An overview of various control benchmarks with a focus on automotive control

Lars ERIKSSON

Vehicular Systems,Department of Electrical Engineering,Link¨oping University,SE-581 83 Linkping,Sweden

Abstract There exists a gap between control theory and control practice, i.e., all control methods suggested by researchers are not implemented in real systems and, on the other hand, many important industrial problems are not studied in the academic research. Benchmark problems can help close this gap and provide many opportunities for members in both the controls theory and application communities. The goal is to survey and give pointers to different general controls and modeling related benchmark problems that can serve as inspiration for future benchmarks and then specifically focus the benchmark coverage on automotive control engineering application.In the paper reflections are given on how different categories of benchmark designers,benchmark solvers and third part users can benefit from providing, solving, and studying benchmark problems. The paper also collects information about several benchmark problems and gives pointers to papers than give more detailed information about different problems that have been presented.

Keywords: Benchmark,automotive control,robust control,robotics,engine control,powertrain control,system identification

1 Introduction

Many texts recognize that there exists a gap between theory and practice in control, i.e., all control methods suggested by researchers are not implemented in real systems and,on the other side,many important industrial problems are not studied in the academic research.This is recognized in, e.g., [1-3] where the need for exchange and balance between theory and practice is expressed.The control practitioners can articulate their needs in benchmark problems to the research community,which can motivate the researchers to tackle problems from real applications,providing a mechanism that“can have a significant impact on increasing the relevance of academic research to engineering practice”.

There is a wide variety of control benchmarks that have been made available [2-47]; e.g., ranging from classical control design challenges,with problems suitable for comparing methods and tools for computer aided control system design, to application areas, such as robotics and automotive applications. Knowledge about and studies of these benchmarks can be beneficial for practicing controls engineers,researchers,and students learning controls as well as the cross fertilization between scientific disciplines.

The goal with this paper is to provide information and pointers to benchmarks where interested scholars can find out and learn more about different control related benchmarks.The focus is on automotive benchmarks as it is the topic of the special issue in which this paper is published. However, since the area encapsulates control,models,and applications there are also benefits to gain from traditional control problems as well as models and other application areas,therefore these will also be mentioned and discussed briefly.

A common theme,and motivation behind the benchmarks is that they were designed to bridge the theorypractice gap in different areas. Many benchmarks have also been designed purposefully so that methodology and theory developers can compare and benchmark their respective methods on relevant and challenging problems. But they also contribute to the technical development in general by increasing the awareness about problems and different approaches to their solutions.

1.1 Contributions and outline

The main contribution of the paper is the collection of benchmark examples that becomes available through the list of references.By studying other benchmarks future benchmark designers can gain inspiration for the benchmark design and evaluation procedures. Furthermore, it also illustrates the usefulness and benefits of constructing and contributing with solutions to benchmark problems.A key benefit for the society is that we can increase the pace of innovation by having the models and benchmarks available.

Section 2 discusses the needs and benefits of benchmark problems for the scientific society as well as the benefits for those who participate in benchmark tests.Section 3 describes some classic control benchmark problems and starts presenting a table that is used to summarize the properties of a collection of benchmarks.Section 4 discusses robotic benchmarks and the subgroup of inverted pendulums. Section 5 discusses benchmark problems related to modeling and system identification and Section 6 discusses several benchmark problems from the automotive control area.

1.2 Def inition of a benchmark

This definition here is an adoption of the definition of the IFAC Technical Committee on Modeling,Identification,and Signal Processing[48],reformulated to fit the context of control.

A benchmark problem is defined as a detailed described reference problem that is used to compare and evaluate the behavior of existing and new control methods,along with a precisely defined evaluation criterion.The system in the problem is preferably provided as a simulation model.The quality of the model and other information should be verifiable by the users.The model should be completed with a detailed description of the system and its usage as well as the model quality.

2 Benef its of benchmark problems

Benchmark problems have several benefits for the communities that they are formulated for. Benchmarks provide an efficient path for introducing control scholars to the application areas as well as introducing application scholars and engineers to the controls subject. To solve industrial and important engineering problems we often need to encapsulate the knowledge from various subject areas and here benchmarks can serve as both a technology bridge and a meeting point for people with different backgrounds and skills.

Systematic control design methods are usually founded on models or direct access to plants, but the latter might be technically difficult or have high costs associated with performing experiments,so models can provide an economic and efficient path for developing control solutions.The modeling of engineering processes often needs a lab for experiments that consumes resources and in addition much engineering and computational time can be spent on the development of models. As a result, benchmark problems in applied areas often provide models for their respective applications. It is therefore, very valuable for the community to get access to models that are made available through challenging problems.

2.1 Why contribute with or participate in benchmarks?

There are several benefits of contributing with or participating in benchmark problems, but how and from what you can benefit depend on the involvement and role. Here follows a short discussion of what benefits different categories can acquire.

Studentsoften contribute as participants. They can then learn how a system works and see the performance specifications in an application area, when analyzing benchmark models they can also study how models are implemented,and understand the fundamentals of the system.Furthermore they have a virtual platform to perform experiments with and test ideas on.Through their work with the benchmark they get hands on experience while learning the problem area and they also get access to the research community seeing both the problems and alternative solutions to a problem they study.

Participating researchersoften contribute as participants.Through the benchmark they get relevant systems to test their newly developed methods on, so they get early results for validation or inspiration for new developments. For example a designer of a method that he believes should test it and compare it with other methods, to evaluate the properties and see the merits compared to other methods, i.e., simply benchmark it.New and interesting solutions to well established and good benchmark problems can be published in scientific journals that are well considered and recognized.Coming up with a good solution to a difficult problem can be very rewarding and give scientific credit.

Benchmark designersResearchers who design benchmarks gets an opportunity to check their problem formulations and the robustness of their models. They also get feedback on their system design and access to solutions that come from outside their box.Constructing a successful and interesting benchmark that many approach, solve, and reuse give an opportunity to gather credit through the problem formulation and receiving citations.

Industry as benchmark solversThe industries who participate and study others benchmarks get an opportunity to study a variety of solutions and methods that can be used as inspiration for solving their own engineering problems. An analysis of variants and competing solutions gives an opportunity for pre-filtering among the possible methods and identify the viable and best performing methods for in their own industrial engineering problems and processes.

Industry as designersIndustry often have access to labs and unique experimental equipment and therefore have a possibility to create interesting and relevant benchmark problems. Those who design the benchmarks get problem solutions prepared for free,through the benchmark evaluations and publications, it can be interpreted as free access to the curious and probing minds of groups of researchers. This can also serve as way of advertising the brand to show that the industry is solving interesting and challenging problems and attract the attention of talented researchers, that can become future employees.

3 Classic control benchmarks

In this section,some classical control benchmarks are collected, these are generic and thus cover many possible application areas. A collection of 13 benchmark problems for control system design was collected over July 1987-July 1990 and written down in a document made available at the IFAC website in the report“Benchmark Problems for Control System Design - Report of the IFAC Theory Committee”,[30].The motivation that was stated was“the committee believed that it would be useful to have a collection of standard problems for comparing the benefits of “new or existing control system design tools”, i.e., currently every new design method is applied to some ad hoc example, and it is difficult to determine a meaningful comparison between existing techniques.”A matrix summarizing the properties of the 13 problems,is also provided in the report.The table presentation is adopted in this paper and it is augmented with other problems collected in this paper. The problems are highlighted in the top section of Table 1 and the interested reader is referred to[30]for the details of the formulations.

Table 1 A summary of some properties of the benchmark problems mentioned in this paper.The number triplet refers to:number of states,number of outputs,and number of control inputs.

Cascade of inverted pendula L,C 20,10,10 Regulation of a“ship’s heading” L,C 3,1,1 Automatic track control of a city bus L,C 5,1,1 Robotic system problem NL,C 12,3,3 Robust control of a“simple process” NL,C n,1,1 Grumman F-14 aircraift L,C 13,1,1 Missile auto pilot,3-variants L,C 13/42/74,4,1 Missile auto pilot,multi-rate system,2-variants L,H 9/26,1,1 Robust two masses and spring L,C 4,1,1 Robust rotating mass and spring damper NL,C 4,1,1 Single link robot L,C 2,1,1 Two link robot NL,C 4,2,2 Acrobot, pendubot, Furuta pendulum NL,C 4,2,1 Throttle control benchmark NL,C 3,1,1 Fuel rail pressure in GDI engines NL,H 3,1,1 Optimal control of diesel electric system NL,C 6,3 PHEV Benchmark NL,H 9,3,8 Look ahead control of a heavy duty truck NL,H 11,4,6 Starting speed control NL,H 37,2,13

Table 1 also outlines and summarizes some of the benchmark problems that are discussed here.In the table the sequence of three numbers represent the number of states,the number of outputs,and the number of control inputs.

3.1 Computer-aided control system design(CACSD)

The IEEE Benchmarks Working Group (BWG) of the Computer-Aided Control System Design(CACSD)Technical Committee,released 3 benchmark problems in the late 80’s. The benchmarks were motivated by “Since about 1980 there has been a rapid development in software for the analysis and design of automatic control systems, with much of the increase being in commercial products. At the present time a software user has a variety of choices that can be made when acquiring software and any unbiased assistance in this task is generally welcome. One way to get such assistance is to have a set of standard problems that can be run on the different software packages under evaluation and this is an approach that is often used by the serious buyer.”First was Grumman F-14 aircraft in 1985,for this problem seven solutions were provided and the problem and the results were discussed and summarized in[43,49].

The second problem is described in [44] as Problem 2, and involved the design of a continuous time missile auto-pilot, for which three models of different complexity were provided having 13,42,and 74 states.Solutions to these problems were discussed and summarized in [44], where a third problem is mentioned.This third problem was later posed in its entirety by[45]and relates to two linear multirate sampled-data models,that were first described by[41].The solutions and results are discussed and summarized in[46].

3.2 Robust and nonlinear control benchmarks

In [12], a linear two mass spring damper system in translational motion is proposed as a benchmark model,and three different control benchmark problems are formulated. In [13] a fourth problem is added to this sequence. The control benchmark tasks are variants of each other that should be robust against different aspects, such as parameter variations in spring stiffness and masses,as well as an external disturbance imposed on one of the masses.

In[11],a benchmark problem with a nonlinear model is developed. It is inspired by the linear model in [13]and involves two interconnected masses and a spring damper where one mass is moving in a rotating system.The benchmark also has a demonstrator test-bed.This is an example of how one benchmark can inspire another benchmark.

4 Robotics and the inverted pendulum

A SISO problem of a robot arm that comes from industry is presented in[50].The SISO benchmark model was experimentally validated and analyzed,showing that the SISO problem can be solved with a PID controller and that it is hard to improve the performance much further.To follow up these results and analyze robotic problems in more detail,[2,15]presents an industrial benchmark problem with the intention to stimulate research in the area of robust control of flexible industrial manipulators.The main difference between the MIMO and SISO is that the realism is increased one step further,including two links,which not only makes the problem multivariable,but also adds some nonlinearities.The aim here was thus to bridge the gap between control theory and practice,and it shows how the problem can adjusted to come closer to the core of the industrial need.

The most common robotic benchmarks are the pendulum based problems that are available in different shapes;Acrobot(Fig.1 left),the Pendubot(Fig.1 right),the Furuta pendulum (Fig. 2), the inverted pendulum,the Reaction Wheel Pendulum (see Fig.2), the bicycle,the VTOL aircraft, the Beam-and-Ball system and the TORA more details about these can easily be found with the aid of search engines.These fall under the family of inverted pendulums as benchmark problem which takes us to the next topic.

Fig. 1 Pendulum examples showing the Acrobot and Pendubot.The name Acrobot derives its name from the gymnast(or Acrobat) hanging on a bar, using his hip action to incur motion.

Fig.2 Illustration of a Furuta pendulum.The motor,M,rotates the disk and the pendulum is hinged to rotate tangentially to the circumference of the rotating disk.

4.1 Inverted pendulums

The inverted pendulum is a robotics related application, that involves interconnected mechanical systems and manipulators. The inverted pendulum and its connection to the development and teaching of control theory is thoroughly discussed in [32], where 150 references are collected. The paper provides an interesting overview of different inverted pendulum systems and the roles that they have played,in particular it“proposes to enhance the wealth of this benchmark and attempt to provide an overall picture of historical,current and trend developments in control theory and robotics based on its simple structure.”The following is an adoption of the discussion in the paper[32].

The evolution of control theory is closely related to education. Therefore, the next generations of control students must receive the scientific and pedagogical supports required to verify conventional techniques,develop new tools and techniques and verify their realization. This can be achieved by integrating, projects that bridge the gap between the acquirement of scientific knowledge and the solution of practical problems into the research laboratories and education curriculums.The inverted pendulum is attractive for education as students can easily relate to and understand its physics and see the result of a stabilizing controller.The inverted pendulum benchmark has, since the 1950s, been used to teach linear feedback control theory to stabilize openloop unstable systems, especially the cart version. This benchmark was considered in many references as a typical root-locus analysis example, and subsequently, it has been used in many books to solve the linear optimal control problem and the complex nonlinear control problem for unstable systems.

In spite of the simplicity of its structure, an inverted pendulum system is a typical nonlinear dynamic system including a stable equilibrium point when the pendulum is at pending position and an unstable equilibrium point when the pendulum is at upright position. When the system is moved up from the pending position to the upright position, the model is strongly nonlinear with the pendulum angle. The interested reader is referred to[32]and its references for more in-depth treatments.

5 Modeling and system identif ication

As discussed,in the introduction and in the definition of control benchmarks, models play an important role in the design and another interesting aspect is that the development of models also have benchmark problems,especially in the system identification area.For example the IFAC Technical Committee on Modeling,Identification, and Signal Processing [48], has developed author and reviewer guidelines for benchmark problems in system identification, that other fields can learn from and adapt to their needs. In addition to the guidelines, the system identification field has worked on collecting datasets that can be used to benchmark system identification methods.A collection of data sets is provided at[35]and a list of hosted nonlinear system data-sets is given below,go to[35]to find a more detailed description of the data-set you are interested in.

1)Coupled electric drives(2017).

2)F-16 ground vibration test(2017).

3)Cascaded tanks system(2016).

4)Wiener-Hammerstein process noise system(2016).

5)Bouc-Wen system(2016).

6)Parallel Wiener-Hammerstein system(2015).

7)Wiener-Hammerstein system(2009).

8)Silverbox(2004).

These data-sets have been used for the IFAC SysID Benchmark Workshops, and the numbers behind the data set titles indicate what year it was used as a benchmark topic. This shows that there is an opportunity to arrange workshops around benchmarks,which can be a good ground for interactions and discussions, as those who are participating share some knowledge about the problem and have ideas about solutions.

Beside the pure system identification benchmarks there have also been modeling and identification benchmarks in application areas for example the paper [5]describes a benchmark for nonlinear identification of automotive engines.

6 Automotive control benchmarks

The automotive industry has been forced to solve serious problems,global warming,air pollution,the energy crisis,and traffic safety.Control technologies have been continuously expanded to develop sustainable solutions to meet these challenges and powertrain controls has thus become highly sophisticated and complex. There is a need for developing reliable solutions with new technologies for the market within a reasonable time time frame and with limited resources. Provided these challenges there is a need for closer collaboration between the academic society and automotive industry.Here, benchmarks can serve the purpose of bridging the gap between the industry and academia. Especially since there are only a few universities that have access to engine dynamometers or vehicle test benches,therefore the benchmarks with the models can be vital testbenches and learning platforms.

There is a range of benchmarks that range from control of components that are crucial for the engine operation or system level benchmarks where several components are included in the problems and need to be handled by the control system for the system.

There has been a sequence of 5 benchmarks arranged under IFAC Technical Committee on Automotive Control (TC 7.1) that have been formulated and arranged before different events.

1)E-CoSM 2009,throttle servo control[36].

2) E-CoSM 2012, fuel optimal control of plug-in hybrid electric vehicle[38].

3) AAC 2016, look-ahead control of a heavy-duty truck[51].

4)E-CoSM 2018,diesel fuel injector control.

5)E-CoSM 2018,control of the common rail pressure in gasoline engines.

Of these,1,4&5 are problems at engine component level while 2 & 3 are at vehicle system level, these will be discussed below,together with other benchmarks at similar levels.

6.1 Engine component benchmarks

The first benchmark at E-CoSM 2009 was a throttle servo control problem [36], that represents an important servo in the engine management system. The problem is a standard DC servo complemented with significant hard nonlinearities such as stick-slip friction and dead-zone.A model was provided to the interested contestants and the solutions were evaluated by the designers at IFP,where one evaluation was performed on a real throttle system hardware.Several solutions were submitted and evaluated at the Conference, the best performing controller was invited to be published in a journal[37].

Two benchmarks were also provided for E-CoSM 2018,one for common rail pressure control of gasoline direct injected engine, and diesel fuel injector control.The common rail pressure control in gasoline direct injection engines,benchmark system has 1 control input,3 states,1 output,as well as 2 measurable exogeneous inputs that act as disturbances.A Matlab/Simulink model is provided for the participants,and the problem with a solution is covered in this special issue of Control Theory and Technology.

6.2 Engine system benchmarks

Beside the IFAC TC 7.1 benchmarks there has also been a sequence of benchmarks that have been organized under the auspices of SICE and JSAE in Japan[3,4,6,7,9,24,40],many of them are attributed to Toyota and Akira Ohata,who has been promoting the application and development of model based control.

The SICE benchmark problem of gasoline engine starting control [4] was proposed to close the gap between the industrial application and the advanced control theory research. More than seven research groups, most from the control community, challenged this problem and after two years from the problem opening,five challenging results are published in several academic conference and journals including IFAC World Congress,Journal of Control Practice, and SAE Journals. A controller resulting from the benchmark has also been validated experimentally on a Toyota engine[6].

The model included in this starting control benchmark has also been used by other authors to study other powertrain control issues and develop solutions to other problems. For example [10] uses the model to design an engine control education system. The system can realize the following functions: it serves to familiarize people with gasoline engine properties and can be applied to carry out engine control simulation, to design engine control logic and to realize engine real-time simulation. This shows that benchmark can reach outside their original scope and serve as learning platforms, to teach students the subjects.

Furthermore the benchmark from [4] is enhanced in [5] with two reduced order models these were designed to facilitate the learning of students who wants to get into the field. This was done since the original problem had been considered to be too detailed and involved for students to grasp the core properties of the system, the general structure was obscured by all the details in the model.

In some applications the optimum performance are attained at the stability limit of operation. Two benchmarks for boundary identification and near boundary control have been presented,to stimulate research and investigations in this direction.The motivation with the models and behind these problems are covered in[3].

Another benchmark, An optimal control benchmark:transient optimization of a diesel-electric powertrain[52], has been designed for testing optimal control software. It lies between component and system level benchmarks and can be seen as a component in a series hybrid vehicle or complete powertrain of a vehicle.This benchmark was designed to bridge the gap between development and testing of optimal control software packages. These are often tested and verified using small problems where the solutions are known and verifiable from theory,while for industrial problems that are more complex and for which there are no theoretical guarantees or known solutions, posing a gap in development and application.To increase the availability of challenging problems a nonlinear model for a diesel engine connected to an electric machine was developed from experimental data and a transient optimal control problem was stated.In the benchmark two solutions are also provided so developers can benchmark against them. The posed problem provides a challenging benchmark for the optimal control software.The model and numerical solutions are available for free at the site pointed out in the paper.

6.3 Powertrain system benchmarks

The second E-CoSM benchmark[38],for the conference in 2012,considered energy management for a plug in hybrid electric vehicle (PHEV). The model provided,described a Chevrolet Volt plug-in hybrid electric vehicle,that is a fairly complex system,having both discrete(Gear,engine on/off)and continuous(electrical machine and combustion engine torques)control variables.In the benchmark the information about distance of the trip was given but no other information was available for the controller. The evaluations were made by the benchmark designers at IFP and the results were presented in two papers,one with the best performing controller[27]and one joint paper with all contributions[22].

The benchmark at AAC 2016 [51] was look-ahead control of a heavy duty truck on open road. A truck model with engine, driveline, chassi, and a road profile was delivered to the benchmark solvers,this model and benchmark is still freely available. This benchmark aimed at developing solutions for problems when the route and road is known beforehand and the controller could plan the driving for minimum fuel consumption.The solutions could use a mix of off-line and on-line computations and the performance of the controllers were evaluated the benchmark designers at LiU.Six solutions were handed in and the results were published in a joint paper[53],it is interesting to note that all six solutions provided were unique, illustrating the complexity and richness of the benchmark problem. Furthermore the complete truck model with driving mission and road slope data is freely available as an open source model,linked to in the publication.

Beside these powertrain control oriented benchmarks there are also benchmarks on Powertrain Verification[29] as well as vehicle dynamics benchmarks. For example the Active Suspension Benchmark [33] and the linear and nonlinear controller design for robust automatic steering[31].

In summary the provided examples of benchmarks show the richness of the benchmarks that are available for students and researchers to approach that enables learning about important application domain problems as well as the capabilities to develop and test new control design methods and tools.

7 Conclusions

Control benchmarks plays the role of extending the controls field by identifying needs and enabling communication opportunities for testing theories and using mixtures of methods,making theory and practice meet.As such they are an important meeting ground for developers of theory and practicing engineers.

In many application areas the experimental equipment is not either readily available or easily accessible for control researchers or it can be the case that experiments are expensive or difficult.There,benchmarks can be an important means for the progression of the research and increase the momentum of research efforts for important and potentially difficult problems.

Control benchmarks can be an inspiration for both new control methods as well as new benchmarks. A goal with the paper has been to provide inspiration for benchmarks by illustrating outcomes and benefits for those that work with benchmarks and also to provide prospective benchmark designers with material where they can learn about how benchmarks can be formulated,arranged,and disseminated.