State Constrained Optimal Control Applied to Supervisory Control in HEVs

State Constrained Optimal Control Applied to Supervisory Control in HEVs — The optimization of the supervisory control of hybrid electric vehicles over predetermined driving cycles has been used as a previous study for determining on-line strategies and also for design and sizing purposes. This problem may be posed as an optimal control problem, in which the energy in the bank of batteries is often the state variable, and the power from any of the system sources is, the control action. As both of these quantities are bounded, the optimal control problem has control constraints or state constraints or both. Usually, the charge-sustaining mode of operation is ensured just by imposing a transversality condition, i.e. a fixed final energy, or including an additional term in the cost functional that penalizes the moving away of the state variable from the nominal value. We considered the problem where the state is allowed to move freely within a band. This led to an optimal control problem with control and state constraints. In this work we describe the difficulties that arise while solving the equations given by the Pontryagin’s Maximum Principle and how these difficulties can be overcome by using the so-called Direct Transcription approach that consists of a programming tool to solve the resultant large-scale finite dimensional optimization problem. Advances in Hybrid Powertrains Évolution des motorisations hybrides IFP International Conference Rencontres Scientifiques de l’IFP 16_ogst09022 18/02/10 11:53 Page 191


INTRODUCTION
The optimization of the supervisory control of hybrid electric vehicles over predetermined driving cycles has been used as a previous study for determining on-line strategies and also for design and sizing purposes.This problem may be posed as an optimal control problem.That has been done in Delprat et al. (2001Delprat et al. ( , 2004) ) and Guzzella and Sciarretta (2005).
Usually, the energy of the bank of batteries is considered the state variable while the power from any of the sources is the control action.As the energy in the batteries and the power from the sources are naturally bounded, these optimal control problems have control constraints or state constraints or both.
In the above references the charge-sustaining mode of operation has been ensured just by imposing a transversality condition, i.e. a fixed final energy in the batteries, equal to the initial energy or close to it.Other authors, such as (Brahma et al., 2000a, b), include an additional term in the cost functional that penalizes the moving away of the state variable from the nominal energy in the batteries.Nevertheless, that may cause a smoothing of the state trajectory which means that the electrical storage system may not be used as exhaustively as it should be.Besides, for certain driving cycles, the energy in the batteries could go beyond its bounds.
Hence, we considered the problem where the state is allowed to move freely, requiring only an upper and a lower bound not to be trespassed.This leads to an optimal control problem with state constraints, a problem whose analytical solution is more involved and that is difficult to translate into an automated algorithm.It has some features of the inventory control problems (Chiang, 1992), since the fuel path can be considered the supplier, the energy in the batteries, the stock and the required power, the demand.However, the reversibility of the electrical path resulting from the possibility of regenerative braking makes it differ from the classical inventory problem, where the demand is always positive.This feature also makes it different from common fuel optimal problems which, also including state constraints, can be found in the bibliography.In this problem the major drawback is that the control and state constraints may become alternatively active and inactive several times during the required driving cycle.Neither the points at which these switchings occur nor the number of them are known.
Several approaches can be chosen to solve this problem.They mainly differ in the way the problem is discretized.The first approach we considered was a complete time discretization together with a quantization of states.This leads to a completely discrete problem that may be solved using dynamic programming (Brahma et al., 2000a, b;Pérez et al., 2006a, b).Alternatively, by performing only the time discretization, it can be posed as a nonlinear programming problem which can be solved by specific software packages.This approach has been called "Direct Transcription" (Ascher and Petzold, 1998;Betts, 1999Betts, , 2001Betts, , 2004;;Pérez and Pilotta, 2007).Finally, the problem can be solved in its continuous form using the optimality conditions of the Pontryagin's Maximum Principle and just performing the discretization to solve the resulting equations at the final stage, using software for differential-algebraic equations (Ascher and Petzold, 1998).
In this work we show the difficulties of obtaining an automated solution by using the latter approach, since the problem led to differential-algebraic equations including inequalities, multipoint boundary conditions and right-handside switchings on unknown points.Software programs to solve these complex problems exist, but many of them require that at least the number of switchings be known, which is not possible in this case.These drawbacks appear even if simplified forms of the problem are considered.We thus show how the direct transcription approach is able to overcome these drawbacks and find a solution.We show that if Euler discretization is used, the direct transcription approach yields the equations of the Euler discretization of the optimality conditions and, hence, both approaches are essentially similar.

Power Flows
In order to consider the supervisory control problem, we used a simplified scheme for the system that represents the vehicle (Brahma et al., 2000a, b;Rizzoni et al., 1999), where the intermediate devices of the powertrain such as rectifiers and converters are replaced by the net power flow from each energy source, the fuel tank and the Electrical Storage System (ESS).Although this simplified scheme may apply for series as well as for parallel hybrids, let us now concentrate on a series configuration.We will use u to indicate the power flow at time t in the fuel tank/engine/generator path (which we shall call fuel-path henceforth, see Fig. 1).

Figure 1
Simplified scheme of a HEV.

Fuel tank Vehicle
We establish the following convention: a positive power flow means power flowing away from the sources towards the vehicle.Consequently, a negative flow will take place in the electrical path during regenerative braking.Besides, the power flow from the fuel tank cannot be negative, as it cannot absorb any power.
The required vehicle velocity profile is considered a given function.The required power can be computed from this profile using a model of the vehicle longitudinal dynamics and will be denoted by r(t) (in the real case, the required power is not known a priori but depends on the transit and road conditions).
The sum of the power from both sources has to be equal to the required power at all times and therefore, the energy flow from the electrical source has to be equal to ru.

Energy from the Sources
Regarding the net energy consumed from each source it must be taken into account that not all the power delivered by the sources can actually be used to supply the demand, since in the intermediate energy conversion processes there are losses.This fact will be represented by means of two functions, f C and f B , that depend on the power flows.We consider them known functions which in practice are obtained by interpolation of a set of values determined by laboratory tests for different power operating points.These functions are normally increasing and nonlinear, particularly f C .
The fuel consumption during a time interval [0, T], where T is known, is represented by the net energy consumed from the fuel source in the interval, i.e.: (1) Our control objective will be minimize this energy or equivalently, maximize the negative of this functional.To compute the net energy in the batteries it has to be considered that the effect of losses implies a power contribution from the batteries greater than that required during acceleration (i.e. for ru > 0), but a power income lower than that produced by regenerative braking (i.e. for ru < 0).Then f B changes at the origin and may not even be smooth at that point.Moreover, the losses in this path will probably also depend on the state, increasing as it deviates from the nominal value, since it is clear that common batteries are less efficient as they get depleted and also when overcharged.The graph for f B used in this work is shown in Figure 2. Thus, the net energy in the batteries at time t is: (2) where x 0 is the initial energy.From Equation (2) we arrive at: (3) This will be considered the state equation with initial condition x 0 .Note that the state equation is nonlinear.
It is not necessary to impose a terminal condition on the state.It may be free, which means that it does not matter what the energy in the batteries is at the end of the cycle (in a charge-depleting operation mode).However, for brevity, we shall limit the exposition to the case of a fixed terminal state x f .If it is equal to the initial state, it represents a "chargesustaining operation" of the vehicle.

Constraints
Clearly, the power flows are physically limited, hence: (4) and (5) In addition, the bank of batteries has to be protected from depletion and from overcharge.This implies that the net energy in the ESS has to be maintained between proper limits.Hence: Summarizing, there are constraints on the control action and bounds on the state variable.

Optimal Control Problem Statement
Find a piecewise continuous control u that maximizes: (7) subject to:

Hamiltonians
The set of constraints is the following: Note that the last two bounds on the control are timedependent.They may be unified by defining: (8) and ( 9) So, the constraint on u becomes: (10) It satisfies the constraint qualification since it is linear in u.For brevity, we will temporarily disregard the upper limit on the state, considering only the lower bound.This may be acceptable for many driving cycles where the energy in the batteries decreases, likely reaching the lower bound, and then recovers, but does not reach the upper limit.Let us then define: (11) So the state constraint is: (12) and also satisfies the constraint qualification.As a first step we will further assume that the state constraint is active in a subinterval [t 1 , t 2 ] and inactive in [0, t 1 ] and [t 2 , T].For this particular problem this means that the batteries get their lowest admissible level of energy at t 1 , remain at that level between t 1 and t 2 and, from t 2 onwards, never reach the lowest limit again.
Unfortunately, this may not be true for many driving cycles, where there may be several binding intervals for the state constraint.
We need to define the Hamiltonian (see [7,10]): (13 where the scalar functions θ -(t) and θ (t) are time-varying Lagrange multipliers.We also need time and space derivatives of G 1 , i.e.: ( to form the augmented Hamiltonian: (15) Here we are using the so-called "indirect approach" to adjoin the state constraint to the Hamiltonian (see [12]).

Finding a Solution
As can be seen, the solution to the above system is not straightforward.Firstly, it depends on the required power.
With the purpose of clarifying the presentation, we shall restrict it to a simple case.Let us neglect the dependence of the functions representing the path efficiency on the state and, further, take f B as equal to the identity function, i.e.: (34) This implies neglecting the electrical losses, since it means that the flow out from the ESS is exactly the necessary difference to satisfy the required power.This approximation may be acceptable, because electrical losses are indeed small with respect to that on the fuel path.We will show that even in this simple case, there are plenty of difficulties obtaining a closed solution from the optimality conditions.
In [0, t 1 ) in order to maximize H and satisfy Equations ( 19) and ( 20 -if θ -= θ = 0 (assuming an interior extreme), it must be: (35 where f C ' indicates the derivatives of f C (.) with respect to its variable.If u can be eliminated from Equation ( 35) in terms of λ, the system (36) where (37) might be solved.Nevertheless, we do not know λ, t 1 and the points where u switches yet.
The situation is simpler in the interval [t 1 , t 2 ] where from Equations ( 28) and (30), it results u = r and then: (38) This means that when the energy in the batteries reaches the lower limit, all the required power has to be provided by the fuel path through the control function u.During this interval, the state remains equal to the lower limit.Note that r being equal to u must satisfy 0 ≤ r ≤ u max .If not, no solution exists.Note also that this imposes conditions on t 2 , which is still not known.
Then, in (t 2 , T] we again have the differential equation conditions we had in [0, t 1 ), but now the boundary conditions are x (t 2 ) = x min , x (T) = x f .
The rest of the optimality conditions say that λ is constant in each of the three intervals, but we still do not know a value for the co-state in any of the intervals, nor μ 0 that determines its jumps.Hence, we face algebraic-differential equations, x r u x x x t x ) 1 2 with right-hand-side switchings and multipoint boundary conditions in unknown points.
It is usual in optimal control with state constraints to use the unconstrained solution to get some information that may be of help in finding the constrained solution.So, let us now assume that the lower bound on the state had also been neglected.In that case we would have got rid of the unknowns in t 1 and t 2 and the optimality conditions reduce to: We still cannot know where u switches among U -, U or the solution of Equation (2.28) in order to be able to solve the state equations, but λ is now constant in the whole interval.Problems of this form, have been addressed in Delprat et al. (2001Delprat et al. ( , 2004) ) and Guzzella and Sciarretta (2005).For certain cycles λ may be found by a dichotomic search.

DIRECT TRANSCRIPTION APPROACH
The above difficulties when using Pontryagin's Maximum Principle optimality conditions led us to address a different approach, usually called a "direct method".This comprises the time discretization of the problem which turns it into a finite dimensional nonlinear constrained optimization problem.Then the solution is found by means of a generalpurpose nonlinear programming code.

Discrete Formulation
Let us divide the interval [0, T] into N stages of length Δt = T/N.Let us define the control function that is being searched for as: (39) Analogously for k = 1, 2, ..., N, let the known required power be represented by: (40) We approximate the integral in Equation (1.7) by the Riemann sum: (41) We will consider the discrete state equation (Euler scheme): (42) As the states depend on the problem unknowns u 1 , ..., u N , we include them as design variables.The constraints on the u k may be expressed in the form: (43) Hence, the optimization problem is posed as follows: Find u 1 , u 2 , ..., u N , x 1 , x 2 , ..., x N -1 that maximize (44) subject to: Hence, the problem becomes a 2N -1 dimensional nonlinear optimization problem with nonlinear constraints.If the final condition is free, x N is also an unknown and the problem size is 2N.As N has to be large for a good approximation, the problem is medium or large-scale.However, the computational complexity order is linear in the number of time steps and in the number of states.This is an advantage over the dynamic programming approach, which is useful if a second electrical storage system such as a bank of ultracapacitors is intended to be added to the vehicle.In the case of dynamic programming approaches, computational complexity increases exponentially with the number of state variables.
To solve this problem we adopted MINOS, which is a well-known Fortran-based computer system designed to solve large-scale nonlinear programming problems.We refer to Murtagh andSaunders (1977, 1978) for details.It uses a projected augmented Lagrangian algorithm.The linear constraints and bounds are treated specially.Now, we will point out some similarities between the direct method we used and the indirect approach that uses the optimality conditions.With this purpose we consider the Lagrangian of the above finite dimensional problem.We leave aside the bounds on the optimization variables (i.e. the inequality constraints in Eq. 46 and Eq.47) because, as stated, they are treated specially by the algorithm used by the code and also to simplify the presentation.Therefore, let: (48) be the Lagrangian of the problem ( 44)-( 45), where λ 1 , λ 2 , ..., λ N are the Lagrange multipliers.The first-order necessary conditions are given by: (49) that is: where ∂f B /∂y k indicates the derivative of f B with respect to the second variable, evaluated at r ku k .Note that Equation ( 52) is the Euler approximation of the state equation and Equation ( 51) is the Euler approximation of the co-state equation backwards in time (see Eq. 16).Hence solving the discretized problem using a Lagrangian algorithm, can be seen as solving the optimality conditions in discrete form.The difficulties arising from the inequality constraints in Equations ( 46) and ( 47) are managed by the special treatment of bounds that the code uses.

EXPERIMENTAL RESULTS
The above method was tested using the parameters of an experimental electric vehicle developed in our laboratory.This prototype was a light-duty vehicle, equipped with a 32 kW electric motor for traction.In previous work (Pérez et al., 2002), the longitudinal dynamics of this prototype was modeled and validated through road tests.This model allows the computation of the power demanded by the vehicle to follow a velocity cycle, including transmission losses and losses from the electrical motor.
The electric storage system is a bank of 20 Yuasa-Exide EV-5 batteries (in series), with nominal charge equal to 197 Amph each.In a first approximation, the voltage was considered constant, equal to its nominal value (U nom = 120 V), then x max = 23.64 kWh.To show results that illustrate the ability of the algorithms to make the state sweep the whole allowed band without trespassing the bounds, hypothetical smaller ESSs were used for the simulations.We indicate in the figures the band imposed between x min and x max .
For the fuel path, a hypothetical fuel converter system, whose efficiency data were taken from [6], was considered.Its maximum power was considered to be 50 kW.
We performed many numerical tests using different duty cycles, discretization steps, discretization schemes, boundary conditions and starting points.We also tested simplified models for the same problem, as described in [18].These simplified models benefit from the facilities of the code, that uses faster algorithms for these cases.In first place, f B may be considered not x, but only power flow-dependent.With this approximation, the problem is also nonlinear but the state equation reduces to a simple integral.If, in addition, f B is taken as the identity function (which, as was said in Section 3, may be a reasonable approximation to begin with), the constraints become linear, and in consequence a linearly constrained problem is obtained.The algorithm that solves these problems is very much faster and converges in many more cases.The third simplification assumes that, in addition, f C is a linear function of the fuel-path power flow.In this case, just a linear programming problem is obtained.For brevity, we only include here the results for the complete problem and the linearly constrained approximation.
The driving cycle used in the figures is the Normalized European Driving Schedule, but reduced to a maximum speed of 60 km/h, which is a design limitation of this vehicle.We adopted Δt = 1 s, since this was the step used in the driving cycle input data.This step may be reduced, but it implies the choice of some interpolation scheme to generate the missing input data in the intermediate points.Note that it also causes the enlargement of the problem size.
In all figures, the top graphs show the required velocity cycle.The central graphs show the optimal control action obtained, along with the required power profile corresponding to the above velocity cycle for this vehicle.The bottom graphs show the corresponding evolution of the state variable within the admissible band.The 1200-sec interval of the cycle was divided into three parts to make the curves visible.Figures 3 to 5 correspond to the free terminal condition case.Figures 6 to 8 correspond to the fixed terminal condition case with x 0 = x f .By comparing the figures for the free and fixed terminal condition cases, it can be seen that the behavior of the optimal control function is similar until the beginning of the boundary interval for the state variable.From that point onwards, the control actions naturally begin to differ, since in the fixed terminal condition case a greater contribution from the fuel path will be required to recover the initial energy value in the batteries.
( , , ..., , , , ..., , , L ( , , ..., , , , ..., , , In Figure 8, the control action shows an oscillating behavior, which means switching the engine on and off.This is clearly not desired.A brief analysis suggests that these oscillations are related neither to the switchings in the control function forced by its time variant bounds U -(t) and U -(t) nor to the boundary interval for the state variable.Thus, they should have a numerical origin.The code allows (and recommends) modifying several parameters such as feasibility and line search tolerance, objective function precision, maximum number of minor and major iterations, penalty parameter in the modified augmented Lagrangian, number of gradient computations and checks, etc. in order to improve the convergence.For the nonlinear cases, one of the first actions suggested is to enlarge the augmented Lagrangian penalty parameter.By doing this, we could improve the objective value from 9.1379 kWh to 9.1169 kWh.In the corresponding solution, the control action not only presents fewer oscillations but also, they are of smaller amplitude and so it does not mean switching the engine on and off.The computation times were also 25% shorter.These results are shown in    still lacking.In the experiments performed, they varied from hours to seconds according to the input specification parameters of the code, the starting point and the approximation used for the function f B .Clearly, the starting point has a major influence.In the example of Figures 3 to 5, the execution time was reduced 95% by using as a starting point the solution obtained for the linear constrained approximation.The choice of the approximation for the function f B is also crucial, since the evaluation of the nonlinear constraints that are determined by f B is the most time-consuming task.In the case of the linearly constrained approximations, instead, just the default specifications of the code needed to be used and the computation times varied between 1.5 and 5 s in all experiments (computations were performed on an Intel Pentium III processor with 384 Mb RAM and 930 MHz speed).

CONCLUSIONS
We found the direct transcription approach was successful in obtaining the solution to this optimal control problem with state control and state inequality constraints.The state trajectories show a wide ride within the bounds imposed, which means a deep use of the ESS against the less efficient    fuel path.The method could overcome the difficulties arising from the optimality conditions when considering inequality constraints in the control as well as in the states, and hence can be used for similar optimal control problems.We found the use of the Euler scheme to discretize the problem very useful for revealing the similarities between the so-called direct and indirect approaches.
We also found that neglecting the electrical losses is a useful first step, to obtain approximated solutions in short computing times.Results from this approximation can be used as initial points for the nonlinear problem.

ACKOWLEDGEMENTS
This work was supported by the Universidad Nacional de Río Cuarto, Consejo Nacional de Investigaciones Científicas y Técnicas, Fondo para la Investigación Científica y Tecnológica and Agencia Nacional de Promoción de Científica y Tecnológica.Results for the linearly constrained approximation, second part.
Figure 14 Results for the linearly constrained approximation, third part.

Figure 2
Figure 2 Net power entering or leaving the ESS as a function of the net power used for traction and of the energy in the ESS.
), we have four possibilities: u may be equal to U -(and θ -= 0); u may be equal to U -(and θ = 0); -θ and θ cannot be simultaneously different from zero (since that implies u = Uand u = U simultaneously); Figure 3Results for the free final state case, first part.

Figure 4
Figure 4Results for the free final state case, second part.

Figure 5
Figure 5Results for the free final state case, third part.

Figure 6
Figure 6Results for the fixed final state case, first part.
Figure 7Results for the fixed final state case, second part.

Figure 8
Figure 8Results for the fixed final state case, third part.

Figure 9
Figure 9Results for the fixed final state case, penalty parameter equal to 5, first part.

Figure 10 Results
Figure 10Results for the fixed final state case, penalty parameter equal to 5, second part.
Figure 11Results for the fixed final state case, penalty parameter equal to 5, third part.

Figure 12
Figure 12Results for the linearly constrained approximation, first part.