Analyze the optimal solutions of optimization problems by means of fractional gradient based system using VIM

In this paper, a class of Nonlinear Programming problem is modeled with gradient based system of fractional order differential equations in Caputo’s sense. To see the overlap between the equilibrium point of the fractional order dynamic system and the optimal solution of the NLP problem in a longer timespan the Multistage Variational teration Method is applied. The comparisons among the multistage variational iteration method, the variational iteration method and the fourth order Runge-Kutta method in fractional and integer order show that fractional order model and techniques can be seen as an effective and reliable tool for finding optimal solutions of Nonlinear Programming problems.


Introduction
Many problems in modern science and technology are commonly encountered with some class of optimization problems.This is the main reason why optimization is an attractive research area for many scientists in various disciplines.In literature most of efficient methods have been developed for finding the optimal solution of these problems.A detailed and modern discussion for these methods can be found in Luenberger and Sun [1,2].
Gradient based method is one of these approaches for solving NLP problems.The main idea behind the method is to replace optimization problem to a system of ordinary differential equations (ODEs), which is equipped with optimality conditions, for getting optimal solutions of the NLP problem.The gradient based method was introduced by Arrow and Hurwicz [3], Fiacco and Mccormick [4], Yamashita [5] and Botsaris [6].In this sense, the method improved by Brown and Bartholomew-Biggs [7], Evtushenko and Zhadan [8] for equality constrained problems.Schropp [9] and Wang et al. [10] improved gradient based method for nonlinear constrained problem using slack variables and Lagrangian formula.Recently, Jin et al. [11,12], Shikhman and Stein [13] and Özdemir and Evirgen [14] have considered a gradient based method for optimization problems.
The fractional calculus, which is one of the other important research areas of science, has been attracting the attention of many researchers because of its interdisciplinary application and physical meaning, e.g.[15].Most of the studies in this area have mainly focused on developing analytical and numerical methods for solving different kind of fractional differential equations (FDEs) in science.Recently, several methods have been proposed for this aim and applied to different areas, e.g.[16][17][18][19][20][21][22][23].The variational iteration method (VIM) is one of these methods, which was introduced by He [24], and applied to FDEs [25].Momani [26,27], also used VIM for solving some FDEs both linear and nonlinear.Only then, multistage technique is adapted to the VIM for getting the essential behavior of the differential equation system for large time t.This technique was introduced by Batiha et al. [28] for a class of nonlinear system of ODEs and applied to delay differential equations by Gökdogan [29].In recent years, a lot of modifications and developments have been proposed for the variational iteration method.For example, in calculation of the Lagrange multiplier [30][31][32], by using a local fractional operators [33,34] and Laplace transform [35].
In this paper, we construct a fractional gradient based system for solving equality constrained optimization problem.The proposed system shows that the steady state solutions x (t) of the system approximate to the optimal solutions x * of optimization problem on a continuous path as t → ∞.The variational iteration method (VIM) and multistage technique are used for achieving the intended results.
The paper is organized as follows.In Section 2, some basic theory and results, which will be useful subsequently in this paper, are discussed.In Section 3, the MVIM is adapted to the fractional gradient based system for solving optimization problem.The applicability and efficiency of MVIM is illustrated by comparison among VIM and fourth order Runge-Kutta (RK4) method on some test problems, in Section 4. And finally the paper is concluded with a summary in Section 5.

Optimization problem
Consider the nonlinear programming problem with equality constraints: where the feasible set is assumed to be non-empty and is defined by where γ > 0 is a constant and η > 0 is an auxiliary penalty variable.It can be shown that the solutions of the constrained problem (ENLP) are solutions of of the following unconstrained one, under some conditions and when η > 0 is sufficiently large.One of the main results connecting the minimizers of the constrained problem (ENLP) and unconstrained problem (UP) is as follows.
Theorem 1. [1, pp.404]Let {x k } be a sequence generated by the penalty method.Then any limit point of the sequence is a solution to the constrained problem.

Fractional calculus
Now we will give some definitions and properties of the fractional calculus [15].We begin with the Riemann-Liouville definition of the fractional integral of order α > 0.
Several definitions of a fractional derivative such as Riemann-Liouville, Caputo, Grünwald-Letnikov, Weyl, Marchaud and Riesz have been proposed.In the following section we formulate the problem in the Caputo sense, which is defined as: The fractional derivative of f (x) in the Caputo sense with m − 1 < α m, m ∈ N, is defined as where f (m) (.) is the usual integer m order derivative of function f .Note that Riemann-Liouville fractional integral and Caputo fractional derivative satisfy following elementary properties: and

Variational iteration method
To describe the solution procedure for variational iteration method (VIM), we consider the following general nonlinear differential equation where L is a linear operator, N is a nonlinear operator and g (t) is a known analytical function.According to the He's variational iteration method [24,25,36], we can construct a correction functional for (4) as follows, where λ is a general Lagrange multiplier, which can be identified optimally via variational theory, u n is the n−th approximate solution.Here ũn is considered as a restricted variation which means δ ũn = 0.The accuracy of the result fully depends on the identification of Lagrange multiplier and initial condition u 0 .Finally, the exact solution may be obtained as

Fractional gradient based system
Consider the NLP problem with equality constraints defined by (ENLP).Generally, these type of problems are usually solved by transforming to the unconstrained optimization problem (UP).In the next step, some traditional methods or dynamical system approaches are used to get optimal solution of the unconstrained optimization problem.
In this article a fractional gradient based dynamical system approach is handled for obtaining optimal solutions of (ENLP) by the help of MVIM.The fractional derivative is described in the Caputo sense, because the initial conditions have the same physical meanings according to the integer order differential equations.The fractional gradient based approach for solving optimization problems was introduced by Evirgen and Özdemir [37,38].Recently, Khader et al. [39][40][41] used fractional finite difference method and Chebyshev Collocation Method for solving system of FDEs, which are generated by optimization problem.
Utilizing the quadratic penalty function (1) to the equality constrained optimization problem (ENLP) with γ = 2, the gradient based fractional dynamical system can be described by the following form: where ∇ x F (x, η) is the gradient vector of the quadratic penalty function (1) with respect to the x ∈ R n .Definition 3. A point x e is called an equilibrium point of (6) if it satisfies the right hand side of the equation (6).
The gradient based fractional dynamic system (6) can be simplified for the readers' convenience as follows, The stable equilibrium point of the fractional order system (7) is acquired with the MVIM algorithm.The MVIM can be described by some modifications of VIM.To ensure the validity of the approximations of the VIM for large t, we need to treat (5) under arbitrary initial conditions.Therefore, we divide [t 0 , t) interval into subinterval of equal length ∆t as [t 0 , t 1 ) , [t 1 , t 2 ) , ..., [t j−1 , t j = t).
The correction functional for the fractional nonlinear differential equations system (7) according to the MVIM can be approximately constructed as where t * is the left end point of each subinterval, λ i , i = 1, 2, ..., n are general Lagrange multiplier, which can be identified optimally via variational theory, and x1 , x2 , ..., xn denote restricted variations that δ xi = 0. Taking variation with respect to the independent variable x i , i = 1, 2, ..., n with δx i (t * ) = 0, and consequently we get following stationary conditions: Therefore, the Lagrange multipliers can be easily identified as Substituting Lagrange multipliers (9) into the correctional functional (8), we acquire the following MVIM formula for i = 1, 2, ..., n.

Numerical implementation
To illustrate the effectiveness of the MVIM according to the VIM and fourth order Runge-Kutta (RK4) method, some test problems are borrowed from Hock and Schittkowski [42,43].
we convert it to an unconstrained optimization problem with quadratic penalty function (1) for γ = 2, then we have where η ∈ R + , η → ∞ is an auxiliary penalty variable.The corresponding nonlinear system of FDEs from ( 6) is defined as where 0 < α 1.By using the MVIM with auxiliary penalty variable η = 800, step size ∆T = 0.00001 and Lagrange multipliers λ i = −1; the terms of the MVIM solutions for fractional order are acquired by for i = 1, 2. In the Figure 1 and Table 1, we see that the fractional MVIM approach the optimal solutions of optimization problem (11) faster than the other methods.Furthermore, MVIM requires only one iteration to reach the optimal solutions for fractional dynamical system.Contrary to this, MVIM for integer order dynamical system requires two iterations.
Example 2. Consider the nonlinear programming problem [42,Problem No: 79 This is a practical problem whose exact solution is not known, but the expected optimal solution is x * 1 = 1.191127, x * 2 = 1.362603, x * 3 = 1.472818, x * 4 = 1.635017, x * 5 = 1.679081.Following the discussion in Section 3, again we set the quadratic penalty function (1) according to the NLP problem ( 13) where η ∈ R + and η → ∞.The corresponding nonlinear system of FDEs can be obtained by way of ( 6) as follows, where 0 < α 1 is order of fractional rivative.Finally, the MVIM algorithm ( 10) is adapted to the fractional dynamical system (14) with auxiliary penalty variable η = 600, step size ∆T = 0.00001 and Lagrange multipliers λ = −1, i = 1, 2, 3, 4, 5. Tables 2-5 show the approximate solutions for optimization problem (13) obtained by different values of α by using methods VIM, MVIM and RK4.The MVIM for the dynamical system of integer and non-integer order is obtained very close solutions to the expected approximate solutions.Again, it should be noted that the MVIM for fractional order system is used by one iteration to reach optimal solutions.Example 3. Consider the nonlinear programming problem [43,Problem No: 320 This is a practical problem and the exact solution is not known, but the expected optimal solution is x * 1 = 9.395, x * 2 = −0.6846.Firstly, the quadratic penalty function ( 1) is used to get unconstrained optimization problem as follows where η ∈ R + and η → ∞ and so that the nonlinear system of FDEs can be given by where 0 < α 1 is order of fractional derivative.The optimal solutions of problem ( 15) are achieved by using the MVIM iteration formula (10) with auxiliary penalty variable η = 10 6 , step size ∆T = 0.5 × 10 −6 and Lagrange multipliers λ i = −1, i = 1, 2. As we see in the previous examples, the approximate solutions in Table 6 obviously show that the MVIM for fractional order system is more effective than the other methods with low iteration calculation.

Conclusions
The main goal of this work is to create a bridge between two attractive research areas, which are optimization and fractional calculus.In this sense, the intersection point is composed through the instrument of fractional order differential equations (FDEs) system.The system of FDEs is become appropriate to solve the underlying optimization problem by means of optimality conditions.
Furthermore, the variational iteration method (VIM) and multistage strategy are successfully composed to obtain the essential behavior of the system of FDEs, which is generated by nonlinear programming (NLP) problems.The numerical comparisons among the fourth order Runge-Kutta (RK4), the MVIM (α = 1 and α = 0.9) and VIM (α = 0.9) verifies the efficiency of the MVIM as a promising tool for solving NLP problems.

Table 1 .
(11)arison of x(t) for problem(11)between VIM and MVIM with RK4 solutions for different value of α.

Table 6 .
(15)arison of x(t) for problem(15)between VIM and MVIM with RK4 solutions for different value of α.