1. Introduction
WITH the rapid development of information technology, the unprecedented popularity of smart mobile devices provides a powerful platform for new applications and also brings many novel challenges [1], [2]. As a typical Internet of things (IoT), Internet of vehicles (IoV) can realize ubiquitous information exchanging and content sharing between vehicles with almost no human intervention through its installed sensors and other smart devices [3]. With the help of LTEV2X and the 5G NR-V2X technologies [4], the on-board units (OBU) installed on the vehicles connect with the roadside units (RSU), IoV can provide an informative travel environment for both drivers and passengers [5]. However, compared with cloud computing center (CCC) servers, OBUs usually have lower computing capacity [6], [7]. The main shortcoming of the direct connection between the vehicle and the CCC servers is the long distance communication delay. Due to the backhaul load during peak hours, it is difficult for CCC servers to meet the needs of various delay-sensitive mobile programs [8]. The tension between resource constrained vehicle-mounted terminals and computing intensive applications has become a bottleneck for improving user satisfaction and service quality in IoV. Vehicular edge computing networks (VECNs) are considered as the promising paradigm to solve the above challenges via deploying servers at the edge of the wireless access network, for example the RSUs [9]. In VECNs, the smart devices and users in vehicles can offload computing tasks to the nearby vehicles or the edge servers deployed at the RSUs, both the communication energy consumption and the delay between edge servers and users become lower.
Normally, the moving vehicles on the road, the parked vehicles at the roadside parking lots, and the RSUs can perform as edge nodes to provide communication and computation resources for the vehicles in VECNs [10]–[12]. However, the dynamic topology change of vehicular network makes the effective communication time duration of both vehicle to vehicle (V2V) and vehicle to roadside unit (V2R) extremely short. Moreover, the locations of the parked vehicles and RSUs are usually fixed, the deployment of MEC servers requires a certain amount of space and cost, which limits the capabilities of the edge servers [13].
In recent years, unmanned aerial vehicles (UAVs) have received extensive attention from academia and industry due to their strong mobility and flexible deployment. In [14], the author designed a data collection scheme based on blockchain, that uses UAVs to obtain data from IoT devices. It helps to improve the security of the IoT and the energy consumption is reduced. The author also extended the above scheme to the UAV swarm system in subsequent research [15]. In [16], the author proposed an AI-authorized pandemic monitoring program based on blockchain.
In cooperation with the CCC center, UAVs can operate as a moving edge server, the vehicles on ground can offload part of the computing tasks to the UAVs. In addition, the vehicles can use UAVs as relays, the tasks will be offloaded to other servers (such as cloud computing centers) to complete. For the situation that the UAV is far away from the mission vehicles, a huge energy and time costs exist when the UAV flies to the mission areas directly.
In this paper, we propose a pre-dispatch UAV-assisted VECN system in an urban traffic environment. The UAVs can be assigned to different mission areas with traffic jams in advance. In detail, an optimal UAV flight trajectory algorithm is proposed based on the traffic situation awareness firstly. We formulate an optimization problem to obtain the optimal UAV flight trajectory, while both the flying and turning energy consumptions of UAVs are mainly considered. Genetic algorithm (GA) is used to find the problem solution. Then, for the UAV flight trajectory is obtained based on the predicted values, when the UAV arrives the mission area, it should find the optimal hovering position. We propose an autonomous deployment and energy efficiency optimization strategy based on deep reinforcement learning (DRL), the real time differentiated computation requests are considered. The main contributions are summarized as follows:
We propose a pre-dispatch UAV-assisted VECN system. The CCC center predicts the short-time traffic conditions periodically, and the UAV can fly to the assigned mission areas in advance.
We propose an optimal UAV flight trajectory algorithm based on the traffic situation awareness, while both the flying and turning energy consumption of the UAV are considered.
We propose a real time control strategy to find the optimal hovering position of UAV in each mission area, a DQNbased hovering algorithm is introduced for the practical computation requests of road vehicles are considered.
The rest of this article is organized as follows. Section II summarizes recent research on UAV-enabled MEC networks. Section III introduces the system model including the UAV to road vehicles’channel and task transmission and computation energy consumption models. In Section IV, the problem formulation of this work is proposed. We give the details of the UAV flight trajectory algorithm and the optimal hovering position algorithm. In Section V, simulation results are presented to evaluate the proposed algorithms. Finally, we conclude the paper in Section VI.
II. RELATED WORK
In recent years, UAVs have received extensive attention in wireless communication networks [17]–[19]. For the mobility and deployment flexibility, UAVs can perform as the temporary base stations (BSs) or relay nodes in areas with limited communication, such as emergency rescue after natural disasters and stadiums during sport events [20], [21]. Installed MEC servers on UAV, the UAV-assisted MEC networks have two advantages: 1) Compared with traditional ground base stations, UAVs can dynamically adjust their hovering positions in the air to provide better services, according to the actual environment and mission requirements. 2) It can establish better line-of-sight (LOS) communication with mobile users, and further shorten the transmission distance. In [22], the author analyzed the energy consumption of UAV as a relay node, the system energy efficiency was maximized via the joint transmission power of UAV and BS, the trajectory, acceleration and flight speed of UAV were considered. In [23], the author studied the joint design of computational offloading and resource allocation, as well as the UAV trajectory optimization, to minimize the UAV energy consumption and task completion time. In [24], the UAV performed computing tasks that the tasks had offloaded from the mobile terminal users (TU). The movement of each TU follows the Gauss-Markov random model. A QoS-based action selection strategy was developed based on the dual deep Q network (DQN) algorithm to maximize system rewards.
Thanks to the development of UAV flying and battery storage technologies [25]–[29], modern UAVs are capable of providing powerful computing capacity in complex environments [30]–[32]. Some research for the UAV-assist VECNs had been proposed [33], [34]. In [33], the authors proposed a software-defined networking(SDN)-enabled UAV-assisted vehicular computation offloading optimization framework to minimize the system cost of vehicle computing tasks. In [34], the authors proposed an edge computing architecture for UAV cluster service vehicles, an efficient multi-mode and multi-task offloading scheme is achieved.
However, the aforementioned studies only considered the energy consumption minimization, the communication and computation resource allocation schemes of the UAV after it reaches the mission area. A big assumption is that the UAVs can provide edge services in the vicinity when the computing requests of users or vehicles exist. For the situation that the UAV is far away from the mission area, the users with tasks need to wait for the dispatch center (i.e. CCC servers) to assign the UAV to task requesting areas. Undoubtedly, this will significantly affect the service quality of UAV, and it is not enough to reflect the superiority of UAV flexible scheduling. Specially, when there are multiple computing tasks in more than one mission area requests for an UAV at the same time, the UAV flight scheduling will become more complex. In addition, in order to enable the UAV to access all target mission areas within the minimum flight energy consumption, most of the existing related works treat the UAV path planning process as an approximate traveling salesman problem (TSP) [135], the energy consumption of the UAV flying between two nodes is usually replaced by the Euclidean distance. However, in practical applications, the flight distance is not the only factor that affects the flight energy consumption of UAV. The flight speed, hovering time and flight angle of the UAV will all affect the flight energy consumption. Especially on the crowded urban roads, the UAVs need to fly along the traffic roads, and the flight turning energy consumption needs to considered mainly. [36] considered the flight turning factors when planning the UAV flying path, however, the proposed algorithm just reduces the number of turns simply, the indepth discussion of energy consumption influences caused by the flight turning hasn’t been discussed.
Above all, few works considered the fly turning energy consumption in UAV path planning, and both the energy and time consumptions of UAV traveling from the control center to the mission locations. Compared with the above research, this paper have two innovations: 1) We propose a pre-scheduling UAV-assisted VECN system, which can schedule UAVs to mission locations in advance, reducing user’s waiting time;
SUMMARY OF NOTATIONS AND SYMBOLS.
2) We plan the UAV’s path under the premise of considering the fly turning energy consumption, which reduces the UAV’s non-service energy consumption.
III. SYSTEM MODEL
A. Framework of System
As shown in Fig. 1, we consider an UAV-assisted VECN system in the urban traffic environment. It has a three-layer network architecture, including the CCC server layer, UAV layer, and ground layer with multiple vehicles on the road. The CCC server acts as the central controller to coordinate the operation of the entire system. UAVs will be charged at the charging station and wait for the next mission instruction. A large number of sensing devices and moving vehicles on the road and the RSUs constitute the underlying equipment
of the intelligent transportation system (ITS), the traffic information is collected in the cloud server through the airto- ground (A2G) and backhaul links. We consider that the vehicles moving on the road generate kinds of computing tasks continuously [37]. During off-peak hours, when there are fewer vehicles on the road, the RSUs can provide sufficient computing and communication services for the covered vehicles. However, when the traffic jams occurs on the road, especially in the road intersections, more vehicles computing task requests exist. Combined with the RSU, the UAV can perform as the temporary BS and provide timely computation and communication services for the ground vehicles in different mission areas, through the arrangement and guidance of the CCC center.
The UAV-assisted VECN system operates in a slot-by-slot fashion with fixed length time durations. At the beginning of each slot, the cloud center predicts the traffic conditions of the covered road intersections, via performing data analysis and processing of the collected historical traffic flow data. Historical data is collected through ITS (such as monitoring equipment at intersections). In our work, we use historical traffic data within 7 days to predict the traffic volume of each intersection every 30 minutes in the next day. The data set comes from the competition website DCLab [43], and the specific data is shown in the Table II (the vehicle ID has been coded to avoid leakage of owner information).
Then, the objective road intersections that may with heavy traffic are obtained. The cloud center schedules UAVs to different mission areas based on the flying distance and the battery capacity. The optimal UAVs assignment scheme will not be discussed in this paper. We focus on the scenario that an UAV is scheduled to provide edge services for a set of road intersections. According to the predicted traffic conditions, we obtain the estimated computing task requests in each mission area and an optimal UAV flying trajectory from the staying station and the assigned road intersections is calculated. When a UAV arrives at a new mission area, supported by the ITS, the UAV can obtain the traffic information, such as the number of ground vehicles, and fly to the optimal position to hover to provide communication and computation services for ground vehicles. After completing the task of the current intersection, the UAV will fly to the next task mission area according to the proposed path plan. When the tasks of all road intersections are completed, the UAV flies back to the staying station and
The specific definition of [TeX:] $$\theta_{g, g+1, g+2}.$$
waits for the next task assignment.
B. Energy Model of UAV
In this work, we consider the UAV flying trajectory and hovering position optimization problem, the energy consumption becomes a key factor affecting the QoS of UAV, in which the flight energy consumption accounts for the highest proportion. In order to analyze the flight energy consumption of UAV, it can be further divided into horizontal flight, vertical flight and turning energy consumption. In the existing UAVassisted MEC system, the flight energy optimization of the UAV only considers the horizontal flight and the vertical flight energy consumption, and does not take into consideration the turning energy consumption. Actually, when the UAV flies over a corner, the additional turning energy consumption occurs. In the urban road environment, the UAV flies between the mission areas along the urban roads. Since frequency lifting and lowering operations cause a lot of energy loss, it is reasonable and necessary to consider the turning energy consumption mainly.
In our work, we set that the UAV is flying at a constant speed at a fixed height h. The energy consumption of the UAV flying in a straight line at a fixed speed is denoted as
where [TeX:] $$d_{n, n+1}$$ is the distance of the UAV flying from mission area n to area [TeX:] $$n+1, n \in\{1,2, \cdots, N\} . \text { and } e_{v}$$ is the unit energy consumption per meter when UAV is flying in a straight line at a constant speed v m/s. In addition, the turning energy consumption is denoted as [38]
where [TeX:] $$\theta_{g, g+1, g+2}$$ is the angle of the UAV when it turns, the specific definition is shown as the Fig. 2. [TeX:] $$\eta_{1} \text { and } \eta_{2}$$ are the fitting parameters obtained from experiments [38].
Hovering energy consumption is related to the computing task requirements in the mission area. Since we consider a predispatch UAV-assisted VECN system, the edge service energy consumption of the mission areas are estimated in advance to facilitate the optimal route planning. In the forecasting stage, we set that the task demands of vehicles in a traffic intersection is related to its historical communication and calculation demands. Set [TeX:] $$E_{n}^{\text {server }}$$ as the energy consumption when the UAV is hovering at the mission area n and provides edge services to the road vehicles. [TeX:] $$E_{n}^{\mathrm{comm}} \text { and } E_{n}^{\text {comp }}$$ denote the energy consumptions of processing the historical communication demand and the historical computation demand, [TeX:] $$E_{n}^{\text {hover }}$$ is the hovering energy consumption, we have:
C. Channel Model of UAV
After the UAV arrives the mission area, it provides edge services for the covered vehicles on the road. The communication channel model between the UAV and the ground vehicles is considered. We define [TeX:] $$\mathcal{M}_{n}$$ as the set of vehicles at mission area [TeX:] $$n, \mathcal{M}_{n}=\left\{1,2, \cdots, m, \cdots, M_{n}\right\}.$$ Since the hovering UAV is regarded as a temporary aerial BS, according to the A2G model in [39], the probability of the LOS link between the m-th vehicle and the UAV is denoted as
where [TeX:] $$\alpha \text { and } \beta$$ are constants related to the operation environment, h represents the height of the UAV, and [TeX:] $$r_{m}$$ is the horizontal distance between the m-th vehicle and the UAV. The calculation of [TeX:] $$r^{2} m$$ is denoted as
where [TeX:] $$\left(m_{x}, m_{y}\right)$$ represents the position of the m-th vehicle on the horizontal, and [TeX:] $$\left(u_{x}, u_{y}\right)$$ is the position of the UAV on the horizontal. In addition, the probability of non-line-of-sight (NLOS) communication link is
Taking into account the long-term changes of the channel and the average path loss, the path loss models of LOS and NLOS in the UAV [39] are denoted as
where [TeX:] $$f_{c}$$ is the transmission carrier frequency, [TeX:] $$d_{m}$$ is the straight-line distance between the UAV and the m-th vehicle. Under both the LOS and NLOS models, the average path loss of A2G link can be denoted as
For a given UAV transmit power Pt, the received power of the m-th vehicle depends on the path loss experienced by its communication link [39], which is denoted as
According to (4) and (10), we analyse the probability of LOS and path loss between UAV and vehicles in different environments (i.e., suburbs, cities, dense cities, and highly dense cities) with the elevation angle of the vehicle to the UAV, based on the public data sets [39], [40]. The simulation results
Probability of LoS in different environments.
Path loss in different environments.
are shown as Fig. 3 and Fig. 4. As the density of ground equipment increases, the LOS communication probability between vehicles and UAVs is decreasing gradually. However, when the UAV is flying directly above the vehicle, it can maintain LOS communication, which also reflects the advantage of UAV as a dynamic temporal edge node. On the other hand, as the elevation angle increases, the probability of line-of-sight communication between the UAV and vehicle also increases, and gradually converges to 1. For the communication path loss between the vehicle and UAV. When the horizontal distance between the UAV and vehicle increases, the path loss becomes higher. It is necessary to find the suitable and optimal hovering position of the UAV to provide edge computing servers.
IV. PROBLEM FORMULATION AND SOLUTION
In the proposed UAV-assisted VECN system, under the guidance of the CCC servers, the UAVs can provide temporary edge services for vehicles on congestion road intersections. We consider a scenario that an UAV is assigned to a set of mission areas. An optimal UAV flight trajectory strategy is proposed based on the traffic situational awareness. When the UAV arrives the mission area, an optimal hovering position selection algorithm is proposed for the UAV. In detail, the system operation includes three main problems: The traffic
Traffic flow prediction flow chart.
flow forecasting in the CCC servers, the UAV flight trajectory optimization and the UAV hovering position optimization problems.
A. Traffic Flow Forecasting
The traffic flow prediction flow chart is shown as Fig. 5, the stpdf of traffic flow prediction are mainly divided into three aspects: Data cleaning, feature engineering, and model training and prediction.
Data cleaning is mainly to clean the raw data, such as duplicate data generated by data collection, abnormal data, etc. Feature engineering is mainly to aggregate data, calculate the traffic volume of each intersection every 30 minutes, and carry out relevant feature extraction and combination, such as the extraction of time information (year, month, day, whether it is a working day, etc). Then, we train the model and test the prediction results.
We use the light-GBM algorithm to predict traffic flow, which is an excellent algorithm proposed by Microsoft in 2017 to quickly process large-scale data [41]. The model utilizes the root mean square error (RMSE) as the evaluation standard. RMSE is the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations. In actual measurement, the number of observations is always limited, and the true obtained value can only be replaced by a reliable value. The RMSE is highly sensitive to the huge large or small errors in a group of measurements, so it can well reflect the precision of the measurement. The calculation formula of RMSE is
where [TeX:] $$z_{\text {pred }}$$ represents the predicted value, [TeX:] $$z_{\text {true }}$$ represents the true value, and m represents the number of observations.
After the prediction, we obtain the traffic flow of multiple different road intersections, including the number of vehicles passing through the intersection in a unit time. We consider that the computing task requests of vehicles are generated continuously, the whole task demands are related to the flow of vehicles. For a given threshold of the traffic volume, when the traffic volume at an intersection reaches this threshold, we consider the intersection is in the communication congestion state and mark it as a mission area.
B. UAV Flight Trajectory Optimization
When the CCC assigns a set of mission areas for the UAV, an optimal UAV flight trajectory algorithm is proposed to guide the UAV. Although the battery capacity has been improved, in order to enable UAVs to provide higher-quality computing or offloading services with less energy consumption, the energy consumption of UAVs is worth considering. In our work, we focus on the urban traffic environment, the UAV flies along the urban roads at a fixed altitude, and needs to turn multiple corners. Thus, in the designed algorithm, not only the regular straight flying energy consumption but also the turning energy consumption also be considered. The quality of service (QoS) function of UAV can be defined as
where Q is the UAV flight trajectory, [TeX:] $$E^{Q}$$ represents the flight energy consumption of the UAV under trajectory Q, Since the estimated service demands are obtained through traffic flow prediction, the optimization problem becomes to maximize the QoS of UAV via optimizing the trajectory energy consumption [TeX:] $$E^{Q},$$ is described the following problem P1.
subject to
where in (14), E is the battery storage of the UAV. It is necessary to ensure that the total service energy consumption and flight energy consumption are less than the energy stored by the battery E. And we have
where [TeX:] $$E^{S}$$ denotes the energy consumption of UAV straight fight, that is associated with the flying distance. [TeX:] $$E^{T}$$ denotes the UAV turning energy consumption. The formulations of [TeX:] $$E_{n, n+1}^{v}(d) \text { and } E_{j, j+1, j+2}^{\text {turn }}$$ are shown as (9) and (10). [TeX:] $$\mathcal{J}$$ denotes the set of UAV flying pass corners. Moreover, [TeX:] $$E_{N, 0}^{v}(d)$$ and [TeX:] $$E_{N-1, N, 0}^{\text {turn }}$$ denote the energy consumption when the UAV completes its mission and returns to the staying station.
In the problem P1, the UAV traverses along the road intersections (including the mission road intersections) and then flies back to the staying station, it is a typical TSP problem.
In order to solve this problem, we propose an optimal UAV Flight Trajectory based on the traditional Genetic Algorithm, named as “UAV FT-GA". The description of the UAV FT-GA structure is given in Algorithm 1.
The fitness function of the UAV FT-GA algorithm in this article is related to the energy consumption of the UAV route. In order to minimize the flight energy consumption, we design the following fitness function,
where [TeX:] $$E^{Q}$$ is the straight-line flight energy consumption in the UAV flight path and the corner energy consumption, [TeX:] $$E_{n}^{\text {hover }}$$ is the hovering energy consumption, [TeX:] $$E_{n}^{\text {hover }}$$ is the service energy consumption, and n is the task location number.
In Algorithm 1, we define and initialize each parameter, such as crossover probability [TeX:] $$P_{c},$$ mutation probability [TeX:] $$P_{m},$$ population size S, number of iterations G. Then we create the initial population. Lines 3 to 15 are the main loop part of the genetic algorithm, including selection, crossover, and mutation operations. First, the fitness value of the individual needs to be calculated, then the individual with the higher fitness value is selected as the parent. Parents need to perform crossover and mutation operations to obtain individual offspring. Crossover is the partial exchange of the gene sequences of the two parents according to different crossover operators, and mutation is the replacement operation of partial values of the parent gene sequence. Repeat the above steps until the iteration stops, and output the UAV flying path with the least energy consumption.
C. UAV Hovering Position Optimization
When the UAV arrives at the n-th road intersection, [TeX:] $$\{1,2, \cdots, N\},$$ it needs to find the best hovering position according to the actual traffic volume and mission requirements. Inspired by the maze game on the OpenAI website,
The cover of UAV for a mission area.
DQN-based hovering algorithm framework.
we modeled the optimization problem of the UAV’s optimal hovering position into an optimization model similar to the maze problem [42]. The maze game is a treasure hunting game. The explorer needs to continuously explore in an unknown maze until he finds the treasure. In this article, we assume that the UAV is a treasure hunter, and the best hovering position is the treasure that the UAV needs to find. The nth mission area is divided into [TeX:] $$\rho \times I$$ units, and the UAV is deployed above the unit and can cover the entire unit, as shown in the Fig. 6. Then, the best position deployment problem of the UAV can be modeled as a maze problem, and the existing maze solving method can be used.
Given the location of a ground vehicle, we use [TeX:] $$\mathcal{U}_{n}=\left\{u_{11}, u_{12}, \cdots, u_{i, j}\right\}$$ to represent the set of indicator variables for the distribution of ground vehicles. When there are k vehicles in the unit {i, j}, the indicator variable [TeX:] $$u_{i, j}=k,$$ otherwise it is 0.
According to the above analysis and the proposed environment, we use the deep reinforcement learning method (i.e. DQN) to find the optimal hovering position of the UAV. The core idea is to use the Q function value network as the evaluation module. Based on the value network, we traverse various actions in the current observation state, and the environment interacts in real time. The state, action, reward and punishment values are stored in the playback memory unit, the Q-learning algorithm is used to repeatedly train the value network, and finally the action that can obtain the best value is selected to deploy the UAV. Fig. 7 shows the framework of DQNbased UAV optimal hovering algorithm. Among them, the DQN model is represented as a set.
Under the DQN framework, we define the following vari-
UAV horizontal movement direction diagram.
ables: State space, action space and reward function.
1) State Space: The status [TeX:] $$s_{t}$$ consists of four parts, namely: location and distribution of ground vehicles [TeX:] $$\mathcal{M}_{n}, \mathcal{U}_{n},$$ the position of the UAV at the initial time [TeX:] $$\left(u_{x}^{0}, u_{y}^{0}, h\right)$$ and the position at time [TeX:] $$\mathrm{t}\left(u_{x}^{t}, u_{y}^{t}, h\right).$$
Through the defined state, the DRL agent can make decisions based on the current distribution of ground terminals, the location of the UAV, the calculation power and energy consumption.
2) Action Space: Corresponding to eight directions, we discretize the UAV movement and define eight horizontal movement directions, as shown in Fig. 8. Assuming that the UAV moves a fixed distance for each action, according to the feedback information of the environment, if the UAV does not reach the optimal position after performing the action, the UAV will continue to take corresponding actions until it arrives the optimal position and completes autonomy deployment. The overall action space is
3) Reward Function: The reward function [TeX:] $$r_{t}$$ is defined as
where [TeX:] $$P_{r}^{t}$$ is the total received power of the vehicles covered by the UAV at time [TeX:] $$t, e_{u}^{t}$$ denotes the flight energy consumption of the UAV from the starting point to the selected location at time t. [TeX:] $$s_{p}, s_{e}$$ are used to standardize the vehicle received power and the UAV flight energy consumption related super parameter. When the reward [TeX:] $$r_{t}$$ becomes larger, the vehicles on the road can get better communication services. At the same time, in order to prevent the UAV flying out of the mission area during the service process, a penalty value [TeX:] $$-r_{t}$$ is given and the current exploration is stopped.
The choice of action is related to the instant reward, but our goal is to obtain the largest future reward, so we define the Q function, the initial value of the Q function is 0, and the update process is as follows
DQN-based hovering algorithm for autonomous deployment position of UAV
where [TeX:] $$Q_{\text {now }}\left(s_{t}, a_{t}\right)$$ represents the Q value of current state, [TeX:] $$Q\left(s_{t+1}, a_{t+1}\right)$$ represents the Q value of the next state, [TeX:] $$\lambda$$ represents the learning rate, and [TeX:] $$\lambda$$ represents the decay rate of the reward. The status update process is as follows
Due to the large state space and action space, directly defining the Q table to solve the problems is not effective. We use a neural network to approximate the Q function as [TeX:] $$\left.\left.Q_{(} s_{t}, a_{t}\right) \approx Q_{\left(s_{t}\right.}, a_{t}, \mu\right).$$ Among them, [TeX:] $$\mu$$ is a network parameter. To minimize the gap between the two values, a loss function is defined for optimization, as
where [TeX:] $$y_{t}$$ is the Q value of the target network, and the calculation method is as follows:
After that, we use the gradient descent method to update the [TeX:] $$\mu$$ as follows:
DQN uses two neural networks: the target network and the main network. The target network is used to generate yt, which is the evaluation benchmark for the loss function of the main network. At regular intervals, the parameters of the main network are assigned to the target network. After introducing the target network, in a certain period of training, the target Q value is kept unchanged, which reduces the correlation between the current and the target Q value to a certain extent, and improves the stability of the algorithm.
In order to avoid reaching the local optimal point, we adopt the [TeX:] $$\epsilon-\text { greedy }$$ policy, which is to choose random actions with probability [TeX:] $$\epsilon.$$ Moreover, in order to speed up the DQN algorithm convergence, a step penalty mechanism is also introduced. The DQN also introduces an experience replay memory, which stores the data obtained from the
system exploration environment, the problems of correlation and non-static distribution are solved. It stores the transferred samples obtained from the interaction between the agent and the environment at each time step and stores them in the playback memory network. For a part of the data (minibatch) is randomly taken out with training, the correlation in the samples is disrupted and the stability of the algorithm is improved.
The description of the DQN-based hovering algorithm structure is given in Algorithm 2. Lines 2 to 9 are the iterative process of DQN. From line 4, UAV explores the environment according to the greedy strategy. When the random number is less than the greedy rate, it chooses a direction to explore randomly. Otherwise, it chooses the action [TeX:] $$a_{t}=\underset{a}{\operatorname{argmax}} Q\left(s_{t}, a, \mu_{t}\right).$$ Line 6 is to store the explored data in the experience replay memory. Lines 7 to 8 are to update network parameters. According to the proposed DQN algorithm, the UAV can find the optimal hovering position automatically.
V. NUMERICAL RESULT
In this section, we validate the effectiveness and convergence of the proposed algorithms through numerical simulation. We use python3.7 and TensorFlow2.0 to build a DQL environment, the experimental running system is window10, the CPU used is AMD R5-3600, and the GPU is NVIDA GTX1660Super. We select the real traffic data set from the competition public dataset [43] for simulation experiments. Other relevant parameter settings are shown in Table III.
A. Traffic Flow Forecasting
Fig. 9 is the visualization of the LightGBM training process. In the figure, the horizontal axis is the number of iterations, and the vertical axis is the evaluation function (i.e., RSME). The red curve named valid_score is the RMSE change of the valid set. We can find that the LightGBM algorithm has a good training effect and convergence performance. After about 350 iterations, the proposed LightGBM becomes convergence, the values of RMSE stabilize at around 2.3. Therefore, the LightGBM can be used to perform the traffic flow forecasting effectively, and the predictions of the traffic computing task requests in road intersections are obtained.
LightGBM training process.
Comparison of energy consumption for the proposed UAV FT-GA scheme and the shortness path scheme.
B. UAV FT-GA
Fig. 10 is a comparison diagram of actual energy consumption of the two schemes to obtain the best path by considering the turning energy consumption or not. In the latter comparison scheme, the UAV flies with the shortest path, and the turning energy consumption is not considered. In this simulation example, the coordinate points are randomly generated on the 1000 × 1000 terrain. GA is used to solve the problem P1, and the optimal flight path is obtained. In Fig. 10, the abscissa is the number of road intersections. From the figure, we can find that compared with the scheme without considering the turning energy consumption, the path planning of the UAV can significantly reduce the flight energy consumption in the proposed UAV GA-FT algorithm. The reason is that we consider the UAV flies along the urban roads, and much more corners should be passed. Considering the turning energy consumption, the proposed flight path planning scheme can be consistent with the actual traffic road situations. Fig. 11 is a schematic diagram of the GA iterative process considering the turning energy consumption. As shown in the figure, when the number of iterations reaches about 600, the proposed UAV FT-GA scheme becomes converge, indicating that the scheme has a good convergence performance.
C. DQN-based Hovering Algorithm
When the UAV arrives the mission area, a DQN algorithm for autonomous deployment position of UAV is proposed to find the optimal hovering position of UAV. In order to validate the effectiveness of the proposed DQN-based hovering algorithm, we compare it with other existing algorithms:
Theoretical maximum: In the simulation experiment, we know the environmental data in advance, so we can obtain the theoretical best hovering position in the current map and the theoretical maximum reward value corresponding to the location through detailed calculations.
Center layout method: In this method, the UAV is deployed at the center of the map.
Q-Learning: Q-Learning is a value-based learning algorithm in RL, which is used to find the optimal hovering position of UAV.
Sarsa: SARSA (State-Action-Reward-State-Action) is an algorithm for learning Markov decision process strategies, which is used to find the optimal hovering position of UAV.
Fig. 12 is a simulation diagram of proposed DQN-based hovering algorithm. From the simulation results, we can find that the UAV reward value presents an oscillating changing at the initial moment. When the number of iterations reaches
Rewards of different height schemes.
Rewards of different transmission power schemes.
around 600, the reward value tends to stabilize. This is because the UAV needs to constantly explore the surrounding environment in the early stage. When the enough corresponding data is collected, the UAV can find the optimal hovering position, complete its autonomous deployment, and provide services for the vehicles on the road.
Before the comparison experiment, we used the parameters in Table III to simulate, and the running time of the simulation program is shown in Table IV. Fig. 13 is the comparison of the reward value of our proposed DQN algorithm and other algorithms under different UAV flight heights. We can find that the proposed DQN algorithm can always obtain higher reward values than others, and be close to the theoretical maximum reward value. Q-Learning and Sarsa algorithms can also obtain the same value as DQN, but they are not stable enough compared with DQN.
Fig. 14 is the comparison of the reward value of our proposed DQN algorithm and other algorithms at different UAV transmission powers. Similar to Fig. 13, we can find that the DQN algorithm we proposed has always been able to obtain higher reward values than others , and be close to the theoretical maximum reward value. The main reason is that the proposed DQN algorithm uses two neural network fitting functions. The experience pools are used to store the explored data is also an important reason for improving stability and
better rewards compared to other algorithms.
VI. CONCLUSION
Taking into account the impact of the UAV flying distance between the staying station to the mission areas, we propose a pre-scheduled UAV-assisted VECN system in urban traffic environment. The CCC uses the powerful data processing capabilities to predict traffic conditions of the covered road intersections, and assign UAVs to different predicted mission areas in advance. An optimal UAV flight trajectory strategy is proposed based on the traffic situational awareness. We design an UAV FT-GA algorithm to find the optimal flight trajectory, while the turning and flying energy consumption is considered. Then, when the UAV arrives the mission area, an optimal DQN-based UAV hovering position selection algorithm is proposed. Simulation results show that our proposed method can help the UAV obtain better service quality.