Optimal 3D UA V Base Station Placement by Considering Autonomous Coverage Hole Detection, Wireless Backhaul and User Demand

Shahriar Abdullah Al-Ahmed , Muhammad Zeeshan Shakir and Syed Ali Raza Zaidi

Abstract

Abstract: The rising number of technological advanced devices making network coverage planning very challenging tasks for network operators. The transmission quality between the transmitter and the end users has to be optimum for the best performance out of any device. Besides, the presence of coverage hole is also an ongoing issue for operators which cannot be ignored throughout the whole operational stage. Any coverage hole in network operators’ coverage region will hamper the communication applications and degrade the reputation of the operator’s services. Presently, there are techniques to detect coverage holes such as drive test or minimization of drive test. However, these approaches have many limitations. The extreme costs, outdated information about the radio environment and high time consumption do not allow to meet the requirement competently. To overcome these problems, we take advantage of Unmanned aerial vehicle (UAV) and Q-learning to autonomously detect coverage hole in a given area and then deploy UAV based base station (UAV-BS) by considering wireless back haul with the core network and users demand. This machine learning mechanism will help the UAV to eliminate human-in-the-loop (HiTL) model. Later, we formulate an optimisation problem for 3D UAV-BS placement at various angular positions to maximise the number of users associated with the UAV-BS. In summary, we have illustrated a cost-effective as well as time saving approach of detecting coverage hole and providing on-demand coverage in this article.

Keywords: 6g , autonomous coverage hole detection , minimization of drive test (MDT) , Q-learning , UAV-BS , unmanned aerial vehicle(UAV) , wirelessbackhaul

1. Introduction

TECHNOLOGICAL advancement and transformation are taking place in every sector of our daily life. One of the sparkling examples of such advancement can be given as today’s diverse communication services to offer global connectivity. These services can be mentioned as human-centric (e.g., mobile devices and complex applications), machine to machine (e.g., autonomous vehicles, artificial intelligence based machines) and human to machine (e.g., telesurgery, immersive virtual reality and so on) [1]. As a consequence, the next generation self- organizing networks (SON) such as sixth generation (6G) need to provide more extensive coverage with high capacity while maintaining low latency everywhere [2].

In favour of providing optimal communication services to the technologically advanced end users, network operators must ensure that there is no coverage hole or poor services. Densification of the base station (BS) is a crucial part of coverage planning for 6G cellular networks. The densification of the BS is not very straight forward for the network operators as there are many environmental situations or many complex parameters exist. Operators also need to identify where and how to densify while utilising the best of their resources [3]. Any coverage hole will hamper the services to the end users as well as the reputation of the operators. Therefore, coverage holes cannot be ignored from the deployment stage to the whole operational stage.

One of the major factors for developing coverage hole is the current deployment methods. In the pre-deployment phase, the mobile operators carry out the analysis of the potential traffic and drive testing of the network sporadically. Consequently, as the infrastructure evolves over time, it faces blockages due to obstacles or many other factors. As a result, the propagated signal deteriorates which leads to very poor received signal. Moreover, the modern cellular networks have very complex design space, densification, agile frequency reuse, split functional planes etc. Therefore, manual and blind adaptation of parameters is difficult warranting self-x features such as self-configuration and diagnostics.

A convenient solution for these concerns can be the utilisation of the mobile robotic platforms. These platforms have become less expensive and can be employed for periodic monitoring of infrastructure. They can also augment existing infrastructure to temporally alleviate severe coverage and capacity issues. Unmanned aerial vehicle (UAV) is a sparkling example of mobile robotic platforms. UAV or commonly known as drones, has many advantages to offer for instance high mobility (it can reach anywhere), provide low cost, and flexibility. They are primarily used for military surveillance which now can be used by the public for many applications, for example, medical supplies, rescue operations, forest fire detection, emergency search, weather monitoring, communication relaying, and so forth [4], [5]. Another advantage of this versatile UAV can be added as assisting to detect coverage hole for the next-generation 6G cellular networks. Consequently, this work focuses on coverage hole detection and providing temporary on-demand coverage by deploying UAV based base station (UAV-BS).

Fig. 1.
Graphical representation of UAV detecting coverage hole and serving user as a UAV-BS after discovering the coverage hole.
A. BACKGROUND

While current cellular networks have envisioned exponential proliferation, several suburban and rural areas still lack connectivity. The Scottish Government and Scottish Futures Trust are working together in order to develop the Scottish 4G Infill Programme [6]. The main objective of the project is to provide long term evolution (LTE) infrastructure and services to around 50-60 complete coverage holes in Scotland. Germany’s 4G coverage is the worst in Europe [7] and ‘dead zones’ are showed by mapping them [8]. In Japan, mobile operators have taken initiative to cover the rural dead zones by 2024 [9].

The optimal communication services include secure connectivity of data and voice. Such reliable key performance indicator (KPI) requires ubiquitous coverage at a certain desired quality of service (QoS) level. Keeping this KPI in mind, the nextgeneration cellular networks will face challenges to meet the demand of current data-hungry services and applications. First, the operators will need to find out where to deploy BS based on the users’ activity and for businesses to generate revenue. Second, the network will become more complex which will increase the capital expenditure (CAPEX) and operating expenditure (OPEX). Sometimes the BS might not work efficiently or breakdown due to many factors, for example, technical fault or disaster. As a result, the coverage will become limited and coverage hole will appear which lead to the degradation of QoS to the end users.

Detecting a coverage hole is a crucial issue to maintain the link reliability of a cell [10]. Classically, an expensive drive test along with the propagation model is used to construct a radio map in order to detect a coverage hole [11]. This radio map is constructed intermittently and always not precise which leads to more complexities such as increased OPEX. Therefore, the 3rd generation partnership project (3GPP) introduced the minimization of drive test (MDT) as a part of the SON to replace the drive test [12]. In the MDT mechanism, the BS has access to the user equipment (UE) measurement report which is called radio link failure (RLF) report if there is any handover issues [13]. Thus, it became easier than before to measure the network performance. Having said that, this report has few limitations such as UEs are unable to give an immediate report at all the time and UE may not provide accurate geographical location due to many factors such as blockages [14], [15]. Besides, the report may be inaccurate or outdated. Another important challenge can be mentioned as external intervention on UE operation. Limited UE battery power, location services may not be available at all times which are managed by the users or even the users might not give consent to the operators to collect information from the devices according to general data protection regulation (GDPR).

In these circumstances, autonomous mobile platforms such as quadcopters or UAVs can assist in detecting coverage holes very efficiently and cost-effectively. Such platforms can be exploited (subject to flying regulations) to assist autonomous network drive testing. The machine learning algorithms will make the platform autonomous where reinforcement learning (RL) will help the platforms to operate under any unknown environment. After detecting the coverage hole, the network operators will decide whether to deploy a fixed BS or to deploy a UAV-BS for temporary on-demand coverage depending on the users’ activity. Furthermore, the operators will have the opportunity to find the optimal place to provide coverage with the best KPIs such as user demand and wireless backhaul.

Being inspired by the above scenarios, we examined the current research related to UAVs and coverage hole detection. As the research of UAV is very trendy in recent years, we are going to mention some of them here which are related to our work. The authors in [11] proposed a deterministic propagation model with coverage hole detection algorithm by constructing a radio map. However, the map may not be accurate as few numbers of RLF report has been considered. The authors in [12] have proposed an automated coverage hole detection technique by constructing a radio environment map with the help of many reports and measurements which is very time consuming. A path-loss model has been created to detect coverage hole in high altitude and tested with the help of UAV measurement data in [15], but UAV has not been utilised to detect any coverage hole. Another related research has been carried out in [16] where authors proposed the use of RL (SARSA) with the aid of value function approximation (VFA) to provide coverage and capacity and compared the result of using of Q-Learning and SARSA. However, the results were not differed much as well as there was no planning of UAV navigation. The authors in [17]–[21] have pre sented the work based on the autonomous navigation of UAVs but not considered to use UAV as a user to detect the mobile signal strength. Additionally, the authors in [22]–[29] have discussed about the efficient 3D aerial BS placement while considering several UAVs, backhaul, an optical link, users’ mobility, millimeter wave (mmWave) and energy constraints with or without machine learning algorithm. Despite all these recent research, UAVs have not been utilised with RL for coverage hole detection. Hence, this article is the first in addressing coverage hole detection with a combination of RL and UAV, to the best of the authors’ knowledge.

B. Our Contributions

The main contribution of this article is in two folds:

We exploit the RL as a trajectory planner to detect coverage hole efficiently. Here, the RL is a suitable approach as the environment is unknown to the UAV completely. UAV will be able to learn and detect coverage hole autonomously from the start position. After finding the coverage hole, the UAV will also determine the available wireless backhaul data rate in several angular positions based on heights.

We present an efficient deployment of on-demand UAV-BS to serve the maximum numbers of users under its coverage area. The similar UAV will work as a flying BS or UAV-BS and it will place itself in an optimal position from the available information (i.e., backhaul data rate, bandwidth) to serve users.

The rest of the article is organized as follows. In Section II, the problem formulation and system model is presented. Then, we discuss our proposed method in Section III. Section IV discusses the simulation results. In the end, we conclude in Section V.

II. SYSTEM MODEL AND PROBLEM FORMULATION

Coverage hole forms when the signal-to-interference-plusnoise ratio (SINR) from any cell or neighbour cell of an area falls below a threshold level required to maintain standard QoS. It can be mathematically defined as area A = x in [TeX:] $$R^{2}$$ where [TeX:] $$\operatorname{SINR}(x)<\gamma_{t h}.$$ This phenomenon occurs for many reasons for example, attenuation due to enormous physical obstacles (such as hills, large buildings) while signal propagates, improper antenna parameters, fault in hardware or improper RF planning [30]. Therefore, when a UE enters into a coverage hole of recent technology with no legacy technology [2G/3G/4G/5G], the user may experience call drop and poor communication services. Alternatively, the UE might experience an audio gap or insufficient service because of the bad air interface. On the other hand, if the UE enters into a coverage hole with legacy technology, then the user will have negative experiences for example, low speed or low latency [31].

In this article, we consider the downlink of the mobile network that serves a range of users distributed in a suburban geographical area with a set of macro BSs distributed according spatial Poisson Point Process. All the BSs use the same frequency with omnidirectional antennas. Small cell BSs has not been taken into consideration as suburban/rural areas are mainly covered with macro BS. This network deployment scenario has one UAV which will act as a user in order to detect the coverage hole. The user association for load balancing in BS has been ignored as we are detecting the coverage hole only. We have considered two different path-loss models for our system model for two scenarios. They are discussed below.

A. Cellular to UAV Path-loss Model

Cellular to UAV path-loss model is required in order to calculate the received signal from BS to various UAV positions which will assist to detect coverage hole and to determine available data rate. Typical terrestrial path-loss models are incompatible for airborne systems [32]. In many cases, airborne coverage appears to be better than terrestrial coverage [33]. Hence, an accessible path-loss model dependent on the depression angles has been considered. In this model, there is an excess path-loss in addition to terrestrial path-loss. So, the path-loss PL from the serving BS to UAV can be expressed according to [33] as:

(1)
[TeX:] $$\begin{aligned} P L(d, \theta)=& 10 \alpha_{t e r r} \log (d)+\mathcal{A}\left(\theta-\theta_{o}\right) \exp \left(-\frac{\theta-\theta_{o}}{\mathcal{B}}\right) \\ &+\eta_{0}+\mathcal{N}\left(0, a \theta+\sigma_{o}\right) . \quad[\mathrm{dB}] \end{aligned}$$

In above expression, d is the 3-D terrestrial distance from BS with height [TeX:] $$h_{B S}$$ and UAV with height [TeX:] $$h_{U A V}$$ can be written as [TeX:] $$d=\sqrt{l^{2}+\left(h_{B S}-h_{U A V}\right)^{2}}.$$ Here, l is the 2-D ground distance between the UAV at position [TeX:] $$\left(x_{U A V}, y_{U A V}\right)$$ pnd BS at position [TeX:] $$\left(x_{B S}, y_{B S}\right)$$ can be written as [TeX:] $$l= \sqrt{\left(x_{U A V}-x_{B S}\right)^{2}+\left(y_{U A V}-y_{B S}\right)^{2}}.$$ The parameter [TeX:] $$\alpha_{t e r r}$$ is the path-loss exponent depends on propagation environment such as urban, suburban, rural. The depression angle can be calculated as [TeX:] $$\theta=\arctan \left(\frac{h_{U A V}-h_{B S}}{l}\right)$$ with the estimation range [TeX:] $$\left\{-3^{\circ}, 10^{\circ}\right\}, \mathcal{N}\left(0, a \theta+\sigma_{o}\right)$$ is an angle dependent standard deviation serving as the shadowing component with fitting parameters [TeX:] $$a \text { and } \sigma_{o}.$$ All the fitting parameters [TeX:] $$\left(\mathcal{A}, \mathcal{B}, \theta_{o}, \eta_{o}, a, \sigma_{o}\right)$$ can be found on the Table 1. The same pathloss model also applies for the received signal from interfering BS to UAV. It is important to note that the authors in [33] discarded any aerial measurements below 5 m as it follows terrestrial propagation solely. Therefore, the terrestrial path-loss or log-distance path-loss model has been considered while detecting coverage hole at user height. Other than that, the expression (1) has been taken into account for any cellular to UAV propagation above 5 m.

B. Air-to-ground Path-loss Model

For the UAV to operate as UAV-BS for any coverage hole, we need to consider an air-to-ground path-loss model. Such pathloss model is required because the propagation characteristics from UAV-BS to ground users are different than cellular to UAV. There are several numbers of air to ground path-loss model has been studied. We have adopted one of the path-loss models which have been presented in [34] and [35]. In that study, there are two major propagation types which are line-of-sight (LoS) receivers and non-line-of-sight (NLoS) receivers. The probability of LoS path-loss model is dependent on the environment (i.e., urban, suburban, rural and so on.) and can be formulated according to [34] and [35] as follows:

(2)
[TeX:] $$P(\operatorname{LoS})=\frac{1}{1+\vartheta \exp \left(-\xi\left(\frac{180}{\pi}\right) \phi-\vartheta\right)},$$

where [TeX:] $$\vartheta \text { and } \xi$$ are the constant values which depends on the environment and [TeX:] $$\phi$$ is the the elevation angle equivalent to [TeX:] $$\arctan \left(h_{U A V} / I\right)$$ where [TeX:] $$h_{U A V}$$ and I parameters represent the altitude of a UAV-BS ranging from 30 m to 1000 m and its horizontal distance from the UE, respect pively. This horizontal distance can be calculated as [TeX:] $$I= \sqrt{\left(x_{U E}-x_{U A V}\right)^{2}+\left(y_{U E}-y_{U A V}\right)^{2}}$$ in a Cartesian coordinate system. Also, this model overlooks the shadowing and presents the average path-loss as:

(3)
[TeX:] $$\operatorname{PL}(d B)=10 \log \left(\frac{4 \pi f_{c} d_{s}}{c}\right)^{\delta}+P(\operatorname{LoS}) \eta_{L o S}+P(\mathrm{NLoS}) \eta_{N \operatorname{LoS}},$$

the first term of (3) denotes free space path-loss (FSPL) as per Friis equation that depends on carrier frequency [TeX:] $$f_{c},$$ speed of light c, path-loss exponent [TeX:] $$\delta$$ and the distance [TeX:] $$d_{s}$$ between the UAV-BS and UE, where [TeX:] $$\eta_{L o S} \text { and } \eta_{N L o s}$$ denotes the average further losses for LoS and NLoS connections, respectively and P(NLoS) = 1 - P(LoS). The parameters in (3) are dependent on the environment.

C. Coverage Hole Detection and Optimisation Problems

Based on the cellular to UAV path-loss model (1), we can calculate the received power and SINR at [TeX:] $$\left(x_{U A V}, y_{U A V}\right)$$ from [36] when the serving cell is a macro BS:

(4)
[TeX:] $$\operatorname{SINR}_{i, k}=\frac{P_{i, k}^{r x}}{\sum_{m=1, m \neq i}^{M} P_{m, k}^{r x}+N},$$

where [TeX:] $$P_{i, k}^{r x}=P_{i, k}^{t x}-P L(d, \theta), P_{i, k}^{r x}$$ represents received signal power from i-th serving BS on subcarrier k at the UAV at any position [TeX:] $$\left(x_{U A V}, y_{U A V}\right), P_{i, k}^{t x}$$ refers to the transmit signal power of i-th serving BS on subcarrier k and [TeX:] $$P L(d, \theta)$$ denotes path loss (PL) between the BS and UAV at any given position. Additionally, [TeX:] $$\sum_{m=1, m \neq i}^{M} P_{m, k}^{r x}$$ represents the sum of the individual interfering signal power received from interfering BS at the UAV at any given position. It is to be pointed out that the channel gain has been ignored here. Variable N implies thermal noise power. SINR level more than 7 dB is assumed as fair to excellent signal, but the SINR level below 7 dB is considered as weak signal and may cause a coverage hole [37].

Once the coverage hole is detected, an efficient angular place is required for the UAV to work as UAV-BS. Therefore, let us consider the transmission of UE data between the UE and BS via UAV-BS. We consider a stochastic geometry approach for users distributed in a spatial Poisson process in a square area where the coverage hole exists. The users are running different applications that require different data rates and bandwidths. There are several limitations while associating UE with BS via UAV-BS. The wireless link running from the BS to the UAV-BS limits the maximum data rate at different depression angles. Hence, we have a different wireless backhaul data rate of [TeX:] $$R_{i}$$ Mbps in that position:

(5)
[TeX:] $$\sum_{i=1}^{N_{\text {uer }}} r_{i} \cdot Z_{i} \leq R_{i}.$$

The rate [TeX:] $$R_{i}$$ can be calculated according to Shannon-Hartley theorem in above equation which is [TeX:] $$R_{i}=B \times \log _{2}\left(1+\operatorname{SINR}_{i, k}\right)$$ where B is the total available bandwidth at UAV. Also, [TeX:] $$N_{\text {use }}$$ denotes the total number of users that are in the coverage hole area, [TeX:] $$r_{i}$$ represents the data rate required by the user and [TeX:] $$Z_{i}$$ denotes the user indicator function which is defined as:

(6)
[TeX:] $$Z_{i}=\left\{\begin{array}{ll} 1, & \text { if user } i \text { is connected with UAV-BS } \\ 0, & \text { otherwise. } \end{array}\right.$$

Another restriction can be mentioned as the bandwidth. It is formulated as follows:

(7)
[TeX:] $$\sum_{i=1}^{N_{\text {user }}} b_{i} \cdot Z_{i} \leq B,$$

where [TeX:] $$b_{i}$$ stands for the bandwidth required by the i-th UE which can be calculated as [TeX:] $$b_{i}=r_{i} / v_{i} \text { where } v_{i}=\log _{2}\left(1+\text { SINR }_{\text {user }}\right)$$ denotes the spectral efficiency and [TeX:] $$\text { SINR }_{\text {usen }}$$ denotes SINR ratio at i-th UE when UAV-BS is serving.

We also had to identify that a user is in the coverage of the UAV-BS if its satisfy the QoS requirement. It is formulated as:

(8)
[TeX:] $$P L_{i} \cdot Z_{i} \leq P L_{\max },$$

where path-loss [TeX:] $$P L_{i}$$ is the path-loss from UAV-BS to user i and [TeX:] $$P L_{\max }$$ is the maximum path-loss that is allowed before the user i falls out of the coverage. By considering all the above mentioned limitations, we formulate the problem as follows:

(9)
[TeX:] $$\max _{\theta,\left\{Z_{i}\right\}} \sum_{i=1}^{N_{\text {user }}} Z_{i},$$

subject to:

(10)
[TeX:] $$\sum_{i=1}^{N_{\text {user }}} r_{i} \cdot Z_{i} \leq R_{i},$$

(11)
[TeX:] $$\sum_{i=1}^{N_{\text {user }}} b_{i} \cdot Z_{i} \leq B,$$

(12)
[TeX:] $$P L_{i} \cdot Z_{i} \leq P L_{\max }, \quad \forall i,$$

(13)
[TeX:] $$\theta_{\min } \leq \theta \leq \theta_{\max },$$

(14)
[TeX:] $$\sum_{i=1}^{N_{\text {uer }}} Z_{i} \leq L_{n},$$

where are the 3D coordinates of the UAV-BS placement. The parameters [TeX:] $$\theta_{\min } \text { and } \theta_{\max }$$ are the range of the depression angle of UAV-BS from the BS. Also, constraint (14) indicates that the UAV-BS can maintain the highest number link of [TeX:] $$L_{n}$$ with users which is more specifically the capacity of the UAV-BS.

III. PROPOSED METHODS

This section presents the concept of using RL to detect coverage hole and then optimise service to the users.

Coverage hole discovery and UAV-BS placement
A. Autonomous Coverage Hole Detection Algorithm

Based on the above scenario to detect coverage hole, we employ the RL in an UAV to utilise at it best. In RL, an agent or a system is given an environment to explore and to learn by itself while taking actions in different states with the association of a reward mechanism [38]. Reward mechanism determines how good or bad the action is. The main target for the agent is to collect the maximum reward.

The sensible algorithm we employ here is called Q-Learning for a model-free environment where ‘Q’ stands for quality [38], [39]. This part of the algorithm is to find the 2D or [TeX:] $$\left(x_{U A V}, y_{U A V}\right)$$ position by maximizing the reward. The Q information information table for this problem has the following factors:

(15)
[TeX:] $$Q(S, A) \leftarrow Q(S, A)+\alpha_{Q}\left[\Gamma_{t+1}+\gamma \max Q^{\prime}\left(S^{\prime}, A^{\prime}\right)-Q(S, A)\right],$$

where [TeX:] $$S \text { and } S^{\prime}$$ are the state of the agent as well as [TeX:] $$A \text { and } A^{\prime}$$ are the available actions that the agent picks at time t and t + 1 respectively. Again, [TeX:] $$\alpha_{Q}$$ is the learning rate at [TeX:] $$\left(0<\alpha_{Q}<\right)$$ that decreases across the learning trials for the convergence of the process. The discount factor [TeX:] $$\gamma \text { at }(0<\gamma<1)$$ defines the precision and speed of the convergence of the process. This algorithm also requires an epsilon-greedy policy which is a method of selecting random actions with uniform distribution for a set of available actions by picking exploration or exploitation strategy. In the exploration learning process, the agent can ignore the successful results at the local level. On the other hand, exploitation takes place when the agent performs the best actions on the basis of its knowledge [40].

A graphical illustration of our proposed method is presented in Fig. 1. We select an area of 15 km15 km and then create a 2020 grid on top of the area. The grid size can be changed for more accurate results if required. In Fig. 2, a snapshot from our simulator is provided. Here, circles graphically represent the BS, the UAV is shown as rectangle and the coverage hole is depicted with the aid of white rectangle. An UAV takes off from to top left corner which is the initial point (home) of the UAV. The initial point can be set in anywhere in the scenario as per requirement.

In each episode of the task, the UAV will start from the initial point and then traversing across the grid randomly by taking actions (left, right, up and down) depending on the epsilon greedy policy in Q-learning and calculate the SINR and SINR related parameters. The SINR value serves as an input to the reward mechanism. The search continues until the UAV finds a reward. If it gets any positive reward which is ‘1’ in our case, it will mark that area as a white rectangle and stores the coordinate of that place. This marked white rectangle will indicate the detected coverage hole where the UAV will act as UAV-BS in order to provide coverage. Another episode will start after following the Q-table update method that will do the same operation. Algorithm 1 explains all these steps that are taken by the UAV. It should be pointed out that the algorithm will converge to an optimal policy over the iterations which will allow to achieve the reward in the shortest path. This shortest path will contribute to energy savings.

B. UAV-BS and User Association Optimisation Algorithm

Here, the algorithm we propose for optimisation is a centralised solution to find the best 3D placement of a UAV-BS. In this algorithm, UAV-BS will be deployed in different frequency as an on-demand basis and serve the maximum users under its coverage. The algorithm is divided into four steps where it considers some constraints to optimise the problem in every step.

1. Every user has to be covered by the UAV-BS and therefore, the user must tolerate the maximum path-loss before going outage. At this step, the algorithm considers the constraint (12). Then, UAV-BS receives a number of association requests from the users i which are in the coverage region of

Fig. 2.
Coverage hole detection simulation exploiting RL.

the UAV-BS. The UAV-BS will check by changing with the maximum number of association requests until it is full of capacity.

2. The UAV-BS selects the maximum number of [TeX:] $$L_{n}$$ association requests out of all the requests by meeting constraint (14). The criteria of the selection is the minimum sum of data rate [TeX:] $$r_{i}$$ and bandwidth [TeX:] $$b_{i}$$ which will assist to serve maximum number of users according to constraint (9) (10) and (11). Therefore, the UAV-BS needs to select the users based on min [TeX:] $$\left(r_{i}+b_{i}\right)$$ and then the next user in higher or ascending [TeX:] $$\left(r_{i}+b_{i}\right)$$ out of the association request list [TeX:] $$L_{n}.$$ The UAV-BS monitor the number of links and associated data rate and bandwidth using counters [TeX:] $$C_{L_{n}}, C_{r} \text { and } C_{b},$$ respectively. In addition, UAV-BS ensure that it stays in the maximum available data rate, bandwidth limit and links and meets the criteria of constraints (10), (11) and (14) (i.e., [TeX:] $$C_{r}+r_{i} \leq R_{i}, C_{b}+b_{i} \leq B \text { and } \left.C_{L_{n}}+1 \leq L_{n}\right)$$ before associating the users i with UAV-BS. After association, all the counters need to be updated. This process will complete if the UAV-BS reaches the maximum number of links or maximum available data rate or bandwidth.

3. In this step, if the UAV-BS receives new association requests and if it has more capacity to serve users, then it will check the list of available wireless backhaul data rate [TeX:] $$R_{i}$$ in different and then move accordingly to that if possible.

4. This process comes into place only that time when the wireless backhaul data rate [TeX:] $$R_{i}$$ or available bandwidth B limit is reached, UAV-BS is in the where the maximum data rate and a new request for association take place (i.e., [TeX:] $$\left.C_{r}=R_{i}, C_{b}=B, C_{L_{n}}L_{n}\right).$$ So, the UAV-BS dissociates some of the users with the maximum data rate and bandwidth after comparing the new required data rate and bandwidth with the maximum one. If the new data rate

Fig. 3.
Heatmap at height 1.5 m.

and bandwidth are lower, then it completes the association with a new user. In this way, more users can be associated comfortably.

The algorithm provides an effective solution to the optimisation problem in (9). All these steps are summarized in the Algorithm 1.

IV. SIMULATION AND DISCUSSIONS

In this section, we discuss the simulation results which were created in the python platform to provide a clear understanding of the concept. An open suburban area was taken into consideration for the set up. Three BSs were deployed in the system as per the system model. The UAV had four actions to take for moving at a constant height- left, right, up (front) and down (back). Besides, the UAV has changeable heights [TeX:] $$\left(h_{U A V}\right)$$ considering various values in particular 1.5 m, 30 m, 60 m, 90 m, 120 m, 360 m, 500 m and 1000 m. User height which is [TeX:] $$h_{U A V}=1.5 \mathrm{~m}$$ is considered for detecting coverage hole only. Rest of the values are considered as the heights of the UAV-BS while providing coverage. The reward was defined as ‘+1’ for any discovery of the coverage hole which means that the SINR value less than 7 dB. Otherwise, it is defined as zero.

Fig. 2 shows simulation of the scenario and Fig. 3 is the SINR heatmap of the environment after deploying the BSs. It has been mentioned before that the MDT has several limitations such as unreliable data, longer time to create database, external intervention and so on. Instead, our methods provides a faster coverage hole detection compared to MDT. The prime reason behind this efficiency is the independent mobility and data collection of the UAV. The result of MDT is entirely dependent on the parameters of UEs. If the users are not scattered evenly in a given area, the network operator will end up creating a coverage map where the users are mostly located. Hence, it will not give an accurate picture of the complete coverage as the users are not going to every place. In some cases, the users may not want to share the parameters with network operators which will eventually create more issues. UAV with RL are free from all these complications

Fig. 4.
Users association in different data rates order: (a) Users association in random order data rates, (b) users association in ascending order data rates, and (c) users association in descending order data rates.
Table 1.
Simulation parameters.

which are providing better results to help in network planning and maintaining.

The simulation for UAV when acting as UAV-BS to provide coverage has also been carried out. To simulate, users are distributed randomly using a spatial Poisson Process in a coverage hole square area where the density [TeX:] $$\lambda=500$$ per unit of time and the UAV will cover from the center region. The data rate is randomly assigned to users starting from 0.1 to 10 Mbps. Finally, the necessary parameters have been passed to the algorithm to figure out the utmost association between UAV-BS and users utilising the optimisation problem presented in (9). Fig. 4 represents the scenario of UAV-BS and users association. As the scenario may change from time to time, we have considered 100 scenarios and figured out the average to get the optimum results.

Fig. 5 plots the average number of users served in random, ascending and descending data rates (required data rates are sorted in ascending and descending order) after averaging out 100 scenarios. For all the scenarios, the random data rate has been considered with random spatial Poisson distribution while keeping the other parameters same such as different wireless backhaul data rates at various depression angles. It can be seen from the plotted bar chart that the highest number of users can be accommodated in UAV-BS while serving the users in their required ascending data rate. Our result considers the constraints (10) and (11) when associating the users with the UAV-BS and maximises the optimisation problem. The total available data rate increases when the depression angle () is in a certain range then it drops. It is occurring because there is a downward tilt in the antenna placement. In our results, the depression angle varies between [TeX:] $$\left\{0.0023^{\circ}, 4.44^{\circ}\right\}$$ for heights from 1.5 m to 1000 m with the coverage hole distance of more than 12 km from the nearest BS. More variations can be expected in the served number of users if the coverage hole is nearer to the BS, e.g., in the case of medium coverage BS with low transmission power [41]. It will happen because of the sharper changes in the depression angle which may contribute either lower or higher path-loss over wireless backhaul. Another point is that the aerial coverage outperforms the terrestrial coverage in some cases. There are many factors associated for this better aerial coverage such as more LoS probability, decay in the path-loss exponent and shadowing effect [32]. It was also noticed but not shown in the article that the number of associated users increases in ascending data rate while the number of total users increases but stay in the limit of the UAV-BS capacity.

In Fig. 6, cumulative distribution function (CDF) of the required data rate by the served users can be found. The CDF curve of the ascending order data rate is above the random and descending order data rate. Therefore, the probability is higher

Fig. 5.
Number of users served in various wireless backhaul data rate.
Fig. 6.
CDF comparison for different data rates order.

for serving users with the low data rates while maximizing the number of users in the coverage of UAV-BS.

V. CONCLUSIONS

In this article, we present an effective way to find out the coverage hole using convenient tools. The result shows that the solution has been successful in detecting a coverage hole when running for a duration of time. Network operators do not have to wait for the report from any UE or person. This process is time convenient and can be a part of the 6G networks. Once the coverage hole is detected, the UAV itself can turn into UAV-BS on the coverage hole and serve maximum users by placing in the optimal place while considering different wireless backhaul rates at a various depression angle of the UAV with respect to BS. Here, we presented the algorithm in a combination with detecting coverage hole and placing the UAV-BS in the optimal position which can be implemented realistically. The further extension of this work could be traffic analysis for mobile coverage planning and collision avoidance by detecting objects. Another essential future work can be added as deploying airborne network if the traffic is quite high. For that scenario, we can detect a coverage hole by using a single UAV. The UAV will collect the coordinate of the start point and end point. Then, the airborne network can use the Markov decision process (MDP) to reach the destination by calculating the shortest path using the start point and end point which is the coordinate of the coverage hole.

Biography

Shahriar Abdullah Al-Ahmed

Shahriar Abdullah Al-Ahmed (S’18) received the B.Eng. degree in Computing and Electronics from the University of East London, London, U.K., in 2008 and the M.Sc. degree in Telecommunication (network pathway) from Queen Mary University of London, London, U.K. in 2012. Currently, he is working toward the Ph.D. degree in Engineering and Computing with the University of the West of Scotland, Paisley, Scotland, U.K. Before joining the Ph.D. degree, he was involved in teaching computer science in secondary educational institutes in U.K. His research interests include artificial intelligence, Internet of Things, and wireless communication.

Biography

Muhammad Zeeshan Shakir

Muhammad Zeeshan Shakir (S’04-M’10-SM’16) is Reader (Associate Professor) at the University of the West of Scotland (UWS), UK, received close to £1.5m funding from bodies such as Innovate UK, ERASMUS and UK industries. With over 10 years of research expertise in design and development of wireless communication systems, he has published over 120 research articles and edited or contributed to 10 books. He is a recipient of STARS (Staff Appreciation and Recognition Scheme) Award 2020 2018 for Outstanding Research and Enterprise performance at UWS. He has been awarded with the Best Journal article award 2019 jointly presented by IEEE Communications Society and China Institute of Communications for research on millimetre wave communications for 5G. He has been/is serving as a Chair and organising committee of several symposiums/workshops in IEEE flagship conferences, including Globecom, ICC and WCNC. He is an Editor of PHYCOM, IEEE Communications Letters and served as a guest editor to EEE Wireless Communications, IEEE Communications and IEEE Access. He is a founding Chair of IEEE ComSoc emerging technical committee on backhaul/fronthaul. He is a Fellow of Higher Education Academy, UK, Senior Member of IEEE and an active member of IEEE ComSoc.

Biography

Syed Ali Raza Zaidi

Syed Ali Raza Zaidi (M’09) is a University Academic Fellow (Assistant Professor) at the University of Leeds in the broad area of Communication Sensing for Robotics and Autonomous Systems. Earlier from 2013-2015, he was associated with the SPCOM research group working on US ARL funded project in the area of Network Science. From 2011-2013, he was associated with the International University of Rabat working as Research Associate. He was also a visiting research scientist at Qatar Innovations and Mobility Centre from October- December 2013 working on QNRF funded project QSON. He completed his Doctoral Degree at the School of Electronic and Electrical Engineering. He was awarded the G. W. and F. W. Carter Prize for best thesis and best research paper. He has published 90+ papers in leading IEEE conferences and journals. From 2014-2015, he was the Editor for IEEE Communication Letters and also Lead Guest Editor for IET Signal Processing Journal’s Special Issue on Signal Processing for Large Scale 5G Wireless Networks. He is also an Editor for IET Access, Front haul and Backhaul book. Currently, he is serving as a Associate Technical Editor for IEEE Communication Magazine. He has been awarded COST IC0902, Royal Academy of Engineering, EPSRC and DAAD grants to promote my research outputs. His current research interests are at the intersection ICT, applied mathematics, mobile computing and embedded systems implementation. Specifically, his current research is geared towards: (i) Design and implementation of communication protocols to enable various applications (rehabilitation, healthcare, manufacturing, surveillance) of future RAS; and (ii) design, implementation and control of RAS for enabling future wireless networks (for e.g. autonomous deployment, management and repair of future cellular networks).

References

  • 1 D. Wang, S. A. Al-Ahmed, M. Z. Shakir, "Optimized link distribution schemes for ultra reliable and low latent communications in multi-layer airborne networks," IEEE Trans. Industrial Informatics, vol. 16, no. 9, Sept, 2019.custom:[[[-]]]
  • 2 M. Mozaffari, W. Saad, M. Bennis, Y. Nam, M. Debbah, "A Tutorial on UA Vs for wireless networks: Applications, challenges, and open problems," CoRR. (Online). Available: http://arxiv.org/abs/1803.00680, vol. abs/1803.00680, 2018.custom:[[[-]]]
  • 3 R. Jurdi, J. G. Andrews, D. Parsons, R. W. Heath, "Identifying coverage holes: Where to densify?," in Proc. IEEE ACSSC, pp. 14171421-14171421, Oct, 2017.custom:[[[-]]]
  • 4 S. A. W. Shah, T. Khattab, M. Z. Shakir, M. O. Hasna, "A distributed approach for networked flying platform association with small cells in 5G+ networks," in Proc. IEEE GLOBECOM. . Fig. 6. CDF comparison for different data rates order, Dec, 2017.custom:[[[-]]]
  • 5 Y. Zeng, R. Zhang, T. J. Lim, "Wireless communications with unmanned aerial vehicles: Opportunities and challenges," IEEE Commun. Mag., vol. 54, no. 5, pp. 36-42, May, 2016.doi:[[[10.1109/MCOM.2016.7470933]]]
  • 6 Scottish Government and Scottish Futures Trust, Scottish 4G Infill Programme Consultation: Request For Information. Edinburgh United Kingdom: Scottish Government, 2018.custom:[[[-]]]
  • 7 T. Jones, DW, December 2018. (Online). Available:, https://www.dw.com/en/germanys-4g-mobile-network-one-of-worst-in-europe/a-46880219
  • 8 The Local, Nov. 2019. (Online). Available:, https://www.thelocal.de/20191111/funklochrepublik
  • 9 T. Ebuchi, Nikkei Asian Review, April 2019. (Online). Available:, https://asia.nikkei.com/Business/Companies/KDDI-and-Rakuten-to-help-eliminate-cellular-dead-zones-by-2024
  • 10 Y. Wang, K. I. Pedersen, F. Frederiksen, "Detection and protection of macro-users in dominant area of co-channel CSG cells," in Proc. IEEE VTC Spring, pp. 1-5, May, 2012.custom:[[[-]]]
  • 11 H.-W. Liang, C.-H. Ho, L.-S. Chen, W.-H. Chung, S.-Y. Yuan, S.Y. Kuo, "Coverage hole detection in cellular networks with deterministic propagation model," in Proc. IEEE ISBSG, pp. 1-6, June, 2016.custom:[[[-]]]
  • 12 A. Galindo-Serrano, B. Sayrac, S. B. Jemaa, J. Riihij¨ arvi, P. M¨ ah¨ onen, "Automated coverage hole detection for cellular networks using radio environment maps," in Proc. IEEE WiOpt, pp. 35-40, May, 2013.custom:[[[-]]]
  • 13 Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Overall description; Stage 2, 3GPP . v12.4.0, Feb, 2015.custom:[[[-]]]
  • 14 I. Akbari, O. Onireti, A. Imran, M. A. Imran, R. Tafazolli, "How reliable is MDT-based autonomous coverage estimation in the presence of user and BS positioning error?," IEEE Wireless Commun. Letters, vol. 5, no. 2, pp. 196-199, Jan, 2016.custom:[[[-]]]
  • 15 E. Teng, J. D. Falc˜ ao, B. Iannucci, "Holes-in-the-Sky: A field study on cellular-connected UAS," in Proc. IEEE ICUAS, pp. 11651174-11651174, June, 2017.custom:[[[-]]]
  • 16 P. V. Klaine, S. Yang, R. D. Souza, J. P. B. Nadas, M. A. Imran, "Reinforcement Learning Enabled Unmanned Aerial Vehicles in Pop-Up Cellular Networks.," in (Online). Available: http://conference2018.chistera.eu/sites/conference2018.chistera.eu/ files/CHIST-ERA Call2018 Conf POSTER Yang.pdf, (Online). Available: http://conference.chistera.eu/sites/conference.chistera.eu/ files/CHIST-ERA Call Conf POSTER Yang.pdf, 2018;custom:[[[-]]]
  • 17 H. X. Pham, H. M. La, D. Feil-Seifer, L. V. Nguyen, "Autonomous UA V navigation using reinforcement learning," arXiv preprint arXiv:1801.05086, 2018.custom:[[[-]]]
  • 18 W. Koch, R. Mancuso, R. West, A. Bestavros, "Reinforcement learning for UA V attitude control," ACM Trans. Cyber-Physical Systemsp. 22, vol. 3, no. 2, Feb, 2019.custom:[[[-]]]
  • 19 R. Sharma, "Fuzzy Q learning based UA V autopilot," in Proc. IEEE CIPECH, pp. 29-33, Nov, 2014.custom:[[[-]]]
  • 20 Z. Yijing, Z. Zheng, Z. Xiaoyi, L. Yang, "Q learning algorithm based UA V path learning and obstacle avoidence approach," in Proc. IEEE CCC, pp. 3397-3402, July, 2017.custom:[[[-]]]
  • 21 H. Bayerlein, P. De Kerret, D. Gesbert, "Trajectory optimization for autonomous flying base station via reinforcement learning," in Proc. IEEE SPAWC, pp. 1-5, June, 2018.custom:[[[-]]]
  • 22 E. Kalantari, H. Yanikomeroglu, A. Yongacoglu, "On the number and 3D placement of drone base stations in wireless cellular networks," in Proc. IEEE VTC-Fall, pp. 1-6, Sept, 2016.doi:[[[10.1109/vtcfall.2016.7881122]]]
  • 23 E. Kalantari, M. Z. Shakir, H. Yanikomeroglu, A. Yongacoglu, "Backhaul-aware robust 3D drone placement in 5G+ wireless networks," in Proc. IEEE ICC workshops, pp. 109-114, May, 2017.custom:[[[-]]]
  • 24 R. I. Bor-Yaliniz, A. El-Keyi, H. Yanikomeroglu, "Efficient 3-D placement of an aerial base station in next generation cellular networks," in Proc. IEEE ICC, pp. 1-5, May, 2016.doi:[[[10.1109/icc.2016.7510820]]]
  • 25 M. Alzenad, M. Z. Shakir, H. Yanikomeroglu, M.-S. Alouini, "FSObased vertical backhaul/fronthaul framework for 5G+ wireless networks," IEEE Commun. Mag., vol. 56, no. 1, pp. 218-224, Jan, 2018.custom:[[[-]]]
  • 26 M. Alzenad, A. El-Keyi, F. Lagum, H. Yanikomeroglu, "3-D placement of an unmanned aerial vehicle base station (UA V-BS) for energyefficient maximal coverage," IEEE Wireless Commun. Letters, vol. 6, no. 4, pp. 434-437, Aug, 2017.custom:[[[-]]]
  • 27 E. Kalantari, I. Bor-Yaliniz, A. Yongacoglu, H. Yanikomeroglu, "User association and bandwidth allocation for terrestrial and aerial base stations with backhaul considerations," in Proc. IEEE PIMRC, pp. 1-6, Oct, 2017.custom:[[[-]]]
  • 28 R. Ghanavi, E. Kalantari, M. Sabbaghian, H. Yanikomeroglu, A. Yongacoglu, "Efficient 3D aerial base station placement considering users mobility by reinforcement learning," in Proc. IEEE WCNC, pp. 16-16, Apr, 2018.custom:[[[-]]]
  • 29 K. T. Feng, L. H. Shen, C. Y. Li, P. T. Huang, S. H. Wu, L. C. Wang, Y. B. Lin, M. C. F. Chang, "3d on-demand flying mobile communication for millimeter-wave heterogeneous networks," IEEE Netw../, vol. 34, no. 5, pp. 198-204, Sept, 2020.custom:[[[-]]]
  • 30 H. Xiao, S. K. Vadgama, "Coverage hole detection in cellular wireless network," Apr. 12 2012uS Patent App.13/249,292, pp. uS Patent A13/249-uS Patent A13/249. custom:[[[-]]]
  • 31 A. Gomez Andrades, R. Barco, I. Serrano, "A method of assessment of LTE coverage holes," EURASIP J. Wireless Commun. Netw.p. 236, vol. 2016, Oct, 2016.doi:[[[10.1186/s13638-016-0733-y]]]
  • 32 R. Amorim, H. Nguyen, P. Mogensen, I. Z. Kov´ acs, J. Wigard, T. B. Sørensen, "Radio channel modeling for uav communication over cellular networks," IEEE Wireless Commun. Letters, vol. 6, no. 4, pp. 514-517, Aug, 2017.doi:[[[10.1109/LWC.2017.2710045]]]
  • 33 A. Al-Hourani, K. Gomez, "Modeling cellular-to-UA V path-loss for suburban environments," IEEE Wireless Commun. Letters, vol. 7, no. 1, pp. 82-85, Feb, 2017.custom:[[[-]]]
  • 34 A. Al-Hourani, S. Kandeepan, A. Jamalipour, "Modeling air-toground path loss for low altitude platforms in urban environments," in Proc. IEEE GLOBECOM, pp. 2898-2904, Dec, 2014.custom:[[[-]]]
  • 35 A. Al-Hourani, S. Kandeepan, S. Lardner, "Optimal LAP altitude for maximum coverage," IEEE Wireless Commun. Letters, vol. 3, no. 6, pp. 569-572, July, 2014.doi:[[[10.1109/LWC.2014.2342736]]]
  • 36 N. A. Ali, H.-A. M. Mourad, H. M. ElSayed, M. El-Soudani, H. H. Amer, R. M. Daoud, "General expressions for downlink signal to interference and noise ratio in homogeneous and heterogeneous LTE-Advanced networks," J. advanced research, vol. 7, no. 6, pp. 923-929, Nov, 2016.custom:[[[-]]]
  • 37 (Online). Available:, https://usatcorp.com/faqs/understanding-lte-signal-strength-values/
  • 38 R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018.custom:[[[-]]]
  • 39 A. Nandy, M. Biswas, Reinforcement Learning: With Open AI, TensorFlow and Keras Using Python. ess, Apr, 2017.custom:[[[-]]]
  • 40 R. Ghanavi, E. Kalantari, M. Sabbaghian, H. Yanikomeroglu, A. Yongacoglu, "Efficient 3D aerial base station placement considering users mobility by reinforcement learning," in Proc. IEEE WCNC, pp. 1-6, Apr, 2018.custom:[[[-]]]
  • 41 3GPP, 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 38.104, 2020, version 15.9.0. (Online). Available:, https://www.etsi.org/deliver/etsits/138100138199/138104/15.09.0060/ts138104v150900p.pdf

Table 1.

Simulation parameters.
Parameter Symbol Value
Simulation environment - suburban
Environment parameters [TeX:] $$\vartheta, \xi, \eta_{L o S}, \eta_{N L o S}$$ 4.88, 0.43, 0.1, 21
LTE frequency band - 850 MHz
Bandwidth B 15 MHz
Transmission power - 43 dBm
Terrestrial path-loss exponent [TeX:] $$\alpha_{\text {terr }}$$ 3.04
Excess path-loss scaler [TeX:] $$\mathcal{A}$$ -23.29
Excess path-loss offset [TeX:] $$\eta_{o}$$ 20.70
Angle offset [TeX:] $$\theta_{o}$$ -3.61
Angle scalar [TeX:] $$\mathcal{B}$$ 4.14
UAV shadowing slope a -0.41
UAV shadowing offset \sigma_{o} 5.86
UAV learning rate [TeX:] $$\alpha_{Q}$$ 0.01
Discount factor [TeX:] $$\gamma$$ 0.9
UAV-BS frequency [TeX:] $$f_{c}$$ 2 GHz
UAV-BS transmission power - 37 dBm
Graphical representation of UAV detecting coverage hole and serving user as a UAV-BS after discovering the coverage hole.
Coverage hole discovery and UAV-BS placement
Coverage hole detection simulation exploiting RL.
Heatmap at height 1.5 m.
Users association in different data rates order: (a) Users association in random order data rates, (b) users association in ascending order data rates, and (c) users association in descending order data rates.
Number of users served in various wireless backhaul data rate.
CDF comparison for different data rates order.