## Dong-Hyoun Na , Ki-Hong Park , Young-Chai Ko and Mohamed-Slim Alouini## |

Notation | Definition |
---|---|

[TeX:] $$L$$ | Number of gateways / feeder link receivers/ antenna feed clusters / beam clusters |

[TeX:] $$B$$ | Number of antenna feed elements/ beams in a beam cluster |

[TeX:] $$G$$ | Number of multicast groups in a beam |

[TeX:] $$K$$ | Number of users in a group |

BM-([TeX:] $$l, b$$) | Beam from the [TeX:] $$b$$th feed element of the [TeX:] $$l$$th antenna feed cluster |

UE-([TeX:] $$l, b, g, k$$) | [TeX:] $$k$$th user of the [TeX:] $$g$$th multicast group in BM-([TeX:] $$l, b$$) |

[TeX:] $$\mathbf{h}_{i, l b g k}^H$$ | Channel coefficient from the ith feed cluster to UE-([TeX:] $$l, b, g, k$$) |

[TeX:] $$G_T, G_R$$ | Maximum satellite antenna gainand user receiver antenna gain |

[TeX:] $$r_{l b q k}$$ | Atmospheric fading of UE-([TeX:] $$l, b, g, k$$) |

[TeX:] $$\mathbf{q}_{i, l b q k}$$ | Vector including the beam pattern and path loss |

[TeX:] $$\xi_{l b a k}, \phi_{l b a k}$$ | Power gain and phase of rain attenuation |

[TeX:] $$\mu, \sigma$$ | Lognormal location and scale of rain attenuation |

[TeX:] $$\lambda$$ | Carrier wavelength |

[TeX:] $$g_{i j, l b q k}$$ | Beam pattern |

[TeX:] $$d_{l b a k}$$ | Distance between the satellite and UE-([TeX:] $$l, b, g, k$$) |

[TeX:] $$\kappa, T_{\mathrm{R}}, B_{\mathrm{ul}}$$ | Boltzmann constant, receiver noise temperature, andbeamwidth |

[TeX:] $$\eta_{i j, l b g k}$$ | Phase variation |

\theta_{i j, l g b k} | Angle between the center of BM-([TeX:] $$i, j$$) andthe user position |

[TeX:] $$\theta_{3 \mathrm{~dB}}$$ | Angle between 3 dB power loss andthe beam center |

[TeX:] $$s_{l b g}$$ | Signal for the gth multicast group users in BM-([TeX:] $$l, b$$) |

[TeX:] $$s_{l b}$$ | Signal for users in BM-([TeX:] $$l, b$$) |

[TeX:] $$\alpha_{l b g}$$ | Power allocation coefficient |

[TeX:] $$\mathbf{W} l b$$ | Precoding vector for users in BM-([TeX:] $$l, b$$) |

[TeX:] $$n_{l b a k}$$ | Additive white Gaussian noise |

[TeX:] $$a_{l b, i j}$$ | Binary variable for decoding order |

[TeX:] $$\operatorname{SINR}_{l b g k}^m$$ | SINR of UE-([TeX:] $$l, b, g, k$$) to decode the message for the mth group |

[TeX:] $$P_1^{\mathrm{GW}}$$ | Maximum power at the [TeX:] $$l$$th gateway |

[TeX:] $$P_{l v}^{\mathrm{Feed}}$$ | Maximum power at the vth feed element of [TeX:] $$l$$th antenna feed cluster |

A. Channel Model

We assume that the feeder links between the gateways and satellite are ideal because the gateways exploit very directive antennas to transmit signals and the SNR of the feeder link is much greater than that of the user link [15], [33]. Therefore, the noise and interference in the feeder links can be negligible.

Thus, [TeX:] $$\mathbf{h}_{i, l b g k}^H \in \mathbb{C}^{1 \times B}$$ represents the channel coefficient from the [TeX:] $$i$$th feed cluster to UE-([TeX:] $$l, b, g, k$$) and can be decom-posed as where [TeX:] $$G_T$$ and [TeX:] $$G_R$$ are the maximum satellite antenna gain and user receiver antenna gain, respectively, and [TeX:] $$\mathbf{q}_{i, l b g k}$$ is a vector including the beam pattern and path loss. The parameter rlbgk denotes the atmospheric fading of UE-([TeX:] $$l, b, g, k$$) because the satellite channels using high-frequency bands, such as Ka- band, are significantly affected by atmospheric fading effects. Among them, rain attenuation is the most dominant factor. The atmospheric fading of UE-([TeX:] $$l,b,g,k$$) due to rain attenuation is given as [34] and [35]

where [TeX:] $$\xi_{l b g k}$$ denotes the power gain of rain attenuation. The power gain in dB, [TeX:] $$\xi_{l b g k}^{\mathrm{dB}}=20 \log _{10}\left(\xi_{l b g k}\right)$$, follows a logonornal distribution, i.e., ln [TeX:] $$\left(\xi_{l b g k}^{\mathrm{dB}}\right) \sim \mathcal{N}\left(\mu, \sigma^2\right)$$, where [TeX:] $$\mu$$ and [TeX:] $$\sigma$$ are the lognormal location and scale pa- rameters, respectively, besides [TeX:] $$\phi_{l b g k}$$ is the uniformly dis-tributed phase over [0, 2π). Furthermore, [TeX:] $$\mathbf{q}_{i, l b g}$$ is defined by [TeX:] $$\mathbf{q}_{i, l b g k}=\left[q_{i 1, l b g k} \cdots q_{i j, l b g k} \cdots q_{i B, l b g k}\right]^T$$ , where [TeX:] $$q_{i j, l b g k}$$ can be written as [35] and [18]

B. Signal Model

Assuming that the message [TeX:] $$s_{l b g}$$ is intended for the [TeX:] $$g$$th mul- ticast group users in BM-([TeX:] $$l, b$$), the signals for BM-([TeX:] $$l, b$$) are superimposed using the NOMA scheme at the lth gateway. It can be written as [TeX:] $$s_{l b}=\sum_{g=1}^G \sqrt{\alpha_{l b g}} s_{l b g}$$, where [TeX:] $$0 \leq \alpha_{l b g} \leq 1$$ is the power allocation coefficient and [TeX:] $$\sum_{g=1}^G \alpha_{l b g}=1$$. The signal [TeX:] $$s_{l b g}$$ has unit average power i.e., [TeX:] $$\left(\text { i.e., } \mathbb{E}\left[\left|s_{l b g}\right|^2\right]=1\right)$$. Thus, the signal for the lth cluster [TeX:] $$\mathbf{S}_l$$ can be expressed as [TeX:] $$\mathbf{s}_l=\left[\begin{array}{llll} s_{l 1} & s_{l 2} & \cdots & s_{l B} \end{array}\right]^T$$. Before transmission, [TeX:] $$\mathbf{s}_l$$ is precoded with matrix [TeX:] $$\mathbf{W}_l \in \mathbb{C}^{B \times B}$$ at the [TeX:] $$l$$th gateway to mitigate intra- cluster and inter-cluster interference. The transmitted signal [TeX:] $$\mathbf{x}_l$$ can be written as

where [TeX:] $$\mathbf{w}_{l b} \in \mathbb{C}^{B \times 1}$$ is a precoding vector for users in M-([TeX:] $$l, b$$).

The [TeX:] $$l$$th feeder link receiver receives the precoded signals in the satellite via the feeder link. Since we consider a transparent payload as in [13]–[15] and [33], the satellite transfers the signals via the antenna feed clusters as a relay. Therefore, the received signal at UE-([TeX:] $$l, b, g, k$$) can be written as

where [TeX:] $$n_{l b g k}$$ denotes the additive white Gaussian noise (AWGN) with zero mean and variance [TeX:] $$\sigma_{l b g k}^2$$. To clarify the lbgk interference due to the SC scheme, intra-cluster interference, and inter-cluster interference, [TeX:] $$y_{l b g k}$$ can be rewritten as

For each beam, some multicast group users adopt SIC to detect the signals. When users in a multicast group directly decode the desired signal without SIC, users in the other multicast groups exploit SIC. The users employing SIC decode the signals intended for other groups, remove them from interference, and obtain the signal desired for themselves. To determine the multicast groups using SIC (i.e., decoding order) in each beam, we introduce binary variable [TeX:] $$a_{l b, i j} \in\{0,1\}$$, where [TeX:] $$a_{l b, i j} = 0$$ indicates that the [TeX:] $$j$$th multicast group users in BM-([TeX:] $$l, b$$) take advantage of SIC and decode the signal for the [TeX:] $$i$$th group, otherwise [TeX:] $$a_{l b, i j}$$ = 1. Note that [TeX:] $$a_{l b, i j}+a_{l b, j i}$$ = 1 and [TeX:] $$a_{l b, i i}=0, i \neq j, \forall i, j$$. Hence, the SINR of UE-([TeX:] $$l, b, g, k$$) to decode the message for the [TeX:] $$m$$th group in BM-([TeX:] $$l, b$$) can be expressed as

where [TeX:] $$\mathcal{W} \triangleq\left\{\mathbf{w}_{l b}, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}\right\}, \mathcal{P}_{l b} \triangleq\left\{\alpha_{l b g}, \forall g \in \mathcal{G}\right\}$$, and [TeX:] $$\mathcal{A}_{l b} \triangleq\left\{a_{l b, i j}, \forall i, j \in \mathcal{G}\right\}$$. An arbitrary large positive num- ber [TeX:] $$M$$ is introduced to exclude the SINR that does not match the decoding order. In other words, if [TeX:] $$a_{l b, m g}=1, \operatorname{SINR}_{l b g k}^m$$ is not calculated according to the decoding order and is not selected in multicast systems. In (8), [TeX:] $$I_{l b g k}(\mathcal{W})$$ is the sum of intra-cluster and inter-cluster interference at UE-([TeX:] $$l, b, g, k$$), which can be written as [TeX:] $$I_{l b g k}(\mathcal{W})=\sum_{j \neq b}^B\left|\mathbf{h}_{l, l b g k}^H \mathbf{w}_{l j}\right|^2+$ $\sum_{i \neq l}^L \sum_{j=1}^B\left|\mathbf{h}_{i, l b g k}^H \mathbf{w}_{i j}\right|^2$$

Our objective is to develop a transmission scheme that Our objective is to develop a transmission scheme that mission in the SatCom system with the NOMA scheme while satisfying perfect SIC and power constraints. Ac- cordingly, we propose a method to optimize the precod- ing vectors [TeX:] $$\mathcal{W}$$, the power allocation coefficients of SC [TeX:] $$\mathcal{P} \triangleq\left\{\mathcal{P}_{l b}, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}\right\}$$, and the selection of multicast groups to employ SIC [TeX:] $$\mathcal{A} \triangleq\left\{\mathcal{A}_{l b}, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}\right\}$$. To suc- cessfully perform SIC in the NOMA multigroup multicast system [36], the optimization problem can be formulated as

for [TeX:] $$\forall l \in \mathcal{L}, \forall b, v \in \mathcal{B}$$, and [TeX:] $$\forall g, i, j \in \mathcal{G} . P_l^{\mathrm{GW}}$$ and [TeX:] $$P_{l n}^{\text {Feed }}$$ are the maximum power at the [TeX:] $$l$$th gateway and the [TeX:] $$n$$th feed element of [TeX:] $$l$$th antenna feed cluster, respectively. In (9c), [TeX:] $$\mathbf{Q}_v$$ is a matrix whose [TeX:] $$v$$th diagonal element is 1 and the other elements are 0. Problem (9) is a non-convex problem due to (9a) and the binary constraints of (9e) although the others are convex constraints. It is difficult to find a globally optimal solution for this problem. Therefore, in the next section, we propose a method to obtain a local optimum by decomposing (9) and approximating it to convex problems.

Since (9) is a non-convex MINLP, we suggest designing an algorithm based on BCD to solve it efficiently. To ap- proximate the problem in each iteration, we need to trans- form it into a more amenable form. Thus, we introduce two sets of variables: [TeX:] $$\Gamma \triangleq\left\{\gamma_{l b}^m, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}, \forall m \in \mathcal{G}\right\}$$ and [TeX:] $$\mathcal{R} \triangleq\left\{R_{l b}^m, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}, \forall m \in \mathcal{G}\right\}$$, and then (9) can be transformed into

for [TeX:] $$\forall l \in \mathcal{L}, \forall b, v \in \mathcal{B}, \forall g, m, i, j \in \mathcal{G} \text {, and } \forall k \in \mathcal{K}$$. Since (10f) and (10g) hold equality at the optimal condition, the equivalence of (9) and (10) is guaranteed and can be easily proved by contradiction.

Even if the (10a) is relaxed to a convex function, (10) is still non-convex owing to (10e) and (10f). To solve the problem effectively, we propose decomposing (10) into three subproblems and solving them by approximating the non- convex constraints iteratively with the help of the BCD al- gorithm. In other words, it is divided into (i) a problem for precoding vector with power allocation and decoding order fixed, (ii) a problem for power allocation with precoding vector and decoding order fixed, and (iii) a problem for decoding order with precoding vector and power allocation fixed. The three subproblems are alternately solved utilizing SCA until they converge.

A. Precoding Optimization

Given [TeX:] $$\mathcal{P}$$ and [TeX:] $$\mathcal{A}$$, the optimization problem for [TeX:] $$\mathcal{W}$$ can be written as

where [TeX:] $$\mathcal{P}_{l b}^{(n)}$$ and [TeX:] $$\mathcal{A}_{l b}^{(n)}$$ denote the solutions obtained at the [TeX:] $$n$$th iteration. The non-convex constraint (11b) can be expressed as

where the right-hand side is a convex function for w. The left-hand side is jointly convex for [TeX:] $$\mathbf{w}_{l b}$$ and [TeX:] $$\gamma_{l b}^m$$ be- cause it is a quadratic-over-linear function [37]. When we define [TeX:] $$f_{l b g k, m}\left(\mathbf{w}_{l b}, \gamma_{l b}^m\right) \triangleq\left|\mathbf{h}_{l, l b g k}^H \mathbf{w}_{l b}\right|^2 \alpha_{l b m}^{(n)} / \gamma_{l b}^m$$ and [TeX:] $$g_{l b, m g}\left(\gamma_{l b}^m\right) \triangleq a_{l b, m g}^{(n)} M / \gamma_{l b}^m$$, they can be lower bounded with the first-order Taylor series expansion around [TeX:] $$\mathbf{w}_{l h}^{(n)}$$ and [TeX:] $$\left(\gamma_{l b}^m\right)^{(n)}$$, which can be expressed as

where [TeX:] $$\mathbf{w}_{l b}^{(n)}$$ and [TeX:] $$\left(\gamma_{l b}^m\right)^{(n)}$$ are the solutions from the [TeX:] $$n$$th iteration. Hence, (12) can be convexified as

Although (10g) is a convex constraint, because log functions are not preferred in optimization solvers (e.g., MOSEK), we change it to a more computationally efficient form [38]. First, we have

Then, by applying the first-order Taylor series expansion, the convex function on the left-hand side can be lower-bounded as

where [TeX:] $$\left(\tau_{l b}^m\right)^{(n)}$$ and [TeX:] $$\left(v_{l b}^m\right)^{(n)}$$ represent

Although the left-hand side in (16) is lower bounded as (17), (16) is non-convex owing to the right-hand side. However, we can convert (16) equivalently into a second-order cone constraint, which can be expressed as

This constraint can be utilized equally in the other two divided optimization problems. Therefore, (11) can be rewritten as

which is a second-order cone programming (SOCP) and can be solved by standard convex optimization software.

B. Power Allocation Optimization

Given [TeX:] $$\mathcal{W}$$ and [TeX:] $$\mathcal{A}$$, the optimization problem to obtain [TeX:] $$\mathcal{P}$$ can be written as

where [TeX:] $$\mathcal{W}^{(n)}$$ represents the solutions obtained at the nth iteration. We can rewrite (21b) as

which is non-convex owing to [TeX:] $$\gamma_{l b}^m \alpha_{l b u}$$ in the first term on the right hand side. When we take advantage of simple algebraic operations and first-order Taylor series expansion around [TeX:] $$\left(\left(\gamma_{l b}^m\right)^{(n)}, \alpha_{l b u}^{(n)}\right)$$, it can be upper bounded as

Therefore, (21b) can be convexified as

The relaxed problem for [TeX:] $$\mathcal{P}$$ can be rewritten as

which is also an SOCP.

C. Decoding Order Optimization

Finally, given [TeX:] $$\mathcal{W}$$ and [TeX:] $$\mathcal{P}$$, the optimization problem for [TeX:] $$\mathcal{A}$$ can be formulated as

where (26b) and (10e) are non-convex constraints. The con-straint (26b) can be expressed as

which can be relaxed similar to (24) as

where [TeX:] $$H_{2, l b u}^m\left(\gamma_{l b}^m, a_{l b, u m} ;\left(\gamma_{l b}^m\right)^{(n)}, a_{l b, u m}^{(n)}\right)$$ is represented as

In (10e), the binary constraint [TeX:] $$a_{l b, i j} \in\{0,1\}$$ can be trans- formed into an equivalently continuous form as follows [39]:

Since (30b) is a form of difference-convex function, we can take advantage of the first-order Taylor series expansion around [TeX:] $$a_{l b, i j}^{(n)}$$ to obtain convex approximation. It is given as

However, if (31) is deployed directly, the problem becomes infeasible and fails to be optimized. Hence, a penalty parameter [TeX:] $$\rho$$ > 0 and slack variable [TeX:] $$\Lambda=\left\{\lambda_{l b, i j} \geq 0, \forall l \in \mathcal{L}, \forall b \in \mathcal{B}, \forall i, j \in \mathcal{G}\right\}$$ are introduced [40]. Therefore, the approximated problem for a can be expressed as

where (32a) and (32b) are equivalent to (26a) and (31), respectively, when [TeX:] $$\lambda_{l b, i j}$$ = 0. (32) is also an SOCP as (20) and (25).

The algorithm based on BCD for solving (10) is summarized in Algorithm 1. For the relaxation of (32b), the initial value of [TeX:] $$\rho$$ is set small, but as in [40], it increases by the constant [TeX:] $$\epsilon$$ > 1 until [TeX:] $$\sum_{l=1}^L \sum_{b=1}^B \sum_{i=1}^G \sum_{j=1}^G \lambda_{l b, i j} \approx 0$$. The convergence of Algorithm 1 including [TeX:] $$\rho_{\max }$$ can be proven in a similar to [40]. If there are [TeX:] $$\rho_{\max }$$ and [TeX:] $$n_1$$ that satisfy [TeX:] $$\sum_{l, b, i, j}\left|\lambda_{l b, i j}^{(n)}\right|=0$$ at the [TeX:] $$n$$th iteration larger than [TeX:] $$n_1$$, then [TeX:] $$\sum_{l, b, m} R_{l b}^m$$ con-verges. Assuming [TeX:] $$p_{l b, i j}^{(n)}$$ and [TeX:] $$q_{l b, i j}^{(n)}$$ are Lagrangian multipliers for (32b) and [TeX:] $$\lambda_{l b, i j} \geq 0$$, the following expressions can be obtained by the Karush-Kuhn-Tucker (KKT) conditions [37], [TeX:] $$\rho^{(n)}-p_{l b, i j}^{(n)}-q_{l b, i j}^{(n)}$$ = 0 and [TeX:] $$q_{l b, i j}^{(n)} \lambda_{l b, i j}^{(n)}$$ = 0. Besides, [TeX:] $$\sum_{l, b, i, j}\left|p_{l b, i j}^{(n)}\right|$$ can be upper bounded [41] and if [TeX:] $$\rho_{\max }$$ is greater than the upper bound, [TeX:] $$n_1$$ satisfying [TeX:] $$p_{l b, i j}^{(n)}<\rho^{(n)}$$ exists for [TeX:] $$\forall n>n$$. Therefore, [TeX:] $$q_{l b, i j}^{(n)}>0$$ and [TeX:] $$\lambda_{l b, i j}^{(n)}=0$$ for [TeX:] $$\forall n>n_1$$. When [TeX:] $$n<n_1, \rho^{(n)}$$ increases at each iteration. Accordingly, (32) converges and the minimum possible [TeX:] $$\lambda^{(n)}$$ is determined. The solution to (32) is also feasible in the next iteration for (20). When [TeX:] $$n>n_1$$, since [TeX:] $$\rho^{(n)}$$ is fixed and [TeX:] $$\lambda^{(n)}$$ = 0, it is a general BCD-based algorithm. Also, since it is bounded by power constraints, Algorithm 1 converges after more than [TeX:] $$n_1$$ iterations.

The complexity of Algorithm 1 based on the BCD method lies in the complexity of subproblems. We note that the subproblems are SOCPs with lower computational complex- ity compared to other nonlinear problems. The complex- ity of SOCP is related to the number of variables and the number of constraints [42]. By using an interior-point method, the numbers of iterations for the subproblems in Section IV-A, IV-B, and IV-C to reach an acceptable dual-ity gap are bounded by [TeX:] $$\mathcal{O}\left(\sqrt{L B G^2 K}\right), \mathcal{O}\left(\sqrt{L B G^2 K}\right)$$, and [TeX:] $$\mathcal{O}\left(\sqrt{L B G^2 K}\right)$$, respectively. In addition, the compu-tational complexity of each iteration for the subproblems i s [TeX:] $$\mathcal{O}\left(L^3 B^3 G^4 K\right), \mathcal{O}\left(L^3 B^3 G^4 K\right), \text { and } \mathcal{O}\left(L^3 B^3 G^6 K\right)$$, re- spectively.

In this section, we present the simulation results to demon- strate the superiority of the proposed NOMA multicast scheme. There is a GEO satellite, [TeX:] $$L$$ = 3 gateways and beam clusters, and each beam cluster has [TeX:] $$B$$ = 7 beams. In terms of the number of users per beam, we simulated a reasonable and sufficient number of users, which is comparable with other multicast multibeam satellite works [5], [18], [33]. As shown in Fig. 3, it is assumed that users are uniformly distributed in the beams and utilize the beam topology considered in [18]. Furthermore, we establish that the maximum power of all antenna feed elements is identical and the maximum power of all gateways is equal to [TeX:] $$B$$ times of [TeX:] $$P_{l v}^{\mathrm{Feed}}$ (i.e., $P_{l n}^{\mathrm{Feed}}=P$ $\left.P_l^{\mathrm{GW}}=B P, \forall l \in \mathcal{L}, \forall n \in \mathcal{B}\right)$$ [18]. Since noise variance does not affect the algorithm and its results, we suppose that all users have the same noise variance [TeX:] $$\sigma_{l b g k}^2=\kappa T_{\mathrm{R}} B_{\mathrm{ul}}$$, where [TeX:] $$\kappa, T_{\mathrm{R},} \text { and } B_{\mathrm{ul}}$$ are the Boltzmann constant, receiver noise temperature, and beam bandwidth, respectively [13], [15]. The system parameters are listed in Table II. The average user throughput per beam [TeX:] $$R_{\mathrm{avg}}$$ used for a multibeam satellite system is given as [14] where β is the roll-off factor.

We compare the proposed multicast scheme with another baseline scheme, the general OMA scheme. The OMA method transmits signals to multiple user groups by dividing the time according to the number of groups within the beam. Similar to the OMA multicast technique, the SCA-SOCP algorithm proposed in [18] is employed in accordance with the situation considered in this paper. The SCA-SOCP algorithm results in higher throughput performance than those suggested in [14] and [17].

Table 1.

PARAMETER | VALUE |
---|---|

Satellite height | 35,786 km (GEO orbit) |

Link frequency band | [TeX:] $$f_c$$ = 20 GHz (Ka-band) |

Boltzmann constant | [TeX:] $$\kappa=1.3807 \times 10^{-23}$$ |

Noise temperature | [TeX:] $$T_{\mathrm{R}}$$ = 517 K |

User link bandwidth | [TeX:] $$B_{\mathrm{ul}}$$ = 500 MHz |

Satellite antenna gain | 52 dBi |

User antenna gain | 41.7 dBi |

Beam diameter | 250 km |

3 dB angle | 0.4◦ |

Rain attenuation | (μ, σ) = (−3.125, 1.591) |

Roll-off factor | β = 0.20 |

Fig. 4 plots the average user throughput per beam vs. the maximum power of the antenna feed element. In each beam, there are two groups with two users per group, i.e., [TeX:] $$G$$ = 2 and [TeX:] $$K$$ = 2. For all two techniques, the throughput increases as a higher power is available. The proposed NOMA scheme has higher performance than the OMA scheme. The main reason for this performance difference is the number of users that can be covered simultaneously. NOMA employs SC to cover [TeX:] $$G$$ user groups in each beam at a time, whereas OMA covers one group at a time in each beam by dividing time with the TDMA technique. That is, during a one-time slot, NOMA covers GK users at the same time, while OMA covers K users. OMA needs [TeX:] $$G$$ times more timeslots to cover the same number of users as NOMA. For the NOMA scheme, each beamforming vector needs to be optimized for multiple groups of users because one beamforming vector is used for one beam. On the other hand, for OMA scheme, it needs to be optimized for only one group of users. In other words, the beamforming vector is obtained for [TeX:] $$G$$ times more user channel information than OMA in the case of NOMA. Nevertheless, NOMA still outperforms OMA.

^{1} The number of users per group was set by referring to other papers on multicast multibeam SatCom [14], [17], [18], [43], [44], and other technical reports on DVB-S2X such as [45]. According to [14], if the number of users per beam to be covered at once is too large, lower performance can be obtained than the conventional frequency reuse techniques without precoding. Therefore, if the number of earth stations that can be covered by a beam at once is exceeded, the users are divided so that the appropriate number of users is covered for every time slot. By properly dividing the users, the number of users per frame is not increased too much, and the number of users to be covered at one time per beam can be kept moderate.

As shown in Fig. 5, we set the number of groups in each beam as [TeX:] $$G$$ = 2 and show the average user throughput per beam for the number of users per group ([TeX:] $$K$$) in the beams, where the antenna feed element power is fixed at 100 W. As [TeX:] $$K$$ increases, [TeX:] $$R_{\text {avg }}$$ decreases in all two techniques because the precoding vector needs to be adjusted for more multigroup multicast channels. The gap between NOMA and OMA schemes also decreases. However, the NOMA scheme still outperforms the OMA scheme. This is because NOMA covers more users simultaneously, even if the difference in the number of users sharing the same precoding vector between NOMA and OMA gradually increases.

Furthermore, the average user throughput per beam with respect to the number of groups per beam is investigated. Each group has two users and the antenna feed element power is set to 100 W. In Fig. 6, as the number of groups increases from 1 to 4, the NOMA scheme can transmit more signals simultaneously, resulting in an increase in the throughput. However, in the case of the OMA scheme, signals can be transmitted to only one group per time slot, so the number of time slots required increases in proportion to the number of groups. Thus, its throughput remains constant regardless of the number of groups.

The accurate channel information may not be obtained due to limited feedback or channel estimation error. It is important to investigate the impact of imperfection in our proposed scheme since the channel imperfection might degrade the performance more severely in our system requiring channel information of more multigroup multicast users. To consider those situations, we examine the performance even in the cases of obtaining channel information of various qualities. We model the imperfect CSI at the gateways as follows, [TeX:] $$\mathbf{h}=\hat{\mathbf{h}}+\tilde{\mathbf{h}}$$, where [TeX:] $$\hat{\mathbf{h}}$$ and [TeX:] $$\hat{\mathbf{h}}$$ are the channel estimate of the channel vector [TeX:] $$\mathbf{h}$$ and the channel estimation errors, respectively. The channel estimation error is independent and identically distributed and independent of the channel estimate. It follows a complex Gaussian distribution [TeX:] $$\mathcal{C N}\left(0, \sigma_e^2\right)$$, where [TeX:] $$\sigma_e^2=(B P)^{-\delta}$$ [43], [44]. A larger [TeX:] $$\delta$$ value means more accurate channel information and a value of [TeX:] $$\delta$$ between 0 and 1 is mainly selected. Additionally, we also investigate the effect of imperfect SIC. In order to calculate the residual interference due to the imperfect SIC, the error ratio [TeX:] $$\varepsilon$$ > 0 is introduced and the performance is calculated for several error ratio values [TeX:] $$\varepsilon \in$$ [0,1], where [TeX:] $$\varepsilon$$ = 0 denotes perfect SIC [46]. The SINR to which the imperfect SIC is applied can be written as where [TeX:] $$\bar{a}_{l b, u m}=\varepsilon \text { if } a_{l b, u m}=0, \bar{a}_{l b, u m}=1$$ otherwise. In Figs. 7 and 8, we compare the proposed NOMA scheme with the OMA scheme in the context of considering the imperfect CSI and imperfect SIC. Fig. 7 shows the average user throughput per beam according to the channel quality, where [TeX:] $$\delta$$ = [0.4,0.6,0.8]. As the quality of channel infor- mation decreases, the average throughput of both schemes decreases, but the proposed NOMA scheme still shows higher performance than the OMA scheme. Fig. 8 plots the effect on the imperfect SIC, where [TeX:] $$\varepsilon=\left[10^{-2}, 10^{-1.5}, 10^{-1}\right]$$. As the error ratio increases, the performance of the proposed NOMA scheme decreases. In the high-power region, the proposed NOMA scheme shows lower throughput than the OMA scheme. This is because residual interference due to the imperfect SIC also increases as the transmit power increases. However, the proposed method can be said to be robust to SIC capability because the antenna feed element power is mainly considered around 100 W in realistic SatCom systems.

Finally, we examine the convergence of Algorithm 1 over a channel realization where we set [TeX:] $$G$$ = 2, [TeX:] $$K$$ = 2, and P=50W. As shown in Fig. 9, we can observe that the objective function value increases and converges after some iterations.

In this work, we investigated the application of NOMA to a multibeam multicast satellite system with multiple gateways. The precoding vector, power allocation factor, and decoding order were optimized to reduce inter-beam interference due to full frequency reuse and improve spectral efficiency. We formulated the problem to maximize the sum rate under power constraints. After the optimization problem was converted into a tractable form, we could find the local optimal solution using the developed algorithm. Simulation results have demonstrated that the proposed NOMA scheme outperforms the OMA method.

When the system of this paper is handled in low earth orbit (LEO) SatCom, the channel model should be considered differently from GEO SatCom. Although LEO SatCom has a shorter propagation delay than GEO SatCom, it experiences a Doppler shift due to a different orbital period than the Earth’s rotation period. If the Doppler shift is predicted and compen- sated effectively as in [47]–[49], the technique presented in this paper can also be applied to LEO SatCom.

Dong-Hyoun Na (S’21) received the B.S. and M.S. degrees in Electrical Engineering from Korea University, Seoul, South Korea, in 2017 and 2019, respectively, where he is currently pursuing the Ph.D. degree with the School of Electrical Engineering. He is also pursuing the Ph.D. degree with the Division of Computer, Electrical, Mathematical Science and Engineering (CEMSE), King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. His current research interests include performance analysis and optimization of satellite communication systems.

Ki-Hong Park (S’06-M’11-SM’19) received the B.Sc. degree in Electrical, Electronic, and Radio Engineering from Korea University, Seoul, South Korea, in 2005, and the joint M.S. and Ph.D. degrees from the School of Electrical Engineering, Korea University, in 2011. He joined the King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia, in 2011, as a Post-Doctoral Fellow. Since 2014, he has been a Research Scientist of Electrical and Computer Engineering with the Division of Computer, Electrical, Mathematical Science and Engineering (CEMSE), KAUST. His research interests include communication theory and its application to the design and performance evaluation of wireless communication systems and networks. His research interests include the application to optical wireless communications, nonterrestrial communications, and physical layer secrecy.

Young-Chai Ko (S’97-M’01-SM’06) received the B.Sc. degree in Electrical and Telecommunication Engineering from the Hanyang University, Seoul, Korea and the M.S.E.E. and Ph.D. degrees in Electrical Engineering from the University of Minnesota, Minneapolis, MN in 1999 and 2001, respectively. He was with Novatel Wireless as a Research Scientist from January 2001 to March 2001. In March 2001, he joined the Texas Instruments, Inc., wireless center, San Diego, CA, as a Senior Engineer. He is now with the school of Electrical Engineering at Korea University as a Professor. His current research interests include the design and evaluations of multi-user cellular system, MODEM architecture, mm-wave and Tera Hz wireless systems.

Mohamed-Slim Alouini (S’94-M’98-SM’03-F’09) was born in Tunis, Tunisia. He received the Ph.D. degree in Electrical Engineering from the California Institute of Technology (Caltech), Pasadena, CA, USA, in 1998. He served as a Faculty member at the University of Minnesota, Minneapolis, MN, USA, then in the Texas A M University at Qatar, Education City, Doha, Qatar before joining King Abdullah University of Science and Technology (KAUST), Thuwal, Makkah Province, Saudi Arabia as a Professor of Electrical Engineering in 2009. His current research interests include modeling, design, and performance analysis of wireless communication systems.

- 1 G. Giambene, S. Kota, and P. Pillai, "Satellite-5G integration: A network perspective," IEEE Netw., vol. 32, no. 5, pp. 25-31, Sep. 2018.doi:[[[10.1109/MNET.2018.1800037]]]
- 2 S. Dang, O. Amin, B. Shihada, and M.-S. Alouini, "What should 6G be?"
*Nat. Electron.*, vol. 3, no. 1, pp. 20-29, Jan. 2020.doi:[[[10.1038/s41928-019-0355-6]]] - 3 E. Yaacoub and M.-S. Alouini, "A key 6G challenge and opportunity—Connecting the base of the pyramid: A survey on rural connectivity,"
*Proc. IEEE*, vol. 108, no. 4, pp. 533-582, Apr. 2020.doi:[[[10.1109/JPROC.2020.2976703]]] - 4 M. Giordani and M. Zorzi, "Non-terrestrial networks in the 6G era: Challenges and opportunities,"
*IEEE Netw.*, vol. 35, no. 2, pp. 244-251, Mar. 2021.doi:[[[10.1109/MNET.011.2000493]]] - 5 M. A. Vazquezetal., "Precoding in multibeam satellite communications: Present and future challenges,"
*IEEE Wireless Commun.*, vol. 23, no. 6, pp. 88-95, Dec. 2016.doi:[[[10.1109/MWC.2016.1500047WC]]] - 6 V . Joroughi et al., "Deploying joint beam hopping and precoding in multibeam satellite networks with time variant traffic,"
*in Proc. GlobalSIP*, 2018.doi:[[[10.1109/GlobalSIP.2018.8646361]]] - 7 M. G. Kibria, E. Lagunas, N. Maturo, D. Spano, and S. Chatzinotas, "Precoded cluster hopping in multi-beam high throughput satellite systems,"
*in Proc. IEEE GLOBECOM*, 2019.doi:[[[10.1109/GLOBECOM38437.2019.9013589]]] - 8 A. M. K., "Channel estimation and detection in hybrid satellite-terrestrial communication systems," IEEE Trans. Veh. Technol., vol. 65, no. 7, pp. 5764-5771, Jul. 2016.doi:[[[10.1109/TVT.2015.2456433]]]
- 9 G. Taricco, "Linear precoding methods for multi-beam broadband satellite systems,"
*in Proc. European Wireless Conf.*, 2014.custom:[[[https://ieeexplore.ieee.org/document/6843054]]] - 10 M. A. Diaz, N. Courville, C. Mosquera, G. Liva, and G. E. Corazza, "Non-linear interference mitigation for broadband multimedia satellite systems,"
*in Proc. IWSSC*, 2007.doi:[[[10.1109/IWSSC.2007.4409391]]] - 11 S. Chatzinotas, G. Zheng, and B. Ottersten, "Energy-efficient MMSE beamforming and power allocation in multibeam satellite systems,"
*in Proc. ASILOMAR*, 2011.doi:[[[10.1109/ACSSC.2011.6190179]]] - 12 C. Qi and X. Wang, "Precoding design for energy efficiency of multibeam satellite communications," IEEE Commun. Lett., vol. 22, no. 9, pp. 1826-1829, Sep. 2018.doi:[[[10.1109/LCOMM.2018.2855935]]]
- 13 G. Zheng, S. Chatzinotas, and B. Ottersten, "Generic optimization of linear precoding in multibeam satellite systems," IEEE Trans. Wireless Commun., vol. 11, no. 6, pp. 2308-2320, Jun. 2012.doi:[[[10.1109/TWC.2012.040412.111629]]]
- 14 D. Christopoulos, S. Chatzinotas, and B. Ottersten, "Multicast multigroup precoding and user scheduling for frame-based satellite communications," IEEE Trans. Wireless Commun., vol. 14, no. 9, pp. 4695-4707, Sep. 2015.doi:[[[10.1109/TWC.2015.2424961]]]
- 15 V . Joroughi, M. ´ A. V´ azquez, and A. I. P´ erez-Neira, "Precoding in multigateway multibeam satellite systems," IEEE Trans. Wireless Commun., vol. 15, no. 7, pp. 4944-4956, Jul. 2016.doi:[[[10.1109/TWC.2016.2550023]]]
- 16 C. Mosquera, R. L´ opez-Valcarce, T. Ram´ ırez, and V . Joroughi, "Distributed precoding systems in multi-gateway multibeam satellites: Regularization and coarse beamforming,"
*IEEE Trans. Wireless Commun.*, vol. 17, no. 10, pp. 6389-6403, Oct. 2018.doi:[[[10.1109/TWC.2018.2859410]]] - 17 D. Christopoulos, H. Pennanen, S. Chatzinotas, and B. Ottersten, "Multicast multigroup precoding for frame-based multi-gateway satellite communications,"
*in Proc. ASMS/SPSC*, 2016.doi:[[[10.1109/ASMS-SPSC.2016.7601466]]] - 18 J. Wang, L. Zhou, K. Yang, X. Wang, and Y . Liu, "Multicast precoding for multigateway multibeam satellite systems with feeder link interference,"
*IEEE Trans. Wireless Commun.*, vol. 18, no. 3, pp. 1637-1650, Mar. 2019.doi:[[[10.1109/TWC.2019.2894823]]] - 19 S. M. R. Islam, N. Avazov, O. A. Dobre, and K. Kwak, "Power-domain non-orthogonal multiple access (NOMA) in 5G systems: Potentials and challenges," IEEE Commun. Surveys Tuts., vol. 19, no. 2, pp. 721-742, 2nd Quart. 2017.doi:[[[10.1109/COMST.2016.2621116]]]
- 20 Y . Liu et al., "Nonorthogonal multiple access for 5G and beyond,"
*Proc. IEEE*, vol. 105, no. 12, pp. 2347-2381, Dec. 2017.doi:[[[10.1109/JPROC.2017.2768666]]] - 21 M. Caus, M. A. V´ azquez, and A. P´ erez-Neira, "NOMA and interference limited satellite scenarios,"
*in Proc. ACSSC*, 2016.doi:[[[10.1109/ACSSC.2016.7869089]]] - 22 X. Yanetal., "The application of power-domain non-orthogonal multiple access in satellite communication networks,"
*IEEE Access*, vol. 7, pp. 63531-63539, May 2019.doi:[[[10.1109/ACCESS.2019.2917060]]] - 23 X. Yan et al., "Performance analysis of NOMA-based land mobile satellite networks," IEEE Access, vol. 6, pp. 31327-31339, Jun. 2018.doi:[[[10.1109/ACCESS.2018.2844783]]]
- 24 S. A. Tegos, P. D. Diamantoulakis, J. Xia, L. Fan, and G. K. Karagiannidis, "Outage performance of uplink NOMA in land mobile satellite communications,"
*IEEE Wireless Commun. Lett.*, vol. 9, no. 10, pp. 1710-1714, Oct. 2020.doi:[[[10.1109/LWC.2020.3001916]]] - 25 X. Zhu, C. Jiang, L. Kuang, N. Ge, and J. Lu, "Non-orthogonal multiple access based integrated terrestrial-satellite networks,"
*IEEE J. Sel. Areas Commun.*, vol. 35, no. 10, pp. 2253-2267, Oct. 2017.doi:[[[10.1109/JSAC.2017.2724478]]] - 26 Z. Lin, M. Lin, J. Wang, T. de Cola, and J. Wang, "Joint beamforming and power allocation for satellite-terrestrial integrated networks with non-orthogonal multiple access," IEEE J. Sel. Topics Signal Process., vol. 13, no. 3, pp. 657-670, Jun. 2019.doi:[[[10.1109/JSTSP.2019.2899731]]]
- 27 A. I. Perez-Neira, M. Caus, and M. A. Vazquez, "Non-orthogonal transmission techniques for multibeam satellite systems,"
*IEEE Commun. Mag.*, vol. 57, no. 12, pp. 58-63, Dec. 2019.custom:[[[-]]] - 28 S. M. Ivari et al., "Power allocation and user clustering in multicast NOMA based satellite communication systems,"
*in Proc. IEEE ICC*, 2020.doi:[[[10.1109/ICC40277.2020.9149308]]] - 29 S. M. Ivari, et al., "Precoding and scheduling in multibeam multicast NOMA based satellite communication systems,"
*in Proc. IEEE ICC Workshops*, 2021.doi:[[[10.1109/ICCWorkshops50388.2021.9473484]]] - 30 Y . Huang, et al., "Signal processing for MIMO-NOMA: Present and future challenges,"
*IEEE Wirel. Commun.*, vol. 25, no. 2, pp. 32-38, Apr. 2018.doi:[[[10.1109/MWC.2018.1700108]]] - 31 D. Hu, Q. Zhang, Q. Li, and J. Qin, "Joint position, decoding order, and power allocation optimization in UAV-based NOMA downlink communications," IEEE Syst. J., vol. 14, no. 2, pp. 2949-2960, Jun. 2020.doi:[[[10.1109/JSYST.2019.2940985]]]
- 32 P. Tseng, "Convergence of a block coordinate descent method for nondifferentiable minimization," J. Optim. Theory Appl., vol. 109, no. 3, pp. 475-494, Jun. 2001.doi:[[[10.1023/A:1017501703105]]]
- 33 V . Joroughi, M. ´ A. V´ azquez, and A. I. P´ erez-Neira, "Generalized multicast multibeam precoding for satellite communications,"
*IEEE Trans. Wireless Commun.*, vol. 16, no. 2, pp. 952-966, Feb. 2017.doi:[[[10.1109/TWC.2016.2635139]]] - 34 G. Zheng, P. Arapoglou, and B. Ottersten, "Physical layer security in multibeam satellite systems,"
*IEEE Trans. Wireless Commun.*, vol. 11, no. 2, pp. 852-863, Feb. 2012.doi:[[[10.1109/TWC.2011.120911.111460]]] - 35 M. Lin, Z. Lin, W. Zhu, and J. Wang, "Joint beamforming for secure communication in cognitive satellite terrestrial networks,"
*IEEE J. Sel. Areas Commun.*, vol. 36, no. 5, pp. 1017-1029, May 2018.doi:[[[10.1109/JSAC.2018.2832819]]] - 36 M. F. Hanif, Z. Ding, T. Ratnarajah, and G. K. Karagiannidis, "A minorization-maximization method for optimizing sum rate in the downlink of non-orthogonal multiple access systems,"
*IEEE Trans. Signal Process.*, vol. 64, no. 1, pp. 76-88, Jan. 2016.doi:[[[10.1109/TSP.2015.2480042]]] - 37 S. Boyd and L. Vandenberghe,
*Convex Optimization*. Cambridge University Press, 2004.custom:[[[https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf]]] - 38 K. Nguyen, Q. Vu, M. Juntti, and L. Tran, "Distributed solutions for energy efficiency fairness in multicell MISO downlink," IEEE Trans. Wireless Commun., vol. 16, no. 9, pp. 6232-6247, Jul. 2017.doi:[[[10.1109/TWC.2017.2721369]]]
- 39 H. Tuy,
*Convex Analysis and Global Optimization*. Springer, 2016.doi:[[[https://link.springer.com/book/10.1007/978-3-319-31484-6]]] - 40 Q. Vu, K. Nguyen, and M. Juntti, "Max-min fairness for multicast multigroup multicell transmission under backhaul constraints,"
*in Proc. IEEE GLOBECOM*, 2016.doi:[[[10.1109/GLOCOM.2016.7841981]]] - 41 T. Van Do, H. A. L. Thi, and N. T. Nguyen,
*Advanced Computational Methods for Knowledge Engineering*. Springer, 2014.custom:[[[https://link.springer.com/book/10.1007/978-3-319-00293-4]]] - 42 M. S. Lobo, l. Vandenberghe, S. Boyd, and H. Lebret, "Applications of second-order cone programming,"
*Linear Algebra Appl.*, vol. 284, no. 1-3, pp. 193-228, Nov. 1998.doi:[[[10.1016/S0024-3795(98)10032-0]]] - 43 L. Yin and B. Clerckx, "Rate-splitting multiple access for multigroup multicast and multibeam satellite systems,"
*IEEE Trans. Commun.*, vol. 69, no. 2, pp. 976-990, Feb. 2021.doi:[[[10.1109/TCOMM.2020.3037596]]] - 44 Z. W. Si, L. Yin, and B. Clerckx, "Rate-splitting multiple access for multigateway multibeam satellite systems with feeder link interference,"
*IEEE Trans. Commun.*, vol. 70, no. 3, pp. 2147-2162, Mar. 2022.doi:[[[10.1109/TCOMM.2022.3144487]]] - 45
*Digital video broadcasting (DVB); Implementation guidelines for the second generation system for broadcasting*, interactive services, news gathering and other broadband satellite applications. part 2: S2 extensions (DVB-S2X). ETSI TR 102 376-2 V1.1.1, European Broadcasting Union, Nov. 2015.custom:[[[https://www.etsi.org/deliver/etsi_tr/102300_102399/10237601/01.02.01_60/tr_10237601v010201p.pdf]]] - 46 Y . Fu, Y . Chen, and C. W. Sung, "Distributed power control for the downlink of multi-cell NOMA systems," IEEE Trans. Wireless Commun., vol. 16, no. 9, pp. 6207-6220, Sep. 2017.doi:[[[10.1109/TWC.2017.2720743]]]
- 47 L. You et al., "Massive MIMO transmission for LEO satellite communications,"
*IEEE J. Sel. Areas Commun.*, vol. 38, no. 8, pp. 1851-1865, Aug. 2020.doi:[[[10.1109/JSAC.2020.3000803]]] - 48 W. Wang et al., "Near optimal timing and frequency offset estimation for 5G integrated LEO satellite communication system,"
*IEEE Access*, vol. 7, pp. 113298-113310, Aug. 2019.doi:[[[10.1109/ACCESS.2019.2935038]]] - 49 S. Amiri and M. Mehdipour, "Accurate doppler frequency shift estimation for any satellite orbit,"
*in Proc. RAST*, 2007.doi:[[[10.1109/RAST.2007.4284064]]]