Traffic Engineering and QoS/QoE Supporting Techniques for Emerging Service-Oriented Software-Defined Network

I. INTRODUCTION

THE quality of service (QoS) management and traffic engineering in communication systems based on a serviceoriented architecture (SOA) paradigm are very important issues to maximize user satisfaction and service provider profit [1]. These days, SOA systems that use the Internet as the communications bus face the technical challenge of guaranteeing QoS based on service response time criteria [2]. Since total service response time consists of the duration of request processing on application servers and network delay introduced by switches and routers, the task of delivering the requested service response time requires proper management of both communication and computing resources. Ensuring effective end-to-end resource and QoS management in such complex heterogeneous network scenarios requires adaptive, and scalable solutions to integrate and coordinate different QoS mechanisms [3]. Achieving user requirements at the lowest operational cost is the essence of building and sustaining future networks [4]. One such problem is adaptive resource management and traffic engineering in network devices. This problem requires elastic solutions based on estimating the state of network nodes depending on the amount of input load in order to perform proper balancing and efficient resource allocation [5]. The use of software-defined network (SDN) technology provides high flexibility in the process of infrastructure management and simplifies the virtualization of network resources [6]. Dynamic configuration of the network through the controller without changing the hardware and software of network devices has led to the fact that today most telecom network operators partially implement this technology, but traffic engineering algorithms and data transmission methods remain virtually unchanged. In this regard, the issue of traffic engineering and QoS support, in accordance with the ordered quality of experience (QoE) requirements of users, remains relevant today. The QoE measurement can be carried out at different scales and include different units of measurement. It can be measured on a qualitative or quantitative scale. For the calculation of the QoE parameter “user satisfaction”, as an example, an ordered (qualitative) scale is used, which includes scores from 1 to 5, where 1 means poor quality and 5 means excellent quality [7]. The QoE gives a broader view of how quality is achieved from an end-user perspective, compared to the more specific view of individual network parameters (bandwidth, delay, loss) that QoS provides. However, QoS has a significant impact on QoE, so in emerging network systems, it is necessary to consider them comprehensively to ensure user requirements.

Thus, the steady growth of diversity and volume of information flows in telecommunication networks prompts the solution to the scientific and practical problem of ensuring the ordered users QoE in conditions of limited network resources by developing a new architecture of service-oriented softwaredefined network (SOSDN). The architecture is based on improved methods and algorithms for the adaptive prioritization of services, server selection, and data transmission route.

The novelty and the main contributions of the work are as follows:

1) We proposed SOSDN architecture to provide coordinated end-to-end QoS in networking and computing domains taking into account individual QoE requirements;

2) We developed an adaptive algorithm for the prioritization of information flows, which, by automating the management at the SDN controller level, dynamically changes the priorities of network flows in case of degradation of the QoS parameters of traffic during transmission through the network to maintain the ordered QoE for individual users.

3) We developed a method of adaptive multi-criteria routing of data flows in SOSDN, which uses an adaptive integral metric for each class of information flows. This metric is based on the measured QoS parameters that characterize each branch of the network topology in the process of its functioning and the necessary requirements for ensuring the QoE of users according to the concluded SLA contract.

4) We proposed an algorithm for solving a multi-criteria optimization problem using an integral criterion to select the optimal data server to support the QoS/QoE of users.

The remainder of the article is organized as follows. Section II presents a brief review of the related works. Section III describes the proposed SOSDN paradigm with adaptive usercentric QoS and resource management. Section IV describes the developed adaptive algorithm for services prioritization. Section V presents the method of adaptive multi-criteria routing of data flows for SOSDN. Section VI describes an algorithm for solving the multi-criteria optimization problem to determine the optimal service server using the integral additive criterion. Section VII demonstrates the simulation results of the proposed solutions. Finally, we conclude in Section VIII.

II. RELATED WORK

This review of work aims to identify and discuss existing approaches to providing QoS in SDN and analyze whether they meet the specific requirements necessary to form a personalized quality of information service delivery in SOAbased SDN.

The paper [8] reviews most of the research on SDN to support end-to-end QoS guarantees, using SDN as a stand-alone system that can provide and enhance QoS functionality. According to the review and analysis in this paper, there are many areas of research where autonomous QoS provisioning can be further improved. For example, SDN-based network monitoring has been investigated in the literature without measuring path or communication delays [9]. In addition, little or no work has been done on QoS routing based on user requirements [10]–[12]. Machine Learning techniques can also be used to build smart QoS routing as well as more powerful analysis functions for autonomous QoS provisioning [10], [13], [14].

In [15], Lee et al. proposed QoS methods based on queue scheduling to ensure the quality of service of cloud applications in SDN. With this approach, applications are defined, and the necessary levels of QoS are established for each type of application. It implements queue scheduling techniques to exclude delay-sensitive data from the queue and send it first. There are three main modules for message control and management during the system design phase. The control message module forwards messages according to flow table rules, and the queue management module configures queues based on configuration information. The queue scheduling module assigns packets from the queues with different priorities. This approach has been evaluated through experimental and theoretical analysis. According to the theoretical analysis, this method can provide differentiated application flow services and provide different levels of QoS. The results showed that if the source interface bandwidth is sufficient, the delay can be reduced by an average of 28

Durner et al. [ 16] determine the impact on network traffic when dynamic QoS mechanisms are applied in SDN switches with OpenFlow support. In the study, the measurements are accompanied by two fundamentally different QoS techniques, named Priority Queue and Bandwidth Guaranteeing Queue. The result shows a marked difference in performance for the different OpenFlow-enabled switches. In addition, different queue implementations, i.e., FIFO queue or SFQ queue, significantly affect network performance. One of the most effective approaches to routing flows in SDN was proposed in [ 17]. The authors developed and implemented a centralized deterministic multicriteria QoS (DMCQR) model in Mininet for SDN research. The experiment results show that the proposed DMCQR algorithm has the best performance in terms of efficient channel load factor utilization, packet loss minimization, bandwidth, and delay compared to the traditional algorithms and the proposed method of multi-path routing for SDN [ 18]. Having considered several routing works, they still have one thing in common: the inability to take into account the changing intentions of users regarding the ordered QoS level. An adaptive flow partitioning scheme has been developed in the paper [ 19] to achieve multipath transmission under dynamic network state changes, using the rule timeout mechanism that comes with the OpenFlow protocol. The authors proposed using a graph neural network to predict link delay to aid in adaptive flow partitioning and intelligent forwarding path selection. Results from the simulations show that the graph neural network has good convergence and generalizability in delay prediction, and the rule timeout-based flow partitioning mechanism implements adaptive flow partitioning according to changes in the network state. This scheme generally outperforms existing generic solutions in terms of processing time, end-to-end delay, flow completion time, and throughput.

Moreover, SDN does not have a unique measurement methodology that can confirm the level of satisfaction with the services provided [20]–[22]. As a result, there is always a gap between the service provider and the user to have service satisfaction information taken into account when deciding on a service provider, which creates a lack of transparency between service providers and consumers [23]. The above comparative studies also demonstrate the absence of a unique QoS standard in SDN that can be adapted to an SOA-based SDN architecture [24]. The paper [25] finds that there is still not enough serious work on applying autonomy features, such as self-management or policy-based QoS management, without human involvement. In the next section, we explain their importance using a use case scenario where a service user enters into an SLA agreement with a service provider.

After analyzing and discussing the above relevant existing research to develop a future SOA-based SDN architecture, the following weaknesses were identified that need to be addressed:

- Do not take into account the requirements of an individual customer and differentiate flows only by traffic classes;

- Cannot automatically cope with the overload of network elements, nodes, or channels in case of QoS degradation of individual flows;

- In practice, flows are routed without differentiating them by the QoS criteria and based on open shortest path first (OSPF) metrics. In most cases, known improvements to routing solutions are theoretically or analytically based without evidence of their effectiveness on real SDN network equipment;

- Lack of a unified system for monitoring the parameters of the telecommunications network, both computing and networking domains;

- A lack of accurate methods for monitoring the QoS parameters between the input and output node of an arbitrary flow.

As such, to ensure maximum QoS provisioning in an SOA-based SDN, a key requirement for requesting applications is to follow a consistent process to help them negotiate, select, monitor, and manage the right type of network with providers who can meet their specific requirements [26]–[28]. If this process is not followed, there is a very high probability that the network QoS provided will not meet the requirements [29]–[31]. Implementing this process to provide QoS in SOA-based SDN requires a paradigm shift in how the network as a service is shaped and managed compared to how it is done in the traditional SDN. In this case, we propose usercentric traffic engineering and QoS/QoE support techniques in emerging SOSDN networks to provide adaptive end-to-end QoS in the networking and computing domain by prioritizing, routing and selecting servers with coordination between these two domains.

III. SOSDN ARCHITECTURE FOR PROVIDING COORDINATED END-TO-END QOS IN NETWORKING AND COMPUTING DOMAINS

The main idea of the work is to develop a SOSDN architecture for coordinated end-to-end QoS in networking and computing domains, taking into account QoE requirements. Our main solutions in the field of traffic engineering and QoS/QoE support methods for new SOSDN are shown in the green plane in Fig. 1. The red arrows show the process of exchanging information about the status of the server and network domains to the SDN controller. Unlike existing SOA solutions, in which the resource management policy at the level of network devices and at the level of computing servers is not coordinated, the proposed approach in this paper allows centralizing these two levels by introducing SDN controller for coordinated adaptive resource management and QoS/QoE support based on improved traffic engineering techniques, such as adaptive prioritization of services, server selection, and QoS/QoE-based routing.

In particular, the Orchestrator monitors the status of the servers and directly decides on the optimal server for a particular service. To ensure the interconnection between the server domain and the network domain, the Orchestrator transmits information to the SDN controller about the selection of the optimal server. Based on this information, the controller can dynamically change data transmission routes to meet individual QoS/QoE requirements of users. Conversely, the controller can send command information to the Orchestrator to reselect a server for a particular type of service, if necessary. The concept provides for the possibility of locating servers both within the local corporate network and the possibility of replicating servers of the various services in the cloud. The servers must be under the control of the operator offering this concept.

In this way, orchestrators and SDN controllers work together to provide efficient network and server resource management. The orchestrator is responsible for the broader context of resource management, including compute resources, virtualization, and application orchestration, while the SDN controller specializes in managing network and data flows at the switch level.

Thus, the proposed integrated SDN/SOA network will allow end-users to make a QoE request from end to end for any service at any given time. This approach is organized by establishing QoE scores by network users, put on a scale of 1 to 5 in the SLA contract. The higher the score, the better the QoS is guaranteed, and the more expensive the provision of this service will be. Practical implementation of this approach can be performed by installing software on the end user’s device (so-called user’s cabinet), where he can request the service of the required quality and make the additional payment. Using this software product, the network adapts to user needs by analyzing requests for the required quality of a certain type of service at a certain time from users.

The SDN/SOA network controller will analyze the requests. By centralizing the state of the network at the control level, the system guarantees the necessary level of service by analyzing the QoE scores of users by dynamically prioritizing services according to the ordered QoE scores, channel resources in the network nodes, load balancing on the servers and developing new routing protocols, based on the selection. optimal node service, analysis, and assessment of network characteristics in real-time. For example, let’s look at various situations that can occur during data transmission in networks and affect QoS.

To provide a guaranteed QoS from end to end, the condition must be satisfied:

where [TeX:] $$Q o S_{E 2 E \text { admissible }}(x)$$ is the acceptable value of the QoS in the telecommunications network according to the SLA and defined as the sum of the acceptable QoS parameters provided by the network domain and the server domain; [TeX:] $$\operatorname{QoS}_{E 2 E}(x)$$ the current QoS provided by the network to the end-user.

The current end-to-end QoS score is determined according to the following formula:

where [TeX:] $$Q o S_{\text {Network }}(x)$$ is QoS, which is provided in the network domain by criterion x, where x can be a delay, packet loss, jitter, available bandwidth, or a complex integral index; [TeX:] $$Q o S_{\text {server }}(x)$$ is QoS, which is provided in the server domain by criterion x, where the role of x can be average request processing time (delay), probability of losing requests, CPU, RAM and network interface loading or a complex integral index.

Accordingly, to provide condition 2, each of the domains must provide the necessary acceptable parameters of the QoS (delay and packet loss). For this purpose, the condition must be satisfied:

According to the following conditions we see that if one of the domains does not provide QoS, it negatively affects the final QoS and as a consequence the users’ QoE.

In modern traditional networks, quite often there is a situation where two domains are functioning with a sufficiently high QoS, but the final QoS for the user is unsatisfactory. Such situations are due to the fact that the traffic of modern networks has self-similarity, characterized by some number of bursts with a relatively small average traffic level [32]–[34]. Due to such bursts of load, the QoS parameters also deteriorate: packet losses increase, and delays in passing through network nodes, which can be solved by logical redistribution of data flow. This in turn requires constant reconfiguration of the network, in real-time, which is impossible to implement in modern networks. Reconfiguration refers to carrying out traffic engineering (routing, reprioritization and load balancing).

As an example, consider a situation where a condition [TeX:] $$Q o S_{E 2 E}(t) \leq 100$$ ms must be met for a critical service according to the packet delay criterion. For example, the delay of packets in the network is [TeX:] $$Q o S_{\text {Network }}(t)=60$$ ms and the delay when the server calculates is [TeX:] $$Q o S_{\text {Server }}(t)=80$$ ms. The total end-to-end delay according to the expression (2) is 140 ms. In the case where the resources of the server machine are used to the maximum, the situation can be corrected by redistributing the traffic only on the network domain, in particular by choosing the best data transmission path according to the criterion of minimum delay, which will reduce the delay from 60 ms, for example, to an average of 20 ms. In this case, the total delay will not exceed the allowable delay of 100 ms from end to end. Conversely, if the best path under the delay criterion of [TeX:] $$Q o S_{\text {Server }}(t)=80$$ ms and [TeX:] $$Q o S_{\text {Server }}(t)=60$$ ms is chosen, you can remedy the situation by choosing a less busy server, which will provide less service delay, e.g., [TeX:] $$Q o S_{\text {Server }}(t)=20$$ ms. The structural and functional diagram of management decisions considered in the work on the implementation of a SOSDN with user-centric traffic engineering and QoS/QoE support techniques is shown in Fig. 2.

According to the analysis in Section 1, the main disadvantage of the existing management methods is the lack of information about the state of the network in real-time. Thus, the first necessary step according to Fig. 2 for carrying out adaptive network reconfiguration is the implementation of monitoring of parameters, which characterize the state of the nodes of the network domain and the server domain. According to the analyzed systems and monitoring methods, the collection of statistics in the existing server monitoring systems takes place with the help of agents installed on the network and server nodes to collect statistical information transmitted at certain intervals to the monitoring center, the socalled sFlow collector [35]. In addition, the openflow protocol allows more parameters to be monitored [36].

The second necessary step is the implementation of an algorithm for adaptive prioritization of information flows in the network. This algorithm, unlike the known ones, allows the SDN controller to automatically change the priority of services provided in the network operation mode according to the needs of individual users according to the concluded SLA contract while taking into account the requirements transmitted in the network. Using the adaptive prioritization of flows will improve the QoS to users at times of need. If the priority is set to the maximum, the stream will have the smallest delay but the largest impact on other streams, and vice versa.

The third step is an algorithm for determining the optimal server to serve the information flows, the choice of which is based on an integral additive criterion describing the state of the server, relative to the ability to handle the required number of requests with the required level of QoS.

The fourth step is a method of adaptive routing of data flows, the metric for determining the optimal path which is based on a multi-criteria analysis of the state of operation of channels and network nodes regarding the current QoS of information flows by solving the problem of multi-criteria optimization using integral additive criterion.

The fifth step is the adaptive formation of weighting coefficients for the criteria by which the optimization occurs at a particular point in time, both for the network and for the server domain, taking into account the acceptable requirements for providing the required ordered quality by users for different types of information services. Accordingly, the last sixth step is the establishment of appropriate rules in the flow table to ensure the ordered QoS.

IV. ADAPTIVE PRIORITIZATION ALGORITHM FOR SERVICES IN SOSDN

Taking into account the limited network resources (buffer space on network nodes, bandwidth capacity of communication channels, design capacity and the time of making the managerial decisions of the SDN controller) telecommunications networks must have effective mechanisms to ensure QoS.

However, the actual task, the solution of which is equally important at the planning stage and at the stage of adaptive management of network resources, is to prioritize classes of traffic in accordance with the levels of QoS. Prioritization involves the distribution of different types of traffic by tightening the requirements on the amount of network resources required to ensure the QoS.

In modern software-defined networks, prioritization of classes of traffic is mainly based on the relevant ITU-T recommendations or the default without priority service. In this case, there is a situation where it is impossible to flexibly manage the resources of service nodes to avoid congestion and improve QoS parameters when the number of traffic classes is not limited to the established ITU-T five varieties and different user needs, especially without considering the possibility to adapt them to changing user needs regarding the importance of service at the current time.

Therefore, the actual task is to develop an algorithm for prioritizing information flows according to the criteria of QoS and importance to the user, because they are the main indicators of the effectiveness of planning and management of network resources in SOSDN.

In this paper, we have developed an adaptive algorithm for prioritizing information flows. This algorithm, by automating control at the SDN controller level, allows to dynamic reprioritizing of network flows in case of degradation of quality parameters of traffic during its transmission over the network to maintain the ordered QoS of specific users.

The algorithm is based on the calculation of the relative flow priority. For this purpose, at the initial stage of operation of the SOSDN the incoming parameters are determined, which are formed on the basis of both service QoS requirements, service demand factor (the so-called priority for the user group), and the priority of the individual user and the network provider policy under the concluded SLA contract.

The next step after obtaining the relative priorities for each type of traffic is to assign to each service a code combination that corresponds to the value of the criterion for this particular class of traffic. After that, the obtained compositions are fixed in the corresponding fields of network protocols type of service (ToS), and differentiated services code point (DSCP). In recommendations, the RFC-2474 ToS field is replaced by the DSCP field, where the lowest 6 bits determine the differentiated services (DS) code and the highest two bits are not yet defined and are subject to zero. We can also use these two bits to indicate the critical priority in cases where the relative priorities of two services are the same.

The algorithm can be used both for differentiating the order of traffic flows in SDN networks using different network protocols, adapted to indicate the priority level in the packet flow headers, and for implementing adaptive multitrack routing of flows.

Table I presents the requirements of telecommunication services to QoS. To date, distinguish a large number of telecommunications services, including the Internet of Things, which have the same QoS requirements, for a more detailed understanding of the alphabet will correspond to a set of their services with the same QoS requirements. Accordingly, A is real-time voice data, automated IoT emergency calls; B is video conferencing, distance learning, IoT real-time monitoring data; C is real-time video, IPTV, video Internet of video things (IoVT); D is Internet data (HTTP, online services), IoT notification (photo/text/email); E is a video on demand unreal-time VoD, IoVT from surveillance cameras; F is a 720p unreal-time video.

Relative coefficients of QoS parameters are calculated by formulas (5-8) as the ratio of minimum values of QoS parameters to the current values obtained by monitoring the network.

Relative packet loss ratio:

Relative packet delay ratio:

Relative jitter ratio:

Relative bandwidth ratio:

We form the importance coefficients of services filling the numbers 1, 2, and 3, which represent respectively low, medium, and high importance of the requirements for QoS parameters (Table II).

For each of the quality parameters QoS we introduce relative coefficients of the importance of the parameter relative to others [TeX:] $$Z_j(P p, P t, P j, P c)$$ which can be changed by the SDN network controller according to the SLA contract. These coefficients are used to weigh the significance of each parameter in the context of the SDN. It’s important to note that the specific values for these coefficients are not universal and can vary significantly from one network or application to another. They are intended to provide a way to tailor network management and resource allocation to meet the specific needs and expectations of the users or services being provided. Network architects and experts may use their experience and judgment to determine these coefficients based on the characteristics of the network and the expected traffic patterns. For example, based on our expert evaluation in concrete experiments these coefficients are defined as follows:

Pp = 0.23 is the relative coefficient of the significance of packet loss;

Pt = 0.18 is the relative coefficient of the significance of packet delay;

Pj = 0.14 is the relative coefficient of the significance of jitter;

Pc = 0.12 is the relative coefficient of the significance of bandwidth.

Moreover, the flow priority approach can take into account an additional parameter, such as the client priority for each traffic type according to the SLA contract. If the priority is not explicitly set in the service provisioning contract, the default is to assign the lowest possible priority to the client. Accordingly, the relative priority for each service category is calculated using the formula:

where [TeX:] $$\operatorname{Pr}_i$$ is the relative priority of the i-th flow type service; j is the number of the QoS parameter; [TeX:] $$X_{i j}$$ is the relative priority of parameter j for service i; [TeX:] $$Y_{i j}$$ is the importance of parameter j for service i.

The final result of the above formula is a fractional number whose value range is between zero and one. The greater the value, the higher the priority of the information flow. The formula can be used for any number of information flows and different QoS requirements. The algorithm for the adaptive prioritization of services is shown in Fig. 3.

The proposed algorithm makes it possible to create a unified and formalized approach to prioritizing the maintenance of various telecommunications network services, including Internet of Things services, required in the process of solving problems of efficient and flexible network resource management with adaptation to the needs of users.

V. ADAPTIVE MULTI-CRITERIA ROUTING METHOD OF DATA FLOWS FOR SOSDN

One of the powerful methods of influencing the efficient use of network resources is traffic engineering (TE) technology, which refers to methods and mechanisms to achieve a balanced utilization of all network resources through the rational choice of paths for transferring traffic through the network [37]. An important issue in this context is the issue of routing. Routing, together with the elements of data flow control, fully determines the functioning of the entire network for a given quality and quantity of services provided. The lack of consideration in routing algorithms of additional network factors, of which there are more and more every day, indicates the need to improve routing protocols by analyzing and evaluating the characteristics of networks, taking into account the fuzzy nature of network evaluation indicators in routing protocols [38]. A multi-criteria approach to information flow routing is key to the proposed conceptual network model that allows the adaptive assignment of an end-to-end data forwarding path taking into account communication parameters (e.g.: packet loss, delay, jitter, available bandwidth, etc.). To achieve this goal, a SOSDN network architecture must continuously monitor and control key network performance parameters to adapt resources as needed. The architecture can also find the best path (or more than one best path in the case of multiple flows) that can meet the needs of the flow using the mathematical model described below.

In this paper, we propose an improved method for routing flows that allows load balancing according to the criteria of maximum network channel utilization and QoS for each flow. For this purpose, the classification of flows according to the proposed service prioritization algorithm, discussed in section 4, is used. Accordingly, the routing task is formed as follows. The first category of service flows should be transmitted optimally according to the integral path cost criterion according to the metric proposed below, taking into account the greater importance of QoS parameters, namely, delay and jitter due to the correlation of criterion importance factors. Such flows are very sensitive to jitter, that’s why routing over several paths with load balancing is forbidden for them, i.e., the whole flow can be transmitted by one path only. The second category of flows is less sensitive to QoS time parameters, but sensitive to data loss, in which case their load balancing on a small number of alternative paths with a small load balancing factor is allowed. The formation of the integral path metric for the cost of forwarding such flows should be based on the greater importance of such criteria as packet loss and jitter, by changing the weighting coefficients in the route metric. Flows of the third category can be supplied in any way without guaranteeing QoS. Therefore, such flows are used for additional loading of sections of paths with low load, which for the above-mentioned flows are impossible in order to equalize the load distribution between channels throughout the network.

Suppose that when a particular path in a particular switch is overloaded, it becomes necessary to redistribute the flows. Given that first-category flows are sensitive to delays and jitter, redistributing these flows may degrade their QoS. Therefore, path offloading starts with redistributing third category flows. Moreover, after each iteration of flow reallocation, the time parameters of QoS are measured first for the first category and then for the second category. If the parameters are as required and the path is not congested, the redistribution algorithm completes its work. The load balancing algorithm for proposed routing is shown in Fig. 4. A metric is the cost of a route, describing a measure of its benefit. The route metric is the most important part of any routing protocol. The better the metric, the more likely it is that the network device will make the optimal decision to send a packet via the route it characterizes.

Modern routing protocols often rely on up to three basic criteria, as regulated by the ITU-T, to calculate routing metrics. These are the percentage of packets lost ratio (IPLR) in the end-to-end link between two devices in the network, the bandwidth, and the network packet transmission delay (IPTD). Each of these, for one reason or another, directly affects the other. These criteria provide only a general assessment of the communication channels, and changes in their values may be a consequence of the influence of other random factors. Consideration of such factors in the future will help to create a more detailed picture of the state of the links in the network and the entire system as a whole.

In the work on solving the multi-criteria optimization problem, the additive optimality criterion method is chosen. Accordingly, the solution of the multi-criteria optimization problem for path selection consists in that a set of criteria characterizing the communication channel, [TeX:] $$f i(x), i=1, n$$ are combined in the form of one integral criterion [TeX:] $$F(x)=\Phi(f i(x), f i(x), \ldots f n(x)) \text {, }$$ after which the maximum or minimum of this target function is found. The target function is constructed by adding the normalized values of the eigen criteria. In the general case the generalized target function on the basis of the integral additive criterion has the following form:

where n is the number of combined partial criteria; [TeX:] $$\alpha_i$$ is the weight coefficient of the i is the partial criterion; [TeX:] $$f_i(x)$$ is the numerical value of the i-th private criterion; [TeX:] $$f_i^0(x)$$ is the normalized divisor of the i-th partial criterion; [TeX:] $$f_i^H(x)$$ is the normalized value of the i-th partial criterion.

In most cases, the optimality criteria are characterized by different physical nature and therefore have different dimensions. Thus, their simple summation is incorrect. According to this statement in the previous formula, the numerical values of optimality criteria are normalized: The maximum (minimum) values of criteria attainable in the field of admissible solutions are taken as normalizing divisors. The dimensions of the eigenvalues of optimality criteria and corresponding normalizing divisors are the same. Accordingly, as a result, the integral additive criterion is a dimensionless value.

To obtain an informative picture of multicriteria routing for the conceptual model of the SDN network domain, we introduce an integral metric [TeX:] $$W_{(i, j)}$$, consisting of four criteria: the percentage of lost packets, bandwidth, jitter, and delay between nodes i, j.

Consider the structure of a communication network consisting of network N SDN devices. The numerical values of the communication channels metric [TeX:] $$W_{(i, j)}$$ can be represented in the form of an adjacency matrix A as [ 39]:

It should be noted that the matrix of parameters [TeX:] $$A^w$$ is not symmetric [TeX:] $$\left(w_{(i, j)} \neq w_{(i, j)}\right) .$$ Minimization of the target function [TeX:] $$W_{(i, j)}$$ on which several constraints or limit values can be imposed is called multicriteria optimization.

As normalizing divisors in this problem we take the best values of the eigen criteria;

- for the number of lost packets:

where [TeX:] $$N_{i, j}$$ is a number of received packets, [TeX:] $$M_{i, j}$$ is the number of transmitted packets and [TeX:] $$M_{i, j}\gt 0.$$

- for the flow delay:

where [TeX:] $$t_{\min }$$ is the minimum delay value in the adjacency matrix, and [TeX:] $$t_{\min }, t_{i, j}\gt 0$$

- for the channel bandwidth:

- for the flow jitter:

where [TeX:] $$j t_{\min }$$ is the minimum value of the jitter in the adjacency matrix, and [TeX:] $$j t_{\min }, j t_{i, j}\gt 0$$

The values of the integral additive criterion are calculated for each communication channel. Thus, the metric of communication channels based on 4 parameters will look like this:

The cost of a route implies the sum of the metrics of each channel. From the set of alternative routes, the one that has a lower cost (i.e., a lower value of the total metric) will be selected.

where [TeX:] $$X_1, X_2, X_3, X_4$$ are weight coefficients varying from 0 to 1, and their sum must be equal to 1:

Changing the value of weight coefficients in metric [TeX:] $$W_{(i, j)} \text {, }$$

we thereby create an apparatus for controlling the significance of one or another metric parameter in the final evaluation of the data transmission channel from node i to node j. From a mathematical point of view as an “ideal” point with coordinates corresponding to [TeX:] $$t_{\min }, P_{\min }, C_{\max }, j t_{\min },$$ with fixed weight coefficients [TeX:] $$X_1, X_{, 2}, X_3, X_4.$$ In reality, taking into account the dynamics of the whole communication system, the choice of weight coefficients depends on the nature of the primary types of traffic.

ITU-T Recommendation regulates five basic classes of QoS (IP QoS) to which the communication system can be attributed. Class 0 networks are the most demanding to the values of performance characteristics of communication channels. As a rule, these are networks providing real-time services, most sensitive to delays (VoIP, videoconferencing, online games, etc.). Currently, there is no unified concept for the construction of communication systems and networks to ensure the provision of all services with high quality.

In this regard, there is a need to develop a flexible mechanism for routing traffic of a separately selected service, taking into account the current capabilities of the communication system and the requirements for the quality of its provision. The QoS provision is inseparably connected with the main parameters, which appear in metric X. The main idea of the developed mechanism for providing flexible and efficient routing is to manipulate the weighting coefficients to achieve the best possible values of performance characteristics defined by the IP QoS class. In the realities of software-defined networks, the ability to implement such a mechanism is the most promising. The core of such a network is a control device known as a software controller, which is responsible for managing the network structure. In this case, it is a device capable of optimally selecting the weight coefficients of metric X. In this work, a block diagram of the developed routing method is presented, along with a description of the operation of its functional blocks (Fig. 5).

The work of the algorithm begins with the definition of the base values of the metric weight coefficients. For public systems, which require equal contribution of each of the primary metric parameters at a time, the base coefficients can be set equal in value. In parallel, the process responsible for the calculation of the primary characteristics that determine the state of all existing communication channels is started. The obtained results go to the block “Calculation of the values of metric”. In this block, according to the formula (16), the corresponding metric values for each channel are calculated. The values are written into the adjacency matrix [TeX:] $$A^w,$$ which is used to determine the cost of routes of routing tables.

The proposed method makes it possible to achieve continuous improvement of the quality of network operation by checking the current values of the operating characteristics of the network, which meet the QoS requirements of the network as a whole. In the variant of non-compliance of the values of operating characteristics with the specified requirements to the communication system, if necessary, it is possible to make adjustments to the values of weight coefficients, in order to adjust the current network topology to the maximum efficiency of operation.

Considering that changing the values can affect the current values of the primary characteristics of the communication channels, it is possible to recalculate the metrics, in order to keep the channel characteristics up-to-date. Decision-making on recalculation of primary network parameters and direct correction of metrics weighting values is the main task of the software-defined network controller. In reality, the selection of weighting coefficients values is not a trivial task, because, given the dynamics of IP-networks, there are no clear rules of its management, for example, in case of unpredictable situations (traffic surge).

The controller operation is an intelligent system, which, taking into account the current state of the network, is able to manage the entire network to achieve the necessary results of traffic transfer. The developed method provides the necessary tool to simplify the management of information and communication system by manipulating the weight coefficients of the metric (16), thereby reducing the tasks to the selection of their “best” values. The working principle of the proposed routing method in software-defined networks is shown in Fig.6. In particular, the red curve represents the established data transmission route between host 1 and host 2. In situations where the controller detects congestion or non-compliance with QoS criteria, it initiates a new data process. This process, using the QoS monitoring system, involves a thorough analysis of alternative routes - depicted by the dashed green curve - and the subsequent formation of a new set of rules in the flow table of switch A that regulate data transmission.

The use of the developed method of adaptive routing in the software-defined telecommunications infrastructure will ensure a guaranteed level of quality of end-to-end information service through dynamic management of network data streams. And also to choose optimal data transmission routes in case of faults or deterioration of one of the QoS parameters in SDN network switches for streams of different priority.

VI. ALGORITHM FOR SOLVING THE MULTI-CRITERIA OPTIMIZATION PROBLEM TO DETERMINE THE OPTIMAL SERVICE SERVER USING THE INTEGRAL ADDITIVE CRITERION

One of the approaches to improving QoS in the proposed conceptual model of the SOSDN telecommunications network is to reduce the load on the end service servers by using the developed adaptive traffic routing. This reduction in the load on the servers of the telecommunications network can be achieved by selecting the optimal service server on the basis of multi-criteria monitoring of important functioning parameters that characterize its dynamic state. In this case, if the load is found to be greater than the server can tolerate, it is necessary to redirect traffic to other servers, with which routing in the telecommunications network or to another virtual machine of the same type with a lower load. Fig. 7 presents a SOSDN architecture designed to coordinate the functioning of two domains in terms of their performance and enable adaptive resource and QoS management. By employing two algorithms for optimizing two objectives, namely, the best path for SDN data and the optimal service server, we amalgamate the decisions made by these two algorithms through the SDN network controller. Specifically, the Orchestrator monitors the state of servers and directly selects the optimal server for a specific service (as denoted by the red, green, and orange arrows representing data flows for three service classes).To establish a connection between the server domain and the network domain, the Orchestrator communicates the chosen optimal server information to the SDN controller. Based on this information, the controller can dynamically adjust data transmission routes to meet individual QoS/QoE user requirements. Conversely, the controller can send command information to the Orchestrator for reselecting a server for a specific type of service, if necessary.

The essence of the offered method of solution of multicriteria optimization for optimal server choice is similar, as well as for routing and consists in that set of criteria characterizing server functionality is combined into one integral criterion and then the maximum or minimum of this target function is found.

The algorithm for solving the multi-criteria optimization problem to determine the optimal data service server using the integral additive criterion is shown in Fig. 8.

According to this algorithm to solve this problem a utility function is constructed to determine the performance of the solution and the preference process is reduced to a comparison of the numbers-values. In this case, the SDN decision-making controller takes into account that one set of local criterion values has an advantage over others, if a larger value of the preference function corresponds to it. If the QoS deteriorates, the SDN controller performs weighting correction to adapt the server domain to provide the required level of QoS by matching it with the QoS level provided by the network domain.

Let’s consider an example of the problem to be solved to find the optimal server using the integral additive criterion. Partial criteria that characterize the functionality of the server at the current time are CPU load, RAM load, the duration of request processing, the probability of loss of requests and the load of the network interface of the server. In this case, the experiment was performed on the basis of the Azure platform, on which 3 separate servers with the same parameters were deployed and on which the same application was deployed. Each server was subjected to a different load under the influence of which the following parameters were evaluated: average request processing time (delay), probability of losing requests, CPU, RAM and network interface loading. The “probability of losing requests” is a criterion used to assess the performance and reliability of a server or a networked system. It refers to the likelihood that incoming requests or transactions will not be successfully processed or completed and, instead, will be lost or dropped. This parameter can be expressed by dividing the number of failed requests by the total number of received requests. Weighting factors were determined according to users’ requirements regarding the importance of a particular parameter’s impact on the quality of service. Accordingly, the best variant of the server will be the one with the lowest CPU load, RAM load, average request processing time, probability of losing requests and server network interface load in the process of monitoring. Initial data for solving the multi-criteria optimization problem are shown in Table III.

The objective function on the basis of the generalized additive criterion is written as follows:

As normalizing divisors in this problem to choose the optimal server we take the best minimum values of the eigen criteria, i.e.

Server 1.

server 2.

Server 3.

Thus, the 1st version of the server is optimal, because it corresponds to the maximum value of the generalized additive criterion.

VII. MODELING AND RESEARCH OF TRAFFIC ENGINEERING AND QOS/QOE SUPPORTING TECHNIQUES IN SOSDN

To study the effectiveness of the proposed traffic engineering and QoS/QoE supporting techniques we developed a simulation model of a SOSDN. The network is simulated using the Java programming language and the Mininet environment version 2.2.1. We used OpenDaylight to manage and control our network infrastructure. OpenDaylight is an open-source software-defined networking controller that provides a centralized platform for managing and automating network tasks. OpenDaylight provides a flexible and extensible platform for developing custom network management applications. SDN controller and Mininet run on the same host physical machine, but they are virtualized in a software environment. We are using an extended version of Mininet called Containernet, which allows deploying containers as hosts and servers. This feature is well-suited for building complex services that will run in containers on hosts and will be connected via the network emulated by the Mininet core. The proposed SOASDN testbed architecture is depicted in Fig. 9 a. The structural and functional diagram of the studied network consists of the following components:

- Monitoring system (based on SDN controller);

- Software-defined network devices (there are 16 switches with a link bandwidth of 100 mbit/s);

- SOA servers to access services (there are 6 servers deployed in Docker containers on a physical machine with an with CPU Intel CORE i5 running 2.5Ghz, 8M RAM memory);

- Personal computers of users (there are 50 users, including 10 IPTV users with individual QoE requirements with a score of 5 according to the SLA).

The central element of the investigated network is the controller, which is responsible for establishing the rules of service flows in the network based on monitoring of the main parameters of the quality of functioning of its elements. The developed automated monitoring system, deployed within the network controller, allows to identify of network bottlenecks without the help of a system administrator and to improves QoE evaluation of users demanding the ordered quality. The developed network is designed for 50 users and will allow to provide them with a good level of such services as VoIP, IPTV, Internet data, Media on Demand. Users use all 4 presented services in this proportion 10 VOIP users, 20 IPTV users, 10 Internet data users and 10 IPTV users.

In the conducted experiment, both synthetic traffic (VoIP) and real traffic (IPTV, VOD, Web) are used. For generating VoIP traffic, Iperf software is employed. By selecting two codecs (G.711, G.729), UDP packets were generated. To generate IPTV traffic, we used prepared video files with a duration of 60 seconds, employing the H.264 codec with a 720p resolution. Fig.9 b shows that two IPTV servers are involved, in particular, we configured server 2 to be more loaded than server 1. In particular, during the simulation, server 1 turned out to be more optimal according to the proposed algorithm than server 2, so it was automatically selected by the system.

In the simulation model, we implemented a monitoring system of the main parameters, such as loss (p), delay (t), and jitter (j). Detailed information on the process of implementing QoS/QoE monitoring systems in SDN can be found in our previous works [40]–[42]. The developed monitoring system will automatically analyze the obtained QoS results and determine by the QoE score of the user at the worst value of communication quality. The QoE score is represented by values from 1 to 5, where 1 is the worst score, 5 is the highest score.

By experimental study, it was found that the main factors affecting the quality of video QoE perception depends on the technical parameters QoS, which are provided by the network, including available bandwidth, packet loss, jitter, delay. The need to relate QoE estimates to network QoS metrics is made in order to anticipate video services QoE by technical QoS parameters. Finding this relationship is important in identifying network problems to further improve the QoE for users through adaptive resource management.

In the simulation model, a script was created to manipulate various QoS parameters in the developed SOSDN network during the transmission of a real-time video stream with a 720p resolution. Accordingly, the following correlation between the QoS parameters and the QoE assessment was experimentally established (Table IV). In particular, when the delay is less than 150 ms, jitter is within 0-20 ms, losses are from 0-1% and bandwidth is more than 2 Mbit/s, then the QoE assessment varies on a scale of 4-5, and user satisfaction with video viewing is considered good.

In the paper, a mathematical model of correlation of user satisfaction level by QoE evaluation depending on technical QoS parameters provided in the network is proposed. As noted above, user satisfaction is influenced by various QoS parameters. In order to mathematically determine the deviation of the QoS parameters in the QoE estimation, a normalization of the QoS calculation procedure is required. For this purpose, the minimum reference values of QoS parameters, at which high quality of perception of the studied video stream is provided in work. The normalized value of the integral additive criterion QoS is calculated by the formula (20):

where [TeX:] $$K_1, K_2, K_3, K_4$$ is the weights of importance of QoS parameters, ranging from 0 to 1, and their sum should be equal to one.

A mathematical model of the correlation between the level of user satisfaction on the QoE evaluation, depending on changes in the integral criterion of the QoS parameters, is presented as a function:

Thus, the task of ensuring the ordered level of quality of service perception by users according to QoE estimates will be to find the required normalized value of the integral additive QoS criterion. This problem is solved by the multi-criteria adaptive routing of data flows, the metric of which is based on the same integral additive QoS criterion.

Thus, using the proposed mathematical model (20) and applying formula (16) to determine the adaptive routing metric described in Section 2, we found the relationship between the path cost given by the target function and the QoE level indicator described in Table V. In particular, if the metric of the route through which the video is transmitted is less than 0.28, the QoE assessment will be in the range of 5-4.

Since the developed monitoring system is automated, it will be able to independently improve the QoS, but only for the network user, which it will not satisfy and who will directly address with this request in writing or by phone. If the network user has asked to improve the provision of service, the system on the basis of the proposed mathematical model of correlation of QoE evaluation from QoS parameters, presented as a function will be able to check whether the QoS provision does not correspond to the proper level of perception of service, signed under the SLA agreement. Our monitoring system marks the QoS values for users with three colors: green, yellow, and red (Fig. 10). Where red means that the ordered individual requirements of users according to the SLA agreement do not correspond to those provided by the network. Accordingly, the red color indicates the need to make certain decisions to improve its service. Yellow color marking indicates that the QoS parameters provided to users in the network are close to the acceptable parameters that they ordered according to the SLA agreement on the quality of service. Green color marking means that the numerical QoS requirements provided in the network to certain users are significantly lower than the values ordered by users under the SLA agreement. This indicates a high quality of user service.

According to Fig. 10 a-c, formula (20) determines that for user 7 the normalized value of the integral indicator QoS(x) is 0.5023, and substituting it into formula (21), we obtain that the QoE score is less than 1. As a result of the calculations, we see that there is indeed a need to improve the quality of IPTV service provisioning for user №7. The basic setup of the simulated network in relation to the existing traffic management tools with which the proposed solutions will be compared is the traditional OSPF routing and queuing algorithm FIFO and Priority Queuing (PQ). OSPF and PQ is commonly used as a baseline for SDN-based network solutions [43]–[45].

Due to the fact that the use of the above-mentioned PQ and OSPF approaches, although they increase the level of user satisfaction, the quality obtained does not correspond to the ordered quality according to the SLA. Thus, the developed method of adaptive routing and prioritization of information flows is proposed to be used to ensure the required level of QoS when watching IPTV video services. In this study it is accepted that 10 users are using IPTV and one or more complain of bad QoS perception, then based on the monitoring system in the SDN network a decision is made to give a higher priority to this group of IPTV service users. According to the adaptive prioritization algorithm, the system will decide to adjust the parameters, in particular giving the probability of using the IPTV service the highest value in relation to other information services.

First of all, to implement this routing method in the network simulation model instead of using the OSPF protocol metric we propose to use the integral metric of adaptive routing proposed in Section II formula (17). This metric is based on a multi-criteria analysis of node functioning state, in particular by monitoring such parameters as delay, packet loss, jitter, and available bandwidth in channels. Accordingly, based on the developed simulation model in the process of its functioning these QoS parameters are monitored and the optimal path for the transmission of IPTV video streams, based on the ordered QoE level, is determined.

After that the work evaluated the QoS parameters of IPTV services, including the user 7, who received an bad QoS perception. Graphical improvement of quality parameters are shown in Figs. 10(d)–(f).

After applying this adaptive routing method, the QoE level provided to user 7 is evaluated (Fig. 11). From the results of modeling, we can see that the complex use of the adaptive multi-criteria routing and prioritization of data flows allowed to provide a high level of quality perception of IPTV service in conditions of overload of individual elements of networks. In particular without using the complex approach the level of the received quality was QoE = 1.9 (PQ algorithm and OSPF), using the proposed method for adaptive routing and services prioritization the perception quality was QoE = 4.7.

Thus, when a user is detected with a violation of its QoS requirements, the SDN controller decides to give it the highest priority of service in the network nodes and select an alternative route. In this case, additional resources were not used, and, accordingly, the network resources provided to other users were reduced by partially redistributing them to user 7. Thus, for other users, packet losses increased within acceptable limits (they do not exceed the requirements set for them), and for user 7, they decreased.

Fig. 11 shows the dynamics of improving the QoE indicator with the introduction of various developed methods and algorithms to improve the QoS provision. Each obtained QoE value shown by a red point on the graph is the average of three similar experiments. As a result, it was determined that the QoE error did not exceed 2%.

VIII. DISCUSSION AND CONCLUSIONS

TheA SOSDN architecture with user-centric traffic engineering and QoS/QoE support techniques is proposed. It has been established that the proposed SOSDN allows to guarantee an ordered level of service by analyzing QoE estimates of users according to the SLA contract. This is achieved by adaptive prioritization of services, allocation of channel resources in network nodes, load balancing on servers and the introduction of a new method of QoE routing.

An adaptive algorithm for prioritizing information flows has been developed, which, by automating control at the SDN controller level, makes it possible to dynamically change the priorities of network flows in case of deterioration of traffic QoS parameters during transmission through the network to maintain the ordered QoS for specific users.

A method for adaptive multi-criteria routing of data flows in SOSDN that uses an adaptive integral metric for each class of information flows has been developed. This metric is based on measured QoS parameters, which characterize each branch of the network topology in the process of its functioning and the necessary requirements to the QoS provision according to the concluded SLA contract. The advantage of the proposed approach to the routing mechanism control is the possibility of deployment, both in the current software-defined networks, and within a separately taken set of communication nodes, united by a single logic of organization of network resources. It is proved that the developed routing method provides a necessary tool to simplify the communication system management by manipulating the weight coefficients of the integral metric, thereby reducing the task of providing the necessary QoS provision to the selection of their “best” values, adapting to the needs of users. Decision-making on recalculation of primary network parameters and direct adjustment of metrics weighting coefficients values is the main task of the software-defined network controller. To solve the multi-criteria optimization problem of finding the optimal SDN data path, a method using a generalized additive optimality criterion is selected and automated on the controller.

An algorithm for solving the multi-criteria optimization problem using the integral criterion is proposed to determine the optimal data service server. According to this algorithm, a utility function is formed to solve this problem, which provides a measure of the effectiveness of the proposed solution, and the preference process is reduced to a comparison of numerical values. In this case, when making a decision, the SDN controller considers that one set of local criterion values has an advantage over others if it corresponds to a larger value of the utility function. In the event that QoS deteriorates, the SDN controller automatically sets optimal weights to adapt the server domain to provide the desired level of QoS by matching the QoS level provided by the network domain.

To study the effectiveness of the proposed methods of traffic engineering and QoS/QoE support, we developed a simulation model of a SOSDN. In our approach, we used Mininet’s extended software known as Containernet, which offers the ability to deploy containers as both hosts and servers. This unique feature is particularly advantageous for the development of intricate services intended to operate within containers on hosts while connecting through an SDN network emulated by the Mininet core. To compare our techniques, we used wellestablished practical solutions such as OSPF and PQ, which are already actively applied in the field of SDN. Also by means of modeling, it was proved that the complex usage of the method of adaptive multi-criteria routing and prioritization of data flows allowed to provide a high-quality level of perception of real-time video service in conditions of overload of separate network elements, in particular, without usage of the complex approach the received quality level was QoE = 1.9, and with usage of the method of adaptive multi-criteria routing and prioritization of data flows the quality level of perception of video service was QoE = 4.7.