Algorithm for Detection of Fire Smoke in a Video Based on Wavelet Energy Slope Fitting

1. Introduction

Thousands of fire accidents take place every day in the world, causing a large number of casualties or destroying vast areas of forest vegetation, and imposing serious threats on the safety of human life, property and natural environment [1]. A fire accident often comes suddenly and fiercely, and impacts a wide area. If it is not discovered in time, once the fire spreads, it will be difficult to control the fire within a short period of time. Therefore, it is very important to have real-time detection of fire accidents [2]. Generally speaking, flame is small at the early stage, but smoke is very obvious [3]. Based on this, a visual image based smoke warning system is used for rapidly improving smoke detection, and this study provide an important basis for judging whether a fire accident takes place in a timely manner [4,5].

A traditional smoke detection system mainly relies on its smoke sensor, which only works when smoke is close. Therefore, the traditional detection system is not applicable in the outdoor space. Moreover, because the sensors are easily distracted by dust, air flow and human factors, these detection systems often have a high false alarm rate. With the rapid development of video processing technology, a video-based algorithm for detection of fire smoke in a video has a promising future for wide applications [6-8].

Being colorful and having irregular textures and shapes are the characteristics of smoke [9,10]. At present, video-based algorithms for detection of fire smoke usually make a decision directly or with a sorter based on one or more characteristics of the smoke. Toreyin et al. [11] use the characteristics of smoke including its movement, flicker, color and blurred edge, to extract the variance between different ranges of the edge of smoke for detection of smoke. Said method needs to analyze the background of the composed scene, thus limiting the scope of application of the algorithm. Fujiwara and Terada [12] propose a method of extracting the smoking area in the image by use of fractal coding concept. However, for smoke images of low contrast or blurred smoke images, the fractal characteristics so extracted are not stable enough. Zhou et al. [13] propose an algorithm for detection of smoke in a video based on the study on both the static characteristics and dynamic characteristics of the smoke. Said method produces a false positive detection result when there is a non-smoke object that is extremely similar to the smoke in color and shape. Wang et al. [14] use a method where many characteristics of the smoke are combined to detect the smoke in the early stage of a fire accident, and said method is sensitive to the non-smoke moving distractors and is likely to produce a false positive alarm. Wang et al. [15] proposes a method of applying both the smoke diffusion model and the sway detection model to smoke detection. Yuan and his colleagues [16,17] use the total number of smoke pixels and a model for accumulating the directions of movement for detection of smoke, and said method has the disadvantage of inaccurate estimation of the directions of movement of the smoke. Yuan [18] also proposes a method for extracting the characteristics of smoke by the dual-mapping structure method and detecting smoke with an AdaBoost sorter. Yu et al. [19] proposes a method for summarizing the characteristics of smoke movement based on the optical flow algorithm and sorting by use of BP neural network, and said method can distinguish smoke from moving distractors. Zhou et al. [20] use an algorithm for detection of fire in a video by use of the characteristics of movement, and said algorithm reduces the number of suspected smoke areas and meets the requirement for real-time detection, but it is prone to produce a false positive alarm when a moving distractor is similar to the smoke in color.

The above methods for detection of smoke in a video have two main problems: (1) they need to spend a large amount of time in calculation when performing analysis and computation for recognition of various characteristics of the smoke, thus making it difficult to apply these algorithms to a real-time fire smoke detection system; and, (2) they are prone to produce a positive alarm for cloud, fog and moving distractors (such as human and vehicles). Therefore, the research objective of this paper is to achieve a more accurate algorithm and a faster processing speed in first detection of smoke than those algorithms that are based on analysis of characteristics and machine learning, while eliminating the distraction of moving objects such as pedestrians and cars. For this reason, we propose an algorithm for detection of fire smoke in a video based on the fitted wavelet energy slope.

This paper mainly contributes to these three aspects: firstly, the algorithm detects the suspected moving smoke object and calculates its wavelet energy; secondly, the algorithm fits the changing slopes of smoke in 20 continuous frames by use the least square method; and lastly, it gives a positive alarm by use of the relation between the changes in the 2 smoke slopes in 40 continuous frames. The experimental results show that the proposed algorithm is superior to other algorithms in both accuracy and processing speed of giving the first smoke alarm, and it is also better in eliminating non-smoke distractors such as pedestrians and cars. This paper is organized in the following structure: Section 2 provides a detailed introduction of the moving object detection based on background updates; Section 3 provides a detailed explanation on the principles and computing steps of the smoke detection algorithm; Section 4 provides the experimental results and analysis; and, Section 5 summarizes the innovations and shortcomings of the algorithm.

2. Detection of Moving Object Area

To judge whether there is any fire smoke in the images of a video, the slow-moving suspected smoke area needs to be extracted. Main methods for detection of moving objects include the frame difference method, the optical flow method and the mixed Gauss method. The optical flow method is insensitive to slow-moving smoke but very sensitive to the change of light, so it is unable to detect the moving smoke area accurately [21]. The frame difference method detects the moving object by use of the difference between several consecutive frames, but said method has a high requirement for the environment and it is only suitable for certain specific scenes and sensitive to environmental noise. Compared with other detection methods, the mixed Gauss method requires complexed and time-consuming computation, which is not suitable for a real-time detection system. Although these methods produce good results in detection of rigid objects, they have difficulties in extracting a complete smoke area of non-rigid smoke that appears to diffuse in movement and easily generates an empty space. The background estimation method is also a commonly-used algorithm for detection of a moving area, which method features a computing speed as fast as that of the frame difference method. In this paper, an improved background dynamic updating method is introduced for extracting the moving area.

2.1 Background Update

The background estimation method is key to background updating. Due to the diffusivity of smoke, traditional methods easily lead to empty space phenomenon [22]. Therefore, in smoke detection, the changes of adjacent frames are considered in background update, and the image of the first frame is also added as reference for the update. The background update matrix is represented as Eq. (1).

In the equation, n represents the current frame number, n–1 represents the previous frame; [TeX:] $$B_{n-1}$$ is the background image matrix of the previous frame, [TeX:] $$B_{n}$$ is the updated background image matrix to be obtained in estimation, [TeX:] $$I_{n}$$ is the current frame image matrix, [TeX:] $$M_{n-1}$$ is binarized image matrix of the moving foreground in the previous frame, α is the background-weighted updating coefficient, [TeX:] $$0<\alpha<1$$. In the actual computation process, the initial value [TeX:] $$M_{0}$$ is zero matrix, and [TeX:] $$B_{0}$$ is the matrix of the first frame image of the video. In this paper, is 0.05.

2.2 Binarization of Moving Foreground

After the background image is updated, the moving foreground area [TeX:] $$M_{n}$$ can be obtained through computation of the difference between the current frame [TeX:] $$I_{n}$$ and the background estimation image matrix B_n (as shown in Eq. (2)). Because the foreground region not only includes the whole moving area of the object, but also produces a lot of noises of which the gray value is not 0. Therefore, a background difference method with a threshold value is adopted, and the binarization of the moving object area is shown as Eq. (3).

In the equation, i and j are the row and column variables of the foreground image M, respectively. The threshold value T is 14 in this paper, and the effect of the divided foreground in the smoke image is shown in Fig. 1.

3. Algorithm for Smoke Detection

In the early stage of a fire accident, smoke is produced due to insufficient combustion. The smoke moves slowly and its volume gradually increases with time [23]. After wavelet conversion, the wavelet energy in the suspected smoke area is also growing bigger slowly. Based on this, this paper proposes an algorithm for detection of smoke in a video based on wavelet energy slope fitting. This algorithm avoids consumption of a lot of computation time for extraction and analysis of the smoke characteristics, thus greatly increasing the processing speed of the algorithm. It is an algorithm for monitoring smoke in the early stage of a fire accident under the scene of a fixed camera. The basic theory is as follows: firstly, the suspected moving object is extracted by use of the background estimation method to form a binarized mask image of the moving object foreground M; secondly, the low-frequency energy Eb decomposed in the monolayer wavelet is calculated by multiplying the binarized mask image M and the current frame image; and, lastly, the least square method is used to fit the 20 consecutive frames of video in sequence as 1 cycle and the slopes in 2 continuous cycles are obtained through fitting. If the fitted slope of the 2 cycles is greater than the set threshold value T, then a smoke alarm signal is given.

3.1 Criterion for Judgment of Smoke

The smoke in the early stage of a fire accident develops from nothing and moves up slowly. With the increase of smoke, the low-frequency wavelet energy of the suspected area also slowly increases. In fire surveillance video, it is difficult to avoid people and vehicles from moving in and causing a false appearance of moving smoke, which is a strong distractor that easily leads to a positive alarm [24]. In order to eliminate the distraction of non-smoke objects such as people and cars, the algorithm fits the change in wavelet energy slope k by use of the least square method in the sequentially consecutive 20 frames in the video. 40 consecutive frames are selected, and 1 slope is obtained by fitting every 20 frames. Total two slopes k1 and k2 are obtained through fitting. When the conditions of [TeX:] $$\left|k_{1}\right|>2 \times 10^{6}$$ and also [TeX:] $$\left|k_{2}\right|>2 \times 10^{6}$$ are met, it can be judged that smoke exists in the video. At the same time, it can overcome the distraction of non-smoke objects like the pedestrians and vehicles in the video.

The following shows the changes in the energy curve of the suspected smoke area in the 2 videos (25 fps and 640 × 480 resolution) and the changes in the fitted slopes. In video1, there are total 27 seconds of white smoke produced by the friction of vehicle tires with the ground, and white smoke appears in the 1st second (the [TeX:] $$$25^{\text {th }}$$$ frame). Video2 is a 22-second video, in which a man dressed in green on top enters from the right side of the picture at the [TeX:] $$$10^{\text {th }}$$$ second (the [TeX:] $$$250^{\text {th }}$$$ frame) in the surveillance video, and the white smoke is released at the [TeX:] $$$18^{\text {th }}$$$ second.

Under no distraction of non-smoke objects, the smoke change curve is a straight line as long as there is no smoke (i.e., no moving object is detected); when smoke appears, the change curve reflecting the smoke in the early stage of fire should be a steep slope (Fig. 2). The Segment A in Fig. 3(b) shows the curve change when a pedestrian enters into the picture. The curve change is similar to that of the smoke. The slope changes greatly, indicating very strong distraction. The Segment B in Fig. 3(b) shows the stage where smoke is rising slowly. Non-smoke distraction can be effectively eliminated through the smoke judgment criterion, and an accurate smoke alarm can be given.

3.2 Smoke Detection Algorithm by Steps

All current smoke detection algorithms need to extract and analyze a large number of characteristics of the suspected smoke area, such as color, roundness and texture [25,26]. Analysis and computation of various characteristics of the suspected smoke area are required, and the algorithms are complexed and time-consuming. Hence it is difficult to use these algorithms in real-time fire monitoring video system [27]. The algorithm proposed in this paper for detection of fire smoke in a video based on wavelet energy slope fitting does not require extraction of these characteristics (Table 1). It calculates the low-frequency wavelet energy of the suspected smoke area and gives a smoke alarm signal through fitting the low-frequency wavelet energy slope change within the given window by use of the least square method.

(1) The input color image is converted into a gray image, and the first frame of smoke-free image is chosen as the background image for updating the reference image B;

(2) The suspected moving smoke area is extracted by use of the background estimation method and the mask M is binarized;

(3) The low-frequency wavelet energy [TeX:] $$E_{b}$$ is calculated by wavelet decomposition of the result of the mask image M multipled by current frame image I;

(4) The value of 20 consecutive frames Eb is selected, and the absolute value of the changing curve slope is obtained through fitting by use of the least square method.

(5) Slope k1 and Slope k2 are obtained through fitting within the cycle of 40 consecutive frames. When the conditions of [TeX:] $$\left|k_{1}\right|>2 \times 10^{6}$$ and also [TeX:] $$\left|k_{2}\right|>2 \times 10^{6}$$ are met under the criterion for smoke judgment, a smoke alarm signal will be given.

4. Analysis of Experimental Results

The algorithm in this paper is to detect the smoke generated in the early stage of a fire accident. During this stage, the smoke volume grows from small to large. Considering the characteristics of such change, a time window of 40 consecutive frames is set, and the slopes of every 20 frames are fitted to judge the change of the two slopes in this time window in order to give a smoke alarm. When a smoke alarm is given under the algorithm proposed in this paper, it does not mean that smoke is produced in every frame within the 40-frame cycle. The algorithm is a process of holistic judgement. The main purpose of the experiment is to test the accuracy and anti-distraction ability of the algorithm. The accuracy refers to the difference between the time when the occurrence of smoke is observed by human eyes and the time when the smoke is detected by the algorithm. The anti-distraction ability means whether the algorithm gives a positive alarm when any non-smoke moving distractor exists in the video. Five videos are selected in the experiment for comparison with [13] and [14], respectively. The videos have the resolution of 640×480 and the frame rate of 25 frames per second.

The videos in the experiment are mainly sourced from three websites, including the website of Yuan Feiniu Lab in Jiangxi University of Finance and Economics (http://staff.ustc.edu.cn/~yfn/vsd.html), the website of the CVPR Lab. at Keimyung University in South Korea (http://cvpr.kmu.ac.kr/), and the website of Internet Online Resources Library (http://imagelab.ing.unimore.it/visor/). The software used in the experiment is MATLAB R2016b, and the computer is configured with CPU G860, dual core 3.0 GHz and 4 G memory.

4.1 Experiment 1

Experiment 1 is to test the accuracy of the algorithm under the 100% smoke condition. Tables 2 and 3, respectively show the video description and detection data, and the rendering of detection result is as shown in Fig. 4.

The smoke-observing frames refer to the number of frames in the video in which the occurrence of smoke is observed by human eyes. The smoke-detecting frames refer to the number of frames in the video in which smoke is detected by the algorithm. The error time is the difference between the smoke-detecting frames and the smoke-observing frames divided by the frame rate. The computation time is the time for the algorithm to complete detection through the entire video. The first image in the test results is the frame that obvious smoke frames observed by human eye in video; the remaining three images are the first smoke detection images of three algorithms in the reference documents and in this paper.

The detection data in Table 3 show that smoke is detected with each of those three algorithms, but we can discover from the error time that the algorithm proposed in this paper results in the minimum error time, which indicates that the algorithm proposed in this paper is more accurate than that of the other two algorithms. In addition, in terms of the time for processing the entire video, the algorithm proposed in this paper results in the shortest processing time, which indicates that the computation speed of the algorithm proposed in this paper is the fastest.

4.2 Experiment 2

Experiment 2 uses the video before smoke occurs, and non-smoke distractors can be seen moving at different speeds in the video. The main purpose of the experiment is to test whether the algorithm gives a false positive alarm (i.e., test of the anti-distraction ability). The results of the experiment are as shown in Tables 4 and 5, and Fig. 5.

Experiment 2 tests the anti-distraction performance of 3 algorithms in the video. The data in the error time column in Table 5 show that moving distractors are considered as smoke and a false alarm is given in [13] and [14]. Only under the algorithm in this paper are the moving distractors effectively avoided and smoke is accurately detected. This is also well proven in Fig. 5. The data in the computation time column in Table 5 show that the algorithm in this paper takes less time in computation than the other two algorithms, which also explains that the algorithm in this paper has the fastest computation speed.

The notation “-” in the column “error time” indicates that the algorithm incorrectly detects the moving distractors as fire smoke and triggers a positive alarm.

4.3 Experiment 3

In Experiment 3, the video shows the change in the density of cloud or fog. The subtitles of the video are described in Table 6. The detection data and results are shown in Table 7 and Fig. 6, respectively.

From Table 7, it shows that the algorithms in both [14] and [13] mistakenly detect clouds and fog as fire smoke, and no smoke is detected by the algorithm in this paper. That is to say, under distraction of cloud, fog and other distractors of a similar color to the smoke, the algorithm in this paper has a stronger anti-distraction ability than those in [14] and [13]. In this algorithm, cloud or fog is not detected as smoke, mainly because the cloud or fog changes more slowly, so the change of the wavelet energy slope of the smoke image is insignificant in detection of the suspected moving smoke.

4.4 Experiment 4

In Experiment 4, smokeless videos with moving distractors similar in color to smoke are selected (Table 8, Fig. 7). The main purpose is to test the ability of the algorithm to distinguish smoke from non-smoke moving distractors.

Table 9 shows that smoke is detected under the algorithms in both [13] and [14]. Under the algorithms in this paper, smoke is only detected in video15 and video16, mainly because that the white moving distractors in video15 and video16 move fast from far to near, causing the algorithm to consider it as the change from having no smoke to having smoke, which is similar to the process of smoke production, thus producing a false positive result.

4.5 Experiment 5

In Experiment 5, smoke videos under weak lighting conditions are selected. The purpose is to test the impact of light intensity on the algorithm (Table 10, Fig. 8).

Table 11 shows that under weak light condition only the algorithm in [14] has detected smoke in video19, while the two algorithms in this paper and the study by Zhou et al. [13] haven’t detected smoke effectively. Therefore, the light has a great impact on the algorithm in this paper.

4.6 Analysis of Experiments

In the above 5 experiments, different types of videos are selected to test the accuracy and anti-distraction ability of the algorithm. In Experiment 1, a video with smoke only and without any moving distractor is selected. Compared with other two algorithms, the algorithm proposed in this paper has the minimum error time (less than 1 second) in smoke detection, and the experimental result shows the highest accuracy in the algorithm proposed in this paper. Moreover, in terms of the density of smoke detected in Fig. 4, the smoke detected by the algorithm in this paper is in the lowest density, which indirectly proves that the algorithm in this paper offers the best accuracy. In Experiment 2, we choose the videos with moving persons as distractors before smoke occurs. According to the detection results in Fig. 5, the algorithm proposed in this paper has the best detection result without giving any false positive alarm, and smoke is shown in every image. In contrast, the other two algorithms mistake the moving person as smoke in a fire. Therefore, the result of Experiment 2 shows that the proposed algorithm has the best anti-distraction ability against moving distractors.

In Experiment 3, the videos with cloud and fog similar to the fire smoke are tested, and the test results show that the algorithm proposed in this paper does not detect fire smoke when the density change is not fast. In contrast, the other two algorithms mistake clouds and fog as fire smoke. The result of this experiment shows that the algorithm proposed in this paper has better anti-distraction ability against cloud and fog than the other two algorithms. In Experiment 4 on non-smoke moving distractors (human and vehicle), the proposed algorithm has been distracted. The main reason is that the pedestrian in video15 suddenly jumps upwards between the 22nd second and the 25th second, which is represented in the slope of the suspected moving smoke object within the 40 consecutive frames being similar to the slope of smoke movement. Similarly, during the longitudinal movement of the car in video16, a slope similar to that of the smoke movement occurs. That is to say, the algorithm detects a false positive result. In Experiment 5, the smoke video under a weak lighting condition is tested. No smoke is detected under all three algorithms, showing the accuracy of the detection by algorithms is much impacted by visible light. The main reason is that most of the current algorithms are designed based on visible light. The algorithm fails to work well due to weak visible light, i.e., the weaker the light is, the worse the detection result is.

5. Conclusion

Specific to the algorithm for detection of smoke in the early stage of a fire accident, a method for fast detection of smoke in the early stage of a fire accident is proposed based on the fitting of wavelet energy change slope of the smoke in the video. The algorithm uses the diffusing characteristic of smoke movement and gives a smoke alarm by fitting the wavelet energy change in the smoke image within the specified time window. This is the innovation of this paper. The experimental results show that said method can accurately detect the smoke in the early stage of a fire accident and eliminate the distraction of some non-smoke moving distractors (e.g., human and vehicle). A shortcoming of this method is that it easily misjudges when some moving distractors have significant movement from far to near. This will be the direction for future improvement of the algorithm proposed in this paper.

Acknowledgement

This paper is funded by Natural Science Foundation Program for Universities and Colleges in Jiangsu Province (No.18KJB520012), Changzhou Sci & Tech Program (No. CE20165049).