FORECASTING AIR QUALITY PARAMETERS USING HYBRID NEURAL NETWORK MODELLING

MIKKO KOLEHMAINEN*, HANNU MARTIKAINEN, TERI HILTUNEN AND JUHANI RUUSKANEN, University of Kuopio, Department of Environmental Sciences, [Link] 1627, FIN-70211 Kuopio, Finland *email: [Link]@[Link]

Abstract. Urban air pollution has emerged as an acute problem in recent years because of its detrimental effects on health and living conditions. The research presented here aims at attaining a better understanding of phenomena associated with atmospheric pollution, and in particular with aerosol particles. The specific goal was to develop a form of air quality modelling which can forecast urban air quality for the next day using airborne pollutant, meteorological and timing variables. Hourly airborne pollutant and meteorological averages collected during the years 1995-1997 were analysed in order to identify air quality episodes having typical and the most probable combinations of air pollutant and meteorological variables. This modelling was done using the Self-Organising Map (SOM) algorithm, Sammon's mapping and fuzzy distance metrics. The clusters of data that were found were characterised by statistics. Several overlapping Multi-Layer Perceptron (MLP) models were then applied to the clustered data, each of which represented one pollution episode. The actual levels for individual pollutants could then be calculated using a combination of the MLP models which were appropriate in that situation. The analysis phase of the modelling gave clear and intuitive results regarding air quality in the area where the data had been collected. The resulting forecast showed that the modelling of gaseous pollutants is more reliable than that of the particles. Key words : air quality, forecasting, neural networks, Self-Organizing Map, Sammon's mapping

1. Introduction Urban air pollution has emerged as the most acute problem in recent years because of its detrimental effects on health and living conditions. To prevent any further decline in air quality, scientific planning of analytical methods and pollution control is required. Within this framework it is necessary; (i) to analyse and specify all pollution sources and their contribution to air quality, (ii) to study the various factors which cause the pollution phenomenon, and (iii) to develop tools for reducing pollution by introducing alternatives to existing practices. Air quality phenomena have traditionally been modelled using physical reality as the starting point, and this information has then been coded into differential equations, for example. Methods of Computational Intelligence offer a
Environmental Monitoring and Assessment 65: 277286, 2000. c 2000 Kluwer Academic Publishers. Printed in the Netherlands.

278
completely different approach, in which the model is constructed entirely using measured data. Computational Intelligence is a fairly new discipline which is based on the hypothesis that reasoning can be realised using computation (Poole et al., 1998). The methods used in computational intelligence include a number of forms of computation, the best known of which are neurocomputing and fuzzy logic. Neurocomputing is based on principles that have been discovered by investigating the brain and its structures (Haykin, 1994), while fuzzy logic is based on fuzzy set theory, which extends traditional bivalent logic into continuous group membership with truth values between 0 and 1 (Zimmermann, 1991). Neural networks are used as a tool in most of these applications. As examples, neural models for ozone concentrations have been constructed (Yi and Prytok, 1996 ; Comrie, 1997) and a model that predicts hourly NOx and NO2 concentrations (Gardner and Dorling, 1999). Most of the work has focused on comparing feed-forward neural networks (especially Multi-Layer Perceptrons) with traditional methods such as the ARIMA model and linear regression. The results show in general that the neural models perform as well as these methods or better (Garder and Dorling, 1998). There has been some criticism for this "black box" modelling, however, on the grounds that it offers too little support for understanding the physical phenomena that are being considered. The research presented here therefore aims at attaining a better understanding of phenomena associated with atmospheric pollution, meteorological parameters, and in particular, aerosol particles. The aim is to show how a set of neural and related methods can be used (i) to understand better the relationship among airborne pollutant and meteorological variables, (ii) to use expert knowledge to support the modelling, and (iii) to create a model which is able to forecast concentrations of airborne pollutant variables.

2. Materials and Methods 2.1 SELF-ORGANIZING MAPS (SOM) One of the best known unsupervised neural learning algorithms is the SelfOrganizing Map (SOM) (Kohonen, 1995), the aim of which is to find prototype vectors that can represent the input data set and at the same time to achieve a continuous mapping from the input space to a lattice. This lattice can be an easily visualised 2-dimensional map, for example. The weight vectors of the SOM are first initialised to random values. With each training pattern the winning neurone (Best-Matching Unit, BMU) is first found by comparing the input (measured) and weight vectors of the neurones by

279
Euclidean distance metrics. The weights of the winning neurone (BMU) and its neighbours are then moved towards the input vector according to a learning rate factor which decreases monotonically towards the end of learning. This unsupervised learning process is summarised in Figure 1 using a 4 by 4 SelfOrganizing Map. A variation of the SOM, called tree-structured SOM, was used in this work (Koikkalainen, 1994). The software implementation consists of several SOMs that are organised hierarchically in several layers in a pyramid-like fashion. The number of neurones at a larger level is four times the number at the previous level. Visual inspection of the measurement data is directed at one level at a time, however, and the results are comparable to those achieved by standard SOM. The level that was selected here for visualisation consists of 1024 neurones.
a) b) c) d)

Self-Organizing Map 1 2 3 4 + + + +
5 + 9 + 13 + 6 + 10 + 14 + 7 + 11 + 15 + 8 + 12 + 16 +

Random configuration

Intermediate configuration

Final configuration

+
+ +
+ + + + +

+ +

+ +

+ +

+ + + + + +
+ + + + + + + + + + +

+ + + + + + + + + +

+ = SOM weight vectors = Original data rows

+ +

Figure 1. Visualisation of the SOM learning process. a) a Self-Organizing Map with 16 neurons. b) Weight vectors (+) are first initialized to random values. c) The weight vectors move towards the final configuration through intermediate configurations. d) In the final configuration the weight vectors represent a number of original data rows.

2.2 SAMMON'S MAPPING Sammons mapping is an iterative method based on a gradient search (Sammon Jr, 1969). The aim is to map points in n-dimensional space usually into 2 dimensions. The algorithm finds the locations in the target space so that as much as possible of the original structure of the measurement vectors in the ndimensional space is conserved. The numerical calculation is more timeconsuming than the SOM algorithm, however, which can be a problem with a massive data set. On the other hand, it is able to represent the relative distances between vectors in a measurement space and is thus useful for determining the shape of clusters and the relative distances between them. It is therefore of benefit to combine these two algorithms. Sammons mapping is thus applied to the stage where the SOM algorithm has already achieved a substantial data reduction by replacing the original data vectors with a smaller number of representative prototype vectors.

280
2.3 MULTI-LAYER PERCEPTRON AND THE BACK-PROPAGATION ALGORITHM Multi-Layer Perceptrons, which represent one type of feed-forward neural network, consist of processing elements and connections (Hecht-Nielsen, 1991). The processing elements, usually called neurones, are arranged in layers. There are three kinds of layers: input layers, hidden layers and output layers. An input layer serves as a buffer that distributes input signals to the hidden layer. Each unit in the hidden layer sums its input, processes it with a transfer function and distributes the result to the next layer, which is usually the output layer. The units in the output layer compute their output in a similar manner. The most common supervised learning algorithm is the back-propagation (BP) algorithm, also called the generalised delta rule (Haykin, 1994). It is a gradient descent algorithm that is normally used to train the MLP network. For an introduction and overview of MLP applications in the atmospheric sciences, see Garder and Dorling (1998). 2.4 GROUP MEMBERSHIP EVALUATION USING FUZZY LOGIC The goal of group membership evaluation is to represent data lines (measurements) in respect to clustered phenomena found in the data set. The process is initiated by finding the kernels of the groups. This can be done using the SOM algorithm and Sammons mapping, for example. The kernels are areas where a certain phenomenon or property is known to be true with certainty. Other areas can now be evaluated using the kernels as reference points. The membership of each data line with respect to the group kernels is described using terms known from fuzzy logic (Zimmermann, 1991). 2.5 STATISTICAL PERFORMANCE INDICATORS Selected statistical indicators were used to describe numerically the goodness of the estimates (Willmott, 1982). With the neural networks, one of the most common indicators is the Root Mean Square Error (RMSE). It can be calculated according to Equation (1).
1 RMSE = N

[P O ]
i =1 i i

(1)

where N = number of data points, Oi = observed data point and Pi = predicted data point.

281
A relative measure of error called Index of Agreement (d) can be calculated according to equation (2). It is a dimensionless measure that is bounded into range 0..1.

d = 1

( P
i =1

i =1 N

(P O )
i i ' i

(2)

Oi' ) 2

where Pi ' = Pi O and Oi' = Oi O .

3. Results and Discussion 3.1 PRE-PROCESSING OF DATA The variables for the modelling were first selected. The meteorological variables used were wind speed, wind direction and temperature. The time information for each hourly data line made it possible to construct time dependence for airborne pollutant and meteorological variables. The airborne pollutant variables selected for modelling were NO, NO2, CO and PM10. As the tools used for neural modelling do not accept missing values, data lines having missing values were simply omitted. Furthermore, variables for wind direction, hour and month were each transformed into two auxiliary variables using the sine and cosine functions. This enabled the neural algorithms to work properly despite of the discontinuities in the original variables. Finally, the data set was split into a training set (90 %) and a test set (10 %). 3.2 SELF-ORGANIZING MAP OF AIR QUALITY The data applied to the city of Kuopio during the years 1995-1997, and the analysis was performed using the Self-Organising Map (SOM) algorithm (Kohonen, 1995), which is able to visualise the resulting mapping from n dimensions to a 2-dimensional grid (Figure 2). The first phase of this process consisted of finding the kernels of the pollution episodes. The locations of the episodes selected are the neurones in the SOM map where the pollutant averages reach their highest values locally. In the next phase, fuzzy membership values were calculated for each data row and for each episode. These new variables described the distance of each data row from the prototypes (kernels) of each episode. Finally, the neurones were collected into several, possibly partially overlapping clusters, each representing one episode. This was done by calculating average values for the memberships of each neurone and comparing

282
these with a pre-set limit value. Thus, the neurones where the average membership for a certain episode was higher that the limit were marked as belonging to the cluster (group of neurones) representing that episode in a fuzzy sense (Zimmermann, 1991).

NO2 CO

PM10

Figure 2. Self-Organizing Map of air quality

3.3 SAMMON'S MAPPING OF EPISODES The clustered data were visualised using Sammon's Mapping algorithm (Sammon Jr, 1969), which is another way of mapping n-dimensional data into two dimensions (Figure 3). This enabled qualitative evaluation of the resulting clusters in two ways. Firstly, the consistency of each cluster could be observed, and secondly, the overlapping of clusters was detectable. By adjusting the limit value for membership, it was thus possible to find a suitable value for it by iteration.

283

Figure 3. Sammons mapping of episodes

3.4 DESCRIPTION OF THE EPISODES In order to identify the clusters (episodes), statistics for the episodes were calculated and the mean values (median for time variables) are given in Table 1. By inspecting the distribution of values for each cluster, it was possible to characterise each episode in terms of time of year, time of day, wind speed and direction and composition of the set of the pollutants. Each episode was then assigned a description (Table 2), which made it possible to convert a forecast into an oral description at a later stage. 3.5 LOCAL MODELING USING MLPS At the next stage, local numerical models were constructed for the episodes. The training data for the Multi-Layer Perceptron (MLP) modelling were first selected from the original data rows taking the same membership limit value as was used for clustering the neurones. Next, a MLP model was constructed for each episode and for each pollutant variable using the back-propagation (BP) algorithm.

284
Table 1 Descriptions of the episodes C1-C7 by mean values (month and hour by median) Variable C1 C2 C3 C4 C5 C6 C7 NO2 65.9 91.3 82.7 48.7 65.1 65.4 59.7 CO 1.1 4.5 3.2 1.0 1.6 0.3 2.2 PM10 79.9 20.7 96.2 128.3 87.5 31.0 23.1 Temperature -4.4 -18.7 -7.9 -7.9 -7.2 18.7 -15.6 Wind sp. 2.2 1.4 1.5 1.1 1.3 2.6 1.4 Wind dir. 309 273 94 332 38 315 302 Hour(med) 21 17 21 7 21 22 11 Month(med) 3 12 3 4 3 8 1 NO2 and PM10 expressed in g/m3, CO in mg/m3, temperature in oC, wind speed in m/s and wind direction in degrees. Table 2 Characterisation of the episodes by textual descriptions Description Springtime evening, low wind from north-west, PM10 and NO2 high Winter afternoon, very low wind (inversion), NO2 and CO high Springtime evening, very low wind, all pollutants high Springtime morning, very low wind, PM10 and NO2 high Springtime evening, very low wind, all pollutants elevated Late evening in late summer, low wind from north-west, NO2 high Winter morning, very low wind, NO2 and CO high

Episode C1 C2 C3 C4 C5 C6 C7

3.6 USING AND TESTING THE HYBRID MODEL The hybrid neural network model (see Figure 4) was tested using the data set which had been put aside for this purpose in the pre-processing stage. The first part consisted of finding the Best Matching Unit (BMU) neurone in the SOM. This calculation consisted of using the meteorological and time variables of the test data line to find the closest neurone in the SOM. An average membership value could then be determined for each episode and the actual levels for individual pollutants calculated using a combination of the MLP models which were appropriate in that situation. Estimates from several submodels were combined by weighting the output of each MLP by the membership value and summing the results into one estimate.

285
History data Training data Test data Weather forecast Self-Organizing Map Dust Traffic x ... MLP n Inversion x y z u + Air forecas quality t MLP 1 x y z u

Figure 4. General principle of the hybrid neural network

3.7 EVALUATING THE GOODNESS OF THE MODEL The goodness of the estimates was evaluated by using general statistical indicators, RMSE and the Index of Agreement (d) for three of the pollutant variables (Table 3). It can be seen that the accuracy of the model was higher for the gaseous pollutant variables than for the particles.
Table 3 Goodness of the model by statistical indicators Variable Min Max Mean Median RMSE NO2 0 123.0 15.2 11 12.2 CO 0 5.7 0.2 0.1 0.3 PM10 0 159.0 12.3 9 11.1 NO2 and PM10 expressed in g/m3 and CO in mg/m3

d 0.66 0.63 0.47

4. Summary and Conclusion The data mining phase of the modelling gives clear and intuitive results regarding air quality in the district where the data had been collected, and also supports the inclusion of expert knowledge in the model. The forecast results with the test data for good air quality were only moderate, which is probably due to the small amount of data available on episodes since the air quality in Kuopio, Finland, is generally good most of the time, with only a few bad pollution episodes per year. The forecast results also showed that the modelling of gaseous pollutants is more reliable than that of particles. By inspecting the distribution of variables of the episodes using data mining, however, it can be deduced that the discriminative power for local models could be improved by using more descriptive meteorological variables, such as atmospheric stability and the height of the atmospheric boundary layer. Another important aspect that should be taken into consideration is the persistence of the

286
weather, and thus also of air quality phenomena. This can be done by time-series forecasting, which will be the most important development aspect of the model for future work.

Acknowledgements This research was funded by Tekes (the Technology Development Centre, Finland) and the University of Kuopio. More information at the web addresses: [Link] (NDA software for neural computing) [Link] (Visual Data software for data mining)

References
Comrie A. C.: 1997, Comparing neural networks and regression models for ozone forecasting, Journal of Air and Waste Management Assiciation 47, 653-663 Gardner M. W., Dorling S. R.: 1998, Artificial neural networks (the multi-layer perceptron) - a review of applications in the atmospheric sciences, Atmospheric Environment 32, 26272636 Gardner M. W., Dorling S. R. : 1999, Neural network modelling and prediction of hourly NOx and NO2 concentrations in urban air in London, Atmospheric Environment 33, 709-719 Haykin S.: 1994, Neural Networks: A Comprehensive Foundation, Prentice Hall, New Jersey Hecht-Nielsen R.: 1991, Neurocomputing, Reprinted with corrections, Addison-Wesley Publishing Company, Inc. Kohonen T.: 1995, Self-Organizing Maps, Springer-Verlag, Berlin Heidelberg, Germany Koikkalainen P.: 1994, Progress with the tree-structured self-organizing map, ECAI'94. Proceedings of the 11th European Conference on Artificial Intelligence, Aug 1994, Editor Cohn, A., Wiley&Sons, 211-215. Poole D., Macworth A., Goebel R.: 1998, Computational Intelligence, A Logical Approach, Oxford University Press Sammon Jr J.W.: 1969, A nonlinear mapping for data structure analysis, IEEE Transactions on Computers, C-18(5), 401-409 Willmott C.: 1982, Some comments on the evaluation of the model performance, Bulletin American Meteorological Society 63, 1309-1313 Yi J., Prybutok V. R.: 1996, A neural network model forecasting for prediction of daily maximum ozone concentration in an industrialized urban area, Environmental Pollution 92, 349-357 Zimmermann H.-J.: 1991, Fuzzy set theory and its applications, Second Edition, Kluwer Academic Publishers