Intelligent checking model of Chinese radiotelephony read-backs in civil aviation air traffic control

Guimin JIA,Fangyuan CHENG,Jinfeng YANG,Dan LI

Tianjin Key Lab for Advanced Signal Processing,Civil Aviation University of China,Tianjin 300300,China

KEYWORDS Air traffic control;Chinese radiotelephony read-backs;LSTM;Mean pooling;MLP;Semantic checking

AbstractFederal Aviation Administration(FAA)and NASA technical reports indicate that the misunderstanding in radiotelephony communications is a primary causal factor associated with operation errors,and a sizable proportion of operation errors lead to read-back errors.We introduce deep learning method to solve this problem and propose a new semantic checking model based on Long Short-Time Memory network(LSTM)for intelligent read-back error checking.A meanpooling layer is added to the traditional LSTM,so as to utilize the information obtained by all the hidden activation vectors,and also to improve the robustness of the semantic vector extracted by LSTM.A MultiLayer Perceptron(MLP)layer,which can maintain the information of different regions in the concatenated vectors obtained by the mean-pooling layer,is applied instead of traditional similarity function in the new model to express the semantic similarity of the read-back pairs quantitatively.The K-Nearest Neighbor(KNN)classifier is used to verify whether the read-back pairs are consistent in semantics according to the output of MLP layer.Extensive experiments are conducted and the results show that the proposed model is more effective and more robust than the traditional checking model to verify the semantic consistency of read-backs automatically.

1.Introduction

Nowadays,radiotelephony communication still plays a very important role in civil aviation transportation which is the primary information transmitting mode for the air traffic controller and fiight crew.1In order to make the aircraft operate safely and efficiently,the air traffic controller and fiight crew must understand each other accurately.During the wholeflight,the Air Traffic Controllers(ATCs)give different instructions to the fight crew by considering the fiight phase,weather conditions,air traffic situations,etc.The fiight crew also has to keep contact with different ATCs in different fiight phases to confirm the received instructions,and report the fiight status and any other necessary information related to fiight.So exchanging the information correctly between the ATCs and the fiight crew is a direct and vital factor for the air traffic safety.

Radiotelephony communication in civil aviation air traffic control is a kind of professional semi-artificial language,2which is built up based on natural languages.To improve the language norms and terminology standard in civil aviation communications,a series of diction and pronunciation standards or proposals have been formulated by the International Civil Aviation Organization(ICAO).The Civil Aviation Administration of China has also set up corresponding standards and requirements that ATCs and pilots have to obey.Although the civil aviation technologies have developed rapidly and communication equipment failures have decreased gradually during the past decades,unsafe aviation incidents related to radiotelephony communication errors,such as inaccurate read-backs,misunderstanding,non-standard diction,and incomplete content,still happen occasionally.3,4According to the investigation reports of NASA,more than fifty percent of the aviation incidents have relation to radiotelephony communication errors.5For example,Tenerife airport disaster in 1977,Urumchi air crash in 1993,and the crash of fiight 965 in 1995 all had close relation to misunderstandings between ATCs and fiight crew.

Many communication problems involve human errors of ATCs or pilots.In spite of the double check required in radiotelephony communication,these human errors are usually difficult to be aware of by ATCs and pilots themselves due to fatigue,tension,tremendous pressure,as well as fast speak speed and unchanged low intonation.With the rapid development of civil aviation transportation industry,controllers and pilots are usually working under enormous pressures.According to surveys,an ATC is usually in charge of more than eighty fiights in one hour at the peak of traffic fiow in the hub airports of China,which means that about tenflights need to be handled at the same time.So it is very likely to make mistakes in radiotelephony communication.Furthermore,pilots and ATCs do not always obey the radiotelephony communication standards strictly and completely in practice.Moreover,the studies of psycholinguistics have shown that in continuous speech perception the speakers will generate potentialpsychologicalexpectation,which ishelpfulto enhance the listening comprehension but is also easy to cause ambiguity.

The data link applications in ATC system are developed to solve this problem partly by exchanging messages between pilots and air traffic controllers.At present,Pre-Departure Clearance(PDC)system and Data link-Automatic Terminal Information Service(D-ATIS),and Controller-Pilot Data Link(CPDL)are typical applications of data link in air traffic control and services.The main benefits of these applications are workload reduction for both pilots and air traffic controllers,as well as the increase in air traffic safety by reducing language and congestion-related communication errors.However,taking account of the construction of data link communication network in differentroutesand airports,the installation of specialized avionics and programs in aircraft,and the technology development of data link communication,the radiotelephony communication is still the primary mode in many countries,including China,and in most fiight phases.In addition,many kinds of information have to be transmitted by air-ground data link system,such as position of aircraft,engine data,important fiight information,and data exchange with airlines,etc.Moreover,the data link communication also grows rapidly and the channel capacity is approaching saturation in some hub airports.Data link communication and radiotelephony communication are usually designed to back up each other in practice.Therefore,detecting the radiotelephony communication errors automatically by computers instead of persons is essential and significant for civil aviation safety.This is the original motivation of our research.

To guarantee that the instructions are received accurately by pilots,read-backs are required according to the air traffic control standard,which means that the pilot reads the instructions back to the controller.It is one of the most important ways to ensure that the instructions are transmitted correctly.It is also crucial for the safety of civil aviation transportation.The common problems in civil aviation radiotelephony communications mentioned above usually result in read-back errors,and FAA technical reports indicate that miscommunications are historically identified as a primary causal factor associated with pilot operation errors.Although read-back errors do not cause pilots’operation errors directly,a sizable proportion of operation errors have relation to read-back errors.6At present,ATCs and fiight crew usually make radiotelephony communication in Chinese for domestic fiights in China,so the main purpose of this paper is to verify whether the Chinese read-backs are consistent in semantics with instructions automatically and intelligently.It has to be noted that,to integrate the proposed method into real ATC system,many scientific and technical problems regarding radiotelephony communications have to be studied furtherly,including continuous speech recognition of radiotelephony communications in real work environment,robust semantic expression,semantic analysis,and situation awareness of their conversations,etc.It involves many fundamental problems in speech recognition and natural language processing.Our research in this paper provides a novel method of semantic matching and verification of the instructions and read-backs,and it is also useful to address other natural language processing problems mentioned above.

Our work is mainly related to the sentence pair modeling based on deep learning.It is an important task to appropriately model the relation between two sentences or texts in many fields of information search and natural language understanding,such as paraphrase identification,query suggestion,machine translation,image retrieval,automatic summarization and semantic textual similarity.7Many algorithms have been proposed for semantic matching which is mainly based on lexical matching,8,9linguistic analysis,10and neural networks.11–16In recent years,some algorithms based on deep neural networks have achieved encouraging results in semantic analysis and matching by using word embedding methods to express the words meaning as vectors in semantic space,17–23which requires no prior knowledge of natural language and no external resource of structured semantic information.

The Recurrent Neural Network(RNN)is a special case of the recursive network in which the structure is simple linear chain.Not only the RNN is primarily used as a language model,but also it has been proved efficient in terms of con-structing sentence representations with a linear structure.17RNN applies a hidden state to represent sentences,and repeatedly feeds the hidden state and word embeddings to the network to update the sentence representations.24RNN may suffer from gradient vanishing and exploding problems which limit the length of reachable context.RNN with Long Short-Time Memory network(LSTM)25,26unit solves such problems by introducing a memory cell and gates into the network.The Convolutional Neural Network(CNN)has also been modified to model the sentence meaning.27,28These developments provide extremely valuable references for us to study the semantic checking problem of Chinese read-backs in civil aviation radiotelephony communication.

In this paper,we propose a novel scheme based on LSTM to address the problem of Chinese read-back verification.A two-channel structure is built up to process the ATC’s instructions and the pilot’s read-backs,respectively.The word embedding method of one-hot vector is used to represent the segmented words of these sentence pairs for the length is usually much shorter and the relation of words is weaker than many other sentence modeling tasks.The traditional LSTM is also improved by adding a mean-pooling layer for the reason that all the words in the read-backs usually have important information related to the fiight in practice.All the hidden activation vectors are combined in concatenation by the meanpooling layer to fully utilize the semantics underlying different sources and also to balance the semantic representation of the former words and the latter words in a sentence.So the semantics extracted by the new model is more robust than traditional LSTM.A MultiLayer Perceptron(MLP)layer is used instead of traditional cosine similarity function to represent the relatedness of the sentence vectors of read-back pairs output by the pooling layer.Then the quantitative similarities of read-back pairs are put into the K-Nearest Neighbor(KNN)classifier to check whether the sentence pair is consistent in semantics.Extensive experiments are conducted based on homemade corpus of Chinese radiotelephony communications in civil aviation.The experimental results show that our new model proposed in this paper is more effective and more robust than the traditional checking model described in our previous work.

2.Semantic representation models

2.1.RNN model

RNN is a natural feed-forward neural network.The structure of RNN is similar to traditional neural network,which is composed of input layer,hidden layer and output layer.The difference among them is that RNN’s hidden units are connected so that it can retain the previous information.The hidden unit of RNN is affected by both the current input sequence and the previous hidden units.RNN’s propagation is related to timestep.29,30The unfolding structure of RNN with time is shown in Fig.1.

In Fig.1,X=[x1,x2,...,xt,...,xT]is the input sequence,and xtis the word vector of the t-th word.K=[k1,k2,...,kt,...,kT]is the hidden sequence of the semantic model,and ktis the hidden activation vector in time-step t.Y=[y1,y2,...,yt,...,yT]is the output sequence.WI,Woand Wrare the input weight matrix,output weight matrix and recurrent weight matrix,respectively.RNN computes the hidden activation vector and output vector at time-step t by the following mathematical formulation:

Fig.1Unfolding structure of RNN with time.

where b is the bias and we set bias as zero in our work;r(?)is the sigmoid function.We use tanh(?)and r(?)as the activation function of hidden layer and output layer,respectively.Though it is theoretical that RNN is able to process arbitrary length sequences,it tends to suffer from the gradient vanishing and exploding problems which limit the length of reachable context.

2.2.LSTM model

In order to learn long-term memory of the input sentences and to address such problems of RNN,the LSTM unit is incorporated into the RNN model.31As shown in Fig.2,the standard LSTM unit contains output gate ot,forget gate ft,input gate it,memory cell ctand hidden state ht.32,33The role of the gates is to control the fiow of information,that is,which information should be input,forgotten and output,respectively.The output gate is used to shield error by taking itself offline,the forget gate is used to empty the memory contents,and the input gate is used to ignore incoming activations.The memory cell is the core of LSTM unit to store information over long-time durations.htis regarded as a representation of the current time step with contexts.In order to generate ht,a temporary result ltis first computed by tanh over the ensemble of input xtand the preceding hidden state ht-1.

Fig.2Architecture of LSTM.

In this paper,the LSTM unit contains peephole connections to fully utilize the presentation of memory cell.The transition functions of LSTM are shown as Eqs.(3)–(8).

where Wiand Wri(r,i=1,2,3,4)are input connections and recurrent connections of output gate,forget gate,input gate and cell,respectively;Wpi(p,i=1,2,3)are peephole connections;tanh(?)and r(?)are activation functions.

3.Proposed checking model

A new model is proposed to solve the checking problems of Chinese radiotelephony read-backs,as illustrated in Fig.3.The ATC’s instructions and the pilot’s read-backs have been transformed into text format,and the semantics of the sentences are extracted and verified by this scheme which has two channels to process the sentences and to learn the semantics of instructions and read-backs,respectively.The procedure of the new model is elaborated as follows:

Step 1.The instructions(SATC)and the read-backs(SP),which have been converted into text format,are segmented into single words by the Chinese word segmentation method.

Step 2.The words of SATCand SPare mapped into vector space by word embedding methods.According to our previous study,34the sentence lengthof read-backs is usually shorter and the relation among the words is weaker than common sentence modeling tasks,so each segmented word is coded with one-hot vector in this paper.

Step 3.The semantic model of SATC(MATC)and the semantic model of SP(MP)are established based on LSTM,respectively.The semantic representation models MATCand MPhave the same structure.They both have two layers which are input layer and hidden layer.The input layer is used to input segmented word vectors of SATCor SP.The hidden layer is applied to generate the semantic vector of the whole sentence.The model of MATCor MPactivates one input word vector for each time-step in sequence,which may obtain the contextual information of sentence.35

Supposing that htis the hidden activation vector in timestep t,it represents the semantic information entered at timestep t and before.When the last word vector xTof the sentence is input into the semantic model,the corresponding hidden activation vector hTis supposed to be the semantic representation vector of the whole sentence,29and it puts more emphasis on the latter words.

Step 4.To represent the whole sentence more properly and express each word more equally,at the last time-step T,the output of LSTM model is put into a mean-pooling layer,whose structure is illustrated in Fig.4.

The pooling layer can help a model retain the most prominent and prevalent features,which is helpful for robustness across examples.In this layer,all the outputs of hidden layer of LSTM are combined in a concatenation as the input of the pooling layer.The widely adopted mean pooling is applied and the output of mean pooling is the average vector of all outputs of hidden layer of LSTM.For the hidden activation vector in time-step t is the semantics of the t-th word,the meanpooling layer can fully utilize the semantics of all words in SATCor SP.The semantics of the former words are also balanced against the semantics of the latter ones.So the output ofthislayeristhefeature representation ofsentence SATC(vATC),or the feature representation of sentence SP(vP).

Step 5.The semantic difference between SATCand SPis transformed into the distance between vATCand vPin the vector space.A MLP layer is adopted to learn and extract the semantic similarity quantitatively.In this part,MLP is constituted by input layer,hidden layer and output layer.vATCand vPare concatenated into a vector,which is the input of the MLP layer.The output of MLP is used to indicate the semantic relatedness of SATCand SP.

Fig.3Flowchart of proposed checking scheme.

Fig.4LSTM model with mean-pooling layer.

Step 6.A classifier layer is applied to classify the output of MLP layer into two classes so as to verify whether the semantics of SATCand SPis consistent or not.According to our previous study,34KNN is used in our work.

The two models of MATCand MPhave to be trained together according to the value and the variance of similarity of vATCand vPin the training process.The corpus used in our work is labelled,and then supervised learning method is utilized to train the model.The training processes of MATCand MPare the same.We train the models by minimizing the cross-entropy error.The error isdefined as

where Sris the semantic relatedness which is the output of MLP layer in our checking model.L is the labelled number,and if the sentence pair is consistent in semantics,L is defined as one,otherwise L is defined as zero.We use Back Propagation Though Time(BPTT)to learn the model parameters and the momentum method is used to accelerate the convergence.36

4.Experiments and analysis

4.1.Corpus of Chinese ATC radiotelephony read-backs

In order to analyze the performance of the new checking model for the consistency verification of ATC sentence pairs,we built an experimental civil aviation radiotelephony communication corpus(ATC Corpus)in Chinese.The corpus is built up according to the original recordings of radiotelephony communication between pilots and air traffic controllers,and also the professional books used for civil aviation radiotelephony training.Since there are lots of proper nouns in the recordings,the pronunciation of numbers and letters is different to common Chinese language,and the actual speed of ATC-pilot communication is much faster than common conversation.We invited two professional air traffic controllers to listen to these recordings and translated them to text format separately so as to ensure the correctness of the corpus.

In this paper,we chose sentence pairs of read-backs(SATC-P)from the corpus to conduct these experiments.800 SATC-Ppairs that are annotated with consistency(the positive samples)were selected from the corpus.For improper read-backs do not occur regularly,we chose and designed different types of improper read-backs according to the investigation of communication problems in civil aviation.500 SATC-Ppairs that are inconsistent in semantics formed the negative samples,so the total samples are 1300 SATC-Ppairs that have covered the common ATC call codes and the conversations during the whole fiight phase.The train set consists of 500 positive samples and 300 negative samples,which are picked up randomly from the total samples.The test set(Total)consists of the other rest samples,in which 300 samples are positive(Positive)and 200 samples are negative(Negative).

4.2.Experimental results based on traditional checking model

According to the traditional RNN and LSTM,a traditional checking model of Chinese read-backs was proposed in Ref.35.To compare the proposed checking model,the experiments based on traditional model were conducted.The procedure of traditional checking model is shown in Fig.5.

The preprocessing of the SATC-Ppairs is the same,while the outputs of RNN/LSTM models are directly put into the cosine similarity layer to compute the relatedness of vATCand vP.The KNN classifier is applied to verify the semantic consistency of sentence pair.The one-hot vector was used to represent segmented words of SATC-Ppairs.The two channels are trained together according to the variance of cosine similarities of vATCand vP.

Similarly,test accuracy and Mean Squared Error(MSE)are used to evaluate the performance of the checking model.For the checking of read-backs is of crucial importance to the safety of aviation transportation,the checking model has to be more sensitive to improper read-backs.So in the experiments,the test accuracy of Total(Pt),the test accuracy of Positive(Pp),and the test accuracy of Negative(Pn)are computed and compared.The experimental results based on the data set described in Section 4.1 are shown in Fig.6 and Table 1.

Fig.5Traditional checking model of Chinese read-backs.

Fig.6Test accuracy of traditional checking model.

Table 1Average test accuracy and MSE of traditional checking models.

In the experiments,the model was trained and tested thirty times randomly.Fig.6 illustrates the test accuracies based on RNN and LSTM.From Fig.6 and Table 1,we can figure out that the test accuracy based on LSTM model is more stable than that based on RNN model,and the traditional checking model based on LSTM achieves the highest average accuracy in Positive,Negative,and Total.The average test accuracy of Positive is higher than that of Negative in both models of RNN and LSTM.Besides,Figs.7 and 8 show the distribution of similarities in one random test of traditional checking model based on RNN and LSTM,respectively.It can be seen that all the similarities acquired by the traditional model based on RNN are very high.But in fact the negative sample’s similarity should be lower than the positive sample’s similarity.There should be obvious distinctions between the positive samples and the negative samples if their semantics is expressed properly enough by the model.By contrast,the similarities acquired by the traditional model based on LSTM have more obvious distinctions.It is indicated that LSTM has more advantages in semantic extraction.So the experimental results give a baseline to be compared with the proposed checking model of readbacks in this paper.

4.3.Experimental results of proposed checking model

To analyze the performance of the new checking model with added mean-pooling layer and MLP layer,the experiments are conducted as follows:(A)only the mean-pooling layer is added to the checking model based on LSTM,and the similarity of vATCand vPis also computed by cosine similarity;(B)only the MLP layer is added to the checking model based on LSTM,and the outputs of hidden layer at the last time-step T are concatenated into a vector,as shown in Fig.9,where[P(1),P(2),...,P(T)]is the input sequence of pilot’s read-back;(C)the proposed model with both mean-pooling layer and MLP layer is tested.

Fig.7Distribution of similarities of traditional checking model based on RNN.

Fig.8Distribution of similarities of traditional checking model based on LSTM.

Fig.9LSTM model with MLP.

Fig.10Test accuracy of checking model with different layers based on LSTM.

The experimental procedure is similar to the traditional checking model.The experiments with different layers are conducted separately,and the data set described in Section 4.1 is used during the experiments.Thirty independent tests are arranged for each experiment,and each test is conducted with randomly chosen train set and test set from the total samples.

The test accuracies of the checking model with different layers based on LSTM are illustrated in Fig.10.The average test accuracy and MSE are shown in Table 2.As indicated in Fig.10,the test accuracy of the proposed model(Meanpooling+MLP)is obviously higher than that of the other two models with different layers,and the model with MLP layer achieves the secondary test accuracy.The fiuctuation of the proposed model is the smallest in the experiments.

From Table 2,we can see that the average test accuracy of Negative is obviously improved in the three experiments compared to the traditional checking model by introducing different layers to the LSTM model.Also,it can be seen that there is a better balance among the average test accuracy of Positive and Negative by adding the pooling layer or/and the MLP layer.That is,the added pooling layer and MLP layer are helpful to improve the performance of the checking model.

The checking model based on LSTM with MLP layer has lower MSE and higher average accuracy than the traditional checking model based on LSTM.It indicates that the relatedness of the semantic vectors(vATCand vP)is expressed moreaccurately by MLP instead of similarity function,and more interaction of vATCand vPare taken into account by this layer.

Table 2Average test accuracy and MSE based on proposed model.

However,the average test accuracy of the checking model with mean-pooling layer is lower than that of the traditional checking model.The primary reason is that the cosine similarity after the mean-pooling layer discards some important information because the particular regions of concatenated vectors,which are obtained by the mean-pooling layer,come from different underlying sources.Moreover,by comparing the experimental results of the proposed model and the model with MLP layer,it can be indicated that by concatenating all the hidden activation vectors the mean-pooling layer introduces more information.The features extracted after mean-pooling layer are more prevalent.So it is easier to learn the relatedness of vATCand vPfor the latter MLP layer.

It can be seen that all the average accuracies of the Total,Positive and Negative have been increased in the experiment,and at the same time the MSE has been reduced,which means that the proposed model is more robust and more superior for the checking task of Chinese read-backs.Figs.11–13 show the distribution of similarities of total samples in one random test based on LSTM with different layers,respectively.It can be apparently seen that the similarities have better discrimination than that obtained by traditional model,and the new model has the best distinction in the similarity of vATCand vP.From the analysis above,we can figure out that by adding a meanpooling layer and a MLP layer to the LSTM the new model is more suitable to extract the semantics of SATC-Ppairs and to represent the similarity relatedness of the SATC-Ppairs.

4.4.Comparison results in extended ATC Corpus

In order to make further test of the proposed model,we extended the ATC Corpus according to real recordings and professional training books of radiotelephony communications.There are 2442 SATC-Ppairs in the extended ATC Corpus,including 1326 positive samples(consistent in semantics)and 1116 negative samples(inconsistent in semantics).The traditional checking models based on RNN and LSTM are compared to theproposed modelin this section.Allthe experimental results shown in this section are conducted on the extended ATC Corpus.For each test,according to the experimental procedure described in Section 4.1,the train set consists of 1952 randomly selected samples,in which there are 1060 positive samples and 892 negative samples.The other 490 samples comprise the test set,in which 266 samples are positive,and 224 samples are negative.Each model was trained and tested 30 times randomly.The test accuracy and MSE are calculated and compared,as shown in Fig.14 and Table 3.

Fig.11Distribution of similarities of total samples based on LSTM with mean-pooling layer.

Fig.12Distribution of similarities of total samples based on LSTM with MLP layer.

Fig.13Distribution of similarities of total samples based on proposed model.

Fig.14Test accuracy of different models in extended ATC Corpus.

Table 3Comparison results of average test accuracy and MSE in extended ATC Corpus.

It can be seen that the proposed model in this paper has greatly improved the performance in read-backs checking of radiotelephony communicationscompared to traditional checking model.The average test accuracies of Total,Positive,and Negative are also improved obviously,and the average test accuracy of Total test set is raised up to 92.47%.The average test accuracy of Negative has been increased to 90.82%which means that the new model makes a better balance among the average test accuracies of Positive and Negative samples.Therefore,the new model is more sensitive to the read-back errors.As shown in Fig.14 and Table 3,the performance of the new model is much more stable than that of traditional checking models based on RNN and LSTM,and the MSE has been reduced obviously.So,as analyzed in Section 4.3,the comparison results in the extended ATC Corpus have further proved the outstanding performance of the proposed model for Chinese read-back checking.

5.Conclusions

To check the radiotelephony read-backs in civil aviation ATC intelligently and automatically,a proposed model is proposed based on LSTM.According to the experimental results and analysis described above,we can make conclusions as follows:

(1)The deep learning method is introduced to model the semantics of the radiotelephony communications in ATC,and a new semantic checking model of readback errors is proposed in this paper to verify whether the meanings of the SATC-Ppairs are consistent.

(2)The traditional LSTM is improved by adding a meanpooling layer to concatenate all the hidden activation vectors.It has advantages in fully utilizing the semantic information underlying different sources.In addition,the mean-pooling layer is helpful to achieve a better balance among the semantic representation of the former words and the latter words.So the semantic feature of the whole sentence is more prevalent than that extracted by the traditional LSTM.

(3)The MLP layer suppliesmore interaction of the semantic vectors of SATC-Ppairs compared to traditional cosine similarity function.It can express the deep relatedness of the SATC-Ppairs more accurately,and as a result the differences of the quantitative similarities acquired by the MLP layer are more distinctive between the negative samples and the positive samples.

(4)The average test accuracy of Negative samples is increased significantly,and the performance of the proposed model is highly improved in terms of test accuracy and MSE.So,by adding a mean-pooling layer and a MLP layer,the proposed checking model is more robust to check the read-back errors intelligently and automatically.The proposed model described in this paper also gives an encouraging way to model the semantics of Chinese radiotelephony communications in civil aviation air traffic control.

Acknowledgements

This work is supported by the National Natural Science Foundation of China(Nos.61502498,U1433120 and 61806208)and the Fundamental Research Funds for the Central Universities,China(No.3122017001).