Training effectiveness evaluation of helicopter emergency relief based on virtual simulation

Xue SUN,Hu LIU,c,*,Guanghui WU,Yaoming ZHOU

aSchool of Aeronautic Science and Engineering,Beihang University,Beijing 100083,China

bKey Laboratory of Aircraft Advanced Design Technology,Ministry of Industry and Information Technology,Beijing 100083,China

cState Key Laboratory of Virtual Reality Technology and Systems,Beijing 100083,China

dCommercial Aircraft Corporation of China,Ltd.,Shanghai 200126,China

KEYWORDS Analytical network process;Emergency relief;Fuzzy comprehensive evaluation;Helicopter;Training effectiveness evaluation;Virtual simulation

Abstract Helicopters are playing an increasingly important part in emergency relief,such as earthquake rescue, fire fighting and medical transport.With the development of virtual simulation technology,virtual simulation-based training is widely used in the training of the helicopter crew especially for the dangerous and costly missions mentioned above.A complete training effectiveness evaluation method is proposed to evaluate the trainees’training effect based on virtual simulation in this paper.A key to this method is regarding the complicated process as a discrete event-activity flow system and establishing the evaluation indicator system.Then expert group and Analytical Network Process(ANP)are applied to determine the weight of indicators.When the training data are processed,there is a novel attempt to apply Fuzzy Comprehensive Evaluation(FCE)model to calculate the two categories of indicators.Eventually,an experiment and the analysis were carried out to validate the evaluation method.

1.Introduction

Earthquake is one of the major natural disasters causing casualties,and when earthquake occurs,how to search and rescue trapped people quickly and effectively is the necessary quality for rescuers1.Statistics have showed that it is hard to predict earthquakes,so earthquake emergency relief is undoubtedly key to reducing casualties after the earthquake.Faced with complex and difficult rescue missions,how to improve the efficiency of search and rescue is the top priority2.Earthquake search&rescue refers to the process of searching and rescuing survivors and carrying out emergency and basic medical care when needed3.Helicopter rescue means applying helicopters into emergency rescue,such as search and rescue work,stock presentation,and airborne command,which is widely used in many countries.Helicopter rescue operations often confront crews with unique challenges in which even minor errors can result in dangerous situations4.The earthquake relief in a complex environment involves two vital factors:(A)the well trained rescuers;(B)specialized rescue equipment5.

Helicopters are playing an increasingly significant role in earthquake relief,which overcome the restrictions of ground traf fic congestion and win the precious time for search and rescue.The mission is not only a battle of equipment but also a battle of rescuers.In order to fully reflect the mission effectiveness,the effectiveness evaluation can not only stay on the level of equipment,but also focus on the impact of man–machine integration on the result,because well-trained crew is a guarantee of successful rescue.There are two widely-used training methods for rescuers.One is live training which is hard to achieve because the reproduction of post-earthquake environment is quite difficult in real life and the cost of helicopter flight is fairly high.The other is virtual simulation-based training.Virtual reality technology is used to create realistic training environment and at the same time the simulation technology is applied to realize the interactive operation which can partially or completely achieve the effect of live training.Virtual simulation-based training is an effective support measure to reduce the training cost and risk,and all kinds of severe environment and disasters can be simulated by using virtual simulation,the application of which has great practical value in the field of disaster relief6.Recent popularity of consumergrade virtual reality devices,such as the Oculus Rift and the HTC Vive,has enabled household users to experience highly immersive virtual environments7.Simulation training provides a promising tool to train the management of complex multidisciplinary setting,and thus reduces the occurrence of fatal errors and increases the safety for both the patient and the Helicopter Emergency Medical Service(HEMS)crew8.The United States Army has heavily leveraged,developed and expanded its use of virtual simulation training,as this class of simulation has been empirically demonstrated to be effective in the transfer of skills to the live environment9.In response to the emergency,an Advanced Disaster Management System(ADMS)has been developed in America.The system provides a powerful simulation training platform for trainees,which simulated all kinds of emergency scenario such as fire and medical rescue.The operation,emergency communication and team coordination abilities of trainees improved a lot by using this training system.Li et al.designed an immersive and novel virtual reality training environment to teach individuals how to survive in earthquakes7.

Effectiveness refers to the degree that a system used to perform a certain mission can reach for the expected goals under certain conditions,and it embodies the ability of system to complete stipulated task and the value of the system as well10,11.Evaluation on the training of the trainees is necessary for live training and so is the virtual simulation training.Virtual reality training systems are commonly used in a variety of domains,and it is important to understand how the realism of a training simulation influences training effectiveness4.There are plenty of literatures to prove the effect of virtual simulation-based training in many fields.Fragomeni et al.described the methodology of the TEE with regard to the effectiveness of augmented virtuality as a platform within the Call for Fire(CFF)task domain and how augment virtuality technologies and methods can impact CFF training12;Gallegos et al.demonstrated that there was a significant improvement in the real assembly performance of subjects who undertook virtual assembly training first when compared to those trained conventionally13;the study of Voelker et al.indicated that curriculum-based mentored VR simulation training improves the performance level of cardiology fellows in coronary interventions14;Tian et al.proved that simulation-based aircraft carrier marshalling training is better than paper-based training by carrying out an experiment15.Meng et al.built the performance evaluation model of virtual equipment maintenance training system based on fuzzy comprehensive evaluation method16.More researches can be found in the literatures17–19on this topic.

But all these studies mentioned above are aimed at the effect of virtual simulation-based training compared with traditional training rather than how to evaluate the trainees’training effect after training based on virtual simulation.Some problems remain unsolved in the assessment of virtual simulation-based training for personnel:How to evaluate the effectiveness of the trained,how to quantify the improvement of the trainees’ability after training,and how to ensure the quality of training.By using the Kirkpatrick model,Chen X M et al.built a training effectiveness evaluation model of helicopter forest fire fighting mission but ignored the other phases of fire fighting process and the interrelation between the indices20.Based on Chen X M’s study,Chen J improved the mission model,but the handling of qualitative indexes and quantitative indexes was not taken into account21.However,there is no standard,complete evaluation method of the training effectiveness evaluation for trainees in virtual simulation-based training.According to the insufficiency of virtual simulation training effectiveness evaluation and taking the Helicopter Earthquake Search and Rescue Mission(HESARM)for example,a novel evaluation model is first proposed to solve practical training effectiveness evaluation problems based on the Analytical Network Process(ANP)model and fuzzy comprehensive evaluation model,and then the quantitative evaluation result can be obtained.At the end of the paper,a detailed simulation case is used to validate the evaluation method.

2.Modeling of HESARM

2.1.A mission model based on discrete event-activity flow

Effectiveness evaluation of HESARM is essentially a complicated multi-index problem,and a reasonable evaluating indicator system is the crucial premise of the scientific effectiveness evaluation22.Discrete event systems can be decomposed into several basic units of ‘‘events-activities-events”,different units are connected by a common event,and thus the event-activity flow model of the system is formed21.HEASRM is such a complex activity which involves so many mission phases and training points that it is impossible to analyze the whole process and construct the evaluating indicator system directly.Fortunately,this mission includes a series of events(E)and activities(A)which are in time sequence and satisfies the definition of discrete event system,so a discrete event model containing a series of events and activities can be built as a replacement for the entire task process.In order to build the right mission model and establish a reasonable index system,we are supposed to have comprehensive understanding and analysis of the mission.

According to the modeling theory of discrete event systems and the characteristics of HESARM,the mission model can be obtained as follows in Fig.1:earthquake occurs(Start),relevant units request rescue(A1),the emergency rescue unit receives the rescue mission(E1),the crew plans the mission including rescuers,rescue equipment, flight path and so on(A2),completes mission planning(E2),checks the equipment such as helicopter,winch and radio facility(A3),completes equipment inspection(E3),the helicopter flies to the disaster area(A4),arrives at the disaster area(E4),searches for trapped people(A5),determines whether or not the trapped are found(G1),carries out rescue if the trapped are found(A6),victims are saved(E5),returns to the base(End).

First,the above discrete event-activity flow model,which decomposes the mission into a series of events and activities,makes the whole process more clearly and the establishment of the index system easier.In addition,in this time-ordered model,upstream events and activities may have an impact on the downstream ones,so ANP is introduced to judge the interaction relationship between different indicators;however downstream events and activities cannot have an impact on upstream ones,and thus the complexity is greatly simplified when the experts analyze the correlation of indicators.

2.2.Simplification of mission model

The research in this paper only applies to a single helicopter.Because in the real earthquake search and rescue activities,several helicopters are often needed to search and rescue in harmony,the resource and task allocation as wellas information sharing and other issues among different helicopters should be considered,which makes the effectiveness evaluation abnormally complicated.

From the above task model,we can see that the HESARM involves many events and activities,which are not conducive to the establishment of the index system.All the events and activities prior to the task planning(A2)were not related to the training of the rescuers,so they were not considered in the simplified task model.Then the whole process of mission can be divided into three main parts:mission planning(AP)(A2,E2),equipment inspection(AC)(A3,E3),search and rescue(ASAR)(from A4 to A6).In the case of a generalized helicopter rescue mission,the mission process can also be reduced to three phases,which not only simplifies the process but also facilitates the establishment of index system.

3.Establishment of training evaluating indicator system of HESARM

3.1.Establishment of two-dimensional index system based on mission model

As mentioned earlier,the HESARM can be divided into three main phases including mission planning(AP),equipment inspection(AC),executive search and rescue mission(ASAR).There are different training focuses and training requirements for trainees in each phrase.

During the mission,the crew should try their best to complete the rescue mission by cooperating with each other,so not only the quality of the rescue task is vital but also the cooperation ability between the crew is crucial examination factors.Then the effectiveness evaluation indicators can be divided into two categories shown in Fig.2:task completeness indicators and collaborative indicators20.Task completeness indicators mean the costs and benefits of completing a certain task,including time,fuel and success rates;collaborative indicators refers to the crew’s ability to work together to complete the task,including quality of task planning,reliability of equipment inspection,formulation of rescue strategy,etc.

Fig.1 Discrete event-activity flow model of HEASRM.

Fig.2 Two-dimensional evaluation indicator system.

3.2.Optimization of indicator system

Using the classic Delphi method,experts were consulted on the rationality of indicator system and their feedback was collated and progressed.Then the feedback was given to experts anonymously and they were asked for advice again.After two rounds of consultation,experts agreed on getting rid of I10,so in the optimized indicator system,emergency disposal ability I10has been removed.There are two reasons: first,as emergency disposal ability is a reflection of the operation ability of the crew,the index system will be redundant and the weight of‘‘operation ability” index may become too high if emergency disposal ability is regarded as an independent indicator;besides,in actual search and rescue mission,the probability of emergency is relatively low,and the simulation of emergency is difficult.Eventually,the hierarchical indicator system is obtained as follows in Fig.3.

4.Weight of evaluation index based on ANP model

4.1.Modeling of ANP based on expert group

ANP,proposed by Wind and Saaty in 1996,is developed on the basis of AHP(Analytic Hierarchy Process)23.And ANP model,which describes the relationship between objective things more accurately,is a more effective and practical decision method adapted to the independent hierarchical structure.

Fig.3 Hierarchical indicator system.

ANP model takes the form of a relative scale and takes full advantage of experts’experience and judgment.In order to improve the credibility of evaluation results,the expert belief map is adopted in this paper to describe experts’expertise and confidence degree.24In Fig.4,horizontal axis represents the expertise of experts,[0.5,1.0],and the ordinate means expert confidence degree,[0.5,1.0].

The experts independently make the judgement and do not interfere with each other,so it can be regarded as a group decision-making process where Bayesian Decision Theory(BDT)can be used25,which integrates the subjective probability estimation of experts,then uses the Bayesian formula to modify probability,and finally re-uses expectations and fixed probability to make optimal and more objective decisions.

One of the most crucial part of ANP modeling is determining the relationship between indicators,i.e.,establishing an indicator correlation matrixindicates the influence of the index i on the index j,and the range is[0,1].In order to simplify the matrix,if the interactional relationship between the two indicators is small or negligible,then γij=0;if the interactional relationship between two indicators cannot be ignored,then γij=1;the impact of the index I on itself is not considered,which means that γiican be replaced by ‘‘”.

According to the analysis of statistical theory,the reliability of evaluation and number of experts follow exponent distribution,namely with the increase of number,the reliability of evaluation will improve.But the credibility will remain almost unchanged when the number of experts is more than 20.26In order to improve the reliability of evaluation as much as possible,the expert group in this paper is made up of 15 experts,including 2 specialists in earthquake search and rescue,9 experienced crew,and 4 postgraduates familiar to the mission.Then the indicator correlation matrix can be obtained as shown in Table 1.

ANP model can be established according to the correlation matrix which is showed in Fig.5,where AP,ACand ASARare three benchmark index clusters Ci(i=1,2,3).

4.2.Construction of judgment matrix based on expert information

The judgement matrix is the basic information of ANP,which is mainly used to compare the dominance of the elements.For a certain layer of factors,the judgement matrix means the comparison of relative importance of the related factors in a sublayer.The comparison must be done between factors which are dependent and have an impact on each other27.Based on the judgement of expert system,a 1–9 scale method is used to determine the relative importance of indicators in this paper.

Fig.4 Expert belief map.

A judgement matrix of single expert k(Ek)is obtained by comparing the importance of all indicators I1,I2,...,In.

The weight vector of experts is

where m means the number of experts.The final judgment matrix is obtained by the convex composite synthesis.

With the increase of number of experts,the above judgment matrix can eventually converge to objective judgment matrix28,which greatly reduces the impact of subjective judgment and owns higher credibility.

5.Progressing of training effectiveness

5.1.Qualitative index processing based on fuzzy comprehensive evaluation

In the index system,qualitative indicators are inevitable.The indicator data,which cannot be got through the virtual simulation training system directly,rely on judgement of the experts,for example,high and low,good and bad,etc.Therefore,in order to ensure the credibility of the results as much as possible,an appropriate method to quantify the qualitative index is essential to minimize the impact of subjective judgement.

Fuzzy comprehensive evaluation method is on the basis of fuzzy mathematics and applies fuzzy relationship synthetic principle29.Using this method,factors without clear boundary and not easy to get quantitatively are quantified.The method gathers all the opinions and reflects the quality of evaluated objects comprehensively,which has been widely used.The specific principles of fuzzy comprehensive evaluation can be found in Refs.16,29,30.

The probability vector of group membership of index j given by expert i is

Then the probability matrix of group membership of index j given by expert group is

Table 1 Indicator correlation matrix.

Fig.5 ANP model.

To integrate expert system’s judgment,the convex composite synthesis method is applied once again and the evaluation result vector of index j is obtained finally.

where ‘‘°” is the fuzzy synthetic operator,and the common M(·,+)model retaining more evaluation content is applied in this paper.And its computing mode is consistent with the matrix operations.

In accordance with the maximum membership principle,the final evaluation grade of index j is grade k thatbelongs to.Then the result vector is mapped to a specific value.

where vkis equal to the evaluation score of the class k.

5.2.Calculation of quantitative indicators

For quantitative indicators,the specific data can be directly got by virtual simulation-based training system.But because of different dimensional scope of transformation and antagonism,using the original index data for calculation directly is unreasonable.

Threshold method is applied to compare the threshold value with the original value of indicators so that the indicators can be converted into standard values.

The following formula is used for indexes whose values are positively correlated with the crew’s performance.

The following formula is used for indexes whose values are negatively correlated with the crew’s performance.

where max（ xi），min（ xi）mean the maximum limit or the minimum limit in the virtual simulation-based training system.

5.3.Calculation of training effectiveness value

According to the weight of each indicator W= ［ω1，ω2，...，ωn］and the values of qualitative and quantitative indicators E=［E1，E2，...，En］T,the effectiveness of HESARM can be calculated as follows:

First,the instructors are able to know the operation ability of trainees and determine whether they meet the requirements by their training effectiveness;besides,by the effectiveness results of single indicator,the trainees can find their training weakness,optimize the training plan,and improve training quality.

The whole evaluation method of helicopter emergency rescue demonstrated above can be described as Fig.6.

6.Case study

6.1.Calculation of indicator weight based on super decision software

ANP model principle and the calculating process are quite complex,so artificial calculation is almost impossible.It is difficult to apply ANP to solve practical decision-making problems if computing software is not used.William and Rozann developed the Super Decision(SD)software,which provides real application for ANP model31.SD software which owns friendly interface can calculate any ANP model and fully express the calculation results,so SD software was used in this paper.

Fig.6 Evaluation method of helicopter emergency rescue.

According to the ANP index system built above,the network model of HESARM was established in SD software,which can be seen in Fig.7.

Taking the ‘‘training effectiveness of HESARM” as the main criterion and the index ‘‘search and rescue” as the subcriterion,we got the judgement matrix by the expert group.And the judgment matrix was input to the SD software in the form of questionnaire.Then the normalized weight vector of indicators and the result of consistency check could be obtained as showed in Fig.8.

After repeated input and calculation,the stability weight of indicators in the limit state was obtained eventually,which is showed in Table 2.

6.2.Comparison with AHP method

In order to compare the ANP model and AHP model,the AHP model was established in the SD software showed in Fig.9 and the weight of indicators was obtained according to the judgement matrix given by experts.

It can be seen from Fig.10 and Fig.11 above that the weight of the two evaluation models differs.The weight of task index and collaborative index was 0.248 and 0.752 respectively in the ANP model,while in the AHP model,the weight of two categories of indicators was 0.491 and 0.509 respectively.Namely,when considering the interplay between indicators,the weight of task indicators declined,while the weight collaborative index increased.The reason was that the collaborative operations between the crew had a significant impact on the completion of the task.In virtual simulation-based training,improving the crew’s collaborative operation ability was an important aspect of evaluation of training effectiveness,so the effectiveness evaluation index system based on ANP model was more reasonable.

6.3.Contrast training experiment and analysis

The whole training system consists of six main parts:task scenarios is used to set different tasks for the crew,including mission time,site and so on;control platform can be used for trainers to control the whole mission;task training part is the core of the system,which supplies an operation platform for the trainees;the situation monitoring is used to observe the whole task in the third angle;effectiveness evaluation part is used to receive the training data and calculate the effectiveness;the resource is a library to save all the models.The system framework is showed in Fig.12.

Fig.7 ANP model in SD software.

Fig.8 Dominance and consistency check of indicators.

There are all kinds of realistic 3D virtual scenes to enhance the vraisemblance of virtual simulation-based training and the pictures in Fig.13 show virtual disaster scene after earthquake.

An experiment was conducted in this paper.In the experiment,there were two groups(A and B)of trainees participating in training of HESARM using a certain virtual simulation-based training system.There were four people in each training group:pilot,co-pilot,operator and lifeguard.The pilot and the lifeguard wear virtual reality helmets which provide a more realistic scene and better sense of immersion when performing a mission.After a round of training,the training data were recorded and the effectiveness of both groups(A and B)was calculated and analyzed.It can be seen in Fig.14.

Table 2 Stable weight of indicators.

Fig.9 AHP model in SD software.

In this paper,there are four grades{excellent,good,medium,bad}in the evaluation set(see Table 3),which conform to the human habits of recognizing and distinguishing things.The HESARM is difficult and dangerous,so the requirement of the crew and the evaluation criterion are relatively high.

According to the training process of group A,membership evaluation matrix can be obtained by the judgement of experts.Now the calculation of I2is taken as an example.

Considering the weight of the expert group,the final fuzzy judgment matrix of I2is obtained by using the method of convex function group.

Fig.10 Single indicator weight contrast between ANP and AHP models.

Fig.11 Weight contrast of two categories of indicators between ANP and AHP models.

Fig.12 Framework of virtual simulation-based training system.

Fig.13 3D earthquake scene.

Fig.14 Virtual simulation-based training of crew.

The membership vector of I2in group A can be got after normalization:where 0.504,0.397,0.083,0.016 respectively represent the possibility that the training result of index I2in group A belongs to the four different grades ‘‘excellent,good,medium,bad”.Obviously,according to the maximum membership degree principle mentioned above,maximum 0.504 means the greatest possibility,so training result of index I2in group A belongs to the ‘‘excellent” grade.By referring to Table 3,the evaluation value is EA2=0.9.As you can see that the judgment some experts made is not consistent with the final evaluation which means fuzzy comprehensive evaluation method is more comprehensive and more credible.

Table 3 Evaluation standard.

Similarly,the evaluation results of other indicators of both groups A and B can be calculated:

Using the maximum membership degree principle again,the evaluation grades and values of all the qualitative indicators are shown in Table 4.

After several rounds of expert consultation,the limit values of all quantitative indicators are got by normalizing(cost indicators focus more on the minimum,while efficiency indicators focus more on the maximum),which are shown in Table 5.

The original data of quantitative indicators obtained by the virtual simulation-based training system are shown in Table 6.

After fuzzy comprehensive evaluation and normalization using Eqs.(8)and(9),the values of the qualitative and quantitative indexes are shown in Table 7.

Table 4 Grades of qualitative indicators of trainees.

Table 5 Limit of quantitative index.

Table 6 Original data of quantitative indicators.

Table 7 Values of evaluation indicators.

The values of the training effectiveness of groups A and B are obtained by bringing the index weight and the index value to the efficiency value calculation formula mentioned above.

Effectiveness is not only reflection of the operation ability of the trainees but also an effective way for faculty to know the trainees’current ability,so the training plan and focus of training can be adjusted to get better training effect.

According to Table 3,EfA=0.823 shows that the overall grade of group A is ‘‘Excellent” which means the trainees are able to complete the task smoothly,but there is room for further improvement.According to Table 7,most of the indicators are of high grades,for instance,they planned the mission almost perfectly and had a high success rates of search and rescue.But there are still several indexes’values under 0.8 which means that some improvement is needed.For example,they spent a little more time in mission planning and equipment inspection.For group B,EfB=0.734 represents that the overall grade of group B is ‘‘medium”which represents that the crew can meet the requirements of task,but there are some unreasonable parts and further training is necessary.It can be seen from Table 7 that the reliability of equipment inspection was high but they spent too much time in searching and rescuing and most of the indicators are between 0.7 and 0.8.Therefore,they are supposed to be more familiar to the whole mission and much training is needed to become more skillful and cooperate better.

By comparing the overall effectiveness evaluation results of two groups,the training effectiveness of group A(grade of‘‘good”)is significantly higher than that of group B(grade of ‘‘medium”)as showed in Fig.15;in terms of the effectiveness of each indicator in Fig.16,the operational indicators’values of I2,I6,I8of group B are much lower than those of group A,which indicates that operation skills of group B should be reinforced.Besides,for group B,in search and rescue phase,the formulation of search and rescue strategy is low and the time consuming is rather high,so training in this regard should be strengthened especially.Furthermore,the evaluation values of cost indicators of both groups are generally low,which means that all trainees are lack of operation proficiency,so constant training is required to improve the operation skill,reduce the task cost and increase the efficiency of training.

Fig.15 Evaluation effectiveness of two groups.

Fig.16 Single indicator effectiveness contrast between two groups.

7.Conclusions

(1)According to the characteristics of HESARM,the discrete activity-event flow model is established,which not only makes the mission process more clear and simpler,but also makes it easier to build indicator system according to the time order.At the same time,the indexes mapped by the discrete model are categorized into two classes:the cost indicators and the coordinative indicators,so as to facilitate the subsequent processing.

(2)The ANP model considering the relation between the indexes is used to calculate the index weight and by comparing with AHP method,the analysis finds that the index weights obtained by ANP model are more reasonable.

(3)As the evaluation indexes are divided into qualitative and quantitative indicators,fuzzy comprehensive evaluation method is applied to effectively reduce the expert subjective influence on the evaluation results,which makes the evaluation results more objective and reliable.

Training effectiveness evaluation is feedback on the training,by analyzing which,the trainees could know their weakness and optimize training plan to yield twice the result with half the effort.A complete evaluation method of trainees’training effectiveness based on virtual simulation is proposed based on ANP-FCE in this paper.A contrast experiment was conducted to demonstrate the application of this method and the results show that this method is reasonable and feasible.This method is valid not only for the HESARM but also universally for general helicopter emergency rescue activities.

Acknowledgments

The authors are grateful to Mr.Sichao YANG and Mr.Yuxuan LIU for their contribution to developing the training system based on virtual simulation,and to experts of China Flying Dragon General Aviation Co.,LTD and Ministry of Transport’s Second Rescue Team for their cooperation with questionnaire.