Team effectiveness evaluation and virtual reality scenario mapping model for helicopter emergency rescue

Xue SUN, Hu LIU,c, Yongling TIAN,*, Gunghui WU, Yun GAO

a School of Aeronautic Science and Engineering, Beihang University, Beijing 100083, China

b Key Laboratory of Aircraft Advanced Design Technology,Ministry of Industry and Information Technology,Beijing 100083,China c State Key Laboratory of Virtual Reality Technology and Systems, Beijing 100083, China

d Commercial Aircraft Corporation of China, Ltd., Shanghai 200126, China

KEYWORDS Helicopter emergency rescue;Scenario mapping model;Team effectiveness evaluation;Virtual reality (VR);Virtual simulation-based training

Abstract The application of helicopter emergency rescue is becoming increasingly widespread,but the flight crew training in this area is still difficult due to high cost and risk.Nevertheless, with the development of Virtual Reality(VR)technology,virtual simulation has become a significant role in crew training of helicopter rescue. During the implementation of VR-based training, how to transform complex real tasks into VR scenarios and how to evaluate the performance of crew are of great importance. To address these issues, a novel VR-based R-E-A-D (Report, Evaluate, Agree, Do)evaluation model for training is proposed, which is suitable for complex missions with multiple tasks,multiple scenarios,and multiple people.Then,a mapping method of VR scenarios is put forward,which can transform the real tasks into virtual scenarios to serve the virtual simulation training. Finally, an experiment is carried out to verify the feasibility of the evaluation method and virtual scenario mapping method.

1. Introduction

Helicopter emergency rescue, especially medical service is increasingly widely used all over the world which is safe,available and equitable.1Its main benefit is evacuation and the maintenance of readiness before rapid transport of the physician to the scene or the patient to the hospital.2Helicopter emergency medical services play a crucial role in providing timely emergency medical services to the critical patients in remote areas3as the provision of ground ambulance services is time-consuming.4However, some studies have shown that providing and developing Helicopter Emergency Medical Services(HEMS)may face serious challenges,5one of which is the safety concerns of HEMS.4So many organizations are looking for ways to make the HEMS safer.6A significant method is to strengthen the training of rescuers. But live training is usually unrealistic because the reproduction of practical harsh environment is hard and the cost of helicopter flight is extremely high.7

With the development of virtual simulation technology,simulation-based training has become a generic method in training situations, which are too dangerous or logistically impossible to have users participate in an actual event.8Virtual Reality (VR) can be used to immerse users in deliberate and repetitive skills practice thus enhance mastery learning and retention.9Training systems based on VR have been used in several areas.10Army and Marine Corps have noted benefits from the use of simulation-based training — both in terms of training effectiveness and in terms of cost savings or cost avoidance.11Currently, the most widely used area is training in the medical field.12-15Simulation-based education has become a ubiquitous teaching approach in nursing.16Patient care simulations have proven to be effective to enhance user learning.17Although the procedure of medical operation is complicated, the virtual scenarios can be concentrated and vivid when focusing on the fidelity of virtual models and the precision of interactive operation. Taking a medical operation as an example, the main training scene is the abdominal cavity,12but helicopter emergency rescue is a kind of complex procedural missions involving multiple scenes, multiple tasks,multiple crew and multiple training points, which is difficult to be realized in a single virtual scene. Nowadays, most simulation-based trainings in aviation are still working on individual operation of aircraft rather than the whole mission and team coordination. In addition, mapping the real mission scenarios to the virtual reality ones is rarely studied. To this end,this study explores new approaches in the evaluation and construction of virtual scenarios.

The evaluation after training is an essential part for improving the skills of trainees.7In,18Kirkpatrick’s model has been applied to various fields of training by using 4 level evaluations like evaluation of learning outcomes in higher education,19,20performance of medical educators.21But it is still hard to realize the full four-level evaluations. Besides,the evaluation model is usually used in enterprises for vocational education and training evaluation of short duration courses17which is not applicable in this study. The Army and Marine Corps is looking into how simulation-based training contributes to soldier training11but no service has established metrics or indicators to assist them in more broadly measuring the impact of simulation-based training on improving the performance or proficiency of service members.11

In this paper,a team effectiveness evaluation model of helicopter emergency rescue is proposed based on VR.In addition,a mapping method of virtual scenarios is put forward to map real missions into virtual reality scenarios for the sake of more reasonable trainings.

2. Training evaluation model of helicopter rescue based on virtual simulation

2.1. Establishment of READ evaluation model

The DECIDE model was proposed by Benner in 197522which includes several aspects of detect, estimate, choose,identifying, do and evaluate. The model was gradually developed into a judgment method in the decision-making process of pilots in CRM (Crew Resource Management) training.23It is a process in which pilots detect, identify, and diagnose possible problems, then determine available solutions and conduct risk assessments. This model can be used to serve the pilot in solving potential problems, but it does not further explain how to evaluate the pilot’s operation and the process from ‘detect’ to ‘evaluate’ is a little complicated.In addition, this model focuses on individual evaluation and crew’s collaboration is not emphasized. In order to solve issues mentioned above, new evaluation approach is needed. However, as DECID model is the most authoritative method for CRM training,23it is convincing to take this model as a pointcut to evaluate the team training effectiveness. Moreover, a large number of cases of helicopter emergency rescue were analyzed and experienced crew members was interviewed. Then, the ‘‘READ” evaluation model,namely ‘report, evaluate, agree, do’ is proposed, the process of which is more feasible.

‘Detect’ in the DECIDE model means the crew’s observation of the external environment, including eye movements,turning heads etc., which is difficult for experts to judge whether the trainees have received the necessary information. Besides, in order to make the right decision, all the crew should report important information they detect to the captain.

‘Estimate,choose and identity’is a process of analyzing and making decisions according to external changes.However,it is hard to distinguish the three steps in an emergency.So we integrated them into ‘evaluate’.

A primary difference between a team and an individual is that the team needs to make decision together. Taking helicopter emergency rescue for example, although the captain,the core of the team plays a major role in decision-making,other crews also need to participate in it. Therefore, before proceeding to the next step, the entire team must reach an agreement in order to operate smoothly.

‘Do’is the direct cause of the change in results.Whether the operation is required and proper affect the mission greatly.And this indicator can be obtained intuitively: one is through the output data of the training system,and the other is judged by the expert.

Based on the analysis above, the interaction among team members and the external environment are shown in Fig. 1.

The data source of effectiveness evaluation is from the output of training system, the external observation and evaluation.24In this model, most data are got from external evaluation except‘do’,which is a combination of both.Therefore,the effectiveness of team(ET)can be expressed as a function of evaluation indexes R, E, A, D.

Fig. 1 Interaction among team members and environment.

It should be noted that the assumption of this training and evaluation is that the team has mastered the basic operation of their respective fields. In other words, the training and model mainly emphasizes the coordination ability between the crew and the captain’s decision-making ability.

2.2. Determination of index system and index weight

Based on the READ model, a complete evaluation index system shown in Fig. 2 is obtained by using Delphi method.25And the assessment indicators may be slightly different with different expert groups involved and using diverse questioning ways. All indices in this system are explained as follows.

R - report: clarity refers to that the reporter’s pronunciation is clear when reporting, and the reported information can be easily identified by the recipient; integrity indicates whether the reporter is able to detect all necessary factors and communicates them to the recipient in full. Professionalism means the reporter is able to use professional briefings and avoid unnecessary misunderstandings; positivity refers to positive tone and attitude of the reporter; interactivity means that the receiver gives feedback to let the reporter know whether he has got the signal or not, meanwhile, the reporter decides to end this round of report or explain the message further.

E - evaluate: information grasping capability indicates the ability of receiver to grasp and extract the received information accurately and to reflect it in the response plan;the scene analysis capability refers to analyzing the current state of the helicopter,the external environment by collecting the information from other reporters;Formulating capability means being able to develop a sound implementation plan based on current conditions.

A - agree: The degree of cooperation in decision-making process includes two main factors, whether the opinions of all members are fully considered, and whether the crew are informed after making the decision; the decisiveness of the decision-making process means that the crew can make decisions quickly without hesitation, which can reflect the proficiency of emergency treatment; the rationality of the decision result means that whether the decision can save the team and helicopter in the crisis and accomplish the task. However, it should be noted that the captain is the core of the team, who has the highest authority. Namely, in regular operations, the crew can make their own decisions. However, in emergency,team members should report their decision to the captain and if no agreement can be reached after discussion, the captain has the final say.

D-do:The correctness of the execution process means the members are able to perform the corresponding operations according to the command of the captain and whether the operation is in place; the decisiveness of the execution process means that whether the member can respond quickly without panic and delay.

Fig. 2 READ evaluation index system.

In the READ model, ‘R’ and ‘D’focus on the independent ability of members,while‘E’tests the comprehensive judgment ability of members and the focus of ‘A’ is on the decisionmaking and collaboration ability of crew; the rescue effect is the goal of the whole operation, which runs throughout the entire operation.

2.3. Calculation of index weight and efficiency value

By using the Delphi method,the index weight questionnaire is designed, and the index weight can be obtained by statistical analysis.

Through multiple rounds of inquiry, the kurtosis of each round can be calculated and Kk≥0 should be guaranteed,which means expert opinions reached a consensus. Then the weight of index can be obtained.

By analyzing the performance of trainees in the training process, which is scored in each stage, the effectiveness value of the personnel can be calculated. Taking indicator R as an example, nomenclatures are listed in Table 1:

Rk-m-ncan be scored directly by the instructor, while Rk-mcan be calculated by the following formula:

3. Hierarchical mapping model of virtual reality scenarios

In virtual reality training,the virtual training scene is the basic unit of human-computer interaction and human-in-loop virtual training. Therefore, ensuring the quality of simulation iscritical.16Besides, evaluation cannot exist without training scenes. Thus, in the design of virtual simulation scenarios,the scene should be taken as the basic design unit.This underlies the significance of approaches to construct authentic and reasonable virtual simulation scenarios.

Table 1 Nomenclature of a mission.

3.1. Principles of scenarios construction

The loop from‘report’ to‘do’is the smallest loop to complete an event, that is, the processing of an event requires at least one loop. The more complex the event is, the more loops are required to complete the event.But if there are too many loops in a virtual scene, it would be difficult to evaluate the performance of the team. Therefore, it is necessary to ensure that each scene is not too complicated. The R-E-A-D loop mentioned above can be described as Fig. 3. For a certain event,the loops develop from the inner to the outer sequentially.

1) The principle of least loops

A scene contains a set of R-E-A-D loops.

Generally,the larger the n is,the more loops the scene contains and the more complex the scene is,which is unfriendly to the assessment. So, the smaller the n, the better.2) The principle of minimal scenes For a particular mission, the total number of scenes contained is roughly constant and can be regarded as a set of scenes

where m is the number of scenes in a mission.

In general,the simpler each scene is,i.e.,the smaller n is,the more scenes there are, i.e., the larger m is and vice-versa.Therefore, the number of loops in a scenario (n) cannot be reduced to one, otherwise the number of scenes in a mission would be too large,which has negative effect on the evaluation and is not conducive to the construction of scenes and training.Therefore, it is necessary to find the best balance between m and l, but there is no definite functional relationship between them, so it cannot be obtained by accurate calculation.

However, we know most of the elements involved in the task, as well as the purpose of training, so we can solve this problem through the organization and decomposition of the mission scenarios.

3.2. The establishment of hierarchical scenario mapping model

3.2.1. Determine the mapping levels

Fig. 3 R-E-A-D loops of a mission.

Helicopter rescue mission is time-ordered7and involves multiscene, multi-tasks, multi-personnel and multi-training points,so the top-down decomposition mode is adopted to design the training content. Initially, there will be some key nodes in the whole process, such as takeoff, search, rescue, which are relatively specific stages, so the next level can be divided according to execution stages of the mission.

Furthermore, many steps are required to complete a stage of the task, so steps are taken as the next level of decomposition. Lastly, each step contains many scenes, which are the bottom level of the scene organization model.

In this way, the emergency rescue mission is disassembled and stratified in an orderly manner, and the whole mission process is divided into four levels: ‘‘mission - stage - step -scene”. Based on that, different training points are integrated into different scenes to facilitate the whole design and evaluation of virtual simulation training.

3.2.2. Establish the discrete event flow model of tasks and determine the task stages

Since tasks are time-ordered, the task flow can be sorted out according to the actual implementation of tasks and a discrete event flow model would be established. Besides, this kind of task involves large terrain and virtual simulation scene is bounded by geography, thus the key locations can be used to divide the task stages. Following this way, the events between the core region of the key location and the adjacent key points are selected as the key subtasks, which are the key events between adjacent points in the discrete event flow model.

3.2.3. Identify the steps of task stage

This stage involves several key steps under each subtask,namely the operation steps needed to complete the subtask.As helicopter emergency rescue is a sequential process, all the stages can be broken down into more specific and executable steps.In other words,the steps should not too complicated to carry out.

3.2.4. Identify typical scenarios

The simulation scenario, the basis of human-computer interaction is the bottom of the model. Typical scenarios are extracted according to task sorting, where theme elements and interaction relationships should be described.

Then,the model can be obtained as follows in Fig.4,which breaks a complex mission down into four distinct layers.

3.3. Bidirectional evaluation based on lantern model

According to the above analysis, ‘READ’ has two meanings:one is a smallest operation loop in a simulation training scene,and the other is an overall indicator system for performance evaluation. In this way, the corresponding relation between simulation training scenes and efficiency evaluation is established.In other words,the evaluation is not only for the whole mission, but can be used in a single scene (or step, stage),which makes the assessment quite flexible. For every scenario,step and stage, we can evaluate the performance of crew in RE-A-D 4 respects. The corresponding relation of scenes and evaluation indicators is shown as the lantern mapping model in Fig. 5.

Fig. 4 Hierarchical mapping model of virtual scenarios.

Till now, two-way evaluation is achieved by mapping scenes and evaluation metrics,which is shown in Fig.6 and treating R-E-A-D as independent indicators.

Longitudinal evaluation indicates a comprehensive evaluation of the R-E-A-D ability within the same level in the training system, so as to obtain the comprehensive ability evaluation under a certain scene (or step, stage and even mission). Then,the performance of a certain step can be obtained by combining the evaluation results in multiple scenes under this step,and so on. This can clearly identify scenes (or steps, stages,missions) in which the trainees perform poorly.

Horizontal evaluation means to evaluate the four aspects R-E-A-D separately and individually. Taking indicator R as an example, the general report ability evaluation results of a crew in the whole operation can be obtained by integrating the reporting ability under several scenes (or steps, stages).Therefore, we can clearly find out their weakness, and then carry out a targeted training. 2-dimentional evaluation greatly increases the pertinence and flexibility of evaluation.Based on different missions and evaluation requirements, general evaluation results of various stages and aspects can be obtained.

It’s worth noting that the virtual simulation scenarios can be updated and modified by the feedbacks of virtual training and evaluation in virtual environment.The whole effectiveness evaluation process and hierarchical mapping method of virtual scenarios proposed in this paper are depicted in Fig. 7.

4. Virtual simulation training experiment model

4.1. Construction of virtual simulation scenarios

Fig. 5 Lantern mapping model of training scenarios and evaluation index.

Fig. 6 Bidirectional READ evaluation model.

Fig. 7 Evaluation model and virtual scene mapping method.

Taking helicopter medical aid for example, the whole rescue process is depicted in Fig. 8 which can be simply described as follows: after receiving the task command, crews and ground staffs begin to prepare and check the equipment, and then take off to fly to the accident site. To clarify the process with more details, the task is refined and decomposed in the form of discrete event flow.28

According to this discrete event flow model,the whole process can be summarized as the following 4 stages: equipment inspection, flying to the accident site, rescue and return. Next,each stage includes several specific steps.For example,the first stage includes external inspection,internal inspection,start-up and pre-departure inspection.After that,each step can be further decomposed into several possible scenes.

By consulting cases of helicopter medical rescue and interviewing the experts of helicopter medical rescue, the training scenarios are summarized, which are often encountered, but difficult to carry out training in real life. By simulating the cases using VR, the team can be trained in the specific VRbased training system with a strong sense of immersion and experience. It is important to notice that the virtual scenarios include normal and emergency operating procedures but not all practical situations. The contents, types and quantities of scenarios can be adjusted according to training requirements.

An important part of the rescue process was selected to illustrate how to determine the hierarchical scenarios, which can be found in Table 2.

Unity 3D,one of the most popular virtual simulation engines,is dedicated to the development of VR and other fields.It is widely used in the development of education and training,industrial simulation, and other VR products.29Therefore,Unity 3D is used in this paper to develop VR training scenarios. According to the construction method and the list above,we established a set of hierarchical VR scenarios, and some typical ones are shown in Fig. 9.

4.2. Calculation of index weight

Super Decision (SD) software30is used in this paper to calculate the indicator weight using which the judgment matrix can be given (see Fig. 10).

Fig. 8 Task discrete event flow model and geographical node division.

Table 2 Scenarios extraction from real mission.

Finally,the normalized weight of R,E,A,D indicators(ω)and weight of sub-indexes(φ)can be got as shown in Table 3.

4.3.The implementation of helicopter rescue training experiment

To begin with, the crew received a call for help that one passenger was seriously injured in a car accident on a freeway.There will be life danger at any time thus emergency rescue is needed.

There are four people in the flight crew including captain,co-pilot, doctor and nurse who need to cooperate to complete the rescue mission.Virtual helmets and rockers are used as system input in the training. The virtual training system contains all the virtual scenarios above. All the details of the process including operation and communication are recorded for the following evaluations.The VR-based training system is shown in Fig. 11.

In this paper, several scenes are selected to train the crew,and the training process is recorded for subsequent evaluation.

4.4. Evaluation results

There are four grades {excellent, good, medium, bad} in the evaluation set, which consists with the human cognition and differentiation habits. The detailed criteria can be found in the Ref.7

Landing on the freeway are taken as an example to evaluate the performance. By recording the performance of the team and analyzing the loops and indexes of R-E-A-D, we can get the results in Table 4.

The instructor grades each part according to the crew’s performance, and the evaluation result is in Table 5.

Similarly,other scenes can be processed in the same way to obtain the evaluation results. Due to the limit of length, this paper only presents an evaluation of three scenes in step‘landing’. By synthesizing the evaluation results of the three scenes according to the bidirectional evaluation method,we can obtain the evaluation results of the four horizontal indicators and the comprehensive effectiveness results of the single scene and step.

Fig. 9 Hierarchy virtual reality scenarios.

We divided 20 volunteers into 5 teams Ti（i=1，2，···，5）,and there are 4 crews in each team. After rounds of VRbased training, we screened 55 groups of valid data from 60 and evaluated the performance of teams with the READ method to get the efficiency value of teams, which is shown in Fig. 12, where the horizontal axes is times of training, and the vertical axes means the evaluation value of performance in a mission.

It can be seen that with the increase of the number of training, the team effectiveness is also constantly improved,although there are slight fluctuations. Even though some of them are of poor skills at the beginning, they can reach a higher level as long as they train enough times. For example,the score of T1for the first round is under 0.6, but they make great progress after rounds of training, the results of whom are higher than others. This is consistent with common sense of training and recognized by experts. Therefore, the READ model is reasonable and can be used to evaluate the performance of team.

Fig. 10 Judgement matrix in SD software.

Table 3 Weight of evaluation indicators.

Then,one of the training results of T4was picked randomly and analyzed, and the results are shown in Figs. 13 and 14 below.

It can be seen from the horizontal evaluation results that,with this step, the four R-E-A-D indexes of the team obtained by combining the three scenarios are 0.682, 0.691, 0.690 and 0.738 respectively, which indicates that the team has the relatively strong execution ability, while the other three aspects are weak. This result is consistent with the situation that the team is a new combination and they are not familiar with each other. Therefore, the execution ability of the single person is acceptable, but the coordination ability between members is poor.

According to the results of the longitudinal evaluation, the comprehensive scores of the three scenes are all around 0.7 under the landing step, which is consistent with the qualification of the team. All the crews are young and lack of experience in performing tasks, so their comprehensive ability scores are generally not high.

Moreover, through the post-training survey, trainees believe that this kind of virtual scenarios organization and training method can make them more interested and involved in the VR-based training, which is of great help to improve their training efficiency and effectiveness.

5. Conclusion

1) Aiming at the characteristics of helicopter emergency rescue, the READ evaluation model of team training effectiveness is proposed. Then the READ bidirectional evaluation method is given by combining evaluation with scenarios, which greatly increases the pertinence and flexibility of evaluation.Through horizontal evaluation,a comprehensive evaluation of the R-E-A-D ability within the same level can be got,so as to obtain the comprehensive ability evaluation under a certain scene; by longitudinal evaluation, the results of R-E-A-D four aspects in the whole training process can be obtained.Based on different tasks and evaluation requirements,evaluations of various stages and aspects can be obtained. Finally, the validity of this method was confirmed by a helicopter rescue training experiment.

Fig. 11 Personnel and equipment in VR-based training experiment.

Table 4 R-E-A-D loops in certain step.

Table 5 Evaluation of R-E-A-D loops.

Fig. 12 Evaluation values trend of 5 teams.

Fig. 13 Horizontal evaluation results of T4.

Fig. 14 Longitudinal evaluation results of T4.

2) The hierarchical mapping model of the virtual simulation scenarios is proposed by using ‘R-E-A-D loop’,where the real mission can be mapped into the virtual reality system and is divided into four levels: mission,stage, step and scene. In this way, the real mission can be transferred to the well-organized VR training system,which is suitable for virtual training.Then,our practices show that this method can reduce the workload of modeling, improve the efficiency and the VR-based training system is friendly to trainees. Finally, this method is not only suitable for helicopter emergency rescue, but can be applied to other complex, multi-person, multiscenes and procedural missions,such as training for firefighters and police officers.