Effects of input method and display mode of situation map on early warning aircraft reconnaissance task performance with different information complexities

Chorn LIANG, Shung LIU, Xioru WANYAN,*, Chengping LIU,Xu XIAO, Yuchen MIN

a School of Aeronautic Science and Engineering, Beihang University, Beijing 100083, China

b China Academy of Electronics and Information Technology, Beijing 100041, China

KEYWORDS Early warning aircraft;Eye tracking;Heart rate/heart rate variations;Human-Machine Interface(HMI);Touch screens

Abstract Early Warning Aircraft (EWA) are the main force for air detection and its Human-Machine Interface(HMI)should be designed to support task efficiency and safety.With the application of advanced input method and interface design in EWA, little is known about their actual usability in terms of human factors and ergonomics. The aim of this study was to investigate the effects of the input method and display mode of the situation map on EWA reconnaissance task performance with different information complexities. Eighteen participants attended a threefactor within-subject design experiment with input method(touch screen and mouse),display mode of the situation map(color and grayscale),and information complexity(high and low)as the independent variables. Participant behavior performance, subjective workload, heart rate/heart rate variability, and eye movements were recorded as the dependent variables. The results suggest that a touch screen requires greater task completion time and has greater physical demands than mouse operation;however,it also facilitates information processing by reducing the average fixation time.Color mode significantly decreases saccade counts compared to grayscale mode and is considered more appropriate for target search tasks as it induces less visual search load. High information complexity produces significant negative effects on behavior performance and subjective workload.It also has significant interaction effects with input method on fixation and saccade counts. The findings have implications in the optimization design of Human-Machine Interface for EWA task systems.

1. Introduction

In warfare,Early Warning Aircraft(EWA)are the main force for air detection and play a critical role in surveillance and reconnaissance systems.1In EWA reconnaissance,an operator monitors the radar search area presented in a situation map and detects potential threats. Thus, the Human-Machine Interface(HMI)for this time-critical mission should be appropriately designed to support task performance, efficiency, and safety.2With the introduction of advanced input methods and interface display designs, evaluation of new technologies in terms of human factors and ergonomics can be valuable in identifying possible advantages and potential risks before actual use.

The input method (mouse, trackball, touch screen, touch pad,and joystick)is an important part of the HMI,with which an operator can select objects, configure options, and enter data into the systems.3With direct input, touch screens have gained considerable interest in flight deck design and command and control (C2) systems.4A touch screen provides a new means of intuitive interaction, with zero displacement between input and output, control and feedback, and hand action and eye gaze.5Compared with indirect input devices(mouse and trackball), a touch screen can potentially reduce the cognitive workload and improve the situation awareness of the pilot.6,7

Recent studies have investigated differences in input methods on the flight deck,8in the main control room,9,10in driving,11and in unmanned vehicle control.12van Zon et al.found that a touch screen produced less movement time for aircraft navigation tasks, but also produced less accurate performance compared to a mode control panel and a control display unit.8In a study of input devices for menu navigation tasks, four styles (track ball, rotary controller, touch pad,and touch screen) were compared; the touch screen produced the shortest task time, lower ratings for hand discomfort,and higher ratings for usability. However, the touch screen was also rated poorly in terms of physical workload and whole-body discomfort.3Although the touch screen offered fast movement for simulated pointing tasks in these studies,it was less clear in the reconnaissance task workspace compared to a mouse.Ulrich et al.suggested that mouse provided an accurate and fast operation for control board tasks.9Noah et al. also indicated that a touch screen provided data entry benefits but required an additional step in navigation compared to a mouse.10The characteristic differences in touch screen and mouse operation in complex reconnaissance tasks are unclear; they are further explored in this study.

User interface display design is another important part of the HMI; good interface design can help an operator perceive and understand the task conditions.13,14In military systems, a situation map is the core of the user interface; many information elements are displayed,such as multi-attribute targets,orientation, navigation, and system status. For EWA reconnaissance tasks, operators must distinguish the object from different targets in the situation map. Thus, the design of the situation map to improve reconnaissance task performance has been a focus of study.

Color-coding of the situation map has generally been recognized as an effective means to improve task performance.15Previous studies have found that color-coding reduced target search time16,17and workload in flight tasks compared to monochrome displays.18Yeh and Wickens reported that the coding of differential classes facilitated performance and reduced the negative influence of clutter.19However, as there are many information elements displayed in a situation map,detailed specification of colors should be considered as it may add system complexity and conflict with previously learned coding conventions.20Zhao et al.suggested that grayscale scenes provided a greater sense of realism than color scenes for rendezvous and docking tasks.21To our knowledge,there is little research on the effects of the situation map display mode on EWA reconnaissance task performance; this issue is further investigated in this study.

Since the first airborne early warning system was introduced, the technologies and functions of EWA have been greatly developed,22extending its management to command,control, communication, computer, and intelligence (C4i) systems and producing problems of information complexity and high workload. Processing of copious information is challenging;the HMI efficiency may deteriorate with information overload,23,24resulting in reduced situation awareness and performance.25Thus, input method and display mode of the situation map should be examined with different information complexities. More frequent actions are usually required in high information complexity conditions; a touch screen may not be appropriate for continuous target selection as it may produce more arm fatigue.

Although a series of previous studies have been performed to investigate the effects of input methods and display modes on the human-machine interaction performance, different characteristics of task scenarios led to different research conclusions.It can be seen that the tasks implemented in the most previous studies required the subjects to move certain objects presented on the interface to their destinations3,8,9and had relatively less interactions with complicated mission systems.However, compared to the simple pointing tasks, more complex information processing is demanded in the EWA reconnaissance tasks, including multi-source target perception,rapid target recognition and judgement, comprehensive decision-making, as well as continuous manipulation with the mission systems. In our study, display modes may have influence on operator’s perception of map situation elements,information complexity may increase the difficulty of operator’s judgment and decision-making,and input methods determine the way in which operator interacts with mission systems.

In EWA reconnaissance task systems, the operator traditionally uses mouse to select menu, scale and drag situation map, and right click to trigger command window, etc. Grayscale mode rather than color mode of situation map is commonly used during task execution. Considering the newly introduced touch screen technology and color mode of situation map, their usability should be further examined, especially under different information complexity conditions.Therefore, the purpose of this study was to investigate the effects of input method (touch screen and mouse) and display mode of the situation map (color and grayscale) on EWA reconnaissance task performance and physiological responses with different information complexities. We tested the following hypotheses:

(1) A touch screen improves task performance due to its direct input but is more physically demanding.3

(2) The color mode of a situation map benefits target searching in reconnaissance tasks.18

(3) Input method and display mode interact with information complexity.

2. Methods

2.1. Participants

Eighteen participants(twelve men,six women,mean age=23.67 years,SD=1.68)were recruited to participate in the experiment,and all of the participants came from the school of aeronautic science and engineering in Beihang University. The participants reported that they were right-handed and had normal or corrected-to-normal vision.A post hoc statistical power analysis using G*Power software indicated that at 0.1 partial η2and a 5% significance level, the number of participants was sufficient to reach a statistical power at 80%.26Due to experiment platform failure, data for one participant was not recorded. Experimental data from 17 subjects were analyzed in this study.All participants provided a written informed consent form before the experiment.

2.3. Experimental design

A three factor (2 × 2 × 2) within-subject design was used in this study. Input method (touch screen and mouse), display mode (color and grayscale) and information complexity (high and low) were the independent variables. The participants completed tasks through mouse or touch screen operation.The situation map was displayed in color mode or grayscale mode, as shown in Fig. 1. Different information complexities were created through the number of targets presented in the radar search area.25With high information complexity,participants searched for a specific target within a whole circle; the low information complexity condition corresponded to the right half-circle. The effects of input method, display mode,and information complexity were assessed by user behavior performance (task completion time and target search reaction time), subjective workload rating (NASA-task load index,NASA-TLX) and physiological measures (eye tracking and heart rate/heart rate variability).

Task completion time was defined as the total time spent by a subject to perform a task. Target search reaction time was the interval between subject hearing the search order and entering the first target parameter. Given the task properties,we focused on task time rather than accuracy as most participants could successfully complete the task after training.NASA-TLX is widely used in subjective workload assessment in human factors evaluation;27six dimensions (mental, physical,temporal,performance, effort,and frustration)were rated with 100-point scales. A previous study suggested that the unweighted version(without pairwise rating)was highly correlated with weighted calculation;28it was used in this study as it was more acceptable to participants.29Eye-tracking measurements included fixation counts, average fixation time, saccade counts,and average saccade peak velocity,as shown in Table 1.Heart rate and heart rate variability (Inter-Beat Interval (IBI)and Standard Deviation of Normal to Normal (SDNN)) were used as physiological measures to evaluate operator workload.

2.4. Experimental tasks

2.2. Apparatus

A high-fidelity human-machine interaction platform for EWA was used in this study,as shown in Fig.1(a).Through the platform,EWA reconnaissance tasks could be simulated and completed by mouse or touch screen operation. The user interface consisted of a menu bar and a situation map with two display modes (color mode and grayscale mode). In color mode, situation map elements were color-coded (ocean, land, city, road)(Fig. 1(b)); in grayscale mode, map elements were distinguished by different grayscale values(Fig.1(c)).The user interface was designed according to the current design for a certain type of EWA.

Eye movement data were recorded using Tobii Glasses 2(Tobii Group, Stockholm, Sweden), a wearable eye-tracking system with a sample rate of 50 Hz and accuracy to less than 0.5 degree. The eye-tracking raw data were processed using Tobii Pro Lab software. An additional Photo Plethsmo Graphy(PPG)device was used concurrently to record subject earlobe pulse data with a sampling frequency of 64 Hz; the data were analyzed using Ergolab software (Kingfar International,Beijing) to assess heart rate and heart rate variability.The selected tasks were determined based on Subject Matter Expert (SME) interviews and operation procedures for EWA reconnaissance tasks. In this study, the scenario simulated EWA reconnaissance tasks in which participants were required to determine software settings and search specific targets. The experimental task was divided into three phases: route planning, radar configuration, and target searching.

In the route planning phase,subjects searched four cities in the situation map and connected them in sequence.During this phase,participants needed to perceive the city information displayed on the situation map according to visual searching,make judgement and determine whether it was the target city needed to be selected, scaled and moved the map until all the target cities were found through touch screen or mouse operation.

In the radar configuration phase, a window was opened by a pointing command in the menu bar, which displayed the radar configuration parameters such as search mode and search area. Subjects deleted the existing parameters and set up new configuration parameters according to the requirements. During this phase, participants needed to firstly perceive the radar configuration parameters information, made comparison and judged whether the parameters satisfied the task requirements, and then input parameters through touch screen or mouse operation.

Fig. 1 Human-machine interaction platform and user interface, color mode of situation map with high information complexity, and grayscale mode of situation map with low information complexity.

Table 1 Eye-tracking measurements and definitions.

In the target search phase, a given numbered target was found in the search area and its parameters were input with a keyboard, as performed by Tseng et al.30Yellow-coded targets were randomly distributed in the red-coded radar search area. Each target had a signage indicating its heading, speed,number, etc. Once a participant prepared and requested a search object, a search order (e.g., reporting the speed and heading of target number 258) was given to the participant;the task was completed when the participant had found two designated objects. During this phase, target information and search order were perceived by participants, and then they should decide whether the number of the yellow targets was consistent with the order and operate the situation map until all the required objects were found.

2.5. Procedures

Participants were familiarized with the study purpose and procedures. Their demographic information was collected before the test. Experiment task instruction was conducted. After becoming familiar with the task scenario, participants were encouraged to practice until they could perform the task skillfully with touch screen and mouse operation. Before the main test,a NASA-TLX questionnaire was explained for consistent understanding. The eye tracker was implemented and calibrated using a one-point calibration method. The PPG device was clipped to one earlobe.

During the main test, participants completed four trials(two input methods and two information complexities) for one display mode,taking a 5 min break before continuing with the other display mode.Participants were asked to complete a NASA-TLX questionnaire after each trial to measure perceived workload. The sequence of trials was counterbalanced to prevent fatigue and the practice effect.All participants were allowed 3 h to finish the experiment.

3. Results

A 2 ×2 ×2 repeated measure analysis of variance (ANOVA)was performed to test the effects of input method, display mode, and information complexity. The significance level was set to 0.5 and partial η2was used to estimate the effect size.Data normality was examined using Shapiro-Wilk tests.Experiment data were analyzed using IBM SPSS version 23 software.Error bars represented standard deviation,and asterisks represented significant differences in conditions.

3.1. Behavior performance

3.1.1. Task completion time

The input method(F(1,16)=17.794,p=0.001,η2=0.527),and information complexity (F(1, 16) = 6.428, p = 0.022,η2= 0.287) had a significant influence on task completion time; the display mode did not (p = 0.9). Task completion time was greater with touch screen operation than with mouse operation, and greater with high information complexity than with low information complexity(Fig.2).No significant interaction effects were observed (all p ＞0.1).

3.1.2. Target search reaction time

Information complexity had a significant influence on target search reaction time (F(1, 16) = 6.369, p = 0.023,η2=0.285).Target search reaction time was greater with high information complexity than with low information complexity(Fig. 3). No significant differences were found for input method (F(1, 16) = 0.463, p = 0.506, η2= 0.028) or display mode (F(1, 16) = 0.018, p = 0 0.896, η2= 0.001). Similar to task completion time, there were no significant interaction effects (all p ＞0.1).

3.2. NASA-task load index

Fig. 2 Scatter plot of individual results of task completion time for each independent variable.

Fig.3 Scatter plot of individual results of target search reaction time for each independent variable.

significant effects (all p ＞0.1). Information complexity had significant effects on workload and most sub-dimensions of the NASA-TLX. Touch screen operation required greater physical demand than mouse operation.In addition,perceived workload and most sub-dimensions(mental,performance,and effort) were increased with high information complexity.

There was only one two-way interaction effect: display mode x information complexity on temporal demand (F(1,16)=6.498,p =0.021,η2=0.289). Fig.4 shows that grayscale mode required more temporal demand than color mode with high information complexity (p = 0.042).

3.3. Heart rate/heart rate variability

Fig.5presents the results for heart rate and heart rate variability for each independent variable. From the graph, input method had significant effects on heart rate (F(1,16) = 16.189, p = 0.001, η2= 0.503) and IBI (F(1,16)=20.114,p ＜0.001,η2=0.557).Touch screen operation increased heart rate and decreased IBI compared with mouse operation.No other significant effects were found(all p ＞0.1).

Input method × display mode had an interaction effect on heart rate (F(1, 16) = 8.813, p = 0.009, η2= 0.355). Heart rate was higher with touch screen operation than with mouse operation in color mode (p = 0.001) and grayscale mode(p = 0.024). The difference in heart rate was greater in color mode (2.47) than in grayscale mode (0.83).

3.4. Eye movements

Eye movement measurements include fixation counts, average fixation time,saccade counts,and average saccade peak velocity. The ANOVA results are shown in Table 3.

3.4.1. Fixation counts

Input method (p ＜ 0.001) and information complexity(p=0.004) had a significant influence on fixation counts; display mode did not (p = 0.845). Touch screen operation produced greater fixation counts than mouse operation. Inaddition, input method interacted with information complexity (F(1, 16) = 5.802, p = 0.028, η2= 0.266). High information complexity produced greater fixation counts than low information complexity, but only with touch screen operation(p=0.001,Fig.6(a)).The difference in information complexity was not significant with mouse operation.

Table 2 ANOVA results for NASA-TLX.

Fig. 4 Interaction effect of information complexity and display mode on temporal demand.

3.4.2. Average fixation time

There was only one significant effect of input method on average fixation time(p=0.006);average fixation time with touch screen operation was lower than that with mouse operation.No other significant effects were found (all p ＞0.5).

3.4.3. Saccade counts

Display mode (p = 0.002) and information complexity(p = 0.019) had a significant influence on saccade counts;input method did not (p = 0.797). Grayscale mode produced more saccades than color mode. Information complexity also interacted with input method (F(1, 16) = 7.926, p = 0.014,η2= 0.361). As with fixation counts, high information complexity produced a greater saccade count than low information complexity with touch screen operation(p=0.001,Fig.6(b));no significant difference was found with mouse operation.

Fig. 5 Heart rate/heart rate variability for each independent variable.

Table 3 ANOVA results for eye tracking measures.

Fig. 6 Interaction effects of input method and information complexity.

3.4.4. Average saccade peak velocity

Input method had a marginal significant effect(p=0.082)on average saccade peak velocity. The interaction of input method × display mode was also marginally significant (F(1,16) = 3.374, p = 0.088, η2= 0.194). Touch screen operation produced greater average saccade peak velocity than mouse operation (p = 0.029) in grayscale mode, but they were comparable in color mode.

4. Discussion

This study was conducted to examine the effects of input method and display mode of the situation map on EWA reconnaissance task performance with different information complexities. The results of the experiment, based on a highfidelity EWA human-machine interaction platform, showed that touch screen operation and high information complexity produced negative effects on operator task performance and workload.Display mode tended to interact with input method and information complexity.

4.1. Effects of input method

Based on the experimental results, hypothesis 1 is partially supported. Unexpectedly, task completion time was greater with touch screen operation than with a mouse, possibly because participants had to keep their arms outstretched with touch screen operation, causing greater physical demand and fatigue. Subjective and objective workload assessments with touch screen operation indicated greater NASA-TLX physical demand and heart rate,and a shorter IBI;an increase in heart rate and decrease in heart rate variability have been correlated with workload.31,32Thus,a touch screen may not be appropriate for tasks that require operators to raise their arms without support for long periods of time.

Another contributor to long task completion time with a touch screen was more repeated operations due to operation error.It was observed that some touch screen interactions can be triggered unintentionally (dragging the map for a long distance may trigger the long press command).Hong et al.found that a touch screen had a significantly higher error rate for dragging tasks that required constant contact with the screen.33In previous studies, Fitts’ law was used to evaluate the movement time for different input devices, and the results showed that a touch screen had the shortest movement time,explained by the direct interaction nature of a touch screen.8,34,35However, in the more complex task environment in our study,participants were required to point at a target and also drag and zoom situation maps.In contrast,mouse operation provides a faster and more accurate means of interacting with the task software system,9,10with important implications for new touch screen technology in actual task environments.

Most previous studies used behavior performance or perceived usability to compare different input devices. In this study,touch screen and mouse input methods were also examined in terms of their differences in eye movement behaviors.The touch screen had a significantly shorter average fixation time and a greater fixation count than mouse operation.Previous studies have indicated that eye movements for a long fixation duration were associated with increased information processing, and increased fixation counts suggested a more active scanning mode.30,36,37The results of this study suggest that a touch screen can greatly facilitate information processing.One explanation may be that with inherent motor skill ability, direct-touch input methods effectively reduce hand-eye coordination problems.4,38Overall, our findings suggest that touch screens can potentially improve information processing as they are intuitive and natural; however, interaction design defects and workload issues should be carefully considered.

4.2. Effects of display mode

Hypothesis 2 was not supported for target search reaction time, but was supported for saccade counts. Generally, we found little evidence of the influence of display mode on task performance and workload; however, the display mode had a significant effect on saccade counts. Grayscale mode produced greater saccade counts than color mode,suggesting that more visual search and location behaviors were required for task completion in grayscale mode, as saccade counts indicate relative information organization and the amount of visual searching.36In color mode, different colors represent map elements (land, ocean, cities), which is more conducive to distinguishing objects and reducing the visual search load. This result was supported by the interaction effect of display mode and information complexity on NASA-TLX temporal demand;color mode required less temporal demand with high information complexity than grayscale mode. The benefits of color mode were consistent with those reported in previous studies.15,17.

We also found that display mode interacted with input method on heart rate and average saccade peak velocity,which was less reported in previous studies. According to the experimental results,a touch screen in grayscale mode reduced the difference in heart rate between input methods and increased average saccade peak velocity.Studies that explored the meaning of saccade velocity indicated that low average saccade peak velocity was correlated with high mental workload39and fatigue.40Our results suggest that grayscale mode can alleviate the difference in workload induced by different input methods.One explanation may be that the brightness and contrast ratio in grayscale mode are softer,reducing visual fatigue.41This has an important design implication,suggesting that grayscale mode is more suitable for a multi-input device task environment.

4.3. Effects of information complexity

Our results indicate obvious adverse effects of high information complexity on task performance and perceived workload.The task completion time and target search reaction time increased significantly with an increase in information complexity.Intuitively,high information complexity requires more time to find the target,as predicted by Hick’s Law.42Previous studies have shown a strong relationship between information complexity and operator workload.23However, although perceived workload was greater with high information complexity according to the NASA-TLX scale,we did not find any difference in heart rate/heart rate variability,which is often used as a psychophysiological index of workload.43The results indicate a divergence between subjective and objective workload measures,as reported in a previous study.44The sensitivity of heart rate/heart rate variability to workload induced by information complexity requires further investigation.

Hypothesis 3 was partially supported for input method.The interaction effects of information complexity and input method were significant for participant eye movements. High information complexity tended to cause greater fixation counts and saccade counts with touch screen operation. This result suggests that mouse operation may be more suitable for a dynamic workload environment to help operators maintain relatively stable visual attention demand.

4.4. Effects of gender difference and user habits

In order to examine the gender difference,the sensitivity analysis of gender factor was performed; the results showed that gender had no significant effect on task completion time(Mmale= 204.6 s, Mfemale= 204.8 s, t15= 0.026, p = 0.98)and target search reaction time (Mmale= 29.5 s, Mfe-male= 32.6 s, t15= - 1.127, p = 0.277).

In addition, the user habits were also considered in our study, as the touch screen technology is newly introduced to EWA reconnaissance tasks,operators’previous use experience of mouse interaction may influence the results. We try to mitigate the daily habits effects from the following three aspects.Firstly, our participants were recruited from the college students. With the widespread use of touch screens in smartphones, participants are familiar with touch screen interaction as well as mouse interaction with computer. Secondly, we derived the simulated scenarios from the actual EWA reconnaissance tasks and properly abstracted the task characteristics. The tasks in our experiment were intuitive to operate and required little skills in use of touch screen or mouse interaction. Thirdly, all the participants were fully trained with different input methods before the formal experiment. They were encouraged to exercise with touch screen or mouse interaction until they could perform the task skillfully.Therefore, basic conclusions can be obtained for future work in an available and efficient way.

4.5. Limitations

There are some limitations in this study. This study mainly assessed the effect of different situation map display modes on search task performance with a 2D display. With developments in 3D display technology in aviation,the effects of a 3D display on reconnaissance tasks should be investigated in the future. This study considered only touch screen and mouse input methods; other input methods such as trackball, voice control, and multimodal input should be considered to determine the optimal input method for EWA mission systems.More complex and dynamic environmental and individual factors can also affect task performance;factors such as vibration may severely diminish touch screen usability35,45; operators’experience of mouse make it more convenient for use.The benefit and training cost of new input method can be examined further. Our conclusions are based on the laboratory experiment results; more detailed influencing factors should be carefully considered in actual task environments.

5. Conclusions

This study investigated the effects of input method (touch screen and mouse) and display mode of the situation map(color and grayscale) on EWA reconnaissance task performance with high and low information complexities. Based on the experimental results, the following conclusions can be drawn.

(1) Touch screen interaction required greater task completion time and produced a higher physical load than mouse operation; it should be used carefully in human-machine interaction with a long duration without other support. However, as an intuitive and natural input method, touch screen interaction could also facilitate the information processing by reducing the average fixation time.

(2) Significantly increased saccade counts and greater temporal demand were observed with high information complexity in grayscale mode,indicating a heavier visual search demand. Color mode is considered more appropriate for target search tasks as it incurs a lighter visual search load. Grayscale mode appeared to alleviate the workload differences of different input methods and is recommended for multi-input device task environments.

(3) High information complexity had significant adverse effects on task performance and workload. As greater fixation and saccade counts were observed with high information complexity in touch screen interactions,mouse operation is more suitable for dynamic task environments to help operators maintain a stable visual attention demand.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This study was co-supported by the National Natural Science Foundation of China and Civil Aviation Administration of China (No. U1733118) and the National Natural Science Foundation of China(No.71301005),as well as the Aeronautical Science Foundation of China (No. 20181330002).