Soft Robotic Glove Controlling Using Brainwave Detection for Continuous Rehabilitation at Home

Talit Jumphoo,Monthippa Uthansakul,Pumin Duangmanee,Naeem Khan and Peerapong Uthansakul,＊

1School of Telecommunication Engineering,Suranaree University of Technology,Nakhon Ratchasima,30000,Thailand

2Faculty of Electrical and Computer Engineering,University of Engineering and Technology Peshawar,Peshawar,25000,Pakistan

Abstract:The patients with brain diseases(e.g.,Stroke and Amyotrophic Lateral Sclerosis(ALS))are often affected by the injury of motor cortex,which causes a muscular weakness.For this reason,they require rehabilitation with continuous physiotherapy as these diseases can be eased within the initial stages of the symptoms.So far,the popular control system for robot-assisted rehabilitation devices is only of two types which consist of passive and active devices.However,if there is a control system that can directly detect the motor functions,it will induce neuroplasticity to facilitate early motor recovery.In this paper,the control system,which is a motor recovery system with the intent of rehabilitation,focuses on the hand organs and utilizes a brain-computer interface(BCI)technology.The final results depict that the brainwave detection for controlling pneumatic glove in real-time has an accuracy up to 82%.Moreover,the motor recovery system enables the feasibility of brainwave classification from the motor cortex with Artificial Neural Networks(ANN).The overall model performance reveals an accuracy up to 96.56% with sensitivity of 94.22% and specificity of 98.8%.Therefore,the proposed system increases the efficiency of the traditional device control system and tends to provide a better rehabilitation than the traditional physiotherapy alone.

Keywords:Rehabilitation;control system;Brain-Computer Interface(BCI);Artificial Neural Networks(ANN)

1 Introduction

The patients with brain diseases(e.g.,Stroke and Amyotrophic Lateral Sclerosis(ALS))are often affected by the injury of motor cortex,which causes a muscular weakness.This is the major cause of disability and affects the Activities of Daily Living(ADL)[1,2].In addition,the report of World Health Organization(WHO)has shown that the stroke is the second leading cause of death worldwide.This is associated with the deaths of approximately 6 million people around the world each year[3–5].Until now,the number of stroke patients has increased as the world population is entering an aging society[6].The most common effect of stroke survivors is muscle weakness or partial paralysis,which is confined to one side of the body and is known as hemiparesis.Likewise,ALS is one of the neurological diseases that affect nerve cells in the brain.When motor function is degraded,the patients may not be able to move some of his muscles.In no time,nerve cells begin to die,which eventually leads to a loss of motion control and then the patient becomes paralyzed[7].However,these diseases can be alleviated in the early stages of symptoms[8–10].

Therefore,post-stroke motor recovery and ALS during the first few months are very important to initiate continued motor regeneration.Repetitive rehabilitation can stimulate motor activity to help the patient in terms of learning to regulate movement again[11,12].However,patients can undergo physical rehabilitation in minimal time as there are no physical therapists and early rehabilitation equipment available.In addition,patients often refuse to go to rehab at hospital.As a result,the patient does not receive physical therapy continuously.Hence,robotic rehabilitation can help solve the problem,as it can record patients’ real-time data for clinical follow-up to improve remote rehabilitation,alongside customizing their own in-house therapies[13–16].Recently,researchers have come up with ideas for developing rehabilitation-assisted robots to improve the effectiveness of physical therapy[17–21].In general,robotic rehabilitation focuses on restoring the upper and lower limbs rather than hand/finger rehabilitation,but the hands are one of the most important parts of the body as they play a fundamental role in the work.Recovering hand function can help patients in terms of ADL.

In addition,the control system of restoring robots is an important type of research.Popular control systems for restoration robots include Continuous Passive Motion(CPM),Active-Assisted Movement(AAM)and Surface electromyography(sEMG)sensors[22–25].CPM is an automated rehabilitation system in which The patient was continually rehabilitated according to the design without any effort.CPM is used during the early stages of rehabilitation as the patient is unable to exert himself.AAM and sEMG are the rehabilitation systems in which the patient must first exert himself,and then use motion aids.Both methods are suitable for the second stage patients,allowing them to move some muscles.In addition,both passive and active control systems can be enhanced in regeneration using brain wave detection.Therefore,if there is a control system that can directly detect motor activity and increase the participation of the patient during the rehabilitation process,it can induce neural resilience to facilitate the early recovery of motor.

In this article,we present a control system named motor recovery system with the intent of rehabilitation utilizing Brain-Computer Interface(BCI)technology.The BCI is a technology that uses brainwaves to analyze and translates them into commands,and then relay to output devices for desired actions[26,27].In BCI at non-invasive stage,the brainwaves that acquire the movement intention of users(e.g.,motor imagery or execution)are decoded in real-time with the help of feature extraction.Then,the neurofeedback system is utilized to increase the user performance by linking the intent to execute a movement[28–30].The neuro-feedback is the measurement of brain activities to produce data for teaching selfregulation of brain function and present it with a positive or negative response to closed-loop control[31].The active motor is intentionally trained and reproduced to stimulate the first cortical motor cortex,providing a better rehabilitation results compared to automatic(passive)movements or limb-only(active)stimulation[32–34].In addition,physical therapy in conjunction with mindfulness is more interesting[35–37].The experience of brain disease negatively impacts patients' mental health.Recently,researchers have revealed that meditation,along with physical therapy,is more effective than conventional physical therapy alone[38].Therefore,the motor recovery system has two steps:The first step is to detect brain wave while the user deliberately meditates to perform robotic-assisted regenerative control.In the second step,the patient can generate some of brain waves from the motions to form a norm.Hence,the motor recovery system detects brain waves in the motor cortex area while the user is controlling the robotassisted rehabilitation.

2 Methodology

In this paper,motor recovery systems focus on affordable,agile,and easy-to-wear solutions for rehabilitation.A commercial low-cost EEG headset was named NeuroskyMindwave and OpenBCI used in the experiment.Both headsets include dry-type electrodes that must be placed on specific scalp locations according to the 10–20 international systems as shown in Fig.1.The EEG headset can measure different brainwave voltage fluctuations due to ionic current in brain neurons.The EEG sensor converts these voltage fluctuations into digital outputs to be sent to the device’s microcontroller using wireless technology.Most observed brain waves range from 1–40 Hz.These waves are categorized into delta,theta,alpha,beta,and gamma.Delta waves(<4 Hz)are most likely to be observed during sleep.Theta wave(5–7 Hz)can be seen during meditation.Alpha waves(8–13 Hz)are the result of the brain's relaxed state.The Mu wave(8–13 Hz)is found above the motor cortex,which tends to be suppressed as each part performs voluntary muscle movement or creates the intention to perform the movement.Beta waves(14–30 Hz)are involved in the brain and active concentration.Low frequency(<22 Hz)beta waves indicate an active and focused brain,while high frequencies(>22Hz)indicate anxiety.The importance of gamma waves(>32)cannot be clearly identified[39].The robot-assisted rehabilitation was developed for the pneumatic system,which uses soft material safely to distribute force along finger length according to the Harvard Glove[40].Besides,the pneumatic glove provides haptic feedback to volunteers as shown in Fig.2.

Figure 1:The international 10–20 systems for electrodes placement

2.1 Data Acquisition

The Neurosky Mindwave device’s dry electrode is approximately positioned at Fp1 and has a sampling rate of 512 samples per second.In the first stage,the motor recovery system is focused on the purpose of controlling the pneumatic gloves.Usually,the intention to do something can generate brain waves involved in meditation.Therefore,the development of the system of early motor recovery uses real-time detection of meditation.The volunteers participating in the trial were 21± 2 years old and in good health.EEG data collection was performed in a closed room,while the volunteers sat comfortably in their chair.Volunteers put their hands on the table and watched the computer screen showing the countdown video.When the computer screen shows “Start”,the volunteer rests their hands in their normal state for about 10 seconds.The volunteer then imagines closing his/her hand for approximately 10 seconds.The measured EEG data set is analyzed offline.

Figure 2:The pneumatic hand glove according to Harvard Glove 2.0

In second stage,the motor recovery system focuses on actuation for pneumatic right-hand glove control.The EEG is recorded by OpenBCI devices with five electrodes comprising C3,Cz,CP1,P3 and Pz positions.These electrode locations are related to motor activity in functional areas of the brain[41].The sampling rate of the OpenBCI device is 250 samples per second.Healthy volunteers,right-handed and aged 21 ± 2 years participated in the experimental procedure.The EEG data retrieval was performed in a closed room,while the volunteers sat in a chair and rested their arms comfortably on the table.The volunteer is asked to meditate and watch the computer screen showing the countdown video.When the computer screen shows “Start”,the volunteer turns their hands on for about 3 seconds and then turns their hands off for about 3 seconds as shown in Fig.3.In this experiment,all EEG data sets were collected from motion and contained approximately 100 data sets.The measured EEG data sets were later used to train Artificial Neural Networks(ANN)learning.

Figure 3:The EEG data acquisition on 5 channels from volunteer

2.2 First Stage EEG Analysis

The EEG data set,measured in the first stage,is used to develop an algorithm for detecting motionintent,which has the following processes.Detecting intentional meditation requires knowing the Baseline Parameter of Meditation(BPM)during normal conditions and can be calculated as follows:

where the meditation is a value of meditation from EEG headset.N is the number of meditation parameters obtained in normal conditions(N = 10).Finding the Peak Value of Meditation(PVM)parameter can be calculated as follows:

where M is the number of meditation parameters obtained during the imagination of slowly closing hands(M = 10).The Threshold Value of Meditation(TVM)is used as a decision-making criterion in meditation detection,which can be calculated as follows:

If Meditation >TVM,Decision is“The pneumatic glove starts to fist”

If Meditation

The development for brainwave detection algorithm of a similar BCI system is named as the assistive communication device in persons with severe disability[42].

2.3 Second Stage EEG Analysis

In the second stage,the signal processing of EEG dataset is digitally filtered using a Butterworth filter between the corresponding Mu and Beta bands,8 to 30 Hz,as this band contains the highest data related to hand movement.Butterworth filters have a flat response to zero pulsation,which is ideal for brain wave analysis.So we filtered the frequency below 8 Hz using the high pass filter and the low pass filter to subtract the frequencies above 30 Hz using the 4th-order Butterworth Digital filters as shown in Eqs.(4)and(5)respectively.

In this study,ANN was used to classify EEG signals to identify hand movement patterns(Open hand and close hand)from the given dataset.The learning process in ANN can be done using a training algorithm named Levenberg-Marquardt.The training phase is to provide the network with sample data and adjust the weights to estimate the activation function in a better way.

In the architecture of ANN,each neuron(j)in the hidden layer sums its input feature vectors(^Xi)after multiplying by the respective connection weight strengths(wji),and computes its output(yj)as a function of the sum:

wherefis the activation function that is significant to transform the weighted sum of all signals impinging onto a neuron.The activation functionfcan be a simple threshold function,a sigmoidal,hyperbolic tangent,or radial basis function.In this paper,for the hidden layer and the output layer,the activation functionfis the tangent-sigmoid(tansig)function.The sum of squared differences between the desired and actual values of the output neuronsEis defined as:

whereydjis the desired value of output neuronjandyjis the actual output of that neuron.Each weight(wji)is adjusted to reduceEas rapid as possible.The adjustment of weightwjiis dependent on the training algorithm.The Mean Square Error(MSE)algorithm is the defined criterion for the network performance.When the MSE value between the network output and the target is less than or equal to the set value,the learning process stops.

To evaluate the performance of the ANN model,the response can be tested through a confusion matrix,Validation performance and Receiver Operating Characteristic(ROC)curves as shown in Fig.4.A confusion matrix was computed to define the accuracy of the classification outcome as shown in Fig.4a.By considering the results of the trained networks compared to the expected results(targets),output network,which are correctly classified according to appropriate reactions,are shown as green squares and red squares.It indicates an incorrect classification by off-base reactions.The gray squares in the lower right show the general accuracy,which can be calculated from true positives(TP),false positives(FP),false negatives(FN)and true negatives(TN)as shown in Eqs.(8)–(10).

The validation performance is used to find the lowest MSE value for every iteration in the training process as shown in Fig.4b.At the point of lowest MSE value,the training should be stopped,and no further iteration should be proceeded.It means that no further training is required,and we could get the wrong results if we perform the training.The ROC curve is another metrics for determining the assessment accuracy via Area Under the Curve(AUC)scores(AUC scores range between 0 and 1.0)as shown in Fig.4c.The AUC scores of discriminations indicate the capability of the ANN to properly categorize samples.This threshold metric of segregation between both classes determines scores of AUC below 0.5(no classification)and 1(perfect classification).Hence,the upper left corner of the ROC figure depicts the perfect curve,which indicates the high accuracy of classification.The following scales are employed to determine the classifier accuracy:excellent = 0.9–1,good = 0.8–0.9,fair = 0.7–0.8,poor=0.6–0.7,and fail 0.5–0.6.

2.4 Motor Recovery System

The aim of this research is to design an embedded system that can be used to classify and control pneumatic glove using the acquired EEG signals.The “STM32” is the embedded system used to fulfill the aim of this research.The EEG data is processed through a program that we develop with the help of C++/C#programming.The result is given as an input to STM32,which is programmed to process the input data for controlling the pneumatic glove.The data processing procedure considers the conditions of Meditation>TVM or Meditation

Figure 4:The performance of ANN response model:(a)Confusion matrix,(b)Validation performance,(c)Receiver Operating Characteristic(ROC)curves

Figure 5:Block diagram of the motor recovery system

Figure 6:The control hardware composition

2.5 Software Development

The software of the developed motor recovery system consists of several sections which are shown in the red box in Fig.7.The red box with number 1 on the top shows the real-time EEG signal as the user imagines it is working to close the right hand.Besides,the level of meditation is also shown in this section.The red box with number 2 on the top shows the command settings including the TVM level device connection,the air pump timing and the constant air pump waiting time.The red box with number 3 on the top shows the working status of the motor recovery system.A red circle means that the system is working.A green circle means that the system is waiting for the command.Additionally,the user can choose to inflate the air to the desired finger by clicking on the small square on the hand image.

3 Results

The pneumatic glove test with control systems developed in our study was performed on 10 normal people(mean age 21±2 years),divided into 5 people to find the suitable TVMs and test the appropriate TVM pneumatic glove control.This test is based on the satisfaction score of the Likert scale technique as shown in Tab.1.The score consists of 5 levels as follows:the range from 4.50 to 5.00,the range from 3.5 to 4.49,the range from 2.5 to 3.49,the range from 1.50 to 2.49 and the range from 1.00 to 1.49.They are referred as Very Good,Good,OK,Poor and Very Poor,respectively.The TVM calculation results show values between 60 and 80.Hence,each volunteer performs 5 trials to find an average satisfaction score and perform 4 trials by changing the TVM from the lowest to the highest value as shown in Tabs.2–6.

Figure 7:The software display of the motor recovery system

Table 1:The satisfaction score of the Likert scale technique

Table 2:The satisfaction score from volunteer 1 experimenting with the pneumatic glove control by changing the threshold value of meditation(TVM)

The best satisfaction score of TVM is used as a condition for device operation.In experiments,a volunteer controls a pneumatic glove to successfully grab and release the ball.There are 10 trials in total,with a time limit of 10 seconds for each trial.The results are shown in Tab.7.

Table 3:The satisfaction score from volunteer 2 experimenting with the pneumatic glove control by changing the threshold value of meditation(TVM)

Table 4:The satisfaction score from volunteer 3 experimenting with the pneumatic glove control by changing the threshold value of meditation(TVM)

Table 5:The satisfaction score from volunteer 4 that trials the pneumatic glove control by changing the threshold value of meditation(TVM)

Table 6:The satisfaction score from volunteer 5 that trials the pneumatic glove control by changing the threshold value of meditation(TVM)

Table 7:The pneumatic glove control test with suitable Threshold Value of Meditation(TVM)

Additionally,our study considers the pneumatic glove control through the brain motor cortex.The EEG data of acquisition from 5 channels(C3,Cz,CP1,P3 and Pz channel)is used to find the classification performance between open and closed hand.We divided the datasets into 70% as training set,15% as validation set and 15% as testing set.Tab.8 shows the ANN results of 5 channels with overall indicators including Accuracy,Sensitivity,and Specificity.

Table 8:The ANN results of 5 channels with the overall indicators

At around 413 epochs of the C3 channel,the MSE validation is stable and the MSE is very low at 0.058982.It means that the training should be stopped because of receiving the best training for pattern recognition.Similarly,for other 4 channels(Cz,CP1,P3 and Pz channels)at different epochs,the best MSE validation performance is 0.05853,0.044409,0.049121 and 0.030337,respectively.Fig.8 shows the ROC results of 5 channels when the AUC score is close to 1.This means that the network has given higher accuracy in hand movement classification.

Figure 8:The ROC results of the 5 channels:(a)C3 channel;(b)Cz channel;(c)CP1 channel;(d)P3 channel;(e)Pz channel

4 Discussion

The results showed that the mean satisfaction score of TVMs at 60 and 65 were 3.36,3.48,respectively,meaning that pneumatic glove control was at an OK level.Although both TVMs can be controlled at an OK level,they can be easily performed using some basic concentration in meditation,which creates a sense of control instability.The average satisfaction score of TVM at 70 is 4.72,which means that the control is a Very Good level.Due to Very Good level,a higher concentration of meditation is required to control the pneumatic gloves as they provide stability in control.Considering the average satisfaction score of TVM at 80,the result shows a score of just 2.28.This means that controlling the pneumatic glove is at a Poor level.Since the value of TVM is high,it requires a very high concentration of intentions in meditation.Although the volunteers can successfully control the pneumatic glove,the intent of practicing a consistent high concentration is very difficult.Therefore,the satisfaction score of 5 volunteers can be concluded that the effectiveness of TVM is 70.This conclusion is used in pneumatic glove control experiments where conditions for catching and placing the ball were successful.The results showed that 82% of the volunteers were able to successfully control the pneumatic gloves.In general,the experience of brain disease negatively affects the mental health of the patient.Therefore,this success shows that meditation,along with physiotherapy,can be performed at the same time and is more effective than conventional physiotherapy alone,which is consistent with the work presented in[38].

Moreover,we have found the feasibility of the brainwave classification in the motor cortex area.Based on the results,the five-channel ANN model had an average accuracy of 96.56%,with an average sensitivity of 94.22% and a mean specificity of 98.8% in terms of Open or Close hand.The result is a possible hand movement in patients with second-stage brain disease.The ANN model then detects the corresponding brain waves as the user moves by hand.The classification results are then translated into commands and sent to the pneumatic glove control system to assist the patient in rehab.Physiotherapy with active and repetitive motor intent can stimulate the activity of the primary cortex.This approach is likely to provide better recovery results than conventional physiotherapy alone[32–34].Although the method of the second stage works.But it is not suitable for real life use as the device is time consuming which can inconvenience the user.

5 Conclusions

In this research,we have discovered the feasibility of a motor recovery system through the aim of rehabilitation using BCI technology.In the first stage of brain disease,the patients cannot generate brain waves related to voluntary movements.Thus,the motor recovery system uses real-time brainwave detection while the user is intending to execute the movement of pneumatic glove.The results show that the volunteers were able to successfully control the pneumatic glove with 82% accuracy.Furthermore,we have found the feasibility of the motor recovery system in the second stage through the brainwave classification in the motor cortex area.Due to the second stage of brain disease,the patients can generate some brain waves related to voluntary movements.Based on the results,the five-channel ANN model had an average accuracy of 96.56% with an average sensitivity of 94.22% and a mean specificity of 98.8% in terms of Open or Close hand.Although the method of the second stage works.But it is not suitable for use in real life as the installation of the device takes time which can inconvenience the user.In the future,we will develop the motor recovery system in the second stage by reducing the number of channels for faster installations to facilitate the users.

Acknowledgement:All subjects gave their informed consent for inclusion before they participated in the study.The study was conducted in accordance with the Declaration of Helsinki,and the protocol was approved by the Ethics Committee of Suranaree University of Technology(License EC-61-14 COA No.16/2561).

Funding Statement:This work is financially supported from the Thailand Research Fund through the Royal Golden Jubilee Ph.D.Program(Grant No.PHD/0148/2557).

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.