The Algorithmic Composition Methodology for Health Using Wearable Device

,

(1. Music College, Shanghai Normal University, Shanghai 200234, China; 2. Department of Information Communications, Kainan University, Taiwan 33857, China)

Abstract： This research would generate the situated music with Algorithmic Composition(AC) system for the exercise activities including running, bicycling riding, going to a gym, etc., with proper incident music. Therefore, it applies the wearing system to integrate the bio-feedback data, such as the brain wave meter and music data, to send the data to the web server for the automated music generative system. The music features including chord, melody, and drum set data will be synthesized and adjusted automatically, to fit the need of the exercise users. Algorithmic composition system is also used to adjust the melody parameters, to integrate the bio-feedback data with the music features. Finally, the brain wave meter is applied as the listeners’ evaluation, with the integration to complete the automated music generative system, for the newly developed interdisciplinary industrial of the health and entertainment with music technology research.

Keywords： algorithmic composition; physiological and psychological conditions; bio-feedback data; brain wave meter; automated music generative system

The wearable devices such as headphones, head-mounted-display, etc. are popularly used for music appreciation as a kind of mobile tools currently[1-3]. More and more wearable device applications combine both music appreciation and health[4-6]. Usually the stream music[7-8]can be used while we are in the movement condition. However it is not easy to select proper music for the wearable devices. Therefore, the algorithmic composition[9-11]can be applied for the wearable devices as the proper situations.

The research can be divided into two parts： Music wear healing and music exercise fitness system. The purpose is to obtain the information instantly obtained through the wearable device, combined with the algorithmic composition system, and feedback the user with music to achieve physiological information sensing, music creation and psychology. For the user, the wearable device can detect bio-feedback and other body information. While maintaining physiological health, it can also generate random music through a set of computer-aided systems with music theory. And the sound, rhythm, etc. of the music can change instantly, so the rich and varied music creation form will have an invisible effect on the user’s recreation effect and even on mental health.

The implementation of the self-action music is performed by using an algorithm. For example： Completely random is the simplest automatic composition model, but there is no regularity in the music produced, and there is no composing criterion. Therefore, we can analyze and use the analyzed results to automatically generate music through the computer. The music features extracted from the music, through the database and rules established by these characteristics, can produce music similar to the original music style. If the analysis of the original music is more detailed, the resulting sample database and the composing rules will be more clear, which will certainly be more conducive to the realization of computer composition software, but there is also a controversy： The result of automated composition must be almost the same as the original style or should be innovative. This is related to whether only the sample database generated by the analysis results is used to arrange the combinations or to create samples different from the samples. The following is the classification of research related to automatic composition so far：

(1) Music parameter setting produces style music： In the 1950s, two professors, Hiller and Isaacson of the University of Illinois, developed a computer program to input stylized music parameters through automatic generation, modification, and selection as well as produce the score of the Illiac Suite[12].

(2) Formulized music： The 1960s composer and mathematician Xenakis used the Stochastic Process’s mathematical algorithm and Sieve Theory to complete the algorithmic music of the formalized music[13].

(3) Music parameter setting combined with formulaic music： Professor Phil Winsor of the University of North Texas has developed automatic composing of “The Composer’s Toolbox” and “Music Sculptor” using screening theory and some mathematical algorithms. Software, in addition to using a formulaic mathematical algorithm, adds core music parameters to produce music of different styles. In 2008, Professor Chih-Fang Huang published the computerized automatic composition study of Taiwanese Hakka mountain song style[14], which improved Professor Phil Winsor’s arguments to study the style characteristics of local folk music and then produced automatic rules that met the parameters of local folk music. Composing program describe the flow of this type of automatic criteria composition, including the analysis of style music, the law of random numbers, the generation of various parameters, the melody features of adding style music, etc. Finally, the intro and endings are added to enhance the style of style music.

(4) Automated analysis and artificial intelligence： After the 1980s, David Cope, a professor at the University of California, Santa Cruz, automated analysis methods to achieve style simulation and creation, and developed the use of artificial intelligence to enhance music algorithms[15-16].

(5) Automated composition using genetic algorithm： In 2002, Alfonso et al. of Universidad Autónoma de Madrid applied genetic algorithm to automatic composition[17]. The concept of gene algorithm： First generate the initial genotype matrix, and define the fitness function with several known single melodies. Through the adaptation function, the excellent genotype can be selected and the above steps can be repeated.

Fig.1 Markov chain analysis for algorithmic composition

(6) Let the computer generate a melody through machine learning. The common way is to use Recurrent Neural Networks(RNN), such as Google’s Magenta[18].

1 Method

The proposed method for automated composition for music is Markov chain, using the possibility control the music melody features, including pitch and rhythm. In addition, the chord can be calculated from the possibility control to simulate the original music piece or segment[19-20]. Please refer to Fig.1.

Tab.1 Music features for algorithmic composition with emotion

For more music features including mode, tempo, harmony, loudness, articulation, pitch, rhythmic complexity, pitch range, pitch variation, pitch contour, note onset, loudness variation, timbre, vibrato, meter, tonality, which can be quantified for algorithmic composition too, with music features mapping, as shown in Tab.1.

The proposed method uses 2-D Emotion coordinate to determine the music arousal and variance[21-22], to generate the music for wearable device using the algorithmic composition techni-ques. As shown in Fig.2, the proposed algorithmic composition system can verify the music features based on 2-D emotion coordi-nate, to generate music automatically. Fig.3(See page 518) shows the music features setting and change for algorithmic composition based on bio-feedback and listeners feedback.

Fig.2 Music features change for algorithmic composition

Fig.3 Music features setting and change for algorithmic compo- sition based on bio-feedback and listeners feedback

In the recent years, some researchers use brain wave meter, such as Neurosky Mindwave, to verify the relation between music features and subjects’ intention and meditation feedback[23-25]. Therefore, the proposed research applies the brain wave meter as the wearable device for algorithmic composition system and listeners.

2 Experiments

2.1 Instant physiological information monitoring

The inertial sensor is used to sense the movements of the brainwave meter wearer’s body, mainly using finger drums and smart shoes to understand and act on the various activities of the wearer. The main application functions of the wearable healing system can be used by Neurosky Mindwave, for the proposed method experiment.

2.2 Activity monitoring

This application uses the brainwave meter worn on the body to analyze the various actions and behaviors of the wearer, such as the daily work record function, which can monitor and record the number of walking days and exercise hours of the wearer to achieve the life record. The effect and suggestions for improvement can be obtained based on the activity monitoring.

2.3 Posture monitoring

Fig.4 The wearable brainwave meter with algorithmic composition system

This application requires more precise “action” sensing technology than the aforementioned activity monitoring. This application needs to be used simultaneously by finger drums and smart shoes, which are “dressed” on the limbs to sense the movement of the wearer. For example, when the patient is recovering, the system can be used to correct the posture.

Eventually, the experiment uses Neurosky Mindwave to verify attention and meditation as the listeners’ bio-feedback data. The proposed research uses the algorithmic composition system to classify the four emotions, such as happy and sorrows, according to positive and negative emotions and energy levels, so that they are located in the four quadrants of the two-dimensional plane. The Mindwave electroen-cephalograph to measure brainwave signals, import them into program as input signals, and then pass the received information to pitch, interval, rhythm, and music features such as articulation, etc., and are determined by the number of notes events, tempo, MIDI channel, timbre, and rest ratio. Let the algorithmic composition system generate the melody corresponding to the brainwave signal. We also automatically generate MIDI files, and can directly read the infor-mation and play the music corresponding to the emotions directly by the program’s MIDI sequencer function, as shown in Fig.4.

3 Results

The movement fitness energy supply system can be roughly divided into an aerobic exercise and anaerobic exercise system. An anaerobic exercise system is for short-term explosive movements, such as sprinting, and an aerobic system is for long-term endurance sports, such as jogging. In the anaerobic movement, focusing on regular movement, if this regular movement can be synchronized with the rhythm speed, then the effect of the movement will be improved, and the aerobic movement will focus on the speed of the heartbeat. When the heart rate reaches 60%—75% of the highest heart rate using the proposed algorithmic composition system for the exercise activities, there will be a good aerobic exercise effect, so if the heartbeat is synchronized with the rhythm beat, it will continue to affect the continuous strength of the athlete and help to stabilize the time between each action and action. In terms of brain wave data, it dropped from an average of 61.11 to 52.50 before the experiment, and the exercise consciousness intensity decreased from an average of 12.72 to 11.82. This means that overall, the subject’s relaxation increased by 14% and the exercise consciousness slowed by 7%. The correlation coefficients of the two groups of data were 0.57 before the experiment and 0.59 after the experiment, all within the statistical correlation range.

4 Conclusion

The result shows a good preliminary relation among music features, emotion, exercise condition, etc. It can be seen from the correlation coefficient that there is a certain correlation between brain wave data and exercise conscious intensity data. For the adaptation of exercise intensity, positive feedback has a more significant effect than negative feedback. Music has different ways of assisting the exercise intensity of different age groups. In the future, in addition to the brain wave meter, some other bio-feedback wearable device such as Heart Rate Variance(HRV) or Galvanic Skin Response(GSR), even with sensor fusion, can be also used for the proposed algorithmic composition system for health.

Acknowledgements： The author would like to thank the Ministry of Science and Technology, Taiwan, China, for financially supporting this research under MOST 107-2410-H-424-012, and 106-2410-H-424-016.