BRAIN DRAIN: USING BRAINWAVES AND MACHINE LEARNING TO DETECT ERRORS IN HUMAN PROBLEM SOLVING KATIE PORTERFIELD. Advisor JOSHUA ECKROTH

Transcription

1 BRAIN DRAIN: USING BRAINWAVES AND MACHINE LEARNING TO DETECT ERRORS IN HUMAN PROBLEM SOLVING by KATIE PORTERFIELD Advisor JOSHUA ECKROTH A senior research proposal submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in the Department of Mathematics and Computer Science in the College of Arts and Science at Stetson University DeLand, Florida Spring Term 2016

2 TABLE OF CONTENTS ABSTRACT INTRODUCTION 3 BACKGROUND. 4 PREVIOUS WORK PROOF OF CONCEPT....7 METHODOLOGY.. 9 RESULTS DISCUSSION CONCLUSION AND NEXT STEPS REFERENCES

3 ABSTRACT The Muse Headband is a simple to use EEG machine that provides real time measurements of brain waves. Using this headband, we built a model using big data and machine learning techniques to interpret brain wave patterns to create a real time feedback system that helps the user understand their cognitive thinking while solving a problem. Once further developed, this model could then be applied in an educational setting to help a student understand how they are progressing through a problem without the interactions of a teacher to support them.

4 1. INTRODUCTION Every student needs different levels of attention when learning and solving a problem. Little interaction between the teacher and student can cause a decrease in the information retention. Affective learning is an important aspect of education. Emotions of students need to be recognized and interpreted so as to motivate students and deepen learning. [1] However, when a student is by himself or herself, the student may experience difficulties when trying to work or take tests. If the student could understand how he or she is thinking, the student could better understand the presented problem. There currently does not exist an application to help a user work through their problems based on the student s brain waves. By monitoring brain waves, an application could provide insight of how the student is thinking and if he or she is on the right track to solving the problem. 2. BACKGROUND There are five types of brain waves: alpha, beta, theta, delta, and gamma. Each wave represents a different kind thinking, whether cognitive or not. Delta waves are a regular, low amplitude wave of 1 5 Hz, and associated with low neuronal firing rate and deep sleep. Theta waves are a medium amplitude, and medium frequency of 5 8Hz that corresponds with drowsiness. Beta waves have an irregular pattern between Hz during alertness and active thinking. Alpha waves are a regular pattern with a frequency of 8 12 Hz. Alpha waves tend to have a high amplitude when awake, but not stressed. [2] Alpha waves can be divided into two main categories, upper and lower. Lower alpha waves tend to reflect attentional demands including alertness and vigilance. Semantic memory

5 processes and higher specific task demands are associated with higher alpha frequencies. [2] What is know as alpha synchronization occurs when the alpha activity increases. When the alpha waves decrease, the waves are desynchronized. Alpha synchronization indicates creative cognition because of the high internal processing demands that creativity requires in comparison to semantic memory. [3] Alpha waves have event related rhythms that help demonstrate different brain processes called oscillations. The oscillations are related to memory processes and movements through different types of events. By understanding the oscillations, alpha waves provide insight to detecting problem solving processes. The Muse Headband was released Winter of 2012 and has currently been used to focus monitor the user s daily brain activity. The Muse is marketed as a meditation application but has also been used to visualize a user s workout progress and overall daily life. Muse is an electroencephalogram (EEG) machine built with three reference sensors in the front, two forehead sensors, two ear sensors, and accelerometer. The headband can sample at a rate of 220 Hz or 500 Hz. Muse provides data regarding alpha, beta, theta, and delta waves amplitudes, along with eye blinks, and raw Fast Fourier Transform (FFT) of the various waves. The Muse s FFT data represent which frequencies are present and how much is present for every channel. When recording the data from the Muse Headset, the data reflected is a limited snapshot data of continuous data. Using a Hamming window allows for a better frequency measurement by reducing the amount of noise taken from the limited snapshot. A Hamming window works by a window views part of the information, then slides the window down to gather the next information, but contains overlap of the previous window. To calculate

6 the FFT, the Muse developers use a Hamming window of 256 samples, then slide over by 22 samples, or 1/10th of a second, to give an overlap window of 90%. Various brain waves and FFT data can allow for a better predictive model as discussed in the next section. 3. PREVIOUS WORKS Previous research suggests Wavelet Transform allows for better analysis of alpha oscillations by including time to keep the shape of the provided data. By having time frequency data, an EEG machine will be more accurately illustrate the event related frequencies. [5] Quiroga and Schurmann used this method to determine functions and sources of the alpha oscillations. [6] Fourier Transform is often used to illustrate two frequencies relationship; however, brain waves are non stationary signals causing the Fourier Transform to lose the time evolution of the frequency patterns. The Wavelet Transform can provide the same results as the Fourier Transform but keep the frequency patterns as the time continues. The Wavelet Transform has also been previously used for pattern recognition. In their results, Quiroga and Schurmann argue the usefulness of the Wavelet Transform due to it s ability to use a varying window to achieve a better time frequency resolution. Wavelet Transform was also used to help find patterns in EEG data to predict seizures in patients. They placed the raw EEG time series and the Wavelet Transform into a Recurrent Neural Network to train each EEG channel separately. [7] They also used bivariate features, quantitative relationship between two signals, that were computed on 5 second windows of every pair of EEG channels. Each window was associated then with a binary feature of seizure or no

7 seizure. After testing out different classifiers, the studied showed the best one was a Convolutional Neural Network with the Wavelet Transform. 4. PROOF OF CONCEPT Our first attempt was focused on determining if there was a difference in correct versus incorrect thinking by developing a prime number Java Application. The application provides basic instructions to click yes or no depending on if they think the number presented is prime. The number shown is a randomly generated number. Time starts when the user clicks next after the instructions are presented. As each number is shown, the timestamp based on the system time, is recorded. Once the question has been answered, the application saves the number presented, whether the user got the answer wrong, and the time answered. This application does not have a time limit associated with each answer to prevent the user from being stressed and causing false results. To associate the data with Muse Headband, the recording is started first in the MuseLab by using the timestamp based also on the system time to be able to line up the data based on the two timestamps.

8 The graph illustrates the relationship between brain waves and correctness. The graph is split in half horizontally to demonstrate correct, 1, and incorrect, 0. Vertically, each brain wave has its own section to show how they are effected individually. The graph is aligned to the right to have the end time all the same due to the variability of each question time response. The graph illustrates the significant noise difference between correct and incorrect thinking. This difference lead us to believe there is something to learn about the brain wave patterns.

9 5. METHODOLOGY The model was created by two different parts, the user interface and the neural network. User Interface The user interface was a Java Application developed to display to the test subject five three digit multiplications and five three digit division equations. The user was selects yes or no to state if he or she believes the presented equation is true. He or she had unlimited time to answer each question and pen and paper. The average time for the test subject to answer all ten problems was around twenty minutes. The time each questions started, each question ended, whether the question was multiplication or division, the numbers presented and if the user got the answer right were all saved in the data set. The Java Application was sync to Muse Lab, as it recorded the brain waves, based on the provided time stamps. Fast Fourier Transform was already calculated by the Muse Headset and save in the Muse data file. Neural Network After gathering the data from the user interface, the files were merged using R and python then sent into a recurrent neural network. The neural network was build in python using Theano, a Python Library used to evaluate mathematical expressions that involve 1 multi dimensional arrays. Graham Taylor s Vanilla RNN was used to create the model because of the simple recurrent neural network script developed to output binary answers. 1 Taylor, Graham. "Vanilla RNN." rnn/blob/master/rnn.py.

10 The neural network was built with 516 input nodes. Each node represented an FFT value gathered from the Muse Headband. The hidden node count was changed to determine the best outcome, which is discussed further in results. Prior work using brain waves and machine learning suggested using a recurrent neural network would be the most effective way. However, we ran the data through other classifiers as well. The data was sent into each classifier the same with each row containing all the fft values gathered at a specific time creating 516 data points per row. Each classifier was trained on the same ⅔ of data. The other clasifiers used were Decision Tree, SMO, nearest neighbor, and random forest. Decision trees create a tree like model to create a bunch of branches that represent the possible choices and their outcomes, till it reaches an end node with the final outcome. K nearest neighbor classifiers the k to the closest training set. Sequential minimal optimization optimizes support vector machines, which takes the data and places it into a higher dimension to determine the best vector to fit between the data. Random forest takes the data and divides it up into smaller subtrees. Each subtree then add weights to each branch, and the main model add weights to each subtree to determine their best options.

11 After the model was developed, a basic feedback system was created. While the user is wearing the headband, the muse headband reads the last twenty five received FFT values through a python script which loads the model and runs predict from Taylor s script. The script then receives an array output which is averaged. If the number calculated is one, the user is thinking correctly, if the number is zero the user is thinking incorrectly. 6. RESULTS After the first model was developed, tests were ran to determine the best way to create the neural network. The best hidden nodes amount was ten, with there being a 65% chance that the prediction is correct. The data was also thrown into weka and classified on other algorithms, seen in the following graph.

12 The recurrent neural network seemed to work the best coming out on top with the 65% in comparison to random forest at 64% and SMO at 63%. One nearest neighbor did the worst out of the tested classifiers. 7. DISCUSSION This experiment demonstrated their is a possible to predict and understand a person's brain wave pattern through classifiers. The data was trained across twelve people s answers and brain waves. Overall, the recurrent neural network proved to be better than chance; however, it is further research needs to be done to determine if the 65% accuracy is significant enough to be placed into real world scenarios along with developing the model more. 8. CONCLUSION AND NEXT STEPS This research focused on predicting brain wave data to determine if a user was thinking correctly about a problem. Using classifiers, with specific interest in recurrent neural networks, a model was developed which predicts accurately 65% of how a user is thinking. The next steps will include improving the accuracy of the predictive model, by testing other recurrent neural networks. Another way is to increase the size and diversity of the training set such as asking test subjects of different ages. Crossfold validation will also be added to create higher accuracy and better training. Lastly, we plan to transform the raw EEG data using wavelet transforms over fast fourier transform. Once a better predictive model is developed, a comparison between asking literary and mathematical based questions should be researched further.

13 9. REFERENCES [1] Li, Xiaowei, Bin Hu, Tingshao Zhu, Jingzhi Yan, and Fang Zheng. "Towards Affective Learning with an EEG Feedback Approach." [2] Dietrich, Arne, and Riam Kanso. "A Review of EEG, ERP, and Neuroimaging Studies of Creativity and Insight." [3] Bendek, Mathias, Sabine Bergner, Tanja Konen, Andreas Fink, and Aljoscha Neubauer. "EEG Alpha Synchronization Is Related to Top down Processing in Convergent and Divergent Thinking." Neuropyschologia, [4] Klimesch, Wolfgang. "Klimesch W. EEG Alpha and Theta Oscillations Reflect Cognitive and Memory Performance: A Review and Analysis." Brain Research Reviews, [5] Polikar, Robi. "The Wavelet Tutorial." The Engineer s Ultimate Guide to Wavelet Transform [6] Q uiroga, R. Quian, and M. Schürmann. "Functions and Sources of Event related EEG Alpha Oscillations Studied with the Wavelet Transform." Clinical Neurophysiology, [7] Mirowski, Piotr, Deepak Madhavan, Yann LeCun, and Ruben Kuzniecky. "Classification of Patterns of EEG Synchronization for Seizure Prediction."