BRAINWAVE PROJECTS A BRAIN COMPUTER INTERFACE FOR SMART HOME CONTROL

Transcription

1 BRAINWAVE PROJECTS A BRAIN COMPUTER INTERFACE FOR SMART HOME CONTROL

2 Abstract The aim of this study is to control home devices using a non invasive brain computer interface (BCI). The Electroencephalographic signals (EEG) recorded from the brain activity using the Mind wave headset are interfaced with the help of Arduino and HC-05 Bluetooth module. The user will control various devices in a smart home by using their attention and eye blink values. This application will be very useful especially for disabled people with special needs.

3 Introduction BCI is a system that captures the brain electrical activity in the form of EEG signal and translates those specific features of the signal that represents the intent of the user into computer readable commands. These commands can control and operate an electronic device. This technology is developing very rapidly, as it has innumerable uses, the most important of which is improving the quality of life of human beings in general and elderly and disabled people in particular. The BCI can be divided into non-invasive and invasive type, where in latter an IC is implanted in the brain by surgery. Hence people prefer non invasive BCI which involves only wearing of a headset or cap equipped with an active electrode system. In this project, our main aim is to develop a thought controlled smart home system. We will use a non-invasive BCI device known as Mind wave headset to capture EEG signals. The EEG signals are transmitted via Bluetooth to the interface Arduino. Hence the electroencephalography (EEG) signals produced by the brain electrical activity can be trained and used to control the home appliances. The selection of application was done by using eye blink signal.

4 Methodology Recent research shows that brain computer interface can be used for motion disabled people, however the mean classification rate achieved is above 80%. This means there is still 10-20% error rate. This error may result in loosing user control. Hence in this study we propose a very simple and effective method for smart home control. Fig. 1. Mind wave headset

5 EEG Signal Acquisition and of Virtual Smart Home System For EEG signal acquisition the Mind wave headset is used. It has one EEG electrode and reference electrode (Ear clip). It has a built in Bluetooth transceiver is connected wirelessly to the Arduino board. Fig.1 shows the Mind wave headset and the location of the electrodes. Mind wave headset includes the sensor that touches the forehead, the contact and reference points located on the ear pad, and the onboard chip that processes all of the data and provides this data to software and applications in digital form. Both the raw brainwaves and the esense Meters (Attention and Meditation) are calculated on the TinkGear chip. The aim of this project is to acquire and identify the EEG signal that is related with the user intention (attention) to operate a device in the smart home. In this study we have used a simple feature i.e. every eye blink to create an event. So whenever the user will create an eye blink a particular device will be selected, after then give the high attention to change the state of selected device (If previous state ON means its goes to OFF state).for that every eye blink will counted and its used for selection of device and the device was controlled by attention levels of Mind wave.

6 Fig. 2. Block Diagram for Smart Home Control

7 In order to control and operate the home using brain signals, a virtual home environment has been created. In the virtual environment there is indoor and outdoor access, it consists of many rooms, each having many devices like TV, MP3, lights, temperature control, and doors to operate. The user will select his desired application using of an eye blink (or a smirk, or a combination of actions if needed and to increase the sensitivity of the system), that will cause a select on the desired control device, as a result the control device will be toggled. For example, the subject can turn on the light of a room by selecting the light device using eye blink and give attention. Fig. 2 shows the block diagram of the brain controlled smart home system. A simple flow chart of the flow of events in a brain controlled smart home system is shown in Fig. 3.

8 Fig. 3. A sample flow chart of event detection and control in a smart home

9 Mind wave Device and Arduino This step-by-step guide is intended to quickly get a typical Arduino board user up and running with the Mind Wave Mobile. This guide will show how to setup the HC-05 Bluetooth Module quickly for communication with the Mind Wave Mobile and gives example code to parse the output stream. The parsed output stream can then be used to perform various tasks such as to control the speed of a motor, to light up LEDs, and to control the position of a servo motor, etc. Features Interfacing the Mind Wave Mobile with an Arduino board using HC-05 Bluetooth module Acquiring the EEG data and esense values from the data stream. Devices are controlled by Attention level of mind wave EEG values. Display the values in serial monitor.

10 Required Materials This application requires the following materials: Hardware s: 1. Mind wave Mobile Device 2. Arduino Board 3. HC-05 Bluetooth Module 4. USB cable (for Arduino) 5. Connecting Wires. 6. Bread board no of LED s no of Resister (100E) 9. Bluetooth dongle, if your computer does not have internal Bluetooth capabilities Software s 1. Arduino r2

11 Project Materials

12 Configure the HC-05 Module for Mind wave Device Find the Unique number of your Mind wave Device 1. Select your Mind wave device

13 2. Right click Properties Bluetooth Then know your Unique Identifier Number (Note it) Example: This module Unique no =9cb7,0d,89e51c

14 Diagram for HC-05 Bluetooth Module with Arduino: Method 1 for AT Commands Note: Here the Arduino Board used only for USB to UART converter. So don t want to download any source code (Sketch) to board.

15 Modes of HC-05 Bluetooth Module: How to HC-05 Module Enter into Command mode: Press and Hold the Push switch in HC-05 module at few secs, when its power ON (In between OFF state to ON state). If HC-05 Module in Command mode means, the LED blink at low toggle (blinking slowly). Buadrate for Command Mode In command mode the HC-05 module works in buadrate (fixed buadrate). How to HC-05 Module Enter into Data mode: In default state, the HC-05 module works as a Data mode operation. If HC-05 Module in Data mode means, the LED blink at High toggle. Buadrate for Data Mode In Data mode the HC-05 module works in variable buadrate.

16 AT Commands for HC-05 Module Initialize for Mindwave Kit First Enter into command mode, after then enter the following commands for HC-05 module to communicate our Mindwave Kit. 1. AT+NAME= PANTECH 2. AT+UART= 57600,0,0 3. AT+ROLE= 1 4. AT+PSWD= AT+CMODE= 0 6. AT+BIND= XXXX,YY,ZZZZZZ (Mindwave Unique Number) 7. AT+IAC= 9E8B33 8. AT+CLASS= 0 9. AT+INQM= 1,9,48

17 Verification commands for HC-05 Module 1. AT OK 2. AT+UART? +UART:57600,0,0 OK 3. AT+ROLE? +ROLE:1 OK 4. AT+PSWD? +PSWD:1234 OK 5. AT+CMODE? +CMODE:0 OK

18 6. AT+BIND? +BIND:9cb7:0d:89e51c OK 7. AT+IAC? +IAC:9E8B33 OK 8. AT+INQM? +INQM:1,7,48 OK Note1: For using above AT commands don t use (Double quotes) only enter In-between commands from double quotes. After the end of every command press Enter Key

19 Open serial communication software (Here we have to use Flashmagic software) 1. Select your Port and set Baudrate is 38400

20 2. Enter the AT commands (Initialize)

21 3. Enter the AT commands (Verification)

22 Some Tips about HC-05 Module Configuration 1. HC-05 module configure as Mater device and Mindwave kit act as Slave device 2. And the HC-05 module was configuring as Specific address mode. Therefore it s Pair with only the specific device. 3. Here the specific device is our Mindwave kit (Using BIND AT command, we configured the Mindwave unique no to HC-05). 4. So the Mindwave and HC-05 module automatically paired, when we switch ON the both device. 5. After this configuration the HC-05 Bluetooth Module will ready for use this project.

23 Diagram for HC-05 with Arduino (with sketch) Method 2 for AT Commands Connection Diagram

24 Steps to Follow 1. Make connections as per the connection diagram. 2. Program (Upload the sketch) the HC-05 with Arduino source code to Arduino board using Arduino compiler 3. Open the Serial monitor with buadrate of Type the AT commands and click Enter in Serial monitor Serial Monitor Screen

25 Connection Diagram for Smart Home Control Note: Keep the Rx (Pin1) line in Open connection

26 Steps to follow [Output at LED_UART] 1. Switch ON the Mind wave mobile and unpaired the last connections, then OFF the Mind wave headset. 2. Program (Upload the sketch) the Mindwave_Arduino_Home_Application source code to Arduino Board using Arduino compiler. 3. Make all the connection as per the connection diagram, and then switch ON the Arduino Board. 4. After that switch ON the Mind wave headset, now the Mind wave device and Arduino Board (with HC-05) will pair automatically. 5. Wear the Mind wave device in Head, and blink your eyes. 6. For Mind wave device need calibration to detect the eye blink values in EEG data s. So far we have to wait few minutes (approximately 2 min) for calibration. 7. After calibration, every eye blink was counted (1 to 7).It s used for selection of home device. 8. If device was selected means you will give concentration (attention) to change the state of particular device. For every high attention the selected device will changed.

27 Tabulation Device Selection Controlling Device There have an indication Led s, which shows the selected, device in 3 bit representation (as shown in tabular Colum). For Example, if you want to control the 5 th device means, initially you can do the device selection as 101 by using eye blink. After that you can give the attention levels and control the status of the selected (5 th ) device. Every high attention the selected device status will be toggled.

28 Real Time View of Smart Home Control

29 Detection of Eye Blink The eye blink was detected by Raw EEG data from Mind wave head set. For every second the eye blink was updated. i) Noisy EEG Raw data wave (Before Calibration)

30 ii) EEG Raw data wave (After Calibration)

31 iii) EEG Raw data wave (When Eye Blink occurs)

32 Some Tips about Project 1. The Indication LED of Mind wave device glow continuously in blue color, when it s paired with arduino HC-05 module. 2. If blue LED in blinking conditions means, it s not pair with device. 3. The Mind wave mobile and HC-05 module automatically paired, when we switch ON the both device. 4. Before sketch the program, you have to disconnect the Tx and Rx lines (Keep in Open connection). 5. If calibrations will take more times means, you have to ensure your Mind wave headset (may its wear improperly). Output Details The output will show in LED s and Serial Monitor of the arduino compiler. For this project the LED s are placed instead of device (with relay circuit).

33 Upload the sketch to Arduino board

34 Hardware output 1. Green LED s indicate the device selection which is control by eye blink 2. Red LED s are devices, which are control by mind wave attention

35 Serial Monitor output values

36 Program for Mindwave_Arduino_Home_Application *************************************************************************************** Title: Mindwave_Arduino_Home_Application *************************************************************************************** // // // Program : Mindwave with Arduino // // Interfacing : HC-05 Bluetooth Module // // Output : Home Application Control by Mindwave // #define BAUDRATE #define SLED1 13 #define SLED2 12 #define SLED3 11 #define DLED1 9 #define DLED2 8 #define DLED3 7 #define DLED4 6 #define DLED5 5 #define DLED6 4 #define DLED7 3 #define Theshold_Eyeblink 120 #define EEG_AVG 60 byte checksum=0,generatedchecksum=0;

37 unsigned long payloaddatas[5] = 0; unsigned long payloaddatab[32] = 0; unsigned long Avg_Raw,Temp,Temp_Avg; unsigned int Raw_data,Raw_EEG,Poorquality,Attention,Plength,On_Flag=0,Off_Flag=1; unsigned int EyeBlink=0,EyeFlag1=0,EyeFlag2=0,EyeFlag3=0; unsigned int Attn_Temp,Att_Avg,Attn_On_Flag=1,Attn_Off_Flag=0,Sel_Device=0; unsigned int j,k=0,n=0; void setup() Serial.begin(BAUDRATE); pinmode(sled1, OUTPUT); pinmode(sled2, OUTPUT); pinmode(sled3, OUTPUT); pinmode(dled1, OUTPUT); pinmode(dled2, OUTPUT); pinmode(dled3, OUTPUT); pinmode(dled4, OUTPUT); pinmode(dled5, OUTPUT); pinmode(dled6, OUTPUT); pinmode(dled7, OUTPUT); byte ReadOneByte() // One Byte Read Function

38 int ByteRead; while(!serial.available()); ByteRead = Serial.read(); return ByteRead; void loop() // Main Function if(readonebyte() == 170) // AA 1 st Sync data if(readonebyte() == 170) // AA 2 st Sync data Plength = ReadOneByte(); if(plength == 32) // Big Packet Big_Packet (Plength); else if(plength == 4) // Small Packet Raw_EEG = Small_Packet (Plength); if (j<512) Temp += Raw_EEG; j++; else

39 Onesec_Rawval_Fun (Temp); Home_App_Fun(EyeBlink,Attention); j=0; Temp=0; void Big_Packet(byte data) generatedchecksum = 0; for(int i = 0; i < data; i++) payloaddatab[i] = ReadOneByte(); //Read payload into memory generatedchecksum += payloaddatab[i] ; generatedchecksum = generatedchecksum; checksum = ReadOneByte(); if(checksum == generatedchecksum) Poorquality = payloaddatab[1]; Attention = payloaddatab[29]; // Varify Checksum

40 unsigned long Small_Packet (byte data) generatedchecksum = 0; for(int i = 0; i < data; i++) payloaddatas[i] = ReadOneByte(); //Read payload into memory generatedchecksum += payloaddatas[i] ; generatedchecksum = generatedchecksum; checksum = ReadOneByte(); if(checksum == generatedchecksum) // Varify Checksum Raw_data = ((payloaddatas[2] <<8) payloaddatas[3]); if(raw_data&0xf000) Raw_data = (((~Raw_data)&0xFFF)+1); else Raw_data = (Raw_data&0xFFF); return Raw_data;

41 void Onesec_Rawval_Fun (unsigned long data) Avg_Raw = data/512; Serial.print(" Rawdata :"); Serial.print(Avg_Raw,DEC); if (On_Flag==0 && Off_Flag==1) if (n<4) Temp_Avg += Avg_Raw; n++; else Temp_Avg = Temp_Avg/4; if (Temp_Avg<EEG_AVG) On_Flag=1;Off_Flag=0; n=0;temp_avg=0; Eye_Blink (Avg_Raw);

42 unsigned int Eye_Blink (unsigned int data) if (Poorquality==0) if (On_Flag==1 && Off_Flag==0) if ((data>theshold_eyeblink) && (data<300)) EyeBlink = 1; else if (data>300) On_Flag==0;Off_Flag==1; EyeBlink = 0; else EyeBlink = 0;

43 else EyeBlink = 0; return EyeBlink; void Home_App_Fun(unsigned long data1,unsigned int data2) Serial.print(" Attention :"); Serial.print(data2,DEC); Serial.print("\n"); Device_Select_Fun(data1); Device_Control_fun(data2); void Device_Select_Fun(unsigned int data1) if (EyeFlag1==0 && EyeFlag2==0 && EyeFlag3==0) if (data1==1) EyeFlag1=0; EyeFlag2=0; EyeFlag3=1; Selection_Disp_Fun(); Sel_Device = 1;

44 Attn_Temp=0; k=0; else if (EyeFlag1==0 && EyeFlag2==0 && EyeFlag3==1) if (data1==1) EyeFlag1=0; EyeFlag2=1; EyeFlag3=0; Selection_Disp_Fun(); Sel_Device = 2; Attn_Temp=0; k=0; else if (EyeFlag1==0 && EyeFlag2==1 && EyeFlag3==0) if (data1==1) EyeFlag1=0; EyeFlag2=1; EyeFlag3=1; Selection_Disp_Fun(); Sel_Device = 3; Attn_Temp=0; k=0;

45 else if (EyeFlag1==0 && EyeFlag2==1 && EyeFlag3==1) if (data1==1) EyeFlag1=1; EyeFlag2=0; EyeFlag3=0; Selection_Disp_Fun(); Sel_Device = 4; Attn_Temp=0; k=0; else if (EyeFlag1==1 && EyeFlag2==0 && EyeFlag3==0) if (data1==1) EyeFlag1=1; EyeFlag2=0; EyeFlag3=1; Selection_Disp_Fun(); Sel_Device = 5; Attn_Temp=0; k=0; else if (EyeFlag1==1 && EyeFlag2==0 && EyeFlag3==1) if (data1==1)

46 EyeFlag1=1; EyeFlag2=1; EyeFlag3=0; Selection_Disp_Fun(); Sel_Device = 6; Attn_Temp=0; k=0; else if (EyeFlag1==1 && EyeFlag2==1 && EyeFlag3==0) if (data1==1) EyeFlag1=1; EyeFlag2=1; EyeFlag3=1; Selection_Disp_Fun(); EyeFlag1=0; EyeFlag2=0; EyeFlag3=0; Sel_Device = 7; Attn_Temp=0; k=0; void Selection_Disp_Fun() digitalwrite(sled1,eyeflag1); digitalwrite(sled2,eyeflag2); digitalwrite(sled3,eyeflag3);

47 void Device_Control_fun(unsigned int data2) switch (Sel_Device) case 1: Attention_Fun(data2,DLED1); break; case 2: Attention_Fun(data2,DLED2); break; case 3: Attention_Fun(data2,DLED3); break; case 4: Attention_Fun(data2,DLED4); break; case 5: Attention_Fun(data2,DLED5); break; case 6: Attention_Fun(data2,DLED6); break; case 7: Attention_Fun(data2,DLED7);

48 break; default : break; void Attention_Fun(unsigned long data1,unsigned long data2) if (k<4) Attn_Temp += data1; k++; else Att_Avg = Attn_Temp/4; if (Att_Avg>80) if(attn_on_flag==1) Attn_On_Flag=0; Attn_Off_Flag=1; digitalwrite(data2,high); else if(attn_off_flag==1)

49 Attn_On_Flag=1; Attn_Off_Flag=0; digitalwrite(data2,low); k=0; Attn_Temp=0;

50 Conclusion and Discussion The main goal of this project is to design, develop and implement a brain controlled smart home system. In this system, the brain EEG signals are acquired using Mind wave headset. A single feature i.e. eye blink and mind attention values are used to control a virtual home environment. Each selection on the desired home application done by eye blink signal (feature).and will control that home item, like turn off the light. All this will happen in real time. It is possible to add more controls to the virtual environment by using a combination of activities like blinking an eye and a single or multiple blinks. The system has been trained and tested with 4 subjects. The preliminary results show a thought controlled smart home system can become a reality in the near future. It will result in a drastic change in the type and quality of life of individuals and disabled and elderly people. In addition it will also result in an increase in demand of consumer electronics devices that can be easily interfaced with BCI systems.

51 Reference Documents Links Reference Video Links ure=youtu.be

52 Did you enjoy the read? Pantech solutions creates information packed technical documents like this one every month. And our website is a rich and trusted resource used by a vibrant online community of more than 1, 00,000 members from organization of all shapes and sizes.

53 What do we sell? Our products range from Various Microcontroller development boards, DSP Boards, FPGA/CPLD boards, Communication Kits, Power electronics, Basic electronics, Robotics, Sensors, Electronic components and much more. Our goal is to make finding the parts and information you need easier and affordable so you can create awesome projects and training from Basic to Cutting edge technology.