GAME AUTOMATORS. - Making Computer Vision and Machine Learning Fun!

Transcription

1 GAME AUTOMATORS - Making Computer Vision and Machine Learning Fun!

2 Hello! I am

3 What is Game Automators? Making learning fun is extremely important. Many of us would have played video games when we were young, and what kept us glued and excited was the desire to make a high score. How awesome would it be to build your own robots and algorithms that play these games? How exciting would it be to watch your algorithm making it through the level that you found very hard?

4 Vision of Game Automators The vision of game automators is to make learning fun and help students collaborate and build cooler things from what they learn. Creativity now is as important in education as literacy, and we should treat it with the same status ~ Sir. Ken Robinson

5 Why are we doing this? Learning should be fun, games interest everyone and the education system doesn t really draw a major correlation between engineering study and its practice. It s an effort to interest more students into STEM courses.

6 Record breaking High Scores More than 15 games automated so far including Piano Tiles, Unblock me and stick hero

7 Automation There are multiple ways in which the automation of these games are done at Game Automators, some of the methods being 1. Using image processing. 2. Using electronics and circuits. 3. Using a combination of image processing and electronic circuits.

8 Work done so far Automating mobile games in android Writing an open source book, publication pending. Creating tutorials on how to develop these games and how to automate more games Building the team size from 2 to 20.

9 Quick Start to Game Automation Lets start with a small sample of Find the Difference Game

10 Find the Difference Basic Algorithm: Pixel by pixel image subtraction of two images P1 and P2 for an output image O! ", $ = & ' ", $ & * ", $

11 Find the Difference Conversion of difference matrix O to binary image by thresholding and enhancing using dilation.! ", $ = & ' ", $ & * ", $

12 Find the Difference Simulate the touch events via the android adb for the given pixel coordinates

13 For the hardware lovers Automating and always winning the game Ready Steady Bang

14 Ready Steady Bang The game has two players, one is system and the other is player controlled. Both the players are allowed to shoot as soon as the word 'BANG' pops on the screen. The objective of the game is to fire the opponent before he does by tapping anywhere on the screen immediately after the 'BANG' appears. We ll look at: 1. Sensor placement 2. Touch simulation 3. Arduino control

15 Ready Steady Bang

16 Using hardware and software (Extreme Hacking) Automating Stick Hero, by Ketchapp

17 Stick Hero In this game, the player need to hold on the screen such that the stick that the man is holding increase its length such that the stick can be used to across the gap between two black pillars. The black pillars are detected using image processing and the distance between them is calculated. This distance is converted to time by using a linear equation. The screen is touched by using the adb tool library.

18 Stick Hero Step 1: Detecting the black pillars

19 Stick Hero Step 2: Determining the duration of the touch

20 Stick Hero Step 3:Simulating Touch Relays are directly connected to the output pin of the Arduino. It is equivalent to a touch if the voltage given is high as there is a path for the current to flow to the ground. It is equivalent to not touching if the voltage given is low. And finally this is how it works!!

21 Stick Hero

22 Other interesting games Free Flow Unblock Me

23 Future Plans

24 Move towards automating games on ios Instruments is a great tools for ios applications and acts as a debug bridge. It could be a great tool for us to move towards automating ios apps. An ios App

25 Web frameworks for game automation Building an MIT AppInventor like environment where the web application connects to the phone and allows the user to automate the process on the phone by typing in adb and Python/CV commands through a web interface. Web Framework for Game Automation

26 Want to Contribute? Fork and enhance existing games Automate newer games Contribute to us in writing the book and using it as a teaching aid Help us by contributing to our website and building tools to automate Help us extend the automation to consoles and other operating systems.

27 The Team behind

28 Thank You! Any questions? You can find me fb.com/sudheesh001 github.com/sudheesh001