I made a Python Script to Automate a Sudoku Game on Android after watching Engineer Man’s Videos on Youtube doing the same for different games.
The script can be divided into 5 parts
- Connecting to an Android device using ADB, and getting the screenshot of the game from it
- Using Pillow to process the screenshot for pytesseract
- Using pytesseract to extract the Sudoku Game Grid to a 2D List in Python.
- Solving the Sudoku Game
- Sending the solved input to your Android Device using Python
Out of the 5, I will be focusing mostly on 2,3 & 5 as 1 & 4 are topics that have been extensively covered.
The complete code is available on the following repository:
You can also watch the script in action on:
1 (a). Using ADB to Connect to your Device
Most of the tutorials on internet use Wired ADB, which discourages many people from using this method. I will be using Wireless ADB, which isn’t very difficult to setup.
Go to your Phone Settings > System > Developer Options (This might vary in different phones, so if it is not the same in your’s, look it up on the internet)
Turn on Android Debugging and ADB over Network.
ADB over Network
Note the IP Address and Port shown under ADB over Network
Install ADB on your computer
Go to your command-line / command prompt and enter
adb connect <ip-address>:<port>
Use the IP Address and Port from Step 3
When connecting for the first time, you will need to authorize the connection on your phone.
Your device should be connected to your PC over WiFi.
1 (b). Using ADB with Python (pure-python-adb)
You can define the following function to connect to the first ADB device connected to your computer using Python
We will be using this function later to return an instance of
ppadb.device.Device which will be used to take a screenshot, and send input to your device.
1 (c). Taking a Screenshot and saving it
pure-python-adb makes it very easy to capture a screenshot of your device. The
screencap function is all that you need to get the screenshot. Use Pythons File IO to save it to `screen.png`
Screenshot of Sudoku
2. Processing the screenshot with Pillow
In the captured screenshot, the accuracy of any OCR will be very low. To increase accuracy, I used Pillow to process the screenshot so that it only shows the numbers in black color on a white background.
To do that, we first convert the image to grayscale (or single channel) using
image.convert('L'). This will make the convert the colors to shades of greys (0-255).
Grayscale Screenshot of Sudoku
After this, we need the numbers (which are the darkest, or very near to black) in black color, and the rest in white. For this, we use
image.point() so that all the greys > 50 become white (255) and the rest (numbers) become 0. I also increased the Contrast and Sharpness a bit to be on the safer side.
Processed Screenshot of Sudoku
3. Extracting the numbers from the image using pytesseract
Using pytesseract on the whole image might give us the numbers, but it won’t tell us in which box the number was present. So, I use Pillow to crop each box and then use pytesseract on the cropped images. Before using pytesseract, I defined some functions to give me the coordinates of each box and to give me a cropped image of each box.
Since Sudoku has a 9x9 grid, I use two for loops from 0 to 8 to loop over each box. The pytesseract wasn’t accurate enough on the default configuration and I had to pass the config
--psm 10 --oem 0.
--psmargument defines the Page Segmentation Method.
Treat the image as a single character. This seemed most appropriate since I am passing cropped images of each box.
--oemargument defines the OCR Engine Mode.
Legacy Engine Only.
The following function will extract the numbers from the passed
image and return a 9x9 2D List with the numbers.
4. Solving the Sudoku Game
Now that we have the 9x9 Sudoku, we need to solve it. Solving Sudoku is a topic that has been covered a lot, and I also copied this code from geeksforgeeks.org.
5. Sending the solved input to your Android Device using Python
To send the input, I first filtered out the input from the solved Sudoku Grid,i.e, only send the values which were missing. I used the
get_coords function from earlier to get the coords of each box and then calculated their centres. I sent a touch at that centre using ADB, and then sent over the solution.
Running the code
All the code that I wrote is in functions and they are called one by one. Note that the grid that I get in step 3 isn’t passed directly to step 4. I use
deepcopy to create a copy of it, so that I can compare the solved grid with the unsolved/original one in step 5.