Autoit Bejeweled bot tutorial part 2: identifying the gem play field

AutoIt Bejeweled bot tutorial part 3This article makes part of the Bejeweled bot tutorial series.

Last time we have set up a framework for our Bejeweled bot and written the first code to locate the Bejeweled window. At this stage we know which window to work with, so we can locate the gem play field and calculate the gem positions.

Identifying the gem play field

The possible solutions

Several methods to position the gem play field exist; they all have their advantages and disadvantages. Possible methods to determine the gem play field borders are:

  1. Fixed start coordinates and fixed end coordinates: start with the x,y coordinates of both the upper left and lower right side of the field.
  2. Fixed start coordinates and fixed dimensions: start with the x,y coordinates of the upper left side of the field together with the length/height of the field.
  3. Searching both start and end coordinates: start with searching for the color which marks both the start and the end of the field on the screen.
I opted for the third method, as that is the most flexible one. Each site can have its bejeweled game positioned on a different location. By searching both start and end each time, we can be sure to have the correct position every time.

Searching the play field corners

In order to search the gem play field, we will search both the upper left and lower right corner of the field. Once we have their coordinates, we can calculate the other corners ourself.

bejeweled gem play fieldbejeweled bot first cornerbejeweled bot last corner

Add following key to the configuration file:

In order to read the newly added configuration keys, add following code to the declaration section:

$cornerColor1=IniRead ( $iniFile, "Common" , "cornerColor1", "0x313029" )
$cornerColor4=IniRead ( $iniFile, "Common" , "cornerColor4", "0x313029" )
Next we will prepare the variables in which to store the coordinates of the four corners. We will also create variables representing three gems: the upper left, upper right, and lower left gem. With these corner and gem locations we can easily process the play field. Add following code to the declaration section



$BlockWidth = 0
Now it's time to search the screen for the color specified in the ini file. Function 'pixelSearch' returns the x,y coordinates of the first found pixel matching that color, from left to right and top to bottom. The coordinates are stored in the &Corner1 variable. If the color cannot be found, the bot stops, else the mouse pointer is moved to the found pixel. Add following code to the main section:

_log("Searching top left position of playfield")
ToolTip('Searching top left position pixel with color ' & hex($CornerColor1, 6) , 0, 0)

$Corner1 = pixelSearch(0,0,@DesktopWidth,@DesktopHeight, $CornerColor1,0)
If @error Then
msgbox(0,"","Top Left pixel with color: " & hex($CornerColor1, 6) & " not found")
Once we have found the first corner we can look for the fourth corner, which is located at the lower bottom of the play field. Again, if the color cannot be found, the bot stops, else the mouse pointer is moved to the found pixel. Add following code to the main section:

_log("Searching bottom right position of playfield")
ToolTip('Searching bottom right position pixel with color ' & hex($CornerColor4, 6) , 0, 0)

$Corner4=_pixelSearchReverse(0,0,@DesktopWidth,@DesktopHeight, $CornerColor4)
If @error Then
msgbox(0,"","Bottom Right pixel with color: " & hex($CornerColor4, 6) & " not found")
Function _pixelSearchReverse doesn't come with Autoit by default. Add this function to the function section:

Func _pixelSearchReverse($left,$top,$right,$bottom,$color)
$i = 0
$j = 0

for $i = @DesktopWidth to 0 Step -1
for $j = @DesktopHeight to 0 Step -1

$currentcolor = PixelGetColor($i,$j)
if $currentcolor = $color then
$coordFoundColor[0] = $i
$coordFoundColor[1] = $j
return $coordFoundColor

With help of corner1 and corner4, we can calculate the x,y coordinates of the other corners, corner2 and corner4. Add this code to the main section:

;Calculate position of remaining corners


Calculating the gem positions

With help of the corner positions we can now calculate the position of three main gems: the upper left, upper right, and lower left gem. Soon we will be needing this exact position of these gems in order to process all gems on the play field.

Add following code to the declaration section:

$AmountGems=IniRead ( $iniFile, "Common" , "AmountGems", 0 )
First we will calculate the distance between each gem. Add following code to the main section:

;Calculate distance between center of each gem
$BlockWidth = ($Corner2[0] - $Corner1[0]) / $AmountGems
Next we will calculate the position of the three main gems. Add following code to the main section:

$Field1[0]=$Corner1[0] + ($BlockWidth / 2)
$Field1[1]=$Corner1[1] + ($BlockWidth / 2)

$Field2[0]=$Corner2[0] - ($BlockWidth / 2)
$Field2[1]=$Corner1[1] + ($BlockWidth / 2)

$Field3[0]=$Corner3[0] + ($BlockWidth / 2)
$Field3[1]=$Corner3[1] - ($BlockWidth / 2)

Proof running the code

At this point you should be able to run you script and see more results. Save your script and configuration file, and run the bot script by pressing F5 in your ScITE editor. If you correctly followed all steps, the bejeweled browser window becomes active. Next the mouse pointer is moved to the upper left corner of the play field, followed by the lower right corner of the play field. Don't worry; searching for the right bottom pixel might take a while. Feel free to improve the _pixelSearchReverse function, which will speed up the initialization process a lot!

If you encounter difficulties with the execution of these steps and you don't succeed in fixing the problem yourself, then leave a message in the comment section. Don't forget to include the error message shown in the ScITE output window. I will answer to your question as soon as possible.

In the next part of this tutorial we will be creating the main loop for the bot and implement the function to read the gem colors.

Continue to the third part of the bejeweled bot tutorial: reading the gem colors

Related Posts by Categories


37 Responses to "Autoit Bejeweled bot tutorial part 2: identifying the gem play field"

Anonymous said... March 27, 2009 at 12:29 PM

I followed your instructions to the tee, I've tried everything I can think of too.

But I keep getting

Error: Unknown function name.

This happens for corner1 to 4 and field1 to 3.

Steven Machtelinckx said... March 27, 2009 at 3:25 PM


It appears that the array functions are not recognized.
Did you add following statements to the declaration section, as described in the first part of the tutorial?

#include <array.au3>
#include <file.au3>

Anonymous said... March 27, 2009 at 6:04 PM

Well what do you know, heh heh.

Thanks. :P

Anonymous said... March 29, 2009 at 2:37 PM

So what next? How do I get it to swap gems?

Can this be made to work on Facebook Bejeweled Blitz?

Steven Machtelinckx said... March 30, 2009 at 6:42 AM

Next article will feature the pixel reading at different locations for each gem. This is needed to ensure a correct recognition of the gems. After that we'll swap the gems.

I haven't tried bejeweled blitz yet. I suppose this bot works as well on bejeweled blitz, hence some modifications. The way of identifying the corners of the playfield for instance should be changed.

Nick said... April 1, 2009 at 5:20 PM

I am trying to make it work on Blitz :)
One problem is the corners. They change color so I give the script the coordinates myself (hardcoded)

Steven Machtelinckx said... April 2, 2009 at 8:03 AM

Automatically identifying the corners of Bejeweled Blitz or Bejeweled 2 seems not to be easy. So in that case using fixed coordinates like you suggest is a much faster solution.

Though I would also recommend this approach:

Try to find a location, of which the color doesn't change, in the neighbourhood of the original color which is changing. Next calculate the horizontal and vertical difference in pixels between them (like an offset) and store the difference in the configuration file.

By applying this technique you are able to detect your "corners" automatically and adjust their location according to your offset in the configuration file.

Nick said... April 3, 2009 at 3:50 PM


That was my other solution. I'll wait untill the other parts are posted and then I'll finish the blitz script

Steven Machtelinckx said... April 3, 2009 at 4:20 PM

That's great! Don't forget to show us the final result when you're ready :-)

Anonymous said... April 5, 2009 at 3:24 PM

I'm interested in the creation of robots for automating actions anyway. It's one of the things I wanted to be able to do when I got into programming. I always wanted to create a minesweeper bot but never knew where to start:

If I could develop an API to be able to read the board and perform the mouse-clicks I could then write the logic engine myself. (Could do the same with Bejeweled). It's just the first part that I have no idea how to do.

Steven Machtelinckx said... April 6, 2009 at 1:30 PM

The first part is usually the hardest part: defining an approach and creating an API to get started. At the same time this is also the most satisfying part, as you have created something you ought to find impossible before.

I think the basis for a bejeweled and minesweeper bot are the same. Only the logic of playing the game will vary. With the idea of this bejeweled bot you can create bots for many other games.

Miguel Jeronimo Alvarez said... April 6, 2009 at 11:53 PM

hi, congratulations for the tutorial, and sorry for my poor English but I'm from Spain
When I tried to play your solution to find the game screen, it doesn't work for the botton rigth
so I create my solution and I share here

My idea is search the corners of the first jewel square and then how we know the number of squares we can know where is going to be the end of the playfield

_log("Searching bottom right position of playfield")
ToolTip('Searching bottom right position pixel with color ' & hex($CornerColor4, 6) , 0, 0)

$Corner4=pixelSearch($corner1[0],$corner1[1],@DesktopWidth,@DesktopHeight, $CornerColor4)
If @error Then
msgbox(0,"","Bottom Right pixel with color: " & hex($CornerColor4, 6) & " not found")


You can see that know the function search reverse is no needed

Miguel Jeronimo said... April 6, 2009 at 11:55 PM

I forget to say that know you need to change that at the config.ini


Steven Machtelinckx said... April 7, 2009 at 9:03 AM

Thank you for your feedback Miguel. Don't worry about your English, I'm not a native English speaker either (I'm from Belgium) :-)

It's weird that the right bottom color couldn't be found. Are you running the bot on the same site as I do? ( Do you mind sending me a (partial) screenshot of the bejeweled game you are running so I can have a look into it? I'm really curious why the color couldn't be found.

On thing is sure: you have found a very nice solution to that problem. Why didn't I come up with that Idea before :-)? I don't have photoshop with me so I'm not able to check which color you are looking for, but I guess it's the color of the second gem background, am I right? I won't change my code at this moment, but when all articles are posted, I plan to improve my code and to make it more fancy as well. I think your code fits well into the improvement plan of the bot :-)

Keep up the good work! It's nice to see motivated people!

Miguel said... April 7, 2009 at 12:44 PM

I run the bot at the same site, the problem is that the bot stay "Searching bottom right position of playfield" for all the time(the function has not end).
Like you say, the color i search is the color of the second gem background(I start to search in the same line is the $corner1 because there is little line arround the gamefield of that color

I will try to continue with the code by my own(I think it will be interesting to see different ways to solve the same problem.

It's my first Autoit code, but I have some experience with other languajes

Miguel Jeronimo said... April 7, 2009 at 10:35 PM

When you will continue with the tutorial, I'm having some problems with the jewels position detect
If you have a complete version i will like to see the code, because I think in a couple of improves to make it looks more "random"

here my mail

Steven Machtelinckx said... April 8, 2009 at 7:24 AM

I vote as well for different solutions, as that will enable me to improve my version as well :-)

This weekend I'll start working on the next part of the tutorial. It will be online somewhere next week if everything goes well :-)
The jewel position detection is a very tricky part and has required me to put a lot of effort into. This topic will be included in the next article.

The bot we're creating with the turial has no randomizer implemented, but that would be a good point for improvement when all articles are posted.

Which problem are you experiencing exactly with the jewel position detection?

Miguel said... April 8, 2009 at 9:21 AM

I split the gamefield in the amount of jewels and I start to make a "pixelgetcolor" in the middle of the jewels. But the middle is not exactly and there are some problems to reconize the color.
Also,i think my method is no enought faster(pixelgetcolor is heavy for the sistem).

Steven Machtelinckx said... April 8, 2009 at 7:24 PM

That's the same problem as I encountered. There are two solutions for this:

1: comparing gems without taking the color code too strictly, by applying a margin. This way the colors don't need to be exactly the same to be "the same gem".

2: reading the pixel color not (always) in the middle of the gem by using an x/y offset.

I chose the second approach, which gives me a 100% success rate and never seemed to fail.

This topic will be covered by the next article

Anonymous said... April 11, 2009 at 10:48 PM

To get the colour is necessary in bejeweled 2/blitz. In blitz you need to be able to determine those with a multiplier which are a similar colour to the one they match with but are not the same shape.

The strategy should also generally be to explode these as soon as possible.

We could allow a "learning" process if it doesn't know what colour it is i.e. if it tries a move and it gets rejected it knows what it isn't.

You also need to know that the power cube shines brighter than the others of its type.

Finally you need to know the difference between a hypercube and the white one.

Steven Machtelinckx said... April 12, 2009 at 10:46 PM

I didn't know that Bejeweled blitz would be so advanced. Deciding on which gem to move based on the color is something we already do with this bot, but recognizing the shape would really take the bot to a next level :-)

I really like the idea of including those gems into the strategy but it will be much harder. Maybe we should try to include shape recognition into this bejeweled I bot when it's finished. At that moment we will have a working bot and we can easily verify whether the new shape recognition functionality works.

Rogee said... May 22, 2009 at 8:38 AM

hi, am a noob in this field and have just moved to stage 2 but need to understand how to determine the co-ords and colours. Could you assit please


Steven Machtelinckx said... May 22, 2009 at 6:20 PM

Rogee, determining the play field corners is explained in this part of the tutorial, while the gem color recognition is treated in part 3. That part is available as well. Which part is it that you don't understand?

Rogee said... May 22, 2009 at 8:58 PM

perhaps I'm jumping the gun a bit and trying to test to early on. will get back to you when i've followedit though :)

Rogee said... May 22, 2009 at 9:31 PM


I can't find the top left corner when i run the script :(

have got a screen dump and analysed the colours in the 4 corners using a graphics package. All 4 corners are different so I took the RGB values and ran them through a html hex code convertor and put the values for corners 1 & 4 into the .ini and declaration but it still won't find them.

Steven Machtelinckx said... May 22, 2009 at 11:18 PM

can you send me the ini file you're using (or at least the content), together with the screen shot you're using? You can send it to It will be much easier to help you that way :-)

Rogee said... May 23, 2009 at 12:07 AM

sent the lot, .ini, script and screen dump

thanks for the help

Steven Machtelinckx said... May 23, 2009 at 7:46 AM

The screenshot you sent me is one from bejeweled blitz :-) So far this bot only supports bejeweled 1, so no bejeweled 2 or bejeweled blitz. That's why it didn't find the corners with the given colors.

I have made it work with bejeweled blitz though, but I needed to make quite some changes. There's no article available for that yet, though. I suggest you first get familiar with the basics of a bejeweled 1 bot. Next you can try to make it work with bejeweled blitz.

Prism said... May 27, 2009 at 1:05 PM

Steven, Any word on when the logic section of the program will be available? I'm new to scripting but your tutorial has been very helpful. The program works great.

Steven Machtelinckx said... May 27, 2009 at 1:17 PM

I'm happy to hear that you find it helpful. That's at least one of the goals of this tutorial :-)

I'm planning to have it ready as soon as possible, but I can't fix a date yet :-) Next section will cover the logic for the moves and will be the last part of the series.

Anonymous said... October 23, 2009 at 1:04 AM

Its pure genius the way you found the top and bottom corners. I went by that in really complicated means the first time I tried... but I guess thats what happens when you just type and don't think.. Good job!

Steven Machtelinckx said... October 23, 2009 at 7:56 AM

Thanks Anonymous :-) I went through the same process as well you know. It's a continuous process of improving what you have. Sometimes you just need some time to think it over :-)

Good luck with the continue of the bot!

PyRoMaNiaC said... November 30, 2009 at 9:39 AM

My code hangs at "Searching buttom right pixel with 2F2F2F"
My game is a bit different than the one on your site. Mine is located at
and have 10 horizontal and 8 vertical gems... I tried the fix as someone posted but then I cant get it at exact Cornor4 and gem count is messed :)
I used 4 hours last night trying to find out where I need to edit it.... but no luck.

I hope you still update this thread :) and point me in the direction. I love this idea :D

Steven Machtelinckx said... December 4, 2009 at 8:03 AM

Usually the pixelsearch is quite slow, especially if you have a high resolution and thus many pixels have to be scanned. But in the end, if the pixelsearch function did not find the pixel of your color, the check on the @error variable should cover this case.
Could it be that you specified the cornercolor in format 2F2F2F instead of this format 0x313129?

Glenn C. said... December 29, 2009 at 2:13 PM

Great tutorial,

I have a small suggestion concerning the search for the bottom right corner. You can also use the pixelSearch function from the API, as with the top left corner. You only have to call it with arguments bottom < top and right < left. That way it will search bottom-to-top and right-to-left. (read more on: So you don't have to write the pixelSearchReverse method yourself.

Vette tutorials, blijven gaan ! :D

Steven Machtelinckx said... December 29, 2009 at 2:16 PM

Thanks for you input Glenn. The approach you suggested is probably much faster than mine...

Cheers ;-)

TestWithUs said... May 7, 2013 at 6:04 AM

SWIFT Interview questions on

For selenium solution visit

For QTP interview questions

Post a Comment

Recent Articles

Top Commenters

Recent Comments