One of the factors that has limited the prevalence of vision systems in amateur robotics has been the lack of affordable processing power in a small package. Another has been the difficulty of obtaining a low-cost vision sensor that is easy to interface with a controller board.
This article describes how to implement a simple monochrome vision system on a robot powered by the Mini RoboMind (MRM). The MRM is a small, powerful, inexpensive 68332-based robot controller designed by Mark Castelluccio. The vision sensor is a CMOS image sensor taken from the GAME BOY® Camera, an accessory for the GAME BOY® handheld gaming device.
This article covers the basics of interfacing the image sensor with the controller board, and includes code samples for both the MRM and a host PC (connected via serial port) to allow captured images to be viewed on the host PC. Image processing techniques are beyond the scope of this article, but I've included a few pointers.
The host PC code for this article is written in JavaTM, and has been tested on Windows® and Linux® platforms. In theory, it should also run on any other operating system that supports JavaTM 2, as long as there is also a working implementation of the JavaTM COMM API available (the JavaTM COMM API is a platform-independent programming interface that JavaTM applications can use to access communication devices such as serial ports and parallel ports).
This article assumes that in addition to basic tools and a host PC, you have the following components:
Although it seems increasingly difficult to obtain the now obsolete GAME BOY® Cameras, it is still possible to mail-order them from some retailers at the time of writing. Try navigating to MySimon.com, and enter GAME BOY Camera in the search field. There are also a few people selling them on the eBay auction site too.
The MRM is available by mail order from the RoboMinds web site in a variety of different configurations and options.
At the time of writing, the basic MRM is available for under $100. The RAM and processor upgrades are well worth the money, and the package bundles represent good value. The MRM MAX RS232 package is a great starter package, and is ideal for this project.
For some nitty-gritty technical details on the MRM, consult the article "Three 68332 Designs Presented" included in the Links section at the end of this article.
At the heart of the GAME BOY® Camera is the M64282FP CMOS image sensor from Mitsubishi. You can download the data sheet for this device from the Links section at the end of the article.
This device, which requires only a single 5-volt power supply, has an effective resolution of 128 pixels across by 123 pixels vertically. The M64282FP interfaces with the host micro via an interesting combination of digital and analog I/O. All control and clocking signals are digital, but the intensity of each image pixel is output as an analog voltage. The MRM's built-in fast analog to digital converter is perfect for this application, enabling the sensor to be connected to the MRM with zero "glue" components.
Figure 3. M64282FP CMOS Image Sensor - Lens Removed.
To access the image sensor, you will need to open the case by removing the four screws in the back (warning: this will obviously void any warranty you may have had when you purchased the GAME BOY® Camera). This is not as easy as it sounds, as these are special screws with three slots. I have managed to get them out with difficulty, using a small flat blade screwdriver, but it's much easier with the proper tri-wing bit tool.
Radio Shack sells a set of security bits that includes the special tri-wing bit for around $13. The part number is 910-5247. You can mail-order this from radioshack.com. Alternatively you can mail-order a suitable screwdriver from A. I. Trading Ltd in Hong Kong (click on GBA, then scroll down to Screw Driver). This costs $7 plus $5 shipping. It's a bit expensive for a small screwdriver, but it works well. Delivery is about a week.
Figure 4. GAME BOY® Camera - Front and Back.
After removing the back of the case, the eyeball may be unplugged from the main board. Whether or not you remove the sensor from the eyeball is up to you; the same awkward three-slot screws hold the eyeball case together. You can see from the picture of my Mini Sumo robot that I removed the eyeball case - this was to keep the weight of the robot down.
If you decide to remove the image sensor PCB from the eyeball housing, you will see that the image sensor is mounted on one side of the board (covered by the plastic lens housing), and the 9-way connector is on the other (see Figures 5 and 6).
Figure 5. Layout of Image Sensor Circuit Board - Sensor Side (M64282FP pin numbers shown)
Figure 6. Layout of Image Sensor Circuit Board - Back (wire numbers shown)
This is probably the trickiest step of the whole operation, but with a little care, this can be accomplished successfully.
Refer to Figure 7 and Table 1 to see how the wires need to be connected to the MRM.
You may think of a better way of doing this than I, but I found that cutting off the connector one wire at a time (so that I could keep track of which wire I was connecting), and soldering directly to ribbon-cable worked well. I used fine-bore shrink-wrap tubing to insulate and strengthen the joints, and I made a careful note of the color of the ribbon-cable wire as I connected each one.
Figure 7. Eyeball Connector Pin-out (M64282FP pin numbers shown)
|Wire||M64282FP Pin(s)||Symbol(s)||Function||MRM Pin|
|1||4,13,15||DVDD,AVDD1,AVDD2||+5V Power Supply||Power (+5V)|
|2||1||START||Start Input||Port E Bit 5|
|3||3||SIN||Data Input||Port E Bit 4|
|4||6||LOAD||Data Set Input||Port E Bit 3|
|5||7,9||Xrst,RESET||System+Memory Reset Input||Port E Bit 2|
|6||8||Xck||System Clock Input||Port E Bit 1|
|7||10||READ||Read Image Output||Port E Bit 0|
|8||14||Vout||Analog Signal Output||ADC Port 0|
Table 1. Eyeball to MRM Connections
At the MRM end of the ribbon cable, you will need to solder or crimp the wires to 0.1" female headers which will plug into the MRM. Figure 8 shows the pins on the MRM on which the headers will be inserted. The numbers in the red circles are the port bit numbers (in the case of Port E), and the analog to digital converter channel (in the case of the A/D port). For Power and GND, you are spoiled for choice.
Figure 8. MRM Connections
The code featured in this article leverages the Open Source software libraries of the MotoRobots project (see the Links section at the end of this article for details). The MotoRobots libraries are an attempt to create a powerful, programmer-friendly collection of C functions (and later C++ classes) to make the MRM (and similar controller boards) more accessible to beginners, and more productive for experienced developers. These libraries are distributed under the GNU LGPL.
The GNU m68k cross-development toolchain, suitable for building the code samples in this article, has been packaged into convenient downloadable installation packages for the MotoRobots project (Windows® and Linux® platforms - Intel x86). These are freely available from the downloads page of the MotoRobots project web site.
Listing 1 shows the main program. This will run on any MRM variant (32KB, 512KB, 16MHz, 25MHz). After initializing the processor clock speed and serial port BAUD rate, the program initializes the camera I/O ports. The program then enters a loop, looking for characters received from the serial port. The program will respond to certain characters as follows:
Before capturing the image, the gbcam_shoot( ) function initializes the M64282FP by writing a set of default values to its registers (please refer to the M64282FP data sheet in the Links section at the end of this article for details of these registers). The default settings will do for most simple image capture applications, but if you want to tweak them, you can alter the values of the members of the structure gbcam_registers (not show in Listing 1 - see the source code file gbcam.h) before calling gbcam_exposure( ) or gbcam_shoot( ).
Listing 1. Target Program
The full target source code is packaged here for download in zip format for Windows®, and tar.gz format for Linux®. The packages include the source code in Listing 1, and the dependent MotoRobots libraries. The compiled binary program in S19 format is also provided, ready to load into the MRM.
The MotoRobots libraries contain ready-made linker scripts for the different MRM configurations (these are provided in the startup/68332/mrm directory). The target code provided here is linked with mrm_rom_32k.ld, which is the script for the 32KB version of the MRM that locates the program code in Flash ROM. The 32KB script also works with the 512KB MRM. However, if you have the expanded MRM and you are planning on expanding on the example code provided here, you may wish to edit the Makefile (in examples/68332/mrm/gbcam) to use mrm_rom_512k.ld and rebuild (using make).
If you have the 25MHz version of the MRM, you may wish to edit camera.c and change the line set_processor_speed(CLOCK16MHZ); to read set_processor_speed(CLOCK25MHZ); and then rebuild.
The MotoRobots libraries are being constantly developed, so by the time you read this, the version included here could well be out of date. Please check the MotoRobots web site for the latest version. However, the code provided here has been well tested, and is known to work, so before downloading any later versions, I would recommend that you eliminate any hardware or wiring bugs using the code provided with this article first.
Listing 2 is the program for the host PC. This is written in JavaTM and just requires a JavaTM 2 runtime environment (version 1.2 or later) and an implementation of the JavaTM COMM API to run. I've successfully tested it using the following configurations:
|Platform||SDK||JavaTM COMM API Implementation|
|Windows® 2000 Professional||JavaTM 2 SDK Standard Edition version 1.3.1||JavaTM Communications for Windows® version 2.0|
|Red Hat® Linux® 7.2||JavaTM 2 SDK Standard Edition version 1.3.1||RXTX version 1.4.13|
Both the source code and compiled binaries of the Camera Image Viewer host software are included in this jar archive file.
To extract the source code files from the archive, you can use
jar xf GBCamView.jar GBCamView.java
The file is also an executable jar, meaning that you can run the program by simply invoking
java -jar GBCamView.jar
Listing 2. Host Program
Here are some instructions for installing the JavaTM Communications API on Windows® and Linux® platforms.
|JavaTM COMM API Installation|
This assumes you are using JavaTM 2 SDK 1.2 or later.
In the steps listed below, <jdk> refers to the root directory of your JavaTM SDK installation, so if you
installed JavaTM SDK in
c:\jdk1.3.1_02 replace all reference to <jdk> with c:\jdk1.3.1_02
I must be honest; I had great difficulty in getting Java Communications working in Linux.
After searching the web, the only implementations I found were IBM's COMM API
implementation for their own Java runtime environment, and an open source package called
RXTX. RXTX looks like it has great potential, but I had problems
getting versions 1.4.15 and 1.5.8
working properly. I did, however, manage to get 1.4.13 working, but only when logged in as root.
Note that versions 1.4.x and 1.5.x are fundamentally different from each other, in that 1.4.x
requires you to obtain
Sun's JavaTM Communications for Solaris,
but the 1.5.x version of RXTX theoretically contains everything needed for COMM API. This means
that the installation techniques for these two versions are quite different, but as 1.4.13 is the only
version I was able to successfully install on my Red Hat Linux 7.2 box, that is the one I will
This assumes you are using JavaTM 2 SDK 1.2 or later. In the steps listed below, <jdk> refers to the root directory of your JavaTM SDK installation, so if you installed JavaTM SDK in /usr/java/jdk1.3.1_02 replace all reference to <jdk> with /usr/java/jdk1.3.1_02
Log in as root to perform the remaining steps, and thereafter to run any COMM API programs (including GBCamView and BlackBox).
If you are feeling adventurous, you can build RXTX version 1.4.13 from the source code. Here is the full package: rxtx-1.4-13.tar.gz.
If you're feeling really adventurous and have plenty of time to waste, you could try to get one of the later versions of RXTX working. At the time of this writing, 1.5 was still experimental.
One final point. The serial port devices on some Linux distros (e.g. Red Hat 7.0 - 7.2) are owned by root and have permissions set so that ordinary users cannot open them. This is easy enough to change. To allow all users to access COM1 for example, enter the following (as root):
and for COM2:
Since you will be running the COMM API code as root anyway, this is probably moot, however.
Here are the steps for running the host PC program. This assumes you have downloaded the host program, installed JavaTM and JavaTM Communications, and tested that it works by running the BlackBox example program bundled with JavaTM Communications.
|Running the Host Program|
With the MRM hooked up to the host PC via the serial port, point the camera at a subject, then click the Set Exposure button on the GUI. By metering the brightness and contrast of the scene, this will automatically set the exposure time. This takes around 15 seconds. Try to keep the camera and subject fairly stationary during this exposure-setting time, as the algorithm takes multiple exposures. After setting the exposure time, clicking the Capture button will then take a picture, download it to the PC, and display it. The download takes about 10 seconds. It is not necessary to reset the exposure time between successive image captures, provided the lighting conditions remain similar.
Capturing an image is only the first step in robot vision. For a robot to do something useful with the image, some processing must be done. The details are beyond the scope of this article, but here are some pointers
To see an example of how statistical analysis can be used to identify the lines of a line-maze, have a look at the article "Line Detection Using a Digital Camera" in the Links section at the end of this article.
Although there is no reason why you couldn't connect two GAME BOY® cameras to the MRM, the MRM is not powerful enough to do stereopsis (extracting depth information from two or more images of the same scene) in real time. Something that certainly is feasible though, and that I am working on in fact, is building a simple stereo vision head using an MRM, two image sensors from GAME BOY® cameras and a pair hobby servos (for pan and tilt). This connects to a PC via a full-speed USB interface (the USB interface has to be implemented using external components - the MRM has no inbuilt USB port). The MRM then acts as a slave to the PC, servoing the head, and capturing genlocked stereo images. The PC (which is much more powerful than the MRM) applies a stereopsis algorithm on the captured images in real time to extract depth information from the scene. When (or if) this work ever gets to a point where I have some meaningful code or circuit designs to share, I certainly intend to make it public.
The technique that my mini-sumo robot uses to detect its opponent is to build a histogram of the sum of pixel intensities for each column, resulting in an array of 256 integer values, and look for clusters of sudden changes in intensity value next to each other. Since the background is so sterile in the mini-sumo arena, such changes tend to occur where there is an object. This is fairly primitive, and far from perfect, but since the technique is so simple, it works in real time (several times per second).
Open Computer Vision Library
If you are feeling adventurous, you might want to take a look at the code in the Open Computer Vision software library (see the Links section at the end of this article), and port some of this code to the MRM. This is an Open Source project, started by Intel and hosted by SourceForge. Their philosophy is stated thus:
Aid commercial uses of computer vision in human-computer interface, robotics, monitoring, biometrics and security by providing a free and open infrastructure where the distributed efforts of the vision community can be consolidated and performance optimized.
A quick search of the web will reveal that there are many places you can obtain CMOS camera modules at fairly affordable prices (although admittedly, not at the prices you can get used GAME BOY® Cameras at currently). A nice source of both color and monochrome devices I found based on OmniVision CMOS Image Sensors was from Amazon Electronics. They sell color camera modules with full digital interface and a choice of lens for under $60.
Many of the photographs and illustrations in this article are courtesy of Daniel Herrington and Benjamin Green. Used with permission. Many thanks to both of them.
Please don't hesitate to email me and let me know if you've found any errors or omissions in this article.
Copyright © 2001, Dafydd Walters
GAME BOY® is a registered trademark of Nintendo of America Inc.
JavaTM is a trademark of Sun Microsystems, Inc.
Windows® is a registered trademark of Microsoft Corporation
Linux® is a registered trademark of Linus Torvalds
Red Hat® is a registered trademark of Red Hat, Inc.