[No, not detecting cheap blobs, but rather blob detection on the cheap.]
Way back in mid-May (seems a very long time ago now), I’d done a bit of exploring using the default Raspberry Pi camera in sight of a possible solution for the DPRG Four Corners Competition. The challenge is to have a robot trace out a large square on the ground marked with orange traffic cones/pylons. Even with well-tuned odometry it’s a difficult trick, as part of the requirements include that the square the robot traces must align with the four markers of the course. You can’t just travel off in any direction, make three 90 degree turns and come back to the start, you have to be able to aim at that first marker accurately. There’s the rub: how to aim at something.
So to address that challenge I’d mounted a Raspberry Pi camera to the front of my KR01 robot and posted a 5mm pink LED as a sentinel three meters in front of the robot. Would it be possible for the robot to aim at that pink LED? The preliminaries from that experiment were largely positive: the robot could at least see the LED at a distance of 3 meters, even in daylight. What I’d managed to achieve was with a deliberately very low resolution image, detection of a blob of pixels indicating a close match to the hue of the pink LED.
The next part would be how to aim the robot at that blob. This post further explores that solution, still just using the Raspberry Pi camera.
Building an LED Beacon
First, one thing about LEDs is that they’re largely directional, a bit like a small laser encased in plastic. Some use translucent rather than transparent plastic to diffuse the light a bit, but if the LED is facing away from the robot it will basically disappear from view, except for perhaps its light shining on something else, which wouldn’t be helpful (see more on floor reflections, below).
As you do, there was this bit of plastic I’d been saving for what, over ten years? that I kept trying to find a use for. It was the diffuser off of a pretty useless LED camp light, useless as it was large, didn’t put out that much light, its plastic case had gone sticky, and I was already carrying a LED headlamp that was smaller and brighter. So I discarded the rest and kept the omnidirectional diffuser. I also had this rather beautiful polished aluminum hub from an old hard drive, and it turns out the plastic diffuser fit exactly onto the hub.
So I wired up a small PC board with three pink LEDs and a potentiometer to control the voltage to the LEDs, and built a small frame out of 3mm black Delrin plastic to hold it and the USB hub providing the 5 volt power supply. Another thing just lying around the house…
I added a tiny bit of cotton wool inside to further diffuse the light, so it would show up on the Pi’s camera as a pink blob. I opened several images taken from the robot in a graphics program and noted the RGB values of various pixels within the pink of the beacon. One of these colors would be the “target color”.
Blob Detection
Now the bright idea I had was that we didn’t really need much in terms of image data to figure out where the pink blob was.
Given my robot is a carpet crawler, we can make some relatively safe assumptions about the environment it will be in, even if I let it out onto the front deck. Lighting and floor covering may vary, but my floors and the front deck are largely horizontal. Since the floor and the robot are mostly level, we can therefore expect to find the beacon near the vertical center of the image, and safely ignore the top and bottom of the image. All we care about is where, horizontally, that blob appears.
I’ll talk more about the Python class I wrote to handle the blob processing below.
I placed the beacon on the rug, plugged it into a power supply and pointed the robot at it. The Raspberry Pi camera’s resolution can be set in software. Set at High Definition (1920×1080) we see the view the robot sees, with the pink beacon exactly one meter away from the camera:
Now, for our purposes there’s no need for that kind of resolution. It’s instead set at what seems to be around the Pi camera’s minimum resolution of 128 x 64 pixels. While irritatingly small for humans, robots really don’t care. Or if they do care, they don’t complain.
The Blob Class
The Python class for the blob detector is just named Blob (and available as with the rest of the code on github). In a nutshell it does the following:
- Configure and start up the Raspberry Pi camera.
- Take a picture as a record, storing it as a JPEG.
- Processing the image by iterating over each row, pixel-by-pixel, by comparing the color distance (hue) of the pixel with the target color (pink). Each pixel’s color distance value is stored in a new array the same size as the image. We enumerate each color distance value and assign a color from a fixed palette, printing a small square character (
0xAA) in that color. We do this row by row, displaying a grid-like image representing color distance. Note that there’s a configuration option to start and stop processing at specified rows, effectively ignoring the top and bottom of the image. - Sum the color distance values of each column, reducing the entire image to a single line array.
- Enumerate the distribution of this array so that there are only ten distinct values, replacing each element with this enumerated value. We also use a low-pass filter to eliminate (set to zero) all elements where there wasn’t enough color match to reach that threshold. Like the color distance image we print this single line after the word “sum” in order to see what this process has turned up.
- Find all the peaks by determining which of the values in this array are the highest of all, given a threshold, and again print this line after the word “peaks”.
- Find the highest peak: If there are multiple peaks within 5% of the image width of each other we assume they’re coming from the same light source, so we average them together, again drawing a line of squares following the word “peak”.
- Return a single value from the function (the index of the highest peak in the array), considered as the center of the light source. If there are too many peaks or they’re too far apart we can’t make any assumptions about the location of the beacon and the function returns -1 as an error value.
The result of this is shown below:
So if the image resolution is 128 x 64, the result will either be a -1 if we can’t determine the location of the beacon, or a value from 0 to 128.
So what to do with this value? If the robot is aiming straight at the beacon the value will be 64. A lower value means the beacon is off to the left (port), higher than 64 the beacon is off to the right (starboard). I’m playing around righ tnow with figuring out what these values mean in terms of turn angle by comparing what 0 and 128 represent when compared against an image taken from the robot of a one meter rule taken at a distance of one meter (see above).
One of the things we’ve talked about in the weekly DPRG videoconferences is the idea that the difference between two PID controller’s velocities as an integral can be used for steering. Or something like that, I can’t pretend to understand the math yet. But it occurs to me that the value returned from the Blob function as a difference from center could be used to steer the robot towards the beacon.
Performance & Reliability
In my earlier tests there was clearly an issue of performance. While the Blob processing time is pretty quick the time the Pi camera was taking to create the image was around 600ms for that 128 x 64 pixel image, with of course more time for larger images. It turns out this was due to a bug in my code: I was creating and warming up the camera for each image. The Pi’s camera can stream images at video speed, at least 30 frames per second, so I’ve got a bit of work to figure out how to grab JPEG images from the camera at speed. So the code posted on github as of this writing is okay for a single image but can’t run at full speed. Until I fix the bug. I’ll update this blog post when I do.
And as to reliability, the pink LED beacon is not very bright. In high ambient light settings it’s more difficult for the robot to discern. Some of the DPRG’s competitions, which are usually held either outdoors or indoors under bright lighting conditions, use a beer can covered in flourescent orange tape. Since the Blob color distance method is designed for hue, if the robot were running in a bright room perhaps an orange beer can would work better than a pink LED beacon. If we tuned that algorithm to also include brightness, perhaps a red LED laser dot might work. All room for experimentation.
I’d noted early on that it was somewhat easy to fool the image processor. My house has a wide variety of colors, rugs, books, all manner of things. That particular pink doesn’t show up much however, and if I threw away the times when multiple peaks showed up, basically gave up when I wasn’t sure, then the reliability was relatively high. If the beacon’s light is reflected on another surface (like my couch) the robot might think that was the beacon so I need to make sure the beacon is not too near another object. I also noted that when placing the beacon on a wood floor the reflection on the floor would show up on camera, but this actually amplified the result, since the reflection will always been directly below the beacon and we’re only concerned with horizontals.
Next Steps
There’s plenty of room to both optimise and improve the Blob class. I’m still getting either too many false positives or abandoning images where the algorithm can’t make out the beacon.
It occurs to me also that we don’t have to train the Blob class to look for pink LEDs or orange cans. It would be relatively trivial to convert it to a line follower by aiming the camera down towards the front of the robot, altering the color distance method to deal solely with brightness, and using the returned value to tell the robot the location of the line.
Maybe next time.