EECE 5554
Lecture 13: Sensing with Cameras:
Dr. Xian Li
Welcome back. Last time, one camera and one plane gave us a homography. Today we add a second camera. We will find matching points between two images automatically, throw out the wrong matches, and turn the matches into depth.
Inspiration
The left camera of a pair on our Mustang's roof
What the pair measures: depth for every pixel near to far
2
Simulated scene: the course rig, 1.7 m up, 5 mm lens, two cameras 0.5 m apart
Here is a simulated view from the left camera of a stereo pair on our Mustang's roof. One camera sees a red car, a blue car, a person, and a wall far away. But from this picture alone, it cannot tell how far any of them are. A small car close by and a big car far away can look the same.
Add a second camera half a meter to the side, and you get this: a distance for every pixel. Warm colors are near, cool colors are far. The road gets farther toward the horizon, and the cars stand out as solid blocks. By the end of today, you will know how this picture is made, and how accurate it is.
Today
Features: corners, descriptors, matching RANSAC: fitting when matches are wrong Two views: epipolar lines and stereo depth Designing the rig: baseline, sync, ROS 2
3
Today has four parts.
First, features: how a computer finds points it can find again in another image, and how it matches them.
Second, RANSAC: how to fit a model when many of those matches are wrong.
Third, two views: epipolar lines, rectification, and how disparity becomes depth.
And last, designing our rig: how far apart to put the cameras, how to keep them in sync, and how ROS 2 runs the whole pipeline.
From four clicks to depth
Lecture 12 You clicked four corners of a parking stall. The DLT turned four point pairs into a homography H.
Today, part 1 The computer finds hundreds of point pairs by itself. Some of them are wrong, and RANSAC copes with that.
Today, part 2 Two cameras that see the same point from two places give its depth, for every pixel, many times a second.
The same idea runs through all of it: find the same point in two images, then use geometry.
4
Let's connect to last time. In Lecture 12, you clicked four corners of a parking stall, and the DLT turned four point pairs into a homography.
Today, the computer finds the point pairs by itself, hundreds of them. Some will be wrong, and we need a method that does not care.
Then we use point pairs from two cameras at once, and get depth for every pixel.
One idea runs through all of it: find the same point in two images, then let geometry do the rest.
Features: what makes a good one Part 1 of 4
Flat no change in any direction Edge changes across it, not along it Corner changes in every direction
A good feature is a patch you can find again, in exactly one place, in the next image.
5
Part one: features. Imagine a small window on the image, and slide it a little. What changes?
On a flat patch, nothing changes, so you could never say where the patch went. On an edge, the patch changes when you move across the edge, but not along it. That is the aperture problem: along an edge, every position looks the same.
At a corner, any move changes what you see. So a corner can be found again in one place only. That is what we want.
So a good feature is a patch that you can find again, in exactly one place, in the next image.
Harris corners
Slide the window by (Δu, Δv). The change in the patch is
E ≈ [Δu Δv] M [Δu Δv]T
M = Σ
image gradients, summed over the window R = det M − k (trace M)2 , k ≈ 0.04 to 0.06
The two eigenvalues of M say how much the patch changes in its strongest and weakest directions. A corner needs both to be large.
λ1 λ2
flat
edge
edge
corner
both large: R > 0
6
Chris Harris and Mike Stephens turned that picture into math in 1988. Slide the window by a small amount, and the change in the patch is a quadratic form with a two by two matrix M. M is built from the image gradients, I x and I y, summed over the window.
The two eigenvalues of M tell the story. Both small: a flat patch. One large and one small: an edge. Both large: a corner.
Computing eigenvalues at every pixel is slow, so Harris uses a shortcut, the score R: the determinant minus a small k times the trace squared. R is large and positive only at corners. Keep the pixels where R is a local maximum, and those are your corners.
Fast features: FAST and ORB
16 1 2 15 3 14 4 13 5 12 6 11 7 10 9 8 p yellow: 9 in a row brighter than p
FAST : is it a corner?
ORB : a fingerprint for each corner
Comparing fingerprints The Hamming distance: count the bits that differ (XOR, then count). A modern CPU does that in one instruction.
7
Rosten and Drummond 2006 (FAST); Rublee et al. 2011 (ORB). Read more:
ORB (OpenCV) Harris is good, but robots want speed. FAST, from Edward Rosten and Tom Drummond, asks a simpler question. Look at the 16 pixels on a small circle around a pixel p.
If nine of them in a row are all brighter than p, or all darker, by some threshold, call p a corner. Most pixels fail after checking two or three of the circle, so FAST can test a whole image very quickly.
Finding a corner is half the job. To match it, we need a fingerprint, called a descriptor. ORB, from OpenCV Labs in 2011, adds an orientation to FAST, then compares the brightness of 256 pairs of pixels around the corner. Each comparison is one bit, so the whole descriptor is 32 bytes. ORB also searches at eight image sizes, so it handles scale.
To compare two descriptors, count the bits that differ. That is the Hamming distance, and a computer does it in one instruction. That is why ORB is a common first choice on robots.
SIFT, and how they compare
SIFT (Lowe, 2004) finds blobs as peaks of a difference of Gaussians across many blur levels, then describes each with 128 numbers: gradient directions in a 4 × 4 grid of cells, 8 directions each.
Method Finds Descriptor Scale and rotation Speed
Harris corners none rotation only fast FAST corners none neither fastest ORB FAST corners at 8 sizes 256 bits, Hamming distance both fast SIFT blobs across scales 128 numbers, Euclidean distance both, very robust slower
On a car at 10 to 30 frames a second, start with ORB. Use SIFT when accuracy matters more than speed, as in mapping offline.
8
The classic descriptor is SIFT, from David Lowe. It finds blobs, not just corners, by blurring the image at many levels and looking for peaks in the difference between levels. Then it describes each blob with 128 numbers: a histogram of gradient directions in a four by four grid around it.
Here is how the four compare. Harris and FAST only find corners and have no descriptor. ORB is fast and handles scale and rotation. SIFT is the most robust, but slower. Its patent ran out in 2020, so it is now in standard OpenCV.
So on a moving car, at ten to thirty frames a second, start with ORB. Reach for SIFT when accuracy matters more than speed, for example when you build a map offline from recorded data.
Matching and the ratio test
Nearest neighbor For each descriptor in image 1, find the closest one in image 2.
Ratio test (Lowe) Keep the match only if the best distance is clearly smaller than the second best: d1 < 0.75 d2 . On a fence or tiled floor, both are close, so the match is dropped.
Still some wrong Even after the test, 10 to 50 percent of matches can be wrong. That is the next part.
See ORB matches on real images
features_ransac.py Copy
import cv2 as cv
import numpy as np
img1 = cv.imread('left.png', cv.IMREAD_GRAYSCALE)
img2 = cv.imread('right.png', cv.IMREAD_GRAYSCALE)
orb = cv.ORB_create(nfeatures=2000)
k1, d1 = orb.detectAndCompute(img1, None)
k2, d2 = orb.detectAndCompute(img2, None)
bf = cv.BFMatcher(cv.NORM_HAMMING)
pairs = bf.knnMatch(d1, d2, k=2) # best two for each
good = [m for m, n in pairs if m.distance < 0.75 * n.distance]
src = np.float32([k1[m.queryIdx].pt for m in good])
dst = np.float32([k2[m.trainIdx].pt for m in good])
H, mask = cv.findHomography(src, dst, cv.RANSAC, 3.0)
print(len(good), 'matches,', int(mask.sum()), 'inliers')
9
Now matching. For each descriptor in image one, find the closest descriptor in image two. With ORB, closest means the smallest Hamming distance.
But the closest is not always right. David Lowe's ratio test helps: keep a match only if the best distance is clearly smaller than the second best, say less than 0.75 times. On a fence or a tiled floor, many patches look alike, the two distances are close, and the match is thrown out. That is what we want: a doubtful match is worse than none.
Even after the test, ten to fifty percent of the matches can still be wrong. Keep that number in mind.
Here is the whole thing in OpenCV: ORB features, a brute force matcher with the Hamming norm, the two nearest neighbors, the ratio test, and then one line that fits a homography with RANSAC. The last line is the Lecture 12 homography, now from automatic matches. Let's see what RANSAC is.
RANSAC: fitting when matches are wrong Part 2 of 4
least squares
RANSAC
Ten points on a line (blue), and three wrong ones (red).
Least squares listens to every point equally. A far-off point pulls hardest, because its error is squared.
RANSAC (Fischler and Bolles, 1981) ignores points that do not agree. The line goes through the blue points only.
10
Part two: RANSAC. Here are ten good points on a line, in blue, and three wrong ones, in red. Fit a line by least squares, as in the regression lecture, and you get the red line. It misses all of them. Least squares listens to every point equally, and a far-off point pulls the hardest, because its error is squared.
RANSAC gets the blue line. It was published by Martin Fischler and Robert Bolles in 1981, and it is still everywhere in robotics. The idea: ignore the points that do not agree.
The RANSAC loop
1 Sample
2 Fit
3 Count
4 Repeat
Why it works: one sample with no wrong points is enough. And it does not matter how wrong the wrong points are.
The threshold is the one number you must choose: about 2 to 3 times the pixel noise of your matches.
11
Here is the RANSAC loop, in four steps.
One: sample. Pick the fewest points the model needs. Two points for a line, four point pairs for a homography.
Two: fit the model to just those points. For a homography, that is the DLT from last lecture.
Three: count the inliers, the points that agree with this model within a small threshold, such as three pixels.
Four: repeat many times, and keep the model with the most inliers. At the end, refit it to all of its inliers, which averages out the noise.
Why does this work? You only need one lucky sample with no wrong points in it. And unlike least squares, a point that is wildly wrong does no more harm than one that is a little wrong.
The one number you choose is the threshold. A good rule is two to three times the pixel noise of your matches.
How many tries?
With a share e of wrong matches and s points per sample, a sample is clean with probability (1 − e)s . To get at least one clean sample with probability p:
N = log(1 − p) / log(1 − (1 − e)s )
p = 0.99 here. Real code updates e as it goes: once it finds many inliers, it stops early.
Model s e = 10% 30% 50% 70%
Line 2 3 7 17 49 Homography 4 5 17 72 567 Fundamental 8 9 78 1177 70188
Small samples win. Each extra point in the sample multiplies the tries. That is why solvers that need fewer points matter.
12
How many tries does RANSAC need? If a share e of the matches is wrong, a sample of s points is completely clean with probability one minus e, to the power s. Solve for the number of tries N that gives at least one clean sample, with 99 percent confidence, and you get this formula.
Here are the numbers. For a homography with four pairs, 30 percent wrong matches needs only 17 tries, and 50 percent needs 72. That is nothing for a computer. But for a model that needs eight points, 50 percent wrong takes more than a thousand tries, and 70 percent takes seventy thousand.
So small samples win. Every extra point in the sample multiplies the tries. That is why people work hard on solvers that need fewer points, like the five point algorithm you will see in a moment.
Try it: matches and RANSAC
13
Try it. Here are sixty matches between two views of a flat scene, so one homography relates them. Green lines are right matches, red lines are wrong ones. The dashed frame shows where image one truly lands in image two.
Press least squares, and the DLT uses every match. With 30 percent wrong matches, the solid frame lands far from the dashed one, off by hundreds of pixels.
Now press RANSAC. It finds the inliers and lands almost on the dashed frame, after only a few dozen tries. Push the wrong share up to 70 percent and try again: it still works, but the tries go up. Add pixel noise and watch the error grow slowly. The demo page has the same code in Python, with a notebook.
Break Time
5 Minutes
Let's take a five minute break. When we come back: two cameras, and depth.
Two views and stereo depth Part 3 of 4
left camera
X? X?
right camera
X
baseline B
One pixel in one camera is a ray: the point X could be anywhere on it. That is the depth we lost in Lecture 12.
A second camera sees the same point along a second ray. Where the rays cross is X: triangulation .
15
Part three: two views. One pixel in one camera tells you a direction, a ray. The point could be anywhere on that ray, near or far. That is exactly the depth we lost when the pinhole divided by Z.
Add a second camera, some distance away. It sees the same point along a second ray. Where the two rays cross is the point. That is triangulation, and the distance between the cameras is the baseline, B.
Epipolar geometry
epipolar plane image 1 image 2
O
O′
X
x
baseline
e
e′ l′
x′
X, O, and O′ make a plane: the epipolar plane .
Each camera sees the other camera's center at its epipole , e and e′.
The plane cuts image 2 in a line, l′. The match x′ must be on it: the epipolar line .
Search one line, not the whole image. Faster, and far fewer wrong matches.
The picture in OpenCV
16
Now let's see what the second camera knows before it even looks. Here are the two cameras, O and O prime, a point X, and its image x in the left camera.
The point X and the two camera centers make a plane, the epipolar plane. The ray from the left camera lies in this plane, wherever X is on it.
The line between the cameras is the baseline. Where it pierces each image is called the epipole, e and e prime: the spot where each camera sees the other one.
The plane cuts the right image in a line, l prime. Wherever X is on the ray, its image in the right camera must fall on that line. That is the epipolar line.
So to find the match of x, search one line, not the whole image. That is faster, and it rules out most wrong matches before they happen.
The algebra: E and F
x′T F x = 0
For every right match, in pixels. F x is the epipolar line l′ in the right image.
Essential matrix E = [t]× R The same rule in normalized coordinates (pixels times K−1 ). It holds the rotation R and the direction of t. 5 free numbers: the length of t cannot be seen. Found from 5 points (Nistér) inside RANSAC.
Fundamental matrix F = K′−T E K−1 E plus both cameras' intrinsics, for raw pixels. 7 free numbers, rank 2. Found from 8 points (Hartley's normalized algorithm) inside RANSAC.
For our rig we do not estimate R and t from every frame: we calibrate them once (Lecture 14), then rectify.
17
The algebra behind the picture fits in one line: x prime transposed, times F, times x, equals zero, for every correct match. F times x is the epipolar line in the right image.
If you first remove the intrinsics, by multiplying pixels by K inverse, the same rule uses the essential matrix, E, which is the cross product matrix of t, times R. It contains the rotation between the cameras and the direction of the translation, but not its length. From one camera's images alone, you cannot tell a small scene close by from a big scene far away. E can be found from just five matches, with David Nistér's five point algorithm, inside RANSAC.
F is E with both cameras' intrinsics folded in, so it works on raw pixels. It has seven free numbers and is found from eight matches, with Richard Hartley's normalized algorithm, again inside RANSAC. Remember the table: eight points means many more tries.
On our rig, we will not estimate R and t from every frame. The cameras are bolted to a bar, so we calibrate R and t once, in Lecture 14, and then use them to rectify the images.
Rectification: matches on the same row
before: epipolar lines slant
xL xR
after: every epipolar line is a row
What it does Turns each image a little, in software, as if the two cameras were exactly side by side and parallel. A match now differs only in its column: the disparity d = xL − xR .
Where it lives in ROS 2 In CameraInfo from Lecture 12: r is this turn for each camera, and p is the new projection. The right camera's p holds Tx = −f B, so the baseline travels with every image.
18
Real cameras are never exactly parallel, so the epipolar lines slant, each one differently.
Rectification fixes that in software. It turns each image a little, as if the two cameras were exactly side by side and pointing the same way. After that, every epipolar line is a row, the same row in both images. A match now differs only in its column. That difference is the disparity, d: x left minus x right.
You have already seen where this lives in ROS 2. In the CameraInfo message from Lecture 12, r is this small rotation for each camera, and p is the new projection matrix. The right camera's p even holds the baseline, as minus f times B. So every image carries what it needs to compute depth.
Depth from disparity
X, depth Z
xL xR
B
f
Similar triangles, as in the pinhole model:
Z = f B / d
Z 3 m 5 m 10 m 15 m 20 m 40 m d 185 111 56 37 28 14
pixels, for f = 1111 px and B = 0.5 m
Near things shift a lot, far things barely move. At 40 m the whole depth hangs on 14 pixels.
19
Now the payoff. After rectification, the two cameras look like this: side by side, a baseline B apart, both with focal length f. A point at depth Z lands at x left in one image and x right in the other.
Similar triangles, the same argument as the pinhole model, give the most important equation of today: Z equals f times B, divided by the disparity d. Depth is inversely proportional to disparity.
For our rig, with f of 1111 pixels and a half meter baseline, a point 3 meters away shifts 185 pixels between the images. At 10 meters, 56 pixels. At 40 meters, only 14.
So near things shift a lot and far things barely move. At 40 meters, the whole depth measurement hangs on 14 pixels. Keep that in mind for the design part.
Finding d for every pixel
left row right row, same row after rectification slide left by d = 0, 1, 2, ...
d cost
best d
Block matching (cv.StereoBM) Compare a small window at each shift, by the sum of absolute differences (SAD), and keep the cheapest. Fast, and noisy.
Semi-global matching (cv.StereoSGBM) Hirschmüller, 2005. Add a cost for jumps between neighbors, along 8 or more directions. Slower, much cleaner: the usual choice.
Knobs you will set window size, disparity range, uniqueness ratio, speckle filter.
20
So how do we find d for every pixel? Take a small window around a pixel in the left row. Because the images are rectified, its match is on the same row in the right image. Slide a window along that row, one pixel at a time, and compute a cost at each shift.
The simplest cost is the sum of absolute differences, SAD. Keep the shift with the lowest cost. That is block matching, StereoBM in OpenCV. It is fast, but noisy.
Semi-global matching, from Heiko Hirschmüller in 2005, adds a penalty when the disparity jumps between neighboring pixels, and adds up those costs along eight or more directions. It is slower, but much cleaner, and it is what most real systems use. In OpenCV it is StereoSGBM.
Either way, you will set a few knobs: the window size, the disparity range to search, a uniqueness ratio, and a speckle filter that removes small islands of bad values. Let's try the window size.
Try it: disparity by block matching
disparity: near to far ; dark = no match
Python demo and notebook
21
Try it. Here is a small synthetic stereo pair: a textured wall far away, a plain painted wall on the left, a box close by in the middle, and a fence with a repeating pattern on the right. The big picture is the disparity that block matching finds.
Start with a small window, five by five. The textured areas come out right, with some speckle. The plain wall is a mess, because every window there looks the same, and only camera noise is left to match.
Now make the window bigger. The speckle goes away, but the edges of the box get fat and blurry, because a big window mixes the near box with the far wall behind it. That is the trade between noise and detail. The plain wall stays wrong whatever you do: there is nothing there to match. And the fence: with a small window it fails, because its stripes repeat every eight pixels and several shifts match equally well. A bigger window reaches the ends of the fence and helps.
Where stereo fails
Plain surfaces White walls, sky, a wet road at night: no texture, so every shift costs the same.
Repeated patterns Fences, tiles, parking lines: several shifts match equally well.
Occlusion Beside every near object, a strip of background that only one camera sees.
Shine and glare Reflections in windows and car paint move with the camera, not with the surface.
Good matchers say "I don't know" instead of guessing: the uniqueness ratio, a texture threshold, and a left to right consistency check mark those pixels invalid.
22
Stereo fails in four typical places.
Plain surfaces: white walls, the sky, a wet road at night. With no texture, every shift costs the same.
Repeated patterns: fences, tiles, even the parking lines in our garage. Several shifts match equally well.
Occlusion: next to every near object there is a strip of background that only one camera can see. It has no match at all.
And shine and glare. A reflection in a window or in car paint moves with the camera, not with the surface, so its depth is wrong.
Good matchers do not guess in these places. A uniqueness ratio, a texture threshold, and a check that left to right and right to left give the same answer, all mark those pixels as invalid. A hole in the depth image is honest. A wrong depth is dangerous.
Design the rig: depth error Part 4 of 4
Matching is off by a fraction of a pixel, δd. Differentiate Z = f B / d:
δZ = Z2 δd / (f B)
Twice as far: four times the error.
Twice the baseline, or twice the focal length: half the error.
0 20 40 60 80 100 0 10 20 30 40 B = 0.3 m B = 0.5 m B = 0.8 m
distance Z (m)
depth error (cm)
23
Part four: designing the rig. Matching is never perfect. Even a good matcher is off by a fraction of a pixel, call it delta d. Differentiate Z equals f B over d, and you get the error: delta Z equals Z squared, times delta d, divided by f times B.
The Z squared is the key. Twice as far means four times the error. Here it is for a quarter pixel of matching error and our 5 millimeter lens. With a 30 centimeter baseline, the error passes 20 centimeters before 20 meters.
Two numbers fight back: the baseline and the focal length. Double either one, and the error halves. Our lens is fixed, so the baseline is the decision.
Picking the baseline
Baseline B at 3 m 5 m 10 m 15 m 20 m 40 m Closest depth
0.12 m 1.7 cm 4.7 cm 19 cm 42 cm 75 cm 3.0 m 0.5 m 0.30 m 0.7 cm 1.9 cm 7.5 cm 17 cm 30 cm 1.2 m 1.3 m 0.50 m 0.4 cm 1.1 cm 4.5 cm 10 cm 18 cm 72 cm 2.2 m 0.80 m 0.3 cm 0.7 cm 2.8 cm 6.3 cm 11 cm 45 cm 3.5 m 1.20 m 0.2 cm 0.5 cm 1.9 cm 4.2 cm 7.5 cm 30 cm 5.2 m
f = 1111 px (5 mm lens), matching error 0.25 px, search of 256 px. Closest depth = f B / 256.
Wider helps far away Error shrinks with B, everywhere.
Wider hurts up close The closest depth grows: at 0.8 m, the 3 m garage test is out of range.
Wider is harder The views differ more, so matching gets harder, and a long bar must not bend.
24
Here is the table that the design review is about. Each row is a baseline, each column a distance, and each cell is the depth error, for our lens and a quarter pixel of matching error. The last column is the closest depth you can still measure, with a disparity search of 256 pixels.
Read it two ways. A wider baseline helps everywhere far away: at 15 meters, half a meter gives 10 centimeters, and 1.2 meters gives 4. But a wider baseline also pushes the closest depth out. At 0.8 meters, anything nearer than three and a half meters has more than 256 pixels of disparity, so our 3 meter garage test falls out of range. And wider views look less alike, so matching gets harder, and a long bar on a moving car must not bend. Half a meter is a good starting point. Team A makes the call.
Try it: pick a baseline
25
Try it. This is the tool Team A brings to the design review. On the left, set the lens, the baseline, the matching error, and the disparity search. Below that, set your requirement: a range, and the accuracy you need there.
The curve is the depth error against distance. The dot is your requirement: green if you meet it, red if not. The red band on the left is too close to measure.
Start at half a meter, and ask for 12 centimeters at 15 meters. It meets it, with a little to spare. Now ask for 10 centimeters at 25 meters, and you need about 1.4 meters of baseline. But then the closest usable distance moves out past 6 meters. You cannot have both from one pair. That is the trade, and now you can put a number on it.
Keeping the pair in sync
left right trigger
shared trigger: both expose together gap
free running: a different gap every time
Global shutter Every row at once (Lecture 12). Our Blackfly S cameras have it, so each image is one instant.
One trigger for both A pulse into both cameras' GPIO input starts both exposures. In ROS 2: spinnaker_synchronized_camera_driver.
What a 10 ms gap costs Turning at 20°/s, the right camera has turned 0.2° more: about 4 px of false disparity, 16 times the matching error.
26
Our numbers: f = 1111 px, so 0.2° = 0.0035 rad × 1111 ≈ 3.9 px
Stereo assumes the two images are taken at the same instant. On a moving car, that takes work. On the left, a shared trigger: one electrical pulse starts both exposures together.
On the right, two cameras running free. Each takes pictures on its own clock, so there is a gap between them, and the gap changes every frame.
Three things keep us safe. First, a global shutter, from Lecture 12. Our Blackfly S cameras have one, so each image is a single instant.
Second, one trigger for both. A pulse into both cameras' GPIO input starts the exposures together, and the ROS 2 driver for this pair is the spinnaker synchronized camera driver.
Third, know what a gap costs. If the car turns at 20 degrees a second, a 10 millisecond gap means the right camera has turned 0.2 degrees more. With our lens, that is about 4 pixels of false disparity, sixteen times our quarter pixel matching error. Software sync is not enough for a turning car.
Stereo in ROS 2: stereo_image_proc
left camera driver
right camera driver
rectify_node image_proc, each side
disparity_node
point_cloud_node
rviz2
image_rect disparity points2 stereo pipeline (ROS 2 Jazzy) Copy
# all of it at once (see its arguments with --show-args):
ros2 launch stereo_image_proc stereo_image_proc.launch.py --show-args
# or node by node, in the shared /stereo namespace:
ros2 run image_proc rectify_node --ros-args \
-r __ns:=/stereo/left -r image:=image_raw
ros2 run image_proc rectify_node --ros-args \
-r __ns:=/stereo/right -r image:=image_raw
ros2 run stereo_image_proc disparity_node --ros-args -r __ns:=/stereo \
-p stereo_algorithm:=1 -p disparity_range:=256 \
-p correlation_window_size:=9
ros2 run stereo_image_proc point_cloud_node --ros-args -r __ns:=/stereo \
-r left/image_rect_color:=left/image_rect
Inputs: the shared names, /stereo/left/image_raw , camera_info, and the same for right.
Outputs: /stereo/disparity and /stereo/points2 , a PointCloud2.
stereo_algorithm: 0 is block matching, 1 is SGBM.
approximate_sync stays off: our pairs share a trigger, so their stamps match exactly.
27
Finally, ROS 2. You don't write the matcher yourself. The image pipeline does it. The two camera drivers publish raw images and camera info. A rectify node from image proc rectifies each side. The disparity node finds d for every pixel, and the point cloud node turns that into 3D points you can see in RViz.
Here are the commands. One launch file starts all of it, and the show args flag lists its options. Or run it node by node in the shared slash stereo namespace. Set stereo algorithm to one for semi-global matching, and the disparity range to 256 pixels, from our table.
The inputs and outputs use the shared names that Team C agreed on: slash stereo slash left slash image raw, and so on, in, and slash stereo slash disparity and points two, out. One more parameter, approximate sync, stays off. Because our pair shares a trigger, the two time stamps match exactly, and that is the best check that the trigger works.
Bonus check-in: M2 design review
Team A: Mount Requirements from M1: range, accuracy, field of view The baseline, picked with the table and the demo CAD version 1: bar length, camera pitch, antenna spot Team B: Calibration Target size: square ≥ npx × Z / f. For 15 px at 10 m: 13.5 cm squares Test distances 3, 5, 10, 15 m and the error each should show Part 1 done with two webcams Team C: ROS 2 The node graph, with the shared names The sync plan: trigger wiring and the driver A recorded bag run through stereo_image_proc
Bring numbers, not opinions. Every choice should point to a row in today's tables.
28
Now the bonus. The next milestone is M2, the design review, and today gave each team what it needs.
Team A brings the requirements from M1, the baseline you picked with today's table and demo, and version one of the CAD: the bar length, the camera pitch, and where the antenna goes.
Team B sizes the calibration target. A square should cover enough pixels at the far test distance: the square size is the pixels you need, times the distance, divided by f. For 15 pixels at 10 meters, that is 13 and a half centimeter squares. Also bring the four test distances and the depth error each one should show, and your webcam practice from Part 1.
Team C brings the node graph with the shared names, the sync plan, meaning the trigger wiring and the driver, and a recorded bag that you have run through stereo image proc.
Bring numbers, not opinions. Every choice should point to a row in today's tables.
Readings (free)
29
Here are free readings. Shree Nayar's short videos are the gentlest start. The three OpenCV tutorials have code you can run today. Szeliski's chapters seven and twelve go deeper. And the three demos from today are on the course site, each with a Python script and a notebook.
Three takeaways
1 Features turn pixels into matches. Detect, describe, match, and use the ratio test. Some matches will still be wrong.
2 RANSAC fits through the wrong ones. Small samples, count the inliers, refit. The tries grow fast with the sample size.
3 Stereo depth is Z = f B / d, and its error grows with Z2 . Rectify, sync, and choose the baseline from the range you need.
30
Three takeaways. First, features turn pixels into matches: detect, describe, match, and use the ratio test. Some matches will still be wrong.
Second, RANSAC fits a model through the wrong matches: sample the fewest points, count the inliers, refit. The number of tries grows fast with the sample size.
Third, stereo depth is f B over d, and its error grows with the distance squared. Rectify the images, keep the pair in sync, and choose the baseline from the range you need.
Next Lecture 14: Sensing with Cameras: Calibration, Mosaicing
How to measure K, R, and t for real, and how to stitch images into a map with RTK.
31
Next time, Lecture 14: calibration and mosaicing. We will measure K, R, and t for real, on the car, and stitch images into a map placed with RTK. See you then.
