EECE 5554
Robotics Sensing and Navigation

Lecture 14: Sensing with Cameras:
Calibration, Mosaicing

Dr. Xian Li
Assistant Teaching Professor
Department of Electrical and Computer Engineering

Inspiration

Photos from a turning camera stitched into one panorama

Photos from a turning camera, stitched together

A roof camera drives one loop of a parking lot. Each image is placed on the map with its RTK pose.

2
Left image: OpenCV homography tutorial. Right: simulated, the course rig 1.7 m up

Today

  1. Intrinsics and lens distortion
  2. Zhang's method: calibrating with a flat target
  3. Extrinsics and time
  4. Mosaicing: panoramas and ground maps
3

Intrinsics: why the spec sheet is not enough

Part 1 of 4

If this is offby this muchyou get, 10 m away
Focal length f1% (11 px)depth off by 1%: 10 cm
Principal point cx10 pxevery bearing off by 0.5°: 9 cm sideways
Lens distortion, ignored0.5%, our “low distortion” lens5 px at the image corners
A spec sheet gets you within a few percent. Stereo needs a quarter of a pixel (Lecture 13). Every robot camera needs calibrating.
4

Lens distortion

xd = x (1 + k1r2 + k2r4 + k3r6) + [2t1xy + t2(r2 + 2x2)],   and the same for y

x, y: normalized coordinates (pixels times K−1); r: distance from the center. Radial terms k bend lines; tangential terms t come from a lens not quite parallel to the sensor.

In CameraInfo this is distortion_model: plumb_bob, and d = [k1, k2, t1, t2, k3]. Calibration finds them; image_proc removes them.
5
Read more: Camera Calibration (OpenCV). A real example

Wide lenses: the fisheye model

A ray at angle θ from the axis lands at distance r from the center:

pinhole:   r = f tan θ

fisheye (equidistant):   r = f θ

θ30°60°80°89°
tan θ0.581.735.6757.3
θ (rad)0.521.051.401.55

r / f for each model

plumb_bob

The model above. Fine up to about 100° across. Our 72° lens and the C920 webcams use it.

rational_polynomial

8 coefficients, for stronger distortion near the edges.

equidistant (fisheye)

For 150° to 200° lenses. OpenCV has a cv.fisheye module for it.
6

Zhang's method: a flat target

Part 2 of 4

The board is a plane, Z = 0. So each photo of it is a homography, exactly as in Lecture 12:

Hi = λ K [ r1   r2   t ]i

Find the corners (cv.findChessboardCorners, then cv.cornerSubPix to a tenth of a pixel), and the DLT gives one H for each view. The same K is inside all of them.
7

From homographies to K

r1 and r2 are columns of a rotation: at right angles, and the same length. With h1, h2 the columns of H and ω = K−TK−1:

h1T ω h2 = 0

h1T ω h1 = h2T ω h2

Why tilt matters: if every view faces the camera straight on, the equations repeat each other, and f cannot be found.

1Stack the equations

ω is symmetric: 6 numbers. Each view gives 2 equations, so 3 or more views are needed.

2Solve V b = 0 with the SVD

The same trick as the DLT. Then K comes out of ω, and each view's R and t out of K−1H.

3Refine everything

Nonlinear least squares on K, distortion, and every R and t, to make the reprojection error small: the regression lecture again.
8

Judging a calibration: reprojection error

Push every board corner through the calibrated model, and measure how far it lands from where it was found:

RMS = √ mean ‖ xfound − xmodel ‖2

Check (the bonus targets)TargetMust pass
Intrinsics, each camera0.3 px0.5 px
Stereo calibration0.5 px0.8 px
Rectified rows line up0.3 px0.5 px
Small error is needed, not enough: 10 views all in the middle can fit well and still be wrong at the edges.
9

Try it: calibrate a virtual camera

Photos of the board

A virtual camera whose true K differs from the spec sheet, as real ones do. Board: 8 × 6 corners, 108 mm squares, 1.2 to 3 m away. Closed form only, no distortion.

Python demo and notebook

10

Good practice for calibration images

Cover the image

Corners in every part of the frame, most of all the corners of the image, where distortion is largest.

Tilt it

Up to about 45°, left and right, up and down. Flat-on views add little.

Near and far

Across the distances you will use. 20 to 40 good views.

Lock the lens

Focus and zoom fixed, taped or set-screwed. Refocusing changes K.

Flat and rigid target

A print on foam board bends. Measure the squares with calipers.

Sharp images

Hold still, and use even light. Blur and glare move the corners.
The ROS calibrator shows four bars, X, Y, Size, and Skew: move the board until all four are full. A ChArUco board (a checkerboard with markers) still works when part of it is out of view.
11

Calibrating in ROS 2: camera_calibration

calibrate (ROS 2 Jazzy)
# one camera: the classroom webcam
ros2 run v4l2_camera v4l2_camera_node
ros2 run camera_calibration cameracalibrator \
    --size 9x6 --square 0.025 --no-service-check \
    --ros-args -r image:=/image_raw

# the stereo pair on the Mustang: large board,
# 8 x 6 inner corners, 108 mm squares
ros2 run camera_calibration cameracalibrator \
    --size 8x6 --square 0.108 --no-service-check \
    --ros-args -r left:=/stereo/left/image_raw \
    -r right:=/stereo/right/image_raw
  • --size counts inner corners, not squares: an 8 × 6 board has 9 × 7 squares.
  • CALIBRATE when the bars are full. Read the error it prints.
  • SAVE writes /tmp/calibrationdata.tar.gz: the images and a YAML file per camera.
  • The driver loads that YAML from its camera_info_url, and publishes it as CameraInfo with every image.
12

In the classroom: calibrate a webcam

1Start it

Webcam driver, then the calibrator, with your board's size and square.

2Fill the bars

One person moves the board, one watches the bars and the coverage.

3Calibrate and save

Write down the RMS error, the number of views, and fx, cx, cy.

4Compare

Post your numbers on the board. Same webcam model: how close are the K's?
Then make it worse on purpose: calibrate again with only flat-on views in the middle. The error may look fine. Is K the same?
13

Break Time

5 Minutes

Extrinsics: every transform the rig needs

Part 3 of 4

Frame (shared names)ParentFrom
roof_barbase_linkTeam A CAD, tape check
stereo_left_optical_frameroof_barCAD, then calibration
stereo_right_optical_frameroof_barstereo calibration
gnss_antennaroof_barCAD, tape check
Each row is one R and t. Together they let every pixel and every RTK fix be said in base_link.
15

Stereo extrinsics: left to right

With both K's known, photos of the board seen by both cameras at once give R and T from the left camera to the right (cv.stereoCalibrate). Check them:

  • |T| matches the bar: 0.500 m in CAD, to a millimeter or two.
  • R is close to identity: the cameras are nearly parallel.
  • After rectification, matches share a row to 0.3 px.

Bumps

A pothole, or a hand on the bar, can move a camera a fraction of a degree.

Refocusing

Changes K, and so the rectification.

Heat

Sun on a dark roof bends the bar and softens plastic.

Remounting

Every time the rig comes off, check again.
The quick check after any of these: match features in a rectified pair and look at their row difference. Over 0.5 px, recalibrate.
16

Camera to vehicle, and the lever arm

1CAD

Team A's model gives every position and angle on the bar.

2Measure

Tape and laser meter from the rear axle center, with a plumb line to the floor and a level on the bar.

3Check with a target

Put a target on the floor at a measured spot ahead. Does the camera put it there?

A small pitch error moves the ground a lot (camera 1.7 m up)

Pitch error5 m ahead10 m ahead20 m ahead
0.2° (target)6 cm21 cm83 cm
0.5°14 cm53 cm2.1 m
Targets: camera to base_link within 1 cm and 0.2°; antenna within 1 cm.
RTK measures the antenna, not base_link. The lever arm between them is a fixed vector in the car, so its direction on the map turns with the car.
17

Publishing the transforms

quick test
# one static transform from the command line
ros2 run tf2_ros static_transform_publisher \
    --x X_BAR --y 0.0 --z Z_BAR \
    --frame-id base_link --child-frame-id roof_bar

# check the tree
ros2 run tf2_tools view_frames
ros2 run tf2_ros tf2_echo base_link gnss_antenna
stereo_rig.urdf.xacro (excerpt)
<joint name="bar_to_left" type="fixed">
  <parent link="roof_bar"/>
  <child link="stereo_left"/>
  <origin xyz="${xl} ${yl} ${zl}" rpy="0 ${pitch} 0"/>
</joint>
<!-- body (x forward, z up) to optical
     (z forward, x right, y down) -->
<joint name="left_to_optical" type="fixed">
  <parent link="stereo_left"/>
  <child link="stereo_left_optical_frame"/>
  <origin xyz="0 0 0" rpy="${-pi/2} 0 ${-pi/2}"/>
</joint>
For the rig, use a URDF: robot_state_publisher publishes all of it, and the numbers live in one file, next to the transforms table. The body to optical turn is the Lecture 12 rotation.
18

Time: when was each image taken?

Speed10 ms off50 ms off
2 m/s (walking pace in the lot)2 cm10 cm
10 m/s (36 km/h)10 cm50 cm

Position error = speed × time offset. RTK is good to about 2 cm.

Two clocks to line up: left with right (one trigger, Lecture 13), and camera with GNSS.

Where offsets come from

Stamping an image when it arrives, not when it was exposed. USB transfer, driver queues, and GNSS message delay.

How to shrink them

Stamp at exposure (the camera's own timestamp). Keep the laptop clock synced to GNSS time.

How to measure what is left

Drive slowly past a line on the ground, both ways. Camera and RTK should place it in the same spot.
19

Mosaicing: panoramas

Part 4 of 4

Why it works

A camera that only turns gives H = K R K−1 for any scene (Lecture 12). Lectures 12 and 13 give every step.

Wide panoramas

Past about 120° a flat canvas stretches too much, so stitchers warp onto a cylinder or a sphere instead.
20
Read more: High level stitching API (OpenCV). A stitched example

Blending, seams, and ghosts

Feathering

Weight each image less toward its edges, so overlaps fade from one to the next.

Seams

Cut between images where they agree, not through a car or a face.

Exposure

Match the brightness of neighbors first, or the sky shows stripes.

Ghosts

A camera that moves, not just turns, sees near things shift: they show up twice.
Our car moves, so a roof camera is not a panorama camera. For the ground we need something else: the plane, and RTK.
21

A ground mosaic with RTK

pmap = pbase(t) + R(ψ(t)) · pground, in base_link

Where heading comes from

One antenna gives position, not heading. Take heading from the track (successive fixes, while moving), or from the IMU as in Lab 4.

Why it can be sharp

Ground points are exact under the homography, and RTK is good to 2 cm. Everything else is calibration and time.
22

Where the map errors come from

SourceExampleA point 10 m ahead moves
RTK positionfixed solutionabout 2 cm
Heading0.5° offabout 10 cm sideways
Camera pitch0.2° off, or the car brakingabout 21 cm along the road
Time offset50 ms at 4 m/s20 cm along the track
Lever arm forgottenantenna 1 m ahead of base_link1 m, and the other way on the return pass
Not flat groundcurbs, ramps, parked carssmeared, as in Lecture 12
Errors in one direction shift the map. Errors that flip with the driving direction double the lines. Look for doubles.
23

Try it: a map from a driving car

Errors

Drive

Python demo and notebook

24

Calibration day plan (M4)

#StepWhat to doPass whenWho
1Mount the rigBolts checked, pull test by hand, cables clear of the doorsnothing moves when tappedA
2Intrinsics checkLoad the classroom YAML files, take a few new views< 0.5 pxB
3Stereo calibrationLarge board, near and far, both cameras on one trigger< 0.8 px; rows < 0.5 pxB
4Depth testTargets at 3, 5, 10, 15 m in the garage, against a tape measurewithin 2× theoryB, C
5Vehicle and lever armCAD, then tape and plumb line, then a floor target2 cm, 0.5°A, B
6RecordBags with images, camera_info, TF, and GNSS, under open skyverify script passesC
The garage blocks the satellites, so RTK will not fix inside. Steps 1 to 5 happen in the garage of the 225 building; the lever arm check with RTK and the mosaic drive need open sky at [[ OPEN-SKY SPOT ]].
25

Bonus: from the bench to the car

Team A: Mount

  • M3: printed parts fit the bar
  • CAD numbers into the transforms table: the starting point for every transform

Team B: Calibration

  • M3: intrinsics done in the classroom, YAML files in the repo
  • M4: run today's procedure on the mounted rig

Team C: ROS 2

  • M3: one launch file runs both cameras on a laptop
  • M4: load the YAML files, publish the TF tree from the URDF, record
The bags from M4 become the M5 showcase: live stereo depth, and a ground mosaic placed with RTK.
26

Readings (free)

27

Three takeaways

1Calibrate every camera, and judge it by more than one number. Zhang's method turns one homography per view into K and distortion. Tilt, coverage, and a flat target matter.
2Every transform needs a source and a check. CAD, then a measurement, then a target. Small angles become big errors on the ground.
3Mosaics show calibration and timing errors. Pose, lever arm, and time must all match, and doubled lines tell you which one is wrong.
28

Next

Lecture 15: [[ LECTURE 15 TOPIC ]]

For the bonus: M3 on the bench, then M4 on the Mustang.

29