Hand-Eye Calibration
robokit.helpers.hand_eye_calibration solves the camera extrinsic (OpenCV T_camera_base) from robot masks and/or depth maps. It renders the robot with nvdiffrast at N known joint positions q and aligns the renders to your images. No calibration board is needed.
Overview
Quick start
From src/robokit/examples/hand_eye_calibration/01_sim_rgb.py (fixed camera, masks only):
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, presets hec = HandEyeCalibrator(presets.rgb, robot, camera_intrinsic=intrinsic, height=H, width=W) predicted = hec.solve(initial, target_masks=masks, q=q)
import torch
from robokit.helpers.hand_eye_calibration import compute_robot_masks
from robokit.robo import Robot
# A one-joint box robot needs no external assets.
robot = Robot.load(
"""<robot name="demo">
<link name="base"><visual><geometry><box size=".3 .2 .1"/></geometry></visual></link>
<link name="arm"><visual><origin xyz=".2 0 0"/><geometry><box size=".4 .1 .1"/></geometry></visual></link>
<joint name="joint" type="revolute">
<parent link="base"/><child link="arm"/><axis xyz="0 0 1"/>
<limit lower="-3.14" upper="3.14" effort="1" velocity="1"/>
</joint>
</robot>""",
load_meshes=True,
)
H = W = 64
q = torch.tensor([[-0.6], [-0.2], [0.2], [0.6]], device="cuda")
intrinsic = torch.tensor([[80.0, 0, W / 2], [0, 80.0, H / 2], [0, 0, 1]], device="cuda")
initial = torch.eye(4, device="cuda")
initial[2, 3] = 1.5
masks = compute_robot_masks(robot, q, intrinsic, initial, H, W)
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, presets
hec = HandEyeCalibrator(presets.rgb, robot, camera_intrinsic=intrinsic, height=H, width=W)
predicted = hec.solve(initial, target_masks=masks, q=q)
assert predicted.shape == (4, 4)
assert torch.isfinite(predicted).all()masks are [N, H, W] float robot silhouettes. q is [N, num_dofs] in robot.spec.actuated_joint_names order. initial is a rough [4, 4] guess; the default config handles up to ~35 cm / 45 deg of error.
Common variants
Mounted camera. For a camera on a moving mount, such as a wrist camera, pass per-sample T_mount_base. The solver then returns T_camera_mount. See src/robokit/examples/hand_eye_calibration/02_sim_mounted.py:
predicted = hec.solve(initial, target_masks=masks, q=q, T_mount_base=T_mount_base)
import torch
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, compute_robot_masks, presets
from robokit.robo import Robot
# A one-joint box robot needs no external assets.
robot = Robot.load(
"""<robot name="demo">
<link name="base"><visual><geometry><box size=".3 .2 .1"/></geometry></visual></link>
<link name="arm"><visual><origin xyz=".2 0 0"/><geometry><box size=".4 .1 .1"/></geometry></visual></link>
<joint name="joint" type="revolute">
<parent link="base"/><child link="arm"/><axis xyz="0 0 1"/>
<limit lower="-3.14" upper="3.14" effort="1" velocity="1"/>
</joint>
</robot>""",
load_meshes=True,
)
H = W = 64
q = torch.tensor([[-0.6], [-0.2], [0.2], [0.6]], device="cuda")
intrinsic = torch.tensor([[80.0, 0, W / 2], [0, 80.0, H / 2], [0, 0, 1]], device="cuda")
initial = torch.eye(4, device="cuda")
initial[2, 3] = 1.5
masks = compute_robot_masks(robot, q, intrinsic, initial, H, W)
T_mount_base = torch.eye(4, device="cuda").repeat(len(q), 1, 1)
hec = HandEyeCalibrator(presets.rgb, robot, camera_intrinsic=intrinsic, height=H, width=W)
predicted = hec.solve(initial, target_masks=masks, q=q, T_mount_base=T_mount_base)
assert predicted.shape == (4, 4)
assert torch.isfinite(predicted).all()Depth. Use presets.rgbd to add depth on the fine levels, or presets.depth for depth only. For depth only, omit masks and pass robot-only depth, with 0 marking invalid pixels. See src/robokit/examples/hand_eye_calibration/03_sim_depth.py:
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, presets hec = HandEyeCalibrator(presets.depth, robot, camera_intrinsic=intrinsic, height=H, width=W) predicted = hec.solve(initial, target_depths=depths, q=q)
import torch
import nvdiffrast.torch as dr
from robokit.helpers.hand_eye_calibration.calibrator import build_robot_render_data
from robokit.helpers.hand_eye_calibration.mask_alignment_task import _precompute_merged_mesh, _render_batch
from robokit.robo import Robot
# A one-joint box robot needs no external assets.
robot = Robot.load(
"""<robot name="demo">
<link name="base"><visual><geometry><box size=".3 .2 .1"/></geometry></visual></link>
<link name="arm"><visual><origin xyz=".2 0 0"/><geometry><box size=".4 .1 .1"/></geometry></visual></link>
<joint name="joint" type="revolute">
<parent link="base"/><child link="arm"/><axis xyz="0 0 1"/>
<limit lower="-3.14" upper="3.14" effort="1" velocity="1"/>
</joint>
</robot>""",
load_meshes=True,
)
H = W = 64
q = torch.tensor([[-0.6], [-0.2], [0.2], [0.6]], device="cuda")
intrinsic = torch.tensor([[80.0, 0, W / 2], [0, 80.0, H / 2], [0, 0, 1]], device="cuda")
initial = torch.eye(4, device="cuda")
initial[2, 3] = 1.5
transforms, vertices, faces = build_robot_render_data(robot, q)
faces, vertices = _precompute_merged_mesh(vertices, faces, transforms)
with torch.no_grad():
masks, positions = _render_batch(dr.RasterizeCudaContext(), initial, intrinsic, faces, vertices, H, W)
depths = positions[..., 2].clamp(min=0) * (masks > 0.5)
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, presets
hec = HandEyeCalibrator(presets.depth, robot, camera_intrinsic=intrinsic, height=H, width=W)
predicted = hec.solve(initial, target_depths=depths, q=q)
assert predicted.shape == (4, 4)
assert torch.isfinite(predicted).all()For a custom schedule, set use_depth per PyramidLevel. Set depth_weight on the mask task in config.terms.
Model offsets. Encoder zero offsets and an unknown mount also shift the rendered robot. To solve for them with the extrinsic, swap the pyramid's mask task for MaskAlignmentOffsetTask: offset_joints frees joint zeros, and offset_link frees the transform a link hangs from. See 05_sim_offsets.py with --offset joints or --offset mount:
from dataclasses import replace import warp as wp from robokit.opt.var_values import VarKey from robokit.helpers.hand_eye_calibration.mask_alignment_task import MaskAlignmentOffsetTask from robokit.terms.dense.configuration_task import ConfigurationTask rest = ConfigurationTask(rest_q=robot.spec.zero_q, weight=150.0, var_key=VarKey.QPOS_OFFSET) mask_term = MaskAlignmentOffsetTask(offset_joints=list(range(robot.num_actuated_joints))) hec = HandEyeCalibrator(replace(presets.rgb, terms=[mask_term, rest]), robot, intrinsic, H, W) predicted = hec.solve(initial, target_masks=masks, q=q) delta_q = wp.to_torch(hec.last_var.get(VarKey.QPOS_OFFSET).x)[0] # radians, add to measured q
import torch
from robokit.helpers.hand_eye_calibration import HandEyeCalibrator, compute_robot_masks, presets
from robokit.robo import Robot
# A one-joint box robot needs no external assets.
robot = Robot.load(
"""<robot name="demo">
<link name="base"><visual><geometry><box size=".3 .2 .1"/></geometry></visual></link>
<link name="arm"><visual><origin xyz=".2 0 0"/><geometry><box size=".4 .1 .1"/></geometry></visual></link>
<joint name="joint" type="revolute">
<parent link="base"/><child link="arm"/><axis xyz="0 0 1"/>
<limit lower="-3.14" upper="3.14" effort="1" velocity="1"/>
</joint>
</robot>""",
load_meshes=True,
)
H = W = 64
q = torch.tensor([[-0.6], [-0.2], [0.2], [0.6]], device="cuda")
intrinsic = torch.tensor([[80.0, 0, W / 2], [0, 80.0, H / 2], [0, 0, 1]], device="cuda")
initial = torch.eye(4, device="cuda")
initial[2, 3] = 1.5
masks = compute_robot_masks(robot, q, intrinsic, initial, H, W)
from dataclasses import replace
import warp as wp
from robokit.opt.var_values import VarKey
from robokit.helpers.hand_eye_calibration.mask_alignment_task import MaskAlignmentOffsetTask
from robokit.terms.dense.configuration_task import ConfigurationTask
rest = ConfigurationTask(rest_q=robot.spec.zero_q, weight=150.0, var_key=VarKey.QPOS_OFFSET)
mask_term = MaskAlignmentOffsetTask(offset_joints=list(range(robot.num_actuated_joints)))
hec = HandEyeCalibrator(replace(presets.rgb, terms=[mask_term, rest]), robot, intrinsic, H, W)
predicted = hec.solve(initial, target_masks=masks, q=q)
delta_q = wp.to_torch(hec.last_var.get(VarKey.QPOS_OFFSET).x)[0] # radians, add to measured q
assert delta_q.shape == (robot.num_actuated_joints,)
assert torch.isfinite(predicted).all() and torch.isfinite(delta_q).all()config.terms holds every cost term, mask alignment included.
Set offset_link_dofs to free only the mount's translation or rotation. Weakly observable dofs, such as roll about a round wrist flange, drift; the ConfigurationTask holds them near nominal.
Real hardware. src/robokit/examples/hand_eye_calibration/04_real_xarm.py runs the full pipeline on an xArm + RealSense. It captures images at several joint configurations, segments the arm with SAM2, takes an initial guess dragged in viser, then solves. For larger initial errors, raise candidate_noise_t/r, num_candidates and iterations, as the example does.
Unknown frame qpos. Fit q from silhouette views of one frame.
| Preset | Optimized variables |
|---|---|
presets.rgb | Camera; q is supplied |
presets.rgb_qpos | Qpos; camera is fixed |
presets.rgb_qpos_camera | Qpos and camera |
Pass target_masks[V,H,W] and an optional initial q[J]. Read the result from last_q[J]. Omit q to search the full joint limits.
Frame and trajectory comparison.
Joint trajectories (experimental). QposTrajectoryCalibrator fits a smooth last_q[T,J] from masks[T,V,H,W] and timestamps[T]. Use presets.rgb_qpos_trajectory for fixed cameras, or presets.rgb_qpos_trajectory_camera to also refine static cameras.
Both APIs also take image keypoints through keypoint_links and target_keypoints. NaNs mark missing points.