Reference/API

Hand-Eye Calibration

class[source]
robokit.helpers.hand_eye_calibration.HandEyeCalibrator(
config: HandEyeCalibratorConfig,
robot: Robot,
camera_intrinsic: Float[torch.Tensor, '*obs 3 3'],
height: int,
width: int,
render_link_names: Optional[List[str]] = None,
target_link_names: Optional[Union[List[str], List[List[str]]]] = None
)

Solve the camera extrinsic from robot silhouette masks and/or depth maps.

Parameters
configHandEyeCalibratorConfigrequired
robotRobotrequired
camera_intrinsicFloat[torch.Tensor, '*obs 3 3']required
heightintrequired
widthintrequired
render_link_namesOptional[List[str]]= None
target_link_namesOptional[Union[List[str], List[List[str]]]]= None
Members
attribute[source]
HandEyeCalibrator.last_q: Optional[torch.Tensor]
method[source]
HandEyeCalibrator.solve(
initial_extrinsic: Float[torch.Tensor, '4 4'],
target_masks: Optional[Float[torch.Tensor, 'n_obs height width']] = None,
target_depths: Optional[Float[torch.Tensor, 'n_obs height width']] = None,
observation_weights: Optional[Float[torch.Tensor, n_obs]] = None,
q: Optional[Float[torch.Tensor, '*batch num_dofs']] = None,
T_mount_base: Optional[Float[torch.Tensor, 'n_obs 4 4']] = None,
keypoint_links: Optional[List[str]] = None,
target_keypoints: Optional[torch.Tensor] = None,
keypoint_weights: Optional[torch.Tensor] = None
) -> Float[torch.Tensor, '4 4']

Solve camera extrinsics from initial_extrinsic.

Lifecycle

  1. Build the solver for new observations, or reuse stored data.
  2. Reset variables and sample camera or joint candidates.
  3. Optionally initialize q from keypoints, then run native population LM.
  4. Retain the best result and restart if mask mismatch remains high.

Data:

  • Any target supplied: replace data and rebuild the solver.
  • No targets supplied: reuse stored data.
Parameters
initial_extrinsicFloat[torch.Tensor, '4 4']required

Initial OpenCV T_camera_base or mounted-camera T_camera_mount, [4,4].

target_masksOptional[Float[torch.Tensor, 'n_obs height width']]= None

Robot silhouette masks [N, H, W] in [0, 1].

target_depthsOptional[Float[torch.Tensor, 'n_obs height width']]= None

Robot-only depth maps [N, H, W] in meters, 0 = invalid.

observation_weightsOptional[Float[torch.Tensor, n_obs]]= None

Optional nonnegative objective weights [N].

qOptional[Float[torch.Tensor, '*batch num_dofs']]= None

Known [N,num_dofs], or one initial [num_dofs] shared across views with optimize_q; None uses midpoints.

T_mount_baseOptional[Float[torch.Tensor, 'n_obs 4 4']]= None

Per-sample mount pose [N, 4, 4] (mounted camera).

keypoint_linksOptional[List[str]]= None

Optional link-frame origins to project, with optimize_q.

target_keypointsOptional[torch.Tensor]= None

Their observed pixels [N,P,2]. NaNs mark missing points.

keypoint_weightsOptional[torch.Tensor]= None

Optional nonnegative confidence weights [N,P].

Returns
Float[torch.Tensor, '4 4']

Optimized extrinsic [4,4] on the input device; last_q stores fitted q and last_var includes model offsets.

class[source]
robokit.helpers.hand_eye_calibration.HandEyeCalibratorConfig(
levels: Sequence[PyramidLevel] = _default_levels(),
terms: Sequence[ResidualTask] = (lambda: [MaskAlignmentNormalEqTask()])(),
solver: PopulationSolverConfig = _default_solver(),
candidate_noise_t: float = 0.35,
candidate_noise_r: float = 0.7,
restart_mismatch: float = 0.5,
max_restarts: int = 4,
seed: int = 0,
optimize_q: bool = False,
optimize_camera: bool = True,
candidate_noise_q: Optional[float] = None,
keypoint_weight: float = 0.03,
keypoint_sigma: float = 8.0,
keypoint_huber_delta: float = 12.0,
keypoint_warmup_iters: int = 24,
candidate_metric: Literal['objective', 'iou'] = 'objective'
)

Eager fitting with pyramid levels, cost terms and candidate sampling.

Example

python
>>> config = HandEyeCalibratorConfig(optimize_q=True, optimize_camera=False)
>>> config.optimize_q, config.optimize_camera, config.candidate_noise_q
(True, False, None)
Parameters
levelsSequence[PyramidLevel]= _default_levels()
termsSequence[ResidualTask]= (lambda: [MaskAlignmentNormalEqTask()])()
solverPopulationSolverConfig= _default_solver()
candidate_noise_tfloat= 0.35
candidate_noise_rfloat= 0.7
restart_mismatchfloat= 0.5
max_restartsint= 4
seedint= 0
optimize_qbool= False
optimize_camerabool= True
candidate_noise_qOptional[float]= None
keypoint_weightfloat= 0.03
keypoint_sigmafloat= 8.0
keypoint_huber_deltafloat= 12.0
keypoint_warmup_itersint= 24
candidate_metricLiteral['objective', 'iou']= 'objective'
Members
attribute[source]
HandEyeCalibratorConfig.levels: Sequence[PyramidLevel] = field(default_factory=_default_levels)

Coarse-to-fine levels, one per solver segment.

attribute[source]
HandEyeCalibratorConfig.terms: Sequence[ResidualTask] = field(default_factory=lambda: [MaskAlignmentNormalEqTask()])

Cost terms; each walks levels itself. MaskAlignmentOffsetTask also calibrates the kinematic model.

attribute[source]
HandEyeCalibratorConfig.solver: PopulationSolverConfig = field(default_factory=_default_solver)

One segment per level; native final ranking returns the best candidate.

attribute[source]
HandEyeCalibratorConfig.candidate_noise_t: float = 0.35

Largest translation (m) of the initial candidates from the initial guess; each draws 0.5-1x of it.

attribute[source]
HandEyeCalibratorConfig.candidate_noise_r: float = 0.7

Largest rotation (rad) of the initial candidates from the initial guess; each draws 0.5-1x of it.

attribute[source]
HandEyeCalibratorConfig.restart_mismatch: float = 0.5

Re-solve with fresh candidates while the final mask mismatch (2 * cost / target mask area) exceeds this.

attribute[source]
HandEyeCalibratorConfig.max_restarts: int = 4

Re-solves allowed after the first; 0 turns restarts off.

attribute[source]
HandEyeCalibratorConfig.seed: int = 0

Seed of the candidate noise.

attribute[source]
HandEyeCalibratorConfig.optimize_q: bool = False

Fit one unknown q vector shared across views; solve(q=...) takes an initial [num_dofs] guess.

attribute[source]
HandEyeCalibratorConfig.optimize_camera: bool = True

Optimize the camera extrinsic; False keeps the supplied extrinsic fixed.

attribute[source]
HandEyeCalibratorConfig.candidate_noise_q: Optional[float] = None

Joint sampling radius in radians/meters; None samples the full finite joint limits. Candidate 0 keeps initial q.

attribute[source]
HandEyeCalibratorConfig.keypoint_weight: float = 0.03

Relative weight of optional 2D link-origin observations.

attribute[source]
HandEyeCalibratorConfig.keypoint_sigma: float = 8.0

Keypoint residual scale in input-image pixels.

attribute[source]
HandEyeCalibratorConfig.keypoint_huber_delta: float = 12.0

Robust-loss transition in input-image pixels.

attribute[source]
HandEyeCalibratorConfig.keypoint_warmup_iters: int = 24

Keypoint-only initialization steps; half the broad samples remain unmodified.

attribute[source]
HandEyeCalibratorConfig.candidate_metric: Literal['objective', 'iou'] = 'objective'

Select by cost or soft IoU; IoU mode uses union-normalized coarse fitting.

class[source]
robokit.helpers.hand_eye_calibration.QposTrajectoryCalibrator(
config: QposTrajectoryCalibratorConfig,
robot: Robot,
camera_intrinsic: torch.Tensor,
height: int,
width: int,
render_link_names: Optional[List[str]] = None
)

Fit unknown q[T,J], optionally refining static cameras[V,4,4] shared over time; temporal penalties apply only to q.

Example (CUDA, robot meshes and observations required):

python
>>> from robokit.helpers.hand_eye_calibration import presets
>>> config = presets.rgb_qpos_trajectory
>>> config.optimize_camera
False
>>> # helper = QposTrajectoryCalibrator(config, robot, K, H, W)
>>> # cameras = helper.solve(cameras, masks, timestamps)
>>> # helper.last_q.shape  # (timestamps, joints)
Parameters
configQposTrajectoryCalibratorConfigrequired
robotRobotrequired
camera_intrinsictorch.Tensorrequired
heightintrequired
widthintrequired
render_link_namesOptional[List[str]]= None
Members
method[source]
QposTrajectoryCalibrator.solve(
initial_extrinsics: torch.Tensor,
target_masks: torch.Tensor,
timestamps: torch.Tensor,
observation_weights: Optional[torch.Tensor] = None,
q: Optional[torch.Tensor] = None,
keypoint_links: Optional[List[str]] = None,
target_keypoints: Optional[torch.Tensor] = None,
keypoint_weights: Optional[torch.Tensor] = None
) -> torch.Tensor

Fit synchronized silhouettes and optional keypoints to a joint trajectory; an empty mask with positive weight constrains absence.

Lifecycle

  1. Validate calibration, timestamps and observations; normalize mask costs by area.
  2. Initialize a coherent trajectory, or use supplied q without frame sampling.
  3. Fit the coupled image, temporal and optional camera system.
  4. Resolve shared joint offsets against source frames and retain joints and cameras.

Outputs

last_frame_q: Independent frame fits, or the supplied q. last_initial_q: Coherent path before coupled refinement, or the supplied q. last_q: Final joint result [T,J] on the input device.

Parameters
initial_extrinsicstorch.Tensorrequired

Static calibrated cameras [V,4,4].

target_maskstorch.Tensorrequired

Silhouettes [T,V,H,W] in [0,1].

timestampstorch.Tensorrequired

Finite, strictly increasing source times [T].

observation_weightsOptional[torch.Tensor]= None

Nonnegative mask weights [T,V]; zero marks a missing view.

qOptional[torch.Tensor]= None

Optional joint trajectory [T,J]; skips frame sampling.

keypoint_linksOptional[List[str]]= None

Link-frame origins [P] observed by the keypoints.

target_keypointsOptional[torch.Tensor]= None

Image coordinates [T,V,P,2]; NaNs mark missing points.

keypoint_weightsOptional[torch.Tensor]= None

Optional confidence weights [T,V,P]; zero excludes a point.

Returns
torch.Tensor

Cameras [V,4,4] on the input device; fixed unless camera refinement is enabled.

class[source]
robokit.helpers.hand_eye_calibration.QposTrajectoryCalibratorConfig(
frame: HandEyeCalibratorConfig = (lambda: HandEyeCalibratorConfig(optimize_q=True, optimize_camera=False))(),
initialization_frames: Optional[int] = None,
initialization_paths: int = 1,
initialization_repairs: int = 0,
initialization_seconds: Optional[float] = None,
levels: Sequence[PyramidLevel] = (lambda: [PyramidLevel(4, 1.0, 5), PyramidLevel(1, 0.0, 5)])(),
solver: PopulationSolverConfig = (lambda: PopulationSolverConfig(stages=[Stage(20, 1, LMOptimizerConfig(lm_lambda=0.001, rho_min=0.001, early_stopping_interval=2)), Stage(20)]))(),
path_weight: float = 0.08,
velocity_weight: float = 0.015,
acceleration_weight: float = 0.02,
optimize_camera: bool = False,
camera_prior_weight: float = 0.0,
camera_prior_t: float = 0.03,
camera_prior_r: float = 0.06
)

Eager frame fitting followed by coupled mask and temporal refinement.

python
>>> config = QposTrajectoryCalibratorConfig(velocity_weight=0.01)
>>> config.optimize_camera, config.velocity_weight
(False, 0.01)
Parameters
frameHandEyeCalibratorConfig= (lambda: HandEyeCalibratorConfig(optimize_q=True, optimize_camera=False))()
initialization_framesOptional[int]= None
initialization_pathsint= 1
initialization_repairsint= 0
initialization_secondsOptional[float]= None
levelsSequence[PyramidLevel]= (lambda: [PyramidLevel(4, 1.0, 5), PyramidLevel(1, 0.0, 5)])()
solverPopulationSolverConfig= (lambda: PopulationSolverConfig(stages=[Stage(20, 1, LMOptimizerConfig(lm_lambda=0.001, rho_min=0.001, early_stopping_interval=2)), Stage(20)]))()
path_weightfloat= 0.08
velocity_weightfloat= 0.015
acceleration_weightfloat= 0.02
optimize_camerabool= False
camera_prior_weightfloat= 0.0
camera_prior_tfloat= 0.03
camera_prior_rfloat= 0.06
Members
attribute[source]
QposTrajectoryCalibratorConfig.frame: HandEyeCalibratorConfig = field(default_factory=lambda: HandEyeCalibratorConfig(optimize_q=True, optimize_camera=False))

Unknown-qpos frame initialization; cameras stay fixed during initialization.

attribute[source]
QposTrajectoryCalibratorConfig.initialization_frames: Optional[int] = None

Limit broad frame searches to this many visible frames; interpolate before fitting every frame. None searches all.

attribute[source]
QposTrajectoryCalibratorConfig.initialization_paths: int = 1

Alternative initial pose branches to rank against every source mask before refinement.

attribute[source]
QposTrajectoryCalibratorConfig.initialization_repairs: int = 0

Additional frame searches at the worst fitted source times; 0 disables repair.

attribute[source]
QposTrajectoryCalibratorConfig.initialization_seconds: Optional[float] = None

Stop starting repairs after this initialization time; the initial searches and ranking still complete.

attribute[source]
QposTrajectoryCalibratorConfig.levels: Sequence[PyramidLevel] = field(default_factory=lambda: [PyramidLevel(4, 1.0, 5), PyramidLevel(1, 0.0, 5)])
attribute[source]
QposTrajectoryCalibratorConfig.solver: PopulationSolverConfig = field(default_factory=lambda: PopulationSolverConfig(stages=[Stage(20, 1, LMOptimizerConfig(lm_lambda=0.001, rho_min=0.001, early_stopping_interval=2)), Stage(20)]))
attribute[source]
QposTrajectoryCalibratorConfig.path_weight: float = 0.08
attribute[source]
QposTrajectoryCalibratorConfig.velocity_weight: float = 0.015
attribute[source]
QposTrajectoryCalibratorConfig.acceleration_weight: float = 0.02
attribute[source]
QposTrajectoryCalibratorConfig.optimize_camera: bool = False

Refine one static extrinsic per view, shared across all frames.

attribute[source]
QposTrajectoryCalibratorConfig.camera_prior_weight: float = 0.0

Prior cost relative to the supplied calibration; 0 disables it.

attribute[source]
QposTrajectoryCalibratorConfig.camera_prior_t: float = 0.03
attribute[source]
QposTrajectoryCalibratorConfig.camera_prior_r: float = 0.06

Translation (m) and rotation (rad) scales of shared camera corrections.