Hand Retargeting
Retarget point streams; reference_origin="wrist" centers floating hands internally, but I/O stays world-frame.
Temporal cost:
- First frame after warmup/reset: none.
- Later frames: smooth against the previous solution, independently of the initial guess.
Example
>>> from robokit.robo.robot_spec import RobotSpec
>>> from robokit.utils.ee_coord_utils import MANOPTH_HAND_COORD_SPEC as coords
>>> from robokit.opt.population_solver import PopulationSolverConfig, Stage
>>> from robokit.opt.lm_optimizer import LMOptimizerConfig
>>> from robokit.utils.warp_utils import wp_vec7
>>> urdf = '''<robot name="finger"><link name="palm"/><link name="tip"/>
... <joint name="finger" type="revolute"><parent link="palm"/><child link="tip"/>
... <origin xyz="0 0.03 0"/><axis xyz="0 0 1"/>
... <limit lower="0" upper="1" effort="1" velocity="1"/></joint></robot>'''
>>> robot = Robot(RobotSpec.parse(urdf, load_meshes=False, mesh_dir=None))
>>> spec = HandSpec(True, ("tip", "wrist"), {"tip": "tip", "wrist": "palm"},
... (("tip",),), "wrist", ("tip",), coords)
>>> config = HandRetargeterOnlineConfig(reference_origin="wrist",
... solver=PopulationSolverConfig(stages=[Stage(2, optimizer=LMOptimizerConfig())],
... cuda_graph_mode="none"))
>>> config.task("positions").weight = {"tip": 24.0, "wrist": 0.1}
>>> solver = HandRetargeterOnline(config, robot, spec, device="cpu")
>>> solver.warmup(1)
>>> points = np.array([[[1, 2.03, 3], [1, 2, 3]]], dtype=np.float32)
>>> packed = solver.solve_numpy(points)
>>> packed.shape
(1, 8)
>>> output = wp.empty((1, 8), dtype=wp.float32, device="cpu")
>>> solver.solve(wp.from_numpy(points, dtype=wp.float32, device="cpu"), out=output) is output
True
>>> state = robot.state(q=wp.from_numpy(packed[:, 7:].copy(), dtype=wp.float32, device="cpu"),
... T_world_base=wp.from_numpy(packed[:, :7].copy(), dtype=wp_vec7, device="cpu"))
>>> all(np.isfinite(cost.numpy()).all() for cost in solver.compute_costs(state).values())
True
>>> solver.reset()import numpy as np
import warp as wp
from robokit.helpers.hand_retargeting.online import HandRetargeterOnline
from robokit.helpers.hand_retargeting.config import HandRetargeterOnlineConfig, HandSpec
from robokit.robo import Robot
from robokit.robo.robot_spec import RobotSpec
from robokit.utils.ee_coord_utils import MANOPTH_HAND_COORD_SPEC as coords
from robokit.opt.population_solver import PopulationSolverConfig, Stage
from robokit.opt.lm_optimizer import LMOptimizerConfig
from robokit.utils.warp_utils import wp_vec7
urdf = '''<robot name="finger"><link name="palm"/><link name="tip"/>
<joint name="finger" type="revolute"><parent link="palm"/><child link="tip"/>
<origin xyz="0 0.03 0"/><axis xyz="0 0 1"/>
<limit lower="0" upper="1" effort="1" velocity="1"/></joint></robot>'''
robot = Robot(RobotSpec.parse(urdf, load_meshes=False, mesh_dir=None))
spec = HandSpec(True, ("tip", "wrist"), {"tip": "tip", "wrist": "palm"},
(("tip",),), "wrist", ("tip",), coords)
config = HandRetargeterOnlineConfig(reference_origin="wrist",
solver=PopulationSolverConfig(stages=[Stage(2, optimizer=LMOptimizerConfig())],
cuda_graph_mode="none"))
config.task("positions").weight = {"tip": 24.0, "wrist": 0.1}
solver = HandRetargeterOnline(config, robot, spec, device="cpu")
solver.warmup(1)
points = np.array([[[1, 2.03, 3], [1, 2, 3]]], dtype=np.float32)
packed = solver.solve_numpy(points)
packed.shape
output = wp.empty((1, 8), dtype=wp.float32, device="cpu")
solver.solve(wp.from_numpy(points, dtype=wp.float32, device="cpu"), out=output) is output
state = robot.state(q=wp.from_numpy(packed[:, 7:].copy(), dtype=wp.float32, device="cpu"),
T_world_base=wp.from_numpy(packed[:, :7].copy(), dtype=wp_vec7, device="cpu"))
all(np.isfinite(cost.numpy()).all() for cost in solver.compute_costs(state).values())
solver.reset()Online solver, objective, filtering, and sampling settings.
Loaded robot model; self-collision skip pairs come from its spec (Robot.load(self_collision_ignore_path=...)).
Ordered semantic point topology.
Warp device. The first CUDA device is used when available.
Build all batch-shaped terms, buffers, and solver state.
Number of independent streams advanced by each solve.
Retarget one batched frame.
Point targets with shape (batch, targets, 3) in spec.target_names order.
Optional state overriding the streaming warm start.
Optional packed output buffer with shape (batch, 7 + dofs).
Packed base transform and joint positions; a helper-owned buffer is reused when out is omitted.
NumPy wrapper for solve().
Clear all streaming history and disable smoothness until a previous frame exists.
Evaluate the current targets for a caller-provided robot state.
Robot state matching the warmed batch and device.
Per-term costs keyed by task cost name.
Frame-level hand-retargeting configuration.
Task templates built at warmup; HandContext lists their inputs.
Vector rows whose target shrinks to near zero while their two targets pinch.
Scale from human to robot hand size applied to vector targets; pinched rows keep their own length.
EMA weight of the newest vector and direction targets, in (0, 1]; 1 disables smoothing.
EMA weight of the newest output joints, in (0, 1]; 1 disables smoothing.
Joint noise around the previous solution; scale is a fraction of each joint's range.
Per-axis base translation noise (m) around the previous solution of a floating hand.
Candidate schedule and LM settings of each frame's solve.
Target distance (m) below which a pinch pair latches.
Target distance (m) above which a latched pinch pair releases.
Solve floating hands relative to the target wrist position ("wrist") or in world; I/O frames are unchanged.
Append a task, optionally assigning its input role.
>>> from robokit.terms.dense.configuration_task import ConfigurationTask
>>> config = HandRetargeterOnlineConfig(terms=[]).add(ConfigurationTask(weight=0.1), name="rest")
>>> config.task("rest").weight
0.1from robokit.helpers.hand_retargeting.config import HandRetargeterOnlineConfig
from robokit.terms.dense.configuration_task import ConfigurationTask
config = HandRetargeterOnlineConfig(terms=[]).add(ConfigurationTask(weight=0.1), name="rest")
config.task("rest").weightReturn a named task; raise KeyError when absent.
>>> from robokit.terms.dense.configuration_task import ConfigurationTask
>>> config = HandRetargeterOnlineConfig(terms=[]).add(ConfigurationTask(weight=0.2), name="rest")
>>> config.task("rest").weight
0.2from robokit.helpers.hand_retargeting.config import HandRetargeterOnlineConfig
from robokit.terms.dense.configuration_task import ConfigurationTask
config = HandRetargeterOnlineConfig(terms=[]).add(ConfigurationTask(weight=0.2), name="rest")
config.task("rest").weightRetarget trajectory batches in HandSpec.target_names order by solving config.terms over HandContext inputs.
Trajectory, contact, and solver settings.
Loaded robot model.
Ordered semantic point topology.
Optional caller-owned collision scene.
Warp device. The first CUDA device is used when available.
Optional static (batch,) or per-frame (batch, frames) scene mapping.
Build buffers and the config's tasks, and capture the solver for one trajectory shape.
Number of independent trajectories.
Frames in each fixed-shape trajectory.
Mask padded frames and temporal edges in every task.
Set one positive pairwise target scale per warmed trajectory without rebuilding the solver.
Example
>>> from robokit.helpers.full_body_retargeting import FullBodyRetargeter
>>> from robokit.helpers.hand_retargeting.presets.inspire import offline, spec
>>> solver = HandRetargeterOffline(offline, FullBodyRetargeter().hand_robot("right"), spec)
>>> solver.warmup(2, 3)
>>> solver.set_target_scales(np.array([0.9, 1.2]))import numpy as np
from robokit.helpers.hand_retargeting.offline import HandRetargeterOffline
from robokit.helpers.full_body_retargeting import FullBodyRetargeter
from robokit.helpers.hand_retargeting.presets.inspire import offline, spec
solver = HandRetargeterOffline(offline, FullBodyRetargeter().hand_robot("right"), spec)
solver.warmup(2, 3)
solver.set_target_scales(np.array([0.9, 1.2]))Retarget one Warp trajectory batch.
Float points shaped (batch, frames, targets, 3) in spec.target_names order.
Optional root rotations shaped (batch, frames, 4).
Optional joint and base warm start.
Optional packed output shaped (batch, frames, 7 + dofs).
Packed base transforms and joint positions.
NumPy wrapper for solve().
Retarget with world-frame contacts; an init_state anchors the solution and its motion to that trajectory.
Named points shaped (batch, frames, targets, 3).
World contacts shaped (batch, frames, contacts, 3).
Active contacts shaped (batch, frames, contacts).
Optional state that anchors the solution around that trajectory.
Optional contacts in each link frame.
Optional caller-owned packed output.
Optional mask for sphere-surface contacts; uses contact_mask when omitted.
Packed base transforms and joint positions.
NumPy wrapper for solve_with_contact().
One whole-trajectory problem; contact_config and refine_config derive the contact ones from tracking.
Task templates of the tracking solve; HandContext lists their inputs.
Scale from human to robot hand size of each trajectory's pairwise targets until set_target_scales.
Optimize a floating hand's base trajectory; False keeps the initial one.
Joints the solve moves; None moves every joint.
Leading frames whose joints keep their initial values; base activity follows optimize_base.
LM settings of the whole-trajectory solve.
Append a task, optionally assigning its input role.
>>> from robokit.terms.dense.configuration_task import ConfigurationTask
>>> config = HandRetargeterOfflineConfig(terms=[]).add(ConfigurationTask(weight=0.1), name="rest")
>>> config.task("rest").weight
0.1from robokit.helpers.hand_retargeting.config import HandRetargeterOfflineConfig
from robokit.terms.dense.configuration_task import ConfigurationTask
config = HandRetargeterOfflineConfig(terms=[]).add(ConfigurationTask(weight=0.1), name="rest")
config.task("rest").weightReturn a named task; raise KeyError when absent.
>>> from robokit.terms.dense.configuration_task import ConfigurationTask
>>> config = HandRetargeterOfflineConfig(terms=[]).add(ConfigurationTask(weight=0.2), name="rest")
>>> config.task("rest").weight
0.2from robokit.helpers.hand_retargeting.config import HandRetargeterOfflineConfig
from robokit.terms.dense.configuration_task import ConfigurationTask
config = HandRetargeterOfflineConfig(terms=[]).add(ConfigurationTask(weight=0.2), name="rest")
config.task("rest").weight