Skip to main content

Wrist RGB Camera

The EgoSuite rig optionally includes cameras mounted on the left and right wrists. In the MCAP files, these cameras are represented by dedicated topics for:

  • Compressed video streams
  • Raw compressed video streams when source raw MP4 files are included
  • Per‑camera intrinsics (calibration)
  • Per‑camera extrinsics (pose per frame transforms)

Topic & Message Type​

The following topics are registered for wrist cameras. All channels use protobuf encoding:

  • Left Wrist Camera

    • Video topic: /sensor/camera/left_wrist/video
    • Message type: foxglove.CompressedVideo
    • Intrinsic topic: /sensor/camera/left_wrist/intrinsic
    • Intrinsic type: foxglove.CameraCalibration
    • Raw video topic: /sensor/camera_raw/left_wrist/video
    • Raw video type: foxglove.CompressedVideo
    • Raw calibration topic: /sensor/camera_raw/left_wrist/calibration
    • Raw calibration type: sensor.camera.RawCameraCalibration

  • Right Wrist Camera

    • Video topic: /sensor/camera/right_wrist/video
    • Message type: foxglove.CompressedVideo
    • Intrinsic topic: /sensor/camera/right_wrist/intrinsic
    • Intrinsic type: foxglove.CameraCalibration
    • Raw video topic: /sensor/camera_raw/right_wrist/video
    • Raw video type: foxglove.CompressedVideo
    • Raw calibration topic: /sensor/camera_raw/right_wrist/calibration
    • Raw calibration type: sensor.camera.RawCameraCalibration

Wrist camera video topics carry compressed video frames as foxglove.CompressedVideo messages. You can inspect, play back using LW-VIZ or export these streams with LW-EgoSuite Devkit.

Camera Intrinsics​

Camera publishes its calibration on a dedicated foxglove.CameraCalibration topic. For field definitions (including width, height, intrinsic matrix K, distortion model and parameters D, rectification matrix R, projection matrix P, and frame_id), see CameraCalibration documentation.

Raw Calibration​

The sensor.camera.RawCameraCalibration message on the /sensor/camera_raw/left_wrist/calibration and /sensor/camera_raw/right_wrist/calibration topics is an EgoSuite-specific proto, not a Foxglove schema. It carries the rig's original (pre-undistortion) calibration for the raw wrist camera video streams:

message RawCameraCalibration {
  google.protobuf.Timestamp timestamp = 1;
  string frame_id = 2;
  fixed32 width = 3;
  fixed32 height = 4;

  // Name of the distortion model as recorded by the rig (e.g. "equiDis62", "brown").
  // Not validated or restricted to any fixed set of values.
  string distortion_model = 5;

  // Distortion parameters, in the order defined by distortion_model.
  repeated double distortion_params = 6;

  Intrinsics intrinsics = 7;  // cx, cy, fx, fy
}

Unlike foxglove.CameraCalibration, distortion_model here is a free-form string (e.g. equiDis62, brown) and is not restricted to the model names Foxglove's Image panel recognizes, so this message cannot be consumed directly by Foxglove's calibration UI. It is intended for offline/algorithmic use that needs the rig's original distortion model and coefficients, paired with the raw (distorted) video on the corresponding /sensor/camera_raw/left_wrist/video or /sensor/camera_raw/right_wrist/video topic.

Camera Extrinsics​

Camera extrinsic info is expressed as foxglove.FrameTransforms message. For field definitions (e.g. parent_frame_id, child_frame_id, translation, rotation), see FrameTransform documentation.

In EgoSuite MCAP files, wrist camera extrinsics are generated from world-frame position and orientation in each wrist camera params file. The message uses parent_frame_id = "world" and child_frame_id set to the concrete camera frame (left_wrist_camera or right_wrist_camera). The converter additionally rotates the wrist camera orientation by 180 degrees around its local Z axis before writing the transform.

For details on computing the W2C (world-to-camera) extrinsic matrix from the C2W pose, refer to the Head Cameras page, which provides the same matrix inversion and projection formulas applicable to all EgoSuite cameras.

Coordinate Frames​

Wrist cameras use the same coordinate conventions as head cameras and the rest of EgoSuite. For detailed descriptions, refer to the Head Cameras - Coordinate frames section.

In summary:

  • World Frame: All pose data is expressed in a shared world frame.
  • Camera Frame (OpenCV convention):
    • zz axis points forward from the camera.
    • xx axis points to the right in the image.
    • yy axis points down in the image.

With intrinsics and extrinsics combined, you can project 3D hand or object points into wrist camera images or lift image measurements back to 3D in the world frame.

Typical Usage​

Common use cases for wrist camera data include:

  • Studying close‑range hand–object interactions from each wrist's egocentric perspective.
  • Aligning wrist camera images with 3D hand keypoints for grasp analysis.
  • Fusing multiple camera views (head and wrist) using the shared transform tree in foxglove.FrameTransforms.