Skip to main content

Overview

MCAP Data is the native EgoSuite recording format. Each .mcap file contains multiple time-synchronized topics: video streams, poses, annotations, session metadata, and optionally audio, IMU measurements, and static transforms. The exact set of topics depends on the capture rig (e.g. head vs. wrist cameras).

EgoSuite topics use Protobuf encoding. Most native messages carry a common Header (common.header.Header): count, ts (Unix time in nanoseconds), sender; sensor.IMUMeasurement uses timestamp and frame_id instead. Message types and proto definitions live in the lw-egosuite-msg package (protos/ and generated code).

Topic convention​

  • Some topics use third-party schemas (e.g. Foxglove foxglove.*, tf2_msgs.*) or are special-cased.
  • Most EgoSuite messages carry a header field of type common.header.Header with:
    • count (uint64)
    • ts (int64, Unix time in nanoseconds)
    • sender (string)
      Third-party messages such as foxglove.* and tf2_msgs.* do not use this header.

Registered topics​

TopicProto Message
/session/metadatasession.SessionMetadata
/annotation/semantic_segmentsannotation.SemanticSegment
/pose/bodypose.BodyFrame
/pose/upper_bodypose.UpperBodyFrame
/pose/lower_bodypose.LowerBodyFrame
/pose/headpose.HeadFrame
/pose/headcampose.HeadCamFrame
/pose/left_handpose.LeftHandFrame
/pose/right_handpose.RightHandFrame
/pose/right_eye_campose.RightEyeCamFrame
/sensor/camera/head_left/videofoxglove.CompressedVideo
/sensor/camera/head_right/videofoxglove.CompressedVideo
/sensor/camera/head_left/intrinsicfoxglove.CameraCalibration
/sensor/camera/head_right/intrinsicfoxglove.CameraCalibration
/sensor/camera/head_left/extrinsicfoxglove.FrameTransforms
/sensor/camera/head_right/extrinsicfoxglove.FrameTransforms
— Optional —
/sensor/camera/left_wrist/videofoxglove.CompressedVideo
/sensor/camera/right_wrist/videofoxglove.CompressedVideo
/sensor/camera/left_wrist/intrinsicfoxglove.CameraCalibration
/sensor/camera/right_wrist/intrinsicfoxglove.CameraCalibration
/sensor/camera_raw/head_left/videofoxglove.CompressedVideo
/sensor/camera_raw/head_right/videofoxglove.CompressedVideo
/sensor/camera_raw/left_wrist/videofoxglove.CompressedVideo
/sensor/camera_raw/right_wrist/videofoxglove.CompressedVideo
/sensor/camera_raw/head_left/calibrationsensor.RawCameraCalibration
/sensor/camera_raw/head_right/calibrationsensor.RawCameraCalibration
/sensor/camera_raw/left_wrist/calibrationsensor.RawCameraCalibration
/sensor/camera_raw/right_wrist/calibrationsensor.RawCameraCalibration
/sensor/imusensor.IMUMeasurement
/tf_staticfoxglove.FrameTransforms
/sensor/camera/head_depth/imagefoxglove.CompressedImage
/sensor/camera/head_depth/intrinsicfoxglove.CameraCalibration
/sensor/camera/head_depth/extrinsicfoxglove.FrameTransforms
/sensor/audiofoxglove.RawAudio
/annotation/bad_frame/pose/handannotation.BadFrameHand
/annotation/bad_frame/pose/upper_bodyannotation.BadFrameBody
/annotation/bad_frame/pose/lower_bodyannotation.BadFrameBody
/annotation/bad_frame/pose/vioannotation.BadFrameVio
/annotation/bad_frame/camera/head_leftannotation.BadFrameImage
/annotation/bad_frame/camera/left_wristannotation.BadFrameImage
/annotation/bad_frame/camera/right_wristannotation.BadFrameImage

Use the sidebar to open detailed docs:

SectionTopics / content
Metadata/session/metadata — session/episode metadata (devices, global time, task info, operator)
PoseHead, hand, body pose and related frames
SensorRGB camera (video, raw video, intrinsic, extrinsic), depth camera (depth image + intrinsic/extrinsic, liftable to 3D point cloud), Audio
Semantic Annotation/annotation/semantic_segments — task description, segmented actions, skill names, and time bounds

For Devkit installation, CLI/API usage (e.g. lw-egosuite convert, lw-egosuite export-video), and visualization, see EgoSuite Devkit.