Overview
MCAP Data is the native EgoSuite recording format. Each .mcap file contains multiple time-synchronized topics: video streams, poses, annotations, session metadata, and optionally audio, IMU measurements, and static transforms. The exact set of topics depends on the capture rig (e.g. head vs. wrist cameras).
EgoSuite topics use Protobuf encoding. Most native messages carry a common Header (common.header.Header): count, ts (Unix time in nanoseconds), sender; sensor.IMUMeasurement uses timestamp and frame_id instead. Message types and proto definitions live in the lw-egosuite-msg package (protos/ and generated code).
Topic convention
- Some topics use third-party schemas (e.g. Foxglove
foxglove.*,tf2_msgs.*) or are special-cased. - Most EgoSuite messages carry a header field of type
common.header.Headerwith:count(uint64)ts(int64, Unix time in nanoseconds)sender(string)
Third-party messages such asfoxglove.*andtf2_msgs.*do not use this header.
Registered topics
| Topic | Proto Message |
|---|---|
/session/metadata | session.SessionMetadata |
/annotation/semantic_segments | annotation.SemanticSegment |
/pose/body | pose.BodyFrame |
/pose/upper_body | pose.UpperBodyFrame |
/pose/lower_body | pose.LowerBodyFrame |
/pose/head | pose.HeadFrame |
/pose/headcam | pose.HeadCamFrame |
/pose/left_hand | pose.LeftHandFrame |
/pose/right_hand | pose.RightHandFrame |
/pose/right_eye_cam | pose.RightEyeCamFrame |
/sensor/camera/head_left/video | foxglove.CompressedVideo |
/sensor/camera/head_right/video | foxglove.CompressedVideo |
/sensor/camera/head_left/intrinsic | foxglove.CameraCalibration |
/sensor/camera/head_right/intrinsic | foxglove.CameraCalibration |
/sensor/camera/head_left/extrinsic | foxglove.FrameTransforms |
/sensor/camera/head_right/extrinsic | foxglove.FrameTransforms |
| — Optional — | |
/sensor/camera/left_wrist/video | foxglove.CompressedVideo |
/sensor/camera/right_wrist/video | foxglove.CompressedVideo |
/sensor/camera/left_wrist/intrinsic | foxglove.CameraCalibration |
/sensor/camera/right_wrist/intrinsic | foxglove.CameraCalibration |
/sensor/camera_raw/head_left/video | foxglove.CompressedVideo |
/sensor/camera_raw/head_right/video | foxglove.CompressedVideo |
/sensor/camera_raw/left_wrist/video | foxglove.CompressedVideo |
/sensor/camera_raw/right_wrist/video | foxglove.CompressedVideo |
/sensor/camera_raw/head_left/calibration | sensor.RawCameraCalibration |
/sensor/camera_raw/head_right/calibration | sensor.RawCameraCalibration |
/sensor/camera_raw/left_wrist/calibration | sensor.RawCameraCalibration |
/sensor/camera_raw/right_wrist/calibration | sensor.RawCameraCalibration |
/sensor/imu | sensor.IMUMeasurement |
/tf_static | foxglove.FrameTransforms |
/sensor/camera/head_depth/image | foxglove.CompressedImage |
/sensor/camera/head_depth/intrinsic | foxglove.CameraCalibration |
/sensor/camera/head_depth/extrinsic | foxglove.FrameTransforms |
/sensor/audio | foxglove.RawAudio |
/annotation/bad_frame/pose/hand | annotation.BadFrameHand |
/annotation/bad_frame/pose/upper_body | annotation.BadFrameBody |
/annotation/bad_frame/pose/lower_body | annotation.BadFrameBody |
/annotation/bad_frame/pose/vio | annotation.BadFrameVio |
/annotation/bad_frame/camera/head_left | annotation.BadFrameImage |
/annotation/bad_frame/camera/left_wrist | annotation.BadFrameImage |
/annotation/bad_frame/camera/right_wrist | annotation.BadFrameImage |
Navigation
Use the sidebar to open detailed docs:
| Section | Topics / content |
|---|---|
| Metadata | /session/metadata — session/episode metadata (devices, global time, task info, operator) |
| Pose | Head, hand, body pose and related frames |
| Sensor | RGB camera (video, raw video, intrinsic, extrinsic), depth camera (depth image + intrinsic/extrinsic, liftable to 3D point cloud), Audio |
| Semantic Annotation | /annotation/semantic_segments — task description, segmented actions, skill names, and time bounds |
For Devkit installation, CLI/API usage (e.g. lw-egosuite convert, lw-egosuite export-video), and visualization, see EgoSuite Devkit.