Skip to main content

Semantic Annotation

EgoSuite MCAP files can include annotation topics that store action-level semantic annotation with tasks, subtasks, and even skill names. Each segment has time bounds. One message per segment; an episode may contain many such messages.

Example: segment caption and hand pose overlay (clean the mirror)

Example: camera image overlaid with semantic annotation (e.g. event "The person cleaned the mirror, comb, and clips", action "Press the nozzle") and hand pose.

Topic & Message Type​

  • Topic: /annotation/semantic_segments
  • Message type: annotation.SemanticSegment
  • Encoding: protobuf

Defined in annotation/semantic_segments.proto; each message contains one segment and includes common.header.Header.

Message definition

syntax = "proto3";

package annotation;

import "common/header.proto";

message SegmentInfo {
  string subtask_description = 1;
  repeated string skill = 2;
  double start_time = 3;
  double end_time = 4;
}

// Main message for topic /annotation/semantic_segments (one message per segment)
message SemanticSegment {
  common.header.Header header = 1;
  // Episode-level task description; same value for all segments in an episode.
  string task_description = 2;
  SegmentInfo segment = 3;
}

The proto supports time-based (start_time/end_time) bounds. task_description stores the global task description for the episode (same across all segments). segment.subtask_description describes the action for a specific time range; an episode contains multiple such segments over time. segment.skill is a repeated field listing skill tags associated with that action segment.