{"data":{"kind":"file","path":"README.md","version_id":"ceunjztc8g1kbl45ioyn9k30","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":10387,"modified_at":"2026-10-27T01:24:57.333000","content_hash":"23fbeeb898719d2fffca410ff74b0a26ce99db790c65dc2d751ab21875371204"},"entries":[],"content":"# Video2Robot Embodied Environment Hub\n\n**Human Video → 2D Landmarks → Robot Task-Space Proxy Demonstrations → Verifiable RL Tasks**\n\n> **Status:** research prototype, runnable core and synthetic benchmark, packaged for Prime Intellect as `taylormino/video2robot-embodied` version `0.2.0`. Not validated against physical robots. It does **not** recover ground-truth 3D geometry from a single RGB camera and does not produce safe robot joint commands.\n\nThis project provides **two complementary components**:\n\n1. **Offline dataset engine:** tracks human hand/arm/foot/body landmarks from a video, normalizes motion to a torso-anchored coordinate frame, filters short holes, derives action deltas, infers tentative motion tags, and exports auditable JSONL/CSV demonstrations for *offline imitation learning research*.\n2. **Native Verifiers v1 benchmark:** an LLM/agent takes noisy, partially occluded human video landmark tracks and predicts cleaned robot task-space trajectories. A deterministic hidden-reference reward grades safety/schema validity, temporal motion fidelity and hand grip semantics. No judge API is needed.\n\nThe baseline is deliberately a **2D intermediate representation**. To create actual robot demonstrations, add calibrated 3D reconstruction, object state/contact tracking, embodiment-specific IK/retargeting, torque limits, collision/self-collision checks, sim validation and expert labeling.\n\n## 1 — Run immediately (Python 3.10+; no video, API or GPU)\n\nFrom the extracted project directory:\n\n```bash\npython -m pip install -e .\npython -m video2robot_env demo --count 25 --split eval --out benchmark_report.json\npython -m pytest -q   # pip install pytest if not already installed\n```\n\nRunning the demo prints a **local baseline** benchmark report. The score is the score of the included simple rule-based geometric model on procedural synthetic clips, **not** the score of a trained robot/LLM or a real-world benchmark.\n\n## 2 — Convert a video into training-ready *proxy* samples\n\nInstall video dependencies (recommended separate Python 3.11 virtual environment):\n\n```bash\npython3.11 -m venv .venv\nsource .venv/bin/activate                  # WSL/Linux\npython -m pip install -e '.[video]'\nvideo2robot from-video --input ./your_consented_demo.mp4 --out ./outputs/episode_001 \\\n  --sample-fps 10 --license 'your-dataset-license'\n```\n\nIf your installed MediaPipe lacks `mp.solutions`, use a compatible legacy 0.10.x release or supply external pose-tracker data via `from-landmarks`. The `video` extra pins a tested *API range*, but system/OS compatibility still depends on available upstream wheels. Video decoding and inference use **local** OpenCV + MediaPipe. Data is **not sent to an API**.\n\n**Already have 2D keypoints from another tracker?** Use the included deterministic fixture:\n\n```bash\nvideo2robot from-landmarks \\\n  --input examples/demo_landmarks.jsonl \\\n  --out outputs/demo_001 \\\n  --source-id consented-clip-001 --license CC0\n```\n\n`source-id` is a stable, non-identifying original clip ID. Its SHA-256 drives source-grouped train/eval allocation, which prevents **frame-level leakage within the same video**.\n\n### Output files\n\n```text\noutputs/episode_001/\n├── metadata.json           # SHA-256 source ID, licensing, split, warnings, coverage\n├── landmarks.jsonl         # video mode only: sampled source 2D pose+hand landmarks\n├── transitions.jsonl       # time, observations, proxy target, delta, mask, tags\n├── actions.csv             # flat columns for data loaders\n├── motion_events.json      # provisional motion candidate segments\n└── trajectory_viewer.html  # offline interactive QC review; open in browser\n```\n\nA `transitions.jsonl` row:\n\n```json\n{\n  \"frame\": 1,\n  \"timestamp_s\": 0.1,\n  \"observation\": {\"human_image_keypoints\": {\"left_wrist\": [0.35, 0.42]}},\n  \"action_proxy\": {\"left_hand_xy\": [-0.25, 0.2], \"left_gripper_open\": 0.75},\n  \"action_mask\": {\"left_hand_xy\": true, \"left_gripper_open\": true},\n  \"action_delta_xy\": {\"left_hand_xy\": [0.01, 0.01]},\n  \"motion_tags\": [\"left_hand_motion\"]\n}\n```\n\n*Illustrative abbreviated row — actual export contains all four limb targets plus both grippers.*\n\n- Coordinates `*_xy`: dimensionless, body-centered 2D, **x positive right, y positive up**, divided by 3× the median image shoulder-to-hip distance. Camera calibration and metric 3D are absent. **Not robot joint angles**.\n- `*_gripper_open`: rough aperture proxy from 2D thumb-index distance relative to palm width; **1 = open, 0 = closed**. No observed hand → `null`, `action_mask=false`.\n- `action_delta_xy` is per-sample difference in normalized target coordinates, computed only when consecutive valid samples exist; missing trajectories are never silently replaced with zeros.\n- `motion_tags` are simple velocity/threshold heuristics and are **not confirmed physical contacts or semantic action labels**.\n- JSONL outputs include masks to train models that ignore absent labels. Split is source-level, not frame-level.\n\n### Expected input for `from-landmarks`\n\nEach line in the `.jsonl` is a frame:\n\n```json\n{\"t\":0.0,\"keypoints\":{\"left_shoulder\":[0.44,0.29,0.96],\"right_shoulder\":[0.56,0.29,0.96],\"left_hip\":[0.46,0.52,0.96],\"right_hip\":[0.54,0.52,0.96],\"left_wrist\":[0.35,0.41,0.9],\"right_wrist\":[0.64,0.42,0.9],\"left_ankle\":[0.45,0.85,0.9],\"right_ankle\":[0.56,0.86,0.9]},\"pinch_open\":{\"left\":1.0,\"right\":1.0}}\n```\n\nEach 2D keypoint is `[x,y,visibility]`, with x/y in normalized image coordinates. At least two frames with valid hips and shoulders are required. Default visibility threshold is 0.5. Short internal gaps of at most 2 samples are linearly interpolated, but long gaps and absent grasp labels remain explicitly masked.\n\n## 3 — Prime Intellect Environments Hub / Verifiers v1\n\nThe native taskset implementation is `video2robot_env/environment.py`. Verifiers v1 is included in the default installation:\n\n- Exported class: `VideoMotionTaskset` (package-level lazy export and `video2robot_embodied` discovery package), compatible with the task-centric Verifiers v1 interface (`vf.TaskData`, `vf.Task`, `vf.Taskset`, `@vf.reward`).\n- Task config: `split` (`train|eval`), `seed` (int), `count` (int), `n_frames` (5–24).\n- Procedural task families (8): `reach_left`, `reach_right_grasp`, `side_step`, `two_hand_lift`, `squat`, `reach_return`, `handover_proxy`, `grasp_release`.\n- Agent input: sampled 2D body observations in the prompt with random camera translation, tracking noise, and per-joint/frame missing values.\n- Agent output: valid JSON `{ \"frames\": [ {\"t\":0,\"left_hand_xy\":[x,y], ..., \"right_gripper_open\":1} ] }`.\n- Hidden ground truth: **only in task data**, not displayed in prompt. Train and eval use disjoint seeds/IDs.\n- Deterministic reward includes trajectory error, gripper correctness, velocity/temporal matching, motion dynamics and endpoint checks; strict schema and displacement limits reject invalid outputs. Details are in `video2robot_env/benchmark.py`.\n- No arbitrary code execution in reward, no network or LLM judge calls, no API secrets.\n\n```bash\npython -m pip install -e '.[dev]'\nvf-eval video2robot-embodied -n 1 --dry-run --no-push\n# Live evaluation requires a configured model endpoint:\nvf-eval video2robot-embodied -m <YOUR_MODEL> -n 5 --no-push\n```\n\n**Compatibility note:** the Hub package targets Verifiers `>=0.4,<0.5`; validation uses Python 3.12. Unit tests, deterministic callback checks and CLI dry-runs do not constitute a live model evaluation. Hosted evaluation and real-world robot performance remain unverified.\n\n## 4 — v2 robustness and evaluation\n\n```bash\nvideo2robot benchmark-suite --count 64 --seed 41 --frames 16 --out examples/benchmark_suite_v2.json\nvideo2robot audit --episode outputs/demo_001\nvideo2robot pack-dataset --episodes outputs/demo_001 --out outputs/dataset\n```\n\nThe suite covers standard/hard/stress noise, correlated missing landmarks and outliers, with baseline-vs-static comparisons and per-task scores. The dataset packer checks consent/licensing, data quality, masks and split-group leakage. `pack-dataset` will reject unconsented clips or unknown licensing, and may demand manual quality review. See `DATASET_CARD.md` for limitations.\n\n## 5 — Safety, consent, scientific validity\n\n- Process **only recordings you own or have explicit permission to use**; do not upload identifiable people or copyrighted recordings without rights. Get permission for reuse and for training models.\n- 2D image coordinates do not uniquely determine 3D trajectory, foot-ground force, joint torque, exact object contact, or safe grasp. Self-occlusion can make a motion ambiguous.\n- Camera motion, handedness flips, clothes, camera angle, blur, truncated body, rolling shutter and untracked objects can cause systematic error. Inspect landmarks, masks and video samples before training.\n- Only use direct video-to-proxy output for research/pretraining/annotation support, *never as direct robot actuation commands*. Adapt to a robot URDF, run IK, smooth joint controls, enforce torque/velocity/collision limits, validate contacts and run simulation before hardware.\n- Do not confuse measured human actions with successful robot trajectories. Real robot trajectories should have synchronized robot state (`q`, `dq`, TCP, gripper, object pose, environment contacts) and accurate actions.\n\n## 6 — Roadmap for competitive robotics bounty\n\nSee [`BOUNTY_RFC.md`](BOUNTY_RFC.md) for a stricter validation plan. Major future milestones: calibrated 3D hand/body/object reconstruction, retargeting to one real robot, contact simulation / kinematics, benchmark on labeled real recordings, robust train/eval splits by person + scene + video, export to LeRobot/robomimic formats, comparisons against published baselines, and reproducible hosted eval.\n\n## External references\n\n- Prime Intellect: https://docs.primeintellect.ai/verifiers/environments and https://github.com/PrimeIntellect-ai/verifiers\n- Prime Intellect Environments Hub: https://app.primeintellect.ai/dashboard/environments\n- SPIDER (general physics-informed human-to-robot retargeting): https://github.com/facebookresearch/spider\n- EgoInfinity (4D hand/object video-to-action research): https://rice-robotpi-lab.github.io/EgoInfinity/\n\n**License:** MIT (project code). Input video, people, and third-party weights have their own licenses and consent requirements.\n","encoding":"utf-8","truncated":false,"total_bytes":10387},"status":null}