{"data":{"kind":"file","path":"README.md","version_id":"hepgzf2kk106fpqm82ql6agv","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":2902,"modified_at":"2026-11-02T22:15:40.212000","content_hash":"38b04d5d9e6bf47e6c2dcb2b0842e868683aa28acb0f2012f427d4a76b606096"},"entries":[],"content":"# SimReadyBench\n\nReconstruct a USD object from one image, then measure its formal, visual and\nkinematic fidelity. SimReadyBench includes 100 NV100 tasks, a formal-feedback\nservice and a separate CPU verifier. Agents have 64 authoring turns and up to\nfive formal-feedback calls. Blender is used for verification, not authoring.\n\n## Install\n\nUse Python 3.12 and Verifiers 0.4.0:\n\n```sh\nuv add 'verifiers[harbor]==0.4.0'\nprime env install chrisvoncsefalvay/simreadybench@2.2.0.post2\npython -m simreadybench tasks\n```\n\nThe command above installs the current private staging package. The team release\nis intended for `hcltech-robotics/simreadybench`.\n\n## Prepare a verifier\n\nEach task needs a completed private Prime verifier image. The scissors task is a\nsmall starting point. Find the team's ID with `prime teams list`, then print its\nbuild command:\n\n```sh\npython -m simreadybench image-plan nv100_198a9e9c4f7ce3f2 --team-id TEAM_ID\n```\n\nRun the printed command to build the image on Prime. This uses billed image\nstorage and downloads that task's reference assets during the verifier build.\nGet the completed image's artefact ID from:\n\n```sh\nPRIME_TEAM_ID=TEAM_ID prime images list --output json\n```\n\nSet `PRIME_API_KEY` on the controller. The actor receives only its\nobservation, instruction and authoring tools. Reference USDs and comparison\nimages stay in the separate verifier, which grades offline.\n\n## Evaluate\n\nCreate a one-task configuration using a multimodal model with tool support:\n\n```sh\npython -m simreadybench config nv100_198a9e9c4f7ce3f2 --team-id TEAM_ID --artifact-id COMPLETED_ARTIFACT_ID --model YOUR_VISION_MODEL --output scissors.toml\nuv run vf-eval @ scissors.toml --dry-run\nuv run vf-eval @ scissors.toml\n```\n\nThe generated configuration runs one rollout on Prime and keeps traces local.\nIt checks private image ownership and artefact identity before execution. Prime\nimage tags must not be overwritten. OCI image bindings can also use immutable\n`@sha256` references.\n\nFor more tasks, add their image bindings to `[env.taskset.verifier_images]` and\nlist the task IDs under `[env.taskset].tasks`. Omitting `tasks` selects all 100.\n\n## Results\n\nVerifiers writes `traces.jsonl`, the run configuration and worker logs under\n`outputs/`. Summarise the benchmark scores with:\n\n```sh\npython -m simreadybench report outputs/RUN_DIRECTORY/traces.jsonl\npython -m simreadybench report outputs/RUN_DIRECTORY/traces.jsonl --json\n```\n\nThe report gives formal, visual and kinematic fidelity separately, together with\nknown/unknown counts and measurement coverage. Unknown measurements remain\nunknown. Verifiers' generic scalar reward is not a SimReadyBench score.\n\nThe package uses evaluator 2.2.0. Task files and evaluator wheels are included;\nverifier builds fetch their reference payloads from the pinned Hugging Face\ndataset. Loading tasks does not require Harbor storage or a Harbor login.\n","encoding":"utf-8","truncated":false,"total_bytes":2902},"status":null}