{"data":{"kind":"file","path":"README.md","version_id":"xqbiwf5c8ybvac4jf8dr7if8","entry":{"name":"README.md","path":"README.md","is_directory":false,"size":5370,"modified_at":"2026-11-07T23:50:40.061000","content_hash":"fc281ee955533b5905eaef40f324f9f9a87076a107fa828fad313085c5767191"},"entries":[],"content":"# bfcl-v4-memory\n\nThe memory category of the Berkeley Function Calling Leaderboard (BFCL V4), with its three backends: key-value (`kv`), embedding search (`vector`) and recursive summary (`rec_sum`). In each scenario, the agent first stores facts from a chain of scripted conversations using memory tools. It then answers recall questions from that memory alone, with no chat history. Scoring uses the official `bfcl_eval` `agentic_checker`; there is no LLM judge.\n\n## Taskset\n\n- **Source:** [`bfcl-eval`](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) at leaderboard commit `f7cf735`, through the same Gorilla fork as `bfcl-v3` (`mikasenghaas/gorilla@898763a`). The fork only relaxes dependency pins; every memory file is identical to `f7cf735`. Data is loaded from the package at runtime, not from an HF dataset.\n- **Size:** 15 tasks: 3 backends × 5 scenarios (`student`, `customer`, `finance`, `healthcare`, `notetaker`). Per backend there are 37 prerequisite sessions and 155 questions, so 465 scored questions in total.\n\n## Running\n\n```bash\nuv pip install -e environments/tool_use/bfcl_v4_memory   # or: prime env install rutvi02/bfcl-v4-memory\n\nuv run vf-run bfcl-v4-memory -m <model> --env.taskset.scenarios notetaker \\\n  --env.prefill.sampling.temperature 0.001 --env.agent.sampling.temperature 0.001 \\\n  --env.max-concurrent-agents 4\n```\n\nOne task is one backend × scenario chain.\n- The env plays the prerequisite sessions in order, as the `prefill` role, which is untrainable and has no reward. Each session starts from the memory the previous one left.\n- It then runs the questions, which are independent and may run concurrently, as the `agent` role. Each question starts from the final memory.\n- The headline reward is therefore question accuracy, the number BFCL reports.\n\nThe two roles can use different models (`--env.prefill.model`, `--env.agent.model`), which separates writing memory from reading it.\n\nOptions:\n- Select a subset with `--env.taskset.backends` and `--env.taskset.scenarios`.\n- `temperature 0.001` is BFCL's setting.\n- Both roles run in local subprocesses by default. `--env.<role>.runtime.type prime` selects a sandbox.\n\nEverything else is BFCL's own code, run as-is:\n- data and system prompts (`add_memory_instruction_system_prompt`);\n- tool schemas (`convert_to_tool`), exposed under their official names;\n- memory backends, run through `execute_multi_turn_func_call`, with snapshots written and read through BFCL's own files;\n- the step limit (21 tool-calling steps per turn, i.e. `count > MAXIMUM_STEP_LIMIT`);\n- the grader (`agentic_checker` on the last text answer).\n\nTool arguments reach the backend raw, without MCP validation or coercion, so a malformed call gets BFCL's own error.\n\nWeight-0 metrics for auditing:\n- `n_tool_calls` and `n_tool_errors`;\n- `answered` and `abstained`;\n- `graded_malformed_call`;\n- `gold_in_prompt` and `gold_in_tool_result`.\n\n`trace.info` records the rollout's test id, role, end memory snapshot, `force_quit`, and the graded message.\n\n## Parity\n\nThe port was checked against official BFCL (its generation and evaluation entry points with the generic `OpenAICompletionsHandler`), with both systems talking to the same model endpoint:\n\n- **Scripted model, full benchmark.** The model misbehaves on purpose: string ids, bad arguments, parallel and duplicate calls, destructive rewrites, a runaway loop up to the step limit, abstentions.\n  - 3,455/3,455 model requests identical;\n  - 111/111 session snapshots identical;\n  - 465/465 verdicts identical.\n  - The check was repeated from a clean install on verifiers 0.4.1.\n- **Qwen3-4B-Instruct-2507, recorded and replayed** (notetaker × 3 backends): 259/259 requests, 15/15 snapshots and 75/75 verdicts identical.\n- **Live, same model:** the port agrees with BFCL on 61–65 of 75 verdicts, while two BFCL runs agree with each other on 65 of 75. Local servers aren't deterministic under concurrent batching.\n\n## Notes\n\nKnown discrepancies from upstream BFCL:\n\n- `vector` similarity scores differ below 1e-6, and ties between duplicate entries (identical text and score) can come back in a different order. The cause is MiniLM/FAISS float noise across library versions (the fork needs numpy 2). Neither changed a snapshot or verdict in any check.\n- Model replies are streamed (BFCL doesn't stream). Some servers format streamed tool arguments with spaces, so the conversation history text can differ while the calls are the same.\n- A failed prerequisite session fails the episode, and it can be retried. BFCL instead warns and lets later sessions start from empty memory.\n- An unknown tool name gets the harness's `error: unknown tool '<name>'` instead of BFCL's Python execution error.\n- BFCL's call rewriter adds its instance prefix, which contains the model name, when a backend method name followed by `(` appears inside stored text. The prefix differs here. This was not seen in any run.\n\nReproduced as in upstream, not discrepancies:\n- An empty `vector` core memory renders as `{}`.\n- Spaces before `(` inside stored text are removed by BFCL's call rewriter.\n\nWhen running official BFCL against a local server: its `OpenAICompletionsHandler` grades every answer as empty when the server sends `tool_calls: []` with a text reply.\n\n## Changelog\n\n- 2026-11-07: Initial release. Requires `verifiers>=0.4.1`.\n","encoding":"utf-8","truncated":false,"total_bytes":5370},"status":null}