VCN #45: BENCH - SETUP Write an eval for your coding agent. Frontier Tower Floor 10, Wednesday 2026-07-29. Doors 19:00 PT, walkthrough 19:30. Deck: https://vcn-45-bench.vercel.app Prose: https://vcn-45-bench.vercel.app/llms.txt RSVP: https://luma.com/vcn-45-bench This file is plain text on purpose: curl it mid-lab, or hand it to your agent. There are NO credentials in this file and it never prints your key. ================================================================================ STEP 0 - CLAIM YOUR TWO CREDITS (do this FIRST, a human has to approve one) ================================================================================ 1. z.ai (GLM) + Claude Code -- the agent under test. https://immersivecommons.com/zai-keys Free Immersive Commons account, then request a workshop key. Keys are approved in waves by the facilitator, so request EARLY and paste later. A key is good for 5 hours from approval. Several GLM models are available on one key; glm-4.6 is the default. 2. Nebius Token Factory -- the lab compute. https://dev.nebius.com/builders Join the Builder Program: $50 Token Factory, $50 AI Cloud, $25 Tavily. Verify your email, then mint a Token Factory API key. This is your fallback if your laptop cannot host a model locally. Instant and self-serve, no approval wait. If you only do one thing before doors, do the z.ai request. ================================================================================ STEP 1 - NODE 18+, CLAUDE CODE, AND THE GATEWAY ================================================================================ Same three steps on every platform. Four correct spellings. -------------------------------------------------------------------------------- macOS (zsh) -------------------------------------------------------------------------------- # Node 18+ : installer from nodejs.org works on Apple Silicon and Intel node -v # want v18 or higher npm install -g @anthropic-ai/claude-code export ANTHROPIC_BASE_URL=https://immersivecommons13.tail5da903.ts.net export ANTHROPIC_AUTH_TOKEN=agt_your_key_here export ANTHROPIC_MODEL=glm-4.6 export ANTHROPIC_SMALL_FAST_MODEL=glm-4.5-air claude # you are live -------------------------------------------------------------------------------- Linux (bash) -------------------------------------------------------------------------------- # Do NOT trust `apt install nodejs` on older LTS: it ships v12. curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - sudo apt-get install -y nodejs node -v # want v18 or higher sudo npm install -g @anthropic-ai/claude-code export ANTHROPIC_BASE_URL=https://immersivecommons13.tail5da903.ts.net export ANTHROPIC_AUTH_TOKEN=agt_your_key_here export ANTHROPIC_MODEL=glm-4.6 export ANTHROPIC_SMALL_FAST_MODEL=glm-4.5-air claude # you are live -------------------------------------------------------------------------------- Windows (PowerShell) -------------------------------------------------------------------------------- # PowerShell has NO `export`. Use $env: instead. # Node 18+ : installer from nodejs.org node -v # want v18 or higher npm install -g @anthropic-ai/claude-code $env:ANTHROPIC_BASE_URL = "https://immersivecommons13.tail5da903.ts.net" $env:ANTHROPIC_AUTH_TOKEN = "agt_your_key_here" $env:ANTHROPIC_MODEL = "glm-4.6" $env:ANTHROPIC_SMALL_FAST_MODEL = "glm-4.5-air" claude # you are live Prefer bash? Git Bash takes the macOS/Linux block verbatim. Chaining: `&&` does not chain in PowerShell 5.1. Use `;` or `cmd1; if ($?) { cmd2 }`. Writing a .sh on Windows? Save it LF, not CRLF, or bash reports: bad interpreter: /usr/bin/env bash^M -------------------------------------------------------------------------------- ChromeOS (Crostini / Linux development environment) -------------------------------------------------------------------------------- Turn the container on FIRST, before anything else: Settings -> Advanced -> Developers -> Linux development environment -> On Then open Terminal. Inside it you are on Debian, so follow the Linux block: sudo apt update curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - sudo apt-get install -y nodejs node -v sudo npm install -g @anthropic-ai/claude-code export ANTHROPIC_BASE_URL=https://immersivecommons13.tail5da903.ts.net export ANTHROPIC_AUTH_TOKEN=agt_your_key_here export ANTHROPIC_MODEL=glm-4.6 export ANTHROPIC_SMALL_FAST_MODEL=glm-4.5-air claude Notes for Chromebooks: - Clone repos into ~ inside the container. Files under /mnt/chromeos are slow. - The container disk may need raising in Settings if apt runs out of space. - Low on RAM? Do NOT try to host a model locally. Use the Nebius hosted fallback from step 0. That is what the credits are for. ================================================================================ STEP 2 - WHAT TO BRING ================================================================================ - A laptop. - A repo with at least ONE test you trust. - A coding task you wish your agent could do reliably: a bug, a refactor, or a feature. Ideally one you have already fixed, so you know the answer. ================================================================================ STEP 3 - THE BENCH YOU WILL BUILD ================================================================================ You WRITE the harness during the session. There is nothing to clone: the code is on the slides and the shape is below. Typing it is the point. bench/ tasks/ 001-fix-null-deref/ prompt.md oracle.sh repo/ 002-refactor-parser/ prompt.md oracle.sh repo/ 003-add-csv-export/ prompt.md oracle.sh repo/ runner.py results/ The oracle is a test, not an opinion. Exit 0 is pass, anything else is fail, and the runner reads nothing else. Probe every oracle twice on the same code before you trust it: if it disagrees with itself, every number after that is noise. Then run each task k times (k=5 is enough for a 90 minute lab) with a FRESH repo copy per attempt, and compute both numbers: pass@k = any(attempt passed) -> "can it ever do this" (optimistic) pass^k = all(attempts passed) -> "can I leave it alone" (reliability) The gap between them is your flakiness. pass^k is the one you act on. ================================================================================ TROUBLESHOOTING ================================================================================ "claude: command not found" npm's global bin is not on PATH. `npm bin -g` prints it; add it to PATH. 401 / invalid token from the gateway Your key is not approved yet, or it expired (5 hours from approval). Ask the facilitator. Check you pasted ANTHROPIC_AUTH_TOKEN and not another variable. "export is not recognized" (Windows) You are in PowerShell. Use $env:NAME = "value", or switch to Git Bash. The variables vanished Environment variables are per window. Run `claude` in the SAME window you exported in, or export again. "bad interpreter: bash^M" A shell script was saved CRLF. Re-save it LF. pytest collected 0 tests, and the oracle still passed That is a false green and the most dangerous failure mode of the night. Make the oracle assert that it actually ran the test you meant. Everything is slow on ChromeOS Expected in the container. Keep task repos small and use Nebius for inference rather than hosting a model locally. Drop to k=3 if you must, and say your k was 3 when you quote a rate.