Did it work?
RoboJev-27B estimates whether a robot completed its task. Give it the instruction and ten frames, plus any final auxiliary-camera views. It returns P(success) in a median 0.26 seconds on one H100.
Ring on the peg. Ring slips off.
Same instruction, on a task held out of training. The overlay shows RoboJev’s answer every half second, from the frames up to that moment.
How it compares
95.2% balanced accuracy on 268 held-out SO-101 runs, at an estimated $0.19 per 1,000 answers. 0.890 AUROC on 459 runs from eight held-out lab/robot domains.
Loading results…
All 29 model and protocol entries, including earlier variants
Loading results…
View individual answers, compare any two models and download the data →
What was measured
SO-101: 268 held-out runs across eight tasks, with 80 runs from two entirely held-out tasks and 188 held out by session. Unseen domains: 459 runs from eight lab/robot domains. Community: 135 clips with a clear outcome, out of 200 labelled by one person. All community evaluation owners are excluded from fitting.
RoboJev and the matched JEV baseline use ten primary-camera frames plus available final auxiliary views. Frontier APIs use four primary-camera frames. The table contains 29 recorded model/protocol entries, including earlier variants; the API comparisons use different amounts of visual evidence.
Balanced accuracy averages success and failure recall at a 0.5 threshold. AUROC is how often a success scores above a failure.
What the differences show
Against GPT-6 Astra, paired bootstrap, 95% intervals: unseen-domain AUROC +0.028 (+0.005 to +0.051); SO-101 balanced accuracy +0.019 (−0.016 to +0.052), which does not resolve a difference. These API comparisons use unequal visual inputs. All paired results are in the metrics file.
Where it doesn’t win: Opus 5.5 has higher community accuracy, and Astra, Sol and Opus have higher community AUROC. The base model with the same ten frames scores 0.900 on unseen labs, compared with RoboJev’s 0.890; the paired interval includes zero. From last frames only, the difference from the previous release is also unresolved (0.776 vs 0.779).
501 hours is already a lot of watching.
We indexed 68,555 episodes in 7,641 public LeRobot datasets: 501 hours of video. One answer per episode at the measured speed is about 4.9 GPU-hours, $13.16 on an H100 at $2.69 an hour. At GPT-6 Astra’s API price: $637.
These are compute estimates, excluding media processing, networking, allocation and idle time. We have not regraded the catalogue with this release.
Read the paper
RoboJev: An Open Vision-Language Judge for Robot Success Detection
Josef Chen · Independent Researcher · October 2026
From real robot attempts to a success probability. The paper documents the architecture, training mixture and calibration, then examines where adaptation helps and where it falls short.
Five scientific figures accompany the recorded evaluation: 20,628 predictions across 29 model/protocol entries. The source package includes the figures and bibliography; the reproducibility bundle includes the plotting inputs and scripts.
Reproduce the figures and tables · BibTeX citation
Preprint. An arXiv link will be added when an identifier is assigned.
The model
JEV-27B-VL, adapted with a rank-32 language LoRA on 52,598 training episode records while the vision encoder stays frozen. The record count is not an independently deduplicated count of physical attempts. It reads the task and ten evenly spaced frames, plus the last frame of each other camera, and returns P(success) with the calibration in decision.json. 54.7 GB in bf16; one 80 GB GPU.
vllm serve josefchen/RoboJev-27B --served-model-name robojev \
--max-model-len 32768 --limit-mm-per-prompt '{"image": 16}' \
--logprobs-mode processed_logprobs --trust-request-chat-template
from robojev import RoboJev
judge = RoboJev("http://localhost:8000")
judge.done("Put the ring on the peg", "attempt.mp4")
robojev_grade.py grades a LeRobot dataset, robojev_overlay.py draws the score on a video or camera feed, and the LeRobot plugin adds a reward model and reward step.
Where it can go wrong
Ten frames can miss a brief button press or an object dropped between frames. Long tasks, wrist-only footage and recordings that stop before the outcome are hard. A confident answer can still be wrong.