Edge inference
ActiveBench2Field
Measures how much of a model optimization survives contact with the robot. The model alone met its deadline on the rover; at default clocks the full frame did not.
Measured results
- Full frame on the rover, default clocks47.8 ms p95Jetson Orin NX at 26 Hz, 33.3 ms deadline, 2026-10-02
- Full frame on the rover, clocks pinned21.1 to 21.5 ms p95Retest 2026-11-11 with jetson_clocks, saved frames, one session; inside the 33.3 ms deadline
- Model alone, same board26.9 msInside the deadline, so a model-only benchmark says GO
- Frames the pipeline sustains22.9 fpsAt default clocks, against a camera delivering about 26 Hz
- Board power at 26 Hz9.2 WOrin NX, against a 15 W budget
- Alpamayo 2 Edge on AGX Thor367 ms median · 427 ms p95Case study 07, TensorRT Edge-LLM FP8, per decision against a 100 ms deadline, 2026-11-12
- SimulationBench boards: RTX 5090, AGX Thor
- PhysicalSame frames from the rover's camera
- Edge computeDecode, preprocess, infer, postprocess
- Edge computeWhole frame timed at the camera's real rate
- PhysicalDeadline verdict on the robot
Hardware
A Jetson Orin NX inside a ROSMASTER M3 Pro rover is the field board. A Jetson AGX Thor and an RTX 5090 are the bench boards. All three run the same frames and the same code, and report into one schema.
Workload
YOLOX-s, exported to FP32 ONNX and run through TensorRT, on frames from the rover’s own camera. The camera is configured for 30 fps, but with the rover’s stack running it delivered about 26 Hz, so 26 Hz is the rate the benchmark follows.
What was measured
The whole frame, not just the model: decode, preprocess, inference, postprocess and the copies in between, timed at the rate the camera really delivers, with board power and repeat-to-repeat spread.
What failed
On the rover, the model alone answered in 26.9 ms, inside the 33.3 ms deadline. The full frame took 47.8 ms at p95 (default clocks), so the robot would drop about one frame in eight. The same frames cleared the deadline on both bench boards.
Corrected 2026-11-14: the original figure was measured at default clocks; see the retest below.
Retest with the clocks pinned
The 47.8 ms figure was measured with the board’s clocks as shipped, with frequency scaling on. In a retest on 2026-11-11 the GPU clock sat mostly at 408 of 1,173 MHz in that state, and the frame again took 48.6 to 49.3 ms at p95. With jetson_clocks pinning the CPU, GPU and memory clocks at maximum, the same frame took 21.1 to 21.5 ms at p95 on the Orin NX, inside the 33.3 ms deadline. On the Jetson AGX Thor the frame went from about 14 ms to about 7 ms, and board power at 30 frames per second was 24.9 to 25.0 W at default clocks against 28.5 to 29.7 W pinned.
The cost is power. Paced at 30 frames per second, the Orin NX drew 14.5 W with the clocks pinned against 8.9 W at default clocks, about 60% more.
Limits: the retest used saved image files, not the live camera with the rover’s stack running. The three clocks were pinned together, not separated, so the share of each is not known. It was one test session per board. The records and the method are in the retest record.
What changed
Version 1.0.1 measured and diagnosed the gap. The first optimization step is now measured as well: at FP16 the same detector keeps its accuracy (mean AP 0.653 at IoU 0.5 on a 500-image COCO subset) and answers in half the time on both Jetson boards. INT8 gives up 1.5 AP points and is faster than FP16 only on the Orin NX. The rest of the plan, including the field-load replay that tests the full frame on the rover again, has not run yet.
Later case studies
The same method, applied to larger models. Each has its own record in the repository.
- 02, Alpamayo 1 (NVIDIA’s 10B driving-reasoning model): median 0.89 s per decision on the RTX 5090 with flash-attn (0.93 s with sdpa) against 4.38 s on the Jetson AGX Thor, on the same eight clips.
- 03, DeepStream 7.1 on the rover’s Orin NX, YOLOv8n in TensorRT FP16, from the rover’s camera and a recorded clip.
- 04, Isaac ROS 4.5 object detection, measured on the RTX 5090 as a lower bound. The Thor run is still to come.
- 05, Cosmos Predict 2.5: one video forecast took 773.9 s and 30.9 GiB at peak on the RTX 5090.
- 06, Alpamayo 2 Edge in PyTorch: on the Thor at MAXN, a 10 Hz stream at 0.331 s median and 0.544 s p95 per decision with NVIDIA’s acceleration, FP8, five diffusion steps and vision reuse together. The 0.1 s deadline is missed on both boards.
- 07, Alpamayo 2 Edge with NVIDIA’s TensorRT Edge-LLM recipe on the Thor: FP8 engines at 367 ms median and 427 ms p95 per decision, still above the 0.1 s deadline. NVFP4 was slower on this stack. The report lists seven issues and their fixes.
Result
A benchmark that times only the model would have shipped this detector without knowing which clock state the robot runs in. The bench-to-field difference is the finding, and every number traces to a committed run.
Evidence
