Corvus ISR Publishes Transparent Benchmark of Tracker Models

Corvus ISR, a leader in wide-area motion imagery (WAMI) exploitation, has released a public tracker benchmark that provides an unprecedented level of transparency and reproducibility. This benchmark compares two distinct tracker models on an identical synthetic scene, with perfect ground truth, ensuring that results are purely due to algorithmic differences rather than external variables.

The first model, v1, employs a simple greedy nearest-neighbour approach with two-pass association, constant-velocity prediction, and fixed 2-second coasting. Despite its simplicity, it serves as a baseline that can run in the archived demo slices 1-2, providing a clear point of comparison for more complex systems. The second model, v2, introduces a sophisticated confirmed-track auction mechanism with three-tier association, velocity-consistency gating, and noise-scaled reservation pricing, available in demo slice 3.

The results demonstrate significant improvements with v2: in the baseline scenario with 150 movers at 2 fps, ID switches per minute drop from 2,042 to 1,183, a reduction of 42.1%. Under high-density conditions of 400 movers, the decrease is from 14,032 to 8,040, a 42.7% reduction. Even in challenging settings such as sensor frame-starvation at 0.5 fps or with 20% occlusion, v2 outperforms v1, reducing ID switches by approximately 18% across these scenarios. It’s important to note that detection rates are held constant, as they are purely a sensor property.

The benchmark uses a strict ID switch metric that counts every change in the track identity assigned to a ground-truth object, including fragmentations and re-acquisitions—making it more demanding than conventional MOT challenge definitions. This rigorous metric underscores the fact that even state-of-the-art models still commit thousands of identity errors per minute under stress, emphasizing the need for honest measurement rather than marketing claims.

Corvus ISR publishes these failure numbers deliberately, leveraging synthetic scenes with perfect ground truth to produce honest, measurable data. Every future tracker must be publicly benchmarked against this same seed, ensuring transparent comparison and progress. As the adage goes, “Vendors who show only successes ask for faith; a published failure matrix asks for measurement.”

From an engineering perspective, v2 achieves impressive efficiency, averaging about 1.2 milliseconds per sensor tick at a density of 400 objects—well within real-time constraints, with worst-case performance around 5ms against a 10ms budget. All results are fully reproducible by anyone through the live demo, where users can press “Run benchmark” without signup or NDA to verify every row.

This entire process is built on a fully synthetic environment—no real persons, vehicles, or locations. Each pixel is computer-generated, ensuring consistent and perfect ground truth for honest evaluation. Furthermore, the v2 tracker was developed by an AI executor, built against a detailed acceptance contract and subjected to independent review before release. The emphasis on fixed-seed reproducibility, byte-identical harnesses, and transparent metrics exemplifies rigorous engineering discipline.

Readers interested in exploring the performance themselves are encouraged to run the benchmark firsthand. Visit the public benchmark page or try to reproduce it live—a fully synthetic, open, and accessible process designed for transparency and honest measurement.

Powered by Thorsten Meyer AI