Simuletic

    Case studies

    Real tests, real footage, full tables.

    What our synthetic data and models do on footage they never trained on, with the method, the misses and the conditions we have not measured yet. Our own tests first; customer tests as they agree to publish.

    Published

    1 study so far

    Real CCTV test frame with the model's box around a person lying on the floor
    Own testFalls

    A lying-on-the-ground detector trained only on synthetic CCTV images, tested on real footage from two homes

    We trained D-FINE, RF-DETR and YOLO on our 1,000-image Fall dataset and nothing else, then scored them on two public real-footage sets. The synthetic-only D-FINE Small catches 97% of people on the floor at 5% false alarms; the model trained on real footage from one room catches 24% of falls in another home.

    97.4%
    lying frames caught at 5% false alarms (real footage)
    24% → 83–87%
    falls caught in a new home: real-only vs synthetic-only
    15
    model and data combinations scored
    Read the study

    How a study is run

    Same rules every time

    1. 01

      Question

      One state or class, one deployment question: does training on our images make a model that works on footage it never saw?

    2. 02

      Setup

      Public real test sets with usable licences, or customer footage under a scoring-only agreement. Nothing from the test set is ever trained on.

    3. 03

      Scoring

      Operating points buyers use: share caught at 5% and 1% false alarms, clip-level alarm rates, mean and range over seeds.

    4. 04

      Report

      Full table, the misses, the false alarms, data licences and a list of what was not measured. Scoring code published.

    FAQ

    About the case studies

    What counts as a case study here?

    A measured test with a stated method: which model, which data, which real footage, which metrics, and what was not measured. Marketing claims without a test set do not qualify, including our own.

    Do you publish customer tests?

    Yes, when the customer agrees. Customer footage is scored under a scoring-only agreement and never leaves their premises or ours; the case study shows the numbers and the method, not the footage, unless they choose otherwise.

    Why show the models' failures?

    Because that is what a buyer has to plan for. Every study lists the misses, the false alarms and the conditions we have not measured yet, and the real test data is public so anyone can rerun the scoring.

    Want your detector in the next study?

    Send us the alarm it keeps getting wrong. We score it on our stress set and, under a scoring-only agreement, on your footage, and publish only what you approve.

    False-Positive Stress Test Talk to us