Human activity recognition datasets: the benchmarks, how they are labelled, and where they fall short

What a human activity recognition dataset is

A human activity recognition dataset is a set of clips — or skeleton sequences, or sensor traces — where each sample carries a label describing what the person is doing. In computer vision the label is usually an activity class plus a temporal window: fall, 3.2s–5.8s. That temporal boundary is the part teams underestimate. Getting the class right is easy; getting the onset right is what makes an alarm useful.

The clip below is one of ours. A synthetic person collapses on a corridor camera; nobody was filmed and nobody was hurt. Clips like this are made to order; the catalogue datasets are labelled images, which is what most CCTV alerting models train on.

HAR vs action recognition

People use the terms interchangeably and then get confused about why their model underperforms. The distinction is practical:

• Action recognition — short atomic movements, one class per trimmed clip. A wave, a kick, a jump. Benchmarks: UCF101, HMDB51, Kinetics.
• Human activity recognition — longer, compound behavior that unfolds over time and often needs temporal segmentation on untrimmed footage. Loitering, shoplifting, a fall and its aftermath.
• Video anomaly detection — the unsupervised cousin, where "abnormal" is anything outside learned normal behavior. Useful when you cannot enumerate your classes.

If your product raises an alert, you are almost certainly doing HAR on untrimmed video, not clean action recognition. Read the wider framing in our video datasets for computer vision guide.

How HAR data gets labeled

Three approaches dominate, and they cost wildly different amounts:

• RGB clip labels. A human watches and tags. Cheapest per clip, weakest temporal precision, and painful at scale.
• Skeleton-based. A pose estimator extracts 17 keypoints per frame and the classifier runs on joint trajectories. Robust to lighting and clothing, and the standard for NTU RGB+D-style work — but it inherits every pose-estimation error.
• Generated labels. In synthetic pipelines the generator already knows where each person is and what they are doing, so labels come out with the image and stay consistent. They still need checking: we verify every set automatically for object presence and box placement and review it visually.

The public HAR benchmarks compared

The four datasets most HAR and surveillance papers report on. Facts come from each paper or project page; the last column is our view for teams building CCTV products.

Sources: Soomro et al., UCF101 (2012); Kay et al., The Kinetics Human Action Video Dataset (2017) and the Kinetics-700-2020 note for split sizes; Shahroudy et al., NTU RGB+D (CVPR 2016); Liu et al., NTU RGB+D 120 (2019); Sultani et al., UCF-Crime (CVPR 2018).

Where the public benchmarks fall short

UCF101, Kinetics, NTU RGB+D and UCF-Crime are great for publishing a number. They are poor for shipping a safety system. The activities you actually care about — a collapse, a weapon draw, a hold-up — appear in a handful of clips (UCF-Crime has 50 fighting and 50 shoplifting videos), mostly filmed from handheld or web video, at framing that looks nothing like a ceiling-mounted camera in a corridor. Licences are the other filter: NTU RGB+D and UCF-Crime are research-only. The mismatch in camera height, lens, compression, and lighting is usually a larger accuracy tax than model choice. We break the tradeoffs down in synthetic data vs real data.

The activity clusters with real demand

These are the HAR classes buyers ask us for. Each links to the dataset you can license today, with the write-up as background.

Building a HAR dataset that survives deployment

1. Fix the camera first. Mount height, tilt, lens, and resolution should match the deployment before a single clip is rendered.
2. Enumerate the negatives. For every positive class, list the three behaviors that look like it and generate them too.
3. Vary the nuisance factors. Lighting, clothing, body type, occluding furniture, crowd density. This is where synthetic pipelines quietly win.
4. Label temporally, not just categorically. Onset accuracy determines alert latency.
5. Validate on real footage. Train mostly synthetic, hold out a small real set from your own cameras for honest evaluation.

How Simuletic ships HAR data

The catalogue datasets are labelled images from CCTV viewpoints: YOLO boxes (most also in COCO), COCO-17 keypoints for the fight dataset, and a fixed train/val split. The shoplifting dataset adds video sequences. Image datasets are licensed as 1,000 images, with a free sample for each.

Temporal HAR data (video clips with per-frame boxes, tracking IDs and event start/end times) is made to order on your own camera views; see custom data.

FAQ