---
title: "Human Activity Recognition Datasets Compared"
description: "What a human activity recognition dataset is, how HAR clips are labelled, the benchmarks teams use, and how synthetic video covers rare events."
canonical: https://simuletic.com/guides/human-activity-recognition-dataset
html: https://simuletic.com/guides/human-activity-recognition-dataset
site: Simuletic, https://simuletic.com/llms.txt
---

# Human activity recognition datasets: the benchmarks, how they are labelled, and where they fall short

What HAR data actually looks like, how it gets labeled, why the public benchmarks stop being useful the moment you deploy, and how synthetic video covers the activities that matter most.

## What a human activity recognition dataset is

A human activity recognition dataset is a set of clips — or skeleton sequences, or sensor traces — where each sample carries a label describing what the person is doing. In computer vision the label is usually an activity class plus a temporal window: *fall, 3.2s–5.8s*. That temporal boundary is the part teams underestimate. Getting the class right is easy; getting the onset right is what makes an alarm useful.

The clip below is one of ours. A synthetic person collapses on a corridor camera; nobody was filmed and nobody was hurt. Clips like this are made to order; the catalogue datasets are labelled images, which is what most CCTV alerting models train on.

Dataset for the most-requested activity

CCTV Fall & Incident Detection Dataset

1,000 images · 2 classes (person, fallen_person) · YOLO + COCO

Synthetic HAR sample: medical collapse captured from a fixed indoor camera, made in our pipeline.

## HAR vs action recognition

People use the terms interchangeably and then get confused about why their model underperforms. The distinction is practical:

- **Action recognition** — short atomic movements, one class per trimmed clip. A wave, a kick, a jump. Benchmarks: UCF101, HMDB51, Kinetics.

- **Human activity recognition** — longer, compound behavior that unfolds over time and often needs temporal segmentation on untrimmed footage. Loitering, shoplifting, a fall and its aftermath.

- **Video anomaly detection** — the unsupervised cousin, where "abnormal" is anything outside learned normal behavior. Useful when you cannot enumerate your classes.

If your product raises an alert, you are almost certainly doing HAR on untrimmed video, not clean action recognition. Read the wider framing in our [video datasets for computer vision guide](https://simuletic.com/guides/video-datasets-for-computer-vision).

## How HAR data gets labeled

Three approaches dominate, and they cost wildly different amounts:

- **RGB clip labels.** A human watches and tags. Cheapest per clip, weakest temporal precision, and painful at scale.

- **Skeleton-based.** A pose estimator extracts 17 keypoints per frame and the classifier runs on joint trajectories. Robust to lighting and clothing, and the standard for NTU RGB+D-style work — but it inherits every pose-estimation error.

- **Generated labels.** In synthetic pipelines the generator already knows where each person is and what they are doing, so labels come out with the image and stay consistent. They still need checking: we verify every set automatically for object presence and box placement and review it visually.

Field note

On our own fall-detection work, the single biggest driver of false alarms was not the fall class — it was the negatives. Sitting down quickly, bending to pick something up, and lying down deliberately all look like a fall to an under-trained model. Adding labelled look-alike negatives did more for precision than adding more falls, which is why about a fifth of our fall dataset is negatives.

## The public HAR benchmarks compared

The four datasets most HAR and surveillance papers report on. Facts come from each paper or project page; the last column is our view for teams building CCTV products.

| Dataset | Clips | Classes | Camera type | Licence | Useful for CCTV? |
| --- | --- | --- | --- | --- | --- |
| [UCF101 (2012)](https://www.crcv.ucf.edu/research/data-sets/ucf101/) | 13,320 clips | 101 actions (sports, instruments, body motion) | YouTube, mostly handheld | No licence stated; clips are third-party YouTube videos | Low: wrong viewpoint, few safety-relevant classes |
| [Kinetics-400 (2017)](https://github.com/cvdfoundation/kinetics-dataset) | ~306K clips of ~10 s at release (246K train); fewer still online | 400 human actions | YouTube, mostly handheld | No licence in the official readme; videos are third-party YouTube content | Low for training, good for pre-training a video backbone |
| [NTU RGB+D 60 / 120 (2016 / 2019)](https://rose1.ntu.edu.sg/dataset/actionRecognition/) | 56,880 / 114,480 samples | 60 / 120 (daily, mutual and health-related, incl. falling) | Three Kinect v2 sensors in a lab; RGB, depth, IR, 3D skeletons | Academic research only; commercial use needs ROSE Lab permission | Medium for skeleton models; lab viewpoint and actors, not CCTV |
| [UCF-Crime (2018)](https://www.crcv.ucf.edu/projects/real-world/) | 1,900 untrimmed videos, ~128 h | 13 anomalies (fighting, shoplifting, robbery…) + normal | Real CCTV from YouTube and LiveLeak | Research use only, per the project page | High viewpoint match, but research-only and weakly labelled |

Sources: Soomro et al., [UCF101](https://arxiv.org/abs/1212.0402) (2012); Kay et al., [The Kinetics Human Action Video Dataset](https://arxiv.org/abs/1705.06950) (2017) and [the Kinetics-700-2020 note](https://arxiv.org/abs/2010.10864) for split sizes; Shahroudy et al., [NTU RGB+D](https://arxiv.org/abs/1604.02808) (CVPR 2016); Liu et al., [NTU RGB+D 120](https://arxiv.org/abs/1905.04757) (2019); Sultani et al., [UCF-Crime](https://arxiv.org/abs/1801.04264) (CVPR 2018).

## Where the public benchmarks fall short

UCF101, Kinetics, NTU RGB+D and UCF-Crime are great for publishing a number. They are poor for shipping a safety system. The activities you actually care about — a collapse, a weapon draw, a hold-up — appear in a handful of clips (UCF-Crime has 50 fighting and 50 shoplifting videos), mostly filmed from handheld or web video, at framing that looks nothing like a ceiling-mounted camera in a corridor. Licences are the other filter: NTU RGB+D and UCF-Crime are research-only. The mismatch in camera height, lens, compression, and lighting is usually a larger accuracy tax than model choice. We break the tradeoffs down in [synthetic data vs real data](https://simuletic.com/guides/synthetic-data-vs-real-data-comparison).

## The activity clusters with real demand

These are the HAR classes buyers ask us for. Each links to the dataset you can license today, with the write-up as background.

### [Falls and medical collapse](https://simuletic.com/datasets/cctv-fall-incident-detection-dataset)

People lying on the floor after a fall, next to look-alikes (sitting, kneeling, crouching) in care, hospital, retail and workplace scenes.

### [Aggression and fighting](https://simuletic.com/datasets/cctv-aggressive-pose-detection-dataset)

Punches, kicks, shoves and grapples with COCO-17 keypoints, plus hugs, dancing and arguments as look-alike negatives.

### [Retail loss behaviours](https://simuletic.com/datasets/cctv-shoplifting-detection-dataset)

Concealment and pocketing from overhead store cameras, with 17-keypoint pose and video sequences.

### [Hold-up and robbery](https://simuletic.com/datasets/cctv-atm-robbery-detection-dataset)

Offender, victim, gun and knife labelled separately at ATMs, so a model learns who is threatening whom.

### [Driver and operator state](https://simuletic.com/datasets/driver-drowsiness-detection-dataset)

Open, closed and drowsy eyes with eye keypoints for in-cabin driver monitoring.

### [Pedestrian attributes](https://simuletic.com/datasets/pedestrian-attribute-recognition-dataset)

Clothing, carried items, age and gender attributes from overhead CCTV, for re-identification and search.

## Building a HAR dataset that survives deployment

- **Fix the camera first.** Mount height, tilt, lens, and resolution should match the deployment before a single clip is rendered.

- **Enumerate the negatives.** For every positive class, list the three behaviors that look like it and generate them too.

- **Vary the nuisance factors.** Lighting, clothing, body type, occluding furniture, crowd density. This is where synthetic pipelines quietly win.

- **Label temporally, not just categorically.** Onset accuracy determines alert latency.

- **Validate on real footage.** Train mostly synthetic, hold out a small real set from your own cameras for honest evaluation.

## How Simuletic ships HAR data

The [catalogue datasets](https://simuletic.com/datasets) are labelled images from CCTV viewpoints: YOLO boxes (most also in COCO), COCO-17 keypoints for the fight dataset, and a fixed train/val split. The shoplifting dataset adds video sequences. Image datasets are licensed as 1,000 images, with a free sample for each.

Temporal HAR data (video clips with per-frame boxes, tracking IDs and event start/end times) is made to order on your own camera views; see [custom data](https://simuletic.com/custom-data).

## FAQ

**What is a human activity recognition dataset?**

A human activity recognition (HAR) dataset is a collection of video clips, skeleton sequences or sensor traces labelled with what a person is doing: walking, falling, fighting, reaching, loitering. Vision HAR datasets pair each clip with an activity class and, ideally, the time the activity starts and ends.

**What is the difference between HAR and action recognition?**

Action recognition classifies short movements in a trimmed clip, such as a wave or a jump. Human activity recognition covers longer behaviour on untrimmed video (loitering, shoplifting, a fall and its aftermath) and usually needs the start and end time, not just a clip label.

**Which human activity recognition datasets are most used?**

UCF101, HMDB51 and Kinetics-400/700 for action recognition, NTU RGB+D 60/120 for skeleton-based HAR, and UCF-Crime for incidents on surveillance video. They are good benchmarks but weak for safety products: rare activities have few examples and the camera angle rarely matches a ceiling-mounted CCTV camera.

**Which HAR dataset is best for CCTV?**

Of the public sets, UCF-Crime is closest to CCTV because its videos come from surveillance cameras, but it is research-only and labelled per video. For a product, train on data from your own camera angle for the activities you alert on, and keep public sets for pre-training and comparison.

**Can I use Kinetics or NTU RGB+D in a commercial product?**

NTU RGB+D is released for academic research only, and commercial use needs the ROSE Lab's permission. Kinetics clips are third-party YouTube videos with no licence in the official readme. Get legal advice before shipping a model trained on either.

**How is a synthetic HAR dataset labelled?**

The generation pipeline knows where each person is and what they are doing, so boxes, keypoints and activity classes come out with the image. Simuletic still checks every set automatically for object presence and box placement and reviews it visually, because generation can place a box wrong.

**Do Simuletic datasets include temporal activity labels?**

The catalogue datasets are labelled images (boxes, and keypoints for the fight dataset); the shoplifting dataset adds video sequences. Video with per-frame boxes, tracking IDs and event start/end times is made to order on your own camera views.

**Is synthetic HAR data GDPR compliant?**

Every person in a synthetic image is computer-generated, so no real person is filmed and no biometric data is collected to make it. That removes the consent and retention questions that come with training on real CCTV footage; your own real test footage is still personal data.

Get a HAR dataset

CCTV Fight & Aggression Detection Dataset

1,000 images · 2 classes (person, fight) · YOLO-pose + COCO

### Need an activity class nobody has filmed?

Send us the behaviour, a few stills of your empty camera views and the label format. We make synthetic HAR images and clips with per-frame labels.

Written by

Fredrik Hegardt, founder of Simuletic

Fredrik founded Simuletic and runs it day to day: the generation pipeline, each dataset release and every customer conversation. Fredrik's background is in computer-vision and video-analytics projects, where the shortage of usable training data first became a daily problem.

---
Markdown version of https://simuletic.com/guides/human-activity-recognition-dataset. Prices in USD, excl. VAT. Contact: contact@simuletic.com