Synthetic Eyes: Fine-Tuning VLMs for Threat Detection with Grounding Data

If you've ever tried to build a safety or security AI, you know the biggest bottleneck isn't the model architecture—it's the data. Specifically, dangerous data.

Collecting thousands of real-world images of weapons in public spaces is not only difficult but fraught with privacy and safety issues. You can't exactly walk around a city center recording "test scenarios" with a knife without getting arrested.

At Simuletic, we focus on bridging this gap with synthetic data. Today, I want to share a behind-the-scenes look at my latest project: creating a fully synthetic dataset of knives and weapons to fine-tune modern Vision Language Models (VLMs) like Qwen2-VL.

The Problem with "Standard" VLM Training

Most off-the-shelf VLMs (like GPT-4o or standard LLaVA) are generalists. They are great at describing a sunset or reading a receipt, but they often hesitate or hallucinate when asked to perform strict security tasks. They might say a scene is "safe" because the weapon is small, or they might refuse to answer entirely due to generic safety filters.

To fix this, I needed to fine-tune a model specifically to act as a specialized security guard: sharp, concise, and accurate.

Building the Dataset: From YOLO to VLM

I started by generating a synthetic dataset of scenes containing people (Class 0) and weapons (Class 1). The raw output gave me standard YOLO-format labels—simple bounding boxes.

In the world of traditional Object Detection (like YOLOv8), that's enough. But for a VLM, we need language.

I wrote a custom pipeline to convert these numerical bounding boxes into a conversation format that a VLM understands. Instead of just teaching the model to output "Danger," I implemented visual grounding.

The difference is huge:

By forcing the model to output the coordinates of the threat, we significantly reduce hallucinations. If the model can't point to it, it likely isn't there.

The Stack: Training on the Edge

What's exciting about 2025 is that you no longer need a data center to do this. I'm running these experiments locally on consumer hardware (specifically the new RTX 5070).

Using tools like Unsloth for efficient backpropagation, I'm able to take a 2B parameter model (like the Qwen-VL series), load it in 4-bit quantization, and fine-tune it on my synthetic "Security Guard" dataset in under an hour.

Why Synthetic Data Wins Here

The beauty of using Simuletic's synthetic data generation is the control. I can generate edge cases that rarely happen in real life but are critical for safety:

• A knife partially obscured by a jacket.
• Metallic objects that look like weapons but aren't (to reduce false positives).
• Low-light scenarios.

By controlling the lighting, angles, and occlusions in the simulation, I can create a robust dataset that outperforms small collections of scraped real-world images.

What's Next: From Vision to Action

This VLM is just the brain. The next step is connecting this "eye" to a "hand." I'm currently experimenting with Vision-Language-Action (VLA) models to not just detect the threat, but to train a robot arm to react to it.

We are entering an era where synthetic data isn't just a backup—it's the primary way we will train embodied AI to handle the dangerous, the rare, and the complex.

Stay tuned for the benchmarks.