Real-world data remains the test for AI video analytics

Artificial intelligence used in video surveillance must operate in conditions that are difficult to reproduce in a laboratory: changing light, poor image quality, unusual camera angles, crowded scenes, and unexpected human behavior.
 
That makes the choice of training data central to how reliably a system performs after installation. Developers can train models using footage collected from operating environments, computer-generated images and video, or a combination of both.
 
Executives from Genetec and Eluviant said synthetic data can expand training datasets and provide examples of rare events. But they also said footage from real environments remains necessary, particularly when an application must interpret behavior or understand activity within a specific location.
 
“Real-world data comes from actual operating environments and reflects the variability that systems encounter after deployment,” said Laurent Villeneuve, Senior Manager, Product and Industry Marketing at Genetec.
 
Those variations can include lighting, weather, camera positioning, image quality, and background movement. Each can affect whether an analytics model correctly detects an object or interprets an event.
 
Synthetic data, by comparison, is created artificially. It allows development teams to produce controlled examples, change specific conditions and generate footage of incidents that may be impractical to record.
 
Villeneuve said both forms of data can contribute to model development. Their value, however, depends on the task being performed and the way the model is tested before deployment.

Where synthetic data helps

Synthetic data is particularly useful when developers need many examples of a clearly defined object or event.
 
For object detection and classification, teams can create variations involving different vehicles, clothing, backgrounds, viewing angles and environmental conditions. This can help fill gaps in datasets without requiring every possible variation to be filmed.
It can also provide examples of incidents that rarely occur in ordinary surveillance footage.
 
“Synthetic data has a genuine role,” said Callum Wilson, Co-Founder and CEO of Eluviant. “It is useful for generating large volumes of controlled examples, filling gaps in a dataset, and recreating rare or dangerous scenarios you could never responsibly film.”
This can be relevant when a model must recognize a specific object, but developers have only a limited number of real examples. A synthetic dataset can broaden the range of conditions represented during training.
 
Villeneuve said this approach can support object detection and classification, as well as applications involving events that occur too infrequently to produce enough training material.
 
Controlled generation also allows developers to isolate particular variables. They can alter lighting, object placement, or camera perspective without changing every other part of the scene.
 
The usefulness of this approach declines, however, as the task becomes more dependent on context.
 
Wilson said functions that identify discrete objects are generally better suited to synthetic expansion than applications that must understand what people are doing over time.
 
“As a rule of thumb, the more a function depends on recognizing discrete objects, the more synthetic data can help,” he said. “The more it depends on reading behavior in a live environment, the more the real world has to do the teaching.”

Behavior requires context

Behavior analysis presents a different training challenge because the meaning of an action often depends on where it occurs.
 
An individual remaining in one place could be considered normal in a waiting area but unusual near a restricted entrance. Movement that appears suspicious in one facility may be part of an ordinary workflow in another.
 
Villeneuve said tasks involving behavior analysis or site-specific processes generally require extensive real-world video because the surrounding context affects how an activity should be interpreted.
 
Testing in operational environments can also reveal weaknesses that were not apparent during development.
 
“Operational environments tend to expose gaps that were not apparent during development,” he said.
 
A model may perform well against a prepared dataset but produce too many false alarms when exposed to reflections, shadows, background movement, weather or an unfamiliar camera position.
Real-world testing is therefore needed to measure both false positives and missed events. It also helps determine whether a model can perform consistently across different sites rather than only under the conditions represented in its training data.
 
Wilson described this as the unavoidable disorder of live video.
“Video AI lives or dies on the messiness of the real world: poor lighting, crowded scenes, awkward camera angles, people doing things nobody could have anticipated,” he said.
 
For this reason, neither executive presented real and synthetic data as mutually exclusive choices.
 
Synthetic data can increase coverage and provide examples that are otherwise unavailable. Real-world data can show whether the resulting model remains useful when installed at an operating site.
 
Wilson said Eluviant prefers real-world datasets because they reflect the conditions its systems encounter. Synthetic data, however, can help broaden coverage while supporting privacy and data-protection requirements.

Errors are not limited to chatbots

The growing use of language-based interfaces in video surveillance has also prompted questions about whether camera AI can hallucinate in the way associated with large language models.
 
The comparison requires care. Traditional video analytics do not generally produce long, open-ended answers in the same way as a chatbot. They can, however, make incorrect classifications, misunderstand behavior, or assign the wrong meaning to an event.
 
Wilson said video intelligence can misclassify both objects and actions. The practical response is not to assume that every detection is correct, but to build systems that help operators assess the result.
 
“No AI is perfect, and nor is any other technology,” he said. “Video intelligence can misclassify objects and behaviors, and it can misread actions.”
 
Eluviant’s approach is to use AI to support human decisions rather than replace them. Wilson described human review as a core design principle.
 
Its Sentry software provides a confidence percentage with each detection instead of presenting every alert as certain. This gives an operator an indication of the system’s level of confidence before deciding how to respond.
 
Testing also takes place at the beginning of a project. Sensitivity settings are adjusted according to the environment and the customer’s preference for receiving more potential alerts or filtering out lower-confidence events.
Eluviant also uses a generative AI component called Aurora as another review stage. Wilson said it checks an alert against the context visible in the scene and provides a separate confidence level.
 
These controls do not eliminate the possibility of error. They are intended to make uncertainty visible and retain human judgment in the decision process.

Validation matters more than dataset size 

The interviews suggest that the amount of data used to train a model is not, by itself, a sufficient measure of quality.
 
A large synthetic dataset may improve coverage, but it cannot demonstrate how the model will respond to every installation. A real-world dataset may reflect operating conditions more closely, but it can still be incomplete or too narrow if it represents only a limited range of sites.
 
The more relevant question is whether the model performs reliably after deployment.
Villeneuve said real-world data remains essential for validation and testing, regardless of how much synthetic material was used during development.
 
“Ultimately, the most meaningful measure is how reliably a model performs under actual operating conditions, and whether it delivers the intended operational outcome,” he said.
 
That emphasizes field testing, false-alarm measurement and the ability to adapt a system to the conditions at each site. For integrators and end users, the distinction between real and synthetic data is therefore less important than the evidence a vendor can provide about performance. Synthetic data can strengthen development, particularly for object recognition and rare scenarios. Real-world footage remains necessary to test whether those gains survive contact with an operating environment.
 
As video analytics take on more context-dependent tasks, the limits of training data become harder to separate from the conditions of the deployment itself. The final test is not how convincingly a model performs in development, but whether operators can rely on its output when an actual event occurs.