How to Evaluate AI in Physical Security
The exploding AI market requires the tools to make smart choices
Organizations are adopting Artificial Intelligence to drive safer, more efficient, and more profitable operations. As they do, they face a daunting task. How is it possible to navigate a marketplace exploding with more than 80 vendors all making similar claims? How do decision-makers vet technology promises without common AI quality benchmarks? Which approaches have staying power as the market evolves? Knowing the right questions to ask and developing a framework to assess vendors are critical to making smart decisions as security operations shift to AI-based solutions.
AI is a probability machine
While there are different flavors of AI, at its core AI is a probability machine. It makes a prediction and learns from its correct guesses and its mistakes. The cycle of predicting, observing, analyzing, and learning is continuous as new training data becomes available. Efficient and highly reliable identification of people, objects and patterns lie at the heart of AI’s value to safety and security applications. Even if budgets allowed it, humans can’t possibly catch every second of potentially important activity.
Proactive threat mitigation
For the first time, it’s economically practical to proactively mitigate threats rather than just record evidence. With the help of AI, machines can alert to the events that warrant human attention and enable real-time response. AI plows through hours of information — whether access control, point-of-sale (POS) or other sensor information — to filter out the noise and isolate what’s important. AI allows better use of resources, enabling the same number of staff to cover more locations by responding to true threats, lowering costs, and increasing ROI.
For example, AI reduces the costs of false alarms by discerning which alarms are triggered by harmless Door Held Opens (DHOs) and which are breaches. So guards can be dispatched exactly when and where needed. AI also produces data, like elapsed response time and incident trend data, that are used to understand problem areas and improve security protocols. But all AI implementations are not equal. A basic understanding of a few core concepts is essential to ensure that solutions align with an organization’s requirements and that a vendor’s claims are accurate.
Understanding precision and recall: AI’s most important quality metric
Precision and recall are the most important factors in determining AI effectiveness. Precision measures the rate of false alarms. It determines the accuracy of AI in correctly identifying a true positive vs. a false positive. Recall measures the rate that AI misses things (false negatives). There is always a tradeoff between precision and recall. Improving precision reduces recall and vice versa. Academically, there may be a perfect balance. But in security, it’s often better to err on the side of false positives to avoid missing something that would require repeating the investigation.
When evaluating AI, a single number doesn’t tell the whole story. An AI vendor may tout a very high precision for facial recognition, for example, but the size of the population being matched and the pixel count on the faces affect the outcome. A test to recognize one of four people at a kiosk is much easier than recognizing one of 4,000 coming through a lobby door. Finding out how a company calculates its accuracy rates helps interpret them.
The cost vs. latency tradeoff
Clear business objectives help optimize AI solutions. With AI, speed requires more compute power, which means higher cost. Applications like automated tailgating detection require real-time notifications. But if the information required is not time sensitive — such as dwell time data that informs better retail marketing decisions — receiving the information in minutes versus seconds is a tremendous cost saver. Tuning service levels to business needs gets the most from security budgets and leads to more effective bandwidth and resource management.
Real-life examples for accurate assessments
How a system performs in real-world conditions often differs significantly from a lab environment. Lighting, camera position and camera resolution all affect AI accuracy. Some companies may not test performance using the instances of blurry video from low exposure frame rates and odd camera angles. So it’s important to test solutions using real data in your own environment. Capture video for a day and export the prediction results for a quick quality rating. Rating the quality of even 100 random samples provides a statistical confidence of +-4%, so the quality evaluation work can take less than an hour.
Asking the right questions
Getting an accurate assessment of an AI feature’s false negative and false positive rates, determining how business requirements match cost vs. latency tradeoffs, and ensuring that a vendor’s technology is flexible to meet future needs are all critical assessment criteria. So is gaining a complete picture of a solution’s total cost of ownership (TCO ). A TCO calculation should include costs such as upfront licensing fees, hardware costs, and deployment dependencies such as new cameras and access control systems.
Future-proofing through extensible AI
The longevity and flexibility of a technology also factors into decision making. Does selecting a particular technology lock an organization into proprietary hardware? Can a vendor quickly deploy new capabilities? How can analysis be extended with additional AI models? Both the pandemic and its associated supply chain problems served as a potent reminder of the value of responding quickly to new demands without waiting for new hardware and installation. If the solution can be extended with new capabilities and new models for continuous improvement, then organizations ensure there is a path for swift response to mitigate new threats.
The PoV: working backwards from the business problem
Since the choice of AI has so many variables across price, quality, latency, features, and deployment topology, it’s most effective to work backwards from the business outcomes needed to converge quickly on evaluation criteria. In addition to a framework of questions, a one-page Proof of Value (PoV) document helps everyone clarify goals. Documenting who the solution is for, why it’s needed, what it is, why it’s better, and the evaluation criteria helps to focus on the problem and the best way to solve it. A PoV helps organizations work backwards from the business problem to manage those variables for a solution with the optimal price/performance ratio. Clarity around desired business outcomes helps to define expected efficiency gains and what’s needed to justify an AI expenditure — whether it’s SOC 2 compliance or reducing unauthorized entries by 20%.
AI is not a magic bullet. But there is magic that comes from a layered approach to security. On their own, layers like guard services, access control systems, and AI are each fallible. But when used together, they produce a remarkable degree of reliability.
Navigating the fast-changing AI landscape requires familiarity with evaluating advances. For more information about how to assess AI, watch to the full “Smart Choices: How to Evaluate the Exploding AI Market” webinar.
Contact info@camio.com to learn more about Camio real-time video search and alerts.
