All posts
-
OWASP LLM Top 10 Mitigation Guide: Controls for Every Risk Category (2025 Edition)
A practitioner's OWASP LLM Top 10 mitigation guide covering all ten 2025 risk categories — prompt injection through unbounded consumption — with concrete controls, tooling pointers, and residual-risk notes.
-
Patronus AI Review: Automated LLM Evaluation and Guardrails
A review of Patronus AI's evaluation platform — the Lynx hallucination model, the Glider custom evaluator, the built-in judge and safety evaluators, and how its self-serve API fits into an AI security stack.
-
Protect AI's ModelScan and NB Defense: Open-Source AI Supply-Chain Scanning
A hands-on review of Protect AI's two best-known open-source tools — ModelScan for model serialization attacks and NB Defense for Jupyter notebooks. What they actually detect, how to run them, and where their limits are.
-
Robust Intelligence (Now Cisco AI Defense): What the Platform Actually Covers
A conservative review of Robust Intelligence — the AI security pioneer now part of Cisco AI Defense. Algorithmic red teaming, AI Validation, model file scanning, and runtime AI Protection, with the public/gated line clearly marked.
-
PyRIT Deep Dive: Microsoft's AI Red Teaming Framework in Practice
A long-form review of PyRIT, Microsoft's open-source AI red teaming framework. Its orchestrator/target/converter/scorer/memory architecture, multi-turn attack support, result persistence, and where it fits versus garak — described from the project's own docs.
-
Garak Deep Dive: Architecture, Probes, and Operating the NVIDIA LLM Scanner
A hands-on, long-form review of garak — NVIDIA's open-source LLM vulnerability scanner. How its probe/detector/generator/buff architecture actually works, which model backends it speaks, what its reports contain, and how to operate it without drowning.
-
Giskard Review: Open-Source Testing and Evaluation for LLM and RAG Apps
A long-form review of Giskard, the open-source Python library for testing AI systems. Its automated Scan for LLM vulnerabilities, the RAGET RAG-evaluation toolkit, the giskard.Model wrapper, and where it fits beside red-team scanners like garak and PyRIT.
-
How to Evaluate AI Security Tools Without Getting Fooled
AI security tool demos are optimized for best-case scenarios. A rigorous evaluation requires adversarial test cases, production-realistic inputs, and honest accounting of false positive costs. Here's the framework.
-
PyRIT: Microsoft's AI Red Teaming Tool in Security Workflows
PyRIT is Microsoft's open-source AI red teaming framework. Built for enterprise security teams, it has better CI/CD integration than research-first tools. The tradeoff is probe breadth.
-
Guardrails AI: Output Validation That Doesn't Require Retraining
Guardrails AI provides a validation layer for LLM outputs — checking format, structure, and content without touching the model. The validator library is extensive. The performance overhead is manageable with the right configuration.
-
Arize Phoenix: LLM Observability That's Actually Free
Arize Phoenix is an open-source LLM observability platform that's evolved well beyond its origins as a drift detector. The security-relevant features — hallucination detection, retrieval quality, prompt monitoring — are production-ready.
-
Rebuff: Open-Source Prompt Injection Defense in Production
Rebuff is a self-hosted prompt injection defense with a multi-layer architecture. The heuristics layer is fast; the LLM-based detection adds coverage. Here's the production configuration that made it viable.
-
Garak LLM Scanner: Production-Grade Red Teaming or Research Tool?
Garak is the most comprehensive open-source LLM vulnerability scanner. It was designed for research. Deploying it in CI/CD requires understanding what it's good at and what it's not.
-
Lakera Guard: Prompt Injection Detection in Practice
Lakera Guard is purpose-built for prompt injection detection rather than general content moderation. After four months in production, here's where it earns its cost and where it doesn't.
-
What this site is for
AI Sec Reviews aggregates and analyzes reviews of AI security products and platforms, drawing on published benchmarks and vendor documentation, with detection and false-positive numbers in the open.