PyRIT: Microsoft's AI Red Teaming Tool in Security Workflows

PyRIT (Python Risk Identification Toolkit for generative AI) is Microsoft’s open-source AI red teaming framework. It was developed by Microsoft’s AI Red Team ↗ — a group with real production AI security experience — and it shows in the design: PyRIT is built for security team workflows rather than ML research.

The comparison with Garak is useful. Garak has more probes and is oriented toward comprehensive research scanning. PyRIT has better workflow integration, better result management, and was designed from the start for the use case of “security engineer running repeatable tests on AI applications.”

Architecture

PyRIT organizes attacks as orchestrators, which combine:

• Prompt targets: The LLM endpoint being tested (OpenAI, Azure OpenAI, or any API endpoint)
• Attackers: Attack strategies (prompt datasets, AI-generated variations, red team LLM)
• Scorers: Evaluation of whether the attack succeeded

A basic red team run:

from pyrit.orchestrator import PromptSendingOrchestrator
from pyrit.prompt_target import AzureOpenAIChatTarget
from pyrit.prompt_converter import TranslationConverter
from pyrit.datasets import fetch_harmbench_examples

target = AzureOpenAIChatTarget()

orchestrator = PromptSendingOrchestrator(
    prompt_target=target,
    prompt_converters=[TranslationConverter(language="Spanish")]
)

harmbench_prompts = fetch_harmbench_examples(harm_category="physical_safety")

result = await orchestrator.send_prompts_async(
    prompt_list=harmbench_prompts
)

What makes it production-friendly

Result persistence. PyRIT saves results to a database (SQLite by default, easily swapped to PostgreSQL). This means you can run scans over time and compare results — did the model’s behavior on a specific probe class change after a model update?

The memory system. PyRIT’s “memory” abstraction tracks conversation context across multi-turn attacks. This enables multi-turn attack patterns that single-turn tools can’t represent.

Score tracking. PyRIT’s scorer system lets you evaluate attack success programmatically. Built-in scorers include an LLM-based evaluation and substring match; custom scorers are easy to implement.

CI integration. PyRIT is designed to be called from a CI pipeline without a complex setup. A focused scan on a specific attack category runs in minutes, not hours.

Coverage comparison with Garak

PyRIT’s probe coverage is narrower than Garak but more curated:

• Jailbreak attacks: comparable coverage of known patterns
• Prompt injection: good coverage, multi-turn patterns
• Data leakage: more focused than Garak
• Encoding-based attacks: less comprehensive than Garak’s encoding probes
• Research-oriented probes (GCG variants, transfer attacks): Garak wins here

The breadth tradeoff: Garak is better for comprehensive vulnerability research; PyRIT is better for production security testing with defined scope.

Enterprise context

PyRIT integrates naturally with Azure OpenAI Service and Azure’s security ecosystem. For organizations running LLM applications on Azure, this is a meaningful advantage — shared identity, logging to Azure Monitor, integration with Microsoft Defender for Cloud AI security findings.

For non-Azure deployments, the Azure integration is irrelevant but not an obstacle — PyRIT works against any API endpoint.

Verdict

PyRIT is the right choice for security teams (as opposed to ML research teams) running regular assessments of LLM applications. The workflow is more polished, the results are more trackable, and the CI integration is better than research-first alternatives.

For teams wanting the broadest possible probe coverage for one-time or quarterly assessments, Garak may still be the better tool. The two are complementary rather than mutually exclusive.

We assessed both against our AI security tool evaluation framework. The comparison data on PyRIT vs. Garak vs. commercial LLM scanners is at bestllmscanners.com ↗.