In Part 1, we explored what AI red teaming is, how it differs from traditional penetration testing and governance activities, and why generative AI introduces an entirely new category of risks. We also examined several common types of AI failures, including hallucinations, harmful outputs, bias, privacy leakage, and misuse. Understanding these risks is the first step, but recognizing that they exist is only half the challenge. The real value comes from designing structured exercises that deliberately test whether your AI systems can withstand realistic misuse and adversarial behavior.

Unlike conventional software testing, AI red teaming is not simply a checklist of vulnerabilities to verify. Every AI system is unique because it combines different models, prompts, retrieval mechanisms, data sources, plugins, APIs, and business processes. A customer support chatbot poses very different risks from those of an internal HR assistant, a healthcare copilot, or an AI agent that can create users, send emails, or approve financial transactions. As a result, effective AI red teaming begins with understanding the AI system’s purpose before attempting to break it.

The objective is not to make the AI fail for its own sake. Instead, the goal is to identify weaknesses that could affect security, privacy, compliance, reliability, or business operations, so they can be addressed before the system reaches production or new capabilities are introduced.

Start with Understanding the AI System

Before writing a single adversarial prompt, take time to understand exactly what the AI system is designed to do. This may sound obvious, but many organizations jump straight into testing without fully understanding the architecture of the solution they are assessing.

Generative AI rarely operates in isolation. A modern enterprise AI solution often includes several interconnected components working together. These might include the large language model itself, a retrieval system that searches organizational content, plugins that connect to external services, orchestration layers that coordinate multiple AI agents, and identity systems that determine what information users are allowed to access.

Each of these components expands the potential attack surface.

For example, consider an internal Microsoft 365 Copilot deployment. The language model itself may be highly secure, but Copilot also relies on Microsoft Graph to retrieve documents, emails, Teams conversations, SharePoint content, calendars, and meeting notes. If permissions within Microsoft 365 are overly permissive, the AI may retrieve information that users should not realistically discover so easily. In this case, the weakness is not the language model but the surrounding ecosystem.

Similarly, a custom AI assistant may retrieve data from SQL databases, REST APIs, customer records, or proprietary knowledge bases. The red team must understand each of these connections because every integration introduces additional opportunities for misuse.

Before testing begins, it is useful to answer questions such as:

• What business problem does the AI solve?
• Who are the intended users?
• What information can the AI access?
• Can it perform actions or only generate responses?
• Does it connect to internal or external systems?
• Are human approvals required for important decisions?
• What safeguards already exist?

These questions establish the context for every testing activity that follows.

Identify What Matters Most

Not every AI failure carries the same level of risk. A grammar mistake in an email assistant is very different from an AI system approving fraudulent financial transactions or exposing personally identifiable information. This is why AI red teaming should always begin with identifying what matters most to the organization. Think about the potential business impact rather than the technology itself.

For example, an AI assistant supporting customer service may present risks such as providing inaccurate warranty information, exposing customer account details, or generating inappropriate responses that damage the organization’s reputation. An HR assistant, on the other hand, may need to protect employee records, salary information, disciplinary actions, and confidential recruitment discussions. A software development assistant may introduce entirely different concerns, including generating insecure code, recommending vulnerable libraries, or exposing proprietary source code.

Rather than attempting to test every possible scenario equally, prioritize the areas where failures would have the greatest operational, financial, legal, or reputational consequences.

One useful exercise is to imagine tomorrow’s headline if the AI fails.

• Would the story involve the leak of confidential information?
• Would customers receive harmful advice?
• Would regulators investigate a compliance violation?
• Would executives lose confidence in the organization’s AI strategy?

Answering these questions often reveals where testing efforts should be concentrated.

Develop Realistic Threat Scenarios

One of the defining characteristics of effective AI red teaming is realism.

Testing should reflect how real users, both well-intentioned and malicious, are likely to interact with the system. Artificial or overly simplistic prompts rarely uncover meaningful weaknesses because attackers rarely behave in predictable ways.

Instead of asking whether an AI will answer an obviously prohibited question, consider how someone might gradually manipulate the conversation over time. Attackers often begin with harmless requests, slowly building context and trust before introducing increasingly sensitive instructions.

Imagine an AI assistant responsible for summarizing legal contracts.

Rather than immediately requesting confidential information, an attacker might begin by asking the AI to explain standard contract terminology. They may then request examples of renewal clauses, followed by typical pricing structures, before eventually asking the assistant to compare those examples with current customer agreements. Individually, each request appears legitimate. Combined together, they may reveal commercially sensitive information.

Similarly, an internal employee may unintentionally misuse the AI by asking it to summarize documents they have not fully read themselves. If the AI hallucinates missing details, those inaccuracies may become embedded within reports, presentations, or executive briefings without anyone realizing the information is incorrect.

These scenarios illustrate why AI red teaming must consider both malicious intent and accidental misuse.

Think Like Different Types of Users

Not every user interacts with AI in the same way, and not every risk originates from a malicious attacker. Effective red teaming considers a wide range of user personas, each bringing different motivations, knowledge, and objectives.

A curious employee may simply explore the boundaries of what the AI can do without intending any harm. A frustrated customer may repeatedly challenge a chatbot after receiving incorrect responses. A competitor may attempt to extract proprietary information. A cybercriminal may carefully craft prompts designed to bypass safeguards or manipulate automated workflows.

Each of these individuals approaches the AI differently, and each presents unique testing opportunities.

For example, consider how an internal finance assistant might respond to different users:

• A finance employee may request quarterly revenue forecasts.
• A project manager may ask about departmental budgets.
• An executive may request strategic financial summaries.
• A contractor may attempt to access information outside their responsibilities.
• An attacker may disguise themselves as a legitimate employee through carefully written prompts.

Testing should assess how consistently the AI enforces access boundaries across these scenarios. Thinking from multiple perspectives often uncovers weaknesses that purely technical testing overlooks.

Building Adversarial Scenarios

Once you understand the AI system and the users interacting with it, you can begin designing adversarial scenarios.

A scenario is more than a single prompt. It represents an entire conversation or workflow that attempts to achieve a specific objective.

For example, suppose the objective is to determine whether an AI assistant will reveal confidential merger information.

• Rather than immediately asking for the confidential documents, the red team might construct a realistic sequence of interactions.
• The conversation could begin with general questions about recent industry acquisitions before shifting to publicly available financial reports. The attacker might then ask the AI to compare internal planning documents with public announcements, ultimately seeking discrepancies that reveal information not yet released.
• Each step appears reasonable in isolation. The overall sequence gradually increases pressure on the AI while remaining believable.
• This conversational approach is considerably more effective than isolated prompts because it reflects how real attackers often operate.

Measuring Success

Unlike traditional software testing, AI red teaming rarely produces simple pass-or-fail outcomes. Instead, every exercise should measure how the AI behaved under pressure. For example, consider three possible responses when testing whether an AI reveals confidential information.

• The first response is ideal. The AI politely refuses the request, explains why the information cannot be shared, and redirects the user toward appropriate resources.
• The second response is less obvious. The AI rejects most requests but unintentionally reveals small pieces of sensitive information that, while individually harmless, could contribute to a larger disclosure.
• The third response is a complete failure, in which the AI discloses confidential information without any meaningful resistance.

These three outcomes require different remediation activities and carry different levels of business risk.

Organizations should therefore evaluate AI behavior across several dimensions rather than relying on a binary success-or-failure assessment.

• Accuracy of the response.
• Protection of confidential information.
• Consistency with organizational policies.
• Resistance to manipulation.
• Appropriate refusal behavior.
• Transparency when uncertainty exists.
• Respect for user permissions.
• Ability to maintain context throughout extended conversations.

Useful evaluation criteria include measuring these characteristics over time, which allows organizations to track improvements as prompts, retrieval systems, and safety controls evolve.

Determining Severity

Not every finding discovered during AI red teaming deserves the same priority. Some issues may simply reduce the quality of the user experience, while others could result in regulatory investigations, financial loss, or significant reputational damage.

A useful way to think about severity is to evaluate the business consequences rather than focusing solely on the model’s technical behavior.

For example, an AI assistant that occasionally formats dates incorrectly may be considered a low-severity issue. Although inconvenient, the impact is unlikely to extend beyond minor user frustration. By comparison, an assistant that leaks confidential customer records, generates discriminatory hiring recommendations, or exposes privileged financial information represents a critical business risk that requires immediate remediation.

When assessing findings, consider questions such as:

• Could confidential information be exposed?
• Would customers receive unsafe or misleading advice?
• Does the issue violate regulatory or contractual obligations?
• Could the organization suffer financial loss?
• Would the organization’s reputation be affected if the behavior became public?
• Is the issue repeatable, or does it occur only under very specific circumstances?

Answering these questions helps organizations prioritize remediation efforts and communicate risks more effectively to business leaders who may not understand the underlying technical details.

AI Red Teaming is a Collaborative Exercise

One of the biggest mistakes organizations make is assuming AI red teaming belongs exclusively to security teams. Unlike infrastructure testing, evaluating AI systems requires expertise from multiple disciplines because AI affects far more than technology alone.

Security professionals bring experience in adversarial thinking and attack simulation. Data scientists understand how models are trained and where limitations may exist. Developers understand the surrounding application architecture. Compliance specialists ensure regulatory requirements are considered throughout testing. Legal teams provide guidance on privacy, intellectual property, and contractual obligations. Business owners contribute operational knowledge that helps identify realistic misuse scenarios.

Perhaps most importantly, end users should also be involved.

Employees frequently interact with AI in ways developers never anticipated. Observing how users naturally phrase questions often uncovers prompt sequences that no formal test plan would have included. Their curiosity, assumptions, and everyday workflows provide valuable insight into how AI will behave once deployed across the organization.

Successful AI red teaming is therefore not a one-time security exercise carried out by a small specialist team. It is a collaborative process that combines technical expertise, business knowledge, governance, and real-world user behavior to build confidence that AI systems remain safe, reliable, and aligned with organizational objectives.