AI Incident Response Playbook: What to Do When Sensitive Data Is Pasted into ChatGPT, Claude, or Copilot

An employee has just pasted customer information, source code, credentials, a contract, or another sensitive document into an AI service.

What should the organization do now?

The first rule is simple: do not automatically call it a data breach. But do treat it as an incident that needs to be assessed.

Submitting sensitive information to ChatGPT, Claude, Copilot, Gemini, or another AI system can create a security, privacy, contractual, or governance incident. Whether it becomes a legally reportable breach depends on what information was involved, which service received it, which account was used, how the provider processes and retains the data, what contractual safeguards apply, whether unauthorized disclosure or access occurred, the jurisdiction, and the actual risk created.

This playbook provides a practical response process for CISOs, security operations teams, DPOs, privacy teams, IT, compliance, and legal.

If sensitive corporate information has just been submitted to an AI service:

1Stop additional submissions. Ask the employee to stop using the affected conversation, workflow, connector, or account for sensitive work until the situation is assessed.

2Do not destroy evidence immediately. Record the service, user, account, device, approximate time, conversation or session, and what was submitted before deleting anything.

3Determine what data was exposed. Identify whether the prompt or upload contained personal data, credentials, source code, customer information, contracts, regulated data, or internal intellectual property.

4Identify the AI environment. Was it ChatGPT Enterprise, ChatGPT Business, a personal ChatGPT account, Claude Enterprise, a personal Claude account, Microsoft 365 Copilot, an API integration, an IDE assistant, or another service?

5Check for credentials. If passwords, API keys, tokens, certificates, private keys, session secrets, or other credentials were included, revoke or rotate them immediately.

6Preserve minimum necessary evidence. Capture metadata, timestamps, classification, event identifiers, relevant screenshots or exports, and the exact affected data where necessary for investigation.

7Check provider controls. Determine whether the organization can delete the conversation, remove uploaded files, use an administrator or compliance interface, or invoke provider-specific retention controls.

8Open an internal incident record. Do not wait for legal classification before documenting the event.

9Escalate based on the data involved. Personal data should involve privacy/DPO functions; credentials should involve security; contracts or privileged information may require legal; regulated information may require sector-specific teams.

10Do not notify regulators or customers reflexively. First establish whether the event meets the relevant legal or contractual notification threshold.

The objective during the first ten minutes is containment, evidence preservation, and classification.

Organizations should distinguish three concepts.

A security incident is the broadest category. It can include an event that threatens confidentiality, integrity, availability, credentials, intellectual property, or security policy.

An employee pasting a production API key into an AI chatbot can therefore be a serious security incident even if no personal data is involved.

Under GDPR, a personal data breach is specifically a security breach leading to accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to personal data.

That definition matters.

The fact that personal data was submitted to an AI provider does not by itself answer every question about whether the event constitutes a personal data breach requiring notification. The organization needs to examine the processing arrangement, authorization, provider/account used, contractual framework, safeguards, and actual disclosure or access circumstances.

An AI governance incident is a useful operational category for an event in which AI was used contrary to organizational policy or outside approved controls.

For example:

confidential information submitted through an unapproved personal AI account;

customer data sent to an AI provider that has not passed vendor review;

source code uploaded despite an internal restriction;

sensitive information sent through an approved AI platform but outside the permitted business use case.

An AI governance incident may overlap with a security incident or personal data breach, but it does not automatically become either.

Use the following sequence before deciding severity or notification.

The important point is that classification should follow investigation, not precede it.

Containment comes first.

Ask the employee to stop adding information to the affected conversation or workflow. If the incident involves an automated integration, connector, AI agent, browser extension, IDE assistant, or API pipeline, temporarily disable the affected data flow if necessary.

Depending on the situation, containment may include:

suspending an AI integration;

removing a file from a shared AI workspace;

restricting the user's access to an unapproved AI service;

temporarily blocking uploads;

disabling a connector;

removing an exposed repository credential;

invalidating active sessions.

Avoid a common mistake: telling the employee to immediately delete everything before the investigation team records what happened.

Deletion may be an appropriate remediation step, but first preserve enough evidence to determine the scope.

Do not investigate an incident based on the description:

> "Someone pasted confidential data into ChatGPT."

You need to know exactly what data was involved.

Determine:

What text was submitted?

Was it one sentence, a customer record, an entire database export, or a large document?

Were files uploaded?

Record filenames, type, approximate size, classification, and relevant hashes where available.

How many records were involved?

One person's email address presents a different risk profile from an exported CRM containing 100,000 customers.

Was the information already public?

Internal classification labels do not always correspond directly to actual confidentiality.

Was the sensitive information necessary for the task?

This becomes important during remediation and policy review.

"ChatGPT" or "Claude" is not enough information.

Determine the exact environment.

For example:

enterprise-managed account;

business/team workspace;

personal/free account;

API;

desktop application;

browser interface;

IDE integration;

third-party application using an underlying model;

organizational AI agent;

SaaS application with embedded AI.

The distinction can materially change the assessment.

For example, OpenAI states that business products such as ChatGPT Business and Enterprise are excluded from model training by default, while consumer services have different data-control arrangements. OpenAI Enterprise also supports organizational retention controls.

Therefore, "the information was submitted to ChatGPT" is not sufficient to determine what happened to the data.

The investigation should also determine whether the user was authenticated through corporate SSO, whether the account was managed by the organization, whether a DPA or enterprise contract applies, and what retention configuration was active at the time.

The type of information determines which response path should be activated.

Multiple categories may apply simultaneously.

A source-code upload, for example, may also contain API credentials, employee names, internal URLs, customer identifiers, and proprietary algorithms.

Incident response becomes dramatically harder when an organization cannot answer basic questions about the AI interaction.

At minimum, try to establish:

Preserving evidence does not mean copying sensitive prompts into multiple ticketing systems.

The objective is to preserve enough evidence to reconstruct the event without unnecessarily multiplying the sensitive data.

Credential exposure is one category where the response should rarely wait for a long legal analysis.

If an employee pasted any of the following into an AI system:

API keys;

passwords;

OAuth tokens;

database credentials;

private keys;

cloud credentials;

session cookies;

access tokens;

certificates or secrets;

assume the credential should no longer be trusted.

Revoke or rotate it.

Then determine whether the credential was used after exposure and what systems it could access.

Deleting the AI conversation is not a substitute for rotating the credential.

Different AI services provide different controls, and even the same provider may behave differently depending on plan and configuration.

Do not rely on assumptions.

OpenAI currently states that normal chats remain until deleted or removed under an applicable workspace retention policy. When a saved chat is deleted, it is removed from the user's view immediately and scheduled for permanent deletion from OpenAI systems within 30 days, subject to stated legal, security, or de-identification exceptions. Enterprise environments can have organizational retention controls and compliance capabilities.

Managed ChatGPT administrators may also have capabilities to access, audit, export, retain, or delete data associated with managed organizational accounts, depending on the product and configuration.

Anthropic's enterprise materials describe organizational retention controls and Compliance API capabilities that can provide enterprise administrators with usage data and selective deletion functionality. Commercial services should be distinguished from consumer Claude accounts, which operate under different terms and retention arrangements.

For supported Microsoft AI services, Microsoft Purview can retain and discover Copilot interaction data. Microsoft also documents workflows using eDiscovery and Microsoft Graph to search for and remove supported AI interaction data, including in response to data-spillage incidents.

The practical lesson is:

Check the specific provider, product, subscription, tenant configuration, contract, and retention policy that applied at the moment of the event.

Provider capabilities change over time. Incident runbooks should therefore link to current provider documentation rather than hard-code assumptions.

External reporting is not the first classification decision.

First determine how the event fits your organization's own policies.

Questions should include:

Was company policy violated?

Was an unapproved AI provider used?

Was a personal account used for corporate data?

Was the data classified as confidential or restricted?

Did a technical control fail?

Did a user bypass an existing warning?

Did the organization lack a necessary control?

Was the action accidental or intentional?

Is the same behavior happening elsewhere?

Even where external notification is unnecessary, the event can reveal a systemic governance gap.

Not every incident needs the same response team.

Credentials and infrastructure secrets generally require security involvement.

Personal data generally requires privacy/DPO assessment.

Contracts, privileged information, trade secrets, and confidentiality obligations may require legal.

Financial institutions, healthcare organizations, public bodies, and other regulated entities may require additional compliance owners.

A mature workflow routes the incident based on data type and impact, rather than sending every AI-related event to the same generic queue.

Do not start with the question:

> "Do we have 72 hours?"

Start with:

> "Which legal regime applies, and does this event meet its definition and reporting threshold?"

Under GDPR Article 4(12), a personal data breach involves a security breach resulting in accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to personal data.

Where a controller determines that a personal data breach has occurred, Article 33 requires notification to the competent supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of it unless the breach is unlikely to result in a risk to individuals' rights and freedoms. Controllers must also document personal data breaches, including their facts, effects, and remedial action.

Communication to affected individuals is a separate threshold: Article 34 applies where the breach is likely to result in a high risk to their rights and freedoms, subject to the exceptions specified in the Regulation.

The EDPB also publishes case-based guidance intended to help controllers assess breach situations and notification decisions.

Therefore:

Do not conclude that every accidental AI submission requires a GDPR notification. Perform and document the actual breach and risk assessment.

NIS2 uses a different test.

For entities within scope, Article 23 requires reporting of incidents that have a significant impact on the provision of their services. The Directive considers factors including severe operational disruption or financial loss and considerable material or non-material damage to other persons.

Where the entity and incident fall within the applicable NIS2 regime and satisfy the significant-incident threshold, the Directive establishes staged reporting requirements, including an early warning and subsequent incident notification. National transposition and the competent authority must also be checked before applying the process to a specific incident.

A routine employee policy violation involving an AI chatbot should therefore not automatically be described as a "NIS2-reportable incident."

DORA applies a separate incident-management framework to covered financial entities.

It defines and requires reporting of major ICT-related incidents, while also requiring financial entities to record ICT-related incidents and maintain processes for identification, classification, handling, follow-up, root-cause analysis, escalation, and remediation.

For incidents that actually qualify as major ICT-related incidents under DORA, Commission Delegated Regulation (EU) 2025/301 specifies current reporting time limits, including an initial notification as early as possible and within four hours after classification as major, subject to an outside limit of 24 hours from awareness, followed by intermediate and final reporting requirements.

Again, those timelines matter only after determining that the organization and incident fall within the relevant DORA requirements.

An AI incident runbook should therefore maintain separate decision paths for:

internal AI governance incident -> security incident -> personal data breach -> sector-regulated/reportable incident.

They are related categories, not synonyms.

Closing the ticket because a conversation was deleted is not enough.

Document:

What happened? Describe the event factually.

Why did it happen? Was AI policy unclear? Was the approved AI tool inconvenient? Was there no technical control? Was the employee unaware of the data classification?

What reduced the risk? Credential rotation, conversation deletion, provider request, account restriction, masking, policy update, or another action.

What remains unresolved? For example, inability to verify provider-side deletion.

What control will prevent recurrence?

A strong post-incident record should allow another security professional or auditor to reconstruct both the event and the response.

Training and acceptable-use policies remain useful, but they should not be the only control.

The strongest intervention point is before sensitive information leaves the employee's environment.

For enterprise AI use, preventive controls can include:

discovering which AI services employees actually use;

distinguishing corporate from personal accounts;

identifying sensitive text and uploaded files before submission;

warning users before risky actions;

masking selected sensitive information;

blocking prohibited data;

applying different policies by AI provider, user, department, or data class;

creating evidence that a control was executed.

This is where AgentID fits.

AgentID provides pre-send controls across employee AI usage, including visibility into AI services, sensitive-data detection, masking or blocking according to organizational policy, and audit evidence around the resulting decision.

The objective is not to prevent employees from using AI.

It is to turn:

"We discovered three weeks later that someone pasted customer data into an AI tool."

into:

"At 10:42, the employee attempted to send customer PII to an unapproved AI account. Policy classified the content as restricted and blocked the submission. The event was recorded without sending the protected data."

That difference fundamentally changes incident response.

The most useful investigation record is often not a permanent archive of every raw employee prompt.

A better model is frequently metadata-first evidence.

For example:

`event_id` `timestamp` `user_id` `device_id` `provider` `account_context` `data_classification` `policy_id` `policy_verdict` `masking_applied` `file_hash` `request_hash` `incident_reference`

Raw content can then be captured selectively when policy, investigation, legal obligations, or evidentiary requirements justify it.

This matters for privacy as well as security.

GDPR's data-minimization principle requires personal data to be adequate, relevant, and limited to what is necessary. Its storage-limitation principle similarly requires identifiable personal data not to be retained longer than necessary for the relevant purposes.

An AI governance system that copies every sensitive prompt forever may therefore create a second sensitive-data repository while attempting to govern the first.

Good governance asks:

What evidence do we need? For what purpose? For how long? Who can access it?

Retention should be configurable by data class, event type, business need, investigation requirement, and applicable law.

An employee using a personal AI account pastes a customer-support spreadsheet containing names, email addresses, account numbers, and case notes into a chatbot to generate a summary.

A sensible response would be:

Minute 0-10: stop further use, preserve basic evidence, determine the provider/account and dataset.

Immediate classification: customer PII is involved, so security and the privacy/DPO function are notified internally.

Containment: determine whether the conversation and uploaded file can be removed and what provider retention conditions apply.

Scope: establish the number of customers and fields involved.

Assessment: determine whether the disclosure was authorized within the organization's processing arrangements and whether the circumstances constitute a personal data breach under applicable law.

Remediation: delete or remove content where appropriate, record the incident, review the personal-account policy, and assess whether similar activity exists elsewhere.

Prevention: require managed AI accounts where appropriate and implement pre-submission detection or blocking for customer data.

What the organization should not do is jump directly from:

"PII was pasted into AI"

to:

"We suffered a legally reportable data breach."

The investigation determines that conclusion.

Stop additional exposure, determine exactly what was submitted, identify the account and service involved, preserve necessary evidence, rotate any credentials, review provider deletion and retention options, and run the event through your security/privacy incident process.

Do not automatically assume the incident is legally reportable.

Not necessarily.

It may be an internal security or governance incident. Whether it constitutes a personal data breach or another legally reportable incident depends on the information, authorization, account/provider context, contractual arrangements, applicable law, exposure, and risk.

Revoke or rotate the key immediately.

Deleting the conversation does not remove the need to rotate the credential. Review logs to determine whether the exposed key was subsequently used.

Usually you should first preserve the evidence necessary to understand the incident.

After evidence preservation, deletion or administrative removal may be an appropriate containment step depending on the provider and investigation requirements.

No enterprise AI service removes the need for organizational data governance.

Enterprise services can provide important privacy, contractual, identity, retention, and administrative controls. For example, OpenAI states that its business products are not used for model training by default.

But organizations still need to determine which information employees are authorized to submit, through which accounts, under which policies, and with what evidence.

There is no universal "AI incident = 72 hours" rule.

GDPR's 72-hour supervisory-authority requirement applies to personal data breaches meeting the conditions in Article 33. NIS2 and DORA use different concepts, thresholds, processes, and reporting timelines.

Identify the applicable legal regime first.

Not necessarily.

Full-content logging can help certain investigations, but it also creates an additional repository containing potentially sensitive information.

A metadata-first model with configurable retention and selective content capture can often provide useful evidence while reducing unnecessary data collection.

When sensitive data is submitted to an AI system, organizations need to answer five questions quickly:

What was sent? Where was it sent? Under which identity and contractual context? What risk did that create? What evidence and remediation exist?

That is the foundation of effective generative AI incident response.

The long-term objective should be to move those questions earlier in the lifecycle.

Instead of discovering sensitive information after it reaches an AI service, enterprises can classify, mask, warn, approve, or block risky interactions before submission-while retaining enough evidence to investigate what happened without creating another uncontrolled archive of sensitive prompts.