AI Audit Logs: Available Does Not Always Mean Useful
In brief: Assess SaaS AI audit logs by testing whether they answer your investigation questions, are enabled for the correct deployment and remain retrievable for the required period. “Logs available” does not establish coverage, identity attribution or useful action details. Generate authorized synthetic events, inspect their records and document gaps before relying on logging as an approval condition.
Key takeaways
Test whether logs answer the actual investigation questions.
Verify enablement, retrieval and retention scope.
Preserve identity and event-coverage limitations.
Symptom: the vendor says logging is included
The security review marks audit logging complete. Months later, an investigator cannot determine which AI action changed a record or which identity approved it. The product may have generated logs, but the earlier review never established whether those logs could answer the question the organization needed to investigate.
Begin with a small set of questions: who invoked the feature, what resource or action was involved, when it happened, whether approval occurred and what outcome followed. Add only fields needed for the workflow. Not every use case needs prompt content, and logging sensitive content can create its own access and retention considerations.
Diagnosis 1: availability was confused with enablement
Identify the product, plan, tenant and event family. Check whether collection requires a setting, entitlement, separate integration or commercial arrangement. Microsoft's Copilot and AI auditing documentation describes product-specific logging context and differences. Use the current applicable documentation rather than assuming all AI products have the same events or retention behavior.
Record the actual configuration and observation time. A documentation page proves a described capability within its scope; it does not establish that your tenant has enabled it or that the reviewer can retrieve records. Verify the authorized search or export path and the roles needed to operate it.
Diagnosis 2: the event does not answer the question
Investigation question | Useful evidence to inspect |
|---|---|
Who acted? | Relevant user or application identity |
What changed? | Action and target identifier within the event scope |
Was approval required and obtained? | Suitable approval and execution linkage |
Did it succeed? | Result or status with documented semantics |
Can we reconstruct the sequence? | Timestamps and correlation identifiers |
The table describes review needs, not a promise that every vendor emits those fields. If a field is absent or ambiguous, ask the vendor what can be established. Do not infer a target object from a generic “AI used” event or attribute an application action to the person who happens to own the integration.
Diagnosis 3: identity attribution is incomplete
Determine whether the workflow operates as a person, shared identity or application. Microsoft's permissions and consent overview supplies context for distinguishing authorization models in its environment. A log showing an application ID may be accurate, yet insufficient to identify the human who requested or approved a specific action.
Record the available linkage and its limitations. If a shared workflow lacks individual attribution, assess that gap with the application owner. Do not create apparent certainty by mapping every event to a mailbox recipient or original administrator. The event's documented semantics should drive the statement you can defend.
Diagnosis 4: retention exists only on paper
Verify how long relevant events remain accessible in your deployment and how the organization retrieves them. Account for product-specific settings and supported export arrangements. Test the query with known event IDs where possible. A configured duration without usable access can still fail an investigation when the reviewer cannot obtain the records.
Have the responsible teams define the required period based on organizational needs and applicable obligations. This article does not prescribe a universal retention duration. Record access controls for the logs, especially where content or sensitive resource names appear. Logging can support accountability while still requiring careful handling of the information it contains.
A small test with a clear expected result
Create an authorized synthetic AI event in a safe test context. Note its time, acting identity, intended action and expected outcome. Search the available logs, inspect the event fields and compare them with the known test facts. Repeat for a representative rejected or failed action if that behavior matters to the approval condition.
Preserve sanitized examples and the retrieval steps. Avoid copying real sensitive prompts into a broadly accessible ticket. If the event is missing, investigate collection, latency and query scope before concluding the service lacks logging. If the event appears but cannot support the required question, record the capability gap precisely.
Turn findings into a use decision
NIST's Generative AI Profile offers wider risk context. In the working review, classify each required question as supported, partially supported or unsupported, with evidence. Assign improvements and decide whether the workflow needs narrower actions, different logging arrangements or a limited interim acceptance.
Waldo's AI Risk & Governance and SOC 2 evidence preparation guide connect the feature assessment to broader evidence practices. Logging availability can inform that assessment, while tenant verification and event tests establish what investigators can actually learn.
The report should preserve its limits
State the tested product, plan, event types, retrieval path and date. List untested actions and missing attribution. Reopen the review after material event-schema, entitlement or workflow changes. A short report that distinguishes capability from verified usefulness is more actionable than a universal green “audit logging” badge.
Keep a sanitized reference event
Retain a small example event and retrieval note that authorized investigators can use to understand the schema. Remove sensitive content and preserve only the fields necessary to demonstrate the supported questions. Include product, event type and observation date, because the reference can become stale after an update. A readable example often communicates limitations more effectively than a logging checkbox.
Assign responsibility for checking material schema or access changes. If a field disappears, an export stops or the entitlement changes, reopen the logging condition tied to the workflow. Preserve the earlier assessment and explain the new gap. That maintenance step keeps logging evidence useful after approval and avoids discovering during an investigation that a once-working retrieval path no longer supports the organization's required questions.
Next step: Explore AI auditability visibility. Start with the evidence gap that affects your current workflow and assign its verification to an accountable owner.
Frequently asked questions
Does “audit logs available” mean logging is enabled?
No. Verify product, plan, tenant settings and retrieval permissions. Preserve the difference between documented capability and observed collection in the deployment you actually review.
Do we need to retain every prompt?
Not necessarily. Define the investigation questions and assess the needed fields. Content logging can introduce sensitive information into another system, so privacy, access and retention decisions should be deliberate.
Can an application ID identify the human requester?
Only if suitable evidence links the request to that person. Record the event semantics and available correlation. Do not infer individual attribution from integration ownership or a shared mailbox address.
What if the test event is missing?
Check collection settings, documented timing, query scope and access before diagnosing the cause. Record the unresolved issue and owner. A missing search result alone does not establish that no event was generated.
What should the final logging assessment include?
Include tested event types, supported investigation questions, retrieval evidence, retention context and gaps. Tie any approval condition to those verified capabilities and list changes that should reopen the review.
Sources and further reading
NIST: SP 800-53 Rev. 5: Security and Privacy Controls. Revision 5 published September 2020. Accessed October 3, 2026.
OWASP: LLM06:2025 Excessive Agency. Accessed October 3, 2026.
Microsoft: Data, Privacy, and Security for Microsoft 365 Copilot. Accessed October 3, 2026.
Microsoft: Audit logs for Copilot and AI applications. Accessed October 3, 2026.
Comments