The False Sense of Security in Vendor Questionnaires

Martin Snyder
Sep 16, 2025
4 min read

Security questionnaires feel reassuring—checklists, green boxes, signatures—but they’re often a snapshot of intent, not proof of protection. Waldo Security gives you proof: we discover every SaaS app, tenant, account, and OAuth connection in minutes, flag SSO/MFA gaps and risky scopes, and export audit-ready evidence so your third-party risk program rests on facts, not forms. Start with Instant SaaS Discovery and keep clean artifacts in our SaaS Compliance Overview.

Why questionnaires lull teams into risk

They’re self-attestations. Standards like the CSA CAIQ (aligned to the Cloud Controls Matrix) and Shared Assessments SIG are helpful baselines, but they record what vendors say they do—usually “yes/no” answers—without demonstrating configuration or control operation. (Cloud Security Alliance, Shared Assessments)

They’re periodic, not continuous. Modern risk guidance stresses ongoing supply-chain monitoring and verification, not one-time forms. NIST’s SP 800-161r1 and CISA’s SCRM guidance both emphasize integrating supplier risk into continuous cybersecurity management and verifying third-party assurance, not just collecting questionnaires. (NIST CSRC, CISA)

They rarely reflect real SaaS usage. Shadow apps, duplicate tenants, and unmanaged OAuth grants won’t show up in a vendor’s questionnaire—yet they’re exactly where data moves. Meanwhile, the 2025 Verizon DBIR again ties a large share of Basic Web App breaches to stolen credentials, meaning “paper SSO” that isn’t enforced (or bypassed via tokens and guests) is still low-hanging fruit. (Verizon)

They don’t prevent cascade events. When a widely used product has a critical flaw (think MOVEit), hundreds of downstream orgs feel it. Questionnaires completed months earlier don’t change today’s exposure; validated patching and monitoring do. (CISA)

What questionnaires are good for (and what they aren’t)

Useful for

Establishing scope and intent (what the vendor claims to do and which controls they map to). CAIQ/SIG are efficient starting points. (Cloud Security Alliance, Shared Assessments)
Anchoring requests for independent reports, e.g., SOC 2 Type II or ISO certifications. (Even then, treat these as inputs—not blanket approvals.) (AICPA & CIMA)

Not sufficient for

Proving SSO/MFA is enforced for your users in your tenant.
Showing OAuth scope hygiene, token lifetimes, or whether offline_access keeps access alive.
Demonstrating data-sharing defaults, guest controls, or public-link posture across collaboration suites.
Continuous assurance that keeps pace with new features, plug-ins, or incidents.

An evidence-first approach (that still uses questionnaires)

Inventory reality before paperwork. Build a living list of the SaaS apps and integrations your people actually use—sanctioned and shadow—by correlating IdP sign-ins, email/collab logs, DNS/proxy, browser extensions, and spend. This aligns with NIST/CISA’s “visibility first” supply-chain guidance. (NIST CSRC, CISA)
With Waldo, discovery takes minutes and shows tenants, accounts, OAuth grants, and SSO coverage.
Request attestations—then verify. Use CAIQ/SIG answers to ask for evidence:
- SSO/MFA enforcement screenshots or policy exports in your tenant.
- OAuth consent and scopes for your integration (avoid tenant-wide *.ReadWrite.All, require verified publishers).
- Audit logs enabled and streaming options.
- Patch SLAs and recent security bulletins (especially for high-profile zero-days like MOVEit). (CISA)
Map claims to independent artifacts. A SOC 2 Type II can corroborate control design/operation over time—read it, don’t just file it. Tie report sections (e.g., logical access, change management, logging) to what your tenant actually uses. (AICPA & CIMA)
Continuously monitor drift. Re-check new admins, new public links, new high-privilege OAuth grants, and non-SSO logins monthly. CISA’s SCRM guidance is explicit: verify third-party assurance and keep doing it. (CISA)

A quick rubric to grade vendor answers (and escalate when needed)

Score each key area 0–2 (0 = weak, 2 = strong) and escalate anything <6/10:

Identity: SSO/MFA enforced in your tenant? Evidence provided?
OAuth: Least-privilege scopes, no offline_access unless justified, verified publisher?
Data controls: External sharing restrictions, guest governance, region & encryption details?
Logging & response: API/audit logs accessible? Clear incident-response commitments?
Patch & disclosure: Timely advisories? Public history of bulletins and mitigations (e.g., MOVEit-style issues)? (CISA)

How to modernize your TPRM workflow (30-day plan)

Week 1 — See what’s real. Run discovery; tag critical vendors by data sensitivity and business impact (payments, HR, customer data, code).Week 2 — Verify claims. For top vendors, turn CAIQ/SIG answers into evidence requests (SSO policies, OAuth scope list, logging config, recent bulletins). Map SOC 2 controls to your use case. (Cloud Security Alliance, Shared Assessments, AICPA & CIMA)Week 3 — Close gaps. Enforce SSO/MFA, restrict end-user consent to verified publishers and low-risk scopes, revoke unused persistent tokens.Week 4 — Make it continuous. Enable alerts for new apps, admins, public links, and high-privilege grants; schedule a monthly evidence packet by vendor.

Waldo automates the loop—find the apps and tenants, fix the identity and OAuth gaps, and prove operation with exportable evidence from the SaaS Compliance Overview.

The takeaway

Questionnaires are starting points, not safety nets. Today’s guidance from NIST and CISA is blunt: treat supply-chain risk as continuous, verified, and evidence-based. Combine CAIQ/SIG and SOC 2 with what actually matters in SaaS—enforced identity, least-privilege scopes, sane sharing defaults, and live logs—and you’ll replace checkbox comfort with measurable assurance. (NIST CSRC, CISA)

If you want the confidence without the busywork, we built it: get your truth map with Instant SaaS Discovery and ship proof, not promises, from the SaaS Compliance Overview.