How to vet a legal AI vendor: the security questionnaire that protects clients

Why SOC 2 alone cannot clear a legal AI vendor for privileged material

A SOC 2 Type II report tells you something real: an independent auditor confirmed that the vendor operated defined security controls over a review period. What it attests, however, is process, not data flow. The report will not tell you which country your prompt lands in, which sub-processors can read an uploaded deposition transcript, whether that transcript enters a training corpus, or how long fragments of it persist in logs. Frameworks like the multi-layer due-diligence model described by Lexitas treat certifications as exactly one layer among several, alongside data handling, model behavior, contractual terms, and ongoing monitoring ^[1].

For lawyers, this gap is not academic. ABA Formal Opinion 512 grounds generative AI use in the Model Rule 1.6 duty of confidentiality: a lawyer must understand how the tool uses and retains client data, and must obtain informed client consent before inputting client information into a self-learning tool ^[3]. A badge cannot satisfy a duty of understanding. To be fair to vendors, SOC 2 remains necessary; it screens out weak corporate perimeters and immature security programs. It is simply not sufficient, because it never claims to answer the question that privilege turns on: where does the client’s data actually go?

It also helps to read the scope statement of any certification before you rely on it. A SOC 2 report defines a boundary, the systems and controls the auditor examined, and an AI product often sits partly outside that boundary. The corporate email system, the customer portal, and the billing stack may be in scope while the inference pipeline, the model host, and the prompt-logging path are not. That mismatch is easy to miss because the badge looks identical either way. When a vendor offers a certification as evidence, ask which components it covers and request the scope section rather than the summary. If the AI pipeline is excluded, the certification is answering a different question than the one you need answered.

A certification tells you a control existed during an audit window. It does not tell you the control still holds today, nor that it covers the part of the system that touches your client’s file.

What "understanding the tool" actually requires of a lawyer

The duty is not satisfied by trusting the tool. It is satisfied by being able to explain, and if needed prove, where the client’s data goes and who can see it.

Opinion 512 sets a standard that sounds simple and turns out to be demanding: before relying on a generative AI tool, a lawyer must understand how it uses and retains client data ^[3]. Understanding here is not a feeling of familiarity from having clicked through a demo. It is the ability to describe, in plain terms, what happens to a client’s information from the moment it enters a prompt to the moment it is provably gone. If you cannot narrate that path, you have not yet met the standard, and a vendor’s marketing page will not close the gap for you.

Practically, that means being able to answer a short set of questions about any tool your firm uses. Where is the data processed, and by whom? Is it used to improve or train a model? How long is it kept, and where does it linger after deletion? Who at the vendor, or at a sub-processor, can read it? What is written down if something goes wrong? These are the same questions the questionnaire below is built to extract, which is the point: the questionnaire is not paperwork for its own sake, it is the mechanism by which you acquire the understanding the ethics rule requires. The consent obligation follows directly, because you cannot obtain informed client consent to a data practice you do not yourself understand ^[3].

The 12 questions every legal AI security questionnaire should ask

A useful questionnaire forces specificity. Vague answers such as “industry-standard encryption” or “we take security seriously” are non-answers; the questions below are designed so that evasion is itself a signal. They align with the layered diligence approach practitioners now recommend ^[1] and with workflow checklists built specifically for law firm AI selection ^[2]. Send them in writing, require written answers, and keep both: they become the representations you later verify and, if needed, enforce.

Score the responses the way structured selection workflows suggest: red, yellow, or green for each answer, with any red on training, retention, or sub-processors treated as disqualifying until cured ^[2]. The last four questions matter more than they first appear. Editable logs cannot support a later dispute about what the tool did, and matters involving trade secrets, estates, or long-lived corporate records can demand confidentiality that outlasts today’s cryptography, which is why quantum-safe record integrity belongs on a legal questionnaire in 2026.

• Where does client data physically go? Name every system, region, and sub-processor that touches a prompt or document.
• Is our data used to train or fine-tune any model, including aggregated or “de-identified” derivatives?
• What is retained, for how long, and can retention be set to zero?
• Can privileged material be withheld, redacted, or tokenized before it ever reaches the model?
• Is a local or single-tenant deployment available for restricted matters?
• How is client informed consent captured and recorded, consistent with ABA Opinion 512 ^[3]?
• What logs exist, and are they tamper-evident or merely editable records?
• Can we independently verify what the system did, rather than audit on trust?
• What happens to our data at contract end, and how is deletion proven?
• Which certifications cover the AI pipeline specifically, not just the corporate perimeter?
• What is the breach-notification commitment in hours, not “promptly”?
• Are signatures and records quantum-safe for data carrying decades-long confidentiality?

How to send the questionnaire so the answers hold up

The value of a questionnaire depends less on the questions than on how you handle the answers, and this is where many firms lose the thread. An answer given verbally on a sales call is not a representation you can later rely on. Insist on written responses to the written questions, from someone at the vendor who is accountable for their accuracy, and preserve the exchange the way you would preserve any other record you might one day need to produce. The reason is straightforward: the questionnaire’s answers are the raw material for your contract terms, and if a dispute arises, the difference between “they told us” and “they wrote this on this date” is the difference between an assertion and evidence.

Ask follow-ups in writing too. When a first answer is thin, a written request for specifics, naming the sub-processor, stating the retention period in days, confirming whether a zero-retention tier exists, does two useful things at once. It gets you the detail you actually need, and it documents that you sought it, which matters for the diligence standard the ethics rule implies ^[3]. Keep the questionnaire, the responses, and the follow-ups together in the matter or vendor file so that the whole record can be read as one narrative later. Structured selection workflows treat this documentation step as part of the process, not an afterthought, precisely because the file is what you fall back on when memory fails ^[2].

1. Send in writingDeliver the questionnaire as a written document and require written responses from a named, accountable person at the vendor.
2. Press vague answersWhere a response is non-specific, follow up in writing for the exact system, region, retention period, or sub-processor name.
3. Score each answerMark every response red, yellow, or green, and treat reds on training, retention, or sub-processors as disqualifying until cured ^[2].
4. Preserve the recordKeep the questionnaire, the answers, and the follow-ups together as the evidentiary baseline you will later compare to the contract.

How to read model-training and data-retention answers

Training answers hide in verbs. “We may use customer content to improve our services” usually means training; press for whether that includes fine-tuning, evaluation sets, and derivative models built on aggregated data. “De-identified” deserves particular skepticism in legal work, because a fact pattern can identify a client even with names stripped. Under Opinion 512, if the tool is self-learning on your inputs, you need informed client consent before client information goes in at all, so the training answer directly determines your consent workflow, not just your risk rating ^[3].

Retention answers need the same dissection. “Thirty days for abuse monitoring” raises follow-ups: who can access data during that window, is it human-reviewed, and is a zero-retention tier available or reserved for enterprise contracts? Ask where deleted data survives in backups and how long. Diligence frameworks treat data handling and model behavior as separate layers precisely because vendors often score well on one and poorly on the other ^[1]. Map each answer to your red/yellow/green sheet, and record the exact language: ambiguity discovered now is a negotiating point; ambiguity discovered after an incident is an exposure.

A third pattern worth naming is the conditional yes. A vendor may confirm that a zero-retention or no-training option exists, then reveal on follow-up that it is limited to a higher pricing tier, a specific deployment mode, or a subset of endpoints. The option is real, but it is not the option you are buying unless you say so in the contract. When you find one of these, note both the capability and its condition, then carry both into the negotiation. The pattern is not necessarily a sign of bad faith; it reflects how many products are built. Your job is simply to make sure the protective setting you were shown is the setting your firm actually runs under, for the matters that need it.

Non-answers and red flags to watch for

Some responses are informative precisely because they avoid the question, and learning to read them saves time. Reliance on adjectives instead of nouns is the most common tell. “Bank-grade security,” “enterprise-ready,” and “fully compliant” describe a posture without naming a control, a scope, or a fact you can check. When you see them, treat the underlying question as unanswered and ask again for specifics. A vendor that is comfortable with its own practices can usually restate them plainly.

A second flag is a certification offered as a conversation-ender. If a vendor answers a data-flow or training question by pointing to a badge rather than describing the practice, the badge is being used to change the subject, and the substantive question still stands ^[1]. A third is resistance to putting a spoken assurance into the contract. A promise not to train on your data is meaningful only if the vendor will commit to it in writing, and reluctance to do so tells you the assurance was softer than it sounded ^[4]. None of these signals proves a vendor is unsafe. Each one tells you where to keep digging before any client material is at stake.

• Adjectives standing in for facts: “bank-grade,” “enterprise,” “fully compliant,” with no named control or scope.
• A certification cited to end a data-flow question rather than to support a specific claim ^[1].
• Reluctance to convert a spoken assurance, especially about training, into a written contract term ^[4].
• “De-identified” or “aggregated” used to imply safety without addressing re-identification from a fact pattern ^[3].
• Retention or deletion described in soft words like “promptly” instead of a defined period you can hold them to.

The contract terms courts and clients now expect

Questionnaire answers only bind anyone once they become contract terms. That shift is no longer optional: in 2026 discovery disputes, courts have required AI tools handling confidential material to carry contractual no-training safeguards, deletion capability, and onward-disclosure limits, with protective orders emerging as a point of dispute when those terms are absent ^[4]. Sophisticated clients increasingly ask outside counsel to certify the same terms in their own vendor stack. The following provisions should be non-negotiable in any legal AI agreement:

Treat these terms as the enforceable mirror of your questionnaire. If a vendor answered “we never train on customer data” but resists a no-training clause, the questionnaire answer was marketing. Structured selection workflows recommend resolving this before any pilot begins, then running the pilot with audit logs enabled and restricted matters routed away from the tool until the contract and technical controls are proven ^[2]. Contract language also gives litigators something concrete to present when a protective order requires proof of safeguards ^[4].

• A no-training clause covering training, fine-tuning, and derivative or aggregated models.
• Deletion capability on demand, with written certification and backup-purge timelines.
• Onward-disclosure limits that bind every sub-processor, not just the prime vendor.
• Advance notice and approval rights for new sub-processors or model changes.
• Breach notification in defined hours, with named contacts and escalation paths.
• Audit and independent verification rights that survive contract termination.

Running a limited pilot before firmwide rollout

Even a strong questionnaire and a well-drafted contract describe intended behavior. A pilot lets you observe actual behavior on a controlled scope before the tool touches your most sensitive files. Workflow checklists built for law firm AI selection treat the pilot as a discrete stage: enable audit logging, route restricted matters away from the tool, and confirm that the technical controls the vendor described actually operate the way the contract now requires ^[2]. The purpose is not to re-run the sales demo. It is to catch the distance between what was promised and what happens, while the blast radius is small and reversible.

During the pilot, keep the questionnaire open beside you and check answers against reality. Does the deployment mode match the one you contracted for? Do the logs record what you were told they record, and can you read them without the vendor’s help? If a zero-retention or no-training setting was conditional, is it in force for the matters you care about? Write down what you observe, because those observations become part of the same vendor file as the questionnaire and the contract. A pilot that surfaces a gap is a success, not a failure; it has done its job before a client’s privileged material was ever exposed to the problem.

A pilot is worth running even when the vendor is reputable. Its value is that it tests the specific configuration your firm will run, not the vendor’s reputation in general.

Who should own vendor vetting inside the firm

A questionnaire only protects clients if someone is clearly responsible for running it every time, and for keeping the answers current. In many firms the natural owner is a small committee that pairs a lawyer who understands the confidentiality duty with a technologist who can read a data-flow answer critically. What matters less is the exact structure and more that ownership is explicit: a named person or group decides which tools are approved, holds the vendor files, and is accountable when a new tool appears in a practice group without review. Diffuse responsibility is how unvetted tools enter through the side door, one enthusiastic associate at a time.

Ownership also has to extend past the initial decision. Vendors change sub-processors and models, and the diligence layer that catches those changes is ongoing monitoring, not the one-time review ^[1]. Assign responsibility for periodic re-checks, for reading the vendor’s change notices, and for re-running the relevant questions when the vendor swaps an underlying model or revises a retention policy. The same owner should hold the line on consent, ensuring that where a tool’s training practice requires client consent under Opinion 512, that consent is actually captured and recorded before client information is entered ^[3]. Clear ownership turns a good questionnaire from a document into a practice.

From questionnaire to proof: verifying vendor claims continuously

Here is the questionnaire’s structural weakness: every answer is a point-in-time representation. Vendors change sub-processors, swap underlying models, and revise retention policies between annual reviews, and a diligence file from January says nothing about what happened to a privileged document in June. Practitioner frameworks acknowledge this by making ongoing monitoring its own diligence layer ^[1], and workflow checklists build in pilots with audit logs and restricted-matter routing for the same reason ^[2]. The question is what ongoing monitoring should look like when the stakes are privilege rather than uptime.

The emerging answer is continuous verification: replacing “trust the badge” with per-interaction attestations, signed records that bind each AI interaction to the approved tool, the isolation method in force, and the consent on file. RankShield Legal produces exactly that record layer, attesting architecture and consent with independently verifiable records rather than editable logs. Honesty requires a caveat: no platform can promise to prevent privilege waiver, because the confidentiality duty under Model Rule 1.6 and Opinion 512 stays with the lawyer ^[3]. What continuous attestation changes is your evidentiary position, so that when a client, regulator, or court asks where the data went, you can prove the answer instead of asserting it ^[4].

The distinction that runs through this entire guide is between self-attestation and verifiable evidence. A vendor’s answer, a certification badge, and a spoken assurance all sit on one side: they are claims. A signed, tamper-evident record of what the system actually did sits on the other: it is evidence. A questionnaire is how you gather the claims and force them into the open; a contract is how you make the claims binding; verification is how you learn, over time, whether the claims held. None of the three replaces the others, and none removes the lawyer’s underlying duty. Together they move a firm from trusting its vendors toward being able to demonstrate, on the day it matters, exactly how a client’s confidential information was handled.