The citation resolved on the first click. It pointed at a compensation review, and directly above the link sat two accurate sentences summarizing what that review said, rendered for a junior analyst who had never been cleared to open the file. There was no hallucination, no dead link, no wrong number, nothing a monitoring dashboard would flag. The model did everything right, and the system committed a data breach. That is the fastest way I know to lose an enterprise deal: answer a question correctly for the wrong person.
I have built RAG over corporate records more than once now: HR files, contracts, finance ledgers, support tickets, the kind of corpus where half the documents have an audience and the other half have a need-to-know. The demo takes an afternoon, same as it always does. Index the documents, embed the question, stuff the top chunks into a prompt. It looks finished, and it is nowhere close, because in the enterprise the hard problems live everywhere except the search itself.
My thesis, which I will defend for the rest of this post: retrieval is an access-control problem wearing a search costume. Relevance ranking is what the tutorials obsess over, and it was mostly solved years ago. What actually decides whether this thing can go live is whether the right person, and only the right person, can see the right record, at the moment they ask, with a citation they can actually open.
The trap is the embedding, not the answer
The first instinct everyone has, including me the first time, is to filter at the end: retrieve broadly, generate the answer, then check whether the user was allowed to see the sources and redact if not. That feels safe right up until you trace what actually happened.
By the time you are redacting, the privileged text has already been embedded into the prompt, sent to the model, and folded into the answer. Strip the citation and the summary in front of the user is still built on content they had no right to. (You will watch this happen the first time a model paraphrases a number from a document it then “declined” to cite. The number is still on the screen.)
Permission cannot live in a post-filter; it has to be a retrieval predicate, meaning the vector search itself must be incapable of returning a chunk the asker cannot read. That is the whole ballgame, unglamorous enough that most RAG content skips it.
Permission-aware retrieval, by hand
You attach the access control list to the chunk at index time, and you pass the user’s principals into the query so the store filters before similarity is computed, not after it.
def retrieve(question, user, k=8, floor=0.74):
# The user's full identity for ACL purposes: their own id plus every
# group/role they inherit. Resolve this fresh per query. People get
# removed from groups, and a stale principal set IS a security bug.
principals = identity.principals_for(user) # {"u:482", "grp:finance-ro", ...}
q = embed(question)
# The ACL filter runs INSIDE the index scan, not as a Python post-step.
# If a chunk's allow-set shares nothing with the user's principals,
# the store never even scores it. It cannot leak what it never returns.
hits = index.search(
q,
k=k,
where=acl_overlaps("allow_principals", principals),
)
keep = [h for h in hits if h.score >= floor and not h.doc.deleted]
if not keep:
# No readable, confident source. Say so. Do not widen the ACL
# to find an answer. That is how you turn a search box into a leak.
return None
return [h.doc for h in keep]The one line to sit with is acl_overlaps living in the where clause. That is the difference between a system you can take to a security review and one that fails it. A vector store that cannot do a metadata filter as part of the search is not a minor inconvenience; it is a reason to pick a different store, or to shard your index by audience so the wrong rows are never physically present in the scan.
Two more things in that snippet earn their keep. One is not h.doc.deleted, which I will come back to. The other is resolving principals per query instead of caching it on a session, because the gap between “person leaves the finance team” and “person stops retrieving finance documents” should be measured in seconds, not in whenever their token expires.
The allow-list lives in a system you do not own
The hard part of enterprise permissions has nothing to do with the allow-list on your chunk and everything to do with where that allow-list comes from. It gets computed somewhere else, by a system you do not own, and it changes constantly. A contract gets reassigned. A deal closes and the room shrinks. Someone goes on leave and their delegate inherits their inbox. Your static allow_principals field is a photograph of a permission that stopped being true a while ago.
You need both halves. Push: when the source system changes an ACL, it emits an event and you re-stamp every affected chunk. Pull: at query time you re-resolve the user’s principals live, so a stale chunk-side ACL still gets intersected against a fresh user-side identity. The intersection is only as safe as the staler of the two. Treat any source that cannot emit permission-change events as one you re-index on a tight clock, because you have no other way to know when its truth moved.
Freshness, deletes, and the right to be forgotten
A record in an enterprise is not a document on a website. It changes, it gets superseded, and sometimes it gets deleted, and not as a nicety: a legal hold lifts, a customer exercises a deletion right, a retention window closes. When that happens, the embedding of that record has to leave your index, and “gone” has to mean gone, not flagged.
This is where “just re-embed everything nightly” goes wrong without anyone noticing. A full nightly rebuild leaves a deleted record answerable for up to a day, which in a regulated shop is not an inconvenience but a finding. Incremental is the fix: the source emits a change, you re-embed that record, and a delete is a hard removal, not a deleted=true you remember to filter on. (Soft-delete-and-filter is how the deleted document slips through the one code path that forgot the filter. Make it physically impossible instead.)
def apply_change(event):
doc_id = event.doc_id
if event.op == "delete":
# Hard delete. The vector and its source text both leave the store.
# A tombstone we forget to filter is a breach with a timestamp.
index.purge(doc_id)
provenance.record_purge(doc_id, reason=event.reason, at=event.ts)
return
record = source.fetch(doc_id) # current truth, not a cached copy
if record.acl_version != event.acl_version:
# The ACL moved between the event and our fetch. Re-fetch, don't guess.
return apply_change(event.refresh())
chunks = chunk(record.blocks)
vectors = embed_all(chunks)
index.upsert(
doc_id,
vectors,
allow_principals=record.allow_principals,
source_version=record.version, # lineage starts here
updated_at=record.ts,
)I keep source_version and updated_at on every chunk because the next demand from any serious buyer is never about whether the answer is good, but about where it came from and whether it was current. That is the last of the boring parts.
Lineage a human can actually check
A citation the user cannot open is decoration. If your answer says “per the FY24 services agreement” and the link 404s, or worse, resolves to a document the user is not allowed to open, you have built something that looks trustworthy and is the opposite. The citation has to resolve, in the user’s own permission context, to the exact version of the record retrieved, the same source_version you stamped at index time. Not “the latest copy,” which may have changed since the answer was generated, and not a deep link that bypasses the source system’s own access check.
The test I use is blunt. Pick any sentence in the answer, click its citation, and land on the specific paragraph of the specific version that produced it, gated by the same permissions the retrieval used. If a sentence has no citation that survives that click, it does not get to be in the answer. An enterprise user is not there for a confident summary; they are there to verify, in ten seconds, that the machine is not lying, and then forward the source to their boss with their own name on it.
None of this is about the model, which has been good enough for this job for a while. What separates a RAG demo from a system a regulated company will run is permissions enforced inside the retrieval, freshness measured in seconds, deletes that are really deletes, and a citation a human can open and trust. All of that is plumbing, and the plumbing is where the product lives.
On the next system I build, the first test I write will not be a relevance benchmark but a query that retrieves as the wrong user and asserts that nothing comes back, wired into CI so it runs on every deploy. My bet is that within a couple of years the enterprise RAG vendors still standing will be the ones who treated that assertion as non-negotiable and the model score as a rounding error. Rerankers will keep improving without any help from me. Whether the wrong person can see the wrong record is what decides who gets to sell into a regulated company at all, and that is where I am spending my time.