Case

LLM gateway for PII

A layer in front of OpenAI-style chat. Raw SSN, email, and phone stay on your side of the call. Policy is checked locally. Internal docs are searched only when the model actually needs them.

The problem

You want a cloud model in the loop. HR files, incident notes, and support threads still have SSN, email, and phone in them. If those strings go out in the prompt, they have already left.

This is not a chatbot. It is a gateway. Callers still speak /v1/chat/completions. A fixed workflow sits around one knowledge agent.

The boundary

Scrub and policy always run. The model cannot skip them. Search is optional: the model decides whether to look anything up.

  1. Auth and session. Tenant key plus X-Session-ID so each session keeps its own map.
  2. Scrub. The gateway finds SSN, email, and phone and swaps them for stable tokens such as [EMAIL_1]. The same value in the same session always gets the same token.
  3. Local policy. After names and numbers are replaced, a local Qwen model reads the security policy. Any deny is 403 for the whole request. If Ollama is down, the request is blocked. We fail closed.
  4. Knowledge agent. When search is on, the cloud model may call search_internal_knowledge. A hello can skip search. An HR or incident question should not. Text that comes back is scrubbed again before the model sees it.
  5. Rehydrate. Tokens are put back so the caller sees real values, in a full reply or in a stream. If a token is split across stream chunks, it is stitched before anyone sees it.

What a walkthrough shows

  1. A PII prompt: tokens in what is sent to OpenAI, real values in the reply.
  2. Project Titan: 403, OpenAI not called.
  3. RAG plus a greeting: policy allows, the search tool is not called.
  4. RAG plus HR or incident: the tool runs, sources are listed.

That sequence is what we demo on a laptop. It is not running on this site. The stack is ASP.NET Core, a small React lab, Ollama, and MongoDB Atlas Local for vectors.

What this is not

  • Not production login. Tenant keys sit in config, not APIM or JWT.
  • Not a swarm of agents. Only the generation step is an agent.
  • Not search on every turn. We do not stuff the whole library into the prompt.
  • Not a live demo on gtac.ai. This site cannot host that boundary.

Back to GTAC