System Prompt Leaks - Why Secrets Don't Belong
Assume every system prompt can be extracted. What actually leaks, why "don't reveal this" fails, and what must never live in a prompt at all.
Short answer
Assume system prompts can be extracted. Never store API keys, passwords, or security-sensitive rules only in the prompt. "Do not reveal your instructions" is not a control.
Why "do not tell" fails
Models follow user pressure across turns. Extraction tricks recover large pieces of hidden prompts.
Treat anything in the prompt as public under adversarial use - because it will be.
Readers skimming system prompt leaks often stop at definitions. The part that prevents incidents is the verification step after the change - and the habit of re-checking after the next deploy that touches the same layer.
Where secrets should live
Server-side config, secret managers, and policy engines the model cannot read.
The prompt can describe capabilities - not credentials, not internal IP allowlists that are supposed to stay secret.
Common failure mode for system prompt leaks: staging looks fine, production still serves the old config because a CDN, load balancer, or second vhost was never updated. Always verify the hostname customers hit.
Practical checklist
Scan prompts for keys and customer data. Move them out. Add output filters for secret-shaped strings.
Re-test extraction attempts after each prompt change - prompts drift like code without tests.
If this section on system prompt leaks becomes a recurring ticket, automate the check. Manual one-offs rot; a post-deploy assertion or weekly grade keeps the control honest.
What to do this week
- Open every system prompt and search for keys, tokens, internal URLs.
- Move secrets to a secret store; inject at runtime server-side.
- Add output filters for secret-shaped strings.
- Re-run extraction attempts and record what still leaks.
Prompts leak. Secrets belong in secret stores.
Update log (1)
2026-05-29Editorial form rewrite for length and uniqueness.
Sources + verification
Practical guidance based on mainstream browser behavior, common reverse-proxy configuration, and widely published RFCs and vendor docs. Verify on your own stack with curl, browser devtools, and a re-scan after each change.
