AI chatbots and agents
Pentest of your AI chatbot or agent
If your chatbot or AI agent talks to customers or touches your systems, someone will try to trick it. We try first: getting it to reveal data, do things it should not or spend without limit.
Pentest of LLM applications to the OWASP Top 10 for LLM and MITRE ATLAS: direct and indirect prompt injection, tool and permission abuse, data and system prompt leakage, and unbounded consumption.
- Duration
- 2 weeks
- Start
- Within 1 week
from US$2,900
Buy it on its own or add it to an audit. Retests: added to an audit, the same as that audit; on its own, 3 months at no charge.
Test my chatbotHas your chatbot already said something it should not? Switch it off or limit it and write to us today.
It is for you if
- You have a chatbot serving customers on your website, on WhatsApp or in your app.
- Your AI agent can read orders, send email, query your database or move money.
- You loaded internal documents so it can answer and you do not know what it might reveal.
What we check
Whether it reveals other customers’ data, internal documents or its own instructions.Sensitive information disclosure (LLM02) and system prompt leakage (LLM07), including the knowledge base (RAG).
Whether it can be talked into doing something with your systems it should not: delete, send, pay.Excessive agency (LLM06): tool abuse, excess permissions and actions without human confirmation.
Whether its answers can break your site or system when they are shown or used.Improper output handling (LLM05): XSS, query and command injection.
Whether someone can run up a fortune in AI costs by asking it questions nonstop.Unbounded consumption (LLM10): usage caps, per-user limits and cost per conversation.
How we work
Week 0
Scope
You tell us what the chatbot can do, with what data and which systems it reaches.
Map of tools, permissions and data sources; test environment with a spending cap and signed authorization.
Week 1
Attack
We try to trick it every way we know, by hand and with test batteries.
Manual and automated tests per OWASP Top 10 for LLM category, mapped to MITRE ATLAS techniques.
Week 2
Report
A second Lop reviews every finding and we hand you what to change to close it.
Review by a second Lop, technical and executive reports, with the conversations that prove it.
What you get
- Every trick that worked, with the exact conversation and what it could cost you.
- Findings with severity, OWASP LLM category, ATLAS technique, reproducible transcript and impact in money.
- What to change: instructions, permissions, caps and confirmations.
- Remediation per finding: least privilege for tools, input and output filters, human confirmation and usage limits.
- Every fix tested again with the same tricks.
- Retest with the same battery; terms depend on how it is bought.
Scope limits
Includes
- 1 chatbot or 1 agent.1 LLM application, with its tools and its knowledge base.
- Up to 2 user types (for example, visitor and logged-in customer).Up to 2 roles.
Does not include
- We do not test the AI provider’s model (OpenAI, Google, Anthropic), but how you use it.Out of scope: the base model and the provider’s infrastructure.
- We do not audit the rest of your website: that is Dig.No pentest of the web application hosting it.
- We do not rewrite your chatbot: Lop Fix can do that.No remediation; available with Lop Fix.
Several agents or one that handles many tools? We give you a fixed price before signing.
Questions before you buy
How is this different from the chatbot test in Cheshire?
Cheshire includes a short test of your website chatbot, as part of a package for your team. This is the full test of a chatbot or agent, with its tools, its data and its permissions.
Will you burn through my AI account?
No. Before we start we agree on a spending cap and, where possible, we work on a test copy.
Is your chatbot already talking to customers?
In 2 weeks we try to trick it the way an attacker would and tell you what to change.