Hello, loyal readers of Saello Write Up!

Today, I’m sharing a finding that demonstrates a bypass of the question limit on a specific AI chatbot This bug allowed me to ask an unlimited number of questions, despite the predefined policy.

Summary

The AI chatbot system is designed to help users by answering questions related to products. Like many services, it likely has a mechanism to enforce a question limit to manage server load and prevent excessive usage. However, my investigation revealed a flaw in how this limit was enforced on the backend, allowing users to effectively bypass it.

Steps to Reproduce

Here are the precise steps that led to the successful bypass of the question limit:

  1. Create an account at https://website.com/.
  2. On the main page, locate and press the “Need help? Ask our AI chatbot a Question” button. This will initiate the chatbot interface.
  3. Set up a proxy tool, such as Burp Suite, to capture HTTP requests. Ensure your browser is configured to route traffic through this proxy.
  4. Turn on intercept in your proxy tool. In the AI chatbot chat interface, make a general question, for example: “What equipment do I need to get started?” Then, press Enter.
  5. You should capture an HTTP POST request similar to this in your proxy:
POST /api/message HTTP/2
Host: website.com
Cookie: 
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://website.com/?embed=true&theme=light
Content-Type: application/json
Content-Length: 157
Origin: https://website.com
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Te: trailers

{"threadId":"7966aefb-d46f-4d57-90fe-3381b142adf8","text":"how do i setup my AI chatbot?","messageId":"8c098e44-dbf1-4824-9722-501cbb574b16","store":"us"}
  1. Make sure to send this request to the Repeater tab in Burp Suite (or similar feature in other proxy tools) for later manipulation.
  2. Once the request is saved in the Repeater, drop the original intercepted request in the intercept tab. This prevents the original request from reaching the server and consuming your question limit immediately.
  3. Go to the Repeater tab and send the saved request. This will send the question to the AI, and you will observe your question limit being reduced (as this is a legitimate request from the server’s perspective).
  4. Now, here’s the bypass. In the Repeater, use the same request you just sent, but change the question (the value for the "text" field in the JSON payload) to something else, for example, “how do i setup my product network?” (or any new question). Send this modified request.
  5. After sending the modified request, refresh the AI chatbot chat page in your browser. You will observe that your question limit has NOT been reduced despite sending a new question via the Repeater. This confirms the bypass.

By repeating step 8, an unauthorized user can continue to ask questions without their limit being decremented.

Impact of the Vulnerability

The ability to bypass question limits on an AI chatbot, while not directly leading to data theft, can have several significant impacts:

  • Increased Operational Costs: If users can bypass limits, the company may face higher infrastructure costs for maintaining the AI service (e.g., increased API calls to LLM providers, higher server utilization).
  • Resource Exhaustion: Malicious actors could potentially flood the AI system with requests, leading to degraded performance or even denial of service for legitimate users.
  • Policy Violation & Fairness Issues: The system is not functioning according to its predefined policy, creating an unfair situation for users who adhere to the limits and potentially frustrating those who face a legitimate limit while others bypass it.
  • Potential for Abuse: While this PoC focused on question limits, similar flaws in validation could potentially be exploited for more severe issues if the AI’s capabilities extend to sensitive operations.
Recommendations

To address this question limit bypass, the following steps are crucial:

  1. Robust Server-Side Rate Limiting: Implement strict server-side rate limiting and quota enforcement that cannot be bypassed by simply replaying or modifying requests. This should track requests based on user session, IP address, or API key, independent of client-side identifiers.
  2. Comprehensive Request Validation: Ensure that every incoming request to the /api/message endpoint is fully validated against current user quotas, regardless of its threadId or messageId if the content (text) is new.
  3. Strict State Management: Properly manage the state of conversations and question counts on the backend. Each distinct question from a user should trigger a decrement in their quota.
  4. Unique Request Identifiers: If messageId or threadId are meant to track unique interactions for quota purposes, ensure they are strictly enforced and new questions generate truly new, validated identifiers.
Conclusion

Findings like this serve as a reminder that even in seemingly benign features like AI chatbots, robust security validation is paramount. Usage limitations, when improperly enforced, can lead to resource abuse and undermine the intended functionality and fairness of a system.

Stay vigilant and happy hunting!

Regards,

Saello

Visited 22 times, 1 visit(s) today