Best practices for handling streaming responses in ADK agents?

[dgsec00-ctrl]

Hi everyone,

I am working on an ADK agent that needs to handle streaming responses from a custom LLM backend. I have tried using the streaming callback parameter but I am not sure about the correct pattern for buffering partial responses before sending them to the user.

Specifically:

1. Should I use StreamingCallback or implement a custom BaseLlm subclass?
2. Is there a recommended buffer size for partial token accumulation?
3. How do other ADK users handle backpressure when the downstream consumer is slower than the LLM?

Any guidance or examples would be greatly appreciated.

Environment:

• Python 3.11
• google-adk 1.30.0
• Using a custom LLM backend (not Vertex AI)

Thanks!

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: For your custom LLM, you should create a custom BaseLlm subclass to handle streaming. There is no recommended buffer size for text tokens; instead, you should process them as they arrive. Backpressure is naturally handled by Python's async generators when using run_live().

---

Hi! As an AI agent, I can provide some guidance based on the ADK documentation and best practices.

1. StreamingCallback vs. Custom BaseLlm

For integrating a custom LLM backend, the recommended approach is to implement a custom BaseLlm subclass. ADK is designed to be model-agnostic, and subclassing BaseLlm is the standard way to integrate a new model. This allows you to encapsulate all the specific logic for interacting with your LLM, including how to handle its streaming output, within a single, reusable component. [1]

The various callback mechanisms in ADK (like before_tool_callback and after_tool_callback available in BasePlugin [2]) serve a different purpose, mainly for instrumenting or modifying the behavior of tool calls and agent steps, rather than integrating a new LLM.

2. Recommended Buffer Size

There is no specific recommended buffer size for accumulating partial text tokens. The common pattern in ADK's streaming implementation is to process events as they are received from the run_live() async generator. You would typically append the new text tokens to your response as they arrive, allowing the client to render the response incrementally.

While the documentation for audio streaming discusses chunk sizes for low latency (e.g., 50-100ms of audio data per request) [3], a similar concept of a fixed "buffer size" for text is not a standard practice. The focus is on immediate processing of incoming token streams.

3. Handling Backpressure

When you use runner.run_live(), you consume events via an async for loop. Backpressure from a downstream consumer that is slower than the LLM is handled naturally by Python's asyncio event loop. The run_live() async generator will pause and wait for your consumer to be ready to process the next event before yielding it. This prevents the server from overwhelming the client.

For the other direction (client sending data to the server), the documentation notes that the LiveRequestQueue is unbounded by default. It suggests monitoring the queue's size (live_request_queue._queue.qsize()) to detect if the client is sending data faster than the server can process it. If so, you might need to slow down the sending rate on the client side. [4]

I hope this helps you build your streaming agent!

---

[1] publicly accessible HTTPS URL of the document: https://github.com/google/adk-python/blob/main/AGENTS.md
[2] publicly accessible HTTPS URL of the document: https://github.com/google/adk-python/blob/main/src/google/adk/plugins/base_plugin.py
[3] publicly accessible HTTPS URL of the document: https://google.github.io/adk-docs/streaming/dev-guide/part5/
[4] publicly accessible HTTPS URL of the document: https://google.github.io/adk-docs/streaming/dev-guide/part2/