Unexpected InvalidArgument error for large response_schema

[seidtgeist]

I’ve encountered an issue with the Gemini 1.5 Pro API where there seems to be an undocumented size limit for the response_schema parameter in GenerationConfig. When attempting to use a schema with a large number of properties or long property names, the API throws an InvalidArgument error without further details.

Environment details

• OS type and version: Ubuntu Linux 24.04 / macOS 14.6.1
• Python version: 3.12.5
• pip version: 24.0
• google-cloud-aiplatform version: 1.63.0

Steps to reproduce

1. Run the below code with ADC set up
2. Observe InvalidArgument exception w/o further details
3. Change the parameters and re-run the case. There's a sweet spot to the number of schema keys and their string length:

• Normal response:
  • num_properties = 42
  • property_name_length = 34
• InvalidArgument:
  • num_properties = 43
  • property_name_length = 34
• InvalidArgument:
  • num_properties = 42
  • property_name_length = 35

Code example

import copy
from pprint import pprint
import random
import string
from vertexai.generative_models import GenerationConfig, GenerativeModel
from google.api_core.exceptions import InvalidArgument


prompt = "Respond according to the JSON schema."
num_properties = 40
property_name_length = 40

properties = [
    "".join(random.choices(string.ascii_lowercase, k=property_name_length))
    for _ in range(num_properties)
]

json_schema = {
    "type": "object",
    "properties": {name: {"type": "string"} for name in properties},
}

model = GenerativeModel("gemini-1.5-pro-001")

try:
    response = await model.generate_content_async(
        contents=prompt,
        generation_config=GenerationConfig(
            temperature=0.0,
            response_mime_type="application/json",
            response_schema=copy.deepcopy(json_schema),
        ),
        stream=True,
    )
except InvalidArgument as e:
    print("Request failed as expected with InvalidArgument error:")
    print(e)
    print(
        f"generation_config.response_schema had {num_properties} properties, {property_name_length} characters each:"
    )
    pprint(json_schema)

Stack trace

---------------------------------------------------------------------------
AioRpcError                               Traceback (most recent call last)
File .venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:77, in _WrappedCall.wait_for_connection(self)
     76 try:
---> 77     await self._call.wait_for_connection()
     78 except grpc.RpcError as rpc_error:

File .venv/lib/python3.12/site-packages/grpc/aio/_call.py:650, in UnaryStreamCall.wait_for_connection(self)
    649 if self.done():
--> 650     await self._raise_for_status()

File .venv/lib/python3.12/site-packages/grpc/aio/_call.py:263, in Call._raise_for_status(self)
    262 if code != grpc.StatusCode.OK:
--> 263     raise _create_rpc_error(
    264         await self.initial_metadata(), await self._cython_call.status()
    265     )

AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Request contains an invalid argument."
	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2a00:1450:4001:828::200a%5D:443 {created_time:"2024-08-27T13:35:59.384259+02:00", grpc_status:3, grpc_message:"Request contains an invalid argument."}"
>

The above exception was the direct cause of the following exception:
…
     77     await self._call.wait_for_connection()
     78 except grpc.RpcError as rpc_error:
---> 79     raise exceptions.from_grpc_error(rpc_error) from rpc_error

InvalidArgument: 400 Request contains an invalid argument.

Note that I've actually also received a 500 AioRpcError response.

---

Is there an internal size limit on the in-memory size of a response_schema? If so, should there be? Is there any documentation on it?

(I'm aware this is likely not an issue with the Python API client but the service itself but hey, the API client is generated from a service description sooo… :D)

[Ark-kun]

Thank you for the report. I'm investigating...
P.S. When you experience some issues it's usually worth check whether non-streaming and non-async behaves differently. I've encountered cases where non-streaming errors are more actionable compared to the streaming mode.

[PintJaguar]

Seconding this. I have a case where i want to ask many attributes of a product and at a certain amount, i also get the error.