Skip to content

fix(gemini): set IncludeServerSideToolInvocations when using built-in tools with function calling#762

Open
jredl-va wants to merge 9 commits into
cloudwego:mainfrom
jredl-va:fix/gemini-server-side-tool-invocations
Open

fix(gemini): set IncludeServerSideToolInvocations when using built-in tools with function calling#762
jredl-va wants to merge 9 commits into
cloudwego:mainfrom
jredl-va:fix/gemini-server-side-tool-invocations

Conversation

@jredl-va

@jredl-va jredl-va commented Apr 6, 2026

Copy link
Copy Markdown

What type of PR is this?

fix: A bug fix

Check the PR title.

  • This PR title match the format: <type>(optional scope): <description>
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Attach the PR updating the user documentation if the current PR requires user awareness at the usage level.

(Optional) More detailed description for this PR(en: English/zh: Chinese).

en:
When built-in tools (GoogleSearch, URLContext, CodeExecution, ComputerUse, FileSearch, GoogleSearchRetrieval, GoogleMaps) are used alongside function calling tools, the Gemini API requires ToolConfig.IncludeServerSideToolInvocations to be set to true. Without this, the API returns INVALID_ARGUMENT: Please enable tool_config.include_server_side_tool_invocations to use Built-in tools with Function calling.

This PR:

  1. Bumps google.golang.org/genai from v1.36.0 to v1.52.1 (the IncludeServerSideToolInvocations field was added in v1.51.0)
  2. Automatically sets IncludeServerSideToolInvocations = true in the ToolConfig when any built-in tool is present alongside function declarations

(Optional) Which issue(s) this PR fixes:

Fixes #761

… tools with function calling

When built-in tools (GoogleSearch, URLContext, CodeExecution, ComputerUse,
FileSearch, GoogleSearchRetrieval, GoogleMaps) are used alongside function
calling, the Gemini API requires IncludeServerSideToolInvocations to be
enabled in the ToolConfig, otherwise it returns INVALID_ARGUMENT.

This bumps google.golang.org/genai to v1.52.1 (field added in v1.51.0)
and automatically sets the flag when any built-in tool is present alongside
function declarations.

Fixes cloudwego#761
@CLAassistant

CLAassistant commented Apr 6, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jredl-va

jredl-va commented Apr 6, 2026

Copy link
Copy Markdown
Author

cc @meguminnnnnnnnn @N3kox @JonXSnow — tagging for visibility as top contributors to the gemini component.

@jredl-va

Copy link
Copy Markdown
Author

@JonXSnow my apologies for the second direct ping, but I was hoping to get a review from you on this. This allows us to set labels on the vertex api calls for better telemetry.

(please let me know if there is a contributors group / process for these tiny types of maintenance adjustments to conform to the changes in the vertex api)

@tangchaojun-bytedance

Copy link
Copy Markdown
Collaborator

@jredl-va Thanks for the fix. I think this needs two more changes before merge.

  1. When include_server_side_tool_invocations is enabled, AUTO mode is not supported. The Gemini docs say it should default to VALIDATED:
    https://ai.google.dev/gemini-api/docs/tool-combination#limitations

Currently WithTools / BindTools can still produce FunctionCallingConfigModeAuto, so built-in tools + function declarations may still generate an invalid config. We should switch AUTO to VALIDATED in this case.

  1. This flag is not supported by the Vertex backend. In google.golang.org/genai@v1.52.1, toolConfigToVertex rejects includeServerSideToolInvocations:
    https://github.com/googleapis/go-genai/blob/v1.52.1/models.go#L3968-L3970

So we should return a clear unsupported error for BackendVertexAI instead of setting the flag unconditionally.

Could you also add regression tests for these two cases?

jredl-va and others added 2 commits June 4, 2026 07:15
…ide tool invocations

AUTO function-calling mode is not supported when
IncludeServerSideToolInvocations is enabled, so coerce AUTO/unspecified
modes to VALIDATED when built-in tools are combined with function
declarations. The flag is also rejected by the Vertex AI backend, so
return a clear error instead of letting the request fail in the SDK.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jredl-va

jredl-va commented Jun 4, 2026

Copy link
Copy Markdown
Author

@tangchaojun-bytedance thanks for the feedback! Both issues addressed with proper test coverage.

  • AUTO → VALIDATED coercion when built-in tools combine with function declarations (handles explicit AUTO, MODE_UNSPECIFIED, and the empty-string default).
  • Vertex backend returns a clear up-front error instead of failing in the SDK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Gemini: INVALID_ARGUMENT when using built-in tools (GoogleSearch, etc.) with function calling

3 participants