3 ways to solve OpenClaw Gemini image recognition failures: common errors in OpenAI compatibility mode and native format configuration guide

When using OpenAI-compatible mode in OpenClaw to invoke Gemini models for image recognition, you'll often run into errors. This is a common pain point for developers configuring multimodal AI agents. This article dives deep into the root cause of the "Invalid JSON payload" error and provides three verified solutions to help you quickly fix OpenClaw Gemini image recognition failures.

Core Value: After reading this, you'll understand the key differences between OpenAI-compatible mode and the native Gemini API, master the correct configuration, and permanently resolve image recognition failures.

Error Symptoms of OpenClaw Gemini Image Recognition Failures

After configuring a Gemini model in OpenClaw, you'll often see the following typical error in the backend logs when attempting image recognition:

Invalid JSON payload received. Unknown name "patternProperties"
at 'tools[0].function_declarations[3].parameters.properties[4].value':
Cannot find field.

Invalid JSON payload received. Unknown name "const"
at 'tools[0].function_declarations[37].parameters.properties[0].value':
Cannot find field.

Key Characteristics of the OpenClaw Gemini Image Recognition Error

Quick Diagnosis of OpenClaw Gemini Image Recognition Failures

A common misconception is that Gemini's image recognition capabilities are flawed. In reality, testing the API directly using Gemini's official visual understanding demo works perfectly. The problem lies in the format incompatibility when OpenClaw forwards requests via OpenAI-compatible mode.

The verification method is simple:

# Direct call to Gemini API to test image recognition — works perfectly
import google.generativeai as genai
import PIL.Image

genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")

image = PIL.Image.open("test.jpg")
response = model.generate_content(["Describe this image", image])
print(response.text)  # ✅ Normal output of image description

🎯 Diagnostic Advice: If you encounter Gemini image recognition issues in OpenClaw, first use the method above to confirm that your API key and the model itself are functioning correctly. You can also quickly test Gemini's visual understanding capabilities via the APIYI (apiyi.com) platform, which automatically handles format compatibility issues.

Root Cause Analysis of OpenClaw Gemini Image Recognition Failures

To choose the right solution, you first need to understand the root cause. The primary reason OpenClaw fails at image recognition when using Gemini is a JSON Schema compatibility issue.

JSON Schema Differences: OpenAI vs. Gemini Tool Calling

When OpenClaw uses the OpenAI-compatible mode (openai-completions) to call Gemini, the request flow looks like this:

OpenClaw builds request (OpenAI format)
    ↓
Includes tool-defined JSON Schema
    ↓
Sent to Gemini OpenAI-compatible endpoint
    ↓
Gemini parses function_declarations
    ↓
❌ Encounters unsupported Schema fields → 400 Error

List of JSON Schema Fields Unsupported by Gemini API

This is the crux of the problem. Gemini's support for function_declarations in JSON Schema is a limited subset. The following fields will trigger a 400 error:

Why Switching to GPT-5.4 Works

This further confirms our root cause analysis. When you switch the model from Gemini to GPT-5.4 in OpenClaw, image recognition works perfectly. That's because the GPT-5.4 API natively supports the full JSON Schema specification, making the tool definition schema generated by OpenClaw fully compatible.

📌 Key Takeaway: This isn't an issue with Gemini's image recognition capabilities, but rather a mismatch between the tool schema sent by OpenClaw's OpenAI-compatible mode and the specific format requirements of the Gemini API.

Solution 1: Switch to Native Gemini Format (Recommended)

The most robust solution is to switch the Gemini API type in OpenClaw from openai-completions to the native google-generative-ai format.

Configuration Steps

Before (OpenAI Compatibility Mode — Issues may occur):

{
  "provider": "google",
  "model": "gemini-2.5-flash",
  "baseUrl": "https://generativelanguage.googleapis.com/v1beta/openai",
  "api": "openai-completions",
  "apiKey": "YOUR_GEMINI_API_KEY"
}

After (Native Gemini Format — Recommended):

{
  "provider": "google",
  "model": "gemini-2.5-flash",
  "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
  "api": "google-generative-ai",
  "apiKey": "YOUR_GEMINI_API_KEY"
}

Key Changes in Native Configuration

Quick Switch Using OpenClaw CLI

# Method 1: Re-initialize Gemini configuration
openclaw onboard --auth-choice gemini-api-key

# Method 2: Manually edit the config file
# Config file location: ~/.openclaw/config.json
# Change the api field from "openai-completions" to "google-generative-ai"

{
  "providers": {
    "google": {
      "apiKey": "YOUR_GEMINI_API_KEY",
      "models": {
        "gemini-2.5-flash": {
          "api": "google-generative-ai",
          "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
          "capabilities": {
            "vision": true,
            "functionCalling": true,
            "streaming": true
          },
          "reasoning": false
        },
        "gemini-2.5-pro": {
          "api": "google-generative-ai",
          "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
          "capabilities": {
            "vision": true,
            "functionCalling": true,
            "streaming": true
          },
          "reasoning": true,
          "thinkingBudget": 8192
        }
      }
    }
  }
}

🚀 Quick Start: If you'd rather not deal with manual configuration compatibility, we recommend using the unified interface provided by APIYI (apiyi.com). The platform automatically converts OpenAI-formatted requests into the native Gemini format, so you don't have to worry about schema differences.

Solution 2: Handle Compatibility via an API Proxy Service

If you prefer to keep using OpenAI compatibility mode in OpenClaw to call multiple models (including Gemini), you can resolve these format compatibility issues by using an API proxy service.

How the API Proxy Service Works

OpenClaw (OpenAI format request)
    ↓
API Proxy Service (e.g., APIYI)
    ↓ Automatically cleans up incompatible JSON Schema fields
    ↓ Automatically converts request format
Gemini API (Native format)
    ↓
✅ Returns image recognition results successfully

Configuration Example

# Call Gemini image recognition via the APIYI proxy service
import openai
import base64

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"  # APIYI unified interface
)

# Read and encode the image
with open("test.jpg", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gemini-2.5-flash",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the content of this image"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}"
                    }
                }
            ]
        }
    ]
)
print(response.choices[0].message.content)

Proxy Service vs. Direct Connection

Configure APIYI proxy in OpenClaw:

{
  "provider": "apiyi",
  "model": "gemini-2.5-flash",
  "baseUrl": "https://api.apiyi.com/v1",
  "api": "openai-completions",
  "apiKey": "YOUR_APIYI_KEY"
}

💡 Recommendation: If you're using multiple models in OpenClaw (GPT-5.4, Claude, Gemini, etc.), managing your model invocations through APIYI (apiyi.com) is a much more efficient choice, as it saves you from having to configure different API formats for every single model.

Solution 3: Manually Cleaning Incompatible JSON Schema Fields

If you need to resolve compatibility issues at the code level, you can manually clean up JSON Schema fields that Gemini doesn't support before sending your request.

JSON Schema Cleaning Function

def clean_schema_for_gemini(schema: dict) -> dict:
    """Clean up JSON Schema fields not supported by Gemini"""
    unsupported_keys = {
        "patternProperties",
        "const",
        "additionalProperties",
        "$schema",
        "exclusiveMaximum",
        "exclusiveMinimum",
        "propertyNames",
    }

    if isinstance(schema, dict):
        return {
            k: clean_schema_for_gemini(v)
            for k, v in schema.items()
            if k not in unsupported_keys
        }
    elif isinstance(schema, list):
        return [clean_schema_for_gemini(item) for item in schema]
    return schema

import openai
import json

def clean_schema_for_gemini(schema):
    """Recursively clean up JSON Schema fields not supported by Gemini"""
    unsupported_keys = {
        "patternProperties", "const", "additionalProperties",
        "$schema", "exclusiveMaximum", "exclusiveMinimum",
        "propertyNames", "if", "then", "else",
        "allOf", "anyOf", "oneOf", "not",
    }

    if isinstance(schema, dict):
        cleaned = {}
        for k, v in schema.items():
            if k not in unsupported_keys:
                cleaned[k] = clean_schema_for_gemini(v)
        return cleaned
    elif isinstance(schema, list):
        return [clean_schema_for_gemini(item) for item in schema]
    return schema

def clean_tools_for_gemini(tools):
    """Clean all schemas within the tool list"""
    cleaned_tools = []
    for tool in tools:
        tool_copy = json.loads(json.dumps(tool))
        if "function" in tool_copy:
            params = tool_copy["function"].get("parameters", {})
            tool_copy["function"]["parameters"] = clean_schema_for_gemini(params)
        cleaned_tools.append(tool_copy)
    return cleaned_tools

# Usage example
tools = [
    {
        "type": "function",
        "function": {
            "name": "analyze_image",
            "description": "Analyze image content",
            "parameters": {
                "type": "object",
                "properties": {
                    "image_url": {"type": "string"},
                    "detail": {"type": "string", "const": "high"}  # Not supported by Gemini
                },
                "patternProperties": {"^x-": {"type": "string"}},  # Not supported by Gemini
                "additionalProperties": False  # Not supported by Gemini
            }
        }
    }
]

# Clean before invocation
cleaned_tools = clean_tools_for_gemini(tools)

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"
)

response = client.chat.completions.create(
    model="gemini-2.5-flash",
    messages=[{"role": "user", "content": "Hello"}],
    tools=cleaned_tools
)

⚠️ Note: The manual schema cleaning approach requires you to process parameter definitions for every tool, which can be high-maintenance. If you have many tools or they change frequently, we recommend prioritizing Solution 1 (Native Format) or Solution 2 (API proxy service).

Comparison and Selection Guide for the 3 Solutions

Selection Decision Tree

• Using Gemini only in OpenClaw → Solution 1 (Native Format), most stable.
• Mixing multiple models in OpenClaw → Solution 2 (APIYI proxy), most effortless.
• Need fine-grained control for custom AI apps → Solution 3 (Manual Cleaning), most flexible.
• Not sure which to pick? → Try Solution 2 first; verify quickly via APIYI (apiyi.com).

FAQ

Summary

The root cause of OpenClaw failing at Gemini image recognition is JSON Schema field incompatibility within the OpenAI-compatible mode, rather than any issue with the Gemini model's visual capabilities. There are three solutions, each suited for different scenarios:

• Native Format (google-generative-ai): The most thorough approach; recommended if you are using Gemini exclusively.
• API Proxy Service: The most hassle-free approach; recommended if you are using a mix of different models.
• Manual Schema Cleaning: The most flexible approach; recommended if you are building your own custom system.

We recommend using APIYI (apiyi.com) to quickly verify Gemini's image recognition performance. The platform supports unified calls for mainstream models like Gemini, GPT, and Claude, automatically handling the API format differences between them.

References

1. Gemini Official Documentation – Image Understanding: Explanation of Gemini's visual understanding capabilities

   • Link: ai.google.dev/gemini-api/docs/image-understanding

2. Gemini Official Documentation – OpenAI Compatibility: Instructions for calling Gemini using the OpenAI SDK

   • Link: ai.google.dev/gemini-api/docs/openai

3. OpenClaw GitHub Issue #21172: patternProperties causing Gemini API 400 errors

   • Link: github.com/openclaw/openclaw/issues/21172

4. OpenClaw GitHub Issue #14456: Gemini 2.5 Flash OpenAI compatibility mode 400 errors

   • Link: github.com/openclaw/openclaw/issues/14456

5. OpenClaw Model Configuration Documentation: Guide to configuring model providers

   • Link: docs.openclaw.ai/concepts/model-providers

📝 Author: APIYI Team — Focused on Large Language Model API integration and technical analysis
🔗 More Tutorials: Visit APIYI at apiyi.com for more model invocation guides and free testing credits