Google Gemini-Pro raises "InternalServerError: 500 Internal error encountered." when passed a FunctionDeclaration that includes an array of object

Question

I am using Gemini-Pro function calling in VertexAI. It works with simple function calls, but when I include a property that is an array of items that are objects, it raises "InternalServerError: 500 Internal error encountered."

The following code gets the error. Note that 'contact' is an array, where the items are objects.

from vertexai.preview.generative_models import FunctionDeclaration, GenerativeModel, GenerationConfig, Tool


function = {
    'name': 'Save',
    'description': 'Saves contact data for a person',
    'parameters': {
        'type_': 'OBJECT',
        'properties': {
            'name': {
                'type_': 'STRING',
                'description': "The person's name"
            },
            'contact': {
                'type_': 'ARRAY',
                'description': "The person's contact information",
                'items': {
                    'type_': 'OBJECT',
                    'description': 'Contact information for the person in a specific context',
                    'properties': {
                        'email': {
                            'type_': 'STRING',
                            'description': "The person's email"
                        },
                        'phone': {
                            'type_': 'STRING',
                            'description': "The person's phone number"
                        },
                        'context': {
                            'type_': 'STRING',
                            'description': 'The context in which the contact information is applicable',
                            'enum': ['work', 'personal', 'other']
                        }
                    },
                }
            }
        }
    }
}

func = FunctionDeclaration(**function)

prompt = (
    "Extract the contact information from the below data and store it using the function provided.\n"
    "John Smith, work: john.smith@smithinc.com, 212-555-5309, home: johnny@thesmiths.com, 917-555-9512"
)

config = GenerationConfig(
    temperature=0.1,
    top_p=0.95,
    top_k=20,
    candidate_count=1,
    max_output_tokens=2048,
    stop_sequences=[]
)

llm = GenerativeModel('gemini-pro', generation_config=config, tools=[Tool(function_declarations=[func])])
llm.generate_content(prompt, stream=False)

However, if I change 'contact' to just be a single object, not an array (as per below), it works

function = {
    'name': 'Save',
    'description': 'Saves contact data for a person',
    'parameters': {
        'type_': 'OBJECT',
        'properties': {
            'name': {
                'type_': 'STRING',
                'description': "The person's name"
            },
            'contact': {
                'type_': 'OBJECT',
                'description': "The person's contact information",
                'properties': {
                    'email': {
                        'type_': 'STRING',
                        'description': "The person's email"
                    },
                    'phone': {
                        'type_': 'STRING',
                        'description': "The person's phone number"
                    },
                    'context': {
                        'type_': 'STRING',
                        'description': 'The context in which the contact information is applicable',
                        'enum': ['work', 'personal', 'other']
                    }
                }
            }
        }
    }
}

It also works if each of 'phone' and 'email' are arrays of string (not of object):

function = {
    'name': 'Save',
    'description': 'Saves contact data for a person',
    'parameters': {
        'type_': 'OBJECT',
        'properties': {
            'name': {
                'type_': 'STRING',
                'description': "The person's name"
            },
            'contact': {
                'type_': 'OBJECT',
                'description': "The person's contact information",
                'properties': {
                    'email': {
                        'type_': 'ARRAY',
                        'description': "The person's email addresses",
                        'items': {'type_': 'STRING'}
                    },
                    'phone': {
                        'type_': 'ARRAY',
                        'description': "The person's phone numbers",
                        'items': {'type_': 'STRING'}
                    }
                }
            }
        }
    }
}

What am I doing wrong in the first function specification?

Answer 1

It sounds like that you're running into a Gemini function calling bug that was reported recently when specifying an array type in the FunctionDeclaration:

https://github.com/GoogleCloudPlatform/generative-ai/issues/418

The bug was reported and a fixed is being worked on:

https://issuetracker.google.com/326497502

So, stay tuned to that issue to know when it's fixed!