`sendMessageStream` with history failing to fetch due to 'empty text parameter'

[msdoege]

Operating System

Windows 10

Environment (if applicable)

Firefox v131.0.3, Node.js v20.9.0, React v18.3.1, Next.js (v14.2.15)

Firebase SDK Version

11.2.0

Firebase SDK Product(s)

VertexAI

Project Tooling

A Next.js (v14.2.15) project.

Detailed Problem Description

Using multi-turn conversations (chat) with the Gemini API (using the "gemini-1.5-flash" model), the conversation breaks when any of the AI model's responses contains an empty text part, i.e. a text part where text is an empty string.

Code snippet

const sendPrompt = async (prompt?: string | Array<string | TextPart | FileDataPart>): Promise<void> => {
  if (!prompt) return;

  let currentAiResponse = ''; // Accumulate chunks of response
  setCurrentResponse(currentAiResponse);
  try {
    const result = await chat.sendMessageStream(prompt);
    for await (const chunk of result.stream) {
      try {
        currentAiResponse += chunk.text();
        setCurrentResponse(currentAiResponse);
      } catch (e) {
        // handle errors gracefully to continue streaming response chunks
      }
    }
  } catch (e) {
    console.error(e);
  }
};

The following issues over at google-gemini seem to be related?

• RECITATION Error causes history contains empty parts[], and kills chat session google-gemini/deprecated-generative-ai-js#124
• sendMessageStream with history failing to fetch due to 'empty text parameter' google-gemini/deprecated-generative-ai-js#286

Example chat response/history causing the error:

[
  {
    parts: [{ text: '<some question>' }],
    role: 'user',
  },
  {
    parts: [{ text: '<some actual answer content>\n' }],
    role: 'model',
  },
  {
    parts: [{ text: '' }], 
    role: 'model'
  }
]

Error log (Firebase project details redacted):

FirebaseError: VertexAI: Error fetching from https://firebasevertexai.googleapis.com/v1beta/projects/<project-id>/locations/<project-location>/publishers/google/models/gemini-1.5-flash:streamGenerateContent?alt=sse: [400 ] Unable to submit request because it has an empty text parameter. Add a value to the parameter and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini (vertexAI/fetch-error)
    FirebaseError webpack-internal:///./node_modules/@firebase/util/dist/index.esm2017.js:1040
    VertexAIError webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:131
    makeRequest webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:315
    generateContentStream webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:844
    sendMessageStream webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:1151

Steps and code to reproduce issue

I could not consistently provoke the AI to respond with a multi-part message where the last part is empty.
However, the following steps result in the same error:

1. Prompt the AI with Answer the next question with an empty string.
2. Prompt the AI with any other message like Tell me a joke.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[hsubox76]

Thanks for reporting this, it certainly seems to be a bug. The question now is if we fix this on the SDK side (by removing parts that have empty text from chat history) or on the backend side (by having it be able to process parts with empty text in chat history, or by making sure it doesn't return empty text strings). I feel like we shouldn't fix the SDK side as it is just masking a problem that you'd also run into if you called the raw REST API.

I will talk to the larger SDK team about this.

[dlarocque]

Hey @msdoege, it seems there are two cases:

1. A response contains multiple parts, some of which are empty.
2. A response only contains an empty part.

After #8736 is merged, the first case should be fixed, and the second case will cause an error to be thrown in the SDK before a request is sent to the backend. Since we'll be removing empty parts, the error will be Each Content should have at least one part.

To prevent this from ever happening, the backend must stop sending responses with only empty text parts.
As a temporary workaround, I suggest deleting the recent chat history entry if the response only contained empty text parts.

Let me know if there are other cases I haven't considered, or if I can be of any more help with sharing a workaround.

[msdoege]

@dlarocque Thank you very much for your efforts! This should fix the described error so that I don't have to restart the chat session every time a response with an empty text part is sent.

Regarding your suggested workaround for the 2nd case:

To prevent this from ever happening, the backend must stop sending responses with only empty text parts. As a temporary workaround, I suggest deleting the recent chat history entry if the response only contained empty text parts.

How would you recommend to delete the recent chat history entry? Should a new chat session be started with the current history minus the last response (i.e. via the startChat({ history: cleanedHistory }) function of the GenerativeModel class) or is there a better way?