How to Disable Safety Settings in Gemini Vision Pro Model Using API?

Question

Question:

I'm currently working with the Gemini Vision Pro model and I'm encountering issues with certain images while using the model through its API. Here's a simplified version of the code I'm using to access the model:

python:-

import google.generativeai as genai
import streamlit as st

def Gemini_vision(prompt, image):
    st.markdown("<p style='text-align:center;'>Image will be used from here, Delete image when you are done ✅</p>", unsafe_allow_html=True)
    genai.configure(api_key=GEMINI_API_KEY)
    model = genai.GenerativeModel('gemini-pro-vision')
    safe = [
        {
            "category": "HARM_CATEGORY_DANGEROUS",
            "threshold": "BLOCK_NONE",
        },
        {
            "category": "HARM_CATEGORY_HARASSMENT",
            "threshold": "BLOCK_NONE",
        },
        {
            "category": "HARM_CATEGORY_HATE_SPEECH",
            "threshold": "BLOCK_NONE",
        },
        {
            "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
            "threshold": "BLOCK_NONE",
        },
        {
            "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
            "threshold": "BLOCK_NONE",
        },
    ]
    response = model.generate_content([image[0], prompt], safety_settings=safe)
    return response.text

The issue arises when I try to generate content using certain images. The model throws an error, but strangely, it works fine with other images. I suspect that the safety settings in the model might be causing this discrepancy.

I got this error while uploading my selfie and asked about something:-

google.api_core.exceptions.InvalidArgument: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you're on Streamlit Cloud, click on 'Manage app' in the lower right of your app).

My Question:

How can I disable or turn off all safety settings in the Gemini Vision Pro model using the API? Are there any additional parameters or configurations I need to consider while making this adjustment? I've tried passing an empty list for the safety_settings parameter, but it doesn't seem to have any effect. Any insights or suggestions on how to handle this situation would be greatly appreciated.

Thank you in advance!

Feel free to customize the question further or let me know if you need any modifications!

Answer 1

For me none of the above code worked as it generate_content function expect SafetySetting object and list below is the code which worked for me

from vertexai.generative_models import (
    GenerativeModel,
    HarmCategory,
    HarmBlockThreshold,
    Part,
    SafetySetting,
    GenerationConfig
)

# Safety config
safety_config = [
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HATE_SPEECH,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
]

model = GenerativeModel("gemini-1.5-flash")
model.generate_content(prompt, stream=False,safety_settings=safety_config)

Answer 2

I find I have to try a few times in a loop because Gemini often throws errors.

def generate_content_with_gemini_from_text(prompt):
model = genai.GenerativeModel(        
    model_name="gemini-1.5-flash",
    generation_config=generation_config,
    safety_settings = safe
)
e = None  # Initialize e before the try-except block
for attempt in range(4):
    try:
        response = model.generate_content(prompt)            
        return response.text
    except Exception as caught_exception:
        e = caught_exception  # Update e with the caught exception
        print(f"Error on attempt {attempt + 1}: {e}")
        time.sleep(1)
if e:
    print(f"Failed after 4 retries. Error: {e}")
else:
    print("Failed after 4 retries due to an unknown error.")
return None

Answer 3

I managed to get it working by modifying the safe setting as described below.
Give it a try and see if it works for you too.

safe = [
    {
        "category": "HARM_CATEGORY_HARASSMENT",
        "threshold": "BLOCK_NONE",
    },
    {
        "category": "HARM_CATEGORY_HATE_SPEECH",
        "threshold": "BLOCK_NONE",
    },
    {
        "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
        "threshold": "BLOCK_NONE",
    },
    {
        "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
        "threshold": "BLOCK_NONE",
    },
]

Answer 4

Without seeing sample images that are causing the problem and the exact error and JSON returned, it is a little difficult to diagnose exactly. However, your guess about the safety settings is probably accurate.

To answer your questions about the safety settings:

How can I turn off all the safety settings?

While you can turn most of them off using the BLOCK_NONE threshold as you've done, there are still some settings that are always enforced (for example, child safety related content), and the Google filters may be a little ambitious on these.

Are there any other parameters I need to consider?

Looking at the other run parameters such as temperature, top p, and top k can sometimes help, but these are less likely. Certainly the text prompt that accompanies the picture also goes a long way in shaping how it handles the output as well, so you might be able to better shape a response that doesn't run afoul of the safety system.

Passing an empty list for the safety parameters doesn't help

Passing an empty list is the same as having it use the defaults, which set it for a moderate safety level.