Databricks Init Script Not Downloading Packages Logs Say Script Ran Successfully

Question

This morning I had to restart our databricks cluster. Upon restarting, our Open AI library code started failing with this error:

from openai import OpenAI

client = OpenAI(
  organization=openai_organization,
  api_key=openai_key
)
TypeError: __init__() got an unexpected keyword argument 'proxies'

Upon research, (https://community.openai.com/t/error-with-openai-1-56-0-client-init-got-an-unexpected-keyword-argument-proxies/1040332) it looks like the latest version of the package solves the issue.

However, when I update the init script and restart the cluster the version is stuck on version 1.42.0 despite the init script fixing a later version, and databricks saying the download was successful.

What happened? How do I fix this?

Here is the scrit:

#!/bin/bash

echo "Starting library installation..."

# Upgrade pip to the latest version
pip install --upgrade pip

# Remove any existing installations of openai to avoid conflicts
pip uninstall -y openai

# Force reinstall openai and all required packages
pip install --no-cache-dir --force-reinstall \
  openai==1.55.3 \
  df2gspread==1.0.4 \
  dropbox==11.21.0 \
  google-api-python-client==1.6.7 \
  google-auth==2.3.0 \
  google-auth-oauthlib==0.4.6 \
  gspread==4.0.1 \
  pandas==1.3.3 \
  paramiko==2.8.0 \
  requests==2.26.0 \
  requests-oauthlib==1.3.0 \
  timedelta==2020.12.3 \
  networkx==2.7 \
  openpyxl==3.1.2 \
  spacy==3.8.0 \
  transformers==4.45.1 \
  torch==2.4.1 \
  db-dtypes==1.3.0 \
  google-cloud-bigquery==1.15.5 \
  google-auth==2.35.0 

# Install specific spaCy model
pip install --no-cache-dir --force-reinstall \
  https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0.tar.gz

echo "Library installation completed."