Vertex AI batch prediction results in ('ValueError', 1)

Question

I have a predictor that works successfully when run on a local container, but when that container is pushed to the Artifact Registry, and imported to a Vertex AI model, and run in Batch Prediction mode, it returns an empty prediction file and an error file that simply says ('ValueError', 1)

The request is made using Flask. code (main.py) is

if 'AIP_HEALTH_ROUTE' in os.environ:
    AIP_HEALTH_ROUTE = os.environ['AIP_HEALTH_ROUTE']
else:
    AIP_HEALTH_ROUTE = '/health'
    
if 'AIP_PREDICT_ROUTE' in os.environ:
    AIP_PREDICT_ROUTE = os.environ['AIP_PREDICT_ROUTE']
else:
    AIP_PREDICT_ROUTE = '/predict'
    
@app.route(AIP_HEALTH_ROUTE, methods=['GET'])
def health():
    response = {"message": "OK", "code": "SUCCESS"}
    return make_response(jsonify(response), 200)

@app.route(AIP_PREDICT_ROUTE, methods=['POST'])
def predict():
    try:
        instances = pd.DataFrame(request.json.get("instances"))
        # creates a model using model and encoders stored at given GCS location
        model = model_active.CustomPredictor('datascience-modelartifacts/my-model/actives')

        processed_data = model.process_score_data(instances)
        predictions = model.predict(processed_data)
        
        response = {"predictions": predictions}
        status = 200

    except Exception as e:
        response = {"error": str(e)}
        status = 500

    return make_response(jsonify(response), status)

For local testing, the data looks like:

dat = {"instances": [
    {"F1": 1000854828492, "F2": 3000076437878, "F3": 19.99, ...},
    {"F1": 1000222326963, "F2": 3000127917915, "F3": 19.99, ...},
    ... ]}

I call this with

import requests
url = 'http://localhost:5000/predict'
headers = {'content-type': 'application/json', 'Accept-Charset': 'UTF-8'}
r = requests.post(url, json=dat, headers=headers)

and in this case everything works as expected (I get a list of predicted classes returned).

I can successfully deploy this model as an endpoint and provide this data as test and get predictions returned.

However, for Vertex batch prediction (which is what I ultimately want to enable since we only need this model once a month), I have tried providing this data as both JSONL and CSV and I keep receiving output with no successful predictions and an error file that simply says ('ValueError', 1)

Does anyone have any suggestions what this error means and what is causing it?

Answer 1

For me this was caused by not wrapping strings in my CSV in quotes, as referenced here. This fixed it:

import csv

df.to_csv("outfile.csv", quoting=csv.QUOTE_NONNUMERIC)