CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-mlgoogle-cloud-vertex-ai
ggk-1
ggk-1

Reputation: 11

Google AutoML in VertexAI Forecasting took a long time for batch prediction and it drop error log

I created the model in AutoML(Google Cloud) with VertexAI Forecasting below the page.

enter image description here

Then I performed batch prediction, it took about 25 minutes. The data for batch prediction is only 7 records in a CSV file and the output destination is BigQuery.

I have two questions.

1. is it normal for it to take about 25. the video on Youtubu over a year ago was done in about 4 minutes.

2. when i checked the logs, the log that started the batch prediction was dropped at 14:15, then the error log was dropped at 14:35. no logs were dropped for 20 min. Why does the error log drop? The following logs are dropping.

W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so. 11.0: cannot open shared object file: No such file or directory

I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

It seems to be a log that says you do not have a GPU on your machine to do batch predictions, but it is AutoML and I can't do anything about it. How to solve this error?

I will greatly appreciate any help.

Upvotes: 0

Views: 580

Answers (1)

Mac
Mac

Reputation: 48

Unfortunately, this may not be a real answer / solution to your problem, but my guess as to why you received this error was that because there was a shortage of GPUs in your chosen region.

I know from discussions with Google and our experience that certain regions have high demand on GPUs, to the point where at times, there are none available. This is especially true since the explosion of LLMs and Generative AI usage.

I would try this again, or move regions. You can check here for what types of GPUs are generally available in any region. Make sure to check the limitations and requirements for your exact use case.

Upvotes: 0

Related Questions