CODEWITHSUNDEEP
airflowgoogle-cloud-mlgoogle-cloud-composer
JY2k
JY2k

Reputation: 2909

Airflow ML Engine Package_URI

What is the package URI used for? Is it mandatory? If so how do I create one? Currently I have my model package into the proper format of:

model.py
task.py
_init_.py

Upvotes: 0

Views: 216

Answers (2)

JY2k
JY2k

Reputation: 2909

I was able to manually build the package and place it in the Cloud composer bucket. I then supplied the path to the file in the bucket:

package_uris=["gs://us-central1-ml-engine/trainer-0.1.tar.gz"]

Upvotes: 0

Younghee Kwon
Younghee Kwon

Reputation: 51

I assume you're asking about package_uris in MLEngineTrainingOperator.

The instructions about it can be found in Cloud ML Engine Documentations. One thing different is that unlike when using gcloud, it is mandatory for Airflow integration that you need to provide/pakcage a package yourself since Airflow operators are running remotely and it cannot package from your local directory.

What you might need here is setup.py based on setuptools, with proper dependencies. (btw, _init_.py is not a valid file, __init__.py is.) When the directories are ready, you can simply run the following command to upload the package.

python setup.py sdist 
gsutil cp dist/<tarfile> gs://<your_bucket>/<folder>/

Or, if you already have a package that's uploaded using gcloud ml-engine jobs submit training command, you can simply provide the uri to reuse it.

Upvotes: 1

Related Questions