Reputation: 325
Set up:
- tracking server openstack instance
- mlflow tracking database on second openstack instance
- traring on on-prem HPC node
Tracking server start
python3 -m mlflow server --dev --app-name oidc-auth --host 0.0.0.0
--port 8080 --backend-store-uri postgresql+psycopg2://$DB_USERNAME:$DB_PASSWORD@$DB_IP_ADDRESS:5432/$DB_NAME
Experiment (i need artifact_location because each research group will want to use their on remote directory
# Create a new experiment
experiment = mlflow.get_experiment_by_name(experiment_name)
if experiment is None:
exp = mlflow.create_experiment(experiment_name, artifact_location='/lustre/scratch127/mlops_artifacts')
set_experiment = mlflow.set_experiment(experiment_id=exp)
else:
set_experiment = mlflow.set_experiment(experiment_name)
From logs
First Experiment Elastic Net Name: exp_EL_research Experiment_id: 8 Artifact Location: /lustre/scratch127/mlops_artifacts Active run id is fff785c73a814fc89102ed7a486de96d Active run name is bedecked-grub-472 Artifact URI is /lustre/scratch127/mlops_artifacts/fff785c73a814fc89102ed7a486de96d/artifacts
first question why artifact_uri is not /lustre/scratch127/mlops_artifacts/8/fff785c73a814fc89102ed7a486de96d/artifacts ??
# Log the model
mlflow.sklearn.log_model(sk_model=lr, artifact_path=artifact_path, registered_model_name=model_name
fff785c73a814fc89102ed7a486de96d
└── artifacts
└── 8
├── conda.yaml
├── metadata
│ ├── conda.yaml
│ ├── MLmodel
│ ├── python_env.yaml
│ └── requirements.txt
├── MLmodel
├── model.pkl
├── python_env.yaml
└── requirements.txt
Upvotes: 0
Views: 47