CODEWITHSUNDEEP
flaskgoogle-apistdoutlong-running-processeslong-running-task
L1ghtsp33D
L1ghtsp33D

Reputation: 119

How to use data streamed to stdout after request to Google API using python SDK and flask

I am using Google's vertex python API and am having trouble understanding how to make use of the output for "LRO" their Long Running Operations. The case I'm stuck on is calling

aiplatform.ImageDataset.create(

Which from what I've found offers no good way to retrieve the operation name through requests after calling it, but the name is streamed to my Flask console immediately after the request closes, because the "create" method is async but even If I call it sync it still gets streamed to stdout in this way. The streamed output looks like this

INFO:google.cloud.aiplatform.datasets.dataset:Creating ImageDataset
INFO:google.cloud.aiplatform.datasets.dataset:Create ImageDataset backing LRO: projects/888888888/locations/us-central1/datasets/88888888/operations/88888888
DEBUG:google.api_core.retry:Retrying due to , sleeping 0.6s ...
DEBUG:google.api_core.retry:Retrying due to , sleeping 1.3s ...
INFO:google.cloud.aiplatform.datasets.dataset:ImageDataset created. Resource name: projects/888888888/locations/us-central1/datasets/888888888
INFO:google.cloud.aiplatform.datasets.dataset:To use this ImageDataset in another session:
INFO:google.cloud.aiplatform.datasets.dataset:ds = aiplatform.ImageDataset('projects/8888888/locations/us-central1/datasets/888888888')

It seems that I am forced to use this streamed output somehow to obtain the LRO operation, I've been looking for weeks for an alternative and havn't found one, but I don't know how to use it and I've found suggestions to use subprocess that I don't know how to spawn and seem difficult to ensure their closure. I am wondering if there is a standard approach to using this type of streamed data, why they would choose to return responses in this manner, what the underlying mechanism is, and if I am on the wrong track trying to use it at all.

Upvotes: 0

Views: 44

Answers (1)

L1ghtsp33D
L1ghtsp33D

Reputation: 119

I found the mechanism that was "streaming" to output is a logging mechanism on the long running operation in the base class, class _Dataset(base.VertexAiResourceNounWithFutureManager): which is extended twice to create the ImageDataset class I'm using, the extra extending was what gave me difficulty tracing back the behavior sooner. The logging calls on the lro look like this

_LOGGER.log_create_with_lro(cls, create_dataset_lro)

created_dataset = create_dataset_lro.result(timeout=None)

_LOGGER.log_create_complete(cls, created_dataset, "ds")

so my solution was to create an alternate class that is the same as ImageDataset but with methods that return create_dataset_lro so that I can access the operation.name property

Upvotes: 0

Related Questions