Dataplex API Tag Policies

I'm on exploration Dataplex API with Python in Google Documentation, there is documentation to Get Lake, Zone, Assets, etc. I've explored that documentation, but I didn't find any documentation related to Tag Policies, for example, I need to attach my Tag Template and add Policy Tag to my BigQuery Table via API.

Is it possible to attach Tag Template and add Policy Tag into BigQuery Table via API?

Here is the link that I've explored:

Dataplex API Service

Dataplex API Metadata Service

Data Catalog API

Upvotes: 0

Answers (2)

Bihag Kashikar

Reputation: 1326

again.

To simulate the pythonic api's behaviour, I used google cloud api explorer to explain in detail. see below.

The entry lookup is to search for object(s) you want to attach a tag/tag templates

Basically here's how I simulated the api calls using api explorer

To attach a tag to a BigQuery table, first step is to search the table up using Datacatalog api url below

LookupEntryRequest

The parameters I passed to get below response is sqlResource: "bigquery.table.myproject.zz_DataSet.tblOne"

Above should give you output as

{
  "name": "projects/myproject/locations/australia-southeast2/entryGroups/@bigquery/entries/mykey",
  "type": "TABLE",
  "schema": {
    "columns": [
      {
        "type": "STRING",
        "mode": "NULLABLE",
        "column": "firstname"
      },
      {
        "type": "STRING",
        "mode": "NULLABLE",
        "column": "lastname"
      }
    ]
  },
  "sourceSystemTimestamps": {
    "createTime": "2023-01-16T04:22:49.397Z",
    "updateTime": "2023-01-16T04:22:49.397Z"
  },
  "linkedResource": "//bigquery.googleapis.com/projects/myproject/datasets/zz_DataSet/tables/tblOne",
  "bigqueryTableSpec": {
    "tableSourceType": "BIGQUERY_TABLE"
  },
  "usageSignal": {
    "updateTime": "2023-02-05T07:59:59.928Z",
    "usageWithinTimeRange": {
      "30D": {
        "totalCompletions": 7,
        "totalFailures": 1,
        "totalExecutionTimeForCompletionsMillis": 7385
      }
    }
  },
  "integratedSystem": "BIGQUERY",
  "fullyQualifiedName": "bigquery:myproject.zz_DataSet.tblOne"
}

The search gives you ability to query multiple tables or attach tags at Dataset level too, see parameters section on link above.

This is why I suggest you use entry look up first as its more scalable code.

API Call two: This is how i simulated the attach tag to resource. If you go to link below

CreateTagRequest

As an example: I pre created an tag template from console and then used the template-id value to pass as a parameter to the request

Input:

parent: projects/myproject/locations/australia-southeast2/entryGroups/@bigquery/entries/mykey from above name element request body:

{
  "template": "projects/myproject/locations/australia-southeast1/tagTemplates/api_call_test_tag_template",
  "fields": {
    "name": {
      "stringValue": "apitestcall"
    }
  }
}

Output: Below is the response generated and if you see in data catalog console, you will see bigquery table with the tag template attached to it with value to name field as "apitestcall" attached to it. see image attached

{
  "name": "projects/myproject/locations/australia-southeast2/entryGroups/@bigquery/entries/mykey/tags/tagsKey",
  "template": "projects/myproject/locations/australia-southeast1/tagTemplates/api_call_test_tag_template",
  "fields": {
    "name": {
      "displayName": "name",
      "stringValue": "apitestcall"
    }
  },
  "templateDisplayName": "api-call-test-tag-template"
}

Finally, please do make sure you have all the correct IAM permissions required for this task.

Upvotes: 0

Bihag Kashikar

Reputation: 1326

for attaching tag templates to BigQuery table, first you will have to lookup the entry in dataplex using api

https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.LookupEntryRequest

and then attach is to table using api

https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.CreateTagRequest

here's sample code, this creates tag template and also attaches it to table in same code base

https://cloud.google.com/data-catalog/docs/samples/data-catalog-quickstart

and to attach policy, use this api

https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.CreatePolicyTagRequest