CODEWITHSUNDEEP
apigoogle-cloud-platformautomationservice-accountsgoogle-cloud-dlp
Nathan McKaskle
Nathan McKaskle

Reputation: 3063

GCP Data Loss Prevention API Authentication: Does it require the use of a service account?

I am trying to automate DLP scans using the API. The only thing holding me back from finishing this project is authentication. It appears that creating and using a service account with BigQuery, Storage and DLP admin rights in each and every single project is the only way to avoid permission denied errors despite my own account having organization owner access to all projects. Is this an IAM issue or some requirement of DLP API and the use of tokens? Why does my token not work? We have a lot of projects and may end up with more in the future and would like to avoid the cumbersome and tedious issue of having to create a service account in each project especially having to do it again and again to keep the script working. Automation should be less work not more.

The error I get with my own account token below on a project were DLP API is definitely enabled.

"error": { "code": 403, "message": "Cloud Data Loss Prevention (DLP) API has not been used in project ###### before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=###### then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.", "status": "PERMISSION_DENIED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developers console API activation", "url": "https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=######"

Upvotes: 0

Views: 1588

Answers (4)

kylejmcintyre
kylejmcintyre

Reputation: 2006

If you find this post because you're wanting to call the DLP API via the Java client using an API Key only, please see this gist of how I got it working (in Scala). Basically had to pass the header in myself and explicitly say no other authentication mechanism was in place

https://gist.github.com/kylejmcintyre/b7ff56cf08addb87ebcf53defa18461f

In my case I'm passing all of the data to inspect into the API and therefore am not concerned with any other authentication other than just calling the inspect API.

Upvotes: 0

Jordanna Chord
Jordanna Chord

Reputation: 995

One thing to note that's a little different with DLP. There are actually two service accounts involved in each request. The one you use to call the API, but the service itself uses the dlp service account to turn around and call other services to read from storage.

When you enable dlp you'll see this account automatically created. It looks like this ... service-[PROJECT_NUMBER]@dlp-api.iam.gserviceaccount.com and is discussed here https://cloud.google.com/dlp/docs/iam-permissions#service_account

A common setup I've seen is a security team will enable DLP in a single project to scan their entire organization. They will do this by granting the service-[PROJECT_NUMBER]@dlp-api.iam.gserviceaccount.com in the project they are running from permission to access all other projects in the org. This will let it read GCS/BQ across the company.

Upvotes: 1

marian.vladoi
marian.vladoi

Reputation: 8066

Yes you can use your own user account(owner) with the api.

  1. gcloud auth login 

  2. gcloud auth list 
    #ACTIVE  ACCOUNT with owner role
    *       [email protected] .

  3. gcloud auth print-access-token
    #xxxxxxxxxxxxxx

  4. curl -s -H 'Content-Type: application/json'   -H 'Authorization: Bearer xxxxxxxxxxx'   'https://dlp.googleapis.com/v2/infoTypes'

  #OR

  5. ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
     curl -s -H 'Content-Type: application/json'   -H "Authorization: Bearer $ACCESS_TOKEN"   'https://dlp.googleapis.com/v2/infoTypes'
   #you need Double Quotes and not Single Quotes for -H flag

3.1.2.2 Single Quotes

Enclosing characters in single quotes (') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

3.1.2.3 Double Quotes

Enclosing characters in double quotes (") preserves the literal value of all characters within the quotes, with the exception of $, , \, and, when history expansion is enabled, !. The characters $ and retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

The special parameters * and @ have special meaning when in double quotes (see Shell Parameter Expansion).

Output:


{
  "infoTypes": [
    {
      "name": "AMERICAN_BANKERS_CUSIP_ID",
      "displayName": "American Bankers CUSIP identifier",
      "supportedBy": [
        "INSPECT"
      ],
      "description": "An American Bankers' Committee on Uniform Security Identification Procedures (CUSIP) number is a 9-character alphanumeric code that identifies a North American financial security."
    },
    {
      "name": "AUSTRALIA_DRIVERS_LICENSE_NUMBER",
      "displayName": "Australia driver's license number",
      "supportedBy": [
        "INSPECT"
      ],
      "description": "An Australian driver's license number."
    },
    {
      "name": "AUSTRALIA_MEDICARE_NUMBER",
      "displayName": "Australia medicare number",
      "supportedBy": [
        "INSPECT"

..................

Upvotes: 1

Serhii
Serhii

Reputation: 4461

Have a look at the documentation Authenticating to the Cloud DLP API:

You must authenticate to the Cloud DLP API in order to use it. The Cloud DLP API can handle both API keys and authentication. The main distinction between these two methods is:

  • API keys identify the calling project—the app or site—that is making the call to an API.
  • Auth tokens identify a user—the person—that is using the project.

and

To use a service account to authenticate to the Cloud DLP API:

Follow the instructions to create a service account. Select JSON as your key type, and grant the user the DLP User role (roles/dlp.user).

In general you should follow these steps:

  1. enable billing
  2. enable DLP API

  3. create a service account, grant role to the service account roles/dlp.user and download the key.json file:

    $ gcloud iam service-accounts create test-dlp --description "test-cloud" --display-name "test-dlp"
$ gcloud projects add-iam-policy-binding class-run --member serviceAccount:[email protected] --role roles/dlp.user
$ gcloud iam service-accounts keys create key.json --iam-account [email protected]

  4. set the env variable:

    export GOOGLE_APPLICATION_CREDENTIALS=[PATH_TO_key.json_FILE]

  5. run your script

In addition, have a look at the Quickstart: Using the command-line tool section Permissions:

Inspecting content requires the serviceusage.services.use permission for the project that's specified in parent. The roles/editor, roles/owner, and roles.dlp.user roles contain the required permission or you can define your own custom role.

If you still have an issue, try to troubleshoot bu following these steps:

  1. check your active account with command gcloud auth list
  2. check if DLP API is enabled gcloud services list --enabled | grep DLP
  3. activate your DLP service account with command gcloud auth activate-service-account and run your script again

and update your question with commands and outputs. Also, please clarify in which way do you use your DLP service account in your the script.

Upvotes: 1

Related Questions