CODEWITHSUNDEEP
jsonbashcsvjq
Milister
Milister

Reputation: 658

Unable to parse JSON file to CSV with jq-string ("") and array cannot be added

Here is output.json: https://1drv.ms/u/s!AizscpxS0QM4hJo5SnYOHAcjng-jww

i have issues in sts:AsumeRole.Principal.Service part when have multiple Services 

Principal": {
                        "Service": [
                            "ssm.amazonaws.com", 
                            "ec2.amazonaws.com"
                        ]
                    }

in my code below, it's .Principal.Service field. If have only one service, no issues

 "InstanceProfileList": [
            {
                "InstanceProfileId": "AIPAJMMLWIVZ2IXTOC3RO", 
                "Roles": [
                    {
                        "AssumeRolePolicyDocument": {
                            "Version": "2012-10-17", 
                            "Statement": [
                                {
                                    "Action": "sts:AssumeRole", 
                                    "Effect": "Allow", 
                                    "Principal": {
                                        "AWS": "*"
                                    }
                                }
                            ]
                        }, 
                        "RoleId": "AROAJPHJ4EDQG3G5ZQZT2", 
                        "CreateDate": "2017-04-04T23:46:47Z", 
                        "RoleName": "dev-instance-role", 
                        "Path": "/", 
                        "Arn": "arn:aws:iam::279052847476:role/dev-instance-role"
                    }
                ], 
                "CreateDate": "2017-04-04T23:46:47Z", 
                "InstanceProfileName": "bastionServerInstanceProfile", 
                "Path": "/", 
                "Arn": "arn:aws:iam::279052847476:instance-profile/bastionServerInstanceProfile"
            }
        ], 
        "RoleName": "dev-instance-role", 
        "Path": "/", 
        "AttachedManagedPolicies": [
            {
                "PolicyName": "dev-instance-role-policy", 
                "PolicyArn": "arn:aws:iam::279052847476:policy/dev-instance-role-policy"
            }
        ], 
        "RolePolicyList": [], 
        "Arn": "arn:aws:iam::279052847476:role/dev-instance-role"
    }, 
    {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": [
                            "ssm.amazonaws.com", 
                            "ec2.amazonaws.com"
                        ]
                    }
                }
            ]
        },

If only one service exists, no issues, but if more than one then getting error string ("") and array (["ssm.amazonaws.com) cannot be added

How to get all values for Principal.Service in one row.

My code:

jq -rc '.RoleDetailList 
  | map(select((.AssumeRolePolicyDocument.Statement | length > 0) and 
        (.AssumeRolePolicyDocument.Statement[].Principal.Service) or 
        (.AssumeRolePolicyDocument.Statement[].Principal.AWS) or 
        (.AssumeRolePolicyDocument.Statement[].Principal.Federated) or 
        (.AttachedManagedPolicies | length >0) or 
        (.RolePolicyList | length > 0)) )[] 
      | [.RoleName,
         ([.RolePolicyList[].PolicyName,
          ([.AttachedManagedPolicies[].PolicyName] | join("--"))] 
         | join(" ")),
       (.AssumeRolePolicyDocument.Statement[] 
        | .Principal.Federated + "" + .Principal.Service + ""+.Principal.AWS)] 
       | @csv' ./output.json

Desired output:

"dev-instance-role","dev-instance-role-policy","ssm.amazonaws.com--ec2.amazonaws.com--*"

Current output:

"dev-instance-role","dev-instance-role-policy","*"

Upvotes: 2

Views: 340

Answers (2)

peak
peak

Reputation: 116870

It appears that .Principal.Service is either a string or an array of strings, so you need to handle both cases. Consider therefore:

def to_s: if type == "string" then . else join("--") end;

You might want to make this more generic to make it more robust or for other reasons.

You might also want to streamline your jq filter to make it more intelligible and maintainable, e.g. by using jq variables. Note also that

.x.a + .x.b + x.c

can be written as:

.x | (.a + .b + .c)

Upvotes: 1

RomanPerekhrest
RomanPerekhrest

Reputation: 92854

Consider adding additional condition to check whether .Principal.Service is type of either array or string:

jq -rc '.RoleDetailList 
        | map(select((.AssumeRolePolicyDocument.Statement | length > 0) and 
        (.AssumeRolePolicyDocument.Statement[].Principal.Service) or 
        (.AssumeRolePolicyDocument.Statement[].Principal.AWS) or 
        (.AssumeRolePolicyDocument.Statement[].Principal.Federated) or 
        (.AttachedManagedPolicies | length >0) or 
        (.RolePolicyList | length > 0)) )[] 
        | [.RoleName,
           ([.RolePolicyList[].PolicyName,
           ([.AttachedManagedPolicies[].PolicyName] | join("--"))] 
        | join(" ")),
         (.AssumeRolePolicyDocument.Statement[] 
        | .Principal.Federated + "" 
          + (.Principal.Service | if type == "array" then join("--") else . end) 
          + "" + .Principal.AWS)] 
       | @csv' ./output.json

The output:

"ADFS-Administrators","Administrator-Access ","arn:aws:iam::279052847476:saml-provider/companyADFS"
"ADFS-amtest-ro","pol-amtest-ro","arn:aws:iam::279052847476:saml-provider/companyADFS"
"adfs-host-role","pol-amtest-ro","ec2.amazonaws.com"
"aws-elasticbeanstalk-ec2-role","AWSElasticBeanstalkWebTier--AWSElasticBeanstalkMulticontainerDocker--AWSElasticBeanstalkWorkerTier","ec2.amazonaws.com"
"aws-elasticbeanstalk-service-role","AWSElasticBeanstalkEnhancedHealth--AWSElasticBeanstalkService","elasticbeanstalk.amazonaws.com"
"AWSAccCorpAdmin","AdministratorAccess","arn:aws:iam::279052847476:saml-provider/LastPass"
"AWScompanyCorpAdmin","AdministratorAccess","arn:aws:iam::279052847476:saml-provider/LastPass"
"AWScompanyCorpPowerUser","PowerUserAccess","arn:aws:iam::279052847476:saml-provider/LastPass"
"AWSServiceRoleForAutoScaling","AutoScalingServiceRolePolicy","autoscaling.amazonaws.com"
"AWSServiceRoleForElasticBeanstalk","AWSElasticBeanstalkServiceRolePolicy","elasticbeanstalk.amazonaws.com"
"AWSServiceRoleForElasticLoadBalancing","AWSElasticLoadBalancingServiceRolePolicy","elasticloadbalancing.amazonaws.com"
"AWSServiceRoleForOrganizations","AWSOrganizationsServiceTrustPolicy","organizations.amazonaws.com"
"AWSServiceRoleForRDS","AmazonRDSServiceRolePolicy","rds.amazonaws.com"
"Cloudyn","ReadOnlyAccess","arn:aws:iam::432263259397:root"
"DatadogAWSIntegrationRole","DatadogAWSIntegrationPolicy","arn:aws:iam::464622532012:root"
"datadog_alert_metrics_role","AWSLambdaBasicExecutionRole-66abe1f2-cee8-4a90-a026-061b24db1b02","lambda.amazonaws.com"
"dev-instance-role","dev-instance-role-policy","*"
"ec2ssmRole","AmazonEC2RoleforSSM","ssm.amazonaws.com--ec2.amazonaws.com"
"ecsInstanceRole","AmazonEC2ContainerServiceforEC2Role","ec2.amazonaws.com"
"ecsServiceRole","AmazonEC2ContainerServiceRole","ecs.amazonaws.com"
"flowlogsRole","oneClick_flowlogsRole_1495032428381 ","vpc-flow-logs.amazonaws.com"
"companyDevShutdownEC2Instaces","oneClick_lambda_basic_execution_1516271285849 ","lambda.amazonaws.com"
"companySAMLUser","AdministratorAccess","arn:aws:iam::279052847476:saml-provider/companyAzureAD"
"irole-matlabscheduler","pol-marketdata-rw","ec2.amazonaws.com"
"jira_role","","*"
"lambda-ec2-ami-role","lambda-ec2-ami-policy","lambda.amazonaws.com"
"lambda_api_gateway_twilio_processor","AWSLambdaBasicExecutionRole-f47a6b57-b716-4740-b2c6-a02fa6480153--AWSLambdaSNSPublishPolicyExecutionRole-d31a9f16-80e7-47c9-868a-f162396cccf6","lambda.amazonaws.com"
"lambda_stop_rundeck_instance","oneClick_lambda_basic_execution_1519651160794 ","lambda.amazonaws.com"
"OneLoginAdmin","AdministratorAccess","arn:aws:iam::279052847476:saml-provider/OneLoginAdmin"
"OneLoginDev","PowerUserAccess","arn:aws:iam::279052847476:saml-provider/OneLoginDev"
"rds-host-role","","ec2.amazonaws.com"
"rds-monitoring-role","AmazonRDSEnhancedMonitoringRole","monitoring.rds.amazonaws.com"
"role-amtest-ro","pol-amtest-ro","ec2.amazonaws.com"
"role-amtest-rw","pol-amtest-rw","ec2.amazonaws.com"
"Stackdriver","ReadOnlyAccess","arn:aws:iam::314658760392:root"
"vmimport","vmimport ","vmie.amazonaws.com"
"workspaces_DefaultRole","SkyLightServiceAccess ","workspaces.amazonaws.com"

Upvotes: 2

Related Questions