CODEWITHSUNDEEP
command-lineamazon-web-servicesfilter
A. Duff
A. Duff

Reputation: 4309

Finding all Amazon AWS Instances That Do Not Have a Certain Tag

I'm trying to use the Amazon AWS Command Line Tools to find all instances that do not have a specified tag.

Finding all instances WITH a tag is simple enough, e.g.

ec2-describe-instances --filter "tag-key=Name"

But how would I invert that filter to return only the instances that have no tag "Name"?

Upvotes: 41

Views: 43136

Answers (11)

Julien4218
Julien4218

Reputation: 11

Replace Owner with the tag name for your needs. This find all the InstanceId in your default region not having that tag defined.

aws ec2 describe-instances \
  --output text \
  --filters Name=instance-state-name,Values=running \
  --query 'Reservations[].Instances[?!not_null(Tags[?Key == `Owner`].Value)] | [].[InstanceId]'

Upvotes: 1

atye52g1
atye52g1

Reputation: 1

I put this together from reviewing some other posts as well. This will get a list of instances that do not have a group of tag keys:

import boto3

def lambda_handler(event, context):

    client = boto3.client('ec2', region_name='us-east-1')

    paginator = client.get_paginator('describe_instances')

    reservations = paginator.paginate()



    filtered_ec2s = reservations.search (

        'Reservations[].Instances[] | [?!not_null(Tags[?Key == `{}`].Value, `{}`)] ||     [?!not_null(Tags[?Key == `{}`].Value, `{}`)] || [?!not_null(Tags[?Key == `{}`].Value, `{}`)] | []'.format(

        'tagkey1', 'tagvalue1',

        'tagkey2', 'tagvalue2',

        'tagkey3', 'tagvalue3')

    )



    for ec2 in filtered_ec2s:

        print(ec2["InstanceId"])

        print(ec2["LaunchTime"])

        print(ec2["State"]["Name"])

I am trying to figure out how to also check for the tag value as well since right now it's ignoring these and returning instances that do not all 3 tag values regardless of the value.

Upvotes: 0

Jordon Phillips
Jordon Phillips

Reputation: 16003

Unfortunately the underlying api call DescribeInstances does not support inverse tag filtering, and so neither does the CLI. You can, however, do client side filtering with the --query parameter which performs a JMESPath search. This will prevent you from having to use pipes as with user2616321's answer.

For example:

aws ec2 describe-instances --query "Reservations[].Instances[?Tags[?Key == 'Name']][]"

Add .InstanceId to the end of that to just get the instance ids.

Upvotes: 2

Chad Geary
Chad Geary

Reputation: 567

I use this python3 / boto script for very large inverse tag filtering operations:

import boto3
from botocore.config import Config

# Attempts
config = Config(
  retries = dict(
    max_attempts = 3
  )
)

# Tag(s)
my_tags = [
  {
    "Key": "backup",
    "Value": "true"
  }
]

# Owner ID Filter
owner_id = 'SOME_OWNER_ID'

# Connection
ec2 = boto3.client("ec2", config=config)

# Instances
def tag_instances():
  # All Reservations [instances] (tagged or untagged)
  all_reservations = ec2.describe_instances(Filters = [{'Name': 'owner-id', 'Values':[owner_id]}])

  # Append each InstanceId in all_reservations to all_instances
  all_instances = []
  for all_reservation in all_reservations['Reservations']:
    for all_instance in all_reservation['Instances']:
      all_instances.append(all_instance['InstanceId'])

  # Append each InstanceId with backup:true or backup:false to tagged_instances
  tagged_reservations = ec2.describe_instances(Filters = [{'Name': 'owner-id', 'Values':[owner_id]},{'Name': 'tag:backup', 'Values':['true','false']}])
  tagged_instances = []
  for tagged_reservation in tagged_reservations['Reservations']:
    for tagged_instance in tagged_reservation['Instances']:
      tagged_instances.append(tagged_instance['InstanceId'])

  # Append each InstanceId in all_instances and not in tagged_instances to untagged_instances
  untagged_instances = [all_instance for all_instance in all_instances if all_instance not in tagged_instances]

  # Print untagged InstanceId
  print("untagged_instanceids:",untagged_instances)

Upvotes: 0

Nate Fox
Nate Fox

Reputation: 2513

You can do that with jmespath (the engine that drives the --query parameter) despite what others say:

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)] | []'

Source: Using Amazon Web Services Command Line Interface (AWS CLI) to Find Instances without a 'Name' Tag.

Upvotes: 36

Sergei
Sergei

Reputation: 2130

Since --filters parameter doesn't seem to support inverse filtering, here's my solution to this problem using --query parameter:

aws ec2 describe-instances \
--query 'Reservations[].Instances[?!contains(Tags[].Key, `Name`)][].InstanceId'

It looks at an array of tag keys for each instance and filters those instance that don't have Tag 'Name' in the array. Then flattens output to array of instance IDs.

  • Advantage over some previous answers: no need for jq or other command to filter output.
  • Disadvantage over true inverse filter: likely to be much slower over large number of instances.

Upvotes: 8

Ramfjord
Ramfjord

Reputation: 939

I too was totally shocked by how difficult this is to do via the CLI. I liked user2616321's answer, but I was having a little trouble making it output the exact fields I wanted per instance. After spending a while messing around and failing with JMESPath in the query syntax, I ended up just making a little ruby script to do this. In case anyone wants to save a few minutes writing one of their own, here it is:

#!/usr/bin/env ruby
require 'json'

# We'll output any instance that doesn't contain all of these tags
desired_tags = if ARGV.empty?
                 %w(Name)
               else
                 ARGV
               end

# Put the keys we want to output per instance/reservation here
reservation_keys = %w(OwnerId RequesterId)
instance_keys = %w(Tags InstanceId InstanceType PublicDnsName LaunchTime PrivateIpAddress KeyName) 
instances_without_tags = []

# Just use CLI here to avoid AWS dependencies
reservations = JSON.parse(
  `aws ec2 describe-instances`
)["Reservations"]

# A reservation is a single call to spin up instances. You could potentially
# have more than one instance in a reservation, but often only one is
# spun up at a time, meaning there is a single instance per reservation.
reservations.each do |reservation|
  reservation["Instances"].each do |instance|
    # Filter instances without the desired tags
    tag_keys = instance["Tags"].map { |t| t["Key"] }
    unless (tag_keys & desired_tags).length == desired_tags.length
      instances_without_tags << 
        reservation.select { |k| reservation_keys.include?(k) }.
          merge(instance.select { |k| instance_keys.include?(k) })
    end
  end
end

puts JSON.pretty_generate(instances_without_tags)

Upvotes: 0

DataGuru
DataGuru

Reputation: 827

I was having the same problem and I figured out how to query on Tag-Values you will most likely have the same tag-key defined for all the instances; I have defined a tag-key "MachineName" on all my instances and I want to filter by the the values of the Tag-key Name

Below is the example to filter where the Name=Machine1

use the option

--filters "Name=tag-key,Values=MachineName" "Name=tag-values,Values=Machine1"

This works fine for me

Upvotes: 1

Jason2616321
Jason2616321

Reputation: 395

This will do what you're asking - find every instance which doesn't contain a tag named "YOUR_KEY_NAME_HERE" (2nd line filters for instances without tags named "Name"):

aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "YOUR_KEY_NAME_HERE"} ]}) | not)' 
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "Name"} ]}) | not)'

If you wanted to filter against the value of the tag, instead of the name of the tag, this query lists all instances which don't contain a tag named YOUR_KEY_NAME_HERE whose value is EXCLUDE_ME. (2nd line lists instances which aren't named "testbox1".)

aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "YOUR_KEY_NAME_HERE"}, {Value: "EXCLUDE_ME"}]}) | not)'
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "Name"}, {Value: "testbox1"}]}) | not)'

Felipe is correct. Parsing the output is the only way to go, since the AWS API does not provide this feature, nor do either of the official AWS CLIs. JSON output is very parseable, especially when compared to the multi-line text records which the old CLI prints by default.

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeInstances.html

The API itself returns JSON, and the new awscli prints that JSON as its default output format. The "jq" program is very useful to parse it, and will even colorize when sent to a terminal, or you can --output text to reduce it back to strings.

Upvotes: 26

Stewie
Stewie

Reputation: 3121

You could always do this: ec2-describe-instances | grep -v "Name" :p

Upvotes: -5

Felipe Garcia
Felipe Garcia

Reputation: 2303

AFAIK directly through the CLI you won't be able to do that.

By the syntax you are using, I can guess you are using the old cli. I suggest you to download the new CLI http://aws.amazon.com/cli/ and call

aws ec2 describe-instances --output json

from python, ruby or any scripting language you may like to parse the json output filtering using the proper regular expression according to your needs

Upvotes: 0

Related Questions