CODEWITHSUNDEEP
securitysplunksplunk-querysplunk-formula
Jon_Snow1
Jon_Snow1

Reputation: 11

How do I check if Splunk has received logs from hundreds of different sources/hosts/devices?

I am relatively new to a company that has used Splunk Professional Services to spin up a Splunk Cloud environment. The company IT has onboarded a lot of AWS, Azure, on-prem and network devices so far. I’m trying to verify that they are in fact sending logs into the Splunk index so that I can eventually apply use cases and alerting on the logs as well as troubleshoot those hosts which aren’t sending but are supposed to be. There isn’t a Splunk resource in the company so I am trying my best to figure it as I go. (classic)

The IT manager gave me a spreadsheet of hostnames and private IP addresses for all the devices which are forwarding logs. At first I thought I could run a search to just compare his list with logs received by hostname but I can’t figure that out. Here’s what I did instead.

Over a 30-day search I run | metadata type=hosts index=* and I exported the results to a csv. I took the ‘hosts’ column (which was a combination of hostnames and IP addresses) from the export and did a diff against the IT managers list of hostnames/IP addresses and where it wasn’t found, presumed it had not sent logs during that time period. The inventory has about ~850 line items in total which are supposedly onboarded and I saw logs from about ~250. Obviously I am second guessing myself because of the delta.

When I spot check some hostnames/IP addresses from the asset inventory spreadsheet from IT in Splunk, there are some that return no results, some that is just DNS or FW traffic from that server (so needs onboarding to get server logs) but others where I get results where the ‘host’ field is a cloud appliance (like Meraki) and the hostname or IP matches to other fields such as ‘dvc_up’, ‘deviceName’ or ‘dvc’ fields. This is really confusing the heck out of me and making me question if there is a better way. So, is there? How do you normally audit and verify that your logs are still being received into your Splunk instance?

Thanks so much for your help and looking forward to learning!

Upvotes: 1

Views: 2321

Answers (1)

warren
warren

Reputation: 33445

Searching for non-existent data is always harder than searching for existing data ... because finding "nothing" isn't possible

However, you can simulate finding nothing like this:

index=ndx sourcetype=srctp ip=* earliest=-7d
| stats count by ip
| append 
    [| inputlookup myListOfIPs
    | fields ip ]
| stats values(count) as filter by ip
| where isnull(filter)

That this search does:

  • do a simple count by all found IPs of the last week
  • append the "master list" of IPs to the stats'd output
  • do a values `stats of all IPs in the table (found in event data and from the lookup table)
  • filter out all items that have some value in the filter field using where

The resultant IPs will all have not been found in the index in question in the last week

Adjust field names, time range, etc as necessary

Upvotes: 0

Related Questions