CODEWITHSUNDEEP
azure-log-analyticsazure-sentinel
Rakim
Rakim

Reputation: 177

TimeGenerated field not taking provided date

I am writting custom logs to log analytics.

Based on the following link:

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api#create-a-request

I should be able to use a field in the request header called: time-generated-field, the documentation says that: " If you specify a field, its contents are used for TimeGenerated. If you don't specify this field, the default for TimeGenerated is the time that the message is ingested. The contents of the message field should follow the ISO 8601 format YYYY-MM-DDThh:mm:ssZ"

I am passing the following value: 2021-11-11T19:52:45Z (as a string, since you can't pass this as a datetime object) but the problem is that when I look in the log analytics workspace, the TimeGenerated field is this (today's date): 2021-12-01T18:41:04.529Z which is the datetime the event is ingested, so basically, it's not taking the real event generated time which is 2021-11-11T19:52:45Z which I am passing in the header.

Am I doing something wrong here?

Any help would be appreciated, I am running out of ideas here.

Upvotes: 1

Views: 1572

Answers (2)

AndrePKI
AndrePKI

Reputation: 41

As stated [here][1], the TimeGenerated value cannot be older than 2 days before received time or more than a day in the future. If in some situation, the value is older than 2 days or more than a day in the future, it would be replaced with the actual received time.

So that is the reason you don't see the date/time you put in, in the TimeGenerated column. I have been told by MS that there will be a feature (planned for Q1 2024) to have an extra column with the actual date/time that is coming in. Until then you could create an ingestion time DCR to

source|extend TimeGenerated_CF = TimeGenerated

which has the same effect. [1]: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-standard-columns#timegenerated

Upvotes: 0

Rakim
Rakim

Reputation: 177

It turns out I misread the documentation and didn't realize that the time-generated-field is not an argument you pass in, but a property in the body of the JSON data that you are sending to the log analytics workspace.

It can be referenced in this manner (this is an example of 1 way of doing it):

def post_data(customer_id, shared_key, body, log_type):
    method = 'POST'
    #The string "raised" assigned to the TimeStampField variable below 
    #is an actual property in the JSON object that I am sending to log 
    #analytics, it contains the datetime in the expected format.
    TimeStampField = "raised" 
    content_type = 'application/json'
    resource = '/api/logs'
    rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    content_length = len(body)
    signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
    uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'
    headers = {
        'content-type': content_type,
        'Authorization': signature,
        'Log-Type': log_type,
        'x-ms-date': rfc1123date,
        'time-generated-field': TimeStampField
    }

Will definitely create a pull request to modify the wording here: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api#create-a-request As I feel it is not very intuitive

Upvotes: 2

Related Questions