CODEWITHSUNDEEP
amazon-web-services.net-coreamazon-cloudwatchlogsaws-cloudwatch-log-insights
Flicco
Flicco

Reputation: 101

AWS with .NET - reading logs from CloudWatch - no log data returned

I am attempting to read Log data using .NET from CloudWatch for Messages sent using SNS.

From the CloudWatch console (CloudWatch \ CloudWatch Logs \ Logs Insights) I enter:

Date range: custom (2w)
LogGroup: sns/ap...../8...../LogName 
Query: fields @timestamp, @message | sort @timestamp desc | limit 20

It returns lots of log records (@timestamp | @message)

I'm trying to do the same from c# using the .net AWS SDK:

public async Task GetLogs()
{
    string logGroupName = "sns/ap...../8...../LogName";

    AWSOptions options = configuration.GetAWSOptions();
    IAmazonCloudWatchLogs logs = options.CreateServiceClient<IAmazonCloudWatchLogs>();

    StartQueryRequest startQueryRequest = new StartQueryRequest();
    startQueryRequest.LogGroupName = logGroupName;
    startQueryRequest.StartTime = 1577850562; //1 Jan 2020
    TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); //Epoch time starts on 1/1/1970
    int secondsSinceEpoch = (int)t.TotalSeconds;
    startQueryRequest.EndTime = secondsSinceEpoch;
    startQueryRequest.QueryString = "fields @timestamp, @message | sort @timestamp desc";
    startQueryRequest.Limit = 1000;
    StartQueryResponse response2 = await logs.StartQueryAsync(startQueryRequest);

    Console.WriteLine();
}

Can't work out why it doesn't return any records.

Response2 = 
   ContentLength: 50
   QueryId: "guid..."
   ResponseMetadata:
      Metadata Count = 0
      RequestId = "guid..."

Any idea what I'm doing wrong? Thanks!

Upvotes: 3

Views: 3304

Answers (3)

Sparafusile
Sparafusile

Reputation: 4956

I have a working example without typos and that will retry if the query takes some time to finish:

            const string LogGroupName = "<your log group name here>";
            const string QueryString = "<your query string here>";

            var epoch = new DateTime( 1970, 1, 1 );
            var s = DateTime.UtcNow.AddMinutes( -15 ) - epoch;
            var e = DateTime.UtcNow - epoch;

            using( var client = new AmazonCloudWatchLogsClient() )
            {
                var startQueryRequest = new StartQueryRequest
                {
                    LogGroupName = LogGroupName,
                    StartTime = ( long ) s.TotalSeconds,
                    EndTime = ( long ) e.TotalSeconds,
                    QueryString = QueryString,
                    Limit = 1000
                };

                var startQueryResponse = await client.StartQueryAsync( startQueryRequest );
                var resultRequest = new GetQueryResultsRequest { QueryId = startQueryResponse.QueryId };
                var result = await client.GetQueryResultsAsync( resultRequest );

                // Retry loop with a timeout (AWS states that queries can run up to 15 minutes)
                for( var i = 0 ; i < 60 && ( result.Status == QueryStatus.Running || result.Status == QueryStatus.Scheduled ) ; i++ )
                {
                    System.Threading.Thread.Sleep( 250 );
                    result = await client.GetQueryResultsAsync( resultRequest );
                }

                // Query failed or timed out
                if( result.Status != QueryStatus.Complete )
                {
                    throw new Exception( "Could not complete query." );
                }

                foreach( var item in result.Results )
                {
                    // Use the results here
                }
            }

Upvotes: 1

Flicco
Flicco

Reputation: 101

In case it helps anyone else - here is my method to get the logs from AWS using C#

public async Task GetLogs()
{
    string logGroupName = "log group name from CloudWatch Log";

    AWSOptions options = configuration.GetAWSOptions();
    IAmazonCloudWatchLogs logs = options.CreateServiceClient<IAmazonCloudWatchLogs>();

    StartQueryRequest startQueryRequest = new StartQueryRequest();
    startQueryRequest.LogGroupName = logGroupName;
    startQueryRequest.StartTime = 1577850562; //1 Jan 2020
    TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); //Epoch time starts on 1/1/1970
    int secondsSinceEpoch = (int)t.TotalSeconds;
    startQueryRequest.EndTime = secondsSinceEpoch;
    startQueryRequest.QueryString = "fields @timestamp, @message | sort @timestamp desc";
    startQueryRequest.Limit = 1000;
    StartQueryResponse startQueryResponse = await logs.StartQueryAsync(startQueryRequest);

    GetQueryResultsRequest getQueryRequest = new GetQueryResultsRequest();
    getQueryRequest.QueryId = startQueryResponse.QueryId;

    GetQueryResultsResponse getQueryResults = await logs.GetQueryResultsAsync(getQueryRequest);
    for (var i=0; i<getQueryResults.Results.Count; i++)
    {
        ResultField timestampResult = getQueryResults.Results[i][0];
        ResultField messageResult = getQueryResults.Results[i][1];
        var message = JsonConvert.DeserializeObject(messageResult.Value);
        Console.WriteLine(message);
    }           
    Console.WriteLine();
}

Upvotes: 6

Dejan Peretin
Dejan Peretin

Reputation: 12089

You are only starting the query, you need to call GetQueryResults to get the data: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetQueryResults.html

Upvotes: 2

Related Questions