CODEWITHSUNDEEP
azure-log-analyticsazure-monitor
ZZZSharePoint
ZZZSharePoint

Reputation: 1351

Log Analytics retention policy and querying on logs

I would like to know how can we address this scenario in Azure Log Analytics where I need to generate Kube-audit logs of different cluster every week and also retain these logs for approx 400 days. Now storing it over Log Analytics will cost me more and its not an optimized architecture as I will not be require that so often. So I would like to know from experts whats the best way to design the architecture, where we get the kube audit logs which can be retained for 400 days and be available for querying when required without incurring too much cost. PS: I also heard in my team that querying 400 days logs always times out in KQL.

Upvotes: 0

Views: 1190

Answers (1)

David Likvornik
David Likvornik

Reputation: 51

Log analytics offerings: Log analytics now provides the capability to manage several service tiers at table scope. Setting your data as archive, with no query capabilities at a much lower cost. offering spans for up to 7 years.

when needed, you can choose to elevate a subset of your data into the Analytics offering, providing you the capability to query it. The action of elevating your data is denoted as - "Search jobs"

Another option is to elevate an entire period in time to the Analytic offering, they call it - "Restore logs".

Table's different service tiers - https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-retention-archive?tabs=api-1%2Capi-2

Search job offering - https://learn.microsoft.com/en-us/azure/azure-monitor/logs/search-jobs?tabs=api-1%2Capi-2%2Capi-3

Restore logs - https://learn.microsoft.com/en-us/azure/azure-monitor/logs/restore?tabs=api-1%2Capi-2

all are under public preview.

both offerings - Search jobs and Restore logs provides you the capability to engage your data on demand, can't comment or suggest regarding the actual cost.

Azure data explorer solution: Another option is to use Azure storage to hold your data (as an example), Azure data explorer provides the capability to create an external table, that table is a logical view on top of your data, the data itself is kept outside of the ADX cluster. you can query your data by using ADX, expect degradation in query performance.

ADX external table offering - https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/schema-entities/externaltables

Upvotes: 1

Related Questions