CODEWITHSUNDEEP
logginggrafanaprometheus
JosMac
JosMac

Reputation: 2332

Monitoring log files using some metrics exporter + Prometheus + Grafana

I need to monitor very different log files for errors, success status etc. And I need to grab corresponding metrics using Prometheus and show in Grafana + set some alerting on it. Prometheus + Grafana are OK I already use them a lot with different exporters like node_exporter or mysql_exporter etc. Also alerting in new Grafana 4.x works very well.

But I have quite a problem to find suitable exporter/ program which could analyze log files "on fly" and extract metrics from them.

So far I tried:

Does any one here has a really running solution for monitoring advanced metrics from log files using "some exporter" + Prometheus + Grafana? Or instead of exporter some program from which I could grab results using Prometheus push gateway. Thanks.

Upvotes: 38

Views: 41679

Answers (5)

Bahubali
Bahubali

Reputation: 1

I installed airflow 2.9.3 and wanted to install exporter where I can collect metrics on promethues, but i have error on the page saying sql row_number() has issue for mysql/mariadb i am using(defsult if sqlite which is not suggested for production)

solution:~/airflow_venv/lib/python3.11/site-packages/airflow_exporter/prometheus_exporter.py"

change the block below to: def get_last_dagrun_info() -> List[DagStatusInfo]: '''get last_dagrun info :return last_dagrun_info ''' assert(Session is not None)

# Subquery to find the latest execution_date for each dag_id
latest_dag_run_subquery = (
    Session.query(
        DagRun.dag_id,
        func.max(DagRun.execution_date).label('latest_execution_date')
    )
    .group_by(DagRun.dag_id)
    .subquery()
)

# Main query to get the last DagRun info using the subquery
sql_res = (
    Session.query(
        DagRun.dag_id,
        DagRun.state,
        DagModel.owners
    )
    .join(latest_dag_run_subquery, 
          (DagRun.dag_id == latest_dag_run_subquery.c.dag_id) & 
          (DagRun.execution_date == latest_dag_run_subquery.c.latest_execution_date))
    .join(DagModel, DagModel.dag_id == DagRun.dag_id)
    .join(SerializedDagModel, SerializedDagModel.dag_id == DagRun.dag_id)
    .all()
)

# Constructing the response with the same format
res = [
    DagStatusInfo(
        dag_id=i.dag_id,
        status=i.state,
        cnt=1,  # The count is always 1 since we are fetching the latest run for each dag_id
        owner=i.owners
    )
    for i in sql_res
]

return res

This gave me my /admin/metrics page and Grafana to scrape metrics as well on 8125 instead of statd on 9102 which is another work around instead of UI given metrics usage. now Iam trying to find a way instead of updating packages might overwrite my metrics exporter or not.

Upvotes: -1

J. Gwinner
J. Gwinner

Reputation: 1577

Slightly newer answer:

I went looking for the same thing, and found "Loki" which is the Grafana log aggregator, and "Promtail" which is what collects the log files and pushes to Grafana (Loki). In effect, Loki is like Prometheus for log files.

Upvotes: 4

yno
yno

Reputation: 61

Try prometheus-python-exporter and write your customized exporter in python grepping whatever you want in your log files, then expose wanted metrics. There are several tutos to help yoy

Upvotes: 2

LazyS
LazyS

Reputation: 356

Take a look at Telegraf. It does support tailing logs using input plugins logparser and tail. To export metrics as prometheus endpoint use prometheus_client output plugin. You also may apply on the fly aggregations. I've found it simpler to configure for multiple log files than grok_exporter or mtail

Upvotes: 19

brian-brazil
brian-brazil

Reputation: 34172

Those are the 3 answers currently for getting log data into Prometheus.

You could also look into getting whatever is producing the logs to expose Prometheus metrics directly.

Upvotes: 2

Related Questions