CODEWITHSUNDEEP
grafana-loki
Haytham
Haytham

Reputation: 175

Unable to ship logs to Grafana Loki with FluentD & Fluent-bit

I am trying to integrate application logs running on EC2 instances with Grafana Loki using FluentD & Fluent-bit.

I am facing the following:

  1. Fluent-bit service is running normally
  2. The application logs don't shipping
  3. I see this info/warning from monitoring fluent-bit.service:

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/csp-nmgr-1/ocomc_IoT.log, inode 100987803

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/event-nmgr-1/ocomc_IoT.log, inode 100727404

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/session-nmgr-1/ocomc_IoT.log, inode 854909

[debug] [input:tail:tail.3] 0 new files found on path '/opt/SP/users/oracle/ocomc/logs/-nmgr/ocomc_IoT.log'

[debug] [input:tail:tail.3] scanning path /opt/SP/users/oracle/ocomc/logs/-nmgr/ocomc_IoT.log

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/collection-nmgr-1/ocomc_IoT.log, inode 84468627

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/csp-nmgr-1/ocomc_IoT.log, inode 100987803

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/event-nmgr-1/ocomc_IoT.log, inode 100727404

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/session-nmgr-1/ocomc_IoT.log, inode 854909

[debug] [input:tail:tail.3] 0 new files found on path '/opt/SP/users/oracle/ocomc/logs/-nmgr/ocomc_IoT.log'

[debug] [input:tail:tail.3] scanning path /opt/SP/users/oracle/ocomc/logs/-nmgr/ocomc_IoT.log

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/collection-nmgr-1/ocomc_IoT.log, inode 84468627

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/csp-nmgr-1/ocomc_IoT.log, inode 100987803

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/event-nmgr-1/ocomc_IoT.log, inode 100727404

[debug] [input:tail:tail.3] scan_blog add(): dismissed: /opt/SP/users/oracle/ocomc/logs/session-nmgr-1/ocomc_IoT.log, inode 854909

[debug] [input:tail:tail.3] 0 new files found on path '/opt/SP/users/oracle/ocomc/logs/-nmgr/ocomc_IoT.log'

Please advise.

Thanks

Upvotes: 0

Views: 345

Answers (1)

Haytham
Haytham

Reputation: 175

The root cause for that issue was another heavy system that was using the same Grafana configurations and putting a heavy load on the monitoring system, which resulted in blocking other systems from accessing Grafana. Once we stopped that system, everything returned to normal.

I wanted to share this information with you in case someone else faces the same issue. The root cause was not initially apparent in our troubleshooting, but we eventually discovered it.

Thanks

Upvotes: 0

Related Questions