CODEWITHSUNDEEP
mailgun
Tom K.
Tom K.

Reputation: 49

Exporting Logs from Mailgun for LTS

Is there a way via the API to export Mailgun's logs to a local file for long term storage? We need to keep our mailing logs for over the 30 days Mailgun provides for.

Thanks!

Upvotes: 4

Views: 8235

Answers (4)

wolfplusplus
wolfplusplus

Reputation: 21

The original answer doesn't work without modifications. Here is the updated code that works:

#!/usr/bin/env python3
# Uses the Mailgun API to save logs to JSON file
# Set environment variables MAILGUN_API_KEY and MAILGUN_SERVER
# Optionally set MAILGUN_LOG_DAYS to number of days to retrieve logs for
# Based on https://stackoverflow.com/a/49825979
# See API guide https://documentation.mailgun.com/en/latest/api-intro.html#introduction
import os
import json
import requests
from datetime import datetime, timedelta
from email import utils

DAYS_TO_GET = os.environ.get("MAILGUN_LOG_DAYS", 7)
MAILGUN_API_KEY = os.environ.get("MAILGUN_API_KEY")
MAILGUN_SERVER = os.environ.get("MAILGUN_SERVER")
if not MAILGUN_API_KEY or not MAILGUN_SERVER:
    print("Set environment variable MAILGUN_API_KEY and MAILGUN_SERVER")
    exit(1)

ITEMS_PER_PAGE = 300  # API is limited to 300

def get_logs(start_date, next_url=None):
    if next_url:
        print(f"Getting next batch of {ITEMS_PER_PAGE} from {next_url}...")
        response = requests.get(next_url,auth=("api", MAILGUN_API_KEY))
    else:
        url = 'https://api.mailgun.net/v3/{0}/events'.format(MAILGUN_SERVER)
        start_date_formatted = utils.format_datetime(start_date)  # Mailgun wants it in RFC 2822
        print(f"Getting first batch of {ITEMS_PER_PAGE} from {url} since {start_date_formatted}...")
        response = requests.get(
            url,
            auth=("api", MAILGUN_API_KEY),
            params={"begin"       : start_date_formatted,
                    "ascending"   : "yes",
                    "pretty"      : "yes",
                    "limit"       : ITEMS_PER_PAGE,
                    "event"       : "accepted",}
        )
    response.raise_for_status()
    return response.json()


start = datetime.now() - timedelta(DAYS_TO_GET)
log_items = []
current_page = get_logs(start)

while current_page.get('items'):
    items = current_page.get('items')
    log_items.extend(items)
    print(f"Retrieved {len(items)} records for a total of {len(log_items)}")
    next_url = current_page.get('paging').get('next', None)
    current_page = get_logs(start, next_url=next_url)

file_out = f"mailgun-logs-{MAILGUN_SERVER}_{start.strftime('%Y-%m-%d')}_to_{datetime.now().strftime('%Y-%m-%d')}.json"
print(f"Writing out {file_out}")
with open(file_out, 'w') as file_out_handle:
    json.dump(log_items, file_out_handle, indent=4)

print("Done.")

Upvotes: 2

pierot
pierot

Reputation: 326

You can have a look at MailgunLogger. It's an open source project that can easily be deployed via Docker to fetch and store Mailgun events in a database. It features a dead simple, although rudimentary, search and allows you to add multiple accounts/domains.

Run via Docker:

docker run -d -p 5050:5050 \
  -e "ML_DB_USER=username" \
  -e "ML_DB_PASSWORD=password" \
  -e "ML_DB_NAME=mailgun_logger" \
  -e "ML_DB_HOST=my_db_host" \
  --name mailgun_logger jackjoe/mailgun_logger

From there on, the interface guides you to configure everything.

In the OP case, this project can be used in a more headless fashion where you only use the database instead of the provided UI.

Upvotes: 0

superdianix
superdianix

Reputation: 98

There's a simple python script to retrieve logs for a domain, however i haven't checked if it hits the events api instead of the now deprecated logs api...

https://github.com/getupcloud/python-mailgunlog

Upvotes: 2

evan_schmevan
evan_schmevan

Reputation: 1691

You can only request 300 events at a time, so you'll have to continue fetching the next page until you run out of results. You can then do whatever you'd like with the log items, such as generate a csv, or add items in your database. Check out https://documentation.mailgun.com/en/latest/api-events.html#events for the API docs. Here's an example in Python:

import requests
import csv
from datetime import datetime, timedelta

DATETIME_FORMAT = '%d %B %Y %H:%M:%S -0000'

def get_logs(start_date, end_date, next_url=None):
    if next_url:
        logs = requests.get(next_url,auth=("api", [YOUR MAILGUN ACCESS KEY]))
    else:
        logs = requests.get(
            'https://api.mailgun.net/v3/{0}/events'.format(
                [YOUR MAILGUN SERVER NAME]
            ),
            auth=("api", [YOUR MAILGUN ACCESS KEY]),
            params={"begin"       : start_date.strftime(DATETIME_FORMAT),
                    "end"         : end_date.strftime(DATETIME_FORMAT),
                    "ascending"   : "yes",
                    "pretty"      : "yes",
                    "limit"       : 300,
                    "event"       : "accepted",}
        )
    return logs.json()


start = datetime.now() - timedelta(2)
end = timezone.now() - timedelta(1)
log_items = []
current_page = get_logs(start, end)

while current_page.get('items'):
    items = current_page.get('items')
    log_items.extend(items)
    next_url = current_page.get('paging').get('next', None)
    current_page = get_logs(start, end, next_url=next_url)

keys = log_items[0].keys()
with open('mailgun{0}.csv'.format(start.strftime('%Y-%M-%d')), 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(log_items)

Upvotes: 5

Related Questions