CODEWITHSUNDEEP
pythonloggingsmtphandlerpython-logging
Jonathan Livni
Jonathan Livni

Reputation: 107082

How to make SMTPHandler not block

I installed a local SMTP server and used logging.handlers.SMTPHandler to log an exception using this code:

import logging
import logging.handlers
import time
gm = logging.handlers.SMTPHandler(("localhost", 25), '[email protected]', ['[email protected]'], 'Hello Exception!',)
gm.setLevel(logging.ERROR)
logger.addHandler(gm)
t0 = time.clock()
try:
    1/0
except:
    logger.exception('testest')
print time.clock()-t0

It took more than 1sec to complete, blocking the python script for this whole time. How come? How can I make it not block the script?

Upvotes: 9

Views: 4093

Answers (7)

Marco Graziano
Marco Graziano

Reputation: 476

Here's the implementation I'm using, which I based on Jonathan Livni code.

import logging.handlers
import smtplib
from threading import Thread

# File with my configuration
import credentials as cr

host = cr.set_logSMTP["host"]
port = cr.set_logSMTP["port"]
user = cr.set_logSMTP["user"]
pwd = cr.set_logSMTP["pwd"]
to = cr.set_logSMTP["to"]



def smtp_at_your_own_leasure(
    mailhost, port, username, password, fromaddr, toaddrs, msg
):
    smtp = smtplib.SMTP(mailhost, port)
    if username:
        smtp.ehlo()  # for tls add this line
        smtp.starttls()  # for tls add this line
        smtp.ehlo()  # for tls add this line
        smtp.login(username, password)
    smtp.sendmail(fromaddr, toaddrs, msg)
    smtp.quit()


class ThreadedTlsSMTPHandler(logging.handlers.SMTPHandler):
    def emit(self, record):
        try:
            # import string  # <<<CHANGE THIS>>>

            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                self.fromaddr,
                ",".join(self.toaddrs),  # <<<CHANGE THIS>>>
                self.getSubject(record),
                formatdate(),
                msg,
            )
            thread = Thread(
                target=smtp_at_your_own_leasure,
                args=(
                    self.mailhost,
                    port,
                    self.username,
                    self.password,
                    self.fromaddr,
                    self.toaddrs,
                    msg,
                ),
            )
            thread.start()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)


# Test
if __name__ == "__main__":
    logger = logging.getLogger()

    gm = ThreadedTlsSMTPHandler((host, port), user, to, "Error!:", (user, pwd))
    gm.setLevel(logging.ERROR)

    logger.addHandler(gm)

    try:
        1 / 0
    except:
        logger.exception("Test ZeroDivisionError: division by zero")

Upvotes: 0

Manu J4
Manu J4

Reputation: 2859

As the OP pointed out, QueueHandler and QueueListener can do the trick! I did some research and adapted code found on this page to provide you with some sample code:

# In your init part,
# assuming your logger is given by the "logger" variable
# and your config is storded in the "config" dictionary

logging_queue = Queue(-1)
queue_handler = QueueHandler(logging_queue)
queue_handler.setLevel(logging.ERROR)
queue_handler.setFormatter(logging_formatter)
logger.addHandler(queue_handler)

smtp_handler = SMTPHandler(mailhost=(config['MAIL_SERVER'], config['MAIL_PORT']),
                           fromaddr=config['MAIL_SENDER'],
                           toaddrs=[config['ERROR_MAIL']],
                           subject='Application error',
                           credentials=(config['MAIL_USERNAME'], config['MAIL_PASSWORD']),
                           secure=tuple())
smtp_handler.setLevel(logging.ERROR)
smtp_handler.setFormatter(logging_formatter)

queue_listener = QueueListener(logging_queue, smtp_handler)
queue_listener.start()

# Let's test it. The warning is not mailed, the error is.
logger.warning('Test warning')
logger.error('Test error')

What I am not sure about is whether it is necessary to use setLevel and setFormatter twice, probably not.

Upvotes: 0

Mikus
Mikus

Reputation: 322

The simplest form of asynchronous smtp handler for me is just to override emit method and use the original method in a new thread. GIL is not a problem in this case because there is an I/O call to SMTP server which releases GIL. The code is as follows

class ThreadedSMTPHandler(SMTPHandler):
    def emit(self, record):
        thread = Thread(target=SMTPHandler.emit, args=(self, record))
        thread.start()

Upvotes: 4

Jonathan Livni
Jonathan Livni

Reputation: 107082

Here's the implementation I'm using, which I based on this Gmail adapted SMTPHandler.
I took the part that sends to SMTP and placed it in a different thread.

import logging.handlers
import smtplib
from threading import Thread

def smtp_at_your_own_leasure(mailhost, port, username, password, fromaddr, toaddrs, msg):
    smtp = smtplib.SMTP(mailhost, port)
    if username:
        smtp.ehlo() # for tls add this line
        smtp.starttls() # for tls add this line
        smtp.ehlo() # for tls add this line
        smtp.login(username, password)
    smtp.sendmail(fromaddr, toaddrs, msg)
    smtp.quit()

class ThreadedTlsSMTPHandler(logging.handlers.SMTPHandler):
    def emit(self, record):
        try:
            import string # for tls add this line
            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                            self.fromaddr,
                            string.join(self.toaddrs, ","),
                            self.getSubject(record),
                            formatdate(), msg)
            thread = Thread(target=smtp_at_your_own_leasure, args=(self.mailhost, port, self.username, self.password, self.fromaddr, self.toaddrs, msg))
            thread.start()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)

Usage example:

logger = logging.getLogger()

gm = ThreadedTlsSMTPHandler(("smtp.gmail.com", 587), 'bugs@my_company.com', ['admin@my_company.com'], 'Error found!', ('[email protected]', 'top_secret_gmail_password'))
gm.setLevel(logging.ERROR)

logger.addHandler(gm)

try:
    1/0
except:
    logger.exception('FFFFFFFFFFFFFFFFFFFFFFFUUUUUUUUUUUUUUUUUUUUUU-')

Upvotes: 13

Drahkar
Drahkar

Reputation: 1671

A thing to keep in mind when coding in Python is the GIL (Global Interpreter Lock). This lock prevents more than one process from happening at the same time. there are many number of things that are 'Blocking' activities in Python. They will stop everything until they completed.

Currently the only way around the GIL is to either push off the action you are attempting to an outside source like aix and MattH are suggesting, or to implement your code using the multiprocessing module (http://docs.python.org/library/multiprocessing.html) so that one process is handling the sending of messages and the rest is being handled by the other process.

Upvotes: 0

Jonathan Livni
Jonathan Livni

Reputation: 107082

You could use QueueHandler and QueueListener. Taken from the docs:

Along with the QueueListener class, QueueHandler can be used to let handlers do their work on a separate thread from the one which does the logging. This is important in Web applications and also other service applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via SMTPHandler) are done on a separate thread.

Alas they are only available from Python 3.2 onward.

Upvotes: 7

NPE
NPE

Reputation: 500217

Most probably you need to write your own logging handler that would do the sending of the email in the background.

Upvotes: 0

Related Questions