Pytest does not seem to match a raised exception in a unit test

Question

Context

I am trying pytest with the cryptography library. In a test, I decrypt and authenticate some data with a purposely corrupted authentication tag. Doing this should raise an 'InvalidTag' exception as written in the following example.

I am using the following way to assert an exception with pytest:

with pytest.raises(Exception, match='a_string'):
    myfunc()

My code example

#!/usr/bin/env python3

import pytest

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

key = bytes.fromhex(
"1111111111111111111111111111111111111111111111111111111111111111")
iv = bytes.fromhex("222222222222222222222222")
aad = bytes.fromhex("33333333333333333333333333333333")
pt = bytes.fromhex("4444444444444444")


def myfunc():
    raise ValueError("Exception 123 raised")


def test_match():
    with pytest.raises(ValueError, match=r".* 123 .*"):
        myfunc()


 def decrypt():
    # encrypt
    encryptor = Cipher(
        algorithms.AES(key),
        modes.GCM(iv),
        backend=default_backend()
    ).encryptor()
    encryptor.authenticate_additional_data(aad)
    ct = encryptor.update(pt) + encryptor.finalize()
    tag = encryptor.tag

    # let us corrupt the tag
    corrupted_tag = bytes.fromhex("55555555555555555555555555555555")

    # decrypt
    decryptor = Cipher(
        algorithms.AES(key),
        modes.GCM(iv, corrupted_tag),
        backend=default_backend()
    ).decryptor()
    decryptor.authenticate_additional_data(aad)

    return decryptor.update(ct) + decryptor.finalize()


 def test_decrypt():
     with pytest.raises(Exception, match='InvalidTag'):
         decrypt()

What really matter here is the function test_decrypt(). At the moment, if I run pytest with the previous code, I get this output:

> ▶ pytest
> 
> ============================================== test session starts ===============================================
> platform darwin -- Python 3.8.2, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
> rootdir: somewhere
> collected 2 items                                                                                                
> 
> tests/test_pytest_exception.py .F                                                                          [100%]
> 
> ==================================================== FAILURES ====================================================
> __________________________________________________ test_decrypt __________________________________________________
> 
>     def test_decrypt():
>         with pytest.raises(Exception, match='InvalidTag'):
> >           decrypt()
> 
> tests/test_pytest_exception.py:52: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
>     def decrypt():
>         # encrypt
>         encryptor = Cipher(
>             algorithms.AES(key),
>             modes.GCM(iv),
>             backend=default_backend()
>         ).encryptor()
>         encryptor.authenticate_additional_data(aad)
>         ct = encryptor.update(pt) + encryptor.finalize()
>         tag = encryptor.tag
>     
>         # let us corrupt the tag
>     
>         corrupted_tag = bytes.fromhex("55555555555555555555555555555555")
>     
>         # decrypt
>         decryptor = Cipher(
>             algorithms.AES(key),
>             modes.GCM(iv, corrupted_tag),
>             backend=default_backend()
>         ).decryptor()
>         decryptor.authenticate_additional_data(aad)
>     
> >       return decryptor.update(ct) + decryptor.finalize()
> 
> tests/test_pytest_exception.py:47: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> self = <cryptography.hazmat.primitives.ciphers.base._AEADCipherContext object at 0x10d2d53a0>
> 
>     def finalize(self):
>         if self._ctx is None:
>             raise AlreadyFinalized("Context was already finalized.")
> >       data = self._ctx.finalize()
> 
> ../../../.local/share/virtualenvs/a_virtual_env/lib/python3.8/site-packages/cryptography/hazmat/primitives/ciphers/base.py:198: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> self = <cryptography.hazmat.backends.openssl.ciphers._CipherContext object at 0x10d2d5040>
> 
>     def finalize(self):
>         # OpenSSL 1.0.1 on Ubuntu 12.04 (and possibly other distributions)
>         # appears to have a bug where you must make at least one call to update
>         # even if you are only using authenticate_additional_data or the
>         # GCM tag will be wrong. An (empty) call to update resolves this
>         # and is harmless for all other versions of OpenSSL.
>         if isinstance(self._mode, modes.GCM):
>             self.update(b"")
>     
>         if (
>             self._operation == self._DECRYPT and
>             isinstance(self._mode, modes.ModeWithAuthenticationTag) and
>             self.tag is None
>         ):
>             raise ValueError(
>                 "Authentication tag must be provided when decrypting."
>             )
>     
>         buf = self._backend._ffi.new("unsigned char[]", self._block_size_bytes)
>         outlen = self._backend._ffi.new("int *")
>         res = self._backend._lib.EVP_CipherFinal_ex(self._ctx, buf, outlen)
>         if res == 0:
>             errors = self._backend._consume_errors()
>     
>             if not errors and isinstance(self._mode, modes.GCM):
> >               raise InvalidTag
> E               cryptography.exceptions.InvalidTag
> 
> ../../../.local/share/virtualenvs/a_virtual_env/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/ciphers.py:170: InvalidTag
> 
> During handling of the above exception, another exception occurred:
> 
>     def test_decrypt():
>         with pytest.raises(Exception, match='InvalidTag'):
> >           decrypt()
> E           AssertionError: Pattern 'InvalidTag' does not match ''
> 
> tests/test_pytest_exception.py:52: AssertionError
> ============================================ short test summary info =============================================
> FAILED tests/test_pytest_exception.py::test_decrypt - AssertionError: Pattern 'InvalidTag' does not match ''
> ========================================== 1 failed, 1 passed in 0.12s ===========================================

From this output it looks like the 'InvalidTag' exception was raised but a second one also for a reason I do not understand. If I do not run the test but I just run the function decrypt() I get the following ouput:

>   python3 tests/test_pytest_exception.py
> Traceback (most recent call last):
>   File "tests/test_pytest_exception.py", line 54, in <module>
>     decrypt()
>   File "tests/test_pytest_exception.py", line 47, in decrypt
>     return decryptor.update(ct) + decryptor.finalize()
>   File "somewhere/a_virtual_env/lib/python3.8/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 198, in finalize
>     data = self._ctx.finalize()
>   File "somewhere/a_virtual_env/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/ciphers.py", line 170, in finalize
>     raise InvalidTag
> cryptography.exceptions.InvalidTag

From this output I have no doubt the 'InvalidTag' exception is raised.

Question

I want to check the 'InvalidTag' exception is raised in a unit test with pytest to pass the test. According to what I explained before, what am I doing wrong?

Answer 1

You are checking it the wrong way. You are validating a generic Exception and try to match the message of the error, not the actual exception. I assume it should be something like this:

from cryptography.exceptions import InvalidTag

def test_decrypt():
     with pytest.raises(InvalidTag, match=''):
         decrypt()

Answer 2

The match parameter to pytest.raises() performs a regex match on the string representation of the exception, which isn't what you want.

Instead, just pass the cryptography.exceptions.InvalidTag type as the first argument:

from cryptography.exceptions import InvalidTag

...

def test_decrypt():
    with pytest.raises(InvalidTag):
        decrypt()