CODEWITHSUNDEEP
pythonunit-testingdatabrickspython-unittest
Szabolcs Magyar
Szabolcs Magyar

Reputation: 409

DataBricks Python unit test error help needed

I have a problem with unit testing in DataBricks. I have not found any similar error message yet. Could someone please help me?

Below you can find my code:

import unittest

def add(n1, n2):
  a = n1+n2
 
  return a

class addTest(unittest.TestCase):
  
  def test_add(self):
    self.assertEqual(add(5,5), 10)

unittest.main()

The error message is this:

EEEEEEEE
======================================================================
ERROR: 36941 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '36941'

======================================================================
ERROR: 0 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '0'

======================================================================
ERROR: 50000 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '50000'

======================================================================
ERROR: 923 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '923'

======================================================================
ERROR: dcc9864f9b334f05b20e25d63062f770 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute 'dcc9864f9b334f05b20e25d63062f770'

======================================================================
ERROR: 3 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '3'

======================================================================
ERROR: 4b8784f2c9e2dd8ccd5890b026495bf12fc0671c88baa508721606e240ca39b2 (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '4b8784f2c9e2dd8ccd5890b026495bf12fc0671c88baa508721606e240ca39b2'

======================================================================
ERROR: unpinned (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute 'unpinned'

Can someone please help me? Thank you!

Upvotes: 4

Views: 1515

Answers (2)

Tom
Tom

Reputation: 616

I had the same issue and could solve it thanks to two articles (see source bellow). But, the most important part, here's the reproductible example that you can execute in DATABRICKS notebook (it worked for me, I have python 3.7):

import unittest

class SayHiAndBye():
    def say_hello(self,name):
      return f"Hello {name}"
    def say_bye(self,name):
      return f"Bye {name}"

class MyNotebookTesting(unittest.TestCase):
    """
    You can test whaatever you want in here
    """
    
    def test_hello(self):
        sayHiBye = SayHiAndBye()
        actual_output = sayHiBye.say_hello("Bratt")
        expected_ouput = "Hello Bratt"
        self.assertEqual(actual_output, expected_ouput)

    def test_bye(self):
        sayHiBye = SayHiAndBye()
        actual_output = sayHiBye.say_bye("Bratt")
        expected_ouput = "Bye Bratt"
        self.assertEqual(actual_output, expected_ouput)

        
suite = unittest.TestLoader().loadTestsFromTestCase(MyNotebookTesting)
runner = unittest.TextTestRunner(verbosity=100)
results = runner.run(suite)
print(results)

As you can see, it will tell you that test_string fails, but test_boolean is OK. These values are hardcoded just for the example (feel free to change them and see what's the output).

Sources:

https://www.linuxtut.com/en/cd2a861b29e00448fc0f/

https://www.journaldev.com/15899/python-unittest-unit-test-example

Upvotes: 2

Alex Ott
Alex Ott

Reputation: 87214

The unittest.main package requires special handling when using Databricks. The main requirement for use with Databricks is to set exit = False in the list of arguments of unittest.main function. It also makes sense to explicitly pass argv as single-element list, to avoid use of sys.argv that on Databricks contains parameters that were used to start Python subprocess. (see documentation on unittest.main).

But it also may not work as desired. I'm usually generating the test suite by doing autodiscovery myself and the executing it. For example with following code:

import unittest

def discover_test_cases(*test_classes):
  suite = unittest.TestSuite()
  for test_class in test_classes:
    for test in unittest.defaultTestLoader.getTestCaseNames(test_class):
      suite.addTest(test_class(test))
      
  return suite

def discover_test_classes():
  classes = [obj for name, obj in globals().items()
    if name.endswith('Test') and 
       obj.__module__ == '__main__' and 
       isinstance(obj, type) and 
       unittest.case.TestCase in set(obj.__bases__)]

  return discover_test_cases(*classes)

and it's used as:

suite = discover_test_classes()
runner = unittest.TextTestRunner()
results = runner.run(suite)

(you can also use unittest-xml-reporting package and then you can get test results in the JUnit XML format that could be intergrated into build process)

P.S. There is also the nutter library from the Microsoft - it's designed specifically for notebooks testing. It has some advantages, like, automatic discovery of tests, generation of results as JUnit XML, etc. I have an example of using it with Databricks Repos functionality.

I'm using following code

Upvotes: 2

Related Questions