test case teardown does not execute when using robot.api but does work when using pybot runner

Question

Background

I'm experimenting with the programmatic API for Robot Framework in order to dynamically create test cases from active data sets.

Problem Summary

While converting my tests to run using the utility libraries provided in the robot.api package I ran into an issue where it does not appear that the teardown keyword for a test case gets executed. This teardown does appear to execute if I use the keyword in a .robot file.

My question is: Why does my test case teardown fail to execute when using robot.api but not pybot?

Example

I've created the following example to demonstrate the issue I'm seeing. To run it you'll need to have the Python robotframework library installed (pip install robotframework).

In the example runs (below) you'll see that the console log line Test Case Teardown Ran appears in the pybot results but not within the robot.api results. I apologize in advance for the format of the lines - this is how the test case runner output them and I didn't want to modify the output and possibly remove some clues in the process.

resource.robot

** Keywords **
TestSuite Setup
    Log To Console  TestSuite Setup Ran 

TestSuite Teardown
    Log To Console  TestSuite Teardown Ran 

TestCase Setup
    Log To Console  TestCase Setup Ran 

TestCase Teardown
    Log To Console  TestCase Teardown Ran 

suite.robot

** Settings **
Resource             resource.robot

Suite Setup     TestSuite Setup
Suite Teardown  TestSuite Teardown

** Test Cases **
Example test case
    [setup]     TestCase Setup
    [teardown]  TestCase Teardown
    Log To Console  Test ran 

suite.py

import robot.api as robot_api

suite = robot_api.TestSuite('Programmatic test suite')
suite.resource.imports.resource('resource.robot')
suite.keywords.create('TestSuite Setup', type='setup')
suite.keywords.create('TestSuite Teardown', type='teardown')

test = suite.tests.create('Example test case')
test.keywords.create('TestCase Setup', type='setup')
test.keywords.create('TestCase Teardown', type='teardown')
test.keywords.create('Log To Console', args=['Test ran'])
result = suite.run(output='output.xml')
robot_api.ResultWriter(result).write_results(
    log='log.html',
    report='report.html'
)

Result (using Pybot)

$ pybot suite.robot
==============================================================================
Suite                                                                         
==============================================================================
TestSuite Setup Ran
Example test case                                                     TestCase Setup Ran
.Test ran
.TestCase Teardown Ran
Example test case                                                     | PASS |
------------------------------------------------------------------------------
TestSuite Teardown Ran
Suite                                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /private/tmp/robot_problem/output.xml
Log:     /private/tmp/robot_problem/log.html
Report:  /private/tmp/robot_problem/report.html

Result (using python-2.7)

$ python suite.py 
==============================================================================
Programmatic test suite                                                       
==============================================================================
TestSuite Setup Ran
Example test case                                                     TestCase Setup Ran
.Test ran
Example test case                                                     | PASS |
------------------------------------------------------------------------------
TestSuite Teardown Ran
Programmatic test suite                                               | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /private/tmp/robot_problem/output.xml

Answer 1

This sounds like a bug to me. Or perhaps an undocumented quirk. The setup keyword has to be added FIRST and the teardown must be added LAST.

test = suite.tests.create('Example test case')
test.keywords.create('TestCase Setup', type='setup')
test.keywords.create('Log To Console', args=['Test ran'])
test.keywords.create('TestCase Teardown', type='teardown')
result = suite.run(output='output.xml')

I figured this out by looking in class Keywords in robot/model/keyword.py:

@property
def setup(self):
    return self[0] if (self and self[0].type == 'setup') else None

@property
def teardown(self):
    return self[-1] if (self and self[-1].type == 'teardown') else None

Note the indicies of 0 and -1 in combination with a check on type.