Test automation with Jenkins + RobotFramework

Question

after a long Internet search I would like to ask you the following question.

We are using Jenkins for building and unit testing of a simulation code which is written in C++.

This works very well. After looking into fitnesse and robotframework I am still not able to run the following test problems.

My program is a command line program which reads some input file and computes some output data. (e.g. simcode.exe -j input##.inp --> output.dat)

I am looking for a way that I can create a test suite via a web interface. Meaning I provide for each test case a input file and some reference output data and the test suite is than executed after a successful build out of Jenkins. Based on the results of the difference between output data and reference output data, a xml file should be created which can be given to Jenkins. This xml file should hold information about all the test case results (e.g. successful or not).

The information of the xml file should be displayed in Jenkins again.

I looking for an approach where I do not need to compile any library to my program.

I would be very thankful for any hint which explains how to achieve that with the RobotFramework. (Is it even possible ?)

Thanks in advance!

Answer 1

What you want seems like a silver bullet. It is feasible, you don't need to compile a thing, but you still need to write some logic around robot framework.

1. As far as I understood you need to call something in CLI - this is trivial, robot already does this
2. It has an input file and some output data - trivial, - robot already does this
3. You need to compare the actual output data with the expected output data - here you need some elbow grease.
4. Package the logs - robot already does this.
5. Present them in Jenkins - trivial - robot already does this.

So all the heavy lifting is already done.

1. Calling CLI - use OperatingSystem

   *** Settings ***
    Library  OperatingSystem
   
   *** Test Cases ***
   Basic test
       Run Me A Command  simcode.exe -j input01.inp
   
   *** keywords ***
   Run Me A Command 
       [Arguments]  ${command} 
       ${rc}   ${output} =     Run and Return RC and Output    ${command}
       Log  ${output}
       Should Be Equal As Integers  ${rc}  0   
   
       Should Not Contain  ${output}  FAIL
   

2. If you want a flexible approach to inputting stuff

   *** settings ***
   Library  OperatingSystem
   
   *** Test Cases ***
   Combinations
       [Template]  Basic test
       input01.inp
       input02.inp
       input03.inp
   
   *** keywords ***
   Basic test
       [Arguments]  ${input}
       Run Me A Command  simcode.exe -j ${input}
   
   
   Run Me A Command 
       [Arguments]  ${command} 
       ${rc}   ${output} =     Run and Return RC and Output    ${command}
      Log  ${output}
      Should Be Equal As Integers  ${rc}  0    
   
       Should Not Contain  ${output}  FAIL
   

   3. Next, we need to compare the content of two files... Assuming that you moved your files around to a convenient location you can write yourself a keyword to compare the content

      *** settings ***
      Library  OperatingSystem
      
      *** Test Cases ***
      Combinations
          [Template]  Basic test
          ../resources/input01.inp  ../resources/expectedoutput01.out
          ../resources/input02.inp  ../resources/expectedoutput02.out
          ../resources/input03.inp  ../resources/expectedoutput03.out
      
      *** keywords ***
      Basic test
          [Arguments]  ${input}  ${expected_output}
          Run Me A Command  simcode.exe -j ${input}
          Log File  output.data
          Compare Files  ${expected_output}  output.data
      

Let's assume that the content is a list of parameters, in the ini format. For example let's assume that you calculate the square root for the numbers present in the input file

How do we store the expected data? Let's assume we have a file called expected.dat

[defaults]
n_1=1
n_4=2
n_9=3

And we have an output.data

[defaults]
n_1=1
n_4=2
n_9=2

Then we need to write ourselves a file comparator. Either you go with diff and the OperatingSystem library, if you are confident that the files should be identical or you can write a simple comparator like:

def get_param(self,theFile)
    config = ConfigParser.RawConfigParser()
    config.read(theFile)
    return config.items("defaults")


def compareMyDict(self, expected, actual):  #There are better ways of doing this 
    for k, v in actual:
        if v != expected[k]:
            print("error message")
            print(k)
            print(v)
            print(expected[v])
            #print whatever you deem fit
            return
        print("Matched OK " + str(k))
        print(k)
        print(v)
        print(expected[v])

def compare_files(self, expectedFile, actualFile):
    '''
    From sourceFile formatted
    [defaults]
    key=value
    '''

    d1 = get_param(expectedFile)
    d1 = get_param(actualFile)

    compareMyDict(d1,d2)
    checkLenghts(d1,d2)
    criteria2(d1,d2)    
    #etc.

4 and 5. After you run, you use the publish artefact and publish robot results plugins to pass the nicely formatted output to Jenkins.

Voila!

PS: The code might not run - I wrote it freestyle in a simple editor without syntax highlighting and I did not tested it.

Answer 2

Robot Framework is a test automation tool... you can do many different things with that...

I don't really understand what you mean with the test suite via a web interface... but in general the functionality you describe seems that can be done with RobotFramework...

In Brief

You can create a Test Suite that can have many Test Cases for example you could have onef or every input file you want to check!

The OperatingSystem Build In Library has the Keyword Run, that you can probably use or if you are running the commands remotely you can use the Execute Command from the optional SSHLibrary

For Every Test Case you could create a step that runs the command and another one that verifies the output file against what you expected. If they match the Test Case is marked as Pass, else it is marked as Fail...

RobotFramework can produce for every Test Suite that you run a log and report file that are in html format.

Jenkins and Hudson have a plugin for RobotFramework that you can use to display these output files in a very nice way! i.e. How many Test Cases have Passed/Failed