CODEWITHSUNDEEP
runit-testingtestthat
nokome
nokome

Reputation: 10304

R testthat not reporting failures or errors

I am finding some weird results using testthat. When running test_file, individual test_that calls are discovered but there is no output to the console other than the name of the context and the returned data.frame does not have the expected results. I suspect I am doing something really dumb but have tried lots of alternatives and get the same outcome.

Here is my tests.r file:

context("The context")

test_that('should pass',function(){
    expect_equal(42,42)
})

test_that('should fail',function(){
    expect_equal(43,42)
})

test_that('should error',function(){
    stop()
})

I run test_file on it:

> require(testthat)
Loading required package: testthat
> test_file('tests.r')
The context : 

> results <- test_file('tests.r')
The context : 

> results
     file     context         test nb failed error user system  real
1 tests.r The context  should pass  0      0 FALSE    0      0 0.002
2 tests.r The context  should fail  0      0 FALSE    0      0 0.001
3 tests.r The context should error  0      0 FALSE    0      0 0.001

As you can see, no console output and in the results data.frame instead of having one failed test and one with an error it looks as though everything has passed.

When I run the function bodies directly in the console I get what is expected:

> expect_equal(42,42)
> expect_equal(43,42)
Error: 43 not equal to 42
Mean relative difference: 0.02380952
> stop()
Error:

This is what I am running:

> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_NZ.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_NZ.UTF-8        LC_COLLATE=en_NZ.UTF-8    
 [5] LC_MONETARY=en_NZ.UTF-8    LC_MESSAGES=en_NZ.UTF-8   
 [7] LC_PAPER=en_NZ.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_NZ.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] testthat_0.9.1

Suspecting that I may have a broken install or something I tried the same tests on Ubuntu Trusty/R 3.12 and Windows 7/R 3.11 and get the same results. I really must be doing something dumb!

When I write analogous tests using Runit:

test.should_pass <- function(){
    checkEquals(42,42)
}

test.should_fail <- function(){
    checkEquals(43,42)
}

test.should_error <- function(){
    stop()
}

I get the expected output and results:

> results <- runTestFile('tests.r')


Executing test function test.should_error  ... Timing stopped at: 0 0 0 
Error in func() : 
    done successfully.



Executing test function test.should_fail  ... Timing stopped at: 0 0 0.001 
Error in checkEquals(43, 42) : Mean relative difference: 0.02325581
done successfully.



Executing test function test.should_pass  ...  done successfully.

> results
Number of test functions: 3 
Number of errors: 1 
Number of failures: 1

Upvotes: 4

Views: 1827

Answers (1)

tonytonov
tonytonov

Reputation: 25608

In your testthat tests you declare an anonymous function like so:

function() {
  expect_equal(43,42)
}

Run this piece of code. Is anything being actually executed? No, because you have given a declaration of function that you do not call. Now run 

{expect_equal(43,42)}

This is a block of code, not a function.

Changing your test file to

context("The context")

test_that('should pass', {
  expect_equal(42,42)
})

test_that('should fail', {
  expect_equal(43,42)
})

test_that('should error', {
  stop()
})

yields

The context : .12


1. Failure(@test.R#8): should fail -----------------------------------------------------------------------------------------------------------------------------------
43 not equal to 42
Mean relative difference: 0.02380952

2. Error: should error -----------------------------------------------------------------------------------------------------------------------------------------------

1: withCallingHandlers(eval(code, new_test_environment), error = capture_calls)
2: eval(code, new_test_environment)
3: eval(expr, envir, enclos)
4: stop() at test.R:12

Upvotes: 4

Related Questions