BTD
build-test-deploy Development Environment

Unit test

Comprised of four classes :
Test.lua
The test runner. Once the tests are written, this class is used to run them and to collect the results. It can be run from the shell :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES PARAMETERS
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.errorIDE = true -- SWITCHES
LuaUnit:run(PARAMETERS)
- SWITCHES
- errorIDE
Turn IDE integration error format on (default = off). In this mode the summary of failed tests is send to 'stderr' instead of 'stdout' and the output format is changed to :
Failed:PATH_TO_TESTPROGRAM/TESTPROGRAM|XX| TESTMETHOD
with 'XX' = line number. From the shell it is turned on by '-e', and from a script as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.errorIDE = true -- default = false
see also jEdit integration
- verbosity
Turn verbosity on (default = off when run from the shell and on when run from a script). From the shell it is turned on by '-v', and from a script it can be turned off as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.verbosity = false -- default = true
- help
Display a help message, only makes sense when run from the shell. Displayed with '-help', '-h' or '-?'.
- silent
Turn off report output (default = on), only makes sense when run from a script. It enables running different testcases, gathering the output and reporting all results in one report. From a script it can be turned on as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.silent = true -- default = false

LuaUnit:run(testcases1) -- no report
LuaUnit:run(testcases2) -- no report
...
LuaUnit:report() -- report with (sorted) results of testcases1 and testcases2
- PARAMETERS
- no parameters DEPRECATED
All test classes and test methods now run in their own environment. This is incompatible with the old functionality of scanning the global environment for test classes and/or methods.
- comma separated list of class names
Test.lua will scan each of these classes for methods who's name start with 'test'. All found methods will be called one by one. This is called from the shell as follows :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES Test1 Test2
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1','Test2')
- comma separated list of method names
Test.lua will call each of these methods one by one. This is called from the shell as follows :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES Test1:test1 Test2:test2
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1:test1','Test2:test2')
- remarks
  • All test classes and/or test methods parameters should be reachable by a corresponding require statement.
    local test = require('btd.lua.testapi)()
    ...
    LuaUnit:run('Test1:test1') -- 'Test1.lua' can be found by require
    LuaUnit:run('sub1.sub2.Test2') -- 'sub1/sub2/Test2.lua' can be found by require
    
  • It is possible to run a single class or method with LuaUnit:run('Test1:test4')
  • Function- and methodnames may be mixed as in LuaUnit:run('Test1','Test2:test1')
  • Tests are run randomly
  • Tests results are sorted alphabetically (first by class, then by method)
  • The mentioned naming conventions are not mandatory, but testmethods will only be found automatically if they are followed. Thus :
    • LuaUnit:run('function') will scan class function for testXXX methods
    • LuaUnit:run('function:method') will run the function:method() test
  • when giving command line SWITCHES these are to come before the PARAMETERS
  • command line switches '-v' and '-e' may be combined
There are a number of built in helper functions :
- test runner methods
- setUpClass (optional)
Any class level initialisation should go here, it is only executed once for each test class.
- tearDownClass (optional)
Any class level clean up should go here, it is only executed once for each different test class.
- setUp (optional)
Any method level initialisation should go here, it is executed before each individual test method.
- tearDown (optional)
Any method level clean up should go here, it is executed after each individual test method.
Remarks
  • It is possible to pass variables between setUp, tearDown and the test methods.
    local test = require('btd.lua.test)()
    ...
    TestClass = {}
    
    TestClass:setUp()
      self.variable = value -- initialise a variable
    end
    
    TestClass:testMethod()
      test.equals(function(self.variable),value2) -- use initialised variable
    end
    
  • It is not possible use self in setUpClass or tearDownClass
Order of execution example :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1:test1','Test1:test2','Test3')
Will run (actual order of test methods might differ) :
  • Test1:setUpClass()
  • Test1:setUp()
  • Test1:test1()
  • Test1:tearDown()
  • Test1:tearDownClass()
  • Test1:setUpClass()
  • Test1:setUp()
  • Test1:test2()
  • Test1:tearDown()
  • Test1:tearDownClass()
  • Test3:setUpClass()
  • Test3:setUp()
  • Test3:test1()
  • Test3:tearDown()
  • Test3:setUp()
  • Test3:test2()
  • Test3:tearDown()
  • ...
  • Test1:tearDownClass()
- test helper methods
- clear
Clear the test statistics. While in one session (of the stand alone interpreter for example) test statitics are accumulated.
- clearAll
Clear all TestXXX classes from the global environment. Of minor importance now because Test.lua will do it's own clean up (except of course if there is for some reason a TestXX class in the global environment).
- report
When in silent mode a report can be triggered by this method. Only the results which have not been output yet will be reported.
local LuaUnit = require('btd.lua.test)()

LuaUnit.silent = true

LuaUnit:run(testcases1)
LuaUnit:run(testcases2)

LuaUnit:report() -- will report results from testcases1 and testcases2
but
local LuaUnit = require('btd.lua.test)()

LuaUnit:run(testcases1) -- report results testcase1
LuaUnit:run(testcases2) -- report results testcas2

LuaUnit:report() -- will report nothing > reports generated after every LuaUnit:run(...)
- reportAll
Reports all gathered test results.
local LuaUnit = require('btd.lua.test)()

LuaUnit:run(testcases1) -- report results testcase1
LuaUnit:run(testcases2) -- report results testcas2

LuaUnit:reportAll() -- will report results from testcase1 and testcase2
TestApi.lua
The methods and switches used to write test classes.
- SWITCHES
- Abort
Activates aborting remaining serialised tests upon failure of one of these methods.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass.Abort = true
TestClass.Serial = true

TestClass:testMethod1() -- will be run before testMethod2
  test.equals(function using common resource,b)
end

TestClass:testMethod2() -- will be run after testMethod1 if testMethod1 didn't fail
  test.equals(function using common resource,b)
end
- Serial (all methods)
Normally all methods will be run in parallel, if they need the same resources (file, socket, screen,...) this can be avoided by setting the serial flag, after which all methods in the test class will be run one by one (and in alphabetical order).
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass.Serial = true

TestClass:testMethod1() -- will be run before (or after) testMethod2
  test.equals(function using common resource,b)
end

TestClass:testMethod2() -- will be run after (or before) testMethod1
  test.equals(function using common resource,b)
end
- Serial (specific methods)
If only some test methods use the same resources they can be individually marked. In this case only one of the marked test methods can be active (all none marked methods keep running in parallel, even to the marked ones).
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass.testMethod1Serial = true
TestClass:testMethod1() -- will be run before testMethod2
  test.equals(function using common resource,b)
end

TestClass.testMethod2Serial = true
TestClass:testMethod2() -- will be run after testMethod1
  test.equals(function using common resource,b)
end

TestClass:testMethod2() -- will be run concurrently with testMethod1 and/or testMethod2
  test.equals(function using common resource,b)
end
- Time
It is possible to limit the maximum time a test method may run. After this time the violating method will be aborted.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass.testMethodTime = 1.5 -- test must terminate within 1.5 s
TestClass:testMethod()
  test.equals(a,b)
end
- METHODS
- differs
Test whether a function does not give the expected result.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass:testMethod()
  a = {1,2,3}
  b = DeepCopy(a)
  test.equals(a,b)
  b[1] = 5
  test.differs(a,b) -- my DeepCopy method works
end
- equals
Test wheter a function gives the expected result.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass:testMethod()
  test.equals(1 + 1,2)
end
For LuaUnit compatibility also known under the synonyms assertEquals and assert_Equals.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}

TestClass:testMethod()
  assertEquals(1 + 1,2)
end
Can also analyse table results, with following remarks :
  • tables are equal if :
    • Both tables have the same length
    • At the same key find the same value
    • or if they are equal (__eq operator)
  • The check is recursive, so tables within tables will also be analysed
- fails
Test whether a function throws an (expected) exception.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass:testMethod()
  test.fails(function() error('error') end)
end
For LuaUnit compatibility also known under the synonyms assertError and assert_Error.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}

TestClass:testMethod()
  assertError(function() error('error') end)
end
- succeeds
Test whether a function does not throw an exception.
local test = require('btd.lua.testapi)
...
TestClass = {}

TestClass:testMethod()
  test.succeeds(function() end)
end
For LuaUnit compatibility also known under the synonyms assertError and assert_Error.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}

TestClass:testMethod()
  assertError(function() error('error') end)
end
- wrap
Wrap a number of test functions in a test class.
local test = require('btd.lua.testapi)
...
function test1()
  assert(1 == 1)
end
...
TestFunction = test.wrap('test1','test2','test3'
For LuaUnit compatibility also known under the synonym wrapFunctions.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
function test1()
  assert(1 == 1)
end
...
TestFunction = wrapFunctions('test1','test2','test3'
- MISCELANEOUS
- Abort
Coupling test methods is not exactly standard unit testing procedure, so here is a way to break that rule...
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.Serial = true

TestClass:test1Method()
  ... -- test1Method test clauses
end
TestClass:test2Method()
  if self.abort then return end
  TestClass.test1Method(self) -- call test1Method (or write it again)
  ... -- (incremental) test2Method test clauses
end
The idea is that, to spare your sore eyes from having to scroll through enormous error listings, testing a run of
Serial'ised test methods can be aborted when one of them fails. This might come in handy when having a lot of incremental test cases (eg. GUI testing). The other test methods will terminate normally. When using this functionality the serialised test methods should have names that reflect the order of the incremental tests (alphabetically sorted). To activate the feature the Abort flag must be used.
TestClass.lua
Internal use : contains test class Lua Lanes code, reporting,... Through use of Lua Lanes all test classes in a LuaUnit:run(...) invocation are run in parallel.
TestMethod.lua
Internal use : contains test method Lua Lanes code, reporting,... Through use of Lua Lanes all test methods in a LuaUnit:run(...) invocation are run in parallel. That is : every test class runs it's contained test methods in parallel. A side effect is that the order in which tests finish can't be controlled/predicted. This behaviour can be changed (for test method invocation within one test class) through the relevant Serial flags.

Best practices

This is meant to be a collection of tips and miscellaneous information which might help and/or give a better understanding of the inner workings (and quirks ?).
preamble
My interest in Lua is as a complete development environment, this is also where unit testing "shines" the most. It replaces Java as my language of choice, which also explains the terminology (classes, classpath,...).
structure of a test
Ideas :
  • no coupling between test methods
  • common parts can be put in setUp and tearDown methods
  • variables can be passed as follows :
    global variables
    sandboxed through use of Lua Lanes, every test method has it's own copy. But conflicts may arise with your class under test.
    local variables
    sandboxed through use of Lua Lanes, every test method has it's own copy
    through the self instance
    valid alternative to local variables. But does not work with setUpClass nor tearDownClass, and they only exist within the test method instance (are not copied back to the test results).
structure of test classes/methods
Generally tests are written on a per class basis, which makes them easier to maintain. So every ClassXX has a TestClassXX test class. These test classes can be run directly from the command line, but easier is to organise them in a TestRunner class. Here you can, for example, group all tests of some specific application including (or excluding) all used "library" functions. Since the actual runner is an instance of btd.lua.Test you can invoke the run method as many times as necessary and the results will be collected in one report (the opposite is also possible of course by instantiating a runner per set of tests). This also makes it possible to chain testrunner classes.
which methods to test
You can of course run all test runners for the entire application and for all libraries included (even if only one function is actually used) and eventually generate enormous error listings. In that case one might opt for only running tests for the actually used library methods.
test results
The test results are reported to 'stdio' and/or to 'stderr' if the errorIDE flag is set. They are however also collected in the testrunner instance to do with as you please :).
class path
You should setup the lua path to include your test environment.

jEdit integration

Instructions for jEdit with 'console' and 'errorlist' plugins :
  • Goto 'Plugins/Plugin Options/Console/Error Patterns'
  • Add new Error Pattern '+'
  • Enter Error Regexp: 'Failed:(.*?):(\d*?):(.*)'
  • Enter Extra lines regexp: '(expected.*)'
  • Enter Filename: '$1'
  • Enter Line number: '$2'
  • Enter Error message: '$3'

Valid XHTML 1.0!

$Id: unittest.html