Unit test
Comprised of four classes :
- Test.lua
-
The test runner. Once the tests are written, this class
is used to run them and to collect the results. It can be
run from the shell :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES PARAMETERS
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.errorIDE = true -- SWITCHES
LuaUnit:run(PARAMETERS)
- - SWITCHES
-
- - errorIDE
-
Turn IDE integration error format on (default =
off). In this mode the summary of failed tests is
send to 'stderr' instead of 'stdout' and the
output format is changed to :
Failed:PATH_TO_TESTPROGRAM/TESTPROGRAM|XX| TESTMETHOD
with 'XX' = line number. From the shell it is
turned on by '-e', and from a script as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.errorIDE = true -- default = false
see also jEdit integration
- - verbosity
-
Turn verbosity on (default = off when run from
the shell and on when run from a script). From the
shell it is turned on by '-v', and from a script
it can be turned off as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.verbosity = false -- default = true
- - help
-
Display a help message, only makes sense when
run from the shell. Displayed with '-help', '-h'
or '-?'.
- - silent
-
Turn off report output (default = on), only makes sense
when run from a script. It enables running different
testcases, gathering the output and reporting all results
in one report. From a script it can be turned on as follows :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit.silent = true -- default = false
LuaUnit:run(testcases1) -- no report
LuaUnit:run(testcases2) -- no report
...
LuaUnit:report() -- report with (sorted) results of testcases1 and testcases2
- - PARAMETERS
-
- - no parameters DEPRECATED
-
All test classes and test methods now run in their own
environment. This is incompatible with the old
functionality of scanning the global environment for
test classes and/or methods.
- - comma separated list of class names
-
Test.lua
will scan each of these classes for
methods who's name start with 'test'
. All
found methods will be called one by one. This is called
from the shell as follows :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES Test1 Test2
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1','Test2')
- - comma separated list of method names
-
Test.lua
will call each of these methods one by
one. This is called from the shell as follows :
lua 'PATH_TO_btd/lua/test.lua'/test.lua SWITCHES Test1:test1 Test2:test2
or from a script :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1:test1','Test2:test2')
- - remarks
-
There are a number of built in helper functions :
- - test runner methods
-
- - setUpClass (optional)
-
Any class level initialisation should go here,
it is only executed once for each test class.
- - tearDownClass (optional)
-
Any class level clean up should go here, it is
only executed once for each different test class.
- - setUp (optional)
-
Any method level initialisation should go here,
it is executed before each individual test method.
- - tearDown (optional)
-
Any method level clean up should go here, it is
executed after each individual test method.
- Remarks
-
Order of execution example :
local LuaUnit = require('btd.lua.test)()
...
LuaUnit:run('Test1:test1','Test1:test2','Test3')
Will run (actual order of test methods might differ) :
Test1:setUpClass()
Test1:setUp()
Test1:test1()
Test1:tearDown()
Test1:tearDownClass()
Test1:setUpClass()
Test1:setUp()
Test1:test2()
Test1:tearDown()
Test1:tearDownClass()
Test3:setUpClass()
Test3:setUp()
Test3:test1()
Test3:tearDown()
Test3:setUp()
Test3:test2()
Test3:tearDown()
- ...
Test1:tearDownClass()
- - test helper methods
-
- - clear
-
Clear the test statistics. While in one session
(of the stand alone interpreter for example) test
statitics are accumulated.
- - clearAll
-
Clear all
TestXXX
classes from the global
environment. Of minor importance now
because Test.lua
will do it's own clean up
(except of course if there is for some reason
a TestXX
class in the global environment).
- - report
-
When in
silent
mode a report can be triggered
by this method. Only the results which have not been
output yet will be reported.
local LuaUnit = require('btd.lua.test)()
LuaUnit.silent = true
LuaUnit:run(testcases1)
LuaUnit:run(testcases2)
LuaUnit:report() -- will report results from testcases1 and testcases2
but
local LuaUnit = require('btd.lua.test)()
LuaUnit:run(testcases1) -- report results testcase1
LuaUnit:run(testcases2) -- report results testcas2
LuaUnit:report() -- will report nothing > reports generated after every LuaUnit:run(...)
- - reportAll
-
Reports all gathered test results.
local LuaUnit = require('btd.lua.test)()
LuaUnit:run(testcases1) -- report results testcase1
LuaUnit:run(testcases2) -- report results testcas2
LuaUnit:reportAll() -- will report results from testcase1 and testcase2
- TestApi.lua
-
The methods and switches used to write test classes.
- - SWITCHES
-
- - Abort
-
Activates aborting remaining serialised tests upon failure
of one of these methods.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.Abort = true
TestClass.Serial = true
TestClass:testMethod1() -- will be run before testMethod2
test.equals(function using common resource,b)
end
TestClass:testMethod2() -- will be run after testMethod1 if testMethod1 didn't fail
test.equals(function using common resource,b)
end
- - Serial (all methods)
-
Normally all methods will be run in parallel, if they
need the same resources (file, socket, screen,...) this
can be avoided by setting the serial flag, after which
all methods in the test class will be run one by one (and
in alphabetical order).
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.Serial = true
TestClass:testMethod1() -- will be run before (or after) testMethod2
test.equals(function using common resource,b)
end
TestClass:testMethod2() -- will be run after (or before) testMethod1
test.equals(function using common resource,b)
end
- - Serial (specific methods)
-
If only some test methods use the same resources they can
be individually marked. In this case only one of the
marked test methods can be active (all none marked methods
keep running in parallel, even to the marked ones).
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.testMethod1Serial = true
TestClass:testMethod1() -- will be run before testMethod2
test.equals(function using common resource,b)
end
TestClass.testMethod2Serial = true
TestClass:testMethod2() -- will be run after testMethod1
test.equals(function using common resource,b)
end
TestClass:testMethod2() -- will be run concurrently with testMethod1 and/or testMethod2
test.equals(function using common resource,b)
end
- - Time
-
It is possible to limit the maximum time a test method
may run. After this time the violating method will be
aborted.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.testMethodTime = 1.5 -- test must terminate within 1.5 s
TestClass:testMethod()
test.equals(a,b)
end
- - METHODS
-
- - differs
-
Test whether a function does not give the expected result.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass:testMethod()
a = {1,2,3}
b = DeepCopy(a)
test.equals(a,b)
b[1] = 5
test.differs(a,b) -- my DeepCopy method works
end
- - equals
-
Test wheter a function gives the expected result.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass:testMethod()
test.equals(1 + 1,2)
end
For LuaUnit compatibility also known under the synonyms
assertEquals
and assert_Equals
.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}
TestClass:testMethod()
assertEquals(1 + 1,2)
end
Can also analyse table results, with following remarks :
- tables are equal if :
- Both tables have the same length
- At the same key find the same value
- or if they are equal (__eq operator)
-
The check is recursive, so tables within tables
will also be analysed
- - fails
-
Test whether a function throws an (expected) exception.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass:testMethod()
test.fails(function() error('error') end)
end
For LuaUnit compatibility also known under the synonyms
assertError
and assert_Error
.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}
TestClass:testMethod()
assertError(function() error('error') end)
end
- - succeeds
-
Test whether a function does not throw an exception.
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass:testMethod()
test.succeeds(function() end)
end
For LuaUnit compatibility also known under the synonyms
assertError
and assert_Error
.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
TestClass = {}
TestClass:testMethod()
assertError(function() error('error') end)
end
- - wrap
-
Wrap a number of test functions in a test class.
local test = require('btd.lua.testapi)
...
function test1()
assert(1 == 1)
end
...
TestFunction = test.wrap('test1','test2','test3'
For LuaUnit compatibility also known under the synonym
wrapFunctions
.
local test = require('btd.lua.testapi) -- necessary for error reporting
...
function test1()
assert(1 == 1)
end
...
TestFunction = wrapFunctions('test1','test2','test3'
- - MISCELANEOUS
-
- - Abort
-
Coupling test methods is not exactly standard unit
testing procedure, so here is a way to break that rule...
local test = require('btd.lua.testapi)
...
TestClass = {}
TestClass.Serial = true
TestClass:test1Method()
... -- test1Method test clauses
end
TestClass:test2Method()
if self.abort then return end
TestClass.test1Method(self) -- call test1Method (or write it again)
... -- (incremental) test2Method test clauses
end
The idea is that, to spare your sore eyes from having to
scroll through enormous error listings, testing a run of
Serial
'ised test
methods can be aborted when one of them fails. This might
come in handy when having a lot of incremental test cases
(eg. GUI testing). The other test methods will terminate
normally. When using this functionality the serialised
test methods should have names that reflect the order of
the incremental tests (alphabetically sorted). To
activate the feature the
Abort
flag must be used.
- TestClass.lua
-
Internal use : contains test class Lua Lanes code, reporting,...
Through use of Lua Lanes all test classes in a
LuaUnit:run(...)
invocation are run in parallel.
- TestMethod.lua
-
Internal use : contains test method Lua Lanes code, reporting,...
Through use of Lua Lanes all test methods in a
LuaUnit:run(...)
invocation are run in parallel. That
is : every test class runs it's contained test methods in
parallel. A side effect is that the order in which tests finish
can't be controlled/predicted. This behaviour can be changed (for
test method invocation within one test class) through the relevant
Serial
flags.
Best practices
This is meant to be a collection of tips and miscellaneous information
which might help and/or give a better understanding of the inner
workings (and quirks ?).
- preamble
-
My interest in Lua is as a complete development environment, this
is also where unit testing "shines" the most. It replaces Java as
my language of choice, which also explains the terminology
(classes, classpath,...).
- structure of a test
-
Ideas :
- no coupling between test methods
- common parts can be put in setUp and tearDown methods
-
variables can be passed as follows :
- global variables
-
sandboxed through use of Lua Lanes, every test method
has it's own copy. But conflicts may arise with your
class under test.
- local variables
-
sandboxed through use of Lua Lanes, every test method
has it's own copy
- through the
self
instance
-
valid alternative to local variables. But does not work
with
setUpClass
nor tearDownClass
,
and they only exist within the test method instance (are
not copied back to the test results).
- structure of test classes/methods
-
Generally tests are written on a per class basis, which makes them
easier to maintain. So every
ClassXX
has a
TestClassXX
test class. These test classes can be
run directly from the command line, but easier is to organise them
in a TestRunner
class. Here you can, for example,
group all tests of some specific application including (or
excluding) all used "library" functions. Since the actual runner
is an instance of btd.lua.Test
you can invoke the
run
method as many times as necessary and the results
will be collected in one report (the opposite is also possible of
course by instantiating a runner per set of tests). This also
makes it possible to chain testrunner classes.
- which methods to test
-
You can of course run all test runners for the entire application
and for all libraries included (even if only one function is
actually used) and eventually generate enormous error listings. In
that case one might opt for only running tests for the actually
used library methods.
- test results
-
The test results are reported to 'stdio' and/or to 'stderr' if the
errorIDE
flag is set. They are however also collected
in the testrunner instance to do with as you please :).
- class path
-
You should setup the lua path to include your test environment.
jEdit integration
Instructions for jEdit with 'console' and 'errorlist' plugins :
- Goto 'Plugins/Plugin Options/Console/Error Patterns'
- Add new Error Pattern '+'
- Enter Error Regexp: 'Failed:(.*?):(\d*?):(.*)'
- Enter Extra lines regexp: '(expected.*)'
- Enter Filename: '$1'
- Enter Line number: '$2'
- Enter Error message: '$3'