Ervin Varga
Sourceforge user name: evarga
This is a first installment, from a series of future articles about the JXUnit framework [1]. JXUnit is an extension of the JUnit framework [2], enabling separation of test data from test code. The JUnit framework does not provide developers with an easy mechanism for managing test data independently of test code. JXUnit is fully based upon XML technology [3], and uses the Quick framework [4] for handling XML related stuff. Not only test data is usually packed inside an XML document, but part of the test case logic is also contained within a special XML document, called JXU file. Only a miniscule part of the whole test suite is required to be implemented in Java, more precisely, some test steps. All in all, JXUnit is an XML based, highly flexible unit testing framework, and this article tries to make it more comprehensible for a large community of software developers.
This paper assumes that 'directory' and 'folder' are synonyms, hence, they are used interchangeably in a text.
Before we delve more deeply into the inner details of the JXUnit framework, note that for a successful mastering of concepts listed further in this paper, you will need a solid understandings of XML technology, the Quick framework, and most notably, the JUnit framework. It would be far out of scope of this document to talk about all these stuff, too. Therefore, read materials listed in the References section ([2, 3 and 4]) in order to become acquainted with the aforementioned technology and frameworks.
In order to avoid repetition, as much as possible, this paper does not contain facts mentioned elsewhere. For example, the [1] is a good jump-start toward gaining knowledge of the JXUnit framework. There, you can read about a general wisdom why it is beneficial to separate test data from test code, as well as, learn about some techniques not duplicated in this paper.
The current release of JXUnit is 3.1.3 and can be downloaded from the following WEB page http://sourceforge.net/projects/jxunit/ . This article assumes that you are working with this particular release, for earlier versions some statements might not apply. The other assumption is that you have a JDK 1.2 (or greater) installed on your machine. The operating system does not matter, however, explanations are given for Windows.
JUnit is a framework for writing unit tests. Numerous extensions are available for different purposes; one of them is a JXUnit framework. JXUnit is a directory-driven test scripting system. A complete test suite is basically comprised from various test suites/cases located in different parts of the whole folder tree in question. Essentially, the terms test suite and test case are only conceptual in JXUnit, since, no explicit differentiation exists on the framework level between them. Nevertheless, in the rest of this paper we will use a term test suite for a test case holding other tests. A folder where they are situated names all test cases.
Using a folder tree for specifying a test suite has a clear advantage over a hard-coded variant, in sense, new suites can be easily constructed by just reorganizing folders (no code changes, no recompilations). Thus, JXUnit effectively utilizes the file system for this particular purpose. Furthermore, as JXUnit builds upon JUnit, tests can be run using the standard TestRunners (textual, AWT or SWING) found in the JUnit framework.
Nonetheless, the Quick framework is the cornerstone of JXUnit's powerfulness and flexibility. Quick is a Java based XML data binding framework, more precisely, a framework for advanced data transformations and mappings. Quick uses QJML for expressing a binding schema. This schema serves as a bridge between XML and Java object representations. By using Quick, one can easily create a Java object from data encapsulated inside an XML document, and vice versa, without ever touching the source code of the class the target Java object belongs to.
Unfortunately, the high flexibility has a price; honestly, the JXUnit framework is not for rookies (those who even struggle with elementary Java), and it do requires some time for getting acquainted with it. However, once you grasp the concepts, you will be, I'm sure, well rewarded for all your learning efforts, though.
As a proof of concept, i.e. its high customizability, take a look at JxWeb [5], which is absolutely built upon the JXUnit framework, and provides testing facility for Web services.
The setup procedure is divided here (I mean, in this paper) into two separate parts: installation and configuration.
JXUnit is shipped as a ZIP archive named 'JXUnit3.1.3.zip'. The installation is quite rudimentary, just unpack the archive into some folder on your hard drive. Let us call this folder as JXUNIT_HOME. The following directory structure will be created:
<JXUNIT_HOME>\bin
\doc
\JARs
\src
The 'bin' folder contains various scripts needed during development of your test suites/cases. Also, this folder holds a script (jxtest.bat) for starting up the JXUnit's engine. The 'doc' folder holds various manuals in HTML format. The 'JARs' contains all required libraries for using JXUnit. The 3.1.3 version of JXUnit has everything it needs inside this folder, no separate bundles should be installed, such as, the JUnit, Quick, etc. At last, in the 'src' folder the JXUnit framework's source code lurks.
The best way to work with JXUnit is to open up a new command window, and add the '<JXUNIT_HOME>\bin' to your PATH environment variable. In this way all scripts will be available to you, no matter where are you currently positioned.
The next compulsory step you should take is to edit the quickSetup.bat file (for LINUX users there is always a corresponding SH script). Actually, the following line should be modified in way depicted below:
set QuickJARs=<JXUNIT_HOME>\JARs
As a last action, execute the previously modified script in order for changes to take effect. If everything was successful, essentially, the installation and configuration of the JXUnit framework has been completed. You are now ready to use JXUnit. As quick self test, do the following:
Open up a new command window (do not forget to set the PATH, and execute the quickSetup.bat file).
Go to the 'src\net\sourceforge\jxunit\testfail' directory (in the rest of this document, all path references, if not told otherwise, will be given relative to the '<JXUNIT_HOME>\' folder).
Start the jxtest.bat script from there.
A JUnit's TestRunner should show up signaling that one test case has been executed, and that there was one failure (of course, this failure is forced, so, do not worry).
If you prefer a textual or AWT TestRunner over a SWING one, freely edit the jxtest.bat file, and put there one of the following alternatives:
junit.textui.TestRunner for a textual runner
junit.swingui.TestRunner for a SWING based runner
junit.awtui.TestRunner for an AWT based runner
For more advanced configurations, such as, changing the underlying XML parser, please, consult the JXUnit's HOME page, and/or read messages on the discussion forum.
Let me make a short digression before continuing about how JXUnit is designed and implemented. The 'docs' folder contains three prominent HTML files, serving as kind of entries, and these are listed below:
The [1] is completely contained in the 'index.html' file.
For a description of all JXU commands (in the rest of this paper, I'll use a term command, opposed to the accepted XML convention called element), please, open the 'jxu.html' file in your browser.
For a description of all JXUC commands, please, open the 'jxuc.html' file in your browser.
Links to further materials explaining the JXUnit framework can be found on the following Web page http://sourceforge.net/forum/forum.php?thread_id=721709&forum_id=67873 (part of the discussion forum). I admit, that documentation for JXUnit is scarce, and a huge task is left to be done pertaining to documenting the API. Nevertheless, the situation is not so hopeless, taking into account JXUnit's intuitive architecture, and the existence of many short examples of its usage (see "The Self Testing Principle" section).
Usually, a test case built using the JXUnit framework is comprised from three different parts:
One 'test.jxu' file (it contains part of the test case logic expressed in the JXU markup language).
Test data, usually put inside a proprietary XML document(s), but this is not mandatory (a simple text file could be equally used).
Test step(s) written in Java.
The following list could be prolonged by an additional extra item called test context. A test context description is conveyed using the JXUC markup language, and stored in a 'test.jxuc' file. As it mentioned before, the JXU and JXUC markup languages are XML based. Their formal definitions can be found in 'jxu.qjml' and 'jxuc.qjml' files, respectively (of course, the corresponding DTD files are also relevant, see 'jxu.dtd' & 'jxuc.dtd'). Please, take a look into the 'src\net\sourceforge\jxunit' folder where these QJML & DTD files are situated. Do not get intimidated by their complexity at first blush, working with QJML files is funny once you get used to them (believe me).
A 'test.jxu' file contains JXU commands, and basically describes the steps needed to perform a particular test case. During the "execution" of a 'test.jxu' file by the JXUnit's engine, test steps will be run, and their results examined. Data between a test case and its steps and/or between test steps themselves are passed by properties. Each property has a name and value.
To parameterize a single 'test.jxu' file (essentially, a test case) with different data files; you would use a 'test.jxuc' file. If this file is present in the same directory where a matching 'test.jxu' file is located, it will establish a specific test context. This context contains JXUC commands describing a directory tree where data files are put, and file filters for those data files. A file filter is defined using a regular expression.
All in all, using JXU and JXUC files, much of the test case's logic could be changed without touching a source code, thus, no need for time consuming recompilations. Other complications implicated by code changes will not be elaborated here, though.
The following UML diagram shows the most significant architectural interfaces and classes of the JXUnit framework.
On the above diagram classificators are colored as follows:
Maybe the easiest way to portray how JXUnit really operates is through use cases. The following, not so formal, white-box "Run Tests" use case shows all phases of JXUnit's operation (the convention for describing a use case has been taken from an excellent book about writing use cases [6]). The underlined sections of the text in a use case description (scenario and extensions parts) are links to lower level use cases. On the next diagram a table of content is depicted (a yellow ellipse symbolizes a "user-goal" level use case, while blue ellipses signify "subfunction" level use cases).
Use Case "Run
Tests"
Primary Actor: Developer
Secondary Actor: JUnit framework's TestRunner
Scope: JXUnit framework
Level: User-goal
Preconditions: The JXUnit framework has been successfully
installed and configured. The directory tree has been created,
with appropriate test cases and data. Developer opened a command
window, and set the working directory to the root of the test
folder tree.
Minimal and Success Guarantees: All 'test.jxu' and
'test.jxuc' files have been taken into consideration (those
situated inside a folder tree in question), and processed
correctly. All tests are run and the results of execution are
presented to the developer. In case of a fatal error a suitable
error report is generated and presented to the developer.
Trigger: Developer starts the jxtest.bat file.
Main Success Scenario:
1. Developer starts the jxtest.bat file.
2. The TestRunner seizes control over execution and
starts the JXUnit framework's engine (calls the static suite() method of the JXTestCase class).
3. The engine creates a test suite and passes back that
suite to the TestRunner.
4. The TestRunner runs a test suite.
5. TestRunner generates an execution history
report and presents it to the developer.
Extensions:
* In case of a fatal JVM error (like out of memory) the use case
fails with decent information presented to a developer.
The following four white-box subfunction level use cases show what is happening inside a JXTestCase.suite() method, in other words, how is a test suite created, and how is that suite processed afterwards.
Use Case
"Create a Test Suite"
Primary Actor: JUnit framework's TestRunner
Scope: JXUnit framework
Level: Subfunction
Preconditions: None
Minimal and Success Guarantees: All 'test.jxu' and
'test.jxuc' files have been taken into consideration (those
situated inside a folder tree in question), and all 'test.jxuc'
files have been processed correctly in order to establish proper
contexts for test cases. In case of a fatal error a suitable
error is thrown.
Trigger: The JUnit framework's TestRunner calls the JXTestCase.suite() static method.
Main Success Scenario:
1. The JUnit framework's TestRunner calls the JXTestCase.suite() static method.
2. The engine creates an instance of a TestSuite class.
3. The engine searches the current directory for a 'test.jxu'
file.
4. The engine creates a junit.framework.Test instance.
5. The engine adds an instance created to a test suite.
6. The engine enters the next subdirectory, sets it to be the
current one, and use case continues from step 3.
Extensions:
* In case of a fatal JVM error (like out of memory) the use case
fails.
3a. There is no 'test.jxu' file in the current directory. Use
case continues from step 6.
3b. There is a 'test.jxuc' file in the current directory. Engine sets
up a test context. Use case continues.
4a. If a property named candidateFiles is null, a single JXTestCase instance is created
"representing" a found 'test.jxu' file.
4b. If a property named candidateFiles is not null,
a new temporary suite is created, and for each data file found, a
new JXTestCase instance is created and added to
that suite. The type of a suite will be a TestSuite in case a property named active is "false", otherwise an ActiveTestSuite will be constructed.
6a. There are no more unvisited subdirectories in the current
directory. If the current folder is a root folder (from where
execution was started) the use case ends, otherwise the engine
moves up one level in a folder tree, and use case continues by
repeating step 6.
Use Case
"Setup a Test Context"
Primary Actor: A JXTestCase class
Secondary Actor: Quick framework
Scope: JXUnit framework
Level: Subfunction
Preconditions: 'test.jxuc' file has been found in the same
directory where a 'test.jxu' file exists.
Minimal and Success Guarantees: The 'test.jxuc' file is
parsed and evaluated successfully. All properties constituting a
test context are setup properly. In case of a fatal error a
suitable error is thrown.
Main Success Scenario:
1. The engine delegates the work of loading and parsing of a
'test.jxuc' file to a Quick framework.
2. The Quick framework constructs an object tree, having an
instance of the JXConfig class as a root, and handles back
that tree to the JXUnit's engine.
3. The engine evaluates the returned result, calling an eval() method of the JXConfig instance.
Extensions:
* In case of a fatal JVM error (like out of memory) the use case
fails.
Use Case "Run
a Test Suite"
Primary Actor: JUnit framework's TestRunner
Secondary Actor: A TestSuite instance
Scope: JXUnit framework
Level: Subfunction
Preconditions: None
Minimal and Success Guarantees: All tests are run and the
results of execution are presented to the developer. In case of a
fatal error a suitable error is thrown.
Trigger: The JUnit framework's TestRunner calls the run() method of the TestSuite instance.
Main Success Scenario:
1. The JUnit framework's TestRunner calls the run()
method of the TestSuite instance.
2. The suite schedules the next junit.framework.Test instance for run.
3. The suite runs the scheduled junit.framework.Test instance (calls its run() method).
....Steps 2-3 are repeated until no further unprocessed instances
are left in a suite.
Extensions:
* In case of a fatal JVM error (like out of memory) the use case
fails.
3a. If an instance is a suite, the use case recursively calls
itself.
Use Case "Run
a Scheduled junit.framework.Test Instance"
Primary Actor: A TestSuite instance
Secondary Actor: A junit.framework.Test instance
Scope: JXUnit framework
Level: Subfunction
Preconditions: The type of the scheduled instance is JXTestCase.
Minimal and Success Guarantees: A 'test.jxu' file has been
parsed and executed successfully. In case of a fatal error a
suitable error is thrown.
Trigger: The TestSuite calls a run()
method of the scheduled instance.
Main Success Scenario:
1. The TestSuite calls a run() method of the scheduled instance.
2. The JXTestCase instance (engine) delegates the
work of loading and parsing of a 'test.jxu' file to a Quick
framework.
3. The Quick framework constructs an object tree, having an
instance of the JXDo class as a root, and handles back
that tree to the JXUnit's engine.
4. The engine evaluates the returned result, calling an eval() method of the JXDo instance.
Extensions:
* In case of a fatal JVM error (like out of memory) the use case
fails.
It is apparent from the elaboration given above, that JXConfig plays a role of a container for JXUC commands, and JXDo for JXU ones, respectively. Both classes are standard composites (see the Composite design pattern [7]). From another angle, JXConfig is an implementation of a jxuc command, and JXDo of a jxu command. Both of these commands are defined as roots in the matching QJML files ('jxuc.qjml' and 'jxu.qjml', correspondingly). As you can see, during execution, for example, of a 'test.jxu' file, a data binding process has a central role (for a good preamble about this process consult [8]). Data found in a 'test.jxu' file is mapped to an instance of a JXDo class using the appropriate binding schema 'jxu.qiml' (QIML files are "compiled" QJML files, suitable for run-time interpretation, although, they are not so readable for humans).
The following section enlists built-in properties, which can be used in JXU files, and/or Java implementations of test steps. The first part of the table (colored with yellow) stands for properties defined for every test case, the other part (colored with light blue) for those, which are set by a test context (more precisely, it lists the effects of executing JXUC commands like directoryScan, includeFiles and excludeFiles). The effects, i.e. properties set by JXUC commands only used internally by a JXUnit's engine, will not be listed here (these will be thoroughly described in the future articles).
Property Name |
Property Value |
|---|---|
| / | the value of a standard Java system property File.separator |
| . | an absolute pathname of a directory from which jxtest.bat has been started |
| testDirectory | an absolute path name of a directory where 'test.jxu' file is situated |
| dataFileName | the name of a ferreted out data file |
| absDataFileName | an absolute pathname of a ferreted out data file |
JXUnit framework employs an extremely useful principle called self testing, as an assurance for its correct functioning. This principle is probably a heritage from a JUnit framework. The story is exceedingly straightforward. JXUnit has a set of unit tests "written" in JXUnit, in the same way, as JUnit has a set of unit test "written" in JUnit. All these unit tests are located beneath the 'src\net\sourceforge\jxunit' folder (see various subfolders whose name begins with 'test'). By running them you can quickly gain confidence that everything, hopefully, behaves as anticipated.
A unit test is a valuable asset not only for testing the software, but for documenting it, as well. I strongly encourage you to analyze the unit tests for JXUnit, as they demonstrate how to use a variety of JXU and JXUC commands.
An astute user of JXUnit had probably noticed that no unit test exists for the JXU command called arg. If there had been one, the arg command would function without problems. Unfortunately, it contains a bug preventing its usage. In the next part, I will show you a route how to fix this bug, and in the same time, show you how to extend the framework. As a good exercise, try to write a unit test for this command, and afterwards to correct the bug (this can be done either by modifying, for example, the JXTestArg class (see 'src\net\sourceforge\jxunit\JXTestArg.java' file), or either by extending the framework without touching the original code base). As a corollary of this discussion, always write unit tests for your code, or better, write unit test(s) even before you begin the implementation of your class(es).
This example is primarily targeted for illustrating contemplation about JXUnit framework, up so far.
Please, go to the 'src\net\sourceforge\jxunit\testconfig' folder. Here you can find a complete example how to work with various data files, i.e. parameterize your test case. By running this example, you can observe the object tree produced representing a whole test suite in a SWING based TestRunner (select a 'Test Hierarchy' tab). For each data file found a separate instance of a JXTestCase class has been created, and named accordingly (the name equals to an absolute pathname of a corresponding data file). Also, each instance has a set of properties prepared, as stated in the table given above.
Here is a listing of a 'test.jxu' file.
<jxu>
<loop count="10" reportLabel="BenchMark">
<set name="data" file="absDataFileName" indirect="true"/>
<ifEqual converse="true" name="data" value="dataFileName" indirect="true">
<save name="data" file="badFileName_.txt"/>
<fail>Each test file must contain only its own name!</fail>
</ifEqual>
</loop>
</jxu>
The above test case assures that each data file contains only its own name. For example, data file 'a.txt' would contain a single line of text 'a.txt', though (of course, apostrophes should be omitted). Let us analyze line by line what is happening here (do not concentrate on black colored lines, as they are not essential for our demonstrational purposes).
The blue line shows you a place where a property named data is set to hold the content of the corresponding data file, whose absolute pathname has been given through a built-in property called absDataFileName. The indirect attribute (when set to "true") signifies a notion, that the file attribute specifies a name of a property, instead of holding a direct value.
The green line realizes an 'if not equal' statement (since, converse attribute is set to "true"). Here a value of a data property is compared against the value of another built-in property named dataFileName. If they are equal everything is OK, otherwise a red colored section will be executed.
Inside an "if" block, a value of a property named data is saved into a textual file called 'badFileName_txt'. This file will be created in the same directory where a 'test.jxu' file is placed. The fail command will signalize a failure, with a decent message printed on output.
A test context for this test case is defined as follows:
<jxuc
jxuSchema="classpath:///net/sourceforge/jxunit/jxu.qiml">
<directoryScan dir="testDirectory" active="true">
<includeFiles regexp=".txt$"/>
<includeFiles regexp=".xml$"/>
</directoryScan>
</jxuc>
Again, concentrate only on blue lines. Here a specification is given for collecting all data files situated in a 'testDirectory' folder, whose extensions are '.txt' or '.xml'. The directory is relative to the one where a 'test.jxu' file is located. The active attribute, when set to "true", means that each instance of a JXTestCase class will be run concurrently in a separate Java thread. This feature is particularly useful for load testing.
As this example proves, there is nothing complicated in the whole framework (so far :)). Of course, in the next installments we will penetrate more deeply into the JXUnit framework, extensively touching the QJML stuff, as well.
Here is a brief bulleted list of things to expect in upcoming articles:
I can not promise exact timings of future articles, but I'll do my best to write them, as soon as, possible. Till then stay tuned...