How to write testcases for UxU? UxU - UnitTest.XUL

• Basic rules of testcases
• Simple testcases (unit-tests)
  • Compatibility for MozUnit(MozLab)'s testcases
• Complex testcases (functional-tests) with deferred operations
  • *Note: before specifying functions or generators for "yield" keyword
• How to assert the number of processed assertions?
• How to run testcases in clean or ready-made profile?
• How to write data-driven tests?
• Skipping of tests by conditions
  • Priorities
  • Target applications
  • Other conditions
• Mapping for URIs

• UxU Home
• Testing Greasemonkey scripts by UxU
• Helper methods for testcases
• How to write testcases with mocks?
• How to control UxU from remote?
• Command line options

Basic rules of testcases

UxU recognizes and runs JavaScript files written in some rules, as testcases.

Testcase files should be named like "<TEST'S NAME>.test.js". When UxU collects testcases from specified folders recursively, only "*.test.js" files will be done if there are both "*.js" and "*.test.js" files. (However, when there are only "*.js" files, UxU tries to parse all of them as testcase.)

In the execution context of codes in testcases, many helper methods introduced by UxU are available. See the list of helper methods.

You can see and try actual testcases written for UxU. There are some testcases which test UxU itself, in the source code tree. Let's checkout UxU's Subversion repository and run tests in tests/uxu/.

Simple testcases (unit-tests)

This is an example. Each file becomes one "testcase".

var description = 'Description of this testcase';

function setUp() {
  // This function is always processed before each test.
  // (ex. creating instances of the class now tested, etc.)
}

function tearDown() {
  // This function is always processed after each test.
  // (ex. destroying instances, etc.)
}

function startUp()
{
  // This function is processed only once before tests.
  // (ex. loading the class you want to test now, etc.)
}

function shutDown()
{
  // This function is processed only once after all tests finish.
}

testWillSuccess.description = 'Successful test';
testWillSuccess.priority    = 'normal';
function testWillSuccess() {
  assert.equals(0, [].length);
  assert.notEquals(10, ''.length);
  assert.isTrue(true);
  assert.isFalse(false);
  assert.isDefined(assert);
  assert.isUndefined(void(0));
  assert.isNull(null);
  assert.raises('TypeError', (function() { null.property = true; }), this);
  assert.matches(/patterns?/, 'pattern');
}

testWillFail.description = 'Failure test';
testWillFail.priority    = 'low';
function testWillFail() {
  assert.isTrue(false);
}

In the global namespace, some functions which have one of special names are recognized as "testcase function". Functions can be recognized not only by names, but by their property.

They are processed in the order like this:

1. codes written on the global namespace
2. startUp (if exists)
3. setUp (if exists)
4. the first test
5. tearDown (if exists)
6. setUp (if exists)
7. the second test
8. tearDown (if exists)
9. ...snip...
10. setUp (if exists)
11. the Nth (final) test
12. tearDown (if exists)
13. shutDown (if exists)

However it is not assured that tests are done in their defined order. The order is possibly shuffled.

The name of the testcase itself or descriptions can be supplied by a global variable, description. It will be shown in the test runner GUI.

*On UxU 0.7.x and older versions, startUp/shutDown was called warmUp/coolDown. For backward compatibility, UxU 0.8.0 and later recognize warmUp/coolDown just same as startUp/shutDown.

Compatibility for MozUnit(MozLab)'s testcases

Testcases written for MozUnit (a part of MozLab) are also available.

Differently from MozUnit, UxU ignores runStrategy option. You don't have to use the option anymore because UxU is designed to run asynchronous tests by default. (But, for compatibility, you can get and call the continuation function as the argument in setUp functions.)

Complex testcases (functional-tests) with deferred operations

Tests written for UxU can do deferred operations easily by utils.wait() or yield. (utils.wait() is available on Firefox 3/Thunderbid 3 and later. You should use it if you write tests only for Firefox 3/Thunderbird 3 and later. When you wish to support Firefox 2/Thunderbird 2, use yield instead.)

When a utils.wait() or an yield appear in tests, setUp, tearDown, startUp, or shutDown, then UxU suspends the test. And, after some conditions are met, he resumes from the next statement of the utils.wait() (or yield). The condition for resuming is changed by the type of the specified value.

var beforeTime = (new Date()).getTime();
utils.wait(1000); // Wait one second (equals to 1000 milliseconds) and resume.
// yield 1000; // Wait one second (equals to 1000 milliseconds) and resume.
var afterTime = (new Date()).getTime();
assert.isTrue((afterTime - beforeTime) > 500);
assert.isTrue((afterTime - beforeTime) < 1500);

var browser;
functionalTest.tests = {
  setUp : function() {
    var loaded = { value : false };
    browser = window.openDialog(
            'chrome://browser/content/browser.xul');
    browser.addEventListener('load', function() {
        loaded.value = true;
    }, false);
    // Wait until the flag is set to true.
    utils.wait(loaded);
    // yield loaded;
    browser.gFindBar.openFindBar();
  },
  tearDown : function() {
    browser.close();
  },
  ...
};

var win = utils.getTestWindw();
var url = win.content.location.href;
// Wait that the page is correctly loaded.
var f = function() {
        return url != win.content.location.href
    };
utils.wait(f);
// yield f;
assert.equals(uri_to_be_redirected,
              win.content.location.href);

// Wait and go
function assert_page_load(aURI) {
  yield utils.loadURI(aURI);
  var win = utils.getTestWindow();
  assert.equals(aURI, win.content.location.href);
}

function assert_window_close() {
  var win = utils.getTestWindow();
  win.close();
  yield 500;
  assert.isTrue(win.closed);
}

utils.wait(assert_page_load('http://www.clear-code.com/'));
// yield assert_page_load('http://www.clear-code.com/');
utils.wait(assert_window_close());
// yield assert_window_close();

function assert_window_close() {
  ...
}

yield assert_window_close; // without "()"

function myDeferredFunc()
{
  var d = new Deferred();
  window.setTimeout(function() {
    ...
    d.call();
  }, 100);
  return d.next(function() { ... })
          .error(function() { ... });
}

utils.wait(myDeferredFunc());
// yield myDeferredFunc();

window.setTimeout(function() {
  MyComponent.autoSubmit();
}, 100);

var result = utils.wait('submit', content.document,
                        'unload', content.document);
assert.equals('submit', result.event.type);

utils.wait({ type    : 'keypress',
             keyCode : Ci.nsIDOMKeyEvent.DOM_VK_RETURN,
             shiftKey : true },
           $('input'),
           { type    : 'keypress',
             keyCode : Ci.nsIDOMKeyEvent.DOM_VK_ESCAPE,
             capturing : true }, // observe the event in the capturing phase
           $('input') );

*Note: before specifying functions or generators for "yield" keyword

This is a note for yield. You don't have to mind it if you use utils.wait() instead.

yield possibly breaks the stack trace when an exception raises from inside of the function handed to yield.

// ex. test of mapping
function assertRedirect(aOriginalURI, aToBeRedirectedTo) {
  yield utils.loadURI(aOriginalURI);
  assert.equals(aToBeRedirectedTo, browser.currentURI.spec);
}
yield assertRedirect('http://myhost/page1',
                     'http://myhost/page1_redirected');
yield assertRedirect('http://myhost/page2',
                     'http://myhost/page2_redirected');
yield assertRedirect('http://myhost/page3',
                     'http://myhost/page3_redirected');

For example, the test above process three assertions, but, even if one of them failed, we cannot know which one failed because the stack-trace ends at the line assert.equals() exists.

A special helper function, Do() solves this problem. Wraping returned value of yield keyword by the function always creates a new stack, so you can detect the line which the exception really raised from.

yield Do(assertRedirect('http://myhost/page1',
                     'http://myhost/page1_redirected'));
yield Do(assertRedirect('http://myhost/page2',
                     'http://myhost/page2_redirected'));
yield Do(assertRedirect('http://myhost/page3',
                     'http://myhost/page3_redirected'));

By the way, Do() does nothing and returns the handed object as-is if it is not a function, generator, or generator-iterator. It solves the problem without any harmful effect. You should write Do() for every yield in your testcases.

How to assert the number of processed assertions?

You can specify how many assertions should be processed for each test, by assertions property.

testRoop.assertions = 10;
function testRoop() {
  for (var i = 0, maxi = data.length; i < maxi; i++)
  {
    assert.equals(expected[i], data[i], i);
  }
}

UxU will report the result as "failed", if the total number of processed assertions is different from the number specified by assertions property, even if there is no failed assertion. This feature will be useful to prevent unexpectedly successes of tests which are in developing. For example, a test will fail and you can realize that you've forgotten to remove comments including assertions, if you specify the number of assertions.

Verification of the number of processed assertions works for assertions written in each test function. Assertions in setUp/tearDown are ignored.

Even if you don't specify the number of assertions, UxU regards them as "success" but reports an warning, for tests which have no processed assertion.

minAssertions, maxAssertions

Additionally, there are two more features, minAssertions property (specifies minimum number of assertions which should succeed) and maxAssertions property (specifies maximum number of assertions which should succeed). They are available like as assertions, and you can use them in same time.

How to run testcases in clean or ready-made profile?

You can run the testcase with a Firefox (or Thunderbird) profile which is made for testing, not only the current profile.

var profile = '../profiles/test-profile/';

function setUp() {
...

The path to the profile for the testcase is supplied by a global variable, profile. If profile is defined and the folder is available, then UxU automatically starts child process of Firefox (or Thunderbird) with the profile and run the testcase in the environment.

The folder will be cloned as a temporary profile, so, the original files are not changed.

var profile = '../profiles/test-profile/';
var application = 'C:\\Program Files\\Mozilla Thunderbird\\thunderbird.exe';

...

If you speficy the path to the executable by a global variable application, UxU starts the specified application with the profile. (*However, application is ignored if you don't define profile.)

How to write data-driven tests?

UxU supports data-driven tests. If you want to do a test for various patterns, you can specify parameters for the test by parameters property, as an array or a hash.

For example, you maybe write a test for a feature which works about webpages, like:

function testMyFunction() {
  utils.wait(utils.loadURI('http://...'));
  assert.equals('result1', myFunc(utils.content));

  utils.wait(utils.loadURI('http://...'));
  assert.equals('result2', myFunc(utils.content));

  utils.wait(utils.loadURI('http://...'));
  assert.equals('result3', myFunc(utils.content));

  ...
}

However, this test will annoy you. You have to copy and paste same codes for new patterns, so, the test will become too long. If the third assertion fails, then all following assertions will be skipped. OK, you'll correct a bug, and redo the test. Then, the fourth assertion fails and followings are skipped again. You have to fix a bug and do the test, again and again. That is too annoying!

You should write those tests in data-driven style. See:

testMyFunction.parameters = [
  { uri: 'http://...', expected: 'result1' },
  { uri: 'http://...', expected: 'result2' },
  { uri: 'http://...', expected: 'result3' },
  ...
];
function testMyFunction(aParameter) {
  utils.wait(utils.loadURI(aParameter.uri));
  assert.equals(aParameter.expected, myFunc(utils.content));
}

If you specify parameters for the test (test function's parameters property) as an array, then, UxU runs the test function for each items of the array. The parameter will be handed as the first argument of the function. (If you specified private setUp or tearDown for the test, they also receive parameter same to the test function itself.) UxU runs the test each time as stand-alone test, with unique name like "testMyFunction (1)", "testMyFunction (2)", "testMyFunction (3)", etc. All tests will be done even if one of them failed.

If you specify parameters as a hash, UxU uses each item of the hash as parameters.

testMyFunction.parameters = {
  google: { uri: 'http://...', expected: 'result1' },
  yahoo:  { uri: 'http://...', expected: 'result2' },
  mixi:   { uri: 'http://...', expected: 'result3' },
  ...
};
function testMyFunction(aParameter) {
  utils.wait(utils.loadURI(aParameter.uri));
  assert.equals(aParameter.expected, myFunc(utils.content));
}

In this case, each test will be named with keys of the hash like "testMyFunction (google)", "testMyFunction (yahoo)", "testMyFunction (mixi)", etc, so you'll easily be able to find out the pattern which causes problem.

To assist your development of data-driven tests, there is a useful helper function which generates hash or array from external CSV files. You can write and maintain testcases which include many many patterns, by OpenOffice.org Calc, Microsoft Excel, etc.

Skipping of tests by conditions

Priorities

You can specify priority of each test, by priority property. For example:

testBasic.priority = 'must';
function testBasic() {
  // This is the test for basic features.
  // They are very important, so must be tested anytime.
  assert.equals(3, calcService.add(1, 2));
  ...
}

testMinor.priority = 'low';
function testMinor() {
  // This is the test for a minor feature.
  // We don't need to test the feature frequently.
  ...
}

testUnderConstruction.priority = 'never';
function testUnderConstruction() {
  // This test always fails because the feature is under construction.
  // We cannot clearly know that other tests succeed or failed.
  // So we should turn off this test before the feature is implemented.
  ...
}

UxU is designed and configured for test-driven development by default. In this development style, you'll run tests again and again. Tests which succeed in the last time often success in the next, so they don't have to be processed every time. So, UxU often skips tests which succeed in the last time, but always runs newly added or failed test anyway. By this feature, each testing-time shortens and you can run tests many times without stresses.

Test's priority is specified in a number, or a string listed below:

By the way, you can specify global priority of the test case by the global variable proprity. UxU applies global priority for each test which has no personal priority.

var priority = 'must';

testFoo.priority = 'never';
function testFoo() { ... } // this will be skipped.

function testBar() { ... } // this will be performed.

Target applications

You can forbid to run Firefox-specific testcases on Thunderbird.

var targetProduct = 'Thunderbird';

function setUp() {
...

If you specify the application name by a global variable targetProduct, UxU skips to run the testcase if the current application isn't it. This feature is useful for addons which works on both Firefox and Thunderbird.

Additionally, if you specify profile in same time, UxU runs the testcase on the application automatically.

You can specify the target application for each test by their targetProduct property. This is useful if your extension works on both Firefox and Thunderbird, and you have to write application-specific tests.

testForFirefox.targetProduct = 'Firefox';
function testForFirefoxSpecificFeature() { ... }

testMinor.targetProduct = 'Thunderbird';
function testForThunderbirdSpecificFeature() { ... }

function testForAllApplication() { ... }

Moreover, you can specify the target application for whole of the test case by the global variable targetProduct. If you specify it, UxU determines that he should skip the test case or not at the startup process. Then, tests which have other applications as their targetProduct will be skipped.

var targetProduct = 'Thunderbird';

testA.targetProduct = 'Firefox';
function testA() { ... } // this will be skipped.

function testB() { ... } // this will be performed.

If you specify both global targetProduct and the profile, UxU automatically runs the test case on the application with the profile.

Other conditions

You can skip each test by other conditions, specifying shouldSkip property of each tests. If you specify a function to shouldSkip property, it will be evaluated just before the test is performed (before setUp() is performed) and UxU skips the test if the returned value is false. Other cases, UxU evaluates the value of shouldSkip property as a boolean.

testA.shouldSkip = true;
function testA() { /* skip this, because this is under construction. */ }

// don't run after 2009-02-09
testB.shouldSkip = function() {
  return Date.now() > (new Date('2009/2/9')).getTime();
};
function testB() { ... }

If you wish to skip whole of the test case, specify the global variable shouldSkip. When the value (or returned value) of shouldSkip is true, UxU completely ignores all of contents of the test case.

var shoudSkip = true;

// they are always skipped!
testA.priority = 'must';
function testA() { ... }
function testB() { ... }

Mapping for URIs

*On UxU 0.8.x, you can use a local HTTP server as mock. It possibly more useful than this simple mapping feature.

For testing of a module which is designed to load resources from fixed URIs, UxU provides an simple mapping feature. For example, you can hand contents of a local file "012345.xml" to your module by loading an URI "http://myservice.example.com/myapi?userid=012345".

var mapping = {
      '*.google.*/*' : baseURL+'../fixtures/google-pages.html',
      '*.yahoo.*/*'  : baseURL+'../fixtures/yahoo-pages.html',
      'https://addons.mozilla.org/ja/firefox/addon/*'
                     : baseURL+'../fixtures/addons/$1.html'
    };

function setUp() { ... }
function tearDown() { ... }
function test_loadInBackground() {
  myModule.loadURL('https://addons.mozilla.org/ja/firefox/addon/6357');
  utils.wait(function() { return myModule.loaded; });
  assert.equals('summary of the loaded document', myModule.getSummary());
}

Like above, mapping rules for the testcase is supplied by a global variable, mapping. If mapping is defined, then any request (including subframes, XMLHttpRequest, and so on) for URIs matching to the rules will be mapped.

UxU's mapping works like replacing of contents by a proxy. In other words, UxU works as a local proxy. When your requesting is mapped, you will get the contents of the document from the URI after mapping, however the response URI will be still the original URI.

var mapping = {
      'http://release.example.com/'  : baseURL+'../fixtures/home.html',
      'http://release.example.com/*' : 'http://test.example.com/$1',
      'http://www??.example.com/*'   : 'http://test.example.com/server$1$2/$3'
    };

var mapping = [
      'http://release.example.com/'      , baseURL+'../fixtures/home.html',
      'http://release.example.com/*'     , 'http://test.example.com/$1',
      /^http://www(\d+).example.com/(.*)', 'http://test.example.com/server$1/$2'
    ];

var mapping = function(aOriginalURI) {
      return aOriginalURI.spec.indexOf('google') > -1 ?
               baseURL+'../fixtures/google-pages.html' :
               null ;
    };

*In old versions of UxU, redirect was used for the feature, instead of mapping. redirect is still available on UxU 0.7.6 and later for backward compatibility.

Additionally, mock features of local HTTP servers will help you to write complex test about applications working with web services.