I have some suggestion about DOH. Overall I think its a great tool, and I think its asynchronous testing support is fabulous. But honestly, I just don't find it that intuitive to get up and running quickly, and I think some simple changes could fix that.
1. Lets make doh a first class package. Pull it out of util and make it just as important as dojox and dijit. Don't you think it is? And by not burying it, I think we could encourage its use.
2. Runner.html: Having the DOH Runner default to running all the tests in dojo is distracting. How about getting rid of:
var _tm = ( (dojo.global.testModule && dojo.global.testModule.length) ? dojo.global.testModule : "dojo.tests.module");
and instead, if there is no defined module, and then there should be a text box available to type it in and then run the tests. A nice, user friendly dialog that pops up in the case of no testModule and gives a little tutorial on how to register a module path (and perhaps offers a widget to allow you to do it on the fly), and specify your test module would be great!
3. I think synchronous and aysnchronous tests are fundamentally different in their structure. mashing them together by returning a doj.Deferred is not intuitive, and seems almost too magical. I think if we explicity stated that a test was an asynchronous test it would force the end user to think through how to write the test. How about runTest() is for synchronous tests and runAsyncTest() is for asynchronous ones?
4. The doh.Deferred.getTestCallback mechanism is great. Its really an implementation of a template pattern, right? But I think its use and name are confusing. And I think there is too much room for error.
So to tie it altogether, how about something like?
name:"testSomethingAsynchronous",
runAsyncTest:function() {
var def = doSomethingDeferred();
return def;
},
callbackAssertions:function(args) {
doh.assertEqual(args.length, 1);
}
errbackAssertions:function(args) {
}
}
The idea here is that if you put your assertions about the result of your deferred in the a second method.
If you expect an error, than you could implement errbackAssertions, but if it is not implemented, an
error would be considered a test failure.
In the background, doh would:
1. create a doh.Deferred
2. run runAsyncTest and get back its deferred object (if the return value was anything but a Deferred, it would be an error condition)
3. run realDeferred.addCallback(doh.Deferred.getTestCallback(callbackAssertions));
you get the idea. The problem here is that I know you want to make DOH independent of dojo. But couldn't you still do that? runAsyncTest could return a doh.Deferred or a dojo.Deferred. So things would be a little more complicated in that case for the test writer. They would have to create a doh.Deferred. They would have to then guarantee that this new deferred object got its callback/errback called form their aynschronous method when it was done. But that is certainly no worse than now, and I think it still makes the tests a lot cleaner without a lot of glue code.
So I would love to have your thoughts on this. If this is a lively thread and we can reach some conclusions about this, I would be happy to help work on this. I just want to see if people agree that this is, in fact, an improvement, and flesh out the ideas a bit.
Thanks in advance!
-maulin
http://blog.medryx.org
A Custom TestFixture
I thought about it some more, and realized I could get a lot of my needs met by creating a TestFixture class that is instantiated to create each test. I've written an entry about it on my blog, so I won't repeat the whole thing here...
http://blog.medryx.org/2008/06/09/simplified-dojo-doh-testing/
Some good ideas
I agree, doh is a good tool but needs just a little iteration to be great.
I'm strongly in favor of removing the default behavior that runs the entire dojo test suite. And we could certainly do more to make it easier to get people up and running writing tests for their own code - I like your suggestion there.
I've also been uncomfortable with the strange action from a distance that seems to take place when you write an async test. And it can be bewildering trying to analyze the output from a mixed bag of tests where some are failing, but not in the order you'd expect. I'm not sure what the right anwer is here, but we do have to be careful about changing the api too much in a point release. It might be something we can accomplish with some simple convention and best practice documentation.
On the test fixtures, many tests are written to be executed in sequence, and make un-documented assumptions about state changes supposedly completed by previous tests. That's not good - it makes it hard to run a test in isolation (say, if its failing), and hard to track down the real cause of a test failure in module with many tests. OTOH, resetting e.g. the DOM in preparation for each test seems overkill and will eventually bring the entire suite to its knees. Again, maybe we just need some more peer review of the tests to ensure they assert any prerequisite state. It would be nice to be able to distinguish an actual test failure, from a state discrepancy probably a result of a previous test, or of a test bug.
To that end, I like your TestFixture class, but I find I need something similar for a test group. A simple way to (re)set state in preparation for a series of related tests. A way to skip a group if some condition is not met. Maybe its there already, maybe its just a tutorial on test-writing best practices I need, maybe doh could help more.
FWIW I support D.O.H.
FWIW I support D.O.H. enhancements, and can help to get it moving. Better late than never.