Nice things about autotest:

  • every test gets run in its own little scratch directory
  • we can poke around in these scratch directories and re-run command by hand when debugging
  • verbose every-command-executed-and-its-output logging
  • can easily run commands, give stdin, get stdout, get stderr, get exit code, do basic unixy things like mkdir, run commands in subdirs, etc.

Things we want:

  • tests have names. none of this weird numbers in output, filenames on disk, need tools to convert between them.
  • proper process handling. start a netsync server, keep a handle on the process, kill it when test is done (whether success or failure)
  • better expectation stuff. "the thing I expected happened (which is good)" = pass, "the thing I expected happened (which is bad)" = known bug, "something weird happened" = user needs to look. autotest collapses the last two, which leads to testrot. be nice if there were better ways to record what desired behavior ir -- "I think this value will be equal to that one, but I'd rather it was equal to this third one"?
  • saner handling of data files -- maybe each test should have its own directory, and when run it can grab data files out of that directory? AT_DATA works, but is really suboptimal.
  • better test filtering -- I want to be able to say "run all tests with expected failures only", for instance. (Though autotest's current functionality here isn't too bad -- by number or by keyword.)
  • reasonable speed on windows. NB: fork() is slow on windows, so shell is terrible here.
  • no 9 megabyte files in our source packages.
Quick Links:    -     Downloads    -     Documentation    -     Wiki    -     Code Forge    -     Build Status