Robert> I've built Maxima from current sources + GCL 2.6.12 (as built from
Robert> gcl-2.6.12.tar.gz) and I'm getting hundreds of errors in the test suite
Post by Raymond Toy
Same here, with gcl 2.6.12. (Not exactly which 2.6.12, I think I built
this myself some time ago.) Most tests don't seem to run due to an
error break somewhere. I get 1057 tests failed out of 5946 tests. I
know there are like 11000 tests, so more than half aren't run. :-(
I think this really says again that we really do need some kind of
continuous integration that prevents check-ins if the testsuite
doesn't pass. Most people (including me) don't have all the lisps
running locally, so some machine with access to these would be really
Gunter> +1 from me. And since I have managed to break plot and draw completely
Gunter> for clisp without anybody noting for weeks: An automatic test that looks
Gunter> if the diagrams gnuplot outputs contain the right kind of graph is next
Gunter> to impossible. But we could add a few tests to the testbench, perhaps,
Well, we could have the test save the plots in png (or other format),
and compare the new output with some previously saved output and note
the difference. If they're different, the submitter will need to look
at the difference and decide if it's ok and as part of the change,
update new expected results.
This is a giant pain.
Gunter> that test if a try to produce a 2d or a 3d plot results in a crash or
Gunter> fails in a similarily horrible way. Also we could perhaps test if all
Gunter> methods of communicating with gnuplot work: The error I had produced
Gunter> wasn't reproducible in wxMaxima.
Gunter> Any objections to that? This step would mean that the testbench will
Gunter> fail on computers that don't contain gnuplot so adding it might be a
Gunter> Wrong Decision. And if we want to add it: Are there any ways of
Gunter> communicating with gnuplot on any platform that are known to fail?
While gnuplot is actually pretty much separate from maxima itself, I,
for one, need gnuplot to work when I'm using maxima.
Gunter> I have just seen that https://git.launchpad.net/maxima seems to have
Gunter> started automatically importing maxima's code again => will try to set
Gunter> up an automatic build that automatically runs all tests on a daily basis
Gunter> on launchpad.
This is cool, but unless the integration is tied to Sourceforge so
that test failures prevent checkins, this doesn't really help. It has
to be automatic and enforced. People who don't check today (including
me!) likely won't check other sites to see if the commit was ok.