Create a comprehensive suite of browser tests

[limi]

Wichert wrote:

For some reason I can't think this has not come up at the summit: I
think it would be extremely useful to have more extensive browsers
tests for Plone. We have seen several problems in Plone 3 that should
have been caught but haven't since we do not have tests for them. I am
thinking of things like creating a collection and some content and test
if the content shows up in the collection in the right order.

[ree]

Proposed roadmap:

1. Extend the kss.demo testrunner to handle application level tests (ree, and I hope to get help or advice from gotcha)

2. Write a howto that explains how to create Selenium tests, and how to run them (ree)

Deadline for 1. and 2.: 2008 March 15.

3. Create application level Selenium tests for Plone. A community effort is needed, we hope to proceed on this during the Sorrento sprint (end of March)

4. Fix and extend the kissbot to enable nightly testrunning on various browsers. (gotcha (?), ree, ...)

5. Organize a number of buildbot slave machines. We need donations from the community. Any computer that has one of the target browser installed, is switched on for the night and accessible from the net can be configured as a slave. It is not clear how many slaves we need exactly, but we probably as much as we can have to assure a reasonable speed.

3, 4, 5 can go on continously and we will set up a reasonable deadline as soon as we see how much resources we have at hand.

[grahamperrin]

I am interested and can probably set aside some G3 PowerPC computers for slave work.

Vintage is  1999-2001 or thereabouts and as a guide to capabilities, most will run Ubuntu or Kubuntu alternate, 6.06 or later. If you think that these old computers will make suitable slaves please let me know ASAP — they're otherwise scheduled for disposal.

[PythonHack]

I may have a couple win XP machines that aren't busy at night. They have FF and IE (at least one likely has IE6 still)

[ree]

Some followup:

I am in a delay comparing the original deadline set for 1. and 2. The new objective is that I prepare 1. (kss.demo testrunner capable of running Plone testsuite) before arriving to the Sorrento sprint.

During the sprint we will write tests as planned originally. We will also write 2. (the howto documentation) together, on the sprint. The objective by the end of the sprint is to have a number of tests that are runnable with a single click, and that developers gain experience with its usage and test creation.

Thanks for everyone who has offered buildbot slaves. We will definitely need even more, but we won't plan to proceed on that points during April. About further questions of technical requirements against such servers, I will ask Godefroid to comment, since he is much more competent with the buildbot issues.

[ree]

Update of progress.

In Sorrento, we succesfully finished 1. and 2.

Beppe Zizza has written a startup documentation that can be found here:

 http://kssproject.org/docs/tutorial/kss-testing-how-to-for-plone-3.1

We will, naturally, need to improve and extend it continously.

We have not written as many tests as we wanted. So the main focus for the upcoming period is point 3: create more application level tests.

It would be also nice to revive the kissbot during some of the upcoming sprints (Godefroid?).

Funittest is also an interesting alternative to recording tests with the Selenium IDE. The attendants on the sprint agreed that it can compliment prerecorded tests, once it becomes more mature and supported on all platforms. So this is also something to consider for the future.

The full sprint report can be read here:

 http://kssproject.org/blog/archive/2008/04/08/plone-app-kss-sprint-in-sorrento

[ree]

A brief progress report:

• Writing more selenium tests. There has been no progress since the Sorrento sprint.

This would be a high priority to proceed with.

• Kissbot, needed for automated nightly tests. I am not aware of any progress here either.

I personally am not so familiar with this technology, so I don't know what steps are needed to fix it or improve it. Dealing with this on a sprint would be desirable.

• Some discussion on has started on the product-developers mailing list recently

about improving the testing infrastructure or develop an entirely new way for doing it. We have to wait for the results and consequences of this, but independently of the outcome we will keep all tests created in plone.app.kss functional and convertable to a possible new testrunner system if needed.

The most important focus in the next period, should be, to increase the number of selenium tests, made by whichever method. We experienced that selenium itself is quite a challenge, and this challenge remains the same with kss.demo, funittest or any similar method that uses Selenium IDE or RC.

Please feel free to comment on this, in case you you have additional information or would like to join the efforts.