Testing an open source project has some advantages compared to proprietary softwares:
- everything is public and can be shared with anyone interested
- you can communicate about your project, issues and features with no fear
- you can use tools that are free only for open source projects (like Atlassian suite, Saucelabs…)
- you may be helped by contributors and benefit from the community of open source lovers
OpenPaaS is an open source software involving several teams of developers and testers around the world (France, Tunisia, Vietnam). Unfortunately, Testers/Dev ratio is way too low and we always need some help to improve the quality of the product. Luckily at the end of year 2015 I saw the following tweet…
Anyone working on an open source project and needs some help with testing?
— Maria Kedemo (@mariakedemo) October 28, 2015
…and I had the good idea to respond…
— Stéphane Colson (@s_colson) October 28, 2015
As any Open Source projects, we are happy to find contributors so I started to talk about this with Maria Kedemo, which is a well known Context Driven Tester. She explains me that she was working for House of Test in Sweden and was at this moment responsible for the education of Software Testing students in Malmö for Hermods. Another class was also interested in Örebro for using OpenPaaS for their training. It’s true that it is always better to work with a real project, with real people working daily on it, and OpenPaaS was fulfilling totally to this demand. We decided that it was a good fit and that each of us could probably benefit from this collaboration. Ok then, let’s do it!
An instance of OpenPaaS has been installed with an account for each ~50 students, divided in 11 groups (domain). OpenPaaS was not really ready for provisioning automatically 50 users with their own passwords and grouped by domain, so we have to create them by directly accessing the Mongo database, which is a bit tedious but help us learn about OpenPaaS to be installed in order to be used by real users.
Communication with students and teacher was eased thanks to a slack team with several channels.
The education is a vocational training, sponsored by the Swedish government, lasting 1.5 year. Here’s the schedule:
- Introduction to test
- Think like a tester: learning to question and explore the product
- Programming: Java
- Test strategy and planning
- Different project contexts
- Test design and techniques
- Bug reporting
- Test tools (jenkins, vm-ware, fiddler…)
At the end, an internship in a company must be done by students during 7 weeks.
Testing is a lot about questioning. So the first work done with the students was to explore and discover the OpenPaaS product, and as a tester ask questions to understand the product, the environment, the customers, the team, etc. The different teams created documents with list of questions, so that I could prepare my answers. General questions, questions from hands-on experience, questions based on claims, project related questions. Then we planned a Hublin video conference and I had to answer questions like, for example:
- What are the biggest competitors OpenPaaS have?
- Anything you want us to focus extra on?
- Screenshots on Website are very different from the product we have, why? (Editor’s note: the website has been updated and is now closer to reality)
- Will it be possible to share files?
- Will OpenPaaS cost anything?
- Which are the most important Quality Criteria?
- Is there any known bugs?
- What is your deadline?
- Is the product W3-verified?
Some questions were easy to answer (“Is there any known bugs?”), some needs more talking (“What is your deadline?”). I learned with this exercise that it is not easy to talk about your project, the one you are working 100% of your work time. Also it’s hard to learn about a product only with an access to the bug tracker, sprint dashboard, a work-in-progress demo product and marketing documents. I had to answer lots of misunderstandings and some unknowns.
Consistencies in User Interface
The second exercise done to the students was to find inconsistencies in the UI of the product. Let’s be honest, there were a lot at this moment of the development. Development teams were progressing quickly but not always by taking care of the global User Interface. The students found several problems with warning and error messages. Some messages can have a template very different in a module and in another one. Some icons and some UX behaviors were not consistent between the different modules. We still have some improvement to do, but these feelings from external users are a great help to have the big picture of our product.
We sometimes have to chose between several mockups. As we are a team of developers and related craft, we sometimes have hard time to agree on the right choice. We asked the student to chose between different UI ideas. This is not really directly related to Software Testing but as a tester, you must understand the impact that can have one idea or another. They were asked to chose between different implementations of the feedback displayed when uploading files in the message composer, especially when an error occurred. Each team made a document explaining their choice. I would like to think that it’s a great exercise to take the place of the UX designer, to act as the user of the product and explain with words why a solution is better than another one. These features are now done in the product and feelings given by the students has been taken into account.
The fourth exercise suggested to students was to do some user testing with strangers that have absolutely no information about the project. I’ve also done this during a Meetup of the Agile CARA group, and also during an OpenPaaS barcamp, and it was very interesting (and depressing at the same time) to see users unable to quickly do some basic actions. Users were asked to create an event, or accept an event, and add a reminder. Some spent a long time to find the grid menu and the calendar inside it. Others couldn’t find how to create an event with the FAB (Floating Action Button from Material Design). Each user has its own experience with different kind of softwares and it’s really hard to find a way to satisfy the most of them.
Testing bug fix or user stories
We use Jira as a bug tracker and to define User Stories/Tasks/Improvements. I asked students to test some User Stories, it was part of the education Schedule, Claim testing (testing regarding what is claimed about the product, in this case, what is described in the user story). They found several issues and filed them in Jira.
They also have to do some re-testing and non-regression testing of bug fixes.
In order to find some new issues, they also did some function testing, domain testing and scenario testing. They also wanted to do some API testing but they did not have enough time to invest.
A good thing about having Swedish students testing your application is that:
- they use some encoding characters we don’t use usually in France (or with English language)
- they use smartphones which are probably not available, or at least not configured like the ones we use
- they use some Windows OS (Windows 10) which is not our preferred daily environment
- they are students and don’t have the same behavior as most of the OpenPaaS team members using a social application
To put it in a nutshell, they opened 22 new Issues, and most of them are now fixed or no more relevant. Because they were well mentored, I didn’t have to ask them some additional information, like screenshots, environment description, etc.
Unfortunately, House of Test won’t be assigned for the education of Hermods 2016–2017 that started a few months ago. If other Software Testing courses wants to use an Open Source project to train their students and read this, we would be happy to help (and be helped) and you’ll easily find how to get in touch. Same for developers, it’s always possible to contribute to an Open Source project.
This article was originally published in Linagora Engineering