What is SpinachCon?
“What the heck is a SpinachCon?” you may ask. Well, the idea is that there are free software projects that have niggling usability issues. Like pointing out to a friend that they have spinach stuck in their teeth, SpinachCon is an event where free software fans can show up and let the participating free software projects know whether or not they have ‘spinach in their teeth,’ and between what teeth that spinach might be. 🙂 It’s basically a fun and informal free software usability testing session open to anyone who would like to drop by and help out by testing software.
Hyperkitty joined three other free software projects – MediaGoblin, LibreOffice, and Inkscape – in the event.
How it worked
SpinachCon was 5 hours long, from noon until 5 PM the Friday before LibrePlanet. Each project had its own table. Ryan and I brought laptops that we set up on the Hyperkitty table, open and logged in as me to the Fedora’s Hyperkitty test server. Deb provided user questionnaire sheets that we stacked on the table, and I had also written out a set of six tasks for users to complete during their testing. I transcribed those into a text file in gedit, and we used USB sticks that Deb brought to transfer the file to the other laptop. We kept the gedit buffer open on each laptop.
Lunch was generously provided by the Open Invention Network, so we both grabbed some pizza and salad and let Deb know when we were ready for our first testers after we finished eating. We had a steady stream of testers throughout the entire length of the event.
Our per-user workflow was as follows:
- Open up the task list text file in gedit and save a copy coded with tester number. (This completely fell apart because the ability to save a gedit buffer with another file name has been taken away in Fedora rawhide. 🙁 So we ended up copy/pasting buffers from one tab to another, it was hectic and stressful and our numbers got out of sync with our testers so we can’t associate questionnaires with tester task text files.)
- Introductions with tester, ask tester to sit down at appropriate laptop.
- Ask tester if they’ve used mailman before, and explain to them a little bit about what Hyperkitty is and how it works.
- Show the tester the gedit buffer and explain it has a list of tasks for them to complete, and that we’d like them to take notes and answer the questions as they complete the test.
- Sit with the tester and watch them complete the tasks, answering any questions they have and noting any issues or comments that came up as they went along.
- Thank tester for completing test, ask them to fill out questionnaire. (We forgot to have one of our testers fill out a questionnaire. Oops!)
Ideas for Running a Smoother Test Next SpinachCon
Based on some of our goofs / mishaps this time around, here are some suggestions for the next time we go to SpinachCon – please feel free to use these to prepare your project for SpinachCon in the future!
- Bring enough Laptops! SpinachCon was held at a co-working space that didn’t have a computer lab. We didn’t realize until the week of that we’d need to bring laptops to the event for testers to use. I was hoping to use my own laptop to take notes, but it turned out to be fine to not have it to use in that capacity. We brought two laptops so we were able to run two tests at once. That worked well, since there was only two of us (me and Ryan) admining the test. I’d make sure you have one laptop per test admin so you’re not stuck waiting on a laptop to open up.
- Grab enough blank questionnaire sheets! We ran out of questionnaire sheets and weren’t sure if there were any more blank ones available. I did track Deb down and get more, but I think for a 5-hour event, if each team has at least 20 sheets, they shouldn’t run out.
- Bring plenty of paper and pens! I brought a paper notebook to take notes in and misplaced it. I ended up using the back of one of the filled out questionnaires to take some notes. Some testers brought laptops and wanted to use their own laptops, so we needed somewhere to write down the URL for them to go to. We also only had one pen at our table. Bring pens! Bring paper notebooks!
- Take photos! I took photos of our testing process, with the permission of the folks sitting at the table. (I’ve used them in this blog post.) This is a good idea to do at the event, because it makes recap blog posts like this more interesting. 🙂
- Assign numbers to testers and use to correlate data collected! If you are saving data about users on the computer and want to correlate that with your testing sheets, assign your testers a number before their test begins, and coordinate with any other test admins. How Ryan and I could have coordinated (and didn’t think to, until it was too late) was one admin gets even numbers, the other gets odd. So my testers would be #1, #3, #5, etc., and his could have been #2, #4, #6, so on and so forth. Write the tester number on the questionnaire sheet at the beginning of the test, save out the blank task question buffer in gedit as ‘test-$NUMBER.txt’ where $NUMBER is the tester’s assigned number, then hand the same sheet to the tester to fill out when they’ve completed the tasks. (A note for me: use vim, not gedit.)
- Collect data on paper rather than electronically! Can you tell I’m frustrated by the issues we ran into collecting the user’s notes in text files? If I had been smart and prepared, I would have typed out the task list into LibreOffice and printed out a stack (At least 20 copies) and had the users refer to it and write their feedback using pen and paper during the test. Then it would have been easier to correlate the task sheets with the questionnaires – just staple them, or write on them, or fold them together, etc.
- Test out your tasks before finalizing them! We realized after the first test that the wording in one of the tasks was confusing. Oops. We were able to update the gedit buffers (with great pain) on both laptops afterwards, but yeah. Test your test before you run it. Doh!
What were the tasks for Hyperkitty?
Ooh! Do you want to try at home and let us know how you did? You know, that would be quite lovely! Anyhow, here’s the list of tasks:
- You’ve got a crazy inbox and you’re really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?
- Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there’s a lot going on – people actively making discussions.
- You’ve heard about Fedora’s development list (“devel”) and decided to look into it. Who are you likely to hear from if you read that list?
- In March 2012, there was a discussion about GPT and Fedora 17 on the devel list. Can you find it?
- Find a post you like an add a tag to it so you can find it later.
- Start a new discussion on the Fedora users list (“users.”)
If you’d like to play at home, visit lists.stg.fedoraproject.org and let us know how you did in the comments!
So, what did you learn about Hyperkitty?
Our full test results, including the full text from the user’s task notes buffers as well as the user questionnaires (listed in full per user as well as across user per question), are available on the Hyperkitty trac wiki:
At a high-level, we drew a few pretty strong conclusions from the testing experience:
1. Hyperkitty’s search needs more work.
Only 1 of the 8 testers was able to find the GPT posting from March 2012 in task #4! Suggestions the users offered here included:
- listing more than 10 results per search results page
- allowing users to sort the search results
- allowing users to search only within a certain time period (e.g., only in March 2012)
- simplifying the search results listing of posts
- making the search box easier to find / more apparent
- have an ‘advanced search’ page with a lot more searching tools
- filter search results to only threads you were involved in
2. The users enjoyed Hyperkitty’s look and feel.
Multiple testers asked me about how to set up Hyperkitty for their own usage after completing the test. Hyperkitty earned the following average user ratings on the user questionnaire:
- Aesthetics: 8.4 out of 10
- Intuitive: 8.0 out of 10
- User-Friendly: 7.6 out of 10
- Professional: 8.6 out of 10
3. The left-hand nav / filters for the mailing list overview weren’t visible to users.
Only one of the 8 testers discovered the left nav on the hyperkitty list overview that would have allowed them to sort the lists by popularity (making task #2 super-easy!)
4. Users didn’t seem to notice the search box.
Only a few of the users noticed the search box. Most users, while on the devel list overview page, noticed the year/month listings in the left nav and clicked to March 2012 when completing task #4. Only a few tried to use search, and most only after browsing to March 2012 first.
5. Tagging needs more work.
Aaron, one of our testers, had a lot of interesting thoughts on making tags useful. For example, we should suggest tags already in use in the system (maybe via tag cloud, maybe via some other mechanism) to prevent users coming up with different tags for the same thing (or at least, preventing that as much as possible.) Some of the users also had a hard time finding the tags – they are associated with threads, not individual posts, so they are on the right-hand side of the thread view. Another issue that came up – some users weren’t sure if the tags were for their own personal usage or if they were viewable to everyone in the system. (The latter is the case!)
Overall we got a ton of great ideas, feedback, and information from SpinachCon, and it was a really valuable experience. Deb mentioned we may be doing another SpinachCon in the Boston area later this spring; I am totally excited to bring Hyperkitty to SpinachCon again then!
BTW, here’s a screenshot of how Hyperkitty looked to testers, for posterity / comparisons at the next SpinachCon:
Thank you the Open Invention Network for sponsoring such a great event!