As part of the greater LibrePlanet 2014 festivities, Deb Nicholson organized the first SpinachCon at Industry Lab in Cambridge MA, sponsored by the Open Invention Network.
What is SpinachCon?
“What the heck is a SpinachCon?” you may ask. Well, the idea is that there are free software projects that have niggling usability issues. Like pointing out to a friend that they have spinach stuck in their teeth, SpinachCon is an event where free software fans can show up and let the participating free software projects know whether or not they have ‘spinach in their teeth,’ and between what teeth that spinach might be. 🙂 It’s basically a fun and informal free software usability testing session open to anyone who would like to drop by and help out by testing software.
Hyperkitty joined three other free software projects – MediaGoblin, LibreOffice, and Inkscape – in the event.
How it worked
SpinachCon was 5 hours long, from noon until 5 PM the Friday before LibrePlanet. Each project had its own table. Ryan and I brought laptops that we set up on the Hyperkitty table, open and logged in as me to the Fedora’s Hyperkitty test server. Deb provided user questionnaire sheets that we stacked on the table, and I had also written out a set of six tasks for users to complete during their testing. I transcribed those into a text file in gedit, and we used USB sticks that Deb brought to transfer the file to the other laptop. We kept the gedit buffer open on each laptop.
Lunch was generously provided by the Open Invention Network, so we both grabbed some pizza and salad and let Deb know when we were ready for our first testers after we finished eating. We had a steady stream of testers throughout the entire length of the event.
Our per-user workflow was as follows:
- Open up the task list text file in gedit and save a copy coded with tester number. (This completely fell apart because the ability to save a gedit buffer with another file name has been taken away in Fedora rawhide. 🙁 So we ended up copy/pasting buffers from one tab to another, it was hectic and stressful and our numbers got out of sync with our testers so we can’t associate questionnaires with tester task text files.)
- Introductions with tester, ask tester to sit down at appropriate laptop.
- Ask tester if they’ve used mailman before, and explain to them a little bit about what Hyperkitty is and how it works.
- Show the tester the gedit buffer and explain it has a list of tasks for them to complete, and that we’d like them to take notes and answer the questions as they complete the test.
- Sit with the tester and watch them complete the tasks, answering any questions they have and noting any issues or comments that came up as they went along.
- Thank tester for completing test, ask them to fill out questionnaire. (We forgot to have one of our testers fill out a questionnaire. Oops!)
Ideas for Running a Smoother Test Next SpinachCon
Based on some of our goofs / mishaps this time around, here are some suggestions for the next time we go to SpinachCon – please feel free to use these to prepare your project for SpinachCon in the future!
- Bring enough Laptops! SpinachCon was held at a co-working space that didn’t have a computer lab. We didn’t realize until the week of that we’d need to bring laptops to the event for testers to use. I was hoping to use my own laptop to take notes, but it turned out to be fine to not have it to use in that capacity. We brought two laptops so we were able to run two tests at once. That worked well, since there was only two of us (me and Ryan) admining the test. I’d make sure you have one laptop per test admin so you’re not stuck waiting on a laptop to open up.
- Grab enough blank questionnaire sheets! We ran out of questionnaire sheets and weren’t sure if there were any more blank ones available. I did track Deb down and get more, but I think for a 5-hour event, if each team has at least 20 sheets, they shouldn’t run out.
- Bring plenty of paper and pens! I brought a paper notebook to take notes in and misplaced it. I ended up using the back of one of the filled out questionnaires to take some notes. Some testers brought laptops and wanted to use their own laptops, so we needed somewhere to write down the URL for them to go to. We also only had one pen at our table. Bring pens! Bring paper notebooks!
- Take photos! I took photos of our testing process, with the permission of the folks sitting at the table. (I’ve used them in this blog post.) This is a good idea to do at the event, because it makes recap blog posts like this more interesting. 🙂
- Assign numbers to testers and use to correlate data collected! If you are saving data about users on the computer and want to correlate that with your testing sheets, assign your testers a number before their test begins, and coordinate with any other test admins. How Ryan and I could have coordinated (and didn’t think to, until it was too late) was one admin gets even numbers, the other gets odd. So my testers would be #1, #3, #5, etc., and his could have been #2, #4, #6, so on and so forth. Write the tester number on the questionnaire sheet at the beginning of the test, save out the blank task question buffer in gedit as ‘test-$NUMBER.txt’ where $NUMBER is the tester’s assigned number, then hand the same sheet to the tester to fill out when they’ve completed the tasks. (A note for me: use vim, not gedit.)
- Collect data on paper rather than electronically! Can you tell I’m frustrated by the issues we ran into collecting the user’s notes in text files? If I had been smart and prepared, I would have typed out the task list into LibreOffice and printed out a stack (At least 20 copies) and had the users refer to it and write their feedback using pen and paper during the test. Then it would have been easier to correlate the task sheets with the questionnaires – just staple them, or write on them, or fold them together, etc.
- Test out your tasks before finalizing them! We realized after the first test that the wording in one of the tasks was confusing. Oops. We were able to update the gedit buffers (with great pain) on both laptops afterwards, but yeah. Test your test before you run it. Doh!
What were the tasks for Hyperkitty?
Ooh! Do you want to try at home and let us know how you did? You know, that would be quite lovely! Anyhow, here’s the list of tasks:
- You’ve got a crazy inbox and you’re really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?
- Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there’s a lot going on – people actively making discussions.
- You’ve heard about Fedora’s development list (“devel”) and decided to look into it. Who are you likely to hear from if you read that list?
- In March 2012, there was a discussion about GPT and Fedora 17 on the devel list. Can you find it?
- Find a post you like an add a tag to it so you can find it later.
- Start a new discussion on the Fedora users list (“users.”)
If you’d like to play at home, visit lists.stg.fedoraproject.org and let us know how you did in the comments!
So, what did you learn about Hyperkitty?
Our full test results, including the full text from the user’s task notes buffers as well as the user questionnaires (listed in full per user as well as across user per question), are available on the Hyperkitty trac wiki:
At a high-level, we drew a few pretty strong conclusions from the testing experience:
1. Hyperkitty’s search needs more work.
Only 1 of the 8 testers was able to find the GPT posting from March 2012 in task #4! Suggestions the users offered here included:
- listing more than 10 results per search results page
- allowing users to sort the search results
- allowing users to search only within a certain time period (e.g., only in March 2012)
- simplifying the search results listing of posts
- making the search box easier to find / more apparent
- have an ‘advanced search’ page with a lot more searching tools
- filter search results to only threads you were involved in
2. The users enjoyed Hyperkitty’s look and feel.
Multiple testers asked me about how to set up Hyperkitty for their own usage after completing the test. Hyperkitty earned the following average user ratings on the user questionnaire:
- Aesthetics: 8.4 out of 10
- Intuitive: 8.0 out of 10
- User-Friendly: 7.6 out of 10
- Professional: 8.6 out of 10
3. The left-hand nav / filters for the mailing list overview weren’t visible to users.
Only one of the 8 testers discovered the left nav on the hyperkitty list overview that would have allowed them to sort the lists by popularity (making task #2 super-easy!)
4. Users didn’t seem to notice the search box.
Only a few of the users noticed the search box. Most users, while on the devel list overview page, noticed the year/month listings in the left nav and clicked to March 2012 when completing task #4. Only a few tried to use search, and most only after browsing to March 2012 first.
5. Tagging needs more work.
Aaron, one of our testers, had a lot of interesting thoughts on making tags useful. For example, we should suggest tags already in use in the system (maybe via tag cloud, maybe via some other mechanism) to prevent users coming up with different tags for the same thing (or at least, preventing that as much as possible.) Some of the users also had a hard time finding the tags – they are associated with threads, not individual posts, so they are on the right-hand side of the thread view. Another issue that came up – some users weren’t sure if the tags were for their own personal usage or if they were viewable to everyone in the system. (The latter is the case!)
Summary
Overall we got a ton of great ideas, feedback, and information from SpinachCon, and it was a really valuable experience. Deb mentioned we may be doing another SpinachCon in the Boston area later this spring; I am totally excited to bring Hyperkitty to SpinachCon again then!
BTW, here’s a screenshot of how Hyperkitty looked to testers, for posterity / comparisons at the next SpinachCon:
Thank you the Open Invention Network for sponsoring such a great event!
“This completely fell apart because the ability to save a gedit buffer with another file name has been taken away in Fedora 20.”
I’m a bit curious as to what you mean by this? In gedit 3.10 the ‘File / Save as…’ menu entry is there just like always. In 3.12 it’s still there (it’s under the cog menu now). Am I missing something?
Good point; I goofed. It was on Ryan’s computer (which is running Rawhide) that we ran into the issue. I mistyped F20, I should have typed Rawhide so I’ve correct it. Just give it a try in Rawhide. Seriously. Open up gedit, type some stuff out, save it as foobar.txt. Now, using the same exact buffer, save it as foobar1.txt. Just try.
If rawhide is on gnome 3.12, you can find the “Save as…” command in gedit lookin in the “gear” (not sure if it has another, proper name) menu, where most other command reside.
Hello Máirín,
First of all, thanks for this mindful blog and work on Hyperkitty.
I made the test at home, and one thing I found not easy to evaluate is the activity graph. All graphs are normalized to the day where there was the most activity on them, so it is difficult to compare the images to know if a list is really active or not (without looking to the total number). For example, the graphs for the list s390x (26 discussions) looks bigger than the one for s390x-builds (1396 discussions). Maybe a color scheme would allow to keep the accuracy of a normalized graph and give a clue about overall activity ? To keep with the blue theme, maybe only a light to dark blue scheme would be ok ?
Also the “hide inactive” check-box being checked by default would have prevented me to find the astronomy list if it had not be the point of a question, so I searched I bit more than I would normally do. Not sure if it can be a problem in real life usage as an inactive lists is useless ?
An advanced search could indeed be helpful when simple keywords are not enough to reduce the volume of results to one or two pages.
Apart from these little points, I really like this interface ! Keep up the good job !
Thanks zeta, this is great feedback!!! I agree about the graph; it came up during the test too. Some folks thought given lists were more active than the were because of the normalization.
Instead of normalising the graphs, they could be logarithmic. For example:
1 block = 1 post
2 blocks = 2 posts
3 blocks = 4 posts
4 blocks = 8 posts
5 blocks = 16 posts
…
10 blocks = 512 posts
15 blocks = 16384 posts
20 blocks = 524288 posts
A block might be 1px or 10px, perhaps depending on screen resolution. (There’s nothing preventing part-blocks — each pixel has 256 levels of opacity.) Maybe the block size could be normalised, but to the busiest list, globally across the system.
This might make each graph flat over a long enough timescale, though I suppose it would be useful to show there’s a constant level of posts. Users may also underestimate just how much of a difference each block makes.
Even with the problems that people noticed, hyperkitty is such a huge improvement over the current archiving in mailman that I’d like to be using it now for the lists I run.
Responses below. Really like the interface.
—-
You’ve got a crazy inbox and you’re really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?
A: Astronomy
Problems: It took me longer than I’d have liked to scroll down and find astronomy after searching the lists as it was below my visual field after searching. Because I saw nothing after the search on the screen I assumed the search had failed, but it was really just folding the lists I hadn’t searched for into small lines. It’s a nice effect but when nothing shows up it’s problematic. I ended up using the global search to find the name of the list to ensure that there was, indeed, something to find. Eventually I thought about scrolling down base don other searches and the visual feedback (under 2 minutes total).
Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there’s a lot going on – people actively making discussions.
A: scm-commits, package-announce, s390-builds, perl-devel, freemedia-trac (?: shows up in most active but discussion numbers hidden from me), java-sig-commits
Problems: None. Clear button top left for most active.
You’ve heard about Fedora’s development list (“devel”) and decided to look into it. Who are you likely to hear from if you read that list?
A: Adam Williamson 54 posts.
Problems: None. Clear “most active posters” section in list.
In March 2012, there was a discussion about GPT and Fedora 17 on the devel list. Can you find it?
A: No (though I gave up after about 3 minutes).
Problems: Search seems to be a problem here. With the volume of posts/pages during that month and year (the navigation to those points is very nice) the search function didn’t seem to drill down to the month/year I was exploring.
Find a post you like an add a tag to it so you can find it later.
A: Wasn’t logged in but this was an easy find.
Start a new discussion on the Fedora users list (“users.”)
A: I wasn’t logged in, but this seemed to be simple.