Notes on the User Experience in Open Source Panel at SIGCHI Boston April 2009

These are my notes from an open source & UX panel session I attended last month at SIGCHI Boston. They’re not completely proofread, and I think I may be missing the last part of my notes. But there is some really great material in here and I wanted to post it before I delayed any longer. 🙂

About this panel session

This session took place at SIGCHI 2009 in Boston, Massaschusetts. It was organized by Scooter Morris and Bonnie John.

Why was this panel put together?

• There are a lot of discussions, panels, SIGs covering open source and HCI at SIGCHI 09 and there have been in past CHIs – it’s not a new topic but generates a lot of interest.
• FOSS is meritocracy – quality of code, how much code you contribute
  • because of this, is there opportunity for HCI involvement? (answer: no)
  • because of this, is there no reason for practitioners to get involved? (answer: no)
  • this does introduce interesting issues though, how do HCI practictioners get involved? Can’t get involved the way we traditionally do
• Open source makes business sense
  • most of the fortune 500 use open source
  • widely used in gov’t
  • open source businesses coming into their own: mysql, mozilla
• Open source projects care about usabilityand need our help
  • flossusability.org
  • season.openusability.org

Panelists

• Alex Faaborg – principle ui designer at mozilla
• Jeff Noyes – IA @ Acquia
• Becca Scallon – fellow at U of Baltimore
• Alonso Vera – chief human systems NASA Ames Research Center
• Bonnie John CMU – moderator

Rules

• First we’ll give each panelist 10 minutes to present
• Then we’ll have a 20 minute moderated discussion
• We’ll finish with 30 minutes of Q&A and audience discussion

Individual Panelist Presentations

Alex Faaborg, Principal Designer, Firefox

The Drupal UX Community

There are 20-30 people on a given day in the drupal ux IRC chat. In the past year there has been more ux community. Example is password checker design. Becca got involved, looking for student project for usability tests, jumped in 4/2008. Had a short deadline, not much community involvement, but tested with 8 subjects and posted to the usability group, got great feedback. The Drupal usability group had made same suggestions but never got so much attention, she put together videos and made a nice looking report.
Recently, 2/2009, worked on first community-run test by Drupal Usability Group. Hang out in #drupalusability. It ran using community input and participation: open process, public archive of all data at drupalusability.org. As we were doing the tests, we put all our notes up, put up categories, participants, videos, eyetracking. Anyone can access the site and use to identify issues.
Becca got involved as a student, given a complex and interesting problem to work with at school “I definitely get a lot out of being in the community, a lot of opportunity, invite other students to get involved.” Work at “crowdsourcing usability tests.”
Usability = adoption = demand.
Usability expertise needed and valued!:

• http://groups.drupal.org/usability
• d7ux.org – created by ext design/ux agency hired to make drupal 7 better
• drupal.org/node/363429

Alonso Vera – NASA HCI Team – Adapting Bugzilla for Use at NASA

NASA thinks of large missions 20-30 years lifetime – every now and then you get a restart moment, tend to come when you get a new program. We’re in transition right now. Shuttle program will be ending in the next year, Constellation will be the new program that will go into 2040. You only get about a 5 year period to set up system to address your needs for the next few decades in these windows / restart moments.
HCI group at NASA has been involved in this: came into this by accident but we realized for this restart, a lot of capabilities are available open source in a way they weren’t even five years ago. Even some capabilities aren’t, controlling rovers on mars, maybe not.

NASA and Bugzilla interface enhancements

We needed to develop a non-conformist software/hardware problem processor. We started with Bugzilla, now we have a two-way relationship with bugzilla community. We have large info systems integration challenges:

• 10 NASA centers plus FFRDCS, facilities, and contractors
• need email, DB, reporting, search , sigs, challenge is that they have to work across a large and divided organization
• when you have an anomaly in space, you have minutes to hours to pull data together to understand the problem: for the shuttle this has been a problem, shuttle started in 70’s, was largely paper-based design.
• HCI group tasked to create single systems across the agency to pull data
• Things get fundamentally different at kennedy vs johnson vs marshal.

NASA’s HCI group developed the Bugzilla solution at NASA. The group is a relatively small HCI group – 15 people. We’re supported a dozen systems across the agency, we developed ourselves (using open source), we’ve had a fairly small team do this over the past 2-3 years, very successfully so fa. NASA has a small subset of bugzilla code they maintain. (10%, bugzilla maintains 90% of what they do) We built an adaptation layer. have another 1.5 years to finish to add fairly significant capabilities. We needed better reporting capabilities, for example. a lot of group decision making associated with the products of these tools.Llinking between data very important, not as important in bz (altho its a target for bz 4.0.) As soon as we could, we started opened sourcing our own code. If bugzilla takes our capabilities, it’s great for us because we are no longer maintaining it on our own, so upstreaming is very beneficial for us: our cost model improves.
Improvement of our cost model has been a tremendous selling point for the open source approach. We have been asked, “did you drop a zero from the cost estimate?” during meetings.
We have not branched from core Bugzilla. Everytime there is an upgrade we can take it. We keep the core code the same for all of the various BZ systems and we maintain configuration diffs between the systems. We can turn configuration on/off. When we first approached Bugzilla, it was at 2.8.  Everything needs to be changeable without changing code, because then have to redo our QA process and other things. $75,000 to change a dropdown item is not acceptable. BZ starting allowing GUI configurability in 3.0. It has made a big difference, especially in us selling Bugzilla to management.
There were slashdoted press releases on NASA drawing on open source for shuttle bug tracking. The Bugzilla community has reacted to what NASA developed as, “we like it, we want to keep it, we like the designs.” This has led to a relationship where NASA devels have become part of the BZ developer community. By getting involved in the community, and producing better reporting functionality from a UI POV,  we’ve helped BZ community.

Lessons Learned

Challenges:

• need to coordinate work with external (FOSS) team
• OS business model / longevity – a potential risk, what if project upstream dries up?
• perceived robustness and security issues (perceived, not true, but have to fight perceptions)

Benefits:

• Very significant cost savings, empowers small teams like our small HCI team
• on-going support from large FOSS community
• open standards – integratability across FOSS capabilities
• configurability

Alonso also had a talk on SIGCHI Thursday about UI Configurability.

Moderated Panel Discussion

Question 1 – Consistency

One of the very dearly held principles to help people learn new apps and work within a large application or system of applications is having consistency. Templates, style guides, compliance groups exist at some orgs to manage consistency. When you have so many people developing so many things, do you just let go of that principle? Or, do you have ways of managing this so end user experience is consistent?
Alex Faaborg: Firefox tries to be consistent across OSes, tries to fit within OS’s conventions / look & feel. of course with extensions, inconsitencies emerge, but i think that’s best solved by letting users, log into their browser and pick their extensinos. we’ll probably move more towards a profile model in the future, your firefox becomes yours.
Jeff Noyes: right now drupal definitely has consistency problems. we’re working towards building a model that lets people go to a specific site, see our standards, try to adhere to them. i dont think it’s going to be any different than at like fidelity, my past employer. have branches and designers all ove rthe world. designers in different locations need to be consistent, use styl eguide. goals and processes can be the same.
Bonnie John: Will there be a compliance committee like several large orgs?
Jeff Noyes: No, more like a place to go to in order to list standards. however the community wraps around that, remains to be seen but they can go there whenever they need to.
Bonnie John: When might that be?
Jeff Noyes: You never know how the comnunity will come together to handle things but the interesting thing is they just do.
Alonso Vera: Consistency issue might be addressed increasingly counter-intuitively by configurability of the system. when we looked at capabilities of bugzilla, and those with a bunch of features stuck together and configurability build in so you can change just about anything – we’ve made our apps consitent. we’ve basically done all the interaction design for it. since 3.0 you can built a front end on top. there has been no committee. we did it ourselves. and i think that’s going to be increasingly true of open source systems.
Bomnie John: So there would be an intermediary group that takes the core the rest of the way. Your group at nasa is the intermediary between the FOSS core and your end users?
Alonso Vera: Yep.

Question 2 – Addressing Usability Issues That Touch Many Parts of the Code

Sometimes usability issues touch may parts of the code. I was just looking at the firefox extensions. downloading and managing vide – any time you touch video, you hit the wrong one, you might want to stop it from playing. cancelling something reaches into many parts of the code. how do you manage that, if it touches 6 different people’s applications… how does the community form itself together to fix it in all the places it has to be fixed?
Alex Faaborg: There will be only one bug, but many people will cc themselves, so you’ll see people cluster together on the bug.
Bonnie John: That brings up another question on prioritization – any single product group has a wishlist of more than they could possibly do with their budget. #1 priority, #2 priority – if people get cc’ed… it’s the 100th priority – does it ever get fixed if on average 1/10 don’ thave the same priority… does that degrade? does it somehow come together?
Alex Faaborg: one aspect is scratch your own itch. some people do that. but i think the way prioritization comes into play – certain issue will block the release and everyone wants to get the release out. that pressure focuses people to tackle just the bugs that are blocking the release. public discussion, regular meetings to determine which bugs are blocker bugs. critical to shipping. that is how we prioritize.
Alonso Vera: slightly hard problem to manage – if you’ve got people in the commnunity – you’ve got to weave that into the QA process. if people external fix bugs, that weren’t blockers, you have non-blocker bugfixes that are coming in that you need to QA.
Alex Faaborg: yep. sometimes though they might not land in that release but we’ll move them to the next release. in general it’s useful to have it constantly coming in.
Scooter Morris: do design bugs often line up as blockers?
Alex Faaborg: it depends. how visible it is, for example.
Bonnie John: Can you give an example of a blocker design bug?
Alex Faaborg: we used the mozilla logo on the start page. it has a bright red dinosaur. not a very good starting-out first impression. we gotta get the red dinosaur out. became a blocker issue.
Jeff Noyes: founder behind drupal, after seeing usability studies, he’s made it a mission for the whole commnuity to make drupal 7 more usable, and he’s committed to not releasing it until it is more usable, so for that reason there have been a lot of design related issues that get prioritized.
Becca Scallon: it’s interesting bc prioritization has recently come up as an issue. it’s challenging because a lot of the major issues right now with drupal is a lot of the big issues are design issues, we don’t have as many designers in the commnunity putting effort into that – those are the harder problems to tackle, the little things are getting tackled, but the bigger issues are a challenge right now. we’re actively working on how do we priotiize the design issues. who is our audience, who are we designing for, cloudy right now.

Question 3: Stamping Out Ugliness

I think for the end user, there is confusion/deflating drupal & themes. some of the themes for drupal i saw are really bad in terms of usability. colors vibrating your eyes. really horrible. “we need your help! from drupal guy” do we have a responsibility to somehow stamp that out? and how could we do that in open source if we felt we had a responsibility .
Jeff Noyes: couple ways to do that. drupal ships with some core themes. we’ve determined those are not all tha tusable either. we’ve got initiatives to go out shpiping with better themes. #1. #2, whole drupal.org website is changing. is really old right now, hard to find info, i can imagine there wil be ways the commnuity could rate different themes. you could search for ones that everyone else is enjoying.
Scooter Morris: someone moved from user experience director to firefox director? he’s the gatekeeper?
Alex Faaborg: yes
Scooter Morris: we have a group leading the next generation bugzilla tracking system? is there a trend? the user experience seems to be taking more of a leadership role in these projects? i’d be interested in the answers to how this came about?
Alex Faaborg: obviously the mission is to try to get a significant chunk of the market using firefox. the only way to achieve that is to create a product people want. then UX becomes critical to the mission. to other projects, users might be the developers and they crae more about the product meeting their needs. best way to win is to have best performance, experience, security, suceced in every category you can. people realiziing user experience is critical.
Alonso Vera: in our case it’s been trying to deliver systems people like. AI guys say, well it’s easy for you, you just give the user things they want. AI trying to give what they don’t want. 🙂 but we’ve done exactly what we’re trained to do. we go, spend a lot of time with the users, when we first put out our project plan for this work, we had x amount of money in for travel, say it was a lot, one of the hecklers asked why do you need all tha tmoney for travel? just write teh reqs at your desk like everyone else. no you better go out and understand what your users are doing – that’s what we did, a little at a time, delivered systems based on what users need, and that slowly brought in more funding, we’ve been growing, getting more control over what we deliver.

Audience Question & Answer

Question 1: Where do UX bugs go?

(Asked by Brian Alice (?) from Google)
Becca, you showed us the bug tracker for a UI design issue. The first comment in the thread was “well a decent idea, this might not be the place for it, sounds like a feature request and not a bug.” How do you avoid ivory tower syndrome that UX bugs go to some special place, segregating them away, as if they are not a bug. How do you bring that back together?
Becca Scallon: It’s nteresting right now, in my experience, in irc, a lot of people in IRC are developers as well and they are really interested in changing and working on the patches themselves. They have interest in usability, some have been to usbaility tests and see them as real bugs. I’ve come in at a time where this is more accepted. I know in the past there was a lot more separation, but right now the team has a lot of support from active developers.
Jeff Noyes: That site was a list of all the problems you found during testing. They weren’t bugs that other people in the community were finding, they were issues found in usability studies. It was a conscious decision not to put then in our bug queue, because some were big-picture issues that needed design thinking first. For that site, the idea is to figure out which ones are needing design solution. Once the design solution is figured out we move it to the bug queue.
Alex Faaborg: It’s important to have developers whose core mission is to improve the user expreience. If something new usability-wise needs to be created, he should do it. How are usability bugs filed for firefox? They are filed against the component the bug is for, eg. the password management component. Developers who work on that section discuss the bug – it’s part of the standard defect tracking system.

Question 2: Who do you design for?

The open source response to usability used to be bad. We’ve moved forward from there. Where does each of the panelists find themselves identifying as to who it is that they are fixing the bugs for? Yeh, there are some bad Drupal themes, but you know, maybe there are people who would like that, it would be cool for someone’s kid… I find myself wondering, the user, users, generically… My question is: are you guys in the 37 signals / basecamp – “we design for ourselves, we’re the users…we know what we want” mindset, or do you have another method for user modeling?
Jeff Noyes: My take is we do, but it’s a work in progress. We are users, but we definitely don’t want to go about designing Ddrupal as if we are the only users. We need a broader spectrum. we have some personas and we try to build towards them, but our usability studies didn’t pick up until laast year. Only 6 months ago we even had a dedicated usability team. We’re really just starting to get some real leadership around usability for Drupal.

Question 3: What are you doing to validate your model and personas?

Becca Scallon: Just out of personal interest, I started doing interviews of Drupal users myself, 22 users. I found a range of people: the more hardcore developer types who tend to work on Drupal or web development teams, but there are a lot of hobbyists too. Hobbyists were the largest group: The schoolteacher who really wants to create a social networking site with his peers and is the techy of the group and is hoping maybe to get income in the future, for example. I found a lot of universities using Drupal. From what  I found, this kind of interviewing is really new in the community and there is a lot of interst in it. This is just he beginning. We need more users, we need to find out who is actually using  it right now? Who can use it? Who wants to? This process is really just starting. There is so much interest in it… we’re starting that.
Jeff Noyes: We hired a UK-based design firm to help get drupal to were it needed to be. That firm has identified a specific group who they think are the users and they assembled a panel. They bounce their designs against that panel. Also we do usability tests – to make sure the designs work for Drupal, Joomla, WordPress users….
Alex Faaborg: One of the big challenges of open source is that people fall back to personal preference as the evidence for everything. Neither is right: let the user decide. The problem is, you need to choose the default and it will be used by the mast majority of people. For instance, the tabs close button placement in Firefox. Andrea did cognitive modeling to prove that one was better than the other, because it’s not about preference. One is better.
Alonso Vera: We did interviews before we even started. we take advantag eof the fact that bugzilla is highly-configurable to then address those user needs. in some ways it sounds like drupal similarly is highly tailorable, and boonie what you were talking aobut is that there were themes, fundamental problems of usbaility problems with themes. i think part of what you’re trying to do with foss is make it so that FOSS is as eneralizable once you get it out there by the users themselves or intervening people who will adapt it.

Question 4: How do you get programmers on board with usability in small projects?

(Asked by Steven (?) from W3c)
You’re all reps of really big projects. Almost by fiat usability has been made part of that. On a lot of small projects, you don’t have that. You talked about scratching your own itch – if it’s not their itch on a small project, they won’t take you as seriously. You’ll suggest clearly better change, but the programmer won’t see it as being so fundamental. Do you have advice about how we can tickle the programmers so they get the itch as well?
Scooter Morris: We have 10 developers worldwide in my project – it’s very difficult for us to get usability help: people who can come and be involved and help us do some user studies in general. Everyone on the team would love it. We have weekly conference calls where we ask, “Should we do this or that? Well, we need user feedback, okay.” And the decision gets postponed. I think that the challenge here is how the smaller open source projects can tap into UX resources in the same way that Drupal has. Becca is a grad student who jumped in. Yes, it was a large open source project, but in this case Becca isn’t paid by the project, so that’s an example of how the smaller projects can avail themselves of this kind of community.
Alex Faaborg: It’s also possible to lead by example. Sometimes an approach to problems is baked in the tools themselves.
Questioner: Scooter, in your small project you really need usability help. That’s no the same problem because you’re listening for UX help. For example, I’ve got a problem with Ubuntu. If the network card doesn’t work, the only thing you have is the help files because you can’t get on the network, and the troubleshooter is atrocious. I reported the steps I did to try to fix the problem and failed. The reply I got made it clear that I offended the person who owned it, and they really didn’t see the problem and didn’t think that I should be so aggressive. I was just trying to help. but they really didn’t want to listen.
Jeff Noyes: It’s amazing how effective video works. The only way I’ve been successful at solving this problem is to put it into a study and see what comes out of it. Usability studies don’t have to be expensive. Eye tracking is full-blown, but you can just walk over to the person next to you.
Jim Gettys: Remember that most of the software you use on the Linux desktop is out-of-the-box GNOME or KDE; those are big projects that have their own usability teams. They will often make changes without necessarily feeling the developer has veto over them. You shouldn’t have reported to  Ubuntu. You should have reported to GNOME. The GNOME project is where that kind of work happens: it crosses many out-of-the-box desktop applications. Ubuntu is not the UI project at all, it’s a whole linux distro.

Question 5: What about this is specific to open source rather than software in general?

(Asked by Joe from the University of Minnesota (?))
I’m rying to figure out what here is unique to open source software. First I was imagining a discussion about Wikipedia – it’s not the way it used to be. We’ve got teams and projects trying to get data into a usable form. Just thinking about software – I’m trying to come up with what attributes are different here? Size? Small software? Most of the small closed-source commercial software I’ve bought has had bad usability. To some extent, what I’m hearing is you may have solved the problem of how to get graphic designers and UX designers interested in projects. If it’s big and lucrative enough, famous enough, then it’s a solution. Then I thought maybe the different was management, but then I’m hearing from you panelists that your management structures are ridiclously diverse. Is the difference here something about the ethos? Some free & open source software communities have transcended the ethos that we’re a community of users developing for ourselves and people like us, and some projects haven’t. If the answer is, “it’s customizable and I can build a whole layer on top of it” – at NASA you’re building something for users, and Bugzilla is building something for geeks. Besides that, is there something about this that is unique to open source software? Or is this all generalizable?
Alex Faaborg: You have to get more momentum behind your ideas in open source. There’s more evangelism involved. In the corporation, you must do this or you’ll get fired – it’s more of a talk-down command structure. You don’t need to evangelize.
Alonso Vera: How do you get to be from a small operation to a big operation? Well, the small ops are less responsive than the big ones. But the big ones get to be big. One of the verykey things we have to attend to is our user interface and take that seriously – it’s a critical component to competing. My perception is that we’ve seen a change in indutry, including FOSS, that being competitive means you have to make your users happy. To your point – I feel like another important difference between software in general and open source – there are roots and mechanisms by which a community can get in there and contribute. That’s not the case with products. Key point – there is access to this [CHI] commnunity. We can get in, demonstrate our benefit, demonstrate how we can impact the usability of software.

Question 6: Who are the target UX folks you want to get involved in open source?

Someone asked, who are the target users? My question – who are the target UX people you want to get involved in your projects? Some people come from computer science & engineering, some people come from design/fine art.  I was involved in the Mozilla design challenge, I’ve done some drupal work. The Mozilla screen design challenge had me very excited, and very motivated. When I started the sessions I then heard things like – “oh, we didn’t really want your type of UX person; we wanted someone who knew about development.” Another example: the new, drupal 7 UX website. When I saw those videos I was very inspired to get involved. It spoke to me and I felt I would be understood. You guys talk a lot about bug reports – that means nothing to me, and it doesn’t inspire me to get involved. Talking about little issues – little things – matter, but being able to change something very small on one module isn’t going to change the entire experience. How do we deal with these cultural issues when we try to bring UX people into the fold?
Jeff Noyes: The first answer is that I want all types of UX people. I dont have a preference. If you are interested, the first step is to go to ir