Collaborative UX Testing: Cardigans Included

Usability Testing

Understanding and responding to user needs has always been at the heart of librarianship, although in recent years this has taken a more intentional approach through the development of library user experience positions and departments.  Such positions are a mere fantasy though for many smaller libraries, where librarian teams of three or four run the entire show.  For the twenty-three member libraries of the Private Academic Library Network of Indiana (PALNI) consortium this is regularly the case, with each school on staff having an average of four librarians.  However, by leveraging existing collaborative relationships, utilizing recent changes in library systems and consortium staffing, and (of course) picking up a few new cardigans, PALNI has begun studying library user experience at scale with a collaborative usability testing model.

With four library testing locations spread over 200 miles in Indiana, multiple facilitators were used to conduct testing for the consortial discovery product, OCLC’s WorldCat Discovery. Using WebEx to screen record and project the testing into a library staff observation room, 30 participants completed three general tasks with multiple parts helping us to assess user needs and participant behavior.

There were clear advantages of collaborative testing over the traditional, siloed approach which were most obviously shown in the amount and type of data we received. The most important opportunity was the ability to test different setups of the same product. This type of comparative data led to conclusive setup recommendations, and showed problems unique to the institutions versus general user problems. The chance to test multiple schools also provided a lot more data, which reduced the likelihood of testing only outliers.

The second major advantage of collaborative testing was the ability to work as a team. From a physical standpoint, working as a team allowed us to spread the testing out, keeping it fresh in our minds and giving enough time in-between to fix scripts and materials. This also allowed us to test before and after technical upgrades. From a relational perspective, the shouldering of the work and continual support reduced burn out during the testing. Upon analyzing the data, different people brought different skill sets. Our particular team consisted of a graphic/interface designer, a sympathetic ear, and a master editor, all of whom played important roles when it came to analyzing and writing the report. Simply put, it was an enjoyable experience which resulted in valuable, comparative data – one that could not have happened if the libraries had taken a siloed approach.

When we were designing our test, we met with Arnold Arcolio, a User Researcher in OCLC’s User Experience and Information Architecture Group. He gave us many great pieces of advice. Some of them we found to work well in our testing, while others we rejected. The most valuable piece of advice he gave us was to start with the end in mind. Make sure you have clear objectives for what data you are trying to obtain. If you leave your objectives open ended, you will spend the rest of your life reviewing the data and learning interesting things about your users every time.

He recommended: We decided:
Test at least two users of the same type. This helps avoid outliers. For us, that meant testing at least two first year students and two seniors.
Test users on their own devices. We found this to be impractical for our purposes, as all devices used for testing had to have web conferencing software which allowed us to record users’ screen.
Have the participants read the tasks out loud. A technique that we used and recommend as well.
Use low-tech solutions for our testing, rather than expensive software and eye tracking software. This was a huge relief to PALNI’s executive director who manages our budget.
Test participants where they would normally do their research, in dorm rooms, faculty offices, etc. We did not take this recommendation due to time and privacy concerns.
He was very concerned about our use of multiple facilitators. We standardized our testing as much as possible.  First, we choose uniforms for our facilitators. Being librarians, the obvious choice was cardigans. We ordered matching, logoed cardigans from Lands’ End and wore those to conduct our testing. This allowed us to look as similar as possible and avoid skewing participants’ impressions.  We chose cardigans in blue because color theory suggests that blue persuades the participants to trust the facilitator while feeling calm and confident. We also worked together to create a very detailed script that was used by each facilitator for each test.

Our next round of usability testing will incorporate many of the same recommendations provided by our usability expert, discussed above, with a few additions and changes. This Fall, we will be including a mobile device portion using a camera mount (Mr. Tappy see http://www.mrtappy.com/) to screen record, testing different tasks, and working with different libraries. Our libraries’ staff also recommended making the report more action-oriented with best setup practices and highlighting instructional needs.  We are also developing a list of common solutions for participant problems, such as when to redirect or correct misspellings. Finally, as much as we love the cardigans, we will be wearing matching logoed polos underneath for those test rooms that mirror the climate of the Sahara Desert.

We have enjoyed our usability experiences immensely–it is a great chance to visit with both library staff, faculty, and students from other institutions in our consortium. Working collaboratively proved to be a success in our consortia where smaller libraries, short staff, and minimal resources made it otherwise impossible to conduct large scale usability testing.   Plus, we welcome having another cardigan in our wardrobe.

More detailed information about our Spring 2015 study can be found in our report, “PALNI WorldCat Discovery Usability Report.”

About our guest authors:

Eric Bradley is Head of Instruction and Reference at Goshen College and an Information Fluency Coordinator for PALNI.  He has been at Goshen since 2013.  He does not moonlight as a Mixed Martial Arts fighter or Los Angeles studio singer.

Ruth Szpunar is an Instruction and Reference Librarian at DePauw University and an Information Fluency Coordinator for PALNI. She has been at DePauw since 2005. In her spare time she can be found munching on chocolate or raiding the aisles at the Container Store.

Megan West has been the Digital Communications Manager at PALNI since 2011. She specializes in graphic design, user experience, project management and has a strange addiction to colored pencils.


Comments are closed.