Kyle Felker

LibraryQuest Levels Up

Almost a year ago, GVSU Libraries launched LibraryQuest, our mobile quest-based game. It was designed to teach users about library spaces and services in a way that (we hoped) would be fun and engaging. The game was released “into the wild” in the last week of August, 2013, which is the beginning of our fall semester. It ran continuously until late November, shortly after midterms (we wanted to end early enough in the semester that we still had students on campus for post-game assessment efforts). For details on the early development of the game, take a look at my earlier ACRL TechConnect post. This article will focus on what happened after launch.

Screenshot of one of the quests. — A screenshot from one of our simpler quests as it was being played.

Running the Game

Once the app was released, we settled on a schedule that would put out between three to five new quests each month the game ran. Designing quests is very time intensive, and 3-5 a month was all we could manage with the man and woman power we had available. We also had short duration quests run at random intervals to encourage students to keep checking the app. Over the course of the game, we created about 30 quests total. Almost all quests were designed with a specific educational objective in mind, such as showing students how a specific library system worked or where something or someone was in the physical building. Quests were chiefly designed by our Digital Initiatives Librarian (me) with help and support from our implementation team and other staff in the library as needed.

For most of the quests, we developed quest write-up sheets like this one: Raiders of the Lost…Bin. The sheets detailed the name of the quest, points, educational objective, steps, completion codes, and any other information that defined the quest. These sheets proved invaluable whenever a staff member needed to know something about a quest, which was often. Even simple quests like the one above required a fair amount of cooperation and coordination. For the raiders quest, we needed a special cataloging record created, we had to tag several plastic crowns and get them into our automatic storage and retrieval system.

For every quest players completed, they earned points. For every thirty points they earned, they were entered once in a drawing to win an iPad. This was a major component of the game’s advertising, since we imagined it would be the biggest draw to play (and we may not have been right, as you’ll see). Once the game closed in November, we held the drawing, publicized the winner, and then commenced a round of post-game assessment.

Thank You for Playing: Post-Game Assessment

When the game wrapped in mid-November, we took some time to examine the statistics the game had collected. One of our very talented design students created a game dashboard that showed all the metrics collected by the game database in graphic form. The final tally of registered players came in at 397. That means 397 people downloaded the app and logged in at least once (in case you’re curious, the total enrollment of GVSU is 25,000 students). This number probably includes a few non-students (since anyone could download the app), but we did some passes throughout the life of the game to remove non-student players from the tally and so feel confident that the vast majority of registered players are students. During development, we set a goal of having at least 300 registered players, based mostly on the cost of the game and how much money we had spent on other outreach efforts. So we did, technically, meet that goal, but a closer examination of the numbers paints a more nuanced picture of student participation.

A screenshot of the LibraryQuest Dashboard — A detail from our game dashboard. This part shows overall numbers and quest completion dates.

Of the registered players, 173 earned points, meaning they completed at least one quest. That means that 224 players downloaded the app and logged in at least once, but then failed to complete any quest content. Clearly, getting players to take the first step and get involved in the game was somehow problematic. There are any number of explanations for this, including encounters with technical problems that may have turned players off (the embedded QR code scanner was a problem throughout the life of the game), an unwillingness to travel to locations to do physical quests, or something else entirely. The maximum number of points you could earn was 625, which was attained by one person, although a few others came close. Players tended to cluster at the lower and middle of the point spectrum, which was entirely expected. Getting the maximum number of points required a high degree of dedication, since it meant paying very close attention to the app for all the temporary, randomly appearing quests.

A shot of a different aspect of the LibraryQuest Game dashboard. — Another detail from the game dashboard, showing acceptance and completion metrics for some of the quests. We used this extensively to determine which quests to retire at the end of the month.

In general, online-only quests were more popular than quests involving physical space, and were taken and completed more often. Of the top five most-completed quests, four are online-only. There are a number of possible explanations for this, including the observation offered by one of our survey recipients that possibly a lot of players were stationed at our downtown campus and didn’t want to travel to our Allendale campus, which is where most of the physical quests were located.

Finally, of our 397 registered users, only 60 registered in the second semester the game ran. The vast majority signed up soon after game launch, and registrations tapered off over time. This reinforced data we collected from other sources that suggested the game ran for too long and the pacing needed to be sped up.

In addition to data collected from the game itself, we also put out two surveys over the course of the game. The first was a mid-game survey that asked questions about quest design (what quests students liked or didn’t like, and why). Responses to this survey were bewilderingly contradictory. Students would cite a quest as their favorite, while others would cite the exact same quest as their least favorite (and often for the same reasons). The qualitative post-game evaluation we did provides some possible explanation for this (see below). The second survey was a simple post-game questionnaire that asked whether students had enjoyed the game, whether they’d learned something, and if this was something we should continue doing. We also asked if they had learned anything, and if so, what they had learned. 90% of the respondents to this survey indicated that they had learned something about the library, that they thought this was a good idea, and that it was something we should do again.

Finally, we offered players points and free coffee to come in to the library and spend 15-20 minutes talking to us about their experience playing the game. We kept questions short and simple to keep within the time window. We asked about overall impressions of the game, if the students would change anything, if they learned anything (and if so, what) and about what quests they liked or didn’t like and why. The general tone of the feedback was very positive. Students seemed intrigued by the idea and appreciated that the library was trying to teach in nontraditional, self-directed ways. When asked to sum up their overall impressions of the game, students said things like “Very well done, but could be improved upon”or “good but needs polish,” or my personal favorite: “an effective use of bribery to learn about the library.”

One of the things we asked people about was whether the game had changed how they thought about the library. They typically answered that it wasn’t so much that the game had changed how they thought about the library so much as it changed the way they thought about themselves in relation to it. They used words like “”aware,” “confident,” and “knowledgeable.” They felt like they knew more about what they could do here and what we could do for them. Their retention of some of the quest content was remarkable, including library-specific lingo and knowledge of specific procedures (like how to use the retrieval system and how document delivery worked).

Players noted a variety of problems with the game. Some were technical in nature. The game app takes a long time to load, likely because of the way the back-end is designed. Some of them didn’t like the facebook login. Stability on android devices was problematic (this is no surprise, as developing the android version was by far the more problematic part of developing the app). Other problems were nontechnical, including quest content that didn’t work or took too long (my own lack of experience designing quests is to blame), communication issues (there’s no way to let us know when quest content doesn’t work), the flow and pacing of new quests (more content faster), and marketing issues. These problems may in part account for the low on boarding numbers in terms of players that actually completed content.

They also had a variety of reasons for playing. While most cited the iPad grand prize as the major motivator, several of them said they wanted to learn about the library or were curious about the game, and that they thought it might be fun. This may explain differing reactions to the quest content survey that so confused me. People who just wanted to have fun were irked by quests that had an overt educational goal. Students who just wanted the iPad didn’t want to do lengthy or complex quests. Students who loved games for the fun wanted very hard quests that challenged them. This diversity of desire is something all game developers struggle to cope with, and it’s a challenge for designing popular games that appeal to a wide variety of people.

Where to Go from Here

Deciding whether or not Library Quest has been successful depends greatly on what angle you look at the results from. On one hand, the game absolutely taught people things. Students in the survey and interviews were able to list concrete things they knew how to do, often in detail and using terminology directly from the game. One student proudly showed us a book she had gotten from ILL, which she hadn’t known how to use before she played. On the other hand, the overall participation was low, especially when contrasted against the expense and staff time of creating and running the game. Looking only at the money spent (approximately $14,700), it’s easy to calculate an output of about $85 per student reached (173 with points) in development, prizes, and advertising. The challenge is creating engaging games that are appealing to a large number of students in a way that’s economical in terms of staff time and resources.

After looking at all of this data and talking to Yeti CGI, our development partners, we feel there is still a great deal for us to learn here, and the results are promising enough that we should continue to experiment. Both organizations feel there is still a great deal to learn about making games in physical space and that we’ve just scratched the surface of what we might be able to do. With the lessons we have learned from this round of the game, we are looking to completely redesign the way the game app works, as well as revise the game into a shorter, leaner experience that does not require as much content or run so long. In addition, we are seeking campus partners who would be interested in using the app in classes, as part of student life events, or in campus orientation. Even if these events don’t directly involve the library, we can learn from the experience how to design better quest content that the library can use. Embedding the app in smaller and more fixed events helps with marketing and cost issues.

Because the app design is so expensive, we are looking into the possibility of a research partnership with Yeti CGI. We could both benefit from learning more about how mobile gaming works in a physical space, and sharing those lessons would get us Yeti’s help rebuilding the app as well as working with us to figure out content creation and pacing, without another huge outlay of development capital. We are also looking at ways to turn the game development itself into an educational opportunity. By working with our campus mobile app development lab, we can provide opportunities for GVSU students to learn app design. Yeti is looking at making more of the game’s technical architecture open (for example, we are thinking about having all quest content marked up in XML) so that students can build custom interfaces and tools for the game.

Finally, we are looking at grants to support running and revising the game. Our initial advertising and incentives budgets were very low, and we are curious to see what would happen if we put significant resources into those areas. Would we see bigger numbers? Would other kinds of rewards in addition to the iPad (something asked for by students) entice players into completing more quest content? Understanding exactly how much money needs to be put into incentives and advertising can help quantify the total cost of running a large, open game for libraries, which would be valuable information for other libraries contemplating running large-scale games.

Get the LibraryQuest App: (iPhone, Android)

About our Guest Author:
Kyle Felker is the Digital Initiatives Librarian at Grand Valley State University Libraries, where he has worked since February of 2012. He is also a longtime gamer. He can be reached at felkerk@gvsu.edu, or on twitter @gwydion9.

Library Quest: Developing a Mobile Game App for A Library

This is the story of Library Quest (iPhone, Android), the App That (Almost) Wasn’t. It’s a (somewhat) cautionary tale of one library’s effort to leverage gamification and mobile devices to create a new and different way of orienting students to library services and collections. Many libraries are interested in the possibilities offered by both games and mobile devices, and they should be. But developing for mobile platforms is new and largely uncharted territory for libraries, and while there have been some encouraging developments in creating games in library instruction, other avenues of game creation are mostly unexplored. This is what we learned developing our first mobile app and our first large-scale game…at the same time!

Login Screen — The login screen for the completed game. We use integrated Facebook login for a host of technical reasons.

Development of the Concept: Questing for Knowledge

The saga of Library Quest began in February of 2012, when I came on board at Grand Valley State University Libraries as Digital Initiatives Librarian. I had been reading some books on gamification and was interested in finding a problem that the concept might solve. I found two. First, we were about to open a new 65 million dollar library building, and we needed ways to take advantage of the upsurge of interest we knew this would create. How could we get people curious about the building to learn more about our services, and to strengthen that into a connection with us? Second, GVSU libraries, like many other libraries, was struggling with service awareness issues. Comments by our users in the service dimension of our latest implementation of Libqual+ indicated that many patrons missed out on using services like inter-library loan because they were unaware that they existed. Students often are not interested in engaging with the library until they need something specific from us, and when that need is filled, their interest declines sharply. How could we orient students to library services and create more awareness of what we could do for them?

We designed a very simple game to address both problems. It would be a quest or task based game, in which students actively engaged with our services and spaces, earning points and rewards as they did so. The game app would offer tasks to students, verify their progress through multistep tasks by asking users to input alphanumeric codes or by scanning QR codes (which we ended up putting on decals that could be stuck to any flat surface). Because this was an active game, it seemed natural to target it at mobile devices, so that people could play as they explored. The mobile marketplace is more or less evenly split between iOS and Android devices, so we knew we wanted the game to be available on both platforms. This became the core concept for Library Quest. Library administration gave the idea their blessing and approval to use our technology development budget, around $12,000, to develop the game. Back up and read that sentence over if you need to, and yes, that entire budget was for one mobile app. The expense of building apps is the first thing to wrap your mind around if you want to create one. While people often think of apps as somehow being smaller and simpler than desktop programs, the reality is very different.

The main game screen. We found a tabbed view worked best, with quests that are available in one tab, quests that have been accepted but not completed in another, and finished quests in the third.

We contracted with Yeti CGI, a outside game development firm, to do the coding. This was essential-app development is complicated and we didn’t have the necessary skills or experience in-house. If we hadn’t used an outside developer, the game app would never have gotten off the ground. We had never worked with a game-development company before, and Yeti had never worked with a library, although they had ties to higher education and were enthusiastic about the project. Working with an outside developer always carries certain risks and advantages, and communication is always an issue.

One thing we could have done more of at this stage was spend time working on game concept and doing paper prototyping of that concept. In his book Game Design Workshop, author Tracey Fullerton stresses two key components in designing a good game: defining the experience you want the player to have, and doing paper prototyping. Defining the game experience from the player’s perspective forces the game designer to ask questions about how the game will play that it might not otherwise occur to them to ask. Will this be a group or a solo experience? Where will the fun come from? How will the player negotiate the rules structure of the game? What choices will they have at what points? As author Jane McGonigal notes, educational games often fail because they do not put the fun first, which is another way of saying that they haven’t fully thought through the player’s experience. Everything in the game: rules, rewards, format, etc. should be shaped from the concept of the experience the designer wants to give the player. Early concepts can and should be tested with paper prototyping. It’s a lot easier to change rules structure for a game made with paper, scissors, and glue than with code and developers (and a lot less expensive). In retrospect, we could have spent more time talking about experience and more time doing paper prototypes before we had Yeti start writing code. While our game is pretty solid, we may have missed opportunities to be more innovative or provide a stronger gameplay experience.

Concept to conception: Wireframing and Usability Testing

The first few months of development were spent creating, approving, and testing paper wireframes of the interface and art concepts. While we perhaps should have done more concept prototyping, we did do plenty of usability testing of the game interface as it developed, starting with the paper prototypes and continuing into the initial beta version of the game. That is certainly something I would recommend that anyone else do as well. Like a website or anything else that people are expected to use, a mobile app interface needs to be intuitive and conform to user expectations about how it should operate, and just as in website design, the only way to create an interface that does so is to engage in cycles of iterative testing with actual users. For games, this is particularly important because they are supposed to be fun, and nothing is less fun than struggling with poor interface design.

A side note related to usability: one of the things that surfaced in doing prototype testing of the app was that giving players tasks involving library resources and watching them try to accomplish those tasks turns out to be an excellent way of testing space and service design as well. There were times when students were struggling not with the interface, but with the library! Insufficient signage, space layout that was not clear, assumed knowledge of or access to information the students had no way of knowing, were all things became apparent in watching students try to do tasks that should have been simple. It serves as a reminder that usability concepts apply to the physical world as much as they do to the web, and that we can and should test services in the real world the same way we test them in virtual spaces.

A quest in progress. We can insert images and links into quest screens, which allows us to use webpages and images as clues.

Development: Where the Rubber Meets the Phone

Involving an outside developer made the game possible, but it also meant that we had to temper our expectations about the scale of app development. This became much more apparent once we’d gotten past paper prototyping and began testing beta versions of the game. There were several ideas that we developed early on, such as notifications of new quests, and an elaborate title system, that had to be put aside as the game evolved because of cost, and because developing other features that were more central to gameplay turned out to be more difficult than anticipated. For example, one of the core concepts of the game was that students would be able to scan QR codes to verify that they had visited specific locations. Because mobile phone users do not typically have QR code reader software installed, Yeti built QR code reader functionality into the game app. This made scanning the code a more seamless part of gameplay, but getting the scanner software to work well on both the android and iOS versions proved a major challenge (and one that’s still vexing us somewhat at launch). Tweaks to improve stability and performance on iOS threw off the android version, and vice versa. Despite the existence of programs like Phonegap and Adobe Air, which will supposedly produce versions of the software that run on both platforms, there can still be a significant amount of work involved in tuning the different versions to get them to work well.

Developing apps that work on the android platform is particularly difficult and expensive. While Apple has been accused of having a fetish for control, their proprietary approach to their mobile operating system produces a development environment that is, compared to android, easy to navigate. This is because android is usually heavily modified by specific carriers and manufacturers to run on their hardware. Which means that if you want to ensure that your app runs well on an android device, the app must be tested and debugged on that specific combination of android version and hardware. Multiply the 12 major versions of android still commonly used by the hundreds of devices that run it, and you begin to have an idea of the scope of the problem facing a developer. While android only accounts for 50% of our potential player base, it easily took up 80% of the time we spent with Yeti debugging, and the result is an app that we are sure works on only a small selection of android devices out there. By contrast, it works perfectly well on all but the very oldest versions of iOS.

Publishing a Mobile App: (Almost) Failure to Launch

When work began on Library Quest, our campus had no formal approval process for mobile apps, and the campus store accounts were controlled by our student mobile app development lab. In the year and a half we spent building it, control of the campus store accounts was moved to our campus IT department, and formal guidelines and a process for publishing mobile apps started to materialize. All of which made perfect sense, as more and more campus entities were starting to develop mobile apps and campus was rightly concerned about branding and quality issues, as well as ensuring any apps that were published furthered the university’s teaching and research mission. However, this resulted in us trying to navigate an approval process as it materialized around us very late in development, with requests coming in for changes to the game appearance to bring it into like with new branding standards when the game was almost complete.

It was here the game almost foundered as it was being launched. During some of the discussions, it surfaced that one of the commercial apps being used by the university for campus orientation bore some superficial resemblance to Library Quest in terms of functionality, and the concern was raised that our app might be viewed as a copy. University counsel got involved. For a while, it seemed the app might be scrapped entirely, before it ever got out to the students! If there had been a clear approval process when we began the app, we could have dealt with this at the outset, when the game was still in the conceptual phase. We could have either modified the concept, or addressed the concern before any development was done. Fortunately, it was decided that the risk was minimal and we were allowed to proceed.

A quest completion screen for one of our test quests. These screens stick around when the quest is done, forming a kind of personalized FAQ about library services and spaces.

Post-Launch: Game On!

As I write this, it’s over a year since Library Quest was conceived and it has just been released “into the wild” on the Apple and Google Play stores. We’ve yet to begin the major advertising push for the game, but it already has over 50 registered users. While we’ve learned a great deal, some of the most important questions about this project are still up in the air. Can we orient students using a game? Will they learn anything? How will they react to an attempt to engage with them on mobile devices? There are not really a lot of established ways to measure success for this kind of project, since very few libraries have done anything remotely like it. We projected early on in development that we wanted to see at least 300 registered users, and that we wanted at least 50 of them to earn the maximum number of points the game offered. Other metrics for success are “squishier,” and involve doing surveys and focus groups once the game wraps to see what reactions students had to the game. If we aren’t satisfied with performance at the end of the year, either because we didn’t have enough users or because the response was not positive, then we will look for ways to repurpose the app, perhaps as part of classroom teaching in our information literacy program, or as part of more focused and smaller scale campus orientation activities.

Even if it’s wildly successful, the game will eventually need to wind down, at least temporarily. While the effort-reward cycle that games create can stimulate engagement, keeping that cycle going requires effort and resources. In the case of Library Quest, this would include the money we’ve spent on our prizes and the effort and time we spend developing quests and promoting the game. If Library Quest endures, we see it having a cyclical life that’s dependent on the academic year. We would start it anew each fall, promoting it to incoming freshmen, and then wrap it up near the end of our winter semester, using the summers to assess and re-engineer quests and tweak the app.

Lessons Learned: How to Avoid Being a Cautionary Tale

Check to see if your campus has an approval process and a set of guidelines for publishing mobile apps. If it doesn’t, do not proceed until they exist. Lack of such a process until very late in development almost killed our game. Volunteer to help draft these guidelines and help create the process, if you need to. There should be some identified campus app experts for you to talk to before you begin work, so you can ask about apps already in use and about any licensing agreements campus may have. There should be a mechanism to get your concept approved at the outset, as well as the finished product.
Do not underestimate the power of paper. Define your game’s concept early, and test it intensively with paper prototypes and actual users. Think about the experience you want the players to have, as well as what you want to teach them. That’s a long way of saying “think about how to make it fun.” Do all of this before you touch a line of code.
Keep testing throughout development. Test your wireframes, test your beta version, test, test, test with actual players. And pay attention to anything your testing might be telling you about things outside the game, especially if the game interfaces with the physical world at all.
Be aware that mobile app development is hard, complex, and expensive. Apps seem smaller because they’re on small devices, but in terms of complexity, they are anything but. Developing cross-platform will be difficult (but probably necessary), and supporting android will be an ongoing challenge. Wherever possible, keep it simple. Define your core functionality (what does the app *have* to do to accomplish its mission) and classify everything else you’d like it to do as potentially droppable features.
Consider your game’s life-cycle at the outset. How long do you need it to run to do what you want it to do? How much effort and money will you need to spend to keep it going for that long? When will it wind down?

References

Fullerton, Tracy. Game Design Workshop (4th Edition). Amsterdam, Morgan Kaufmann. 2008

McGonigal, Jane. Reality is Broken: Why Games Make us Better and How they Can Change the World. Penguin Press, New York. 2011