|

Assessment of Student Outcomes and Systemic Analytics

By Steven Bell

1 Leave a comment on paragraph 1 0 Assessment may strike the reader as questionable subject matter in a 2015 collection of essays about new roles for academic librarians. Academic librarians, after all, are hardly strangers to the topic, and some fill campus leadership roles in adapting assessment mechanisms to the delivery of services. In the past decade the position of Assessment Librarian or User Experience Librarian has emerged as one of those most frequently added to the organization chart, conferences are dedicated to assessment, articles abound in the professional literature and the Association of College & Research Libraries is developing proficiency standards for assessment librarians. Thus it would seem that assessing student learning, resource effectiveness or any other of dozens of quantifiable and qualitative library services is firmly fixed territory within the academic library landscape. Granted, as a profession we can improve our mastery of collecting and analyzing data in support of better decision making. Despite all the gains made in the assessment arena, from conferences to dedicated discussion lists and hundreds of journal articles, there is still more work needed to deliver the type of assessment that will enable academic librarians to target their support services to those most in need of it – the academically struggling student at risk of losing it all. Looking down the new road, the signposts point the way to an infusion of new technologies that academic librarians will adopt to aid them in assessment.

2 Leave a comment on paragraph 2 0 Much of the current assessment taking place in academic libraries is the summative type; the focus is on assessing our impact at the end of the learning or service delivery sequence. To what extent did our intervention impact the student’s grade or their grade point average? Did our instruction session cause the student to choose better resources to include in their research paper bibliography? Does the arrangement of the study area furniture contribute to an improvement in study habits? There are endless questions for assessment projects. Owing to new technology developments in the area of learning analytics and automated assessment software, academic librarians may be able to shift to more formative assessment that allows them to intervene at the point of need when students are struggling academically. As higher education assessment becomes more systematic and predictive, academic librarians can add a new dimension to their portfolio of assessment activity.

3 Leave a comment on paragraph 3 0 Higher education institutions began experimenting with learning analytic software a few years ago. Purdue University, an early adopter of this technology, developed a system in 2009 called Course Signals. Using a combination of data, including grades, demographics and interaction with learning material, analytics software uses algorithms to produce, on demand for a student, an indicator signaling an instructor to take action. For example, a yellow signal may prompt the instructor to contact the student by email or arrange for a meeting to review that student’s course performance. Research on Course Signals shows that this type of formative assessment increases student retention by using early warnings to intervene at the point where a student is struggling (Arnold & Pistilli, 2012).

4 Leave a comment on paragraph 4 0  

SIDEBAR –  New Roles – Preemptive Support and Response Specialist – While they are somewhat rare today, more colleges and universities are looking into software systems that provide analytical analysis of student performance for early warning of academic difficulty and potential failure. Both two and four year institutions are adding “Fly in Four” type programs where at-risk students are pooled and assigned to advisers that help them get past the initial fear of failure that often leads students to dropout as freshmen when they first encounter the academic rigor of college that differs so greatly from high school. Research from Project Information Literacy demonstrates that when it comes to doing college-level research, many students lack confidence in their abilities and it leads to procrastination and failure. Just as analytical and early warning systems look for signs of potential failure, such as missing classes, a poor midterm grade, too few submissions to the course discussion group, etc., the Library Preemptive Support and Response Specialist receives and monitors student performance on research assignments and identifies as-risk students who needs additional attention and personalized assistance. This Specialist helps the institution to retain students by equipping with the skills needed to be a successful researcher as well as helping those students build a relationship with a librarian.

5 Leave a comment on paragraph 5 0 Using both library-generated and institutional data, academic librarians could develop similar analytics systems or collaborate with faculty who use them as well. As library assessment efforts go beyond the counting of inputs and outputs, they shift to measuring the extent to which librarian services and resources contribute to student retention, persistence to graduation, research productivity and overall academic success. Harnessing analytic technology could allow librarians to establish an intervention role when the algorithm identifies students struggling with research assignments. While students drop out for many reasons, from financial to family challenges, academic failure is one of the top contributors to an early departure. It is more common among low-income and first generation students who may arrive at college feeling prepared, but who often give up early on if they suffer academic setbacks.

6 Leave a comment on paragraph 6 0 Findings from Project Information Literacy reinforce that the transition to college-level research can easily overwhelm college freshmen. The 2013 report “Learning the Ropes” studied the freshman transition to the college library and found that the lack of confidence in research skills and dependency on Google search contributes to an aversion to research (Head, 2013). According to the report, many freshmen do eventually develop the capacity to exceed the limitations of their high school research experience. What about those who remain limited to Google and Wikipedia searching, and as a result fail to develop the necessary research skills for success? The Report makes good suggestions, such as better connections between high school and college librarians, but better assessment through analytics is not among them. That’s not to fault the PIL researchers though because too few academic librarians have adopted a preemptive approach to identify the students who need the most support making the transition to college-level research. But the PIL research clearly points to a need for such intervention if it were technologically possible.

7 Leave a comment on paragraph 7 0 American higher education is under great pressure to introduce reforms that will make a college diploma accessible to all those who desire it. That means keeping tuition affordable, helping students persist to graduation and developing new paths for earning degrees on campus and online. If colleges and universities are unable to do this voluntarily, they can expect even more pressure to make it happen thanks to some new “tough love” initiatives, as evidenced by the Obama administration’s proposed college rating system (Gardner, 2014). By design it will impose more stringent accountability measures while forcing institutions to improve accessibility, graduation rates and even post-graduate success if they want to improve their ratings. New approaches to assessment are an important piece of the strategy to help students avoid dropping out too soon or staying on too long, both of which are costly to colleges and universities. The rise of special intervention programs such as the Texas Interdisciplinary Program at the University of Texas, which uses analytics to identify at-risk students and track their academic performance so advisers can be notified if students need special assistance, are likely to become more common at both public and private institutions. Those academic librarians who embrace assessment should adapt well to an evolving role in which analytic methods are used to identify students when they are at the point-of-need for research support.  That evolving role may expand to include new responsibilities for academic intervention or working on teams with administrative or college-linked assessment professionals to coordinate the delivery of support services to students.

8 Leave a comment on paragraph 8 0 However, our profession’s inherent concerns about the importance of protecting privacy and the unwarranted mining of student data might challenge our ability to adopt this new role. It is part of our professional DNA to safeguard any data about our community members’ library activity. These fears are understandable owing to repeated higher education security breaches, many owing to human error, that have exposed private student data. But should we allow those fears to create roadblocks to potentially beneficial services to students, particularly if we could introduce the types of security measures needed to protect our community members’ right to privacy? In an opinion piece for EdSurge in 2014, Steve Rappaport addresses these exact issues of the data debate. He acknowledges the profound distrust for those who seek to mine big data in any sector of American education. There is another narrative that he says is often overlooked, in which data use is central to the mission of the education system whether for the proper administration of courses or managing financial aid. The desired outcome is properly managing data for use in the service of teaching and learning without sacrificing privacy or security.

9 Leave a comment on paragraph 9 0 There are two scenarios in which academic librarians could manage the use of student data in learning analytics technology to balance the need to help students achieve academic success with concerns about the library tracking student data, and correlating it with academic records to create intervention opportunities. First, it could be an entirely opt-in choice for students. Many students who achieve academic success independently and demonstrate no need for additional support could completely opt out. First-generation students, those from low-income households and others in at-risk situations may prefer to opt-in to allow monitoring by the library in order to give themselves an advantage when it comes to getting personalized research assistance at the point of need. Second, learning analytics will likely be a requirement for students that enroll in specialized support programs like the Texas Interdisciplinary Program. When colleges and universities invest funds in these programs to boost student success and persistence to graduation, they will seek to implement analytics to keep students on track and it is a trade-off students will likely accept. In order to receive a scholarship, extra support or preferred advising, students will understand that their academic performance will be closely monitored. In exchange for allowing their data to be mined by predictive analytics systems, the students receive a better shot at getting a diploma.

10 Leave a comment on paragraph 10 0 The transition from counting inputs and outputs to focusing on the library impact on academic performance is still in an early phase. At present, there are multiple academic libraries that are slicing and dicing student record data along with library usage data in order to demonstrate that library use by students contributes to higher grade point averages, better retention and any other indicators where the academic library does matter when it comes to student academic success (Soria, Fransen & Nackerud, 2013).  If academic librarians are able to master predictive assessment techniques it may complement efforts to promote adaptive learning for library research skills. It is, quite simply, software that allows a more personal form of learning, offering an individualized consultation activity.  Imagine an adaptive learning system that, using analytics, could detect when a student requires additional instructional content on locating scholarly articles or avoiding plagiarism, and could deliver librarian-produced tutorials at the point of need or automatically create an appointment for a consultation with an academic librarian.

11 Leave a comment on paragraph 11 0  

SIDEBAR – New Roles – Adaptive Learning Specialist – May work closely with the Preemptive Support and Response Specialist to identify students in need of personal attention because their individual learning analytics indicate they need more academic support. The Specialist works with instructors to customize an adaptive learning process for the student. While adaptive learning systems are relatively new in American higher education, more colleges and universities are planning to adopt them. The Specialist is a library staff member who focuses on creating adaptive learning activities for college students who need to improve their research skills. The Specialist adaptive learning technology skills to the academic library.

12 Leave a comment on paragraph 12 0 Adaptive learning systems can pave the way for learners to complete their degrees with greater independence and a curriculum more finely tuned to their academic interests. In the Chronicle of Higher Education’s special report on The Innovative University: What College Presidents Think About Change in Higher Education, when presidents were asked which innovation would have the most positive impact on learning, 61% responded that adaptive learning would revolutionize personal learning (figure 8, p.19) (Selingo, 2014b). Presently, most adaptive learning technology is produced by commercial publishers and is used primarily by for-profit higher education institutions to allow their students to customize their learning while reducing the need for routine presence by instructors (Fain, 2014). In this setting computers are being used to accomplish learning tasks conducted by humans in traditional higher education, a highly controversial application. When or to what extent the rest of higher education would implement these types of personalized learning systems is uncertain, but if they prove successful in reducing cost while boosting graduation rates, it is likely that college presidents would encourage adoption. When that happens, academic librarians may be able to use these systems to offer a more personalized approach to library instruction that incorporates better technology for assessment and learning analytics. As is often the case with what we see traveling down the new road, it will require us to rethink our roles and determine how we can best preserve our noble past as we adapt to a radically different higher education future and find a balance between the two.

13 Leave a comment on paragraph 13 0  

SIDEBAR: Flashpoint – Analytics and Adaptive Learning: Beneficial or Boondoggle.Steven: While I understand the concerns that academic librarians have about keeping student data private, I think you have to look at the big picture of what’s happening in higher education. When you do that you see that it’s a change we have to seriously consider, and the use of these tools is already happening. It’s just a question of how does the academic librarian leverage these technologies in a way that create a balance between do something that benefits the learning in need while keeping private data secure. That said, I do believe that the application of such technologies should be transparent and allow students to opt in or out. Not every student will need these technologies to be academically successful but for the ones that do, I believe they are beneficial.

Barbara: As Edward Snowden said during an alternative to the Queen’s Christmas speech in 2013, “privacy matters.” Some citizens may have been lulled by the entertainment value of “free” social platforms (actually paid for with loads of aggregatable personal information and targeted advertising) into believing privacy is a thing of the past. Facebook officials have said that privacy is an archaic social norm and is no longer relevant and if you are so concerned about it, you’re probably doing something you shouldn’t. They weren’t so casual about privacy when we learned that the NSA was data mining massive amounts of personal information, but the problem wasn’t invasion of privacy; it was the damage done to Internet security protocols and to the reputations of American companies. As librarians, we know privacy matters because it’s a condition necessary for intellectual freedom. Participating in schemes to use personal behavioral data to “improve the customer experience student learning” is capitulating to the unproven notion that analytics are smart and subtle enough to identify and fix difficulties students are having. I would argue that human beings are much better at doing that (as unpopular as that notion is in the era of adjunctification), and that learning is more complex than algorithms might suggest. Libraries should be places where learners pose problems of their own and practice freedom. Yes, I’m alluding to Freire’s Pedagogy of the Oppressed, but that’s because this current obsession with analytics is a modern-day high-finance banking concept of education. Try reading Gradgrind’s speech in Hard Times by Charles Dickens, substituting the word “metrics” for “facts” and you’ll see where I’m coming from. [1]
[1] “‘Now, what I want is, Facts.  Teach these boys and girls nothing but Facts.  Facts alone are wanted in life.  Plant nothing else, and root out everything else.  You can only form the minds of reasoning animals upon Facts: nothing else will ever be of any service to them.  This is the principle on which I bring up my own children, and this is the principle on which I bring up these children.  Stick to Facts, sir!’”
Page 14

Source: https://acrl.ala.org/newroles/?page_id=261