National Institute for Learning Outcomes Assessment |

National Institute for Learning Outcomes Assessment

NILOA Guest Paper Response

We’ve invited thought leaders to provide a response to Occasional Papers that NILOA crafts. We hope that these pieces will spark further conversations and actions that help advance the field. Please sign up here to receive monthly newsletters that headline these pieces along with NILOA updates, current news items, and upcoming conferences.

Response to Equity and Assessment: Moving towards Culturally Responsive Assessment

Steps in the Right Direction: Additional Strategies for Fostering Culturally Responsive Assessment
By Melissa Wright
University at Buffalo

Introduction

Institutions of higher learning have made significant strides over the past ten years in designing intentional learning experiences around the issues of equity. Supported by the work of organizations like the Association of American Colleges and Universities (AAC&U) and the Council for the Advancement of Standards (CAS), many institutions have created learning outcome domains and/or general education requirements around the broader categories of global awareness, diversity, and social responsibility. Though important milestones, the recent surge of student protests and demands for increased administrative accountability suggest that colleges and universities—like U.S. society at large—still have much to consider and improve upon in this area. Moreover, while institutions are increasingly articulating and assessing learning outcomes related to intercultural awareness, institutional discrimination, and intersectionality, equity is not consistently at the forefront of learning outcomes assessment practices or discussions. The National Institute for Learning Outcomes Assessment’s (NILOA) Occasional Paper, Equity and Assessment: Moving Towards Culturally Responsive Assessment, by Erick Montenegro and Natasha A. Jankowski identifies this surprising gap in the field of assessment and provides a useful model for conducting assessment with equity at the center.

Putting Students at the Center: Collaboration and Informal Assessment

Though Montenegro and Jankowski offer at least three concrete strategies for bringing equity into focus in the learning outcomes assessment process, the model they outline also entails a paradigm shift in the way college leaders, administrators, and practitioners think about and describe equity issues. Specifically, the authors observe that “Higher education has a tendency to group student differences and issues around race under the term ‘diversity,’ which is often discussed in relation to benefits to White students as opposed to African Americans, Latinx, Asian Americans, and Native Americans who continue to be underrepresented in higher education” (p. 8). Rather than using the word ‘diversity’, which fails to account for the way marginalized students experience life in or outside of college, Montenegro and Jankowski suggest a shift to the word ‘culture’ instead. ‘Culture’, they argue, is a more inclusive concept, one that extends beyond the race/ethnicity divide and embraces everything from “language, housing region, heritage, race/ethnicity, rituals, [and] religion” to a group’s “beliefs, values, ethics, gender identity, sexual orientation, [and] common experiences (e.g., military veterans and foster children)” (p. 9). Their definition of culture also includes “cognitive elements,” which comprise all those unifying ways in which groups know, behave, or perceive in the world.

While the terminological nuances are important, perhaps even more vital is their call to include students in the articulation of learning outcomes statements. Especially because denotation and connotation can shift over time, involving students in discussions about what language is inclusive and representative of their learning experience is necessarily an ongoing and cyclical process. Moreover, in Montenegro and Jankowski’s definition of culturally responsive assessment, the authors clarify that such assessment “does not simply mean being mindful of students.” “Instead,” they write, “being student-focused calls for student involvement throughout the entire assessment process including the development of learning outcomes statements, assessment tool selection/development process, data collection and interpretation, and use of results” (p. 10). In one of my recent experiences teaching a first-year writing class at the University at Buffalo, one that was largely comprised of students enrolled in the Educational Opportunity Program, I invited students to assist me in composing and refining the rubrics used to grade their assignments. I was surprised to find that students were both engaged in the activities—asking questions, suggesting revisions—and that many acknowledged the learning happening in both their reflections on the assignments and in their course evaluations comments. Indeed, I was encouraged most by the meta-cognitive awareness they demonstrated in these observations, especially as it related to their involvement with the assessment tools—as commentators, co-authors, and peer-raters.

While there has been a shift over the past few years to collect more meaningful campus climate data, at least in part driven by the White House’s 2014 Task Force to Prevent Sexual Assault, institutions could overall strive to more regularly include students in the assessment cycle, as Montenegro and Jankowski recommend. It bears noting, however, that many institutions and practitioners have embraced elements of a culturally responsive assessment framework. At the University at Buffalo, the Division of Student Affairs has undergone a thorough review process to articulate their learning outcome framework, inviting leaders and staff from across the university to provide feedback. Mindful that the language should be student-friendly and inclusive of multiple student viewpoints, they also invited student leaders to comment on the framework from their own perspectives. Such student involvement can also extend to the selection of appropriate assessments and assignments, especially when culturally-biased assignments can unintentionally privilege certain groups of students and “reinforce for marginalized students that they do not belong because their learning ‘doesn’t count’” (p. 8). As the authors observe, several European institutions have implemented alternative assignment options, where students can choose how they want to demonstrate their learning (e.g., prepare a presentation in lieu of an exam), but the reality today is that many, if not most, programs and courses in institutions across the U.S. assess students the same way for the same learning outcome.

While more flexible assignment structures overall would better serve all students, Montenegro and Jankowski also highlight informal assessment as a useful strategy in encouraging culturally responsive assessment. In my experience in the classroom, informal or formative assessment is instrumental in scaffolding learning for non-native speakers, first-generation students, and marginalized groups who may not be as prepared to write traditional college essays. As Brownie and Horstmanshof (2014) observe, instructor comments on discussion board posts leading up to a writing assignment were particularly beneficial for students who may need more time to respond to questions or prepare their writing (p. 64). If a common assignment is necessary or required for all students, scaffolding the learning leading up to that assignment—with multiple points for quick feedback from instructors—can make all the difference in student achievement and self-efficacy.

Problem- and Inquiry-Based Assessment

While Montenegro and Jankowski do not cite problem- or inquiry-based assessment directly, it seems that such models of assessment would complement the culturally responsive assessment approach they recommend. In Assessing for Learning (2010), Peggy Maki defines problem-based assessment as a process that “encourages faculty and other educators to collaboratively identify and pursue student-focused learning problems or issues translated into open-ended research or study questions” (p. 124). Both Lesley University and Marquette University offer examples of collaborative, interdisciplinary, and inquiry-driven assessment approaches that complement culturally responsive assessment to the extent that they bring educators’ questions about student learning to the center of the assessment process. At Lesley, a culture of inquiry guides assessment both at the academic program-level and in the assessment of general education outcomes and required student experiences, such as internships or practicums. In addition to serving on assessment teams to rate student artifacts, faculty are instrumental in determining the questions related to student learning that they wish to explore through assessment. At Marquette, the Office of the Provost has a portion of their website dedicated to inquiry-based assessment, inspired by Maki’s (2010) model of problem-based assessment. Though the institution does have a robust annual assessment plan process for academic programs, they also encourage faculty members and administrators who may be feeling “tapped out” of their current annual assessment plan to identify problems and formulate questions based on their work with students. Often working in interdisciplinary teams, this inquiry-based approach is intended to push assessment beyond the simple analysis of whether a threshold was met and explore the conditions behind those thresholds. By asking not only ‘what was learned and to what extent?’, but also ‘who learned, who didn’t, and why/why not?’ Marquette’s inquiry-based assessment approach seems like a useful model for motivating campus practitioners to examine their role in creating inclusive and equitable learning environments.

The cultures of assessment at both Lesley and Marquette suggest models for how other institutions might combine disaggregation of learning outcomes data with a thoughtful process of inquiry, one that corresponds to the authors’ suggestion that the disaggregation of data “should explore why the condition exists in the first place, and then be used to inform/develop possible solutions” (p. 14). And while some regional accrediting bodies, such as the Western Association of Schools and Colleges, have encouraged the disaggregation of assessment data by multiple demographic variables, many institutions still struggle to move beyond what Montenegro and Jankowski call “surface-level findings.” As the authors argue, “the disaggregation of assessment data should not only be used to uncover surface-level findings such as Latinx students excel at ‘ABC’ while first-generation students struggle with ‘XYZ’” (p. 14). Indeed, without looking at multiple student characteristics or the conditions that reproduce inequity among certain populations, the disaggregation of data along race/ethnicity, for example, might unconsciously be used to justify or reinforce cultural stereotypes rather than transform them.

Conclusion

The potential problems of disaggregation only emphasize the importance of including students and practitioners—both of whom have context for the learning situation and conditions underlying it—throughout the assessment cycle. At the same time, as Montenegro and Jankowski observe, campus leaders, administrators, and practitioners must continue to be mindful of the ways their cultural experiences influence the interpretation of data. Though it is essential that students and practitioners be invited to the assessment table, it also seems that further guidelines for analysis and discussion of data would be useful. This seems especially true as campuses contend with the results of and responses to the recent U.S. election and its implications for marginalized students. Moreover, because bias cuts across all cultural viewpoints and is not limited to any one group, guidelines for facilitating discussions with campus stakeholders about sensitive assessment data may help alleviate anxieties about potential biases and encourage a richer and more transparent dialogue about equity on campus.

References

Brownie, S., Horstmanshof, L., & Garbutt, R. (2013). A scaffolded approach to discussion board use for formative assessment of academic writing skills. Assessment & Evaluation in Higher Education, 38(1), 61-73.

Maki, P. L. (2010). Assessing for learning: Building a sustainable commitment across the institution. (2nd Ed.) Herndon, VA: Stylus Publishing, LLC.

About the author:

Melissa Wright is a PhD Candidate, Clint Goodson Research Fellow, and Teaching Assistant in the Department of Comparative Literature at the University at Buffalo. In addition to her studies at UB, Melissa is also a Senior Consultant in the Division of Campus Success at Campus Labs, where she guides colleges and universities in the northeast region in the implementation of strategic planning, learning outcomes assessment, and course evaluation software and educates member campuses on topics and trends in the larger field of institutional effectiveness. She is also a co-editor of theory@buffalo 19: Difference: Sexual, Cultural and Universal (2017).

For additional responses to this paper please click below:

Video response by Dr. Eboni Zamani-Gallaher

Response by Dr. Thomas F. Nelson Laird & Dr. Allison BrckaLorenz

Response by Dr. Pamela Felder