National Institute for Learning Outcomes Assessment |

National Institute for Learning Outcomes Assessment

NILOA Guest Viewpoints

We’ve invited learning outcomes experts and thought leaders to craft a Viewpoint. We hope that these pieces will spark further conversations and actions that help advance the field. To join the conversation, click the link below the Viewpoint. You can also sign up here to receive monthly newsletters that headline these pieces along with NILOA updates, current news items, and upcoming conferences.

 

Challenges and Opportunities in Assessing the Capstone Experience in Australia
Nicolette Lee, Australian Government Office for Learning and Teaching National Senior Teaching Fellow, Associate Professor and Acting Pro-Vice Chancellor (Learning and Teaching), Centre for Collaborative Learning and Teaching, Victoria University

 

Over the past few years in Australia, there has been a growing interest in assessing and benchmarking the quality of undergraduate programs and student learning outcomes.  At the same time, concerns have been expressed about graduate employability. This has led to a sector-wide focus on the potential of capstone curriculum as a tool for enhancing final year student experience and transition to post-graduate life. However, while final year projects are required in some disciplines such as engineering, capstones have not been a common feature of the Australian undergraduate degree. This situation is rapidly changing, as many universities are implementing capstones in an effort to respond to the graduate employability and quality assessment imperatives. 

Students in Australia, as in the US, generally undertake one capstone in the final year of their degree. However, in some cases, there may be a capstone for each major or for a major and core curricular stream. Capstones generally occupy a relatively small proportion of the final year credit: they may be a single subject worth one quarter or one half of the credit for one semester or spread across two semesters in two subjects worth one quarter of the credit for each semester. There is some variation in the curriculum format. Industry and research projects are predominant. Other capstones utilise simulations of industry activities, work placements and the like. Many, but not all, of these activities bear the hallmarks of what we understand to be a capstone experience: integration, extension and application of prior learning; transition from student to independent professional, and a closure and celebration of achievement in the degree.

In late 2013, I received a National Senior Teaching Fellowship from the Australian Government Office for Learning and Teaching to run a national program on the topic of capstones. The goal of this work is to provide faculty and universities with a contextual understanding of capstones in Australia and internationally, to provide targeted resources for capstone coordinators, to support decision-making at both institutional and curriculum design levels in Australia, and ultimately, to improve the student experience in capstones.

Drawing on prior studies, the research component of the Fellowship was designed to answer four central questions:

  • What is the range of capstone models in use across the sector?
  • What are the key characteristics and dimensions of capstone curriculum?
  • What challenges do faculty face in reconciling capstone innovation with policy and standards expectations?
  • What tools or resources are needed by staff to support curriculum design?

In June 2014, we completed the major data collection phase of the Fellowship, including interviews with capstone coordinators and a survey of current practices. The survey was wide ranging, covering program structure and timing, delivery and scope, key purposes, assessment and perspectives on the benefits and challenges of capstones. The final sample comprises some 216 capstones. The majority (88%) of the universities in Australia are represented, as well as contributions from New Zealand, the United Kingdom and the United States, and a small group from Canada and Singapore. There is also a wide spread of disciplines, with strong representation in social sciences and humanities, business and economics, health and biomedical sciences, and engineering. Smaller groups from creative and performing arts, physical sciences, architecture and law reflect the relative numbers of programs in those areas. Only four of the capstones were not attributed to a single broad field of education – that is, they were intrinsically generic or cross-disciplinary.

The early analysis of data is revealing some evocative findings on the topic of assessment, both assessment of student learning and capstones for program quality assessment.

We found a wide range of assessment activities are being used in combination. These were dominated by some of the conventional written and verbal outcomes we might expect of capstones, such as reports (71%) and presentations (69%). Evidence related to engagement domains (such as evidence of behavior, participation and reflective journals) was around 40%. Assessment of developmental or multiple outcomes was next, at around 30%, as were essays. Physical products such as manufacturing prototypes, websites or other media, along with exams and tests, occupied the lowest band at 23% and 22% respectively, the former possibly reflecting the limited number of respondents for whom physical products are the natural outcome of the profession, and the latter likely reflecting a move to assessment of evidence of student capacity to do the work, rather than declarative knowledge in examination conditions.

What is instructive is the number of cases that demonstrated a clear departure from conventional academic assessment evidence. There were individual cases of blogs, wikis, websites, videos and diaries; interviews, journal articles, student delivery of seminars or lectures and conference papers. In addition, the evidence seems to be being used as a way of demonstrating the quality of the work itself, particularly in performance scenarios where some learning activities might be more difficult to capture. Almost across the board, these kinds of approaches reflected a desire to have students demonstrate their best work and assess their performance in a public manner that would prepare them for the transition from being a student to a post-university setting

Faculty do not necessarily find less conventional assessment an easy path, however. In interviews, and anecdotally, several coordinators commented that one of the greatest struggles they experience is defending assessment approaches of authentic student work to conservative course approval committees. They also reported struggling with assessment policy where the number of assessments was limited and progressive or graded formative assessment did not easily fit within the required framework. On the complexity of assessment design, faculty reported challenges in simply working out how to capture the depth of learning outcomes (particularly in the affective and engagement domains) in assessment evidence and how to grade diverse project outcomes.

They also reported some struggles with how to balance peer and self-assessment between assessment for learning, assessment for grading, and equity. In the capstones we collected data about, teachers assess in 95% of cases. In 40% of cases, teacher assessment was the only assessment mechanism used. Peer assessment came in at a frequency of 47%, self-assessment at 38% and external assessment at 35%. The assessments carried out by non-teacher assessors were weighted inversely to their frequency: external assessments were weighted in the final capstone grade at a mean of 30%, self at 17%, and peers at 14%. This suggests that while faculty believed that using other assessors, particularly peer assessors, is important, teachers are still considered to be by far the most important verifiers of the quality of student work. We might also infer that they believe peers to be the most unreliable assessors.

Recent work in Australia has pointed to the use of student assessments in capstone subjects as evidence for quality assessment of programs, and as a tool for a tuning process in which staff from multiple universities review assessments for each other. On the topic of using capstones for this process, faculty could perhaps be best characterised as pragmatic. They indicate some logical challenges: capstone outcomes do not necessarily capture all of the learning (or not-learning) in a program, particularly where the capstone is weighted at only one eighth of the final year full time load, the program is very broad or students have numerous elective choices. Designing capstones that create the conditions for a substantial integration of skills and knowledge, as well as supporting transition, is clearly challenging under these conditions.

Further, faculty point to a teleological issue; students are also taking other subjects during the capstone period, meaning that unless there is a fully integrated capstone through which all final semester activities are assessed, we are not capturing all of the final year learning outcomes. Nonetheless, most faculty agree that using capstones as the focus for a ‘bird’s eye’ quality assessment ‘is better than nothing’, and the benefits of having a focus for delivering on program learning outcomes, as well as for external moderation, outweigh the limitations.

More information on the fellowship and associated research activities, as well as the resources for capstone curriculum design, can be found at the fellowship website: www.capstone.curriculum.com.au

The views expressed in this article do not necessarily reflect the views of the Australian Government Office for Learning and Teaching.

 

We invite you to contribute your thoughts regarding this Viewpoint here.

Check out our past Viewpoints:

Challenges and Opportunities in Assessing the Capstone Experience in Australia
Nicolette Lee

Making Assessment Count
Maggie Bailey

Some Thoughts on Assessing Intercultural Competence
Darla K. Deardorff

Catalyst for Learning: ePortfolio-Based Outcomes Assessment
Laura M. Gambino and Bret Eynon

The Interstate Passport: A New Framework for Transfer
Peter Quigley, Patricia Shea, and Robert Turner

College Ratings: What Lessons Can We Learn from Other Sectors?
Nicholas Hillman

Guidelines to Consider in Being Strategic about Assessment
Larry A. Braskamp and Mark E. Engberg

An "Uncommon" View of the Common Core
Paul L. Gaston

Involving Undergraduates in Assessment: Documenting Student Engagement in Flipped Classrooms
Adriana Signorini & Robert Oschner

The Surprisingly Useful Practice of Meta-Assessment
Keston H. Fulcher & Megan Rodgers Good

Student Invovlement in Assessment: A 3-Way Win
Josie Welsh

Internships: Fertile Ground for Cultivating Integrative Learning
Alan W. Grose

What if the VSA Morphed into the VST?
George Kuh

Where is Culture in Higher Education Assessment and Evaluation?
Nora Gannon-Slater, Stafford Hood, and Thomas Schwandt

Embedded Assessment and Evidence-Based Curriculum Mapping: The Promise of Learning Analytics
Jane M. Souza

The DQP and the Creation of the Transformative Education Program at St. Augustine University
St. Augustine University

Why Student Learning Outcomes Assessment is Key to the Future of MOOCs

Wallace Boston & Jennifer Stephens

Measuring Success in Internationalization: What are Students Learning?
Madeleine F. Green

Demonstrating How Career Services Contribute to Student Learning
Julia Panke Makela & Gail S. Rooney

The Culture Change Imperative for Learning Assessment
Richard H. Hersh & Richard P. Keeling

Comments on the Commentaries about "Seven Red Herrings"
Roger Benjamin

Ethics and Assessment: When the Test is Life Itself
Edward L. Queen

Discussing the Data, Making Meaning of the Results
Anne Goodsell Love

Faculty Concerns About Student Learning Outcomes Assessment
Janet Fontenot

What to Consider When Selecting an Assessment Management System
R. Stephen RiCharde

AAHE Principles of Good Practice: Aging Nicely A Letter from Pat Hutchings, Peter Ewell, and Trudy Banta

The State of Assessment of Learning Outcomes Eduardo M. Ochoa

What is Satisfactory Performance? Measuring Students and Measuring Programs with Rubrics
Patricia DeWitt

Being Confident about Results from Rubrics Thomas P. Judd, Charles Secolsky & Clayton Allen

What Assessment Personnel Need to Know About IRBs
Curtis R. Naser

How Assessment and Institutional Research Staff Can Help Faculty with Student Learning Outcomes Assessment
Laura Blasi

Why Assess Student Learning? What the Measuring Stick Series Revealed
Gloria F. Shenoy

Putting Myself to the Test
Ama Nyamekye

From Uniformity to Personalization: How to Get the Most Out of Assessment
Peter Stokes

Transparency Drives Learning at Rio Salado College
Vernon Smith

Navigating a Perfect Storm
Robert Connor

Avoiding a Tragedy of the Commons in Postsecondary Education
Roger Benjamin

In Search for Standard of Quality
Michael Bassis

It is Time to Make our Academic Standards Clear
Paul E. Lingenfelter