NILOA Guest Viewpoints
We’ve invited learning outcomes experts and thought leaders to craft a Viewpoint. We hope that these pieces will spark further conversations and actions that help advance the field. To join the conversation, click the link below the Viewpoint. You can also sign up here to receive monthly newsletters that headline these pieces along with NILOA updates, current news items, and upcoming conferences.
Making Assessment Count
Dr. Maggie Bailey, Vice Provost Program Development and Accreditation, Point Loma Nazarene University
Most universities have highly developed processes for using assessment data to inform faculty how well students are achieving the program’s desired learning objectives, as well as identifying those areas needing improvement. However, this key evidence is too often not incorporated in other institutional decision-making processes such as program review, budgeting, program development, and strategic planning. PLNU’s Assessment and Program Review committees decided to tackle this issue by linking assessment findings directly into the program review process. The program review findings, including assessment, are then embedded in the program review Memorandum of Understanding (MOU), and used to inform budgeting and strategic planning.
When the two Committees began their discussions around harmonizing their processes, there were already key practices embedded in the PLNU assessment culture that made this “next step” in closing the assessment loop an easy transition. First, in spring 2012 PLNU adopted the NILOA Transparency Framework for curricular and co-curricular assessment reporting on the web. The PLNU adaptation of the NILOA Framework we refer to as the PLNU Assessment Wheel. Through the use of the wheel, the culture of accountability and transparency has been widely accepted in the PLNU ethos. Second, PLNU was well into revising both the Program Review and Assessment policies and practices to move the university away from a compliance model toward a continuous improvement cycle based on a culture of evidence. Third, the Program Review Committee had recently adopted a culminating Memorandum of Understanding (MOU) linking the outcomes of the program review self-study and external review to university budgeting and strategic planning. Therefore, a stronger and more intentional integration of the Assessment Committee’s review into the program review process was a natural next step in the development of the two Committees’ collaboration.
The PLNU Assessment and Program Review Committee faculty noted that when academic units go through program review, the focus shifts from internal course level assessment to external program level analysis, and the academic program self-study rarely closes the loop in articulating how assessment evidence is informing program improvements. Additionally, the Program Review Committee is focused on academic unit level issues informed by financial, enrollment, retention rates and other data and the important assessment findings are not always fully considered in the Program Review Committee’s decision-making. The message sent to faculty is that while assessment data is very important to the faculty culture and curriculum improvement, it is not essential to resourcing decisions or strategic priorities of the institution.
The Academic Assessment and Program Review Committees agreed the first step would be to harmonize the rubrics the committees use to conduct their evaluations. The Committees found in their respective rubrics that there were gaps and overlaps, as well as confusion in the assessment language. The committees delineated responsibilities and agreed to revise the rubrics to avoid redundancy, mixed messages, and gaps in accountability. In addition, the Committees decided to sequence and coordinate their reviews so that academic units moving through program review would first receive an in-depth Assessment Committee review and report. The Assessment Committee report (4 to 6 pages) includes commendations, recommendations for improvement, and a summary of the overall quality of the assessment work being done in the program. The report is then used by the program faculty to assist them in revising their assessment planning and activities as part of the program review process. The assessment report must be received by the Program Review Committee before the Program Self-study will be accepted by the Program Review Committee.
The revised process includes the following steps:
STEP 1 - ASSESSMENT WHEEL: Academic unit maintains all assessment planning and activities documentaton in their Assessment Wheel (NILOA Transparency Framework) on the PLNU website.
STEP 2: ASSESSMENT REVIEW AND SEQUENCING: PLNU has a three-year assessment cycle and a six-year Program Review Cycle. Each program goes through two complete assessment cycles between each program review. All major curriculum changes must be accompanied by a current Assessment Committee report and Program Review MOU.
STEP 3: ASSESSMENT REPORT: At the end of the second 3-year assessment cycle the Assessment Committee conducts an in-depth review followed by a Committee report (4-6 pages) on their findings, commendations, and recommendations. The Committee report is sent to the academic unit leadership, Program review Committee, curriculum committees, College Dean(s), and Provost.
STEP 4: PROGRAM REVIEW COMMITTEE: The Program Review Committee receives the Assessment Committee report and Assessment Committee rubric criteria scores on the “Quality of Assessment” are then embedded in the Program Review Self-Study Rubric.
STEP 5: MEMORANDUM OF UNDERSTANDING: The Program Review process culminates with a MOU and is linked to future budgeting, planning and resource allocation as well as a program improvement time line. The MOU contains three parts: (1) summary of academic unit and program improvements needed including assessment, (2) the action plan for the academic unit to improve the program with specific targets, and (3) a commitment from the University to provide resources, support, and faculty to achieve the agreed upon student learning outcomes.
The assessment of student learning is the foundation for program review with the focus of improving educational quality based on the culture of evidence. This assessment process is a continual, formative process of data collection, analysis, reflection and improvement. Below is a graphic depiction of the new reporting structure linking assessment to strategic planning.
PLNU began using the revised process in spring 2014, and while the evidence is still preliminary there are some early indications that this more coordinated approach holds out a promise to more directly align resources to academic assessment findings or at the very least having assessment data as important evidence informing Institutional decision-making.
We invite you to contribute your thoughts regarding this Viewpoint here.