Standard Operating Procedures for Reviewing Academic Program Assessment Plans and Assessment Reports

Title

University of North Carolina at Chapel Hill Standard Operating Procedures for Academic Program Assessment Plans and Assessment Reports

Introduction

Purpose

This Standard Operating Procedure (SOP) describes the process of preparing and reviewing outcomes assessment reports for the University of North Carolina at Chapel Hill (“UNC-Chapel Hill” or “University”) and designates University staff responsible for the effectiveness of outcomes assessment in improving programs.

Scope

This Procedure applies to the Office of the Executive Vice Chancellor and Provost, Academic Deans, Assessment Coordinators, Office of Institutional Research and Assessment (OIRA) and anyone else involved in the review of Academic Program Assessment Plans and Reports for the following types of programs and units at UNC-Chapel Hill:

• Academic degree major (undergraduate, graduate, and professional), and
• Stand-alone certificates (undergraduate, graduate, and professional) which are designed to provide non-degree-seeking students with an introduction to a field of study or preparation to seek licensure in a specific professional area. Certificates consisting of courses that are part of existing degree programs or are designed primarily for currently matriculated students are not defined as stand-alone.

Standard Operating Procedures (SOP)

Programs Required to Engage in Student Learning Outcomes Assessment

UNC-Chapel Hill requires that all academic degree and stand-alone certificate programs undergo outcomes assessment annually, according to the following:

• Graduate programs that admit students to pursue a doctoral degree but award the non-terminal master’s degree as progress is made toward the doctorate prepare one Assessment Report for the doctoral program.
• In cases where a program is offered both via distance education and on campus, each delivery mode is viewed as a component of a single, integrated program that is assessed on the program level. However, Assessment Reports must include comparisons of the knowledge and skills attained by graduates across the modes of delivery.
• A program may petition the Office of Institutional Research and Assessment (OIRA) to report on Student Learning Outcomes every two years. The two-year report cycle is reserved for programs that may have difficulty with annual reporting because of small enrollments, generally considered 10 or fewer graduates within a reporting period. To request a two-year reporting cycle contact the University Director of Assessment at assessment@unc.edu.
• A new program must prepare their initial Student Learning Outcomes report for the year in which the first cohort of students graduates. If the first cohort is 10 or fewer students, the program may petition to delay their first report until the total number of graduates is greater than 10. To request a delay in reporting, contact the University Director of Assessment at assessment@unc.edu.
• Free-standing certificate programs are required to submit Assessment Reports. These certificate programs are generally designed to provide students with an introduction to a field of study and may enroll students not currently associated with the University. Certificate programs that are extensions of a current or completed course of study are not required to complete Assessment Reports. Reporting requirements for certificate programs are the same as those for other Academic Programs.

Documenting Student Learning Outcomes Assessment

The Assessment Plan and the Assessment Report are essential documents for reporting the Student Learning Outcomes assessment process.

The Assessment Plan

The Assessment Plan is a crucial tool for managing the annual assessment process. The plan maps out the Student Learning Outcomes for the program, and how and when the outcomes will be measured. Each Academic Program must have a set of comprehensive Student Learning Outcomes and include direct learning measures aligned with these outcomes. While not required, programs may include program outcomes measured and reported through the assessment cycle.

Specifically, the UNC-Chapel Hill Assessment Plan template requires that the Academic Program provide:

• The mission of the program;
• The individuals responsible for conducting the assessment and the process for reviewing the results;
• The Student Learning Outcomes to be assessed (For undergraduate programs, the Student Learning Outcomes on the Assessment Plan should match the Student Learning Outcomes in the catalog);
• Brief explanations of the assessment methods, including the source(s) of data and the direct assessment measures used to demonstrate the attainment of each outcome;
• A description of any targets or thresholds for each Student Learning Outcome; and
• The intervals or timelines at which data will be collected and reviewed.

OIRA reviews and approves plans before their implementation. Documentation about preparing the assessment plan is available through the university assessment website, including the Assessment Plan template and examples of completed plans.

The plan is multi-year and can be amended when needed. A new plan should be submitted to OIRA by the academic program Assessment Coordinator for review and approval when there have been changes to:

• The Student Learning Outcomes being assessed;
• The methods used to assess Student Learning Outcomes; and/or
• The implementation schedule.

All Academic Programs complete a full assessment cycle every three years, with rare exceptions for programs that have aligned their assessment effort with external criteria such as professional accreditation. In this model, Academic Programs engage in and report on assessment activity annually, in such a way that they can assess all their Student Learning Outcomes within a three-year cycle. Exceptions to the three-year cycle model are made in consultation with the University Director of Assessment in OIRA. To file a request for an exemption, email assessment@unc.edu.

Student learning outcomes should be:

• Measurable – describing what can be observed and verified with evidence.
• Student-Centered – measuring what students learn by the time they graduate.
• Faculty-Driven – reflecting what department faculty agree a student should know, or be able to do, as a result of completing the program.

Each program must include direct measures of learning that are aligned with the outcomes. Examples of appropriate direct measures include, but are not limited to:

• Written or oral comprehensive examinations
• Licensure examinations
• Standardized tests
• Final exam performance in courses mapped to a program-level outcome
• Research paper evaluations
• Capstone course project evaluations
• Internship performance evaluations
• Problem sets
• Research project performance
• Pre- and post-tests
• Clinical skills/competency assessments
• Portfolio evaluations
• Oral presentation evaluations
• Artistic performance and exhibit ratings by experts
• Thesis/dissertation proposal and defense evaluations
• Grading with scoring rubrics assessing dimensions of the learning outcome
• Faculty committee annual reviews of graduate students’ professional development: milestones reached, publications, conference presentations, grants, competitive fellowships received, etc.
• Analysis of Graduate Teaching Fellows’ instructional evaluations

Please contact the OIRA for assistance in identifying appropriate direct measures for your program by email assessment@unc.edu.

Publishing New Assessment Plans

1. Assessment Coordinator and Department Head create an Assessment Plan that will comply with SACSCOC reporting requirements.
2. Assessment Coordinator uploads the Assessment Plan to the OIRA Repository by Friday of the first week after Spring Break (see UNC-Chapel Hill academic calendar) and notifies OIRA that the Assessment Plan has been uploaded to the Repository.
3. OIRA reviews the Assessment Plan, provides feedback, and works with Assessment Coordinators to make any necessary changes.
4. Assessment Plans must be approved by OIRA by the end of the Spring semester.

Updating Existing Assessment Plans

An existing Assessment Plan only requires further review or acceptance by OIRA if the Academic Program changes the student learning outcome, method, or timeline. This can be on a rolling basis. If modifications are needed:

1. Assessment Coordinator and Department Head create an updated Assessment Plan that will comply with SACSCOC reporting requirements.
2. Dean or designee confirms the Assessment Plan complies with SACSCOC reporting requirements.
3. Assessment Coordinator uploads the updated Assessment Plan to the OIRA Repository and notifies OIRA by emailing assessent@unc.edu that the updated Assessment Plan has been uploaded to the Repository.
4. OIRA reviews the Assessment Plan, provides feedback, and works with Assessment Coordinators to make any necessary changes.
5. Any planned changes to assessment plans for the upcoming academic year should be in place by Friday of the first week after Spring Break.

The Assessment Report

The Assessment Report is submitted by each Academic Program by the Assessment Coordinator to OIRA through the Repository every Fall semester. This Report should reflect the assessment of student learning during the fall and spring semesters of the previous academic year, encompassing the collection of student learning data, analysis of results, discussion of the results and their implications, and recommendations for actions going forward.

Documentation about preparing the assessment report is available through the http://assessment.unc.edu website. This includes the assessment report template and a detailed guidebook for completing reports with examples.

The the UNC-Chapel Hill Assessment Report template requires that the Academic Program provide:

• Outlines the Student Learning Outcomes assessed during the assessment cycle period;
• Identifies and describes the specific assessment method(s) and tools used to gather evidence for the outcomes;
• Identifies the specific source(s) of the evidence;
• Provides brief results of each method and the extent to which the SLOs were achieved (using tables or figures where appropriate);
• Provides a summary or conclusions regarding the assessment results;
• Notes strengths and weaknesses observed in student performance that might inform improvements;
• Identifies how the results were shared and with whom;
• Identifies how the assessment data contributes to decision-making and the actions taken because of the information; and
• Provides evidence of continuously seeking improvement in student learning.

Each Academic Program makes decisions based on the review of collected data. Depending on the program, this may occur during faculty meetings, retreats, curriculum and instruction committees, or graduate committee meetings. The program faculty review the data and make decisions regarding what improvements to make. The Assessment Report form is structured so that Academic Programs describe actions taken or planned to improve, enhance, or sustain student success in achieving the learning outcomes assessed in that year, as well as how changes made in previous years have made an impact on student learning achievement over time.

While standard templates are used for Assessment Plans and Assessment Reports, these documents may be modified to reflect the assessment practices within an Academic Program. Modifications of the standard templates are made in consultation with the University Director of Assessment.

Roles and Responsibilities

Office of the Provost

• The Executive Vice Chancellor and Provost is responsible for overseeing the assessment processes of Academic Programs throughout the University.
• The Assistant Provost for Institutional Research, Planning, and Assessment has been entrusted by the Provost to lead the Student Learning Outcomes assessment process to ensure accountability and quality.
• The University Director of Assessment reports to the Assistant Provost for Institutional Research, Planning, and Assessment. The University Director of Assessment, who leads the Assessment Team in OIRA, is responsible for maintaining the necessary systems and infrastructure to support Student Learning Outcomes assessment as well as providing Academic Programs with direct assistance and professional development.

Academic Deans

• Deans are responsible for ensuring that all Academic Programs within their schools:
  • Have Assessment Plans,
  • Carry out assessments that meet prescribed standards, and
  • Submit Assessment Reports that document improvements made based on assessment results.
• Deans may set standard operating procedures for the Student Learning Outcomes assessment process within their school.
• Deans may designate Associate Deans or other professional staff to oversee aspects of the assessment process and review assessment reports on their behalf.
• Each Dean must appoint Assessment Coordinators to manage internal assessment processes for individual Academic Programs and to serve as liaisons to OIRA.

Assessment Coordinators

• Assessment Coordinators are responsible for:
  • Collecting and reviewing Assessment Plans and Assessment Reports,
  • Providing support to faculty within the department to improve the quality of assessment,
  • Ensuring that the Assessment Plans and Assessment Reports are submitted to the Repository maintained by OIRA, and
  • Participating in periodic training and professional development activities sponsored by the Office of the Provost.
• Coordinators of Academic Program assessment must be full-time faculty members and are often in the program coordinator role.
• Department chairs may designate or appoint additional faculty members to support the Assessment Coordinators in assessment activities.

OIRA Assessment Team

The Assessment Team in the OIRA:

• Provides guidance and training to Assessment Coordinators and other program faculty on effective assessment practices,
• Maintains an annual calendar of due dates for plans and reports and makes templates and other assessment resources available through its website,
• Maintaining a central Repository for Assessment Plans and Assessment Reports,
• Reviews these documents to ensure they conform to standards and provides feedback to Academic Programs on necessary improvements,
• Reports regularly to the Office of the Provost on compliance and opportunities for process enhancement.

Infrastructure

Repository

OIRA created and maintains a central repository of all Assessment Plans and Assessment Reports that have been submitted to ensure transparency and quality throughout the assessment process. The Repository is used to document initial Assessment Plans and Assessment Reports, and any feedback or changes to the approved Assessment Plan and Assessment Report.

Dashboard

OIRA maintains an Assessment Plans and Assessment Reports Dashboard to ensure accountability throughout the assessment process. The Dashboard is used to track the status of Assessment Plans and Assessment Reports. The Dashboard indicates when an Assessment Report for each program has been:

• Submitted,
• Reviewed,
• Returned for the program to “Revise and Resubmit”, and
• Accepted.

The Annual Review Process

Timeline

The University follows a specified timeline for assessment reporting, as delineated in Table 1 below.

Submission of Documents

• The forms for Student Learning Outcomes Reports and Plans are in Microsoft Word and are available through UNC-Chapel Hill's Assessment website.
• OIRA manages the receipt and review of reports on behalf of the Office of the Provost.
• The OIRA Assessment Team maintains a dedicated Microsoft Teams site that is used as the Repository for all documents related to the annual assessment process. Within the Teams site, each Academic Program has a unique folder where Assessment Coordinators may upload their program documentation. Student Learning Outcomes reports are submitted to the Office of the Provost through this Teams site.
• Department chairs and Deans are asked to review the documents submitted to the site and confirm that they are complete and meet expectations.
• The OIRA Assessment Team then reviews all submitted reports.

Evaluation Rubric

To fulfill the University outcomes assessment documentation requirement, OIRA established an Evaluation Rubric. This Evaluation Rubric sets forth basic standards and expectations for Student Learning Outcome Assessment Plans and Assessment Reports. The Evaluation Rubric closely aligns with the components of the Student Learning Outcome Assessment Plan and Assessment Report forms. The Evaluation Rubric will be provided to Assessment Coordinators through the Repository.

Assessment Plan Criteria

• When the Plan was last reviewed and updated,
• The clarity of the student learning outcomes,
• The explanation of methods and measures to be used for assessing each student learning outcome,
• A description of performance targets, and
• The proposed schedule for assessing each Student Learning Outcome.

Assessment Report Criteria

• How consistent the report is with the assessment plan,
• The description of the procedures used to assess student work,
• The discussion of the results, analysis, and interpretation,
• The discussion of improvements made in response to assessment results, and
• How findings from prior years have been addressed over time.

The OIRA Assessment Team evaluates Assessment Reports according to the Evaluation Rubric. When criteria are not met:

• The OIRA Assessment Team provides Academic Programs a worksheet delineating what changes need to be made to the reports,
• The University Director of Assessment meets with individual Assessment Coordinators to identify what additional support may be needed to fulfill Assessment Report requirements,
• Academic Programs may receive additional feedback by notations in the submitted documents, and
• The OIRA Assessment Team provides technical support to Academic Programs to ensure that criteria are met, including assisting with identifying appropriate assessment methods and data sources, as well as providing support with data analysis.

Upon the completion of the review process and acceptance of all Assessment Reports within a school, the Assessment Team is responsible for producing a summary for each dean. This summary comprises:

• An assessment of the quality of the school's reports,
• Identification of any significant findings that may be of interest to the dean, and
• An overview of how assessment results have been employed to enhance Academic Programs.

Additionally, the Summary Report will highlight any pending Assessment Reports that are yet to be submitted and indicate if any Assessment Reports are missing. These Summary Reports are shared with the respective deans by the Provost.

In addition to the summary report prepared for each dean, the Assessment Team also creates a comprehensive summary report for the Provost's Office. This Summary Report:

• Evaluates the quality of reports submitted from all academic programs across the university,
• Identifies any significant findings that may be of interest to the Provost, and
• Provides an overview of how assessment results have been employed to enhance Academic Programs.

The Assistant Provost for Institutional Research, Planning, and Assessment informs the Provost's Office when Assessment Reports have been deemed complete and acceptable and shares the Summary Report with the office. If necessary, the Provost may contact deans to request missing Assessment Reports based on the information provided in the Summary Report.

Definitions

This SOP incorporates definitions included in the UNC-Chapel Hill Policy on Student Learning Outcomes Assessment in Academic Programs.

Related Requirements

External Regulations

This SOP was developed to ensure UNC-Chapel Hill's continued compliance with the Southern Association of Colleges and Schools Commission on Colleges Principles of Accreditation, Comprehensive Standard 8.2.a.

The consequences of failure to comply with SACSCOC requirements include sanctions and possible loss of accreditation.

University Policies, Standards, and Procedures

• UNC-Chapel Hill Policy on Student Learning Outcomes Assessment in Academic Programs

Contact Information

Unit: Office of Institutional Research and Assessment

Title: University Director of Assessment

Email: assessment@unc.edu