Based on a decade of data from comprehensive examination scores in an online doctoral program, it was determined change needed to occur to strive for greater consistency in assessment of online exams. A Comprehensive Exam Assessment Team (CEAT) was developed to create a collaborative and reliable process for assessing exams.

Doing anything online in higher education can be considered a bit more challenging and completing comprehensive exams in a strictly online setting with committee members from across the country is no different. In a time where more and more programs are moving to an online format, there are many facets of teaching, learning, and assessment that program directors and administrators need to focus on more carefully. For this online doctoral program, comprehensive examinations have recently risen to the top as a priority. This lightening session will address the motivation behind creating a Comprehensive Examination Assessment Team (CEAT) to improve consistency in assessment of online exams.

For ten years, the student’s comprehensive examination was assessed by their dissertation committee, which consisted of three doctorly prepared members. The students were assigned a chair and a second member, who were faculty, staff, or graduates of the EdD program. In addition to these two members, the students were asked to select an outside member for their committee. This member could be an expert in their field or on their topic or research design. Many students selected colleagues or used references to find their third committee member. This practice resulted in dissertation committees that brought a wealth of experience to the group, but that experience was not always consistent with the EdD program and resulted in varied scores.

After a thorough review of comprehensive exam scores and anecdotal evidence from previous years, it was determined that the current structure of assessment could be improved to strengthen the consistency of the comprehensive exam grading process. A proposal was submitted and approved by leadership to create the Comprehensive Examination Assessment Team (CEAT). The pilot took place in spring of 2024.

Team members were carefully selected from a pool of former graduates of the EdD program. It was decided that graduates would be strong candidates for the team because of their knowledge of the program content and experience with the exam process. Each team was led by a full-time faculty member in the EdD program. All CEAT members received training on how to assess the comprehensive examination to improve inter-rater reliability for each of the exams, which included assessing former examinations for practice.

This lightening session will focus on the implementation of the Comprehensive Exam Assessment Team in its pilot year. This includes the training that took place prior to the exam, the comprehensive exam assessment timeline, and the follow-up procedures for reviewing and analyzing scores for greater consensus. Due to the limited time in the lightening session, engagement for this session will take place by sharing tools and resources used throughout the pilot year of implementing the CEAT. By providing examples, session participants can make personal connections to their current online assessment practices of significant milestones, such as comprehensive exams, in their programs.