This workshop focuses on critically evaluating research studies in technology-integrated teaching and learning. Participants will explore the Quality Indicators Tool for assessing e-learning research quality, gaining insights into its application and significance in informing teaching practices. Discussions on standardized criteria and study planning offer practical insights for educators and practitioners.

With technological advancement, various formats of e-learning have been growing exponentially. While quality indicators for online delivery of instruction and sustainable development of the courses were well-established through multiple educational studies (Hafeez et al., 2022; Moore, 2002; Timbi-Sisalima et al., 2022), assessing the quality of e-learning educational research has been overlooked, and the literature lacks research quality indicators or standardized appraisal criteria in reference to e-learning educational research. Further, theoretical frameworks are also underutilized in educational research, and experimental research is still poorly present due to the difficulties of its implementation in e-learning contexts. At this point, it is crucial to analyze the evolution of educational research on e-learning and appraise the research quality. Quality indicators in e-learning research refer to the criteria or measures used to assess the quality and rigor of research studies related to e-learning, which is the use of web-based technologies to deliver educational content and facilitate learning. These indicators are used to evaluate the methodological soundness, validity, trustworthiness, and reliability of research findings in the field of e-learning. In essence, good research in education is defined by evidence that is reliable, applicable to (many) practical situations, consistent, and neutral (unbiased) regardless of whether a qualitative or a quantitative technique is taken. The design and operationalization of the quality requirements may vary between qualitative and quantitative research, even though they share similar standards for good evidence (quality criteria). However, issues such as choosing the “right” research method (qualitative, quantitative, mixed method, etc.) and strategies for rigorous data collection/data consistency (collect, record, and process) and analysis procedures are still common concerns (Cohen et al., 2017; Fraenkel et al., 2018; Panke, 2018). To address the gap mentioned above, online educators, researchers, and instructional designers need a practical framework and tool tailored to the unique aspects of e-learning research. Thus, we developed the Quality Indicators Tool for Evaluating E-Learning Research to address the need for assessing the rigor and relevance of research studies in the field of e-learning in a systematic way. The tool comprises 11 sections, each designed to evaluate specific aspects of a published peer-reviewed article related to e-learning research. These sections cover everything from the clarity of the title and abstract to the research methodology, results, discussion, conclusion, references, and additional information. Each section includes specific questions that researchers or reviewers can use to evaluate the quality of a published peer-reviewed article. To ensure the tool's relevance and comprehensiveness, the study group members reviewed the tool and provided valuable input regarding the inclusion of relevant sections and questions, as well as the refinement of existing criteria. Face validity was established through discussions within the workgroup, ensuring that the tool appeared to measure what it intended to assess. In addition to the workgroup validation, input from experts in the field of e-learning research was sought to enhance the tool's content validity. Experts provided feedback on the clarity, specificity, and appropriateness of the criteria and questions included in each section. Their expertise ensured that the tool accurately captured the essential elements of rigorous and relevant e-learning research. This workshop will focus on how to critically evaluate the quality and rigor of research studies to ensure that evidence-based practices are being used to inform decision-making and can contribute to the advancement of the technology-integrated teaching and learning field. Participants will explore the essential aspects of evaluating e-learning research studies using the newly developed Quality Indicators Tool. Through interactive discussions and hands-on activities, participants will gain insights into the significance of research evaluation in enhancing online teaching and learning practices. They will learn how to apply the Quality Indicators Tool to critically assess the quality of e-learning research studies and make informed decisions about their applicability to their own teaching contexts. Through large group participation, the session will use examples from specific studies to discuss standardized criteria and appraisal scales for best practices in the scope of methodological and reporting quality of e-learning educational research. The presenter will also share tips on study planning and manuscript preparation. Workshop Agenda, Topics/Activities (90 Minutes) Step 1: Opening (5 minutes) • Welcome and overview of the workshop agenda and objectives. • Importance of evaluating e-learning educational research studies Step 2: Warm Up - Quick knowledge check! (5 min) • An interactive quiz and polling to check the audience's familiarity with the quality indicators for assessing e-learning education studies. Step 3: Understanding E-Learning Research (10 minutes) • Definition and Scope of E-Learning Research • Types of E-Learning Research Studies • Challenges and Considerations in Evaluating E-Learning Research Step 4: Introduction to the Quality Indicators Tool (15 minutes) • Overview of the Newly Developed Quality Indicators Tool • Explanation of Each Indicator and Its Importance • Demonstration of How to Use the Tool for Evaluation Step 5: Small Group Activity - Hands-on Practice (30 minutes) • Participants will be divided into small groups. • Each group will review a sample research paper related to e-learning in education using the newly developed "Research Quality Indicators in e-Learning Publications" tool. • Groups will assess the research paper's quality and identify strengths and weaknesses. Step 6: Large Group Discussion (10 min) • Each small group will share their findings and insights with the larger group. • The presenter will also facilitate a discussion on common challenges and observations in evaluating research quality. Step 7: Standardized Criteria and Appraisal Scales (5 min) • Discuss the importance of standardized criteria and appraisal scales in improving research quality. • Q&A: Participants’ questions or concerns about the use of standardized criteria. Step 8: Tips for Study Planning and Manuscript Preparation (5 min) • Provide practical tips and advice for participants on planning and publishing e-learning research. Step 9: Conclusion and Wrap-Up (5 min) • Summary and key takeaways from the workshop • Questions & Answers Session Goal: The session aims to introduce participants to the Quality Indicators Tool for Evaluating E-Learning Research and provide them with the necessary knowledge and skills to effectively utilize the tool in evaluating research articles in the field of e-learning. By the end of the session, participants should understand the structure and content of the tool, be able to apply it to assess the quality of e-learning research articles and recognize its importance in promoting rigorous and relevant research practices in the field. Level of Participation: The workshop session is structured to create a mutual learning experience that encourages active participation from all attendees, fostering engagement through interactive discussions, hands-on activities, and opportunities for reflection. Participants are encouraged to ask questions, share insights from their own experiences, and collaborate with peers in applying the Quality Indicators Tool to sample research articles. The level of participation is designed to be inclusive, accommodating various learning preferences, and promoting a supportive learning environment where everyone's contributions are valued and respected. Learning Objectives By the end of the session, participants will be able to: • Describe quality indicators for assessing e-learning educational studies, including various criteria with different appraisal scales for methodological quality. • Apply the newly developed tool to evaluate the quality of e-learning research studies in eleven sections of a manuscript. • Discuss standardized criteria and appraisal scales for best practices in the scope of methodological and reporting quality of e-learning educational research. References • Cohen, L., Manion, L., & Morrison, K. (2017). Research Methods in Education. Routledge. • Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2018). How to Design and Evaluate Research in Education. 8th Edition. McGraw Hill. • Hafeez, M., Naureen, S., & Sultan, S. (2022). Quality Indicators and Models for Online Learning Quality Assurance in Higher Education. Electronic Journal of E-Learning, 20(4), pp374-385. https://doi.org/10.34190/ejel.20.4.2553 • Moore, J. C. (2002). Elements of Quality: The Sloan-C Tm Framework. Olin College - Sloan-C. • Panke, D. (2018a). Research Design & Method Selection: Making Good Choices in the Social Sciences. SAGE. • Timbi-Sisalima, C., Sánchez-Gordón, M., Hilera-Gonzalez, J. R., & Otón, S. (2022). Quality Assurance in E-Learning: A Proposal from Accessibility to Sustainability. Sustainability, 14(5), 3052. https://doi.org/10.3390/su14053052