Blended-Learning Training and Evaluation: A Qualitative Study

Introduction

The current research follows the Research Master Plan of Jabal Ghafur University, Indonesia. To support the strategic research plan, one of the macro policies that should be addressed for institutional development is the e-learning policy. This policy is designed by considering the changing world towards the era of Society 5.0, a concept of a human-centred and technology-based society developed by the Japanese government (Gultom et al., 2021). This concept was born as a development of the Industrial Revolution 4.0, which is considered to have the potential to reduce the role of humans. The Industrial Revolution 4.0 is a strategic initiative introduced by the German government to transform the manufacturing industry through digitisation and exploit the potential of new technologies (Rojko, 2017; Herman et al., 2022). The presence of Society 5.0 will be one of the innovations that help solve the problems we face in the future.

In response to the issues, lecturers need to understand technological developments and changes in learning instructions in line with current technological advances. The change in learning from traditional to digital technology approaches is now an essential aspect of the current Indonesian curriculum, i.e., Merdeka Belajar-Kampus Merdeka (MBKM, hereafter, Merdeka curriculum) in universities. Digital competency is also one of the crucial 21st-century skills that should be incorporated into teacher education programs (Yesmakhanova et al., 2022). Thus, educators should be able to answer this challenge by empowering their students with technology in all aspects of life (Larson & Miller, 2011; Silalahi et al., 2022). Since the issuance of four ministerial decrees in 2020 regarding learning implementation in the 2021 academic year during the COVID-19 pandemic––to wit, the use of online and offline combinations––some lecturers at Jabal Ghafur University were found to conduct lectures through WhatsApp groups. Whereas this application is mainly designated for communication instead of digital-based learning. Many students complained that such learning was less effective. This case indicates that, in practice, not all lecturersunderstand appropriate e-learning lectures. Hence, they prefer teaching offline, albeit prone to the spread of COVID-19.

Moreover, many lecturers have insufficient knowledge and skills in digital-based learning, including planning, implementation, and evaluation (Herman et al., 2023). If this phenomenon is left unattended without any intention from the lecturers to improve the quality of their lectures in the digital-based learning environment, of course, the macro policy of institutional development that has been proclaimed through the e-learning policy will be hampered. Whereas e-learning is effective during the pandemic since distance, place, and time are not obstacles, mobile devices, such as laptops and tablets, can mediate online learning. Although the COVID-19 pandemic has somewhat eased and the restrictions imposed on community activities have been lowered, e-learning and blended learning, which combines face-to-face interaction via the internet and interactive media as a means for transferring knowledge, are still necessary (Chaiyama, 2019). E-learning or information and communication technologies (ICT) have many advantages. Notably, students can learn at their own level and speed as well as manage their learning. ICT promotes active rather than passive learning (Özerol, 2009), enhances students’ physical and mental abilities, and encourages them to develop their thinking processes (Farahani et al., 2015). Chiu and Churchill (2016) state that children have no fear of the digital world and are quick to embrace innovations and technology products. Bwalya (2019) believes integrating technology into education is a positive step, and other researchers have found that technology plays a crucial role in changing the classroom environment.

Blended learning highly supports the digitalisation era of society 5.0 and has the following characteristics: using information and communication technology, adapting to society, community participation, shared values, emergence, inclusiveness, effectiveness, the power of intelligence, and economic development (Salgues, 2018). Those characteristics show that the world of education serves a vital role in improving the quality of human resources in the future. Therefore, students should be prepared for the era of education through blended learning lectures. The mixed learning model is virtual learning accompanied by detailed activities to improve lecturers’ knowledge and skills through Zoom meetings, Google Meet, and Google Classroom that can be applied in the blended learning process (Beard & Wilson, 2006).

Furthermore, hybrid learning is an e-learning development that combines online and synchronous learning. On the other hand, e-learning is only asynchronous, while blended learning constitutes both. Therefore, blended learning is an effective learning model in online learning (Mufidah & Surjanti, 2021). Blended learning has been widely used for learning instruction and has been reported to improve student learning outcomes (Saritepeci & Cakir, 2015). Based on the above explanation, we need a learning model that can directly combine online, offline, and face-to-face learning. Therefore, there is a need for innovative learning and training for lecturers after the pandemic to support Merdeka curriculum policies. Through the training on blended learning, the lecturers are expected to be able to develop their knowledge and skills in operating digital-based learning.

Blended learning is a learning model that combines face-to-face learning with online learning (Fitriasari et al., 2018). This learning model has two learning systems: direct synchronous learning and synchronous virtual learning. Direct synchronous learning is a face-to-face mode at the same time and place. In contrast, synchronous virtual learning is a face-to-face mode, virtually and simultaneously, but in a different place (Ngabidin, 2021). Meanwhile, asynchronous learning activities are carried out indirectly at other places and times. With this learning model, students are expected to meet directly with their classmates and lecturers through an application from electronic media. They can also meet face-to-face in class without electronic media. Thus, one may state that this learning model can directly combine online, offline, and face-to-face modes. A blended learning approach is a learning approach that integrates traditional face-to-face learning and distance learning using online learning resources and a variety of communication options that educators and students can use. This kind of learning allows a combination of online and face-to-face learning (Harding et al., 2012).

Many researchers have studied blended learning worldwide at the higher education level. Anthony et al. (2020) reviewed the body of literature on blended learning in higher education from 2004 to 2020. They found that blended learning has been studied globally in the context of students, lecturers, and administrators. Among them, blended learning studies concerning lecturers (9.10%) still receive a small portion compared to those conducted for students (59.62%). Meanwhile, blended learning studies for students and lecturers simultaneously are also considered limited (10.10%). In Indonesia, in the context of students, blended learning was adopted to investigate students’ perception (Rianto, 2020), attitude (Muhayyang et al., 2021), responses (Tarihoran, 2020), its effectiveness for students (Rachmadtullah et al. 2020), etc. Meanwhile, other studies implemented blended learning to find lecturers’ perspectives (Mulyadi et al., 2020), participation (Shamad & Wekke, 2019), attitude (Musdariah et al., 2020), and so on. Others were concerned with blended learning instructional design, such as Chaeruman et al. (2020), Munthe et al. (2021) and Suartama et al. (2019).

Concerning the issues, it is essential to conduct more blended learning studies in the context of lecturers or lecturers-students concurrently. Lecturers’ competence, including knowledge and skills in using blended learning, is vital to investigate since they contribute to successful and effective blended learning (Le et a., 2022). Relevant to our case, as previously mentioned, insufficient knowledge and skills of the lecturers in using blended learning make the learning less effective or even discourage them from employing it. For that reason, the researchers conducted training on the blended learning model to improve lecturers' competence in using blended learning. We subsequently investigated their competence and satisfaction in using blended learning and its impact on students' abilities after lecturing in the blended learning environment. To be specific, this study aims to investigate: (1) lecturers' understanding of the concept of blended learning and their skills in using several blended learning platforms through the training; and (2) the effectiveness of blended learning model training in improving lecturers’ competence.

Additionally, this study differs from previous studies on lecturers’ competence (Arianto et al., 2021; Martin et al., 2020). Arianto et al. (2021) investigated the effect of blended-learning training on the lecturers’ competence in developing teaching instructions but did not assess the implementation of the teaching instruction on students. Martin et al. (2020) also investigated the faculty’s competence in the use of technology, but they did not organize training. Therefore, this study will contribute to the literature on blended learning in the context of lecturers and students.

Methods and Materials

Research Design and Sampling

This research aims to assess the lecturers’ competence in blended learning before and after training. This study constitutes an experimental study, with the one-group pretest-posttest design, in which the effect of training given to one research group was investigated after the treatment. The competence of lecturers measured includes knowledge and skills regarding blended learning and its use, including planning, implementing, and evaluating digital-based lectures. Meanwhile, the skills refer to designing a lesson plan integrating Google Classroom, Google Forms, and YouTube. The samples in this study involved all lecturers from 20 departments at Jabal Ghafur University, where one lecturer represented each department. Thus, initially, 20 lecturers participated in this study. Subsequently, ten lecturers were added, resulting in 30 lecturers in total selected by purposive sampling. Besides that, 67 mathematics education students taking the analytic geometry course were also involved in this study. They were also chosen by purposive sampling. All participants in this study had given their consent to take part in this study.

Data Collection

The data in this study were collected using tests and a questionnaire. The tests consist of the lecturer competence test and the student competence test. The lecturer competence test aims to assess lecturers’ knowledge of using blended learning before and after training; it includes planning, implementation, and assessment components. This test also assesses lecturers’ skills in designing lesson plans using Google Classroom, Google Forms, and YouTube. On the other hand, the student competence test intends to evaluate students’ understanding, applying, and analyzing abilities after lectures in the blended learning environment. It also aims to measure the effectiveness of blended learning lectures for improving students' mathematics learning outcomes. The test was designed by the researchers and tested for its validity and reliability. The test was administered online to students and took 60 minutes to complete. The test consists of four problems; each problem scores 25 points if correct and zero if incorrect. As such, the maximum score is 100 points. The post-test problems for students are presented below.

  1. A mosque dome has a diameter of 5 m. If the dome's outer surface is painted at the cost of Rp. 27,000.00/m2 and 2 m2 requires 1 kg of paint. How much does it cost, and how many kg of paint will you need to paint the dome?
  2. When the radius of a cone is increased by 26 cm, the increased volume of the cone is equal to its increased volume when the height of the cone is increased by 26 cm. If the initial height is 4 cm, determine the initial radius. How do you predict the answer? Please check.
  3. Given a cube ABCD.EFGH, B1 is the sphere outside the cube, and B2 is the sphere inside the cube. Anne has calculated that the ratio of the B1 and B2 is 2:1. What is your opinion? Is it right? Please check.
  4. Given a right triangle ABC with right-angled at A, BC2 = AB2 + AC2. Is the conclusion of the statement correct? Justify your answer.

In addition, the questionnaire was administered to investigate lecturers’ and students’ satisfaction with the blended learning model. It was adopted from Suana et al. (2017), which had been theoretically validated by three experts. The questionnaire consists of 18 items with five scales, from 1 (strongly disagree) to 5 (strongly agree) (Sugiono, 2017; Van Thao et al., 2021).

Research Procedures

This research was carried out for four weeks, from the middle of September to the middle of October 2022. This study has several activities, including focus group discussion, training, assessment, and implementation. In detail, the research procedures are described as follows.

Focus Group Discussion (FGD) on the Blended Learning Model

This activity was carried out to increase lecturers' knowledge of the blended learning process, including planning, implementing, and assessing blended learning. The FGDs were divided into two sessions, each conducted for a group of 15 lecturers. At the beginning of the meeting, the lecturers were given an initial test on understanding the blended learning model. Then, the lecturers were provided with guidance to prepare lesson plans by formulating learning objectives and developing appropriate materials to carry out the blended learning lectures. Further, the lecturers were given an understanding of the concept of designing, implementing, and assessing the lesson through blended learning models by providing information about platforms that could be used for blended learning. On this occasion, the lecturers were also given an understanding of the implementation of blended learning using Google Meet and Google Classroom. Furthermore, lecturers were provided with knowledge of a blended learning assessment using Google Forms and affective and psychomotor assessments by recording videos entered into YouTube. At the end of the activity, a final test was administered to determine the lecturers' understanding and knowledge of the concept of the blended learning model.

Training on the Application of Blended Learning

This activity aims to improve the lecturers’ skills in planning, implementing, and designing assessments in blended learning. The lecturers were requested to design one of the topics they taught. They were then trained to use Google Classroom, Google Forms, and YouTube for learning assessment. At the end of the activity, the lecturers were assessed to determine the level of their skills in using the platforms.

Assessment of the Lecturer's Comprehension After Training

The lecturers were given a final test and asked to fill out a questionnaire via Google Forms. The success of this activity is indicated by the average result of the final competence tests regarding the concept of blended learning and skills exceeding 80. Data on the test were quantitatively analyzed using descriptive statistics on a scale of 10 to 100. Table 1 presents an example of the three-week training activities on the blended learning model.

Blended Learning Implementation by Lecturers

After training, the lecturers were instructed to apply online-based learning to their respective students. In general, the departments have implemented a blended learning model for students. As such, the data on students’ learning outcomes and satisfaction were collected from 67 students, with satisfactory results. Students’ abilities consist of understanding, applying, and analyzing the lesson that has been taught.

Subtopic Knowledge and Skills Mode Time
Blended learning model activities FGD on understanding: Face to face 7 hours*
Blended learning Face to face 7 hours*
Blended learning planning concept Face to face 7hours*
Blended learning implementation concept Face to face 7 hours*
Blended learning assessment concept Face to face 7 hours*
The final test on lecturers’ knowledge Online (synchronous virtually) 3 days**
Lesson plan design Online (synchronous virtually) 3 days**
Using the Google Classroom and Google Meet applications Online (synchronous virtually) 3 days**
The use of Google Forms and YouTube applications Online (synchronous virtually) 3 days**
The final test on lecturers’ skills Online (synchronous virtually) 3 days**
Applying online-based learning to their respective students Online (asynchronous) 2 weeks
Table 1.Examples of training activities on blended learningNote: * and ** Done at the same time.Source: Training activities on blended learning (2022)

Data Analysis

As previously mentioned, the instruments of this study comprise lecturer and student competence tests and a questionnaire. The lecturer competence test was analyzed using descriptive statistics on a scale of 10 to 100. The student competence test was analyzed using a scale of 0 to 100. All quantitative data were analyzed using a standard deviation, mean, and percentage. While data on lecturers’ and students’ satisfaction with blended learning was analyzed using three categories: attractiveness, easiness, and benefit, developed by Suana et al. (2017).

Results

In the beginning, the lecturers attended blended learning training for one week. The training aims to assess lecturers' competence before and after training as well as their satisfaction with the blended learning model after treatment. The competence includes understanding the concepts of planning, implementing, and assessing blended learning and skills in a lesson plan design and using Google Classroom, Google Form, and YouTube. After training, the lecturers implemented blended learning in lectures for four meetings (two weeks). It attempts to determine students’ competence, including understanding, applying, and analyzing abilities concerning the lesson taught using a blended learning model. After the initial and final tests were administered to the lecturers, they were requested to complete the questionnaire distributed online via Google Forms. The test results are presented in Table 2 and Table 3.

N o. Competence Mean (N=30) Complete (%) Incomplete (%)
1. Planning blended learning 57.8 53.3 46.7
2. Implementation of blended learning 48.6 46.6 53.4
3. Assessment of blended learning 45.7 43.3 56.7
Average score 50.7 47.7 52.3
Table 2.Lecturers’ Competence on Initial tests regarding their knowledge of blended learningSource: Questionnaire distributed online via Google Forms (2022)
N o. Competence Average (N=30) Complete (%) Incomplete (%)
1. Planning of blended learning 87.9 100 0
2. Implementation of blended learning 85.9 100 0
3. Assessment ofBlended learning 86.5 100 0
Average score 86.8 100 0
Table 3.Lecturers’ competence on the final test regarding their knowledge of blended learningSource: Questionnaire distributed online via Google Forms (2022)

Table 2 shows that initially, the lecturers did not have sufficient knowledge about the blended learning model. It is evident from the average score of 50.7, meaning that only half of the lecturers understood the concept of the blended learning model. However, Table 3 reveals that after the training, the lecturers’ average final test score exceeds 80.00. One may conclude that the lecturers had good knowledge of blended learning models after treatment. The improvement in the lecturers’ knowledge is illustrated in Figure 1.

Figure 1.The results of the initial test and final test.

Lesson plan design Google Classroom Google Forms YouTube Overall (N=30)
Pre-test 53.4 (12.6) 52.7 (14.3) 48.7 (14.2) 39.7 (11.2) 48.6 (11.8)
Post-test 80.9 (16.7) 86.3 (15.5) 84.3 (16.8) 82.2 (16.5) 84.02 (15.7)
Information complete complete complete complete complete
Table 4.The Lecturers’ skills in blended learningSource: Results of lecturer’s skill in blended learning (2022)
Aspect Item Scale (1 to 5)
Score Average
Attractiveness Online discussion is an interesting activity 3.88 3.98
Practice questions are interesting to do online 4.03
Teaching materials are interesting to learn online 4.03
Overall, online classes with google classroom make lectures more attractive 3.81
Overall, the blended learning model with a pre-online–face-to-face–online learning mode is interesting to follow 4.16
Easiness Communicating with peers and lecturers in the online class can be done easily 3.83 3.59
I can easily participate in post-online learning 3.52
Learning tools in online classes can be found easily 3.85
Practice questions in the online class can be learned easily 3.56
I have difficulty with ICT/internet skills to participate in online learning in the blended learning model 3.38
Overall, ‘pre-online learning – face-to-face – Online learning can be easily followed without any significant obstacles 3.42
Benefit The blended learning system increased my interest in today's lectures 3.66 3.77
Online learning activities improve my concept mastery 3.88
Problem-solving activities improve my concepts mastery 4.02
I became more motivated to attend lectures using the blended learning model 3.82
Based on my experience, I want to give lectures with a blended learning approach 3.50
The blended learning model increases my self-directed 3.90
Overall, the blended learning approach is more beneficial than face-to-face learning 3.62
Table 5.The lecturers' satisfaction scores with the blended learning model.Source: Results of lecturers' satisfaction scores with the blended learning model (2022)

Table 4 presents the lecturers’ skills in carrying out the blended learning process by utilizing various platforms yielded a score of 84.02. It signifies that the lecturers can use multiple platforms in the blended learning process. Moreover, Table 5 illustrates that the lecturers' satisfaction with blended learning is positive in all aspects. Out of the three aspects, attractiveness receives the highest score. Furthermore, the results of students’ abilities, including understanding, applying, and analyzing the lesson in the analytic geometry course, and their satisfaction are presented in Table 6.

No Measured variables (N=67)
Learning Outcome Score Satisfaction Score
1. Understanding 68.4 (15.4) Attractiveness 4.21
2. Applying 58.7 (16.6) Easiness 4.03
3. Analyzing 47.2 (16.2) Benefit 4.13
Average 58.1 (16.7) 4.12
Table 6.The results of test scores and students’ satisfaction with blended learningSource: Results of test scores and students’ satisfaction with blended learning (202

Discussion

The Lecturers' Competence in Blended Learning and their Satisfaction

This study has revealed the competence of lecturers at Jabal Ghafur University regarding the blended learning process. Blended learning is a learning process that combines a face-to-face system with an online learning process. Thus, blended learning is considered suitable during a pandemic or post-pandemic period. The results showed that the lecturers' knowledge and skills in blended learning during the post-pandemic COVID-19 had increased. In the FGD, the lecturers also responded positively to the concept of lectures and learning design using various platforms in blended learning. On average, the knowledge score obtained by the lecturers on the initial test is 50.7, and on the final test is 86.8. Meanwhile, regarding the blended learning skills, the lecturers yielded a score of 48.6 on the initial test and 84.02 on the final test. This shows a significant increase in both knowledge and skills of lecturers concerning the blended learning process.

Even though nearly all participants encountered internet connection-related issues, their satisfaction with attractiveness, ease of use, and benefits of the learning design and materials were all considered in the "good" category (3.78). It reveals a correlation between the knowledge and skills of lecturers in the blended learning process and their satisfaction with using blended learning in the learning process. Among the three aspects, the attractiveness aspect scores higher than the other two. The findings also indicate that the lecturers' achievements in training and their satisfaction using a blended learning model platform become high when problems are solved via the internet. This finding is in line with the increase in student’s cognitive abilities, as Chen and Yao (2016) stated that the level of students' perceived satisfaction with blended learning had a vital role in its effectiveness. This finding is similar to the previous research, reporting that blended learning is more effective in improving students’ learning outcomes (Suana et al., 2017). Finally, it is recommended that lecturers solve problems in their fields of study by integrating lecture models, such as problem-based and project-based learning, into a blended learning system.

The Students’ Competence and Satisfaction after using Blended Learning

After training, lecturers were instructed to apply online-based learning to their students. However, the departments have generally implemented the blended learning model for students in lectures. Student achievement and satisfaction data were collected from 67 participants, with satisfactory results. This is because lecturers' use of the internet in the learning process related to their satisfaction of attractiveness, ease of use, and benefits of the learning design and materials were all in a ‘good’ category. It is aligned with Gunes (2019), who revealed that university students favored the blended learning process and gave positive responses. This result also follows the statement of the ICT experts who state that integrating technology in education is a positive step and technology has a key role in changing the classroom environment–and indeed entire schools–since it encourages outcome-oriented learning (Bwalya, 2019). Thus, school administration and national educational policies have set out to integrate ICT into the learning process (Passey, 2011).

We recall that three students' abilities were observed in this study: understanding, applying and analyzing. The 2013 curriculum emphasizes students' ability to be able to solve problems in everyday life. Problem-solving is an aspect of higher-order thinking skills (HOTs). HOTs is a thinking skill that requires the ability to remember and other abilities, such as the ability to understand, apply, analyze, and use that information as a solution in solving problems (Sara et al., 2020). One cognitive aspect that needs to be improved in learning mathematics is the ability to understand. Understanding ability is an initial ability that students should master to achieve other higher cognitive abilities (National Council of Teachers of Mathematics [NCTM], 2000). The importance of this mathematical understanding ability is stated in NCTM (2000). To wit, the vision in school mathematics learning is that students understand what they are learning. Therefore, the requirements for a student can be said to be proficient in mathematics if they master mathematical concepts (NCTM, 2000).

According to Bloom's Taxonomy, the abilities to understand and apply constitute lower-order thinking, but the abilities to analyze, evaluate and create are included in HOTs (Anderson et al., 2001). When students solve a higher-order thinking problem, the students begin to understand the problem and think of a solution, then make connections between various variables in the problem, elaborate a solution strategy, and perform calculations (Ansari & Sulastri, 2018). In this current study, the four problems given to students are considered HOT problems. Regarding students' ability to solve problems, this study reveals that the understanding ability (68.4) is higher than other abilities, applying ability (58.7), and analyzing ability (47.2). This means that students' abilities are only at moderate and low levels (58.1 on average). Nalurita et al. (2013) reported that in learning, students tend to be given problems only at the level of understanding and little application, which may explain why their higher-order thinking skills do not develop. This is not comparable to their satisfaction in applying the blended learning model, which is considered in the good category.

Conclusion

The results showed an increase in the lecturers' knowledge and skills in using the blended learning model. The findings also revealed that the blended learning training in improving lecturer competence was considered effective and impacted their students’ learning outcomes. Furthermore, the lecturers’ and students’ satisfaction with the blended learning model is positive. However, student satisfaction with the blended learning model is not comparable to the scores of understanding, applying, and analyzing abilities they get. This means that their satisfaction is in the good category, but their cognitive abilities are in the medium and low categories. The researchers hope that students from various universities can use the blended learning model as a current learning method to support the current curriculum. This blended learning activity can be continued periodically to attract students’ interest in learning during the post-COVID-19 pandemic.

Acknowledgment Statement: The authors would like to thank the reviewers for providing comments in helping this manuscript to completion.

Conflicts of Interest: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Author contribution statements: Bansu Irianto Ansari, Junaidi Junaidi, and Abdul Rahman contribute to conceiving, designing, conducting the research, analyzing the data, and drafting the initial manuscript. Suci Maulina and Ilham Kamaruddin deal with methodology and data analysis. Nanda Saputra and Bansu Irianto Ansari focus on collecting data, drafting, formatting, and analysing, and Herman Herman is responsible for drafting, reviewers responding, and methodology.

Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Ethical Consideration statement: Not applicable. This study did not involve human and animal studies.

Data Availability Statement:Data is available at request. Please contact the corresponding author for any additional information on data access or usage.

Disclaimer: The views and opinions expressed in this article are those of the author(s) and contributor(s) and do not necessarily reflect JICC's or editors' official policy or position. All liability for harm done to individuals or property as a result of any ideas, methods, instructions, or products mentioned in the content is expressly disclaimed.