Indonesian health professions students’ perceptions toward an interprofessional education program: Findings after five years of implementation

Background: The Interprofessional Education (IPE) program is important for preparing health professions students to provide future interprofessional collaborative practice. The Universitas Indonesia Health Sciences Cluster has been implementing the IPE program since 2013. A comprehensive evaluation is required following the implementation of the IPE program. The aim of the study was to evaluate the IPE course based on perceptions of undergraduate students at the Universitas Indonesia from 2013 to 2017. Methods: A mixed-methods study utilizing a semi-structured questionnaire was conducted with first year students following the completion of the first stage of the IPE course. A total of 2355 students (56.35%) from the 2013–2017 academic years completed the questionnaire. Results: The results showed that students’ perceptions of the IPE course improved each year on the domains of clear and relevant learning objectives, studentcentered teaching methods, staff support, and supporting infrastructures. However, some room for improvement was identified, such as the need for tutors to have a more neutral attitude toward every student despite their background differences. Conclusion: The IPE course has been well perceived by students because of its comprehensive instructional design and principles of IPE that are implemented in the curriculum. The continuous cycles of improvement to maintain the quality of the IPE program will remain a challenge.

Indonesian health professions students' perceptions toward an interprofessional Indonesian health professions students' perceptions toward an interprofessional education program: Findings after five years of implementation education program: Findings after five years of implementation Cover Page Footnote Cover Page Footnote

Introduction
Interprofessional collaborative care is necessary for high-quality patient care. It improves compliance of patients with chronic diseases, increases patient-referral and patient care system efficiency, and reduces complications and mortality. [1][2][3][4] Therefore, health professionals need to communicate effectively, recognize the roles and responsibilities of each team member, reflect on their capabilities, collaborate effectively, and conduct ethical health practice and patient-centered care as an interprofessional team. 5 The attempts to equip health professionals with these abilities in an interprofessional collaborative practice can be made explicitly in the health professions' education institutions through Interprofessional Education (IPE).
IPE is a curriculum approach in which students from various health professional backgrounds collaborate in one setting to learn from each other, recognize and appreciate each other's role, to enhance collaboration and teamwork skills for better health care. 6 Merely putting students from different health professions with diverse educational backgrounds in one course without any emphasis on the achievement of interprofessional competencies is not considered IPE. 7 Curriculum developers and organizers should pay attention to several factors that can influence the implementation of IPE, such as the characteristics of students and teachers, 8 learning outcomes agreed to by the stakeholders, and the availability of learning experiences in a real-world setting. 9 The IPE curriculum should consider learning outcomes, background, subject content, teaching-learning methods, evaluation format, and required resources. 5,10 Some learning principles, such as collaborative, egalitarian, group-directed, experiential, reflective, and applied Indonesian students' perceptions on IPE program 105 Makara J Health Res.
August 2020 | Vol. 24 | No. 2 learning, are the cornerstones of IPE implementation. 11 These IPE learning principles are in concordance with adult learning theory, where students are encouraged to be responsible to direct their learning, learn from experiences, and focus on problem solving. 12 In undergraduate IPE, the relevance of the topics and contexts being highlighted in the implementation is critical since it can help students have a realistic nature of teamwork collaboration, as reflected in clinical practice. 9,13,14 Another essential part of learning in IPE is the reflection on the learning processes and related competencies. 14,15 The outcomes of the IPE curricula can be evaluated using Kirkpatrick's framework. 16 This framework is widely used in program and curriculum evaluation. It consists of four levels of evaluation: reaction or satisfaction of students (level 1), learning or outcomes of the course based on the assessment results (level 2), results or evaluation on the implementation of the competencies/behavior in a real setting (level 3), impact or health/other relevant outcomes in real practice following the placement of the graduates completing the program/curriculum. 16 A previous study conducted a systematic review of 46 studies. 13 The outcomes of the IPE curricula were evaluated at all levels and showed: learner reactions (65%), changes to learners' attitude (56%), changes to knowledge/skills (56%), changes in individual participants' behavior (43%), changes to organizational practice (30%), and changes patient/client care outcomes (24%). The systematic reviews included studies that mainly originated from North America, Canada, and European countries. Some studies in Asian countries, i.e., Japan, also evaluated the implementation of IPE from the change in the attitudes of different stakeholders toward IPE: medical schools' deans, 17 first-and third-year undergraduate medical students, 18 undergraduate students, and alumni. 19 Studies on the implementation of the IPE curriculum have been completed in different settings, including in Indonesia. Studies on the evaluation of student readiness to participate in IPE involving medical, nursing, pharmacy, dentistry, and midwifery students show that students are ready for IPE implementation. This is supported by students' cognitive abilities, early exposure to IPE, socialization impact, 20 the consideration of students' motivation, and implementation of best practices in the IPE learning processes. 21 Further studies on the implementation of IPE in some health professions education schools in Indonesia also highlight that students learn interprofessional competencies in the longitudinal community-based program [22][23] and disaster management course. 24 An evaluation of students' perceptions of their interprofessional competencies following IPE in the clinical stage involving health professions education schools in a university in Indonesia also shows good results and emphasizes the recommendation to incorporate a well-structured IPE curriculum in the clinical year and at a later stage of the health professions education. 25 The studies conducted in Indonesia, as mentioned before, are mostly done as one-time studies following completion of the programs. Given the contextual challenges of IPE curriculum implementation and the importance of continuous improvement of such a program, a study that shows evaluation over several years is necessary. Therefore, this current study aims to evaluate the IPE course performance based on perceptions of undergraduate health professions' students at the Universitas Indonesia from 2013 to 2017 on the domains of course objectives, teaching methods, staff support, supporting infrastructure of the course, and course management, and discuss in-depth the changes and adaptations in the IPE curriculum following the evaluation.

Methods
Context. The health sciences cluster at the Universitas Indonesia has been implementing IPE course for all five faculties: medicine, dentistry, nursing, pharmacy, and public health since the 2012/2013 academic year. The course is divided into two stages, the first in semester two and the second in the last semester. The course in the first stage focuses on the achievement of all learning outcomes in a more controlled and/or simulated condition(s). Around twenty students from different faculties are grouped and engage in various learning activities: games for improving the group dynamics, collaborative learning sessions, self-reflection, teambased learning, group projects, and presentations. In the second stage, the learning activities are a combination of field practices in community health centers and group projects. This context provided students with the opportunity to engage with other students from different faculties and future professional settings. The course organizers, comprising members of the teaching staff from the five faculties, created the guidebook for this course. At the end of the first stage of the IPE, students completed a semi-structured questionnaire that reflected their reaction to and satisfaction with the course. This evaluation can be considered the level 1 evaluation. 16 Study design. This study employed a mixed-method with an observational retrospective design by obtaining and collating the first-year students' evaluation questionnaire data following the completion of the first stage of Health Interprofessional Collaboration and Teamwork course from the academic year 2013-2017. The response rate in 2013-2017 was 56.35% (2355 out of 4139 students). Since student participation was voluntary, the variation in response rate per year might be influenced by the consistency of course organizers to remind the students. Table 1 shows the distribution of 0.6 *respondents did not fill in their faculty of origin respondents based on the academic year and the faculty of origin. The highest responses were in 2014 (30.3%), and the lowest was 10.7% in 2015. While most respondents were from the faculty of public health (29.5%), the faculty of dentistry contributed the least respondents (11.3%).

Instrument.
The semi-structured anonymous questionnaire consisted of 21 questions that were categorized into several domains and related to the instructional design of an educational program ( Table  2). The domains were the course's objectives, teaching methods, staff support, infrastructure to support learning, and course management. For each domain, several statement items and students were asked to indicate their level of agreement with each statement using a 4-point Likert scale from "strongly disagree" to "strongly agree." At the end of the questionnaire, a place was provided for students to write comments, and demographic data (faculty of origin and student batch), was provided. The questionnaire was distributed online using the university learning management system called the Student Centered e-Learning Environment.
Data analysis. The score of students' perceptions of the course performance was presented as the mean, minimum, maximum, and standard deviation. Different perceptions of the students toward the course were analyzed based on the faculty of origin and student batch using SPSS version 22. Qualitative data obtained from the open question was analyzed thematically by encoding the responses into predefined themes.
Ethical consideration. Participation in this study was voluntary, and it did not affect student assessment in any way. This study has been approved by The Research Ethics Committee of the Faculty of Nursing Universitas Indonesia (No.79/UN2.F12.D/HKP.02.04/2018).

Results
The distribution of students' responses across all items in the questionnaire was grouped into several domains, as elaborated in Table 3. Students' active comments were categorized and fell under the domain of either learning objectives, instructional design, teaching methods, the role of staff, facilities for learning, or management of the course. These comments are presented after the quantitative results to highlight the prominent results in the quantitative part and further elaborate them.
Perceptions of course objectives. Most students in 2013 disagreed with the statement that the objectives of the course could be well understood. While in the next four years, students mostly agreed that they had a good understanding of the course objectives. Similar trends were found when students were asked about their understanding of the course instructional design, and when they had to indicate the similarities between actual and planned learning activities.
Students from batch 2013 revealed that they did not derive any significant benefits from this course. The content was too normative and abstract, thus limiting its application. One of the comments is as follows: "Materials being discussed were mostly very basic and normative in nature; [It was] not [applied], therefore students cannot understand [how to apply or use it in] the real situation of collaboration because it was difficult to find out about it from the references" (q2013_147) More positive comments were provided by students of the next batches (example below). They thought that the course was conferred with direct benefits on students because they could collaborate and understand the roles of other professions.

"The course was good at providing the instructional design. Students [must] understand the roles of other health professions. Hopefully, the collaboration can improve and become the lessons for application in the workplace later" (q2014_561)
Perceptions of teaching methods. The students of 2013 were almost evenly divided with respect to those who agreed and disagreed regarding the time allocated for group discussion. For other items in the domain (as presented in Table 3), most students of 2013 (between 66.3-75.7% of the total students in that year) disagreed on the statements.

Indonesian students' perceptions on IPE program 107
Makara J Health Res.
August 2020 | Vol. 24 | No. 2 Learning activities were conducted as planned Students' perceptions of teaching methods used in the course Q4 There was sufficient time allocated for group discussion Q5 Steps within group discussion were well performed Q6 There was a good teamwork during group discussion Q7 Group discussion was useful for learning Q9 There was a sufficient time allocated for plenary presentation Q10 Plenary presentation activities were useful Q11 Practicum activities was useful to achieve learning objectives Q15 There was a sufficient time allocated for self-study activities Students' perceptions on staff support within the course Q8 Tutor facilitated group discussion well Q12 Practicum instructor guided the practicum well Q19 There was a good communication between course organizers and students Q20 Secretariat staff were helpful in running course activities Students' perceptions of the infrastructure provided in the course Q13 Practicum guidebook was easily understood Q14 E-learning system was useful Q16 References suggested covered all topics in the course Q17 References suggested were available Q18 There was sufficient equipment for practicum activities Students' perceptions of the management of the course Q21 In general, course management was good enough   From 2014 to 2017, the results fluctuated, but the trends remained consistent. Most students felt that the steps within the group discussions were well conducted, and there was good teamwork in the group. Over 50% of students thought that group discussion and plenary presentations were useful activities for learning. The practicum activity, in the form of a group project, was viewed as helpful for their learning. In particular, more than 85% of the 2017 students considered the practicum as a useful learning opportunity.
Although the teaching methods were perceived as relatively good, the students indicated areas that needed improvement. Some plenary presentation sessions were not effective and conducive since they involved too many students. It was challenging to supervise students closely, and many students did not pay attention. The teachers were also expected to provide feedback during the plenary session. Two suggestions are as follows: "Input for the future, during plenary students need to be supervised more closely because many students did not follow the plenary" (q2015_100)

"Other than that [stated above]… during plenary, there should be clarification from teachers" (q2015_190)
Since 2016, the course implemented team-based learning as a form of group discussion. Students commented that during Team-Based Learning (TBL), groups continued to make decisions or provide answers based on voting, not having a discussion. For example:

"In my opinion, the TBL conducted was a little bit ineffective because there were still groups which made the decisions based on voting, not discussion" (q2016_278)
Perceptions of staff support. There are several types of staff involved in the course, including a tutor, practicum instructor, course organizer, and secretarial staff. Students of 2013 consistently disagreed on each item in the domain. More than 50% of students felt that tutors did not facilitate the group discussion well, and there was a lack of good communication between the course organizers and students. They also mentioned the lack of support from the secretarial staff. On the contrary, students from 2014 to 2017 indicated otherwise.
Concerning the roles of tutors and instructors, students felt that sometimes there was a lack of communication between the tutors and course organizers that led to misperceptions of the tutors, especially regarding the tasks to be done. The students highlighted this issue as follows: "Often, there were misperceptions and differences in the task instructions. The students became confused and had to ask the same things repeatedly to the tutors" (q2015_55) Another problem with the tutors, according to the students, was that some tutors did not guide students' discussion. The students further elaborated that the tutors needed to create a conducive and interactive learning atmosphere and be neutral to every health profession student, as illustrated by the example of a comment provided below.
"Perhaps the tutors can be more interactive with students to make it less boring and increase students' spirits." (q2016_11) The course organizers consistently reminded the students about their tasks and permitted the students to obtain the information they needed. However, during the group project, some students felt that communication needed improvement because there were misperceptions about certain aspects of the group project.

"My class got a place for field work in restaurant A; however, when we got there, it turned out that there was no communication yet between the restaurant and the course organizer" (q2017_448)
Perceptions of supporting infrastructures of the course. Running a course requires that particular infrastructures are available to support each learning activity. For this course, the main infrastructures are the e-learning system, practicum guidebook and equipment, and relevant references. The students mostly agreed on the understandability of the practicum guidebook and the sufficiency of the practicum equipment. However, the year 2013 students considered them very differently by indicating their disagreement on all items under this domain (except for practicum-related items since there was no practicum for students of 2013).
The course has used an e-learning system; however, at some places on the campus, the quality of the wireless internet was poor. References were provided in the course, but students felt that they preferred if some references were in the Indonesian language. Also, some references needed to be updated. An example of a student comment is as follows:

"Please make the references easier [to read] by providing references in [the] Indonesian language and do not cover the same materials, because many references have the same content" (q2017_331)
Perceptions of course management. For the overall management of the course, most students from 2014 to 2017 provided their agreement that it was good enough.

Indonesian students' perceptions on IPE program 109
Makara J Health Res.
August 2020 | Vol. 24 | No. 2 However, students of 2013 indicated otherwise. The sum of students who agreed and strongly agreed increased from 2014 to 2017, except for 2016, when there was a slight decrease (around 4%), before increasing again to reach 95.5% in 2017.
Most students felt that the course was well managed, considering the high number of students who participated in the course every year. This opinion is exemplified by the following comment.
"In my opinion I think the overall management of the course is good enough, related to the guidebook, evaluation system, program administration, facilities, and more importantly, the learning sessions in the course are very interesting and should be attended as experience in collaborating with future health professionals" (q2017_186) Each response in the questionnaire was then treated like a Likert scale, and a score was assigned to each possible response with a maximum raw score of 80 (item number 21 was excluded in the total score calculation because it was more of a general statement that did not correspond to any domain). Then, the raw score was converted, so that 80 is equal to 100. Table 4 below provides the distribution of scores of students' perceptions of course performance based on student batch and faculty of origin.
There was a significant increase in mean scores from batch 2013 to 2014. From 2014 onwards, the increase was not as much as before, and there was even a very slight decrease from 2014 to 2015, only 0.09 points. The number of respondents in 2015 was also very low. The mean score in 2017 was the highest among all with 76.43, coming from 589 respondents in total. Based on faculty of origin, it was found that public health students had the highest mean score of 69.90, whereas the lowest mean score was from dentistry students with a score of 65.44.

Discussion
This study applied the first level of Kirkpatrick's framework 14 to evaluate an educational course by exploring the perceptions of students who have completed the course in five areas. These five areas are the course objectives, teaching methods, staff support, supporting infrastructure, and course management.
Course objectives. The learning objectives of the IPE course were explained in the first week by the course organizers and were written in the guidebook. Hence, they can be easily understood by students. The student from batch 2013 disagreed with the statement that the learning objectives could be well understood, which might be because it was the first time that the IPE course was implemented. The course organizers realized that since the IPE course is undertaken at the undergraduate level, it can enhance attitudes, skills, and knowledge for the collaborative practice among healthcare students. 26 It also provides opportunities for students from different professions to interact and know each other. However, there may be complexities in learning interprofessional communication, leadership, teamwork, and conflict management skills when the course is conducted with a large cohort of students, as happened in this study. 13 However, despite the limitations we had in the first year, we launched the IPE course.
Teaching methods. The results demonstrated that students perceived the learning activities conducted in the course to be in line with what was written in the course guidebook. The students from batch 2014 onwards felt that the course was beneficial for them because they learned to understand different roles of different professions and how to collaborate. Therefore, there was an overlap between the declared curriculum in the guidebook, the taught curriculum in the class, and the learned curriculum by the students, 27 as expected. However, the continued anticipation against the existence of a hidden curriculum is required because it cannot be denied that students also learn from observing role models. 28 Especially when they have entered clinical practice, students need to see that teamwork takes place. Otherwise, interprofessional learning in the previous stage would be meaningless.
The course implemented student-centered active learning methods. A newer method, such as teambased learning, was also introduced in 2016. The use of small group active learning methods is based on adult learning theory, where students are asked to participate actively to solve the problem presented, drawing from the knowledge they have obtained, and their own experiences. Teaching methods in the IPE course should include principles of adult learning theory and contextual, situated learning. 5,29 Previous predictions also emphasized the need for authenticity in interprofessional learning. 8,13 The current IPE course improved the teaching strategies by immersing students in a real situation in which they had to assess and analyze common health problems in communities throughout the university. This limited real-life experience, although not yet in the clinical setting, had been considered beneficial in improving students' perceptions of this course.
Staff support. The IPE program for many health professions students required the participation of a large number and a wide range of health professions education teachers. However, the involvement of teachers with different backgrounds requires adequate knowledge and skills in IPE. 30 Teachers need to have a positive attitude toward IPE before they could become role models for students. Facilitation skills and faculty development are important in the IPE implementation. 13 The IPE developers at the Universitas Indonesia realized that involving teachers was not merely gathering them since their clinical and academic experiences alone might not be enough for IPE, and none of them were exposed to a systematic IPE program before. 31 The students noted that tutors should have tried harder to encourage a positive learning environment. The fact that some tutors were "picking sides" to students from certain professional backgrounds was considered a challenge in the current study. In addition to IPE core knowledge and facilitation skills, it is imperative that the tutors can provide a positive and equal learning environment. 32 Negative attitudes held by teachers can be revealed to students through unconscious cues and non-verbal behaviors. 33 Therefore, faculty development at the Universitas Indonesia is necessary for teachers to learn how to be role models in interprofessional leadership, which is strongly suggested as one of the key successes in the IPE development. 30,31 Supporting infrastructure of the course. Successful implementation of the course also depends on infrastructures, such as adequate discussion rooms, tables and chairs, availability of audiovisual, proper internet connection, and adequate references. School administrators play essential roles in curriculum construction, budget allocation, and decision-making regarding infrastructure construction. 17 In the five years of IPE implementation, students' perceptions of the infrastructure provided in the course improved from 2013 to 2017. This improvement is partly because of the role of the course organizers who evaluated the course each year and reviewed the evaluation results with the university leaders.
The IPE in the present study was conducted at the very early stage of health professions education program (year 1), and students felt the benefits of learning how to collaborate and perform good teamwork. According to the concept of professional identity formation as "an ongoing process of interpretation and re-interpretation of experiences", 34 introducing IPE early means embedding the interprofessional identity, as an integrated process of professional identity development, 35 and avoiding stereotyping. 36 The evidence has not yet provided any strong recommendations on the best timing of IPE in the health professions education program or how the timing will influence the professional and interprofessional identity formation. 36 However, by conducting the course at the early stage, students could have "socialization" 37 as one of the key processes in their professional development.
Course management. Students' perceptions of the management of this course improved each year. This perception stems from their perceptions of each specific domain of the course, such as teaching methods and staff support, which have been improving from 2013 to 2017. Despite the very large number of students participating in the course, the course organizer team has managed to run the course well. The course organizers have considered all the intertwining factors related to successful IPE implementation. 38 These factors are learner-focused factors, facilitator focused factors, and organization focused factors. For all these interrelated factors to work cohesively, proper coordination and communication are necessary. A systematic review also showed that dedicated leaders are essential in the running of IPE. 13 Therefore, it is imperative to nurture more teachers to be committed to IPE for the sustainability of the program.
Limitations of the study. The authors are aware of the study limitations. First, the relatively low response rate might lead to an underrepresentation of students' perceptions and sampling bias. Second, given the nature of the current evaluation, which used students' perceptions as the primary data source, the authors realized that future study is required to strengthen the evidence of the IPE program by evaluating the attainment of learning outcomes and performance in real practice. However, this study reflects students' perceptions over five years of IPE implementation. In addition, the evaluation focused on the ongoing process of IPE implementation; hence, the feedback was taken further into actions for improvement.

Conclusion
The current study showed that the IPE course had been well perceived by health professions students. The program has been developed according to the components of instructional design and principles of IPE. This includes the application of student-centered active learning strategies. However, a continuous cycle of improvement is the key for successful IPE. Thus, obtaining stakeholders' feedback and translating it into actions for improvement should be a best practice in the course delivery. Faculty development is also paramount to enable teachers to be role models in interprofessional collaboration.