• If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

Evaluation Report


Introduction

The Personalised Curriculum Creation through Coaching (PC3) project originally aimed to develop a process, together with supporting technology structure and necessary regulatory change, to allow students to design their own curriculum, based on their learning needs, supported by coaching. However, seismic changes in institutional management from almost the start of the project, as discussed in the Institutional Story, meant that we had to make considerable changes throughout the project to both the focus of activities and evaluation of those activities.

 

The project initially started with a top-down approach, supported by the then current institutional policies which were moving towards flexible learning frameworks. There was a recognition that not all students wanted to or were able to follow the traditional routes for gaining further qualifications. The initial project idea was to develop a core module onto which students seeking to develop a personalised course would enrol. This module, Personalised Learning through Coaching (PLC), would then provide them with the skills and knowledge, via the use of coaching principles and supported via a number of technologies, to identify what further learning they needed to attain their personal or professional goals. As part of the assessment for this module students would produce a learning plan of the modules they required from across the institution for the qualification they want to achieve. The initial focus here was Masters level provision with the intention of moving to undergraduate provision as the project progressed.

 

Due to substantial changes in both institutional policy and structure the initial plan to roll out and evaluate this module was abandoned. The project refocused on a number of activities working directly with staff and students to embed coaching principles directly into existing modules and courses. These activities were developed inline with staff requests and student needs giving the project the ability to follow the trail of interest in the use of coaching and take advantage of serendipitous opportunities. This effectively switched the project to a bottom-up approach enabling it to more closely evaluate the impact of coaching principles on both staff and students. A more flexible approach to the evaluation process was also adopted, akin to action research, directing the focus of investigation and evidence gathering towards areas of interest that were generated by the process.

 

Evaluation Process

The project applied two different approaches to evaluating the process and outcomes although they both applied the same set of methods. The first approach, used to evaluate the PLC module, was well defined and followed more traditional methods. It is represented in the project's initial evaluation plan. Data was captured from student assessments deigned to provide the necessary evaluation data through a series of surveys and reflective questions on their experiences with coaching and the technology. The intention here was to continue to capture data from the students via regular email reflections and surveys over the course of their personalised course. This latter half was dropped due to poor student numbers and changes in university policy.

 

The second approach was more flexible and less well defined at the outset. The intention was to be able to move with the interests and needs of the staff and students as they applied coaching in a number of different ways. Methods were used where it appeared most effective for capturing the type of data that would provide evidence towards the research questions. Using a series of questionnaires, group discussions and interviews (semi-structured and video) to capture the data, relevant and interesting aspects were identified in the interviews or group discussions which were further explored in the surveys and visa-versa. This generated qualitative data from a number of perspectives which was supported by opinion ratings gathered in the surveys. This approach was used to capture the thoughts and ideas of a number of staff who had attended the coaching workshops, exploring how they were using coaching and which aspects they embed in their modules and courses. It was also used throughout the Coaching Ambassador initiative to track their ideas for the promotion of coaching and capture their own experiences of coaching. 

 

Assumptions 

Our assumption was that by using coaching principles of the students would be more confident of what they wanted from their educational experience and more confident of the decisions that they made during that experience. Resulting in students taking more responsibility and ownership for their learning process and becoming more self-aware of how to utilise their strengths and overcome challenges. 

 

Stakeholders

The stakeholders were considered to be the coaching team, the administrative staff, course teams, the institution and the students. During the lifetime of the project the emphases on these stakeholders shifted from administration and course teams towards individual staff and students. The initial plan focused on evaluating the module and its role out across a number of faculties. This was to be done by considering the cohorts involved, the administrative process required, the effectiveness of the coaching team and the process of negotiation required  

 

Due to changes to the project documented elsewhere (see Institutional Story), the stakeholders actually involved in the project, how they were involved and the evaluation methods used with each group are summarised in Figure 1. 

 

Figure 1:  Links between activities, stakeholders involved and research methods used with each 

 

Methods

Three methods were used throughout the project to gather evidence. These were surveys, semi-structured interviews and student assessments. To gather evidence of how students perceived the effectiveness of coaching principles and the technologies they used the module assessment was designed to capture their reflections on these aspects. This method was used both for the PLC module and for some parts of the assessments included in sports business management course. There were a number of reasons for using assessment data as part of the evaluation. One of the reasons for using student assessments was to reduce the burden on the students asking them to complete separate evaluation material. The assessments also provided a genuine opportunity for students to reflect on both how coaching had impacted on their learning and how they had used the technology to support that.

 

To capture the process undertaken by the coaching ambassadors each of the group sessions was recorded and reviewed to identify key areas of interest. While not analysed directly, due to some of the discussions being more personal to the students involved, these recordings indicated where further questions could be asked to dig deeper into their perceptions of coaching and how they thought it could be presented to future students. From this a number of surveys were designed and completed by students between meetings to capture the individual stories of the students. The main reason for using this process was that it was uncertain at the beginning of the coaching ambassador initiative what direction the students would take. By using this process of session review and survey unexpected perceptions or ideas could be explored. The surveys also gave the students an opportunity to express their individual sorts on particular concepts and ideas that they may not have been so willing to share within the group sessions.

 

The final method used throughout this project was that of a semi-structured interview. The aim of these interviews was to capture what individual members of staff wanted to improve in their module course and their expectations regarding the use of coaching practice. Questions asked these interviews were generally along the same themes but not necessarily the same questions. This was primarily due to the migrating change of direction the project as a whole as well as the growing understanding of the interviewer. One of the benefits of using interviews over surveys is the generation of much more expressive qualitative data. It also provides the interviewer with the opportunity to ask questions about unexpected aspect that come to light during the interview process.

 

To complement the written stories of both staff and students a number of video shorts were produced which expressed how staff and students perceived coaching and the potential benefits within an educational setting. This anecdotal evidence can be viewed here.

 

In the event, the evaluation of the project was captured through case studies, so the main findings are discussed in that context and summarised.

 

Phase 1: Developing Flexible Curriculum

Case study: Personalised Learning through Coaching module

 

Phase 2: Working with staff

Case Study: Sport Business Management

Mini Case study: Media students

 

Phase 3: Working with Students 

Case Study: Coaching Ambassadors

 

Conclusion

Each of the case studies presents specific conclusions in relation what each study sought to explore. Here we summarise the key benefits and challenges.

 

Benefits have been delivered to both staff and students through the project, with some of these being intensely personal: changing perspectives and behaviour of individuals who have engaged in coaching, with subsequent impact on others. More than 420 students have benefited from coaching at some level through the project, with over 150 receiving significant coaching training. Evidence from the Sports Management module indicates an improvement in the quality of the assessment in the year that coaching was introduced, with students becoming more engaged in their learning and, as a result, increasing their grades. Students found the process helped them to reflect on their learning more deeply and we had several examples of them transferring the principles of coaching from one area of their learning and life to another.  Students reported increase in confidence as one of the outcomes. Staff using coaching within their curriculum saw better engagement with students and, in some cases, a reduction in the time they personally spent supporting students, as peer coaching became established, although there was an initial investment in time to train the peer coaches.

 

However, the use of coaching raises a number of challenges which need to be addressed. Students and staff reported the importance of education about coaching on the success of it: students needed to understand the nature and benefits of coaching otherwise their expectations of getting "answers" could cause frustration. Once understood, students invariably commented on the benefits of the coaching approach.  This ongoing training need for both staff and students raises a sustainability challenge: coaching needs to become embedded into the mainstream curriculum and staff development programme to be widely effective. Introducing coaching to students early in their university career is recommended so that they see self-directed learning as the norm for university.  Single coaching interventions are less valuable than an ongoing embedded programme. 

 

Technology can provide valuable support for the coaching process, with students successfully using Facebook and messaging to support their peer coaching and both audio and video based coaching having a role in supporting distance coaching. In the choice of technology it is important to ensure that all parties are comfortable and familiar with the system being used: offering students choice is appropriate. 

 

 

 

 

Comments (0)

You don't have permission to comment on this page.