SCHOLAR Evaluation Report - Executive Summary
Introduction
As part of the national evaluation of projects supported through the Scottish Executive Education Department's Future Learning and Teaching Programme (FLaT), the Quality in Education Centre, University of Strathclyde was commissioned to evaluate the SCHOLAR programme.
Background
The SCHOLAR programme, launched in June 2000 by Heriot-Watt University, offers an online learning programme to schools and colleges in a range of subjects at Higher, Advanced Higher and Higher National levels. The programme set out to provide common educational resources and a 'virtual college' support network in order to improve student choice and flexibility and help students progress between school, college and university. The key stakeholders are members of staff in local authorities, schools (state and independent), further education colleges and the higher education sector (Heriot-Watt University), as well as the students within these institutions. The partnership approach to the implementation of the programme aims to create links between the students' study in their final years at school and in further and higher education. The programme began with a pilot phase involving four local authorities.
Aims
The evaluation aimed to determine the impact of SCHOLAR on the learning experience and achievements of students in a range of subjects in schools and further education colleges and its impact on the teaching strategies of teachers and lecturers. In relation to this overall aim the evaluation sought to:
- determine the impact and value of e-learning on those involved
- determine how the management of organisational processes within the partner institutions/authorities is meeting the needs of an e-learning approach
- identify patterns of usage, experiences of and attitudes to e-learning
- evaluate the impact SCHOLAR is making at the school/FE/HE interface.
Methodology
The evaluation was undertaken in the academic session 2002-2003 and conducted in two main phases. Quantitative and qualitative data were gathered. Phase 1 included an initial familiarisation period when documentation from the SCHOLAR website and from key members of the SCHOLAR programme team was collected and analysed. In addition, a questionnaire survey of key participants' views (students, teachers and coordinators in schools and FE colleges) was carried out in November/December 2002.
The secondary schools involved in the survey were drawn from all those registered with SCHOLAR to reflect the numbers of students using it at Higher and Advanced Higher across the subject areas supported by the programme. In addition, schools were invited to participate such that the number sampled in each local authority reflected the total registered within that authority. The target number was set at 75 - approximately 20% of the total number of schools registered. In total 53 schools and 7 FE colleges returned questionnaires.
In Phase 2 of the evaluation 25 interviews and 9 focus groups were conducted in three case study local authorities, two colleges and Heriot-Watt University in order to extend and enrich the data already gathered in the survey.
An additional dimension was added to the project in order to gain an insight into the impact of SCHOLAR on national examination results. Results for the academic years 2001-2002 and 2002-2003 were obtained from the Scottish Qualifications Authority for SCHOLAR and non-SCHOLAR students in the range of subjects included in the programme. The percentages of SCHOLAR and non-SCHOLAR candidates achieving each grade on the relevant examinations within each of the subjects specified were calculated and compared statistically. In addition, an analysis by gender was undertaken.
E-learning programmes are available in a range of formats, from stand-alone, distance learning with little or no face-to-face contact with a teacher, to materials and activities that are designed to complement and supplement more traditional teaching methods. The following definition of e-learning was used during the analysis of the data: 'a range of activities, from the effective use of digital resources and learning technologies in the classroom, through to a personal learning experience enabled through individual access at home or elsewhere.' (DfES, 2002, page 6).
Findings
Schools and colleges became involved with SCHOLAR for a variety of reasons. For most schools, it was the local authority initiative and funding that provided the initial impetus. For colleges and some schools, the prospect of being involved in new developments in ICT, at the forefront of innovation in online learning, was paramount. Many coordinators, teachers and students commented on problems with administration, communication and registration processes at the start of the SCHOLAR programme. Problems with registration were related to the issuing of student passwords and the distribution of the study guides. The findings suggest that these problems have to a large extent now been addressed.
The SCHOLAR coordinator's role was essentially an administrative one, providing liaison with Heriot-Watt University and managing student registration and the supply of materials. The majority of coordinators reported that they spent little time on SCHOLAR activities compared to other aspects of their work. For the majority of teachers, the key activities were working with students, mainly using the study guides to complement classroom teaching. The amount of time spent on SCHOLAR-related activity varied significantly according to how the teacher interpreted the role, which elements of the programme were used and the reasons for using SCHOLAR in that particular subject.
Many of the coordinators and teachers who were interviewed said they had insufficient time to learn how to use SCHOLAR properly and had not been able to explore the full potential of the resources.
The majority of coordinators, teachers and students indicated that they had adequate technical support in schools and colleges. However, some problems were reported concerning the speed of the systems, especially when using animations and simulations.
Most coordinators, teachers and students had ready access to computers out of school/college and were regular users of the more common applications. Students were more likely to play games, use simulations and use the World Wide Web than teachers. Both teachers and, more particularly, students were confident that they had sufficient ICT skills to cope with the demands of SCHOLAR.
Of the key components -study guides, online, assessment and revision materials - students reported using all considerably more than their teachers thought; this is in part explained by their greater home use. Most students found the study guides clearly presented, easy to use and at the right level of demand. Similarly, most students found the online materials easy to negotiate and readily understood although some needed the text booklets to support their study; the diagrams and simulations were especially appreciated.
The students made frequent use of the assessment activities. They tended to use these as part of the revision process and appreciated the immediate feedback on their performance. Students' views of the assessment materials varied a little with the specific subject. Some students expressed frustration at the way some answers had to be entered in the online assessments.
The online reporting facility and the online discussion forum/noticeboard were not used as frequently as might be expected; in part, this may be explained by the fact that many did not know of their existence, possibly as the result of a limited induction process.
Amongst students, the preference was for a blend of learning experiences, using both online learning and more conventional teaching and text-based materials. While students appreciated the opportunity to continue to study or revise at home through the SCHOLAR website, a few expressed concerns about the expense of being online and that it kept the telephone line busy for extended periods.
Some coordinators and teachers considered that students coped well with the SCHOLAR environment, showing increased levels of independence and motivation, although just over half did not think that it had made any impact on achievement.
The FE sector had slightly different needs and expectations of an online learning system from those of schools. While the latter focused on resources and materials to support classroom learning, the colleges were looking for higher levels of interactivity and smaller units.
Around half of the coordinators and teachers had experienced a formal induction session organised by the University. Most of those who had attended one session found it very helpful and enjoyable. However, experiences of induction varied considerably and for some it was a haphazard and partial introduction that often did not familiarise them with all of the components of the programme. Teachers were generally positive about the support offered in the teacher handbook, although less so than the coordinators.
The majority of coordinators and teachers did not feel that they had been well prepared for the programme. Half of the coordinators would have liked more staff development and almost half of the teachers would have liked more on the online elements of the programme.
The interview responses suggested that not all the coordinators or the teachers were aware of the range of staff development activities available.
There was little sharing of experience and good practice amongst coordinators and teachers and there appeared to be few established processes for facilitating this.
The majority of the students considered the induction good enough to get started on the SCHOLAR programme. However, a number of students said that they had only been given the web address and their password and would have liked a demonstration of the programme.
The SCHOLAR management team receive considerable feedback from schools and colleges on their efficiency and effectiveness and are aware of many of the issues and problems raised by the respondents to the survey. They have taken steps to deal with these with some evidence of success.
The findings from the analysis of external certification indicate a degree of superior performance from those students registered with SCHOLAR across the subjects studied. This varies with the subject area but is present to some extent in all. Boys and girls appear to benefit similarly in most instances.
The evidence collected indicates that SCHOLAR provided students with an opportunity for greater flexibility in and control of their learning in that they were able to choose to learn in their own time, at their own pace, and in their own environment. Some students were able to use SCHOLAR to personalise their learning to meet their own needs and engage in self-regulated learning.
The independent use of the materials by students, particularly when it was self-directed rather than prompted by their teachers and when they selected the mode of study provided an opportunity to experience and develop learning strategies that will be useful if they continue into further and higher education.
Links between schools and colleges had benefited slightly from involvement in SCHOLAR, although it was generally considered a little early to measure the impact. The team recognise the need to respond to colleges and schools differently, acknowledging their different stages of development with regard to ICT and staff readiness for new developments.
There is not yet a body of evidence to indicate whether the aim of improving application rates to university, and Heriot-Watt in particular, is making an impact, although several felt that they had anecdotal evidence that indicated some improvement. This should be an area for further study.
Conclusions
The evidence suggests that the majority of coordinators, teachers and students have welcomed the SCHOLAR programme. Seventy-eight percent of the teachers said they would recommend SCHOLAR to colleagues and 85% of the students said they would recommend it to other students. It has provided teachers with high quality resources to complement traditional classroom teaching approaches. However, the potential of the SCHOLAR programme to develop e-learning approaches has not been fully explored by teachers.
There is some evidence to suggest that SCHOLAR has impacted positively on raising the students' standard of educational attainment. Overall, the SCHOLAR students performed at a higher level in 2001 - 2002 and 2002 - 2003 in the SQA examination subjects included in the programme than non-SCHOLAR students. However, the factors underpinning this finding are more complex than simply being registered with SCHOLAR or not. It is possible that there are important differences in the two groups of students (SCHOLAR and non-SCHOLAR registered) in that the former may have been more motivated to study anyway and the opportunities provided by the programme made this more effective. (No data were available to check for other factors in this study). It is not possible to generalise from findings relating to the use of an online learning programme and improvement in examination results with the group of students in this evaluation to the whole school population in Scotland. The students involved in this study were engaging in Higher and Advanced Higher and were therefore likely to be more able students.
The findings suggest that the SCHOLAR Programme impacted on the way students learned. It provided students with an opportunity for greater flexibility in and control of their learning in that they were able to choose to learn in their own time, at their own pace, and in their own environment. Some students were able to use SCHOLAR to personalise their learning to meet their own needs and engage in self-regulated learning.
Students in schools and colleges used SCHOLAR more than their teachers appreciated. The majority of teachers were not monitoring the students' use of the programme or engaging in follow-up discussions. This missed the opportunity to add depth to the learning experience by creating a bridge between classroom and independent study. Not all students were aware of all the facilities of the SCHOLAR, in particular, most did not appear to know about the discussion board. Better information is needed for students and teachers when starting out on the programme in order to ensure they are aware of all the facilities on offer. A more effective online discussion facility, along with ongoing support to develop the necessary skills for teachers and students to use it would provide a mechanism for improved networking.
Although the majority of students appeared to have sufficient technical skills to use SCHOLAR, this is not the same as understanding how to use it to its full potential to support and enhance learning. Similarly, the majority of teachers said they needed to know more about online learning.
Almost half of the coordinators and teachers would like more staff development. The focus of staff development appeared to be on introducing the SCHOLAR programme at a very general level. There was little evidence of staff development in e-learning or in pedagogical approaches. This is an area that should be addressed. The majority of teachers said that they had made no changes to learning and teaching approaches in order to implement SCHOLAR. However, it was clear from the analysis of all the data collected (student and teacher) there was blending of established and ICT-based teaching and learning which gave students the opportunity to make decisions about how they used the range of SCHOLAR materials to support classroom learning.
Problems with administration, communication and registration were reported at the start of SCHOLAR. However, the SCHOLAR management team are aware of the issues and are working to address them.
One of the aims of the programme is to improve links between schools, FE colleges and the universities. The evidence collected from coordinators, teachers and students in the schools and colleges does not suggest that there has been any significant impact at the school/FE/HE interface. Where links already existed, these had been maintained or slightly strengthened.
The independent and self-regulated use of the materials by students gave them an opportunity to experience and develop learning strategies that will be useful if they continue into further and higher education. In this way, SCHOLAR may contribute to creating a link between the students' study in their final years at school and further and higher education.
The majority of students indicated that they had already decided to continue into further and higher education and that being registered with the SCHOLAR programme had not particularly increased their interest in further study. It was not possible to identify any greater interest in going to Heriot-Watt University due to the SCHOLAR Programme.
Further Research
It would be of interest to investigate the impact of the development of self-regulated learning and independent study in the final years on learning and achievement in further and higher education. Examination results were analysed for the academic years 2001-2002 and 2002-2003. However, as the teachers become more acquainted with SCHOLAR it would be valuable to carry out a further investigation of the impact of SCHOLAR on examination results. The blending of traditional and e-learning approaches needs to be more fully explored. In particular, issues to do with the appropriate balance between the two and methods of optimising the links between teacher-directed and independent study require further research. The evaluation gathered a rich set of data. Although the findings suggest that some developments have been made in the implementation of e-learning approaches, much more needs to be understood in order to maximise the benefits for students.
The report can be downloaded in PDF format by clicking on the link below:
SCHOLAR Evaluation Report - Full Report