Learning for Living and Earning Evaluation Report
Executive Summary
Introduction
This is a report of the evaluation of the Learning for Living and Earning (LLE) project conducted by the Centre for Studies in Enterprise, Career Development and Work, University of Strathclyde, as part of the national evaluation of projects supported through the Scottish Executive Education Department's Future Learning and Teaching programme (FLaT). The evaluation was undertaken between May 2004 and December 2004.
The stated aims of the Learning for Living and Earning programme were:
¤ to enable participants to reflect on the purpose, rationale or narrative of lifelong learning in the 21st century
¤ to enable all participants, via interrogation of stimulus video material, presented documentation and discussion, to develop a clear understanding of the value and importance of high aspirations and a sense of self-efficacy for young people
¤ to demonstrate via a series of practical projects as stimulus material, the values of focused and planned collaboration between education and external partners including the business world
¤ to enable school managers and policy makers to consider programmes in which young people acquire and demonstrate core skills that promote creativity and ambition.
Aims of Evaluation
The aims of the evaluation were:
¤ to identify any impact on participants attitudes and views on the purposes of education
¤ to determine the subsequent impact on establishments.
¤ to assess its impact on the delivery of the National Priorities
¤ to assess the overall impact of cross-sectoral working.
Evaluation Methods
The evaluation comprised of five stages.
During stage one it was necessary to examine the rationale behind the initiative itself and the preconditions existing at the time within schools and businesses. It was also important to understand the nature and content of the staff development sessions, including the ‘messages’ that the LLE team hoped to pass on to the participants. Semi-structured interviews were held with the LLE team, who also provided copies of all materials which were used on the programme. All members of the evaluation team viewed all relevant documentation relating to the programme: setting up and developing the initiative, materials used as part of the programme and participant evaluations of the programme. The only exception to this was a CD of the final rearranged session which was viewed at a later date,
Stage Two involved interviews with all participants, designed to identify participants’ perceptions of the programme and to establish any immediate impact which it had on their attitudes and actions.
In order to examine these issues, the research team aimed to work with a 100% sample of the schools where participants had attended most or all of the sessions. Although the original list of head teacher participants totalled nineteen it was only possible to interview fifteen. Background data on each school and its locality was also examined, for example attainment, attendance, socio-economic factors, parental and family involvement, inclusion and deprivation issues, local post-school opportunity structure including lifelong learning.
At this point the contributors from business were interviewed.
During stage three all interview data was analysed and issues identified for further exploration.
By stage four, it was expected that any changes (in attitude or behaviour) as a result of the programme could be clearly identified. It was hoped to identify changes in individual head teachers, in SMTs within these schools and the impact on staff/pupils within the schools. Such questions were addressed through a second interview with head teachers of each establishment, and questionnaires which were sent to one person from the SMT in each establishment, in order to establish any impact on action. A very low percentage of these questionnaires were returned.
During stage five all second interview data was analysed, and developments identified. A further meeting was held with the LLE manager and her colleagues. An interview with the Depute Director of Education took place in order to establish the Local Authority perspective of the programme, its aims and possible subsequent development. A questionnaire was sent to two local authority representatives whose area of responsibility had potential links with the programme. Despite requests from the evaluation team, these questionnaires were not returned by the time of analysis of findings.
During Stage Five of the evaluation process, the two invited speakers were also interviewed.
Finding and Conclusions
The main conclusions were as follows:
The Programme
Successes
1. The programme succeeded in engaging the interest of the participants.
2. The cross-sectoral approach was seen as a positive one in terms of awareness raising and highlighting possibilities for collaboration.
3. Those participants who took part in the research generally felt that the programme was of a high quality and allowed considerable scope for reflection. They recommended that the programme would be useful to others, for example in supporting head teachers in developing more flexible teaching and learning in response to current policy papers. The programme succeeded in engaging the interest of participants.
4. The involvement of business stimulated debate and provided a balanced view of the purpose of education. The business view that attributes were as important as (or more important than) attainment was well received, although some participants were not fully convinced of this.
Challenges
5. Even after completing the programme, a significant minority of participants were unclear about its purpose and had difficulty understanding the central concept which was one of a narrative.
6. At the time of the evaluation, participants noted that they were left with a general feeling that there was no clear indication of what the next steps might be, or how they might be supported in the future.
7. The programme proved to be expensive in terms of staff costs.
Impact of the programme
8. The programme appears to have reinforced previously held beliefs and increased the confidence of individual head teachers, rather than change attitudes.
9. In terms of subsequent action, very little can be seen to be solely attributable to the programme. The LLE team and local authority directorate appear to be comfortable with this and see the course as part of a ‘bigger picture’.
10. The cross-sectoral approach was primarily useful in terms of awareness raising and highlighting the benefits of close working relationships. However, evidence that the various establishments were working together more closely in practice as a direct result of the programme was not apparent. The development of Learning Communities has reinforced some aspects of what the programme hoped to achieve.
11. Other than increasing confidence in head teachers and providing a business perspective, no immediate or short-term effects of the programme were discernable to the evaluation team. It is probably too soon to tell what longer term impact may have made at the level of the school experience for pupils. Although the LLE team regard the programme as part of a longer term development, perhaps over ten years, there are some issues about the monitoring and evaluation of its impact. The LLE team may wish to consider how they will monitor the longer term effects of the programme if it continues, or is run again.
12 Participants still identify several barriers to progress in implementing the messages of the LLE programme. These include: overcrowding and inflexibility of the curriculum, the need for target setting, staff resistance to change and rigid local authority structures. A dilemma still remains for some head teachers who feel that they have been encouraged through the programme to challenge accepted structures within education, yet are still expected to work within given frameworks.
Recommendations
Key recommendations of the evaluation team are as follows:
1. It is likely that the selective nature of the participant group ensured a positive response to the programme content. Being described as ‘thinkers’ or ‘leaders in their field’ may also have disposed them to be positive (‘the halo effect’). Therefore, while the programme was felt to be of value to this particular group, it should be run with a less selective group in order to test general replicability.
2. Any future run of this programme would be more effective if the purpose of the programme was explored more clearly at the beginning, with a sense of the expectations and possible outcomes.
3. The cross-sectoral aspect of the programme is one of its main strengths and should be retained. However the LLE team should examine ways of encouraging further development of the links made as a result of the programme.
4. Participants indicated that there were elements of repetition within the programme, therefore the organisation should be reviewed in light of comments e.g. number of sessions, length of time between sessions. This might have the effect of making the programme more cost-effective in the use of staff time.
5. In terms of content, there should be regular review of input - personnel, reading material and cameo materials. While extensive reading material is provided for participants and a bibliography provided, the LLE team should consider a more specific form of referencing of the extracts. This would assist participants who wished to target the exact text in the original documents.
7. While the programme is intended to be ‘open-ended’ and non-prescriptive in its aims so that participants may be empowered to develop their own strategies, it is nonetheless important for the LLE team to be able to identify the outcomes of an investment of central funds, both in the short and long term. The LLE team may wish to consider how they will monitor the longer term effects of the programme
Learning for Living and Earning Programme: Evaluation Report