St Stephen's Enterprise Centre - Evaluation Report - Executive Summary

The external evaluation

In October 2004, under the Framework Agreement (ref QLC/2/5/41) between the Scottish Executive Education Department (SEED) and Professor J. E. Wilkinson of the Faculty of Education in the University of Glasgow, the SEED commissioned an external evaluation of the St Stephen’s Enterprise Centre Project. The evaluation was conducted by a team from the Scottish Council for Research in Education (SCRE) Centre and the Department of Educational Studies.

The main aim of the study was to provide an independent external evaluation that focused on key outcomes of the St Stephen’s Enterprise Centre. The specific evaluation objectives were to assess:

• The extent to which the Centre has promoted and stimulated lifelong learning and employability in the wider community of Port Glasgow, particularly concerning targeted groups who are traditionally difficult to attract into learning and training.

• The extent to which the Centre has developed a flexible and tailored curriculum that caters for the needs of all learners.

• The impact of the Centre on participating secondary pupils, including:

      – The specific impact of the Virtual Work Experience (VWE) course on pupils in relation to their application and attainment when they return to mainstream education
      – The impact of the Centre on disaffected pupils. This includes any improvement concerning their achievement, attendance, employability and motivation to learn. 

• The extent to which the St Stephen’s and Port Glasgow High Schools and the Enterprise Centre have worked together to develop relevant school development plans.

To address these aims, a range of quantitative and qualitative methods were seen as appropriate. The overarching research design for the project was based upon three overlapping stages with a final fourth stage, involving a synthesis and analysis of collected data and information.

Stage 1: Preparatory phase

The first stage entailed: 

• Refining the research design with the SEED and the Research Advisory Group, and holding initial meetings with project management at the Enterprise Centre and other key informants to gather contextual information and further details necessary to inform the design of methods used in subsequent stages.

•  Establishing an electronic forum for stakeholders accessed via the SCRE Centre website to allow them to contribute additional information and to provide feedback.

Stage 2: Assessing the nature of provision and participation

The second stage provided an assessment of the courses at the Centre, of underlying planning, and of patterns in the participation figures to date. To do this, the research involved:

• Face-to-face and telephone interviews to gather information from key informants involved with the management and administration of the Project and the delivery of the courses. Relevant individuals in the local community, partner organisations and businesses were also interviewed to obtain information on the development and planning of the Project.

• Examination of documentary information held at the Centre to establish: the nature and scope of participation and the nature of the curriculum (including bespoke courses available through the project, the use of ICT and the range of learning and teaching approaches).

Stage 3: Assessing the impact of the Project

The third stage focussed on the impact of the Project to date and involved:

• Interviews and focus groups with key stakeholders, including participating pupils from St Stephen’s and Port Glasgow schools, teachers from both schools and representatives from local businesses and relevant agencies in the community.

• Collation and analysis of pre- and post-course questionnaires, administered by staff at the Centre

• A brief follow-up survey of participants from both St Stephen’s and Port Glasgow schools. This questionnaire was administered by researchers approximately six months after pupils had taken part in the VWE course.

• Telephone interviews with a sample of parents of participating pupils from St Stephen’s and Port Glasgow schools. The researchers had wanted to conduct 50 telephone interviews with parents of pupils who had participated in the VWE (half from St Stephen’s school and half from Port Glasgow). The final number of interviews was just under half this total. However, the fact that the information emerging from all interviews was revealing similar patterns and perspectives regardless of the school at which the their child attended meant that further parental interviews would have provided limited additional insights.

Stage 4: Analysis and reporting

The fourth stage involved final synthesis, analysis and reporting of the information from Stages 1, 2 and 3 to address the main research objectives set out in the specification and associated research questions. The original research objectives and questions set out in the Research Specification provided the overall framework for the analysis.

Summary of key finding

The main findings to emerge from the study are presented below and are arranged using the main research objectives.

A shared vision of aims and purposes

• Stakeholders generally see the aims and purposes of the Centre and VWE programme as developing relevant core skills in participating pupils and adults, to promote their life chances and employability, and to foster positive attitudes towards lifelong learning and work across the community.

• A strong theme to emerge was that these aims, particularly those concerning promoting employability and attitudes to learning and in the wider community, were ambitious and would require time and sustained effort to address.

Providing an appropriate and relevant curriculum

• Overall, the content of programmes offered by the Centre was seen as appropriate to meet the aims of the Project. The VWE programme is seen as particularly well designed to engage with disaffected pupils and those who struggle with traditional lessons.

• Pupils greatly valued the opportunity to work in a ‘business-like’ environment, and the fact that they were treated like young adults with high levels of autonomy.

• The Centre is seen as a unique environment with contemporary surroundings and high quality ICT resources. This motivates pupils and facilitates their access to learning approaches currently unavailable in many of their lessons.

• Interviewees highlighted the relevance of programmes for many adults, with reference to the high levels of participation in those courses offered. However, some interviewees suggested that the Centre should continue to monitor whether popular computer courses such as the European Computer Driving Licence (ECDL) were suitable for those people in the community who were traditionally difficult to attract.
  

The impact of the Project

a) Impact of the Virtual Work Experience (VWE) course on pupils

• Since the beginning of the VWE programme at the Centre, 608 pupils have participated in the VWE course (209 from Port Glasgow High School and 399 from St Stephen’s High School). This figure includes 20 St Stephens’ pupils who participated in the pilot version of the VWE course in 2003.

• Overall, the VWE was seen as having a positive impact, at least in the short-term, on pupil participants, including those identified as disaffected with school.

• Initial positive outcomes for pupils were seen as increased communication skills, self-confidence, team working abilities and specific ICT skills.

The majority of pupils who were seen as having problematic behaviours were more motivated to study and work on tasks. However, some interviewees reported that often these pupils’ behaviour patterns re-emerged soon after returning to their normal lessons in school.

b) Impact of the courses on adult participants 

• Initial analysis of the Centre’s participation data reveals that between the start of operations at the Centre and November 2005, 229 adults have participated in courses. One hundred and eighty-one (181: 79%) of these registered learners were female.
  

Data available from the Centre on types of participant further reveals that:

– The largest group of registered learners (37%) is in the 31–45 age bracket; and the second largest group is the 46–60 age range, with 26% of the total. 10% of all course participants are aged over 61.
– 25% of registered learners are aged 16–30, and within this group, those aged 16–20 represent 10% of all participants.
– 53% of registered learners are in employment.
– 9% of learners are students, mainly between the ages of 16 and 20.
– 7% of learners are disabled.
– The Centre combines retired and unemployed participants into a single category, and these made up 31% of learners. There was no available gender or age breakdown for this category.

• Adult participants reported an initial positive impact concerning their computing skills, self-confidence, ability to help others learn and opportunities for further learning and employment. It must be stressed, however, that only a small proportion of those involved in the focus groups were participating for vocational reasons.

Promoting lifelong learning and employability in the wider community

• There was consensus that because the Project has been operating for a relatively short time it was unrealistic at this stage to expect a notable impact on promoting lifelong learning and employability in the wider community. Nevertheless, there was consensus across stakeholders that the Centre was largely developing in line with its main objectives and had the potential to make a substantial and positive impact if longer-term support was available.

The above participation figures for adults indicate that the Centre faces a challenge in terms of attracting male participants to do courses. This is particularly so for the 16–30 age group, one of the target groups, where only 3% of participants were males. Such challenges were recognised by the management at the Centre, partner organisations and participating adults. There was consensus that the Centre was not fully engaging with local adults who could benefit most from the courses on offer, ie those who traditionally do not participate in learning and training such as disaffected males who were long-term unemployed with few or no qualifications.

• Qualitative information gathered from adult participants and partner organisations working in the community highlighted that, for adults in the target groups, a lack of self-confidence was an important barrier to returning to learning. Providers also highlighted “territorial/geographical issues” which meant people were reticent about travelling outwith their neighbourhoods to access services and facilities.

• Suggestions from stakeholders, including those adults who had participated in courses at the Centre, for promoting greater participation among target groups included the following:

- Using more outreach work and targeted marketing would help encourage those who lacked confidence to participate or approach an education provider. Word of mouth was seen as one of the best ways to promote interest in the Centre among local adults. Interviewees believed that this was how most of the adult participants had become aware of the Centre and courses. Adults also suggested that encouraging whole peer groups to participate would address the self-confidence and peer pressure factors that often acted as barriers to participation.

 - Any impact on promoting positive attitudes to employment and learning among target groups would have to take into account local culture, including challenging the views of some who see no utility in education or lacked the confidence to participate in learning, training and employment. Again this process would take time and require sustained and appropriate provision.

 - Education, guidance and other relevant partners should continue to combine their efforts and resources to focus on particular groups to offer advice on accessing training/education and/or employment. (Examples were given of the local Job Centre and Careers Scotland services increasingly working with the Centre to target particular groups and develop targeted programmes.)

– Setting aside part of one day per week as a drop-in time at the Centre for people to discuss returning to study or employment with relevant agencies.

– Some interviewees suggested that more marketing of the Centre was required. Adult participants and business partners, in particular, stressed that they had seen little active marketing and advertising of the Centre and courses.

– Providers suggested that greater use of the Centre by pupils outwith the Virtual Work Experience programme could be encouraged by teachers in the two High Schools.

 
– Parents of pupils at the two schools were also seen as potential participants and a Communications Strategy group was being established as part of local Determined to Succeed initiatives, with the aim of targeting parents. The two High Schools were seen as playing a key role in encouraging parents to use the Centre.

Developing a flexible and tailored curriculum

• Some interviewees from partner businesses stated that they had limited knowledge of the precise curriculum and methods adopted across the courses. Teachers, course providers and others from partner organisations were better placed to assess the curriculum and methods used in the courses. There was consensus across these stakeholders that the range and nature of the courses were appropriate for all learners. Importantly, participating pupils and adults also concurred, highlighting the varied and stimulating course content and methods of delivery.

• The Senior Management Team members at both High Schools believed that the VWE course addressed most of the National Priorities for Education and articulated well with curriculum areas. The course and other opportunities provided via the Centre were also seen as relating to initiatives on citizenship and enterprise. Teachers generally believed that the programme was designed to suit the needs of all pupils, commented on the excellent environment of the Centre, and stressed how much the pupils enjoyed using it.

• Most stakeholders, including participating adults, saw the courses offered by the Centre as relevant for many local adults and highlighted the high levels of participation in those courses offered. However, some interviewees, including providers at the Centre suggested that the Centre should continue to monitor whether popular computer courses such as the ECDL were suitable for those people in the community who were traditionally difficult to attract.

Partnership working

• There was consensus that the varied range of partners involved with the St Stephen’s Enterprise Centre Project had worked well together from the inception and planning stage through to supporting the operation of the Centre.

• Some professional stakeholders reported that there was room for further development in terms of the amount of partnership and levels of collaboration between the Centre and the two High Schools. Teachers and providers reported that teachers and their pupils at the two High Schools could make greater use of the Centre. Some departments were said to make more use than others of the facilities and services at the Centre to enhance their lessons (ie English, Art, Modern Languages).

• The already well-established partnerships and networks involving public and private organisations in the Port Glasgow area were seen as an important factor in supporting the Centre.

• Most of the representatives of supporting partner organisations and businesses cited the work and commitment of the Centre’s manager and his staff as being pivotal in sustaining these partnerships and in the efficient running of the Centre.

Issues and recommendations

The findings raise particular issues for innovative programmes and centres that address aims pertinent to the Determined to Succeed initiative and, indeed, the wider Curriculum for Excellence.

Promoting greater participation among target groups in the wider community

The Centre’s own participation data indicated that certain computer-based courses are very popular with local adults and even oversubscribed. This data also reveals, however, that there are issues concerning attracting males of all ages to do courses. A strong theme in the interviews with providers and adult participants was that a culture of apathy and/or low self-confidence exists across a ‘core’ of people in the community. This meant that such people rarely sought out education and training or even employment.

Recommendations

• Providers at the Centre and partners in the local authority, support services and business should continue to explore measures that aim to promote the participation of target groups in the local community. Recent developments in interagency partnership working have seen the implementation of collaborative projects that could impact on target groups. Careful monitoring of the impact of these projects, including highlighting effective practice (eg the approaches and projects that work best to encourage participation and effective outcomes for adults), would be extremely valuable and should be incorporated into the design of the projects.

• Providers should take cognisance of the views of adults who currently participate and use these to inform measures to attract others. The evaluation found that adult learners at the Centre possessed useful insights concerning why their peers did not participate and provided sophisticated suggestions concerning possible strategies to address non-participation. For example, greater use of outreach approaches and working with adult peer groups to encourage participation could be considered.

Encouraging Port Glasgow High School pupils to use the Centre in their own time

The Centre encourages pupils to use facilities after school hours. However, the Centre’s capacity to provide pupils with access after school at any one time is limited to relatively small numbers (approximately 12 pupils).

While some pupils have used the Centre after school, a theme arising from focus groups and the VWE participant survey findings indicate that Port Glasgow High School pupils were more likely than those from St Stephen’s to be reticent about using the Centre in their own time. Some Port Glasgow High School pupils saw the Centre as a resource for St Stephen’s High School only. Others believed that pupils at their neighbouring school would not welcome them dropping in to use the facilities during lunchtime or after school. This theme also emerged in two of the parental interviews.

Promotion of the Centre within Port Glasgow High School is ongoing with leaflets to all senior pupils encouraging them to use the Centre to assist them with employability issues. Staff and pupils are encouraged to consider computing courses between the hours of 3.30 and 8.30 on Tuesdays and Thursdays.

Previously, both the Careers Adviser and the Principal Teacher for Enterprise within Port Glasgow High School have also booked time at the Centre, to provide Port Glasgow High School pupils with career planning lessons and assistance with PowerPoint presentations.

Recommendations

• While Port Glasgow High School and staff at the Centre are clearly working to promote pupil access and use of the facilities, the views of some Port Glasgow High School pupils highlight that such measures need to continue. Particular effort is required to address pupil perceptions about the Centre as a resource for their use.

• To address those attitudes and perceptions that can act as barriers to participation, staff at the two High Schools and the Centre could look at ways to challenge the views of some pupils concerning the ‘ownership’ of the Centre. Some form of organised access, possibly facilitated by staff at the Centre or teachers, might promote participation by Port Glasgow pupils, or using pupil facilitators or ‘buddies’ from St Stephen’s High School to accompany peers from their neighbouring school while they attend the Centre. In time, it would be hoped that such strategies would not be needed and pupils from both High Schools would use the Centre without inhibition.

Meeting the long term objectives of the Centre

The majority of those involved with Centre believed that it was having a positive impact on those pupils who participated in the VWE programme and on the adults who were attending courses. Those interviewed, however, stated that to address the longer-term aims of the Centre, such as promoting employability and attitudes towards learning and work across the community, would require many years of sustained work.

Recommendations

• If the Centre is to meet those objectives that focus on promoting lifelong learning and employability among the local community, then adequate longterm funding will be required to maintain appropriate programmes and necessary staffing.

• The Centre will require systematic and appropriate monitoring of provision and impact and continued partnership working to make efficient use of resources, services and provider skills.

• The Centre should continue to develop partnerships with other providers and services that play a key role in developing programmes to address lifelong learning and employability for key groups in the community.

• Follow-up courses for pupils, possibly variations on the VWE programme, could be provided every two years to ensure that improvements to pupils’ self-esteem, team-working, communication and ICT skills were not lost.

• Teachers could look at ways to use the centre to enhance their lessons and curriculum and consider ways of promoting greater access to the Centre by pupils during school hours.

Maintaining positive outcomes for disaffected pupils

A strong theme to emerge from the findings was that those pupils seen by teachers as disaffected with school had largely engaged with the VWE programme, with reported improvements to their core skills and, initially, their behaviour. However, stakeholders were not sure how permanent such changes were. For example, the initial benefits for disaffected pupils, who were often frustrated by ‘traditional teaching approaches’ and resources, were lost when they went back to lessons in school.

Recommendation

• It is suggested that teachers at the two High Schools could consider ways of incorporating aspects of those learning and teaching styles and resources that disaffected pupils have found particularly enjoyable and productive in other areas of the curriculum. Making school lessons more like the approaches used in the VWE presents certain challenges and might require greater resources, but all pupils, not just those disaffected with school, reported positive outcomes and high levels of engagement. Such goals are also in line with the aims of the Determined to Succeed initiative and the Curriculum for Excellence.

Promoting links between the VWE course and the schools’ curriculum and methods

Senior management in both High Schools stated that, where appropriate, they were looking at how the school curriculum could relate more closely to the content of courses at the Centre. In some departments (for example, English and Art) teachers in St Stephen’s High School were beginning to place a greater emphasis on resources such as ICT.

Recommendation

• More teachers should be encouraged and supported to become familiar with the pedagogical arrangements at the Centre. This could be facilitated through discussions between teachers and the Centre staff to ensure more continuity between Centre and school approaches to learning.

• Again, related to reinforcing positive outcomes from the VWE, teachers at the two High Schools and the staff at the Centre should, where appropriate, continue to explore ways to articulate the content of the VWE and any similar courses with the wider school curriculum.

The role of key personnel in the success of the Centre

Almost all of the partner organisations and school managers highlighted the role of the Manager and staff at the Centre in the efficient set up and running of the Centre. Furthermore, they stated that the initially positive outcomes observed for pupils such as increased motivation and communication skills were largely attributable to the learning and teaching styles and approaches of the Manager and his staff. This raises two issues: firstly, other centres and initiatives, particularly those that target disaffected pupils might benefit from sharing practice and ideas with the Centre; secondly, these key personnel appear to be playing a very important role in the success of the Centre, and, therefore, a key questions is, to what extent would the Centre be successful if these personnel moved on?

Recommendation

• The historical factors and circumstances that have led to the setting up of the Centre, and the mix of staff who have developed it to date, may be unique. However, given its initial success, it seems likely that the St Stephen’s Enterprise Centre provides a good model for wider adoption in similar social and economic contexts. It is recommended that similar centres be developed with the caveat that sustained funding is required if the principal objective of such centres is to work with others to promote employability and lifelong learning. Careful selection of managers and presenters will also be required to ensure appropriate and effective pedagogical approaches are used.

Addressing parents’ awareness and involvement

The very low level of parents’ awareness of the VWE course, demonstrated during the telephone survey of parents and in pupils’ accounts, suggests that an opportunity was lost for promoting the image of the Centre and its value among the wider community. Parents limited awareness also meant any positive support parents might have given to pupils concerning the course was reduced.

Recommendation

• Staff at the two High Schools, the Centre and other partner organisations should work to raise parents’ awareness of programmes provided by the Centre and other associated services. A range of publicity and dissemination could be considered, including information leaflets sent home with pupils, and making reference to the work of the Centre in general correspondence to parents. Open evenings/days or hosting celebration of achievement awards at the Centre might also be an appropriate way to involve some parents.

Maintaining greater interest and involvement by teachers

Some of the professional stakeholders suggested that greater teacher involvement with the Centre could further increase teachers’ awareness of the programme and the facilities at the Centre. This could then help promote greater use of the Centre by teachers at both High Schools, to enhance their provision and help them to build on the outcomes of the VWE course.

Recommendation

• Teachers are busy and have many demands on their time; however, pupils greatly appreciated the presence of their teachers at the end of course presentations. Pupils also saw teachers dropping into the course as evidence of their interest in what pupils were doing and as an indicator of teachers’ value for the course. It is recommended that teachers and staff at the Centre explore ways in which teachers could play a greater role in using the Centre and observing their pupils while they do the VWE.

Long-term objectives require long-term evaluation

The majority of professional stakeholders and adult participants in courses stressed that the aims and objectives of the Centre that concerned promoting lifelong learning and employability among the local community would take years to demonstrate any significant impact. The scale of the challenge and the complex range of factors involved in such a task, therefore, meant that the current evaluation could not adequately measure the impact of the Centre in terms of these criteria.

Recommendation

• As stakeholders have stressed, any meaningful evaluation of the Enterprise Centre’s objectives that concern promoting lifelong learning and employability among the local community will require long-term evaluation. Such an evaluation should include on-going monitoring of data on the types of participants in courses at the Centre, and a sufficiently rigorous and in-depth exploration of the benefits for participants and ‘distance travelled’. For example, any further research should be able to demonstrate whether those who participate in courses at the Centre benefit in terms of developing more positive attitudes towards lifelong learning and have greater life chances and employment prospects. This would likely involve tracking participants for some time to assess whether their participation had been beneficial, in what ways it has helped them and what factors facilitated or inhibited their using what they had learned on their courses.

PDF file of St Stephen's Enterprise Centre Final Evaluation Report (825 KB)

• Email this page
• Print friendly format
• Digg this
• Save to del.icio.us