What's new Site map Search Content FAQ Help Complaints Feedback Form
corner
Home Arrow Updates Arrow Good Practice from Red Tray - Evaluating e-learning

Good Practice from Red Tray - Evaluating e-learning

Tools

  • font size iconText size Increase font size Decrease font size
  • PrintPrint

DateJuly 11, 2008 Posted by: Nige Howarth   Keywords: benefits, learner engagement, measuring value, strategy

Introduction  Over the last few years, Mike Ditchburn @ Bourne Training (now part of Red Tray) has written a number of articles and papers on Good (Best) Practice which dovetails with the ongoing findings of the Towards Maturity team. As an active member of the Towards Maturity Virtual Advisory Network (VAN) Mike has kindly agreed that many of his findings should be shared, and we are publishing in two parts. The complete articles are available through Bourne Training info@bournetraining.co.uk. Supporting Good Practice (Part 1) - Evaluating e-Learning For Evaluating e-Learning, Mike identifies 10 key considerations to assess the value of your e-Learning project: 1. Clarify your business driversThis is obvious – if they weren’t clear then you wouldn’t be starting! Whilst much e-Learning happens because it has to (health & safety, induction, product knowledge etc.), there’s still the discretionary element – the solutions that need a business case. Regardless of the origin of the business need, you should make sure you can answer the following questions:
  • What will the business gain in measurable terms as a result of this training?
  • How will I know when the business has achieved these measures?
  • Who are the senior stakeholders and what are their motives for this succeeding?
 Don’t assume you know your business drivers – take time to think them through so you can come back to them after the solution has been implemented. 2. Identify the learning objectivesThe first step in identifying learning objectives is to understand your audience (job roles, locations, age range, experience levels, learning preferences etc.). We all know about making objectives SMART but what type of objectives are you looking for? Learning Objectives (or learning outcomes) are statements that describe what a learner will be able to do as a result of learning. They are aimed at the three domains of learning: knowledge, skills and attitudes (or cognitive, psychomotor and affective domains of learning). If you haven’t clearly identified the learning outcomes for your e-Learning programme, you can’t effectively evaluate whether the solution meets the business need. 3. Internal marketing – whet the learners appetite!Although this doesn’t immediately link to evaluation, the important point to realise is that unless learners are ‘warmed up’ to the idea of e-Learning its impact can be severely impaired. This is more important for e-Learning that happens because it has to. Think about how you can market your programme, for example:
  • Sending out flyers advertising the event (like you’re advertising a film)
  • Using testaments from a pilot review group if you’ve used one
  • Hold lunchtime ‘surgeries’ to demonstrate the programme during development
  • Develop a ‘taster’ version of your course and make it available to staff before launch.
 4. Evaluation during developmentAfter the programme is built is too late to start evaluating its effectiveness. Think about what you can do during development and use this feedback to mould the solution. Learner review pilot exercises and stakeholder reviews should be built into the development process and help answer questions like:
  • Is the content relevant to the audience in coverage, pitch, style and interactivity?
  • Is the interface design and navigation intuitive and appropriate for the audience?
  • Is the learning material effective – have learners learnt from the course?
  • Is the assessment approach effective – does it provide evidence of knowledge and skills in the right areas?
 5. Learner reactionsWhilst this may not be your primary concern, the learners are your customers so their experiences are important. Just because there is no trainer doesn’t mean that you shouldn’t bother with this stage of the formal evaluation process. Take a straw poll, issue questionnaires or build in an on-line response mechanism, but make sure you communicate with the most important people soon after they’ve completed the training. 6. What have they learnt?e-Learning lends itself to automated assessment, but this part of the evaluation process still needs a lot of thought to make sure the data is relevant and timely, and the approach is appropriate to the audience and learning objectives. Areas to consider when establishing the approach to this stage include:
  • How formal do I want the course assessment to be i.e. is it a test or a few simple exercises, will scores be calculated, is there a pass/fail point, do I need questions to be delivered on a random basis?
  • What questioning approach best suits the learning outcomes i.e. simple multiple-choice, scenario-based exercises, role play, activities with a branching structure to enable learners to learn through making mistakes?
  • How will assessment data be collected i.e. through an LMS or HR system, through a bespoke database, by simple email to a central point or printed locally to be held as a personal record?
 7. Transfer of learningIt now starts getting difficult – you know what they know after the assessment but how can you know what they’re going to do with it? The answer to this is in how you structure your e-Learning solution in the first place and to what extent it provides a real link to the workplace. Consider the following approaches:
  • Building in the learners line manager during the e-Learning course – agreeing personal learning objectives and outcomes at the start of the course, developing a programme that includes trigger points for discussion (perhaps following course exercises), or formal reviews of assessment outcomes to discuss areas of development and support required for the learner.
  • Incorporating a personal development plan within the e-Learning programme for the learner to complete at the end of the course - they can identify what they have learnt from the course and how they will apply that learning to the workplace. This provides another opportunity for a structured discussion with their line manager.
 Whatever method you use, make sure you build in evaluation of the learning transfer into the process and leave it until a few weeks after the training so the learner can illustrate measurable improvement. 8. The business results!The Holy Grail! It’s easy if you’ve clearly identified business drivers and objectives at the start. You need to make sure you can justify your approach and illustrate success to the stakeholders and learners, and this can only be done after a realistic timeframe has elapsed following implementation. The findings you present here are critical – and a focus on the ‘bottom line’ is essential. 9. Documenting resultsThis goes without saying really, but it’s a discipline that starts at the first stage. Consider creating an ‘audit trail’ of your evaluation objectives and approach which clearly states the business objectives and measures, and the learning objectives – and then relate all areas of your evaluation feedback to these pre-requisites. When collating feedback, whether it be reaction, learning, transfer or business results, keep the document succinct – a high level overview – and add detail as appendices. That way you can circulate different versions to different people with the least amount of effort. 10. Telling your storySee yourself as your own business. After a successful implementation you need to ‘market’ your story and tell people what you and they have achieved – remember, you got them involved early on and they’ll want to know how you did.Think about what would work in your organisation. You might want to try some of the following:
  • Publicise your success – put up posters with evaluation feedback quotes or more formal business benefits
  • Send out an internal ‘press release’
  • Find a way of thanking people – is there something of value that you can offer learners?
  • Consider publicising the project in your internal magazine – or in your industry journals
  • What about a training award – this will help recognise the hard work you’ve put in?
 And finally…evaluate your own performance and that of your team. You need to know what went well so you can incorporate it next time, and what you’d do differently next time. And make sure you take time out to celebrate what you’ve achieved! Thanks again to Mike for his insight and thoughts. You may also find it valuable to review some of the case studies and research on the Towards Maturity website which provide further insight on evaluation strategies. Pertinent research includes Linking Learning to Business and the original Towards Maturity research findings. A selection of relevant case studies include Transforming a Business @ Cable & Wireless, Cruising for Success at Royal Caribbean, eNVQ’s for Contact Centre staff at UKi Partnerships and finally Expanding Training for Radiologists at the NHS. We can also cross reference Mike’s 10 Key Considerations against the relevant strand in the Towards Maturity model www.towardsmaturity.org 
Evaluating e-learning considerationsStrand in Towards Maturity Model
1. Clarify your business driversDefining need
2. Identify the learning objectivesDefining need
3. Internal MarketingEnsuring engagement
4. Evaluation during developmentEnsuring engagement
5. Learner reactionsLearner context
6. What have they learnt?Demonstrating value
7. Transfer of learningDemonstrating value
8. The business results

Demonstrating value

9. Documenting results

Demonstrating value

10. Telling your storyDemonstrating value
  While on the theme of Top Ten’s you can find on the Towards Maturity website 10 Tips for Communicating with Learners, a recent Podcast with Gordon Bull on Top Tips for Building Learning Strategies and finally 50 Ideas for Free e-learning!  Nige HowarthTowards Maturity NB: Some content used with the kind permission of Bourne Training

Also do complete our minipoll.

Did you like this article? Rate it!

You might also like...

Name:
Your homepage:
Email:
Do you have any suggestions or information that would help us improve this article?
 
Name:
Email:
Your homepage:
Do you have any questions that you would like to ask one of our many experts?
 

Hot off the press

Amazing industry support for TM’s unique e-learning benchmark

Take part today. Don’t miss the chance to receive free personalised feedback & practical advice to help accelerate the benefits of learning technologies in your workplace.

corner