Evaluation framework EFI for measuringthe impact of learning，education and training

C.M.STRACKE

（TELIT Research Institute for Technology－Enhanced Learning and Economic and Vocational Education and Training，University of Duisburg－Essen，45141 Essen，Germany）

0 Introduction

The Evaluation Framework for Impact Measurement（EFI）focuses the important purpose and demand to assess the internal and external impact by learning，education，and training：Impact measuring is becoming more and more crucial due to economic cost pressures and international competition.The Evaluation Framework for Impact Measurement was developed for specifying the evaluation phase and its objectives and tasks within the IDEAL Reference Model for the introduction and optimization of quality development within learning，education and training.First，a description of the Evaluation Framework for Impact Measurement will be provided by explaining the combination of the two targets for impact measurement：the internal development and the external impact.Then，the IDEAL Reference Model will be introduced to explain the broader picture and background for the development of the Evaluation Framework for Impact Measurement.Afterwards the example for the introduction and usage of the Evaluation Framework for Impact Measurement within the ARISTOTELE project is described.

1 Quality Development in Learning，Education and Training

Evaluation is an important and necessary part for the holistic approach of Quality Development to assure and improve the quality of learning，education and training：Quality development is a crucial task for any learning，education and training（LET）.A long－term debate on quality development regarding the different quality issues，aspects and approaches has taken place（cf.Deming 1982；Juran1951 and 1992；and for an overview Stracke 2006a）.This article focuses on the special support that evaluation can provide and thus，we can only highlight the main characteristics of quality development and its relevance in learning，education and training.

Quality development in its broad sense can be defined as follows（cf.Stracke 2006b）：

Quality development covers every kind of strategy，analysis，design，realisation，evaluation，and continuous improvement of the quality within given systems.

Quality development needs a long process to be established and integrated throughout a whole organisation.Once started，it has to be a continuous on－going circle to be successful.Quality cannot be described and fixed by a simple definition，because in itself quality is too abstract to have any impact.Therefore，quality has to be defined and specified according to the given context and situation considering the perspectives of stakeholders involved.It is important to identify the relevant aspects and to specify the suitable criteria.And it is necessary to find a consensus among the different views and perspectives to gain a common understanding of quality for the given context and situation due to different and sometimes contradictory needs and definitions of quality by all stakeholders（Crosby 1980；Deming 1986；Donabedian 1980）.Evaluation plays a key role for the quality development and the continuous improvement cycle，in particular in learning opportunities and proces－ses as intangible products and services.

2 The Evaluation Framework for Impact Measurement EFI

The Evaluation Framework for Impact Measurement EFI was developed to close a gap for assessing and optimizing the holistic quality development within learning，education and training.It combines the traditional（internal）evaluation of the processes and developed products with the（external）evaluation concerning the strategic objectives and impact that is becoming more and more crucial due to economic cost pressures and international competition.Through this connection，the Evaluation Framework for Impact Measurement EFI offers an adaptable model for the definition and specification of indicators for both，the internal lifecycle and the external relations.

The Evaluation Framework for Impact Measurement EFI is combining the measurement of two dimensions：

1.（Internal）Impact of（direct）Results as output and

2.（External）Impact of Outcomes as indirect results.

Using the Evaluation Framework for Impact Measurement EFI，the following theoretical procedure has to be applied in general：

First，the impact of the internal development and output as direct results will be measured by operative indicators.Within one given project or process the operative indicators will be related to the planned products of the project or process.The measurement of the operative indicators has to focus the two dimensions of the pilot implementations：（1）the internal processes and activities and（2）the internal results（to be tested）.

Second，the impact of the external relations and outcomes as indirect results will be measured by strategic indicators.They will be related to the strategic objectives of a given project or process：The measurement of the strategic indicators has to focus the two dimensions of the given project or process：（1）the external processes and activities（within the whole organization and external relations）and（2）the outcomes and their impact and external relations.

The following figure presents the overview of the Evaluation Framework for Impact Measurement EFI and demonstrates its relations between the two dimensions of impact measurement（internal impact of pilots assessed by the operative indicators and external impact of outcomes assesses by the strategic objectives）：

Through the combination to measure the internal impact of the internal processes and results by operative indicators and the external impact of the external processes and impact by strategic objectives，the Evaluation Framework for Impact Measurement offers a holistic approach for assessing and optimizing the tasks of quality and competence development.

Thus，the Evaluation Framework for Impact Measurement can be used to fulfil the third phase（“Evaluate”）of the generic IDEAL Reference Model for quality development in learning，education and training that will be introduced in the following chapter.

Fig.1 The Evaluation Framework for Impact Measurement EFI

3 The IDEAL Reference Model

The IDEAL Reference Model is a generic approach for the introduction and improvement of quality development in learning，education and training（cf.Stracke 2010a and 2010b）：It can be used e.g.for the implementation of the Reference Process Model of ISO／IEC19796－1 to establish a sustainable quality management including a continuous improvement cycle.Moreover the Evaluation Framework for Impact Measurement EFI was developed and designed to cover one specific phase of the IDEAL Reference Model：the third phase“Evaluate”.Therefore a short overview of the IDEAL Reference Model for quality development will be provided in the following.

To achieve a holistic quality development the needs and requirements of all stakeholders of the current learning scenario have to be considered（cf.Feigenbaum1986；Ishikawa 1985；Soin1992）.This assumption is also valid for the adoption and introduction of the IDEAL Reference Model：A systematic plan is needed for its adaptation to a specific organisation including all stakeholders（like the Evaluation Framework for Impact Measurement EFI is providing for the evaluation phase）.

The following figure shows the IDEAL Reference Model in an overview：

The abbreviation IDEAL stands for the four main phases and in addition for the learner who should be in centre of all activities and objectives for quality development.

Fig.2 The IDEAL Reference Model

The IDEAL Reference Model consists of four main phases to introduce and optimize quality development（according to our definition above）：

1.Initiate！

2.Do！

3.Evaluate！and

4.Act！

The four phases and their tasks were developed according to the Deming cycle and the ISO standard family ISO9000ff.（cf.Stracke 2006a）.They have to be fulfilled for the implementation of quality development in LET：These phases and their tasks can be realized and applied according to the specific needs and the given situation of the organization including feedback loops，individually adapted sequences and parallel implementation.

The four phases and their tasks can be described and defined as follows：

1.Phase：Initiate！

First the raising of the awareness of all stakeholders and their full involvement and participation is needed.In transparent procedures the vision for the quality development should be defined based on a common and shared understanding of quality for learning，education，and training and the innovations that should be achieved by the implementation of quality development.The long－term policies and strategies will be discussed and agreed upon the approved vision.

2.Phase：Do！

For the starting implementation the quality model has to be selected（here in our case：RFDQ）and to be adapted as an application profile.Concerning RFDQ that means the selection of the appropriate and relevant processes in a quality profile.Then the specific definitions and success criteria has to be described，for RFDQ those are the attributes of the descriptive model，mainly the methods，actors，metrics and criteria.The preparation of the implementation will include all needed decisions and developments for the organizations.And finally the realization of all planning for the quality development and its integration into organizational processes is completing this task.

3.Phase：Evaluate！

The evaluation will focus on three distinctive objects：First，the realization of the implementation of quality development itself as the main outcome of phase 2.Second，the adaptation of the quality model selected and adapted at the beginning of phase 2.And third，the evaluation of the initiation phase 1 including the revision of the vision，strategy and policies for the quality development.

4.Phase：Act！

The fourth phase is dedicated to the sustainability and long－term impact of the adaptation and implementation of the quality development.The communication and further discourse with all stakeholders will guarantee the ongoing debate and consensus building on the quality definitions and common understandings.Based on the evaluation results，the adaptation of the quality model will be revised and the vision and strategy will be reviewed.Finally it should lead the whole organization to the establishment of a continuous improvement cycle for the quality development related to all phases and their tasks.

The following figure shows the overview of the four phases of the IDEAL Reference Model and their tasks：

The IDEAL Reference Model can be used for the introduction and adaptation of process－oriented quality management and quality development：It could be demonstrated by using the example of introducing the ISO quality standard RFDQ （ISO／IEC19796－1）that the IDEAL Reference Model is an appropriate model for establishing a continuous improvement cycle based on the principles of ISO9000ff.and the Deming cycle，in particular for the implementation of sustainable quality development in learning，education and training by adapting RFDQ，the unique ISO quality standard for LET，to the specific organization and given situation（cf.Stracke 2010a and 2010b）.

4 The Usage of EFI in the ARISTOTELE project

The Evaluation Framework for Impact Measurement EFI that has been selected and adapted for the ARISTOTELE project1For more information about the ARISTOTELE project cf.www.aristotele－ip.eu.by developing the ARISTOTELE Evaluation Framework that is combining the measurement of two dimensions：

Fig.3 The Phases of the IDEAL Reference Model and their tasks

1.Impact Measurement of Pilot Implementations by outcome indicators and

2.Impact Measurement of Outcomes by usage indicators.

First，the impact of pilot implementations will be measured by so called“outcome indicators”.Within ARISTOTELE the outcome indicators were related to the planned products of the project and its processes.The measurement of the outcome indicators has to focus the two dimensions of the pilot implementations：（1）the activities（within the pilot implementation）and（2）the（implemented interim）outcomes（to be tested）.

Second，the impact of outcomes will be measured by so called “usage indicators”.They were related to the strategic objectives of ARISTOTELE：The measurement of usage indicators has to focus the two dimensions of the project and its processes：（1）the activities（within the whole organization or within specific processes）and（2）the（developed and improved）outcomes.

The following figure shows the adaptation of the Evaluation Framework for Impact Measurement EFI to the ARISTOTELE project and demonstrates its relations between the two dimensions of impact measurement（usage impact and product validation）：

In the following the structure and design of the ARISTOTELE Evaluation Framework is described in brief providing ageneral overview how the Evaluation Framework for Impact Measurement EFI was adapted for ARISTOTELE.

The ARISTOTELE Evaluation Framework combines four different evaluation categories and methods to be used and assessed：

Fig.4 The Adaptation of EFI for the ARISTOTELE project

i）Conceptual Validation through Experts＇Reviews ＆ Focus Groups；

ii）Usability Check of ARISTOTELE tools through Cognitive Walkthrough；

iii）User Validation of the ARISTOTELE tools through Functional Testing ＆ Quantitative Survey（Online Questionnaire）；

iv）Software Validation and Impact Measurement－Usability of the Platform and Impact of whole project through Functional testing（verification and validation），Qualitative Semi－Structured Interviews ＆ Quantitative Survey（Online Questionnaire）.

The methods and tools applied in each of the evaluation levels have been chosen according to the principles of“What is state of the art in evaluation？”as well as to efficiency and applicability considerations（“What is possible to measure within the application partners in the given timeframe and with the given resources？”）.The mix of methods，the different aspects to be evaluated and their order follow the principles of the IDEAL Reference Model and the EFI Evaluation Framework for Impact Measurement.

For the evaluation of the satisfaction of the target groups with ARISTOTELE tools，models and methodologies as well as for the evaluation of the integrated ARISTOTELE Platform and Tools and their impact，the following Evaluation Levels，Pilot Trials and Iteration Cycles were distinguished and defined within ARISTOTELE：

·Evaluation Level 1：Conceptual Validation，

·Evaluation Level 2：Usability Study，

·Evaluation Level 3：User Validation of ARISTOTELE Tools，

·Evaluation Level 4：Software Validation and Impact Measurement.

For the evaluation of the satisfaction of the target groups with ARISTOTELE tools，models and methodologies as well as for the evaluation of the ARISTOTELE platform and project impact，the following evaluation methods are used within ARISTOTELE：

·Level 1：a two phase process including individual experts＇assessment of the concepts and a SWOT Analysis based on the feedback by the internal and external experts（using SWOT analysis），

·Level 2：a usability study using a cognitive walkthrough approach for representatives of both application partners，

·Level 3：a standardized online survey and functional testing for tool users from the APs in the 1st evaluation iteration cycle，

·Level 4：functional testing，a semi－structured interview on usability and on impact and a standardized online survey for ARISTOTELE platform users by APs in the 2nd evaluation iteration cycle.

The following table presents the overview of the ARISTOTELE Evaluation Phases，their Evaluation Levels and Methodologies.

Tab.1 Overview of ARISTOTELE Evaluation Phases，Levels and Methodologies

The first three evaluation categories and ARISTOTELE Evaluation Levels 1－3 were selected to cover and ensure the broadest evaluation and validation of the ARISTOTELE tools and platform.

Within ARISTOTELE，not only the impact of the interim results during the Pilot Trials is evaluated，but also the impact of the Integrated ARISTOTELE Platform as thefinal ARISTOTELE result（the“outcome”according the Evaluation Framework EFI）.Therefore the ARISTOTELE Evaluation Level 4 was defined for the impact measurement of Integrated ARISTOTELE platform through In－depth Interviews and Online Survey.

Through their careful combination the different Evaluation Levels and their methodologies and instruments，ARISTOTELE is covering both categories of the Evaluation Framework for Impact Measurement EFI：（1）the internal evaluation，that are the processes and outcomes for measuring the product validation within ARISTOTELE （in the Evaluation Levels 1，2，and 3）as well as（2）the external evaluation that are the activities and outcomes for measuring the usage impact within ARISTOTELE（in the Evaluation Level 4）.

ARISTOTELE has selected a strong mixture of different methodologies within the four Evaluation Levels to ensure rich and diverse evaluation perspectives and results：In particular the mix of quantitative and qualitative methodologies was addressed and established to achieve both：baseline data as well as quality feedback.

By using the Evaluation Framework for Impact Measurement EFI as the basis for the ARISTOTELE Evaluation Framework，ARISTOTELE could assess and optimize the internal development processes and results as well as the external impact.Thus，ARISTOTELE could prove that the introduction and adaptation of the Evaluation Framework for Impact Measurement EFI is not only feasible but leading to a valuable and sustainable evaluation and validation.

5 Summary and Future Prospects

This article has presented and analysed the Evaluation Framework for Impact Measurement EFI for the combination of internal and external evaluation and for the realization of the third phase of the IDEAL Reference Model for sustainable quality development as a valuable instrument for its introduction and implementation.

Quality development is always depending on the given situation.The definition of quality development and the benefits of quality standards were presented in particular relation to LET.The establishment of a continuous improvement cycle is one main important benefit as the involvement of all stakeholders is one main aim for the introduction of quality development.It has been shown that the Evaluation Framework for Impact Measurement EFI is offering a valuable support for the adoption and implementation of quality development.It is an appropriate means for assessing and optimizing the quality and impact and in consequence，for supporting the adoption and implementation of a sustainable quality development that is covering all learning and business processes.We have pointed out the main tasks and potentials for its adaptation that is always needed for gaining a longterm impact leading to overall quality development with continuous improvement.

For the facilitation and improvement of the broad application of quality development，the IDEAL Reference Model was introduced and presented as an instrument for adoptingand implementing quality development by establishing a continuous improvement cycle.

Finally，the adaptation of the Evaluation Framework for Impact Measurement EFI within the ARISTOTELE project for the ARISTOTELE Evaluation Framework was explained demonstrating its feasibility and benefits for internal and external evaluation.

In summary，it can be stated that the Evaluation Framework for Impact Measurement EFI is a suitable and valuable instrument for the introduction and implementation of internal and external evaluation to assess and optimize the quality development and impact within learning，education and training.EFI is a holistic evaluation approach facilitating the adaptation and implementation of the IDEAL Reference Model as a helpful instrument for its objective as well as supporting sustainable quality development in learning，education，and training in general.

［1］ ARISTOTELE （2012）.The ARISTOTELE Evaluation Framework：Evaluation Methodologies and Pilot Trials Definitions（D9.2）.

［2］ BERG B L.（1989）.Qualitative Research Methods for the Social Sciences.Allyn and Bacon，Boston.

［3］ CHERMACK T J，KASSHANNA B K.The Use and Misuse of SWOT Analysis and Implications for HRD Professionals.In：Human Resource Development International 2007，10（4）：383－399.

［4］ CHESBROUGH H W，WEST J，VANHAVERBEKE W.（2006）.Open Innovation：Researching a New Paradigm.Oxford：Oxford University Press.

［5］ CROSBY P B.（1980）.Quality is Free.The art of making quality certain.New York：McGraw－Hill.

［6］ DEMING W E.（1986）.Out of the Crisis.Cambridge，MA：MIT.

［7］ DEMING W E.（1982）.Quality，productivity and competitive position.Cambridge，MA：MIT.

［8］ DONABEDIAN A.（1980）.The Definition of Quality and Approaches to Its Assessment［＝ Explorations in Quality Assessment and Monitoring，vol.1］.Ann Arbor：Health Administration Press，1980.

［9］ DRUCKER P F.（1999）.Management Challenges of the 21st Century.Harper Business，New York.

［10］ FEIGENBAUM A.（1986）.Total Quality Control.Engineering and management.New York：McGraw－Hill.

［11］ GASSMANN O，ENKEL E，CHESBROUGH H W.The future of open innovation.R＆D Management，2010，40（3）：213－221.

［12］ ISHIKAWA K.（1985）.What is Total Quality Control？The Japanese Way.Englewood Cliffs，NJ：Prentice－Hall.

［13］ ISO9000：2005 （2005）.Quality Management Systems.Fundamentals and Vocabulary.Geneva：International Organisation of Standardization（ISO）.

［14］ ISO9001：2008 （2008）.Quality Management Systems.Requirements.Geneva：International Organisation of Standardization（ISO）.

［15］ ISO／IEC19796－1：2005 （2005）.Information Technology－Learning，Education，and Training—Quality Management，Assurance and Metrics—Part 1：General Approach.Geneva：International Organisation for Standardization（ISO）.

［16］ JURAN J M.（1992）.Juran on quality by design.The new steps for planning quality into goods and services.New York：Free Press.

［17］ JURAN J M.（Ed.）（1951）.Quality Control Handbook.New York：McGraw－Hill.Patton，M.Q.（2002）.Quantitative evaluation and research methods.Newbury Park：Sage Publications.

［18］ SINGH K.（2007）.Quantitative Social research methods.New Delhi：Sage Publication.

［19］ SOIN S S.（1992）.Total Quality Essentials.New York：McGraw－Hill.

［20］ STRACKE C M.（2013）.“Open Learning：The Concept for Modernizing School Education and Lifelong Learning through the Combination of Learning Innovations and Quality”；in：Stracke，Christian M.（Ed.）：Learning Innovations and Quality：The Future of Digital Resources，Berlin：Logos.p.15－28.

［21］ STRACKE C M.（2012）.“Learning Innovations and Learning Quality：Relations，Interdependences，and Future”；in：Stracke，Christian M.（Ed.）：The Future of Learning Innovations and Learning Quality.How do they fit together？，Berlin：Gito.p.13－25.

［22］ STRACKE C M.（2010a）.“Quality development and standards in learning，education，and training：adaptation model and guidelines for implementations”；in：Информатизацияобразованияинауки［＝Информике（Informika），ISSN2073－7572］；Vol.7（3），2010.Moscow （Russian Federation）：Informika.p.136－146.

［23］ STRACKE C M.（2010b）.“Quality and Standards in Learning，Education，and Training：The Adaptation Model IDEA for the Introduction of Quality Development”；in：Proceedings of the International Conference on the Past and Future of e－Learning Standards.Tokyo（Japan）.p.26－36.

［24］ STRACKE C M.（2009）.“Quality Development and Standards in e－Learning：Benefits and Guidelines for Implementations”；in：Proceedings of the ASEM Lifelong Learning Conference：e－Learning and Workplace Learning.Bangkok（Thailand）［online available at：http：／／www.qualitydevelopment.eu／docs］.

［25］ STRACKE C M.（2006a）.Process－oriented Quality Management.In Pawlowski J.＆ Ehlers，U.（Eds.）European Handbook on Quality and Standardisation in E－Learning.Berlin：Springer，79－96.

［26］ STRACKE C M.（2006b）.Interoperability and Quality Development in e－Learning.In Proceedings of the Asia－Europe e－Learning Colloquy.Seoul：e－ASEM ［online available at：http：／／www.qualitydevelopment.eu／docs］.

［27］ THOMPSON J L，MARTIN F.（2005）.Strategic management：Awareness and change.London：Thomson learning.