Evaluering – oppskrift på TOR

Evaluering – oppskrift på TOR
Bistandstorget Evaluering – oppskrift på TOR w w w. R e s u l t a t s t y r i n g . n o Anette P. Simonsen 24. oktober 2012 9:00-16:00

w w w. R e s u l t a t s t y r i n g . n o
INTRO Evaluering som en del av M&E w w w. R e s u l t a t s t y r i n g . n o

Monitoring and Evaluation
WHO? Internal, involving project implementers Can be internal or external to organisation WHEN? Ongoing During project Periodic and after project WHY Check progress, inform project decisions and remedial action update project plans support accountability Assess progress and worth, identify lessons and recommendations for longer term planning and organisational learning provide accountability Link to LFA Focus on activities, outputs and short term outcomes Focuses on outcomes and overall goal. Mål- og resultatstyring Anette P. Simonsen

Definitions M&E Monitoring:
refers to the regular follow-up and assessments during the implementation of a project or programme. normally provides information on expenditure, activities and outputs and a certain extent outcome is used for adjusting the course of the project/programme as well as writing progress reports. Evaluation is the systematic and independent assessment of a project/ program that is performed at certain stages of the project (e.g. mid-term, end of project). would provide information on output, outcome and impact is used for adjusting the course or assessing continuation or ending a project. Mål- og resultatstyring Anette P. Simonsen

M&E definition again.. Evaluation approaches often have high methodological standards, a separate budget and often scientific requirements for measurements and analysis. Monitoring on the other hand is a routine task, regularly ‘checking up on how things are’ and recording this information in a way that is useful both w w w. R e s u l t a t s t y r i n g . n o Observing Change N-P-Aid

Norad Regelverk for støtte til sivilsamfunnsaktører
Se spesielt side 3: «Beslutning om støtte skal tas etter en helhetsbedømmelse av søknaden. Hovedsakelig er kriteriene delt i to; de som gjelder for vurdering av organisasjonen og de som gjelder vurdering av bistandsvirksomheten. Sentrale punkter i vurderingen av organisasjon er: Kapasitet og kompetanse til å nå organisasjonens mål Organisasjonens kostnadseffektivitet, det vil si evnen til å gjennomføre organisasjonens mål med lavest mulige kostnader. Systemer for resultatoppfølging Evne til å identifisere og håndtere risiko w w w. R e s u l t a t s t y r i n g . n o

Økonomireglementet sier:
§ 16 Evalueringer Alle virksomheter skal sørge for at det gjennomføres evalueringer for å få informasjon om effektivitet, måloppnåelse og resultater innenfor hele eller deler av virksomhetens ansvarsområde og aktiviteter. Evalueringene skal belyse hensiktsmessighet av eksempelvis eierskap, organisering og virkemidler, herunder tilskuddsordninger. Frekvens og omfang av evalueringene skal bestemmes ut fra virksomhetens egenart, risiko og vesentlighet. w w w. R e s u l t a t s t y r i n g . n o

M&E har 3 formål M&E skal bidra til: Kontroll av virksomheten - Forsikre interessenter om at midlene blir brukt på riktig måte, for avtalte mål og at de ulike samarbeidspartene oppfyller sine forpliktelser til beslutningsprosesser – gi informasjon som underbygger de valg organisasjonene står overfor, når de må fatte beslutninger om prosjekter/ programmer skal fortsette, avsluttes eller endres til å fremme organisatorisk læring – ved å gi aktuelle aktører kunnskap og informasjon om prosjekter/ programmer samt den sammenhengen de virker inn i. Ingen begrensning i hva som er mulig læring. w w w. R e s u l t a t s t y r i n g . n o Kruse & Forss (2000)

Preparing for Evaluation
Look back at: Does the project/programme have a clear goal hierarchy and a set of M&E plans? Have the plans been monitored? Are there good and regular monitoring reports? Are the project staff available /still contactable? Are the stakeholders involved? Are the donor(s) involved? w w w. R e s u l t a t s t y r i n g . n o From Local Livelyhoods

Action Plan for Evaluation
- Approving the activity and the budget w w w. R e s u l t a t s t y r i n g . n o

Action Plan versus TOR The Action plan Formulates the internal “project” of performing an evaluation It documents the approval of the decision to perform an evaluation and the budget for it. The TOR is a work description for the team performing the evaluation It constitutes only one part of the Action Plan w w w. R e s u l t a t s t y r i n g . n o

Action Plan for Evaluation
Background and justification Need for evaluation at this point Previous relevant evaluations / reviews Who will be involved in evaluation / cooperation with other stakeholders/donors on the evaluation Target group (audience) Objective of evaluation Any special requirements to team composition, method etc.? Outputs/main activities Planning phase Implementation Follow up Risk and uncertainties Organisation and responsibilities Budget w w w. R e s u l t a t s t y r i n g . n o

Justification for evaluation
Donor Donor Donor Donor Norwegian NGO Partner Tanzania Stake- holders Project Stake- holder w w w. R e s u l t a t s t y r i n g . n o IMPLEMENTATION LEVEL Stake- holders Stake- holder

1. Background and Justification
Do we really need an evaluation at this point? What is the problem we are trying to solve? E.g. The M&E system is weak so we need to increase level of information on progress and or achievement of results? The target group is complaining on quality of services? We need to secure funding by a more objective voice on achievement of results? w w w. R e s u l t a t s t y r i n g . n o

M&E in project Cycle w w w. R e s u l t a t s t y r i n g . n o

When? Reporting to donors
Example from 5 year programme: Program year Deadline and type of Report Reviews and evaluations 2009 May 2010 –Progress report 2010 May Progress report 2011 May Progress report 2012 May Periodic Results Report 2013 May 2014 –Final Report w w w. R e s u l t a t s t y r i n g . n o

Why? Status report Norad
3.1 Describe the reviews/evaluations that have been carried out during the year In what way(s) did the review/evaluation contribute to learning and improvement of the programmes? Are external reviews submitted to Norad’s data base for evaluations? (state type of evaluation – external, internal or mix external/internal) w w w. R e s u l t a t s t y r i n g . n o

Timing of evaluations Appraisal/ evaluation – before start-up Mid-Term Evaluation – towards middle of programme period End-Term evaluation – at the end of programme Formative evaluations – during programme Summative evaluations – at the end Ex-post evaluations – some time after Impact evaluations – at the end – some time after w w w. R e s u l t a t s t y r i n g . n o

OECD evaluation on DAC principles 1998:
Evaluations most useful when they are linked to current and future programmes rather than historical perspectives (although accountability requires the latter type) w w w. R e s u l t a t s t y r i n g . n o

2. Who will be involved in evaluation / coop with other stakeholders
Discuss evaluation plans and design with the stakeholders (staff, partners, benficiaries, others) and include the primary target group. Who in the commissioning organisation is familiar with project/programme Who are the stakeholders? Which other donors are involved in Project/programme or even thematic area og geographical area. Others? w w w. R e s u l t a t s t y r i n g . n o

3. The target group (audience)
Who will read, learn from and use the evaluation report should be one of the a determining questions in developing the terms of reference and designing the evaluation. In other words who is the primary target group that should be able to make use of the report? Should e.g. Norwegian NGO / the partner /stakeholders / members / taxpayers/ donors make use of the evaluation? w w w. R e s u l t a t s t y r i n g . n o

OECD evaluation on DAC principles 1998:
Evaluations are performed mostly because donors ask for it. Little attention is paid to the demand, the use and usability of evaluations, and differentiation according to different audiences w w w. R e s u l t a t s t y r i n g . n o

4. Objective of the evaluation
The purpose of evaluations is normally a combination of a need for control (accountability) to contribute to decision making and for learning Partner – all 3 Norwegian all 3 Donor…. Who cares? w w w. R e s u l t a t s t y r i n g . n o

Objective of Evaluation
The implementing partners will have increased knowledge and lessons learned in their continuous work within the thematic areas in their respective countries. The Norwegian NGO will be better equipped for future disasters of similar order of magnitude, both in the initial and recovery phase. The private and corporate donors and the Ministry of Foreign Affairs will gain knowledge about the results of their donations. w w w. R e s u l t a t s t y r i n g . n o Who is the target group here?

Objective of Base line study
The main objectives of the baseline survey are: To get an initial baseline standing/situation/ context of the current and existing governance and livelihood situation, systems and structures in the impact areas To establish baseline indicators for the programme that will imply a clear monitoring and evaluation plan (M&E) and platform for impact evaluation (IE) To source information that is required to describe qualitatively and quantitatively the outcome indicators and output indicators for the Caritas Malawi programme on governance and livelihood. w w w. R e s u l t a t s t y r i n g . n o Who is the target group here?

Have conversations on the internal functions, including financial management, in each organisation involved in the visit in order to strengthen the organisation Have conversations about the cooperation (communication, interaction) between the organisations in order to improve cooperation The review should be summarized in a report in which the strengths and weaknesses are described, and proposed changes in procedures and communication within and between organisations are described. w w w. R e s u l t a t s t y r i n g . n o Who is the target group here?

Conducting a mid-term external review in 2011 will establish an overall picture of whether the five-year programme is on track towards 2013 and create a basis for corrective actions and the planning ahead for the final 2 years of implementation. The review will be an input to a mid-term meeting we want to hold in the second part of 2011. w w w. R e s u l t a t s t y r i n g . n o Who is the target group here?

Formålet med evalueringen er å: å gi LNU og bevilgende myndigheter økt kunnskap om i hvilken grad de tiltakene som får støtte fra LNU bidrar til å nå støtteordningens prosjektmål (måloppnåelse). å gi LNU og bevilgende myndigheter økt kunnskap om eventuelle ikke-planlagte effekter (positive eller negative) av de tiltakene som får tilskudd. å gi LNU et grunnlag for å videreutvikle sin forvaltning av støtteordningen. Formålet med evalueringen er avgrenset på bakgrunn av de økonomiske rammene for evalueringen. w w w. R e s u l t a t s t y r i n g . n o Who is the target group here?

Organisational review:
w w w. R e s u l t a t s t y r i n g . n o PREBEN LINDØE for BN 2006

5. Any special requirements to Team, method etc.
Team: Internal / external Special expertise necessary Method: Level of data collection Special tools to be used NB! The action plan should not be too detailed at this point. Difficult to change. w w w. R e s u l t a t s t y r i n g . n o

6. Outputs/ main activities
The Action plan is split into Planning phase Implementation Follow up w w w. R e s u l t a t s t y r i n g . n o

Did you forget about Learning?
Example of Outputs and activities in Action Plan Planning phase Outputs and activities Timeframe Responsible Plan of Action AKV,KH;DH Approval Plan of Action AKV TOR AKV,KH,DH Select internal team members SLM Go out for tender TM/DH Implementation phase Consultant chosen AKV,KH Prepare for evaluation Evaluation (by team) Report draft Completion phase Quality assurance of the reports are done by relevant stakeholders Final report Did you forget about Learning?

7. Risk and uncertainties
Any project meets uncertainties: e.g Risk factors Mitigating activities An evaluation in four parts over several months lead to tiredness and lack of interest of the process among stakeholders and organization Ensure internal communication about process and sub results along the way and continuous involvement of stakeholders. Reputational risk in case of exposure of mistakes made by NorCross during the programme Communication strategy is developed. Follow-up of recommendations and plan for learning is developed. NorCross initiatives to expose and learn from own mistakes decrease reputational risks (precautionary action). Low participation by the appointed resource group and ToR groups Involvement in the plan for the process by the members to ensure ownership. Appoint one responsible in each ToR group. Announce workshop in due time. Limited time frame and occasions for participation Context in host countries hinders and/or complicates evaluation implementation Research ahead and involvement of NSs Difficulties in finding qualified consultants for evaluation implementation (technical knowledge in various areas is necessary) Tender process through Mercell. Separate invitations (through Mercell) to tender for companies recognized by NorCross after research w w w. R e s u l t a t s t y r i n g . n o

9. Organisation and responsibillities
Function Title Tasks Owner of evaluation Director International Division Responsible Anne Kirsti Vartdal Kjersti Haraldseide Dagne Hordvei Carry out activities in accordance with this PoA: -Initiate, choose consultant and team, prepare, assist evaluation team, comment on report, follow up findings. Steering committee Management in International division Approve PoA and budget Approve Report Resource group Adviser 1 Adviser 2 Adviser 3 Adviser 4 Rep delegate section w w w. R e s u l t a t s t y r i n g . n o

Læring – referansegruppe evalueringsavdelingen norad
Det er evalueringsavdelingen sin målsetning at evalueringsprosesser skal være deltakende og inklusive og fremme størst mulig læring. Det etableres derfor ofte en referansegruppe bestående av interessenter, berørte parter, ev. faglige ressurspersoner, fagavdeling og evalueringsavdelingen. Disse kommenterer arbeidet underveis. I den grad det er hensiktsmessig ønsker også evalueringsavdelingen å arrangere seminarer underveis og i etterkant. w w w. R e s u l t a t s t y r i n g . n o

OECD evaluation on DAC principles
There are reason for concern regarding the way in which lessons and experience, gained during the evaluation process, are transmitted to current operation managers. w w w. R e s u l t a t s t y r i n g . n o

10. Budget example 1 Items Activity number Budget cost (NOK) Consultant Haiti 66% Sri Lanka 34% Travels 55.000 Misc 5.000 Total w w w. R e s u l t a t s t y r i n g . n o

10. Budget example 2 Items Activity number Budget cost (NOK) PREPARATION -Stakeholder meeting -M&E system update IMPLEMENTATION -Consultant -Travels Haiti 66% Sri Lanka 34% 55.000 FOLLOW UP -Dissemination -internalisation -Misc 5000 50.000 5.000 Total w w w. R e s u l t a t s t y r i n g . n o

TOR – Terms of reference w w w. R e s u l t a t s t y r i n g . n o

Action Plan versus TOR The Action plan Formulates the internal “project” of performing an evaluation It documents the approval of the decision to perform an evaluation and the budget for it. The TOR is a work description for the team performing the evaluation It constitutes only one part of the Action Plan w w w. R e s u l t a t s t y r i n g . n o

Tor Norad 1 Background • Short description of the programme that will be reviewed, based on the Agreement, Programme Document(s) and appraisal • Why the review is initiated • Team composition and leadership. (e.g. Embassy/Norad team, joint donor/Partner, external team or combination of these) 2 Purpose, context and intended use • Description of the main purpose, context and intended use (stakeholders) 3 Scope of work Type of assessments may include: (for definition of the review criteria, see Annex III) • Efficiency • Effectiveness • Impact (if the programme has been operating for some years) • Relevance • Sustainability • Risk management • Particular concerns to be investigated • Audit • Anti-corruption measures 4 Implementation of the review • Sources of information and methodology to be employed • Division of responsibility between the consultant/team, the Embassy, other donors and the Partner(s) • Timetable for preparation, field work and finalisation of report • Budget 5 Reporting • Description of required report format • The need for an introduction summary with main conclusion on lessons learned and recommendation(s) • Report in electronic form and/or paper, language w w w. R e s u l t a t s t y r i n g . n o

Evalueringssprinsipper (Eval Norad web)
Evalueringsprinsipper Objektive, etterprøvbare og gjennomsiktige: Vurderinger skal være basert på faktiske utsagn; utsagn skal være basert på troverdige (reliable) data eller observasjoner. Relevante interessenter i Norge eller mottakerlandet skal konsulteres i forbindelse med evalueringers tilrettelegging og gjennomføring, inklusive i utarbeiding av oppgavebeskrivelse og diskusjon av rapportutkast. Upartiske: Evalueringer skal gi en avbalansert fremstilling av styrker og svakheter. I den grad interessenter har forskjellige synspunkter skal disse tas med i evalueringen. Uavhengige: Medlemmer av evalueringsteamet må ikke ha vært personlig engasjert i de aktiviteter som skal evalueres. Virksomheter som gjennomfører evalueringer må ikke ha vært involvert i forberedelse eller gjennomføring av aktivitetene. w w w. R e s u l t a t s t y r i n g . n o

Description of Project /issue to be evaluated
w w w. R e s u l t a t s t y r i n g . n o

Description of project to be evaluated
Basically a summary of project document Especially important is the goal hierarchy and any changes in that along the way M&E System must be described and commented on Previous or other relevant evaluations should be mentioned here Budgets must be included too. w w w. R e s u l t a t s t y r i n g . n o

Scope of evaluation The boundaries of the evaluation (e.g time period, area) Evaluation criteria and evaluation questions w w w. R e s u l t a t s t y r i n g . n o

Scope of evaluation The formulation of “Evaluation questions” is one method that will end up with a precise definition of what should be evaluated. Evaluation questions represent “what one wants to know through evaluation.” When formulation the evaluation questions it is useful to consider which of the DAC Principles for Evaluation of Development Assistance that should be covered w w w. R e s u l t a t s t y r i n g . n o

Development Goal (expected Impact) Project Goal (expected Outcome)
One goal -LFA spread out Development Goal (expected Impact) Project Goal (expected Outcome) OUTPUT 1 OUTPUT 2 Mål- og resultatstyring Anette P. Simonsen OUTPUT 3 OUTPUT 4 Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Activities: 1. 2. 3 4 5 Inputs Inputs Inputs Inputs

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) efficiency, effectiveness, impact relevance, sustainability. w w w. R e s u l t a t s t y r i n g . n o

Evalueringskriterier(web Norads eval avd)
Den enkelte evaluering bør normalt belyse følgende: Relevans: Er aktiviteten relevant i forhold til norsk bistandspolitikks målsetninger og strategier? Er aktiviteten relevant i forhold til mottagerlandets behov og prioriteringer? Er den relevant i forhold til det utviklingsproblem den skal se på? Måloppnåelse (effectiveness): Er de primære målsetninger for aktiviteten nådd? Er de planlagte resultater nådd? Bærekraft (sustainability): Hva er langtidsvirkningen av aktiviteten? Vil aktiviteten kunne videreføres også etter at bistandsfinansieringen er avsluttet? Er det lokalt eierskap? Produktivitet (efficiency): Er investeringene og driftsomkostningene berettighet? Kunne de samme resultater ha blitt oppnådd med færre midler? Virkning (impact): Positive og negative primære og sekundære langtidseffekter produsert av aktivitetene, direkte eller indirekte, tilsiktede eller utilsiktede. w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Evaluation criteria: Efficiency Effectiveness Impact Relevance Sustainability w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Efficiency Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a programme or a project, it is useful to consider the following questions: • Were activities cost-efficient? • Were objectives achieved on time? • Was the programme or project implemented in the most efficient way compared to alternatives? Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a programme or a project, it is useful to consider the following questions: • What has happened as a result of the programme or project? • What real difference has the activity made to the beneficiaries? • How many people have been affected? Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions: • To what extent did the benefits of a programme or project continue after donor funding ceased? • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Effectiveness A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a programme or a project, it is useful to consider the following questions: • To what extent were the objectives achieved / are likely to be achieved? • What were the major factors influencing the achievement or non-achievement of the objectives? w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. When evaluating the impact of a programme or a project, it is useful to consider the following questions: • What has happened as a result of the programme or project? • What real difference has the activity made to the beneficiaries? • How many people have been affected? w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Relevance The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a programme or a project, it is useful to consider the following questions: • To what extent are the objectives of the programme still valid? • Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives? • Are the activities and outputs of the programme consistent with the intended impacts and effects? w w w. R e s u l t a t s t y r i n g . n o

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions: • To what extent did the benefits of a programme or project continue after donor funding ceased? • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? w w w. R e s u l t a t s t y r i n g . n o

Evalueringsavdelingen Norad
Målene for evalueringsavdelingen er å: vurdere måloppnåelse og resultater i forhold til vedtatte planer vurdere forutsette og uforutsette virkninger av bistandstiltak vurdere om ressursbruken står i rimelig forhold til oppnådde resultater systematisere erfaringer og kunnskap for å kvalitetssikre og forbedre fremtidig bistandstiltak gjennom gode læringsprosesser kartlegge kunnskapshull/ vurdere de underliggende antakelsene for ulike deler av utviklingssamarbeidet gi informasjon til dem som bevilger og utformer ny politikk, og allmennheten w w w. R e s u l t a t s t y r i n g . n o

Methodology w w w. R e s u l t a t s t y r i n g . n o

Methods for collection of info/data
Quantitativ Quantitative Surveys Rating Qualitative Desk reviews Case studies Qualitative surveys Interviews w w w. R e s u l t a t s t y r i n g . n o

Ex: Methodology LNU ønsker at følgende metoder skal benyttes: Intervju med utvalgte brukere av støtteordningen og deres partnere i perioden 2004 – 2008 for å vurdere effekten av de tiltakene som har fått støtte i forhold til støtteordningens mål. Intervju eller spørreskjema for alle organisasjoner i Norge som har brukt ordningen i perioden 2004 – 2008 for å vurdere LNUs forvaltning av støtteordningen. Intervju med saksbehandlere Intervju med medlemmer av fordelingsutvalget for støtteordningen. Dokumentanalyse av styringsdokumentene fortøtteordningen, herunder retningslinjene. w w w. R e s u l t a t s t y r i n g . n o

Collecting true qualitative data
By nature , much Qualitative data are more sensitive and personal than Quantitative data. While surveys can be administrated to more people in a shorter time (breadth), interviews probe for more information from fewer individuals (depth). w w w. R e s u l t a t s t y r i n g . n o From: Real life evaluation

Collecting true qualitative data
Interviewees may feel uncomfortable for a variety of reasons; including cultural linguistic, age related, gender related and political reasons. Interviewees who are less than fully forthcoming obstruct data collection. w w w. R e s u l t a t s t y r i n g . n o From: Real world Evaluation

A bit of both please Quantitative – breadth in insight quantitative surveys can be administrated to more people in a shorter time Qualitative – depth in insight qualitative interviews probe for more information from fewer individuals (depth). w w w. R e s u l t a t s t y r i n g . n o

NB! Please Note! NB! Please note: there is a continuum between quantitative and qualitative data definition and collection. w w w. R e s u l t a t s t y r i n g . n o QUANTITATIVE QUALITATIVE

Qualitative / Quantitative
Indicators / data: Qualitiative Quantitative Collection methods: Qualiatiative w w w. R e s u l t a t s t y r i n g . n o

Qualitative data collection
Interviews are often characterized by: Structured (all asked the same questions). Semi-structured (questions vary). Unstructured (no protocol). Qualitative interviewing is typically semi- structured (B) which is useful as it offers focus and flexibility but also most demanding (as one must record data while adjusting wording & order of questions, preserving focus, and follow up on conversational leads). w w w. R e s u l t a t s t y r i n g . n o From: Real world Evaluation

Methodology – Evaluation Norad East Africa NGO Eval. April 2011
The evaluation team shall develop an appropriate methodology that can respond to the information needs arising from the list of evaluation questions. The evaluation team may make use of various empirical methods, e.g. questionnaire surveys, interviews, focus group discussions and participatory rural appraisal. The presentation of the methodology should indicate any significant advantages and limitations of the chosen approach. w w w. R e s u l t a t s t y r i n g . n o

Ex of Methodology The methods employed in this evaluation will include the following: review of available documents and literature, individual interviews and focus groups discussions. It will be important to allow intended beneficiaries to tell their stories without necessarily sticking to a structured questionnaire as this will allow interviewees to release their emotions alongside the needed information. The purpose of employing several methods to gather data (triangulation) is to see whether the different sets of data material confirm or may complement each other, and hence provide a deeper understanding of the programme, its activities, effects and challenges. w w w. R e s u l t a t s t y r i n g . n o

Sikre bruk av M&E Systemet I evalueringen

Ex: Monitoring health by indicators
Outcome 2: Risk Groups adopt good practices of prevention and treatment of HIV/AIDS # of people referred for councelling and testing # of People reached by HIV/AIDS projects in he frame of the programme (CORE indicator) # of test taken at local clinic pr year. Output 1: NS engage youth in youth peer education poagramme # of youth certified as peer educators # of youth who have received training in peer education Output 2: NS provide education for risk groups on sexual health (STD and HIV/AIDS) # of people educated on sexual health % of partcipants satisfied with the courses. Output 3: NS have adequatly trained staff and volunteers to adress HIV/AIDS according to org. standards and in gender sensitive manner. # of volunteers and staff participating in HIV/AIDS prevention activities in the frame of this programme (CORE Indicator) Output 4: NS promotes HIV/AIDS voluntary testing and councelling # of information campaigns

Administrativ Controller visits
Eksempel på M&E plan Årlig rapport m/ regnskap (Partner) Indikator survey Partner Administrativ Controller visits (Donor) Resultat gjennomgang / (Joint) 2011 Baseline Jan 2012 2012 Årsrapport for 2012 i mars 2013 Februar 2013 Oktober 2012 Controller visit August 2012 Workshop om baseline og indikatorer 2013 Årsrapport for 2013 i mars 2014 Februar 2014 Oktober 2013 Gjennomgang av organisasjonens administrative systemer 2014 Sluttrapport for i mars 2015 Februar 2015 Slutten av 2014 gjennomgang av prosjekt med fokus på måloppnåelse Mål- og resultatstyring Anette P. Simonsen

Team composition w w w. R e s u l t a t s t y r i n g . n o

Team composition Control versus learning: Internal team versus external to project Origin/language Norwegian, international/local base Qualifications w w w. R e s u l t a t s t y r i n g . n o

Internal versus External Evaluator
Advantages INTERNAL Advantages EXTERNAL Knows the situation, history, problems, processes etc Not personally involved -easy to be objective Understands and can interpret personal behaviour and attitudes Free from local bias Is known to stakeholders so pose no threat anxiety or disruption Can bring fresh perspective and insights Can follow up on recommendation May have broader experience Is often less expensive and Quick to hire ( no negotiations etc) Can serve as arbitrator or facilitator between parties May provide mor oportunity to build local evaluation capacity Can bring local stakeholders into contact with additional sources Mål- og resultatstyring Anette P. Simonsen ADVANTAGES

Internal versus External Evaluator
Disadavantages INTERNAL Disadavantages EXTERNAL Hard to be objective May not know local situation May avoid looking for facts or forming nagative conclusions tha reflect badly on community individuals Ignorant of constraints affecting feasibility of recommendations Tends to accept the assumptions of the stakeholders and community May be perceived as adversary and arouse anxiety May be too busy to participate fully May be expensive May be part of the local community and may be constrained by conflicts of interest Requires more time for contract negotiations, orientation and monitoring May not be trained in evaluation methods and technical expertice May be unfamiliar with local political and economic environment Mål- og resultatstyring Anette P. Simonsen DISADVANTAGES

The TOR The TOR: In planning phase the TOR is part of the tender documents and e.g. thus describes the qualifications that is saught for the evaluation. After the team has been selected and contracted, the TOR is updated with the names of the team members and enclosed the Evaluation as an annex w w w. R e s u l t a t s t y r i n g . n o

Ex of team selected team composition
Team composition / representation: Team leader, external consultant YRCS programs’ coordinator Representative of the YRCS OD department YRCS health Coordinator Resource persons: The YRCS CBHD programme manager The YRCS CBHD programme officer President / Secretary general of Hajjah and Hodeidah branches and sub-branches (4 people) NRC OD advisor SRC health advisor DRC OD advisor IFRC team member w w w. R e s u l t a t s t y r i n g . n o

Reporting w w w. R e s u l t a t s t y r i n g . n o

Reporting deliverables &formats
An inception report detailing the strategy to be utilised, defining clear roles and responsibilities of the consultant, work plan, budget and a list of stakeholders to be consulted. This is expected within the first two days of commencement of contract; A draft baseline report for the governance and livelihood programme to be presented to Caritas Malawi. This is to be presented on the 8th day of the commencement of the consultancy. A draft report discussed between Caritas Malawi and the Consultant on the 9th day of the consultancy A final baseline report on governance and livelihood after incorporating comments from Caritas Malawi and Caritas Norway. This should be submitted in two hard copies and one electronic copy on the 12th day of the commencement of the consultancy. w w w. R e s u l t a t s t y r i n g . n o

Learning w w w. R e s u l t a t s t y r i n g . n o

Dissemination - variations
Transparent and complete dissemination. Evaluation results should be placed in public domain and widely disseminated, ensuring that information is readily accessible to all stakeholders. An initial dissemination list (Practice 5.10) should be employed to ensure the evaluation report or summaryreaches its intended audience (per the TOR Appropriate dissemination. The dissemination of the evaluation report may take a variety of forms that are appropriate to the specific audience. This can include posting reports or excerpts/summaries on a community notice board or on the internet, and presentations at planning meetings, community meetings, and industry conferences. Such considerations are especially important when sharing reports with communities. w w w. R e s u l t a t s t y r i n g . n o

Dissemination - variations
Differentiate between Internal and external dissemination. This may be done for multiple reasons: with sensitive issues that should remain internal, to protect the identity and avoid or reduce any harm to evaluation subjects, and to improve the comprehensibility of the report in a summary or simplified form for external audiences. Disseminate through Evaluation Database. All evaluation reports should be submitted for record in international Database on evaluation w w w. R e s u l t a t s t y r i n g . n o

Bruk av M&E info For at resultatmålingen skal ha noen hensikt, må resultatene drøftes, vurderes og brukes av ledelsen til å ta beslutninger om tiltak som skal forbedre virksomheten og gi bedre måloppnåelse. w w w. R e s u l t a t s t y r i n g . n o Mål- og resultatstyring SSØ s 8

Learning Quality of learning depends on enough slow time. (opportunities) w w w. R e s u l t a t s t y r i n g . n o Fritt etter Knut Harald Ulland Utviklingsfondet

Learning – a crime? ‘ it is almost as if my organisation considers learning as a crime rather than a behaviour we’re trying to encourage’. A crime (or learning) needs MMO to incur: Motive Means Opportunity w w w. R e s u l t a t s t y r i n g . n o Bruce Britton (2005)

MMO for Learning Motive: Develop systems for acknowledging and rewarding learning Means: Map out where expertise lies in organisation. Develop team working E.g introduce methods such as mentoring, coaching, action learning and communities of practice Opportunity: Create space for learning Build time and resources for reflection and learning intor project/programme proposals Involve staff/partners alongside external consultants in review and evaluation teams. w w w. R e s u l t a t s t y r i n g . n o B. Britton ( 2005)

Time Schedule w w w. R e s u l t a t s t y r i n g . n o

Ex Time schedual for external mid tern review
Task/Activity Timeline Procure Team By week 12: Contract signed by Team By week 14: Preparation work by Teamleader in Oslo (finalization of evaluation plan, document collection, consultations with resource persons, and initial document review) Weeks 15: Desk study/document review and fieldwork preparations Week 17: Field interviews and visits Uganda - week 21: Zambia – week 22: Mozambique – week 23: Data analysis and drafting the report Week 24: Submission of draft to SOS Children’s Villages Draft assessed by SOS Children’s Villages Week 25: Incorporation of comments and inputs from SOS Children’s Villages Week 26: Submission of final report to SOS Children’s Villages w w w. R e s u l t a t s t y r i n g . n o

TOR – Terms of reference Vi finner «Den beste» w w w. R e s u l t a t s t y r i n g . n o

- Oppfølging - har målgruppen nytte av evalueringen?

Thank you for your contribution today! 

Evaluering – oppskrift på TOR

Liknende presentasjoner

Presentasjon om: "Evaluering – oppskrift på TOR"— Utskrift av presentasjonen:

Liknende presentasjoner

Om prosjektet

Tilbakemelding

Logg inn

Logg deg på via sosiale nettverk:

Evaluering – oppskrift på TOR

Liknende presentasjoner

Presentasjon om: "Evaluering – oppskrift på TOR"— Utskrift av presentasjonen:

Liknende presentasjoner

Om prosjektet

Tilbakemelding