Presentasjon lastes. Vennligst vent

Presentasjon lastes. Vennligst vent

Evaluering – oppskrift på TOR Anette P. Simonsen 24. oktober 2012 9:00-16:00 Bistandstorget w w w. R e s u l t a t s t y r i.

Liknende presentasjoner


Presentasjon om: "Evaluering – oppskrift på TOR Anette P. Simonsen 24. oktober 2012 9:00-16:00 Bistandstorget w w w. R e s u l t a t s t y r i."— Utskrift av presentasjonen:

1 Evaluering – oppskrift på TOR Anette P. Simonsen 24. oktober :00-16:00 Bistandstorget w w w. R e s u l t a t s t y r i n g. n o

2 INTRO Evaluering som en del av M&E w w w. R e s u l t a t s t y r i n g. n o

3 MonitoringEvaluation WHO?Internal, involving project implementers Can be internal or external to organisation WHEN?Ongoing During project Ongoing Periodic and after project WHY • Check progress, • inform project decisions and remedial action • update project plans • support accountability • Assess progress and worth, • identify lessons and recommendations for longer term planning and organisational learning • provide accountability Link to LFA Focus on activities, outputs and short term outcomes Focuses on outcomes and overall goal. Monitoring and Evaluation Mål- og resultatstyring Anette P. Simonsen

4 Monitoring: • refers to the regular follow-up and assessments during the implementation of a project or programme. • normally provides information on expenditure, activities and outputs and a certain extent outcome • is used for adjusting the course of the project/programme as well as writing progress reports. Evaluation • is the systematic and independent assessment of a project/ program that is performed at certain stages of the project (e.g. mid-term, end of project). • would provide information on output, outcome and impact • is used for adjusting the course or assessing continuation or ending a project. Definitions M&E Mål- og resultatstyring Anette P. Simonsen

5 M&E definition again.. Evaluation approaches often have high methodological standards, a separate budget and often scientific requirements for measurements and analysis. Monitoring on the other hand is a routine task, regularly ‘checking up on how things are’ and recording this information in a way that is useful both w w w. R e s u l t a t s t y r i n g. n o Observing Change N-P-Aid

6 st%C3%B8tte/_attachment/153487?=true&_ts=12fd9e2bf57http://www.norad.no/no/tilskudd/s%C3%B8k- st%C3%B8tte/_attachment/153487?=true&_ts=12fd9e2bf57 Se spesielt side 3: «Beslutning om støtte skal tas etter en helhetsbedømmelse av søknaden. Hovedsakelig er kriteriene delt i to; de som gjelder for vurdering av organisasjonen og de som gjelder vurdering av bistandsvirksomheten. Sentrale punkter i vurderingen av organisasjon er: • Kapasitet og kompetanse til å nå organisasjonens mål • Organisasjonens kostnadseffektivitet, det vil si evnen til å gjennomføre organisasjonens mål med lavest mulige kostnader. • Systemer for resultatoppfølging • Evne til å identifisere og håndtere risiko Norad Regelverk for støtte til sivilsamfunnsaktører w w w. R e s u l t a t s t y r i n g. n o

7 § 16 Evalueringer • Alle virksomheter skal sørge for at det gjennomføres evalueringer for å få informasjon om effektivitet, måloppnåelse og resultater innenfor hele eller deler av virksomhetens ansvarsområde og aktiviteter. Evalueringene skal belyse hensiktsmessighet av eksempelvis eierskap, organisering og virkemidler, herunder tilskuddsordninger. • Frekvens og omfang av evalueringene skal bestemmes ut fra virksomhetens egenart, risiko og vesentlighet. Økonomireglementet sier: w w w. R e s u l t a t s t y r i n g. n o

8 M&E har 3 formål M&E skal bidra til: • Kontroll av virksomheten - Forsikre interessenter om at midlene blir brukt på riktig måte, for avtalte mål og at de ulike samarbeidspartene oppfyller sine forpliktelser • til beslutningsprosesser – gi informasjon som underbygger de valg organisasjonene står overfor, når de må fatte beslutninger om prosjekter/ programmer skal fortsette, avsluttes eller endres • til å fremme organisatorisk læring – ved å gi aktuelle aktører kunnskap og informasjon om prosjekter/ programmer samt den sammenhengen de virker inn i. Ingen begrensning i hva som er mulig læring. w w w. R e s u l t a t s t y r i n g. n o Kruse & Forss (2000)

9 Look back at: • Does the project/programme have a clear goal hierarchy and a set of M&E plans? • Have the plans been monitored? • Are there good and regular monitoring reports? • Are the project staff available /still contactable? • Are the stakeholders involved? • Are the donor(s) involved? Preparing for Evaluation From Local Livelyhoods w w w. R e s u l t a t s t y r i n g. n o

10 Action Plan for Evaluation - Approving the activity and the budget w w w. R e s u l t a t s t y r i n g. n o

11 Action Plan versus TOR The Action plan • Formulates the internal “project” of performing an evaluation • It documents the approval of the decision to perform an evaluation and the budget for it. The TOR • is a work description for the team performing the evaluation • It constitutes only one part of the Action Plan w w w. R e s u l t a t s t y r i n g. n o

12 Action Plan for Evaluation 1.Background and justification – Need for evaluation at this point – Previous relevant evaluations / reviews 2.Who will be involved in evaluation / cooperation with other stakeholders/donors on the evaluation 3.Target group (audience) 4.Objective of evaluation 5.Any special requirements to team composition, method etc.? 6.Outputs/main activities – Planning phase – Implementation – Follow up 7.Risk and uncertainties 8.Organisation and responsibilities 9.Budget w w w. R e s u l t a t s t y r i n g. n o

13 Donor Justification for evaluation w w w. R e s u l t a t s t y r i n g. n o Stake- holder Norwegian NGO Stake- holder Stake- holders Partner Tanzania Project IMPLEMENTATION LEVEL

14 1. Background and Justification Do we really need an evaluation at this point? What is the problem we are trying to solve? E.g. • The M&E system is weak so we need to increase level of information on progress and or achievement of results? • The target group is complaining on quality of services? • We need to secure funding by a more objective voice on achievement of results? w w w. R e s u l t a t s t y r i n g. n o

15 M&E in project Cycle w w w. R e s u l t a t s t y r i n g. n o

16 When? Reporting to donors Program yearDeadline and type of Report Reviews and evaluations 2009May 2010 –Progress report 2010May Progress report 2011May Progress report 2012May Periodic Results Report 2013May 2014 –Final Report w w w. R e s u l t a t s t y r i n g. n o Example from 5 year programme:

17 Why? Status report Norad • 3.1 Describe the reviews/evaluations that have been carried out during the year • In what way(s) did the review/evaluation contribute to learning and improvement of the programmes? • Are external reviews submitted to Norad’s data base for evaluations? (state type of evaluation – external, internal or mix external/internal) w w w. R e s u l t a t s t y r i n g. n o

18 Timing of evaluations Appraisal/ evaluation – before start-up Mid-Term Evaluation – towards middle of programme period End-Term evaluation – at the end of programme Formative evaluations – during programme Summative evaluations – at the end Ex-post evaluations – some time after Impact evaluations – at the end – some time after w w w. R e s u l t a t s t y r i n g. n o

19 OECD evaluation on DAC principles 1998: • Evaluations most useful when they are linked to current and future programmes rather than historical perspectives (although accountability requires the latter type) w w w. R e s u l t a t s t y r i n g. n o

20 2. Who will be involved in evaluation / coop with other stakeholders Discuss evaluation plans and design with the stakeholders (staff, partners, benficiaries, others) and include the primary target group. • Who in the commissioning organisation is familiar with project/programme • Who are the stakeholders? • Which other donors are involved in Project/programme or even thematic area og geographical area. • Others? w w w. R e s u l t a t s t y r i n g. n o

21 3. The target group (audience) Who will read, learn from and use the evaluation report should be one of the a determining questions in developing the terms of reference and designing the evaluation. In other words who is the primary target group that should be able to make use of the report? Should e.g. Norwegian NGO / the partner /stakeholders / members / taxpayers/ donors make use of the evaluation? w w w. R e s u l t a t s t y r i n g. n o

22 OECD evaluation on DAC principles 1998: • Evaluations are performed mostly because donors ask for it. Little attention is paid to the demand, the use and usability of evaluations, and differentiation according to different audiences w w w. R e s u l t a t s t y r i n g. n o

23 4. Objective of the evaluation The purpose of evaluations is normally a combination of a need for control (accountability) to contribute to decision making and for learning • Partner – all 3 • Norwegian all 3 • Donor…. Who cares? w w w. R e s u l t a t s t y r i n g. n o

24 Objective of Evaluation • The implementing partners will have increased knowledge and lessons learned in their continuous work within the thematic areas in their respective countries. • The Norwegian NGO will be better equipped for future disasters of similar order of magnitude, both in the initial and recovery phase. • The private and corporate donors and the Ministry of Foreign Affairs will gain knowledge about the results of their donations. w w w. R e s u l t a t s t y r i n g. n o Who is the target group here?

25 Objective of Base line study The main objectives of the baseline survey are: • To get an initial baseline standing/situation/ context of the current and existing governance and livelihood situation, systems and structures in the impact areas • To establish baseline indicators for the programme that will imply a clear monitoring and evaluation plan (M&E) and platform for impact evaluation (IE) • To source information that is required to describe qualitatively and quantitatively the outcome indicators and output indicators for the Caritas Malawi programme on governance and livelihood. w w w. R e s u l t a t s t y r i n g. n o Who is the target group here?

26 Objective of Evaluation • Have conversations on the internal functions, including financial management, in each organisation involved in the visit in order to strengthen the organisation • Have conversations about the cooperation (communication, interaction) between the organisations in order to improve cooperation • The review should be summarized in a report in which the strengths and weaknesses are described, and proposed changes in procedures and communication within and between organisations are described. w w w. R e s u l t a t s t y r i n g. n o Who is the target group here?

27 Objective of Evaluation • Conducting a mid-term external review in 2011 will establish an overall picture of whether the five-year programme is on track towards 2013 and create a basis for corrective actions and the planning ahead for the final 2 years of implementation. The review will be an input to a mid-term meeting we want to hold in the second part of w w w. R e s u l t a t s t y r i n g. n o Who is the target group here?

28 Objective of Evaluation Formålet med evalueringen er å: • å gi LNU og bevilgende myndigheter økt kunnskap om i hvilken grad de tiltakene som får støtte fra LNU bidrar til å nå støtteordningens prosjektmål (måloppnåelse). • å gi LNU og bevilgende myndigheter økt kunnskap om eventuelle ikke-planlagte effekter (positive eller negative) av de tiltakene som får tilskudd. • å gi LNU et grunnlag for å videreutvikle sin forvaltning av støtteordningen. Formålet med evalueringen er avgrenset på bakgrunn av de økonomiske rammene for evalueringen. w w w. R e s u l t a t s t y r i n g. n o Who is the target group here?

29 Organisational review: w w w. R e s u l t a t s t y r i n g. n o PREBEN LINDØE for BN 2006

30 5. Any special requirements to Team, method etc. Team: Internal / external Special expertise necessary Method: Level of data collection Special tools to be used NB! The action plan should not be too detailed at this point. Difficult to change. w w w. R e s u l t a t s t y r i n g. n o

31 6. Outputs/ main activities The Action plan is split into • Planning phase • Implementation • Follow up w w w. R e s u l t a t s t y r i n g. n o

32 Planning phase Outputs and activitiesTimeframeResponsible 1.Plan of Action AKV,KH;DH 1.Approval Plan of Action AKV 1.TOR AKV,KH,DH 1.Select internal team members SLM 1.Go out for tender TM/DH Implementation phase 1.Consultant chosen AKV,KH 1.Prepare for evaluation 1.Evaluation (by team) Report draft Completion phase 1.Quality assurance of the reports are done by relevant stakeholders Final report Example of Outputs and activities in Action Plan Did you forget about Learning?

33 7. Risk and uncertainties Any project meets uncertainties: e.g w w w. R e s u l t a t s t y r i n g. n o Risk factors Mitigating activities An evaluation in four parts over several months lead to tiredness and lack of interest of the process among stakeholders and organization Ensure internal communication about process and sub results along the way and continuous involvement of stakeholders. Reputational risk in case of exposure of mistakes made by NorCross during the programme Communication strategy is developed. Follow-up of recommendations and plan for learning is developed. NorCross initiatives to expose and learn from own mistakes decrease reputational risks (precautionary action). Low participation by the appointed resource group and ToR groups Involvement in the plan for the process by the members to ensure ownership. Appoint one responsible in each ToR group. Announce workshop in due time. Limited time frame and occasions for participation Context in host countries hinders and/or complicates evaluation implementation Research ahead and involvement of NSs Difficulties in finding qualified consultants for evaluation implementation (technical knowledge in various areas is necessary) Tender process through Mercell. Separate invitations (through Mercell) to tender for companies recognized by NorCross after research

34 9. Organisation and responsibillities Function Title Tasks Owner of evaluation Director International Division Responsible Anne Kirsti Vartdal Kjersti Haraldseide Dagne Hordvei Carry out activities in accordance with this PoA: -Initiate, choose consultant and team, prepare, assist evaluation team, comment on report, follow up findings. Steering committee Management in International division Approve PoA and budget Approve Report Resource groupAdviser 1 Adviser 2 Adviser 3 Adviser 4 Rep delegate section w w w. R e s u l t a t s t y r i n g. n o

35 Læring – referansegruppe evalueringsavdelingen norad Det er evalueringsavdelingen sin målsetning at evalueringsprosesser skal være deltakende og inklusive og fremme størst mulig læring. Det etableres derfor ofte en referansegruppe bestående av interessenter, berørte parter, ev. faglige ressurspersoner, fagavdeling og evalueringsavdelingen. Disse kommenterer arbeidet underveis. I den grad det er hensiktsmessig ønsker også evalueringsavdelingen å arrangere seminarer underveis og i etterkant. w w w. R e s u l t a t s t y r i n g. n o

36 OECD evaluation on DAC principles • There are reason for concern regarding the way in which lessons and experience, gained during the evaluation process, are transmitted to current operation managers. w w w. R e s u l t a t s t y r i n g. n o

37 10. Budget example 1 ItemsActivity numberBudget cost (NOK) Consultant Haiti 66% Sri Lanka 34% Travels Misc Total w w w. R e s u l t a t s t y r i n g. n o

38 10. Budget example 2 ItemsActivity numberBudget cost (NOK) PREPARATION -Stakeholder meeting -M&E system update IMPLEMENTATION -Consultant -Travels Haiti 66% Sri Lanka 34% FOLLOW UP -Dissemination -internalisation -Misc Total w w w. R e s u l t a t s t y r i n g. n o

39 TOR – Terms of reference w w w. R e s u l t a t s t y r i n g. n o

40 Action Plan versus TOR The Action plan • Formulates the internal “project” of performing an evaluation • It documents the approval of the decision to perform an evaluation and the budget for it. The TOR • is a work description for the team performing the evaluation • It constitutes only one part of the Action Plan w w w. R e s u l t a t s t y r i n g. n o

41 Tor Norad 1 Background • Short description of the programme that will be reviewed, based on the Agreement, Programme Document(s) and appraisal • Why the review is initiated • Team composition and leadership. (e.g. Embassy/Norad team, joint donor/Partner, external team or combination of these) 2 Purpose, context and intended use •Description of the main purpose, context and intended use (stakeholders) 3 Scope of work Type of assessments may include: (for definition of the review criteria, see Annex III) • Efficiency •Effectiveness • Impact (if the programme has been operating for some years) •Relevance • Sustainability • Risk management •Particular concerns to be investigated • Audit •Anti-corruption measures 4 Implementation of the review • Sources of information and methodology to be employed • Division of responsibility between the consultant/team, the Embassy, other donors and the Partner(s) •Timetable for preparation, field work and finalisation of report •Budget 5 Reporting • Description of required report format • The need for an introduction summary with main conclusion on lessons learned and recommendation(s) • Report in electronic form and/or paper, language w w w. R e s u l t a t s t y r i n g. n o

42 Evalueringssprinsipper (Eval Norad web) Evalueringsprinsipper • Objektive, etterprøvbare og gjennomsiktige: Vurderinger skal være basert på faktiske utsagn; utsagn skal være basert på troverdige (reliable) data eller observasjoner. Relevante interessenter i Norge eller mottakerlandet skal konsulteres i forbindelse med evalueringers tilrettelegging og gjennomføring, inklusive i utarbeiding av oppgavebeskrivelse og diskusjon av rapportutkast. • Upartiske: Evalueringer skal gi en avbalansert fremstilling av styrker og svakheter. I den grad interessenter har forskjellige synspunkter skal disse tas med i evalueringen. • Uavhengige: Medlemmer av evalueringsteamet må ikke ha vært personlig engasjert i de aktiviteter som skal evalueres. Virksomheter som gjennomfører evalueringer må ikke ha vært involvert i forberedelse eller gjennomføring av aktivitetene. w w w. R e s u l t a t s t y r i n g. n o

43 Description of Project /issue to be evaluated w w w. R e s u l t a t s t y r i n g. n o

44 Description of project to be evaluated Basically a summary of project document • Especially important is the goal hierarchy and any changes in that along the way • M&E System must be described and commented on • Previous or other relevant evaluations should be mentioned here • Budgets must be included too. w w w. R e s u l t a t s t y r i n g. n o

45 Scope of evaluation – The boundaries of the evaluation (e.g time period, area) – Evaluation criteria and evaluation questions w w w. R e s u l t a t s t y r i n g. n o

46 Scope of evaluation The formulation of “Evaluation questions” is one method that will end up with a precise definition of what should be evaluated. Evaluation questions represent “what one wants to know through evaluation.” • When formulation the evaluation questions it is useful to consider which of the DAC Principles for Evaluation of Development Assistance that should be covered w w w. R e s u l t a t s t y r i n g. n o

47 OUTPUT 1 OUTPUT 3 OUTPUT 2 One goal -LFA spread out Activities: Activities: Activities: Project Goal (expected Outcome) Development Goal (expected Impact) OUTPUT 4 Activities: Mål- og resultatstyring Anette P. Simonsen Inputs

48 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) • efficiency, • effectiveness, • impact • relevance, • sustainability. w w w. R e s u l t a t s t y r i n g. n o

49 Evalueringskriterier(web Norads eval avd) • Den enkelte evaluering bør normalt belyse følgende: • Relevans: Er aktiviteten relevant i forhold til norsk bistandspolitikks målsetninger og strategier? Er aktiviteten relevant i forhold til mottagerlandets behov og prioriteringer? Er den relevant i forhold til det utviklingsproblem den skal se på? • Måloppnåelse (effectiveness): Er de primære målsetninger for aktiviteten nådd? Er de planlagte resultater nådd? • Bærekraft (sustainability): Hva er langtidsvirkningen av aktiviteten? Vil aktiviteten kunne videreføres også etter at bistandsfinansieringen er avsluttet? Er det lokalt eierskap? • Produktivitet (efficiency): Er investeringene og driftsomkostningene berettighet? Kunne de samme resultater ha blitt oppnådd med færre midler? • Virkning (impact): Positive og negative primære og sekundære langtidseffekter produsert av aktivitetene, direkte eller indirekte, tilsiktede eller utilsiktede. w w w. R e s u l t a t s t y r i n g. n o

50 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Efficiency Effectiveness Impact Relevance Sustainability Evaluation criteria: w w w. R e s u l t a t s t y r i n g. n o

51 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Efficiency Efficiency measures the outputs -- qualitative and quantitative -- in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a programme or a project, it is useful to consider the following questions: •Were activities cost-efficient? •Were objectives achieved on time? •Was the programme or project implemented in the most efficient way compared to alternatives? Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a programme or a project, it is useful to consider the following questions: •What has happened as a result of the programme or project? •What real difference has the activity made to the beneficiaries? •How many people have been affected? Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions: •To what extent did the benefits of a programme or project continue after donor funding ceased? •What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? w w w. R e s u l t a t s t y r i n g. n o

52 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Effectiveness A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a programme or a project, it is useful to consider the following questions: •To what extent were the objectives achieved / are likely to be achieved? •What were the major factors influencing the achievement or non-achievement of the objectives? w w w. R e s u l t a t s t y r i n g. n o

53 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. When evaluating the impact of a programme or a project, it is useful to consider the following questions: •What has happened as a result of the programme or project? •What real difference has the activity made to the beneficiaries? •How many people have been affected? w w w. R e s u l t a t s t y r i n g. n o

54 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Relevance The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a programme or a project, it is useful to consider the following questions: •To what extent are the objectives of the programme still valid? •Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives? •Are the activities and outputs of the programme consistent with the intended impacts and effects? w w w. R e s u l t a t s t y r i n g. n o

55 DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria) Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions: •To what extent did the benefits of a programme or project continue after donor funding ceased? •What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project? w w w. R e s u l t a t s t y r i n g. n o

56 Evalueringsavdelingen Norad Målene for evalueringsavdelingen er å: • vurdere måloppnåelse og resultater i forhold til vedtatte planer • vurdere forutsette og uforutsette virkninger av bistandstiltak • vurdere om ressursbruken står i rimelig forhold til oppnådde resultater • systematisere erfaringer og kunnskap for å kvalitetssikre og forbedre fremtidig bistandstiltak gjennom gode læringsprosesser • kartlegge kunnskapshull/ vurdere de underliggende antakelsene for ulike deler av utviklingssamarbeidet • gi informasjon til dem som bevilger og utformer ny politikk, og allmennheten w w w. R e s u l t a t s t y r i n g. n o

57 Methodology w w w. R e s u l t a t s t y r i n g. n o

58 Methods for collection of info/data Quantitativ • Quantitative Surveys • Rating Qualitative • Desk reviews • Case studies • Qualitative surveys • Interviews w w w. R e s u l t a t s t y r i n g. n o

59 Ex: Methodology LNU ønsker at følgende metoder skal benyttes: • Intervju med utvalgte brukere av støtteordningen og deres partnere i perioden 2004 – 2008 for å vurdere effekten av de tiltakene som har fått støtte i forhold til støtteordningens mål. • Intervju eller spørreskjema for alle organisasjoner i Norge som har brukt ordningen i perioden 2004 – 2008 for å vurdere LNUs forvaltning av støtteordningen. • Intervju med saksbehandlere • Intervju med medlemmer av fordelingsutvalget for støtteordningen. • Dokumentanalyse av styringsdokumentene fortøtteordningen, herunder retningslinjene. w w w. R e s u l t a t s t y r i n g. n o

60 Collecting true qualitative data • By nature, much Qualitative data are more sensitive and personal than Quantitative data. • While surveys can be administrated to more people in a shorter time (breadth), interviews probe for more information from fewer individuals (depth). w w w. R e s u l t a t s t y r i n g. n o From: Real life evaluation

61 Collecting true qualitative data • Interviewees may feel uncomfortable for a variety of reasons; including cultural linguistic, age related, gender related and political reasons. • Interviewees who are less than fully forthcoming obstruct data collection. w w w. R e s u l t a t s t y r i n g. n o From: Real world Evaluation

62 A bit of both please  Quantitative – breadth in insight quantitative surveys can be administrated to more people in a shorter time  Qualitative – depth in insight qualitative interviews probe for more information from fewer individuals (depth). w w w. R e s u l t a t s t y r i n g. n o

63 NB! Please Note! NB! Please note: • there is a continuum between quantitative and qualitative data definition and collection. w w w. R e s u l t a t s t y r i n g. n o QUALITATIVE QUANTITATIVE

64 Qualitative / Quantitative Indicators / data: – Qualitiative – Quantitative Collection methods: – Qualiatiative – Quantitative w w w. R e s u l t a t s t y r i n g. n o

65 Qualitative data collection • Interviews are often characterized by: A.Structured (all asked the same questions). B.Semi-structured (questions vary). C.Unstructured (no protocol). • Qualitative interviewing is typically semi- structured (B) which is useful as it offers focus and flexibility but also most demanding (as one must record data while adjusting wording & order of questions, preserving focus, and follow up on conversational leads). w w w. R e s u l t a t s t y r i n g. n o From: Real world Evaluation

66 Methodology – Evaluation Norad East Africa NGO Eval. April 2011 Methodology • The evaluation team shall develop an appropriate methodology that can respond to the information needs arising from the list of evaluation questions. The evaluation team may make use of various empirical methods, e.g. questionnaire surveys, interviews, focus group discussions and participatory rural appraisal. The presentation of the methodology should indicate any significant advantages and limitations of the chosen approach. w w w. R e s u l t a t s t y r i n g. n o

67 Ex of Methodology The methods employed in this evaluation will include the following: review of available documents and literature, individual interviews and focus groups discussions. It will be important to allow intended beneficiaries to tell their stories without necessarily sticking to a structured questionnaire as this will allow interviewees to release their emotions alongside the needed information. The purpose of employing several methods to gather data (triangulation) is to see whether the different sets of data material confirm or may complement each other, and hence provide a deeper understanding of the programme, its activities, effects and challenges. w w w. R e s u l t a t s t y r i n g. n o

68 Sikre bruk av M&E Systemet I evalueringen w w w. R e s u l t a t s t y r i n g. n o

69 Ex: Monitoring health by indicators Outcome 2: Risk Groups adopt good practices of prevention and treatment of HIV/AIDS # of people referred for councelling and testing # of People reached by HIV/AIDS projects in he frame of the programme (CORE indicator) # of test taken at local clinic pr year. Output 1: NS engage youth in youth peer education poagramme # of youth certified as peer educators # of youth who have received training in peer education Output 2: NS provide education for risk groups on sexual health (STD and HIV/AIDS) # of people educated on sexual health % of partcipants satisfied with the courses. Output 3: NS have adequatly trained staff and volunteers to adress HIV/AIDS according to org. standards and in gender sensitive manner. # of volunteers and staff participating in HIV/AIDS prevention activities in the frame of this programme (CORE Indicator) Output 4: NS promotes HIV/AIDS voluntary testing and councelling # of information campaigns

70 Eksempel på M&E plan Mål- og resultatstyring Anette P. Simonsen Årlig rapport m/ regnskap (Partner) Indikator survey Partner Administrativ Controller visits (Donor) Resultat gjennomgang / (Joint) 2011Baseline Jan Årsrapport for 2012 i mars 2013 Februar 2013 Oktober 2012 Controller visit August 2012 Workshop om baseline og indikatorer 2013 Årsrapport for 2013 i mars 2014 Februar 2014 Oktober 2013 Gjennomgang av organisasjonens administrative systemer 2014 Sluttrapport for i mars 2015 Februar 2015 Slutten av 2014 gjennomgang av prosjekt med fokus på måloppnåelse

71 Team composition w w w. R e s u l t a t s t y r i n g. n o

72 Team composition Control versus learning: Internal team versus external to project Origin/language Norwegian, international/local base Qualifications w w w. R e s u l t a t s t y r i n g. n o

73 Internal versus External Evaluator Mål- og resultatstyring Anette P. Simonsen Advantages INTERNALAdvantages EXTERNAL Knows the situation, history, problems, processes etc Not personally involved -easy to be objective Understands and can interpret personal behaviour and attitudes Free from local bias Is known to stakeholders so pose no threat anxiety or disruption Can bring fresh perspective and insights Can follow up on recommendationMay have broader experience Is often less expensive and Quick to hire ( no negotiations etc) Can serve as arbitrator or facilitator between parties May provide mor oportunity to build local evaluation capacity Can bring local stakeholders into contact with additional sources ADVANTAGES

74 Internal versus External Evaluator Mål- og resultatstyring Anette P. Simonsen Disadavantages INTERNALDisadavantages EXTERNAL Hard to be objectiveMay not know local situation May avoid looking for facts or forming nagative conclusions tha reflect badly on community individuals Ignorant of constraints affecting feasibility of recommendations Tends to accept the assumptions of the stakeholders and community May be perceived as adversary and arouse anxiety May be too busy to participate fullyMay be expensive May be part of the local community and may be constrained by conflicts of interest Requires more time for contract negotiations, orientation and monitoring May not be trained in evaluation methods and technical expertice May be unfamiliar with local political and economic environment DISADVANTAGES

75 The TOR The TOR: • In planning phase the TOR is part of the tender documents and e.g. thus describes the qualifications that is saught for the evaluation. • After the team has been selected and contracted, the TOR is updated with the names of the team members and enclosed the Evaluation as an annex w w w. R e s u l t a t s t y r i n g. n o

76 Ex of team selected team composition Team composition / representation: • Team leader, external consultant • YRCS programs’ coordinator • Representative of the YRCS OD department • YRCS health Coordinator • Resource persons: • The YRCS CBHD programme manager • The YRCS CBHD programme officer • President / Secretary general of Hajjah and Hodeidah branches and sub-branches (4 people) • NRC OD advisor • SRC health advisor • DRC OD advisor • IFRC team member w w w. R e s u l t a t s t y r i n g. n o

77 Reporting w w w. R e s u l t a t s t y r i n g. n o

78 Reporting deliverables &formats • An inception report detailing the strategy to be utilised, defining clear roles and responsibilities of the consultant, work plan, budget and a list of stakeholders to be consulted. This is expected within the first two days of commencement of contract; • A draft baseline report for the governance and livelihood programme to be presented to Caritas Malawi. This is to be presented on the 8 th day of the commencement of the consultancy. • A draft report discussed between Caritas Malawi and the Consultant on the 9 th day of the consultancy • A final baseline report on governance and livelihood after incorporating comments from Caritas Malawi and Caritas Norway. This should be submitted in two hard copies and one electronic copy on the 12 th day of the commencement of the consultancy. w w w. R e s u l t a t s t y r i n g. n o

79 Learning w w w. R e s u l t a t s t y r i n g. n o

80 Dissemination - variations Transparent and complete dissemination. Evaluation results should be placed in public domain and widely disseminated, ensuring that information is readily accessible to all stakeholders. An initial dissemination list (Practice 5.10) should be employed to ensure the evaluation report or summaryreaches its intended audience (per the TOR Appropriate dissemination. The dissemination of the evaluation report may take a variety of forms that are appropriate to the specific audience. This can include posting reports or excerpts/summaries on a community notice board or on the internet, and presentations at planning meetings, community meetings, and industry conferences. Such considerations are especially important when sharing reports with communities. w w w. R e s u l t a t s t y r i n g. n o

81 Dissemination - variations Differentiate between Internal and external dissemination. This may be done for multiple reasons: 1)with sensitive issues that should remain internal, 2)to protect the identity and avoid or reduce any harm to evaluation subjects, and 3)to improve the comprehensibility of the report in a summary or simplified form for external audiences. Disseminate through Evaluation Database. All evaluation reports should be submitted for record in international Database on evaluation w w w. R e s u l t a t s t y r i n g. n o

82 Bruk av M&E info • For at resultatmålingen skal ha noen hensikt, må resultatene drøftes, vurderes og brukes av ledelsen til å ta beslutninger om tiltak som skal forbedre virksomheten og gi bedre måloppnåelse. w w w. R e s u l t a t s t y r i n g. n o Mål- og resultatstyring SSØ s 8

83 Learning 1.Quality of learning depends on enough slow time. (opportunities) w w w. R e s u l t a t s t y r i n g. n o Fritt etter Knut Harald Ulland Utviklingsfondet

84 Learning – a crime? ‘ it is almost as if my organisation considers learning as a crime rather than a behaviour we’re trying to encourage’. A crime (or learning) needs MMO to incur: 1.Motive 2.Means 3.Opportunity w w w. R e s u l t a t s t y r i n g. n o Bruce Britton (2005)

85 MMO for Learning Motive: • Develop systems for acknowledging and rewarding learning Means: • Map out where expertise lies in organisation. • Develop team working • E.g introduce methods such as mentoring, coaching, action learning and communities of practice Opportunity: • Create space for learning • Build time and resources for reflection and learning intor project/programme proposals • Involve staff/partners alongside external consultants in review and evaluation teams. w w w. R e s u l t a t s t y r i n g. n o B. Britton ( 2005)

86 Time Schedule w w w. R e s u l t a t s t y r i n g. n o

87 Ex Time schedual for external mid tern review Task/ActivityTimeline 1.Procure TeamBy week 12: Contract signed by TeamBy week 14: Preparation work by Teamleader in Oslo (finalization of evaluation plan, document collection, consultations with resource persons, and initial document review) Weeks 15: Desk study/document review and fieldwork preparations Week 17: Field interviews and visits Uganda - week 21: Zambia – week 22: Mozambique – week 23: Data analysis and drafting the reportWeek 24: Submission of draft to SOS Children’s Villages Draft assessed by SOS Children’s VillagesWeek 25: Incorporation of comments and inputs from SOS Children’s Villages Week 26: Submission of final report to SOS Children’s VillagesWeek 26: w w w. R e s u l t a t s t y r i n g. n o

88 TOR – Terms of reference Vi finner «Den beste» w w w. R e s u l t a t s t y r i n g. n o

89 - Oppfølging - har målgruppen nytte av evalueringen? w w w. R e s u l t a t s t y r i n g. n o

90 Thank you for your contribution today!  w w w. R e s u l t a t s t y r i n g. n o


Laste ned ppt "Evaluering – oppskrift på TOR Anette P. Simonsen 24. oktober 2012 9:00-16:00 Bistandstorget w w w. R e s u l t a t s t y r i."

Liknende presentasjoner


Annonser fra Google