The Green Book: Appraisal & Evaluation in Central Government
HM Treasury
.
 

 

1. Introduction & background
2. Overview of appraisal & evaluation
3. Justifying action
4. Setting objectives
5. Appraising the options
6. Developing & implementing the solution
7. Evaluation

Annex 1: Government intervention
Annex 2: Valuing non-market impacts
[Annex 3: Land & buildings]
Annex 4: Risk & uncertainty
Annex 5: Distributional impacts
Annex 6: Discount rate

 

PREFACE

The Treasury has, for many years, provided guidance to other public sector bodies on how proposals should be appraised, before significant funds are committed – and how past and present activities should be evaluated. Decisions taken at the appraisal stage affect the whole lifecycle of new policies, programmes and projects. Similarly, the proper evaluation of previous initiatives is essential in avoiding past mistakes and to enable us to learn from experience. The Green Book therefore constitutes binding guidance for departments and executive agencies.

 

1 INTRODUCTION AND BACKGROUND

INTRODUCTION

1.1 All new policies, programmes and projects, whether revenue, capital or regulatory, should be subject to comprehensive but proportionate assessment, wherever it is practicable, so as best to promote the public interest.

1.2 The guidance emphasises the need to take account of the wider social costs and benefits of proposals, and the need to ensure the proper use of public resources.

1.3 This is achieved through:

  • Identifying other possible approaches which may achieve similar results;
  • Wherever feasible, attributing monetary values to all impacts of any proposed policy, project and programme; and
  • Performing an assessment of the costs and benefits for relevant options.

The Green Book describes how the economic, financial, social and environmental assessments of a policy, programme or project should be combined.

BOX 1: ACTIVITIES COVERED BY THE GREEN BOOK

  • Policy and programme development: Decisions on ... the extent of regulation.
  • Specification of regulations: Decisions, for example, on standards for health and safety, environment quality, sustainability, or to balance the costs and benefits of regulatory standards and how they can be implemented.

1.8 This guidance applies:

  • At the start ... to any analysis used to support a government decision to adopt a new policy, or to initiate, renew, expand or re-orientate programmes or projects, which would result in measurable benefits and/or costs to the public. This is the appraisal part of the process.
  • And at the finish ... to retrospective analysis of a policy, programme or project at its completion, conclusion or revision. This is the evaluation part of the process.

 

2 OVERVIEW OF APPRAISAL AND EVALUATION

THE APPRAISAL AND EVALUATION CYCLE

2.2 Appraisal and evaluation often form stages of a broad policy cycle that some departments and agencies formalise in the acronym ROAMEF (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback).

THE ROLE OF APPRAISAL

2.3 Appraisals should provide an assessment of whether a proposal is worthwhile, and clearly communicate conclusions and recommendations. The essential technique is option appraisal, whereby government intervention is validated, objectives are set, and options are created and reviewed, by analysing their costs and benefits. Within this framework, cost-benefit analysis is recommended, as contrasted with cost-effectiveness analysis below, with supplementary techniques to be used for weighing up those costs and benefits that remain unvalued.

COST-BENEFIT ANALYSIS

Analysis which quantifies in monetary terms as many of the costs and benefits of a proposal as feasible, including items for which the market does not provide a satisfactory measure of economic value.

COST-EFFECTIVENESS ANALYSIS

Analysis that compares the costs of alternative ways of producing the same or similar outputs.

PROCESS FOR APPRAISAL AND EVALUATION

2.4 Appraisals are often iterated a number of times before their proposals are implemented in full. Therefore the stages set out below may be repeated, and they may not always be followed sequentially. In particular, as options are developed, it will usually be important to review more than once the impact of risks, uncertainties and inherent biases. This helps to avoid spurious accuracy, and to provide a reasonable understanding of whether, in the light of changing circumstances, the proposal is likely to remain good value for money.

Chapter 3 – Justifying Action

2.6 The first step is to carry out an overview to ensure that two pre-requisites are met: firstly, that there is a clearly identified need; and secondly, that any proposed intervention is likely to be worth the cost. This overview must include an analysis of the negative consequences of intervention, as well as the results of not intervening, both of which must be outweighed to justify action.

Chapter 4 – Setting Objectives

2.7 The second step is to set out clearly the desired outcomes and objectives of an intervention in order to identify the full range of options that may be available to deliver them. Targets should be set to help progress towards meeting objectives.

Chapter 5 – Option Appraisal

2.8 The third step is to carry out an option appraisal. This is often the most significant part of the analysis. Initially a wide range of options should be created and reviewed. This helps to set the parameters of an appropriate solution. A shortlist may then be created to keep the process manageable, by applying the techniques summarised below to high level estimates or summary data. The ‘do minimum’ option should always be carried forward in the shortlist, to act as a check against more interventionist action.

2.9 Each option is then appraised by establishing a Base Case [1].This is the best estimate of its costs and benefits. These estimates can then be adjusted by considering different scenarios, or the option’s sensitivity to changes can be modelled by changing key variables. More fully, the appraisal may develop as follows:

  • Identify and value the costs of each option.
  • Identify and value the benefits of each option.
  • If required, adjust the valued costs and benefits for:
  • Distributional impacts (the effects of proposals on different sections of society);
  • Relative price movements.
  • Adjust for the timing of the incidence of costs and benefits by discounting them, to obtain their present values.
  • If necessary, adjust for material differences in tax between options.
  • Adjust for risk and optimism to provide the Base Case, and consider the impacts of changes in key variables and of different future scenarios on the Base Case.
  • Consider unvalued impacts (both costs and benefits), using weighting and scoring techniques if appropriate.

Chapter 6 – Developing and implementing a solution

2.10 Following option appraisal, decision criteria and judgement should be used to select the best option or options, which should then be refined into a solution. Consultation is important at this stage, regardless of whether it has taken place earlier.

2.11 Issues that may have a material impact on the successful implementation of proposals must be considered during the appraisal stage, before significant funds are committed. This is to ensure that the outcome envisaged in the appraisal is close to what eventually happens.

Chapter 7 – Evaluation

2.12 Evaluation is similar in technique to appraisal, although it obviously uses historic (actual or estimated) rather than forecast data, and takes place after the event. Its main purpose is to ensure that lessons are widely learned, communicated and applied when assessing new proposals.

[1] The term ‘Base Case’ is sometimes used to refer to the ‘do minimum’ option, but it is not used in this way in the Green Book.

PRESENTING THE RESULTS

2.13 The ultimate outcome of any appraisal is a decision whether or not to proceed with a proposal or a particular option. As these decisions will often have far reaching consequences, the presentation of the conclusions and recommendations to decision makers and key stakeholders can be as important as the analysis itself. In all cases, transparency is vital. Presentations and reports should be clear, logical, well founded, and geared towards helping the decision at hand.

2.14 Reports should provide sufficient evidence to support their conclusions and recommendations. They should provide an easy audit trail for the reader to check calculations, supporting evidence and assumptions. Major costs and benefits should be described, and the values attached to each clearly shown rather than netted off in the presentation of the analysis. This should help to ensure that decision makers understand the assumptions underlying the conclusions of the analysis, and the recommendations put forward. Appraisal reports should contain sufficient information to support the conduct of any later evaluation.

MANAGING APPRAISALS AND EVALUATIONS

2.17 Conducting an assessment can be resource-intensive. Appraisals and evaluations should therefore be carried out collaboratively wherever possible between stakeholders, but lead responsibilities need to be well defined, and accountability for accuracy and thoroughness clearly understood.

FRAMEWORKS

2.20 The frameworks below are particularly relevant to appraisals and evaluations:

  • The OGC Gateway Review (mainly for programmes and projects);
  • The Regulatory Impact Assessment (mainly for policies involving regulatory impacts); and
  • The Centre for Management and Policy Studies (CMPS) Policy Hub.

Office of Government Commerce Gateway Process

2.21 Gateway is a review process for civil procurement projects conducted by the Office of Government Commerce. It examines policies and projects at critical stages in their lifecycle to provide assurance that they can progress successfully to the next stage. Compliance with the Green Book is incorporated into the first and second gateways. Detailed information is available from the OGC website.

Regulatory Impact Assessment

2.22 A regulatory impact assessment (RIA) is a policy tool that assesses the impact, in terms of costs, benefits and risks of any proposed regulation that could affect businesses, charities or the voluntary sector. It is Government policy that all government departments and agencies where they exercise statutory powers and make rules with general effect on others must produce an RIA. They should also produce an RIA for proposed European legislation that will have an effect on businesses, the public sector, charities or the voluntary sector in the UK.

2.23 Although the trigger for producing an RIA is that the proposal could affect businesses, charities or the voluntary sector, the RIA itself should cover the full range of economic, social and environmental effects, in line with the Green Book methodology.

The CMPS Policy Hub

2.24 The CMPS Policy Hub aims to improve policy making and delivery, by providing:

  • Tailored access to resources and activities from the UK and abroad that help formulate, develop and evaluate policy more efficiently and effectively;
  • Innovative examples of improved policy making and delivery;
  • Tools to help break down organisational and geographical barriers, and improve collaborative working within and beyond government; and,
  • A platform for promoting the highest standards of research and evaluation.

ISSUES RELEVANT TO APPRAISAL AND EVALUATION

2.25 There is a wide range of generic issues that may need to be considered as part of any assessment. The following list should be checked for relevance to options under appraisal, and used for later evaluations:

  • Strategic impact – new proposals can be said to have strategic impacts on organisations if they significantly affect the whole or major part of an organisation over the medium to long term. Proposals should therefore be considered in terms of their potential scale of impact, and how they fit in with the strategy of the organisation(s) they affect.
  • Economic rationale – proposals need to be underpinned by sound economic analysis, which should be provided by a cost benefit analysis in an option appraisal. See Chapter 5 in particular.
  • Financial arrangements and affordability – proposals need to be affordable, and an affordable financial plan needs to be developed. See Chapter 6.
  • Achievability – all proposals should be assessed for their achievability, and recognised programme and project management arrangements set up as necessary. See Chapter 6.
  • Regulatory impact – as discussed previously, the impacts of new proposals on businesses, voluntary sector and charities should be assessed. See Chapter 2.
  • Legislation – consideration should be given to legislation specific to the case in hand, as well as statutes that affect many proposals, such as the Human Rights Act, or the Data Protection and Freedom of Information Acts.
  • Information management and control – The information requirements of proposals, including the data needed for later evaluation, and the supporting IT that may be required. Further guidance is available from the OGC [10].
  • Environmental impacts – The effects on the environment should be considered, including air and water quality, land use, noise pollution, and waste production, recycling and disposal. Further guidance is available from ODPM, Defra and DfT.
  • Equality – Impacts on various groups in society should be considered as part of an appraisal. Chapter 5 describes how distributional impacts should be brought into the appraisal process.
  • Health – the impacts of proposals on health should be considered, and evaluation made of the impact on health of poverty, deprivation and unemployment, as well as poor housing or workplace conditions. The Department of Health can provide further advice12, or can be accessed via the policy hub [13].
  • Health and safety – the health and safety of people at work or arising from work activity may need to be safeguarded. Obviously this is of particular concern in construction. The Health and Safety Commission can provide further advice [14].
  • Consumer focus – Assessments may need to involve consideration of the cost and quality of goods and services, as well as access to, choice of, and information about them [15].
  • European Union – It will often be important to take account of proposals and activities in other European Union countries, as well as specific legislation and regulations. State aid rules are particularly important to consider, as these prescribe the extent to which government can intervene [17].

 

3 JUSTIFYING ACTION

REASONS FOR GOVERNMENT INTERVENTION

3.2 This underlying rationale is usually founded either in market failure or where there are clear government distributional objectives that need to be met. Market failure refers to where the market has not and cannot of itself be expected to deliver an efficient outcome; the intervention that is contemplated will seek to redress this. Distributional objectives are self-explanatory and are based on equity considerations.

3.3 Government intervention can incur costs and create economic distortions. These must be taken into account to determine whether intervention is warranted. For example, a regulation may be successful in addressing a particular market failure, but might also involve other costs that mean that overall it is not worthwhile.

 

4 SETTING OBJECTIVES

OBJECTIVES, OUTCOMES, OUTPUTS, AND TARGETS

4.2 Objectives should be stated so that it is clear what proposals are intended to achieve. Objectives may be expressed in general terms so that the range of options to meet them can be considered. The objectives of individual proposals should be consistent with statements of government policy, departmental or agency objectives, departmental Public Service Agreements (PSAs), and wider macro-economic objectives.

4.3 There is usually a hierarchy of outcomes, outputs, and targets that should be clearly set out in an appraisal. Outcomes are the eventual benefits to society that proposals are intended to achieve. Often, objectives will be expressed in terms of the outcomes that are desired. But outcomes sometimes cannot be directly measured, in which case it will often be appropriate to specify outputs, as intermediate steps along the way. Outputs are the results of activities that can be clearly stated or measured and which relate in some way to the outcomes desired.

4.4 Targets can be used to help progress in terms of producing outputs, delivering outcomes, and meeting objectives. Targets should be SMART;

  • Specific,
  • Measurable,
  • Achievable,
  • Relevant, and,
  • Time-bound.

BOX 5: SETTING OBJECTIVES AND TARGETS

The following questions may help to set suitable objectives and targets:

  • What are we trying to achieve? What are our objectives? What would constitute a successful outcome or set of outcomes?
  • Have similar objectives been set in other contexts that could be adapted?
  • Are our objectives consistent with strategic aims and objectives as set out, for example, in the department’s Public Service Agreements (PSA’s)?
  • Are our objectives defined to reflect outcomes (e.g., improved health, crime reduction or enhanced sustainable economic growth,) rather than the outputs (e.g. operations, prosecutions or job placements), which will be the focus of particular projects?
  • How might our objectives and outcomes be measured?
  • Are our objectives defined in such a way that progress toward meeting them can be monitored?
  • What factors are critical to success?
  • What SMART targets can we then set? What targets do we need to meet?

 

5 APPRAISING THE OPTIONS

CREATING OPTIONS

5.3 This step involves preparing a list of the range of actions which government could possibly take to achieve the identified objectives. The list should include an option where government takes the minimum amount of action necessary (the ‘do minimum option’), so that the reasons for more interventionist actions can be judged.

5.4 The range of options depends on the nature of the objectives. For a major programme, a wide range should be considered before short-listing for detailed appraisal. Both new and current policies, programmes and projects should be included as options. At the early stages, it is usually important to consult widely, either formally or informally, as this is often the best way of creating an appropriate set of options.

BOX 8: CREATING OPTIONS

Establishing a range of options can be challenging. The following actions are suggested:

  • Consider the full range of issues likely to affect the objective.
  • Identify the full range of policy instruments or projects that may be used to meet the objectives. This may span different sorts or scales of intervention; regulatory (or deregulatory) solutions may be compared with self-regulation, spending or tax options.
  • Develop and consider radical options. These options may not become part of the formal appraisal but can be helpful to test the parameters of feasible solutions. Well-run brainstorming sessions can help to generate such a range of ideas.

VALUING THE COSTS AND BENEFITS OF OPTIONS

Introduction

5.8 The relevant costs and benefits to government and society of all options should be valued, and the net benefits or costs calculated. The decision maker can then compare the results between options to help select the best.

5.12 Wider social and environmental costs and benefits for which there is no market price also need to be brought into any assessment. They will often be more difficult to assess but are often important and should not be ignored simply because they cannot easily be costed. Annex 2 provides more information on how to take into account the wider impacts of proposals.

Estimating costs

5.14 Costs should be expressed in terms of relevant opportunity costs. It is important to explore what opportunities may exist. An example of an opportunity is to use land in a different, more valuable, way than in its current use.

5.18 For substantial proposals, the relevant costs are likely to equate to the full economic cost of providing the associated goods and services, and for these proposals, the full economic cost should be calculated, net of any expected revenues, for each option. The full cost includes direct and indirect costs, and attributable overheads.

Estimating the value of benefits

5.24 The purpose of valuing benefits is to consider whether an option’s benefits are worth its costs, and to allow alternative options to be systematically compared in terms of their net benefits or net costs. The general rule is that benefits should be valued unless it is clearly not practicable to do so. Even if it is not feasible or practicable to value all the benefits of a proposal, it is important to consider valuing the differences between options.

5.25 In principle, appraisals should take account of all benefits to the UK. This means that as well as taking into account the direct effects of interventions, the wider effects on other areas of the economy should also be considered. These effects should be analysed carefully as there may be associated indirect costs, such as environmental costs, which would also need to be included in an appraisal. In all cases, these wider effects should be clearly described and considered.

Valuing costs and benefits where there is no market value

5.30 Most appraisals will identify some costs and benefits for which there is no readily available market data. In these cases, a range of techniques can be applied to elicit values, even though they may in some cases be subjective. There will be some impacts, such as environmental, social or health impacts, which have no market price, but are still important enough to value separately.

ADJUSTMENTS TO VALUES OF COSTS AND BENEFITS

Distributional analysis

5.33 It is important that the distributional implications of each option are considered during appraisal. This type of analysis enhances the understanding of the fairness of proposals, their social impacts and their scale.

5.34 The impact of a policy, programme or project on an individual’s well-being will vary according to his or her income; the rationale being that an extra pound will give more benefit to a person who is deprived than to someone who is well off. In economics, this concept is known as the ‘diminishing marginal utility of additional consumption’.

5.35 Other distributional issues may also arise, and should be considered during appraisal. A proposal may have differing impacts according to age, gender, ethnic group, health, skill, or location. These effects should be explicitly stated and quantified wherever feasible. For example, the costs and benefits of a proposal might be broken down according to the ethnic group they accrue to, providing appraisers with a basis for comparison and analysis.

5.41 Where appraisers decide not to adjust explicitly for distributional impacts, they must provide a justification for this decision. This judgement should be informed by the following considerations:

  • The significance of the impact of distributional analysis to the proposal under consideration;
  • The ease with which distributional impacts can be measured; and
  • The scale of the impact associated with a particular project or proposal.

ADJUST FOR DIFFERENCES IN TAX BETWEEN OPTIONS

5.55 The adjustment of market prices for taxes in appraisal is appropriate where it may make a material difference to the decision. In practice, it is relatively rare that adjustments for taxation are required, because similar tax regimes usually apply to different options. It can also be difficult in practice to estimate costs net of tax. However, where the tax regimes applying to different options vary substantially, this should not be allowed to distort option choice. In such cases it is important to adjust for any differences between options in the incidence of tax arising from different contractual arrangements, such as in-house supply versus buying in, or lease versus purchase.

ADJUSTING FOR BIAS AND RISKS

Optimism bias

5.61 There is a demonstrated, systematic, tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors [11]. Many project parameters are affected by optimism – appraisers tend to overstate benefits, and understate timings and costs, both capital and operational.

5.62 To redress this tendency, appraisers should make explicit adjustments for this bias. These will take the form of increasing estimates of the costs and decreasing, and delaying the receipt of, estimated benefits. Sensitivity analysis should be used to test assumptions about operating costs and expected benefits.

ASSESSING UNCERTAINTY

Sensitivity analysis

5.69 Sensitivity analysis is fundamental to appraisal. It is used to test the vulnerability of options to unavoidable future uncertainties. Spurious accuracy should be avoided, and it is essential to consider how conclusions may alter, given the likely range of values that key variables may take. Therefore, the need for sensitivity analysis should always be considered, and, in practice, dispensed with only in exceptional cases.

5.70 The calculation of switching values shows by how much a variable would have to fall (if it is a benefit) or rise (if it is a cost) to make it not worth undertaking an option. This should be considered a crucial input into the decision as to whether a proposal should proceed. It therefore needs to be a prominent part of an appraisal.

5.71 Examples of variables that are likely to be both inherently uncertain and fundamental to an appraisal are the growth of real wages, forecast revenues, demand, prices, and assumptions about the transfer of risks. A prior analysis of costs into fixed, step, variable, and semi variable categories can help in understanding the sensitivity of the total costs of proposals.

Scenarios

5.72 Scenarios are also useful in considering how options may be affected by future uncertainty. Scenarios should be chosen to draw attention to the major technical, economic and political uncertainties upon which the success of a proposal depends. Considering scenarios needs to be proportionate. It may take the form of asking simple ‘what if ’ questions for small and medium sized projects, but extend to creating detailed models of future states of the world for major policies and large programmes. The expected NPV can be calculated for each scenario. It may also be helpful to undertake some sensitivity analysis within a scenario.

CONSIDERING UNVALUED COSTS AND BENEFITS

5.76 Costs and benefits that have not been valued should also be appraised; they should not be ignored simply because they cannot easily be valued. All costs and benefits must therefore be clearly described in an appraisal, and should be quantified where this is possible and meaningful.

5.77 Research may need to be undertaken to determine the best unit of measurement. Alternative non-monetary measures might be considered most appropriate (See Box 17).

5.78 The most common technique used to compare both unvalued costs and benefits is weighting and scoring (sometimes called multi-criteria analysis). The basic approach to weighting and scoring involves assigning weights to criteria, and then scoring options in terms of how well they perform against those weighted criteria. The weighted scores are then summed, and these sums can be used to rank options. An even simpler method is to list the required performance criteria (sometimes called ‘critical success factors’), and assess options in terms of whether they meet them or not.

5.79 In practice, the weight to give to factors that are thought to be important by key players cannot be decided by ‘experts’. They inevitably incorporate the judgements of stakeholders and decision makers. The risk that they are weighted towards acceptance of more expensive solutions by those who would enjoy the potential benefits should be tempered by at least one stakeholder representing the opportunities that an expensive solution would be foregone elsewhere. There are other pitfalls to avoid in carrying out this type of analysis, and reference should be made to guidance on multi-criteria analysis.

 

6 DEVELOPING AND IMPLEMENTING THE SOLUTION

INTRODUCTION

6.1 Following the identification and description of all costs, benefits and risks, their valuation where feasible, and their testing through sensitivity and scenario analysis, the best option should be selected. Transparency is important at this stage, so that it is clear on what basis decisions are taken.

6.2 Once an option has been selected, it will need to be refined into a solution. Consultation is important at this stage. Further consideration will need to be given to the implementation of the proposal, including the involvement of the private sector, procurement options and processes, and the programme and project management arrangements that may be required.

SELECTING THE BEST OPTION

Decision guidelines

6.3 If a full cost benefit analysis has been undertaken, the best option is likely to be the one with the highest risk adjusted net present value. To the extent that all costs, benefits and risks have been robustly valued, this guideline can be applied with more certainty. In cost effectiveness analysis, the option with the lowest net present cost should be the best, again assuming that the cost estimates are as accurate and reliable as possible.

6.5 Other decision criteria can be used to help select options where risk is an important consideration. The ‘maximin-return’ option is the most important to consider. It is the most risk averse option, as it is the option that provides the least bad outcome if the worst possible conditions prevail.

BOX 20: EXAMPLE – MAXIMIN RETURN

Two government services are being considered, which are mutually exclusive. Their NPVs under different market conditions are shown below:

  Low demand (£) Expected value (£) High demand (£)
Service A 1,000,000 1,200,000 1,600,000
Service B 100,000 1,250,000 2,000,000

The maximin criteria points to Service A, as it provides the highest value in the worst market conditions.

6.6 In practice, other factors will also affect the selection of the best option, in particular the consideration of unvalued costs and benefits. Weighting and scoring techniques are useful in comparing different options in terms of the same criteria. However, as scores are not expressed in monetary terms, judgement is then required to compare the results of weighting and scoring with the cost benefit or cost effectiveness analysis. The two analyses should complement each other, and may indicate that further analysis is required before a decision can be reached. Annex 2 provides further information on how weighting and scoring can be brought into the decision making process. Fully involving stakeholders is very important in making judgements between monetised and non-monetised effects.

6.7 There is always a value imputed by decisions to proceed, and this value should always be clearly identified and analysed.

Affordability, funding, and cashflows

6.10 The affordability of options should always be considered when developing and selecting options. In addition to the analysis of economic costs and benefits, appraisals usually need three major financial statements, at least for the lead options:

  • A budget statement. This should be based on resource accounting and budgeting (RAB) principles, and show the resource costs over the lifetime of the proposal. For strategic initiatives, the budget will often comprise the forecast RAB financial statements of a whole organisation over a number of years.
  • A cashflow statement. This should show the additional cash that will be spent on the lead option if it goes ahead.
  • A funding statement. This should show which internal departments, partners and external organisations would provide the resources (and in some cases cash) required.

DEVELOPING THE SOLUTION

Introduction

6.12 The best option is likely to require further refinement before a solution emerges. Options are rarely completely mutually exclusive, so it is useful to review the other options to see if their good parts can be grafted onto the leading option.

Consultation

6.13 Consultation with external experts and with those affected is very important at this stage, whether or not formal or informal consultation has taken place earlier on.

6.14 Consultation on projects will usually be on one or two lead proposals; whereas consultation on policy and programme proposals that have more widespread effects should usually be undertaken both earlier, and on a wide range of options and alternatives.

6.15 Analysis of who is affected by a proposal, undertaken as part of the appraisal, may be very useful in determining who should be consulted, and also in considering the details of implementation. Attention should be drawn to the key assumptions, options and implementation issues. Consultation exercises should be drawn up in line with the following best practice guidelines [3]:

  • Use the most appropriate approach. Written consultation may not the best way to canvass views on a policy or project option. Methods include meetings with interested parties and user surveys.
  • Consultation should be easy to respond to (e.g., by electronic means).
  • Check if statutory obligations apply.
  • Allow sufficient time; consultation should be built into the planning process at the start.
  • Be clear about who is being consulted, about what, in what time-scale, for what purpose.
  • Consider joining up with other consultations, for instance in other government departments.
  • Consultation documents should be clear, concise and focused.
  • Ensure that the process reaches the target audience.
  • Ensure that people are told the results, and the reasons for decisions taken.

IMPLEMENTATION

6.23 Implementation plans should be sufficiently complete to enable decisions to be taken on whether or not to proceed. So that evaluations can be completed satisfactorily later on, it is important that during implementation, performance is tracked and measured, and data captured for later analysis.

Programme and project management

6.24 Economically justifiable and financially affordable proposals are of no value if realistically they cannot be implemented. The implementation of proposals must be considered as part of the appraisal process, enough to ensure at least that proposals are viable, risks are manageable, and that benefits can be realised, before significant funds are committed. These aspects of appraisal develop iteratively as with the analysis of costs and benefits.

Performance management and measurement

6.27 Performance management concerns tracking the success of a policy, programme or project in achieving its objectives and in securing the expected benefits. For appraisal and evaluation purposes, it involves the systematic collection of data relating to the financial management and outcomes of the policy, programme or project during implementation.

6.29 Effective performance measurement and monitoring means tracking all categories of benefit and ensuring that:

  • Projects have defined target benefits and outputs;
  • Ownership of the delivery of benefits remains with the programme manager;
  • Outputs of a project or policy remain consistent with changing government objectives;
  • Targets and achieved benefits are measured, reported and communicated;
  • Costs are closely monitored and managed; and,
  • Forecast costs and benefits are frequently reviewed.

6.30 A monitoring system should establish:

  • Whether management data is actually measuring what it purports to measure; and,
  • Put in place sufficient controls to ensure that the data is accurate.

Financial reporting

6.31 Regular financial reporting on policies, programmes and projects should be performed. Reports may be integrated into the normal financial reporting cycle of an organisation, issued separately, or possibly combined with the reporting of progress against plan, benefits, and risks.

Benefits realisation management

6.33 Benefits realisation management is the identification of potential benefits, their planning, modelling and tracking, the assignment of responsibilities and authorities and their actual realisation. In many cases, benefits realisation management should be carried out as a duty separate from day to day project management.

6.34 Benefits fall into four main categories, which are described below.

BOX 24: BENEFIT CATEGORIES

Benefit   Example
Financial Quantitative Operating cost reduction, revenue increase
Non-financial Quantitative Number of customer complaints, reduction in road accidents, percentage of government departments on-line
Non-financial Qualitative Staff skills, staff morale
Outcomes Quantitative and qualitative Improved standards of healthcare

6.35 It is also useful to identify financial savings that release cash for other uses.

 

7 EVALUATION

INTRODUCTION

7.1 When any policy, programme or project is completed or has advanced to a pre-determined degree, it should undergo a comprehensive evaluation. Major or on-going programmes, involving a series of smaller capital projects, must also be subject to ex post evaluations.

7.2 Evaluation examines the outturn of a policy, programme or project against what was expected, and is designed to ensure that the lessons learned are fed back into the decision-making process. This ensures government action is continually refined to reflect what best achieves objectives and promotes the public interest.

7.3 Evaluation comprises a robust analysis, conducted in the same manner as an economic appraisal, and to which almost identical procedures apply. It focuses on conducting a cost benefit analysis, in the knowledge of what actually occurred rather than what is forecast to happen.

EVALUATION PROCESS

7.5 The evaluation itself would normally follow this sequence:

  1. Establish exactly what is to be evaluated and how past outturns can be measured.
  2. Choose alternative states of the world and/or alternative management decisions as counterfactuals.
  3. Compare the outturn with the target outturn, and with the effects of the chosen alternative states of the world and/or management decisions.
  4. Present the results and recommendations.
  5. Disseminate and use the results and recommendations.

Evaluation requires management initiative (sometimes political commitment) and intensive monitoring. The thoroughness of an evaluation should depend upon the scale of the impact of a policy, programme or project, and to some extent on the level of public interest. There may be a high level of media interest around a project which has required a significant degree of expenditure, or one which is highly complex, novel, or represents a pilot for future large scale programmes. Evaluation reports should be widely disseminated and published, where appropriate, to contribute to the knowledge base upon which future decisions will be taken.

Establish what is to be evaluated

7.6 The activity to be evaluated needs to be clearly specified. The evaluation might be of a project, programme or policy, particular aspects of the activity, or of key common issues affecting a number of activities. It might also be a pilot designed especially for evaluation.

7.7 Objectives, outcomes and outputs should be defined and quantified as precisely as possible for use in step three below. It is important to distinguish between the objectives and outcomes, and the outputs and targets.

7.8 The availability of output and performance measures and targets, and other monitoring data, and how they relate to the objectives should be reviewed. If this information is inadequate, consideration should be given to the collection of additional data, although ideally, data needs would have been considered at the outset of the project.

Alternative States / Management Decisions

7.9 The definition of exactly what needs to be compared with what needs to be clearly stated. The outturn of any complex activity will never be exactly as projected in advance. However, the reasons for the outturn being better or worse than expected may be attributable to the ‘state of the world’, or to actions of the responsible body. These might include the management of the project, forecasting assumptions, or the inherent design of the policy.

Compare the Outturn with Targets

7.10 As discussed earlier, the technical methodologies used for appraisal and evaluation are similar. Each should identify and measure, where possible, both the direct and indirect benefits of the policy, programme or project. The main difference is that evaluation tends to be based on actual data, and appraisal on forecasts and projections.

7.11 The evaluation should include the following:

  • An assessment, quantified where possible, of what happened;
  • A comparison with the target outturn; and
  • A comparative assessment of one or more counterfactuals (i.e. alternative outturns given different states of the world, or different management decisions).

7.12 Where possible the comparative assessment should include a ‘control group’, to whom the activity was not applied.

7.13 It is usual to take as a benchmark for comparison, what would have happened if the activity under consideration had not been implemented. It is also useful to consider the consequences of implementing one or more of the alternatives considered during appraisal. Occasionally it may be appropriate to consider an option that was not originally appraised, as long as it was feasible at the time of implementation.

7.14 The evaluation should assess the success of the project, programme or policy in achieving its objectives, and also how this achievement has contributed to the wider outcomes. If the objectives were not achieved, the evaluation should establish why that was the case.

Presentation of results and recommendations

7.15 The results of an evaluation should summarise:

  • Why the outturn differed from that foreseen in the appraisal;
  • How effective the activity was in achieving its objectives, and why;
  • The cost effectiveness of the activity; and
  • What the results imply for future management or policy decisions.

7.16 The results obtained should generally lead to recommendations for the future. These may include, for example, changes in procurement practice, delivery, or the continuation, modification, or replacement of a programme.

Disseminate the results and recommendations

7.17 The results and recommendations from evaluation should feed into future decision making. The methods used to achieve this will generally require senior management endorsement. Efforts should be made to disseminate the results widely, and, for this purpose, it may be helpful to use summaries of the main points, and reports which synthesise the results from a number of evaluations with common features.

7.18 Evaluation reports and the research that informs them should be placed in the public domain unless there are good reasons, in terms of security or commercial confidentiality, for not doing so.

Comparison of Appraisal and Evaluation

7.19 Box 25 sets out the differences between undertaking an assessment at the outset, in support of government intervention – appraisal – and undertaking an assessment to evaluate how successful such action has been – evaluation.

BOX 25: COMPARISON OF APPRAISAL AND EVALUATION

  Appraisal Evaluation
Aim Ex ante assessment of whether action is worthwhile and impacts Ex post assessment of whether action was worthwhile and impacts
Use of Output Project procurement, policy and programme design Feedback for:
(a) future procurement, project management,
(b) wider policy debate, and
(c) future programme management.
Application Projects, policies and programmes Projects, policies and programmes
Timing Always prior to implementation . During implementation (‘formative’)
. After implementation (‘summative’)
Data Forecasted Historic and current, estimated and actual.
Estimates of counterfactuals
Method Comparison of options against ‘do nothing’ option

Estimated assessment of risk

Comparison of results against ‘do nothing’ option.
Comparison of actual outturns against target outturns/ alternative outturns.
Assessment of risks that did or did not materialise.
Analytical

Techniques

 Cost Benefit/ Effectiveness Analysis.
Discounted cash flow analysis.
Multi-criteria analysis.
Other statistical analysis.		 Cost Benefit/ Effectiveness Analysis.
Discounted cash flow analysis.
Multi-criteria analysis.
Other statistical analysis – e.g.: analysis of performance indicators.
Decision

Criteria

Comparison of NPV, NPC for different options.
Non quantifiable factors may be included if quantification impossible		 Consideration of whether correct criteria were used
Audit and

Enforcement

 Public Accounts Committee (PAC), NAO, HMT, OGC Gateways 0, 1. Departmental arrangements PAC, NAO, HMT, OGC Gateway 5.

Departmental arrangements