After an intensive phase of implementing activities and with the monitoring data output at hand, it is time to step back and evaluate what has been achieved. The data collected through monitoring are used for evaluating both the results obtained through implementation and the way the management cycle is working.

Evaluation and reporting is the last step of the cycle, but provides the basis for starting a new year with a new cycle. It analyzes what has happened during the year in order to understand why things happened or failed to succeed. It provides the politicians with a basis for taking further decisions on the targets and actions for the next year. It provides the stakeholders, including the public, with a report on what the city has done during the year and how they have succeeded in fulfilling their targets. The importance of this step is the actual city council decision on how to act on the results of the evaluation process. How will the knowledge gained be used to adapt or set the short-term targets for the next year? What actions should be implemented next year? Is there a need to revise the baseline review because of major changes in the city or its surroundings? In any case, a decision should be taken and a new annual cycle should begin.

An effective monitoring and evaluation process provides ongoing, systematic information that strengthens project implementation. The monitoring and evaluation process provide an opportunity to:

a) Compare implementation efforts with original goals and targets,
b) Determine whether sufficient progress is being made towards achieving expected results and,
c) Determine whether the time schedule is observed.

Only when the data collected through the monitoring process is analyzed and evaluated can it help to improve the management system by identifying flaws, failures and particularly successful elements in both the way the process was organized and implemented and the outcome.

An effective reporting system combines the data collected within the monitoring system of all institutions with implementation responsibilities. It provides a system of accountability for all responsible parties on how well they are achieving the objectives and targets established in the strategic programme.

Monitoring and evaluation are not “events” that occur at the end of a project, but rather are ongoing processes that help decision-makers better understand the effectiveness of actions. An effective monitoring and evaluation programme requires collecting and analyzing important data on a periodic basis throughout the life of the integrated management system. This process often involves collecting baseline data on existing conditions, reporting on progress toward the preset targets, making connections between actions and intended outcomes, and making mid-course changes in the strategic programme and/or action plan. A good monitoring and evaluation process engages all stakeholders and is useful to those ultimately responsible for improving the integrated management system. Evaluation is also an important public awareness and educational tool.

The evaluation analyzes the data of the monitoring exercise and makes a statement as to whether progress is as expected.

The evaluation contains a summary of all actions undertaken as well as comments to and interpretations of the specific achievements and deviations from the targets. It must refer to city statutes as well as to strategic objectives and targets (ideally already addressed in the strategic programme).The evaluation is an important input to drawing lessons from the implementation experience and using the experience to improve the next cycle. It should formulate concrete recommendations for changes in the strategic programme and action plan. As with monitoring, both

• the process (How have actions been implemented, how can the process be improved?)
• and the results (Have the targets been reached? Why/why not? What changes would be needed to reach the targets? Or: Do the targets need to be adapted?)

need to be analyzed and evaluated.

There are several methods of evaluation. One can either ask the various responsible collaborators to give a written account of things, leading them more or less strictly with guiding questions. A questionnaire can be developed to get precise answers on specific items, or an evaluation form can be developed. The advantage of the latter is that the answers can easily be compared to one another: on the other hand, some explanations might not be captured with a standardized format.

An important component of the integrated management system is to be able to continuously improve. Carrying out audits is one strong element of this. An audit is carried out to check the situation according to some preset criteria. Audits involve people interviewing other people and thus enable a very close connection to the people actually working with the system. During an audit, it is possible to find explanations that cannot be captured with a standardized format. An audit can also be seen as a good way to get input and experiences from another person/department/organization that can actually help in solving a problem. It is usually carried out as a combination of a desk review and in the form of interviews.

There is a difference between internal audits and external audits. In an internal audit, the auditor/s is usually a person or persons from within the administration, but not directly involved in the implementation process. An external audit is usually an assessment by an independent, certified environmental verifier. An external audit is for example required to obtain an EMAS registration or an ISO 14001 certification.

A particular method for evaluating is to carry out an internal audit. The auditor is usually a person from within the administration, but not directly implied in the implementation process. Another possibility to conduct an internal audit is a peer review: a colleague or team from another city realizes the audit in your city and vice versa. The auditor follows a given scheme and evaluates all elements of the system at a given point in the cycle.

Carrying out internal audits requires effective planning and setting up an audit programme in addition to an audit system. It requires that the person/s working with the audits are well familiar with the management system and the thematic issues they are going to investigate about. However, if it is well-planned and carried out, it can be of real benefit for a city. Internal audits can be used as a way of getting the whole city involved in the work with the integrated management system and can be seen as an educational tool. It will help them in seeing the potentials with the integrated management system and could therefore strengthen the work as a whole with the integrated management system. It also provides a great deal of input for how to improve the management system.

• Read more: What should be checked in an internal audit?

• Example: Peer review between Stockholm and Oslo

The EMAS and ISO 14001 certification processes require an assessment by an independent, certified environmental verifier. It is only after this person has "validated" the management system that it may be termed an approved environmental management system according to EMAS and ISO 14001 criteria.

The integrated management system facilitates the “external audit” – the verification by an accredited auditor according to the requirements of EMAS and ISO 14001 – as well. Even if the scope of integrated management is wider and not limited to environmental aspects, it fulfils all elements and criteria of EMAS and ISO 14001.

On the basis of the experiences and results within the Managing Urban Europe-25 project, some wording and procedures have been further developed and adopted to the reality and the structure of local authorities, and probably the official auditor is not familiar with this adaptation of EMAS and ISO 14001 to the needs of local authorities. Therefore, it is necessary to discuss with the verifier and to explain the differences.

The table “Comparison EMAS/ISO 14001 and MUE-25 integrated management system” includes all differences and can be useful to inform the accredited auditor and help him/her to understand the integrated approach and how it relates to EMAS/ISO 14001.

• Read more: Comparison EMAS/ISO 14001 and MUE-25 integrated management system

The environmental verifier validates the environmental statement or the environmental part of the sustainability report, i.e., s/he checks that the information it contains is correct and reliable and confirms its validity. It lies within the powers of the environmental verifier to determine whether an on-site reassessment must be conducted every one or three years. S/he makes this decision on the basis of the number of employees and the impact the organization has on the environment.

The long periods of time required in the field of municipal planning and development to realize objectives means that changes are therefore only measurable in the long term. This ought to convince the verifier that a validation cycle of 36 months is appropriate for an environmental management system in this field.

The environmental verifier carries out the validation in two steps: a review of the documents and an on-site visit. The coordinator is responsible for preparing for the verifier's on-site visit. He or she must ensure that the environmental verifier is given an appointment with the "top executive" (i.e., the mayor) in order to talk about the strategic programme and development of action plans, the organizational set up and the monitoring and evaluation procedure and results.

The verifier takes random data samples to see information related to the environmental management system (protocols from coordination team meetings, updates to reference data /indicators, the process by which the observation of legal requirements relevant to the environment is ensured, the organizational set up, internal communication and training of staff and the communication to the public, i.e., the environmental or sustainability report).

The results of the audit as well as necessary corrections are discussed at the end of the visit. After the on-site assessment, the mayor receives a detailed audit report. If specific EMAS/ISO 14001 requirements have not been met, the municipal administration normally is granted a period of time to make corrections. In the case of serious deviations that cannot be adequately corrected within this period, the environmental verifier does not grant the validation.

If the city was verified according to EMAS, the procedure is that once the environmental verifier has declared the validity of the environmental statement, the local authority can apply to be entered into the EMAS register. To this end, the validated environmental statement must be handed in and a registration fee paid. The administration responsible for the registration asks the supervisory authorities to assess the environmental statement and to confirm, that the local authority concerned fulfils all legal requirements. If no objections arise, then the local authority is entered into the official EMAS registry and is granted the right to use the EMAS logo in its public relations. The revalidation is conducted according to the same scheme.

The coordinator must consider how the evaluation results will be used at the outset of the evaluation process. Implementing institutions are more likely to use information generated from an evaluation if they understand, participate, and have ownership over the evaluation process. Therefore, the more people who have been actively consulted in the process, the easier it will be to use the results for project improvement.

One of the most important aspects of an evaluation process is that it actually provides usable results to project implementers – information that can be utilized by project managers and staff to improve results. Useful evaluation results inform decisions and provide information on how to improve project performance. Thus, if the city failed to meet a certain target within a specified time frame, evaluation results can provide critical information in helping to revise your actions. For example, suppose that the city established a target to reduce solid waste 10% annually for each of the next five years. At the end of the first year, the city discovered that waste disposal had been reduced by only 5%. Further, the evaluation revealed widespread confusion among residents on when and how to recycle. This information strongly indicates the need to significantly improve the educational component of the program in order to achieve target levels.

The results of the evaluation should be used for reporting. The report of the result has essentially two goals: to inform the decision makers about the consequences of their actions and to inform the public about the progress of the city towards sustainability. The report must be presented differently to different target audiences. It implies a different format, different language and various modes of distribution.

The evaluation report needs to be presented to, discussed and approved by the city council. It should serve as a basis for taking decisions regarding the next annual cycle. The evaluation report should provide politicians with a basis for deciding on potential adjustments of targets or actions, responsibilities, etc., that are needed.

Evaluation results summarize the findings of the evaluation. Reports are ideally circulated to a variety of audiences. Relevant stakeholders should also have the opportunity to influence the evaluation report (public involvement).

The language and style of reports may change depending on the audience; however, the information conveyed needs to be consistent and accurate. It is important to keep the public informed of the progress toward achieving the targets set in the strategic programme and action plan. Citizens need to be informed about the improvements made, not the least to learn what action individuals can undertake to help achieve the city’s sustainability goals. This will require an effective communication programme to provide regular information, but also to report their reactions to implementing institutions (collect feedback!).

An effective evaluation can help ensure that the actions selected achieve your goals and targets. The monitoring and evaluation process provides an important process not only for determining whether sustainability targets are being achieved, but also why/why not, and what modifications are necessary to keep the efforts on track. As a critical feedback loop, it is then important to share programme results with community members, the municipal council, and other stakeholders.

Although this step appears at the end of the cycle, it is only the beginning of the next cycle. The next step would be to consider if the baseline review or strategic programme needs to be adapted. If not, the short-term targets and actions for the next year need to be confirmed/decided on, and then the cycle starts again.

• Checklist: Implementation and evaluation

Regional Environmental Center (REC): “Guide to Implementing Local Environmental Action Programs in Central and Eastern Europe” (author: Paul Markowitz, 2000)

UNEP (2005): Geocities Application Manual

Baseline review

• Description of scope of the integrated management system
• Overview regarding departments and roles /responsibilities with relevance to sustainability
• All departments of the local government informed /involved
• Mapping of relevant stakeholder
• All relevant (significant) environmental or sustainability aspects covered
• Documents/programmes/strategies/plans currently in use identified
• Analysis of legal compliance regarding environmental /sustainability aspects included
• Use of meaningful indicators to describe the current situation
• Documentation of gaps with regard to data and indicators
• State of sustainability for different aspects/issues has been assessed objectively and transparently
• Results of BR considered within priority setting
• Stakeholder and departments informed about BR results
• Procedure to update Baseline Review according to Internal Audit results and new information established

System audit:

• Currently implemented management and monitoring structures to run plans and programmes identified and analyzed
• Gaps identified and missing elements developed

Organizational setup

• Decision of council / senior management to administration to draft organizational setup and necessary managerial directives
• Coordinator with appropriate task description and assigned managerial authority
• Co-ordination team with representatives of city departments, external organizations and stakeholder and with task description and responsibilities/managerial directives
• Other relevant bodies (e.g., community platform, Agenda 21) involved with appropriate task description and responsibilities
• Linkages/frameworks of cooperation with other levels of government
• Establishment of formal and informal cooperation with neighbouring municipalities
• Organizational chart of departments and bodies involved in the integrated management system developed
• Financial budgeting requirements and routines considered
• Clear monitoring and reporting structure and rules established
• Clear structures and rules for internal and external communication established
• Checklist and/or standardized format for collection and interpretation of data according to indicators selected
• Capacity development requirements reviewed
• Training plan developed and approved
• Relevant documentation regarding the integrated management accessible for the staff and involved stakeholder
• Checklist for Internal audit developed and internal auditor identified
• Organizational setup and adequate financial resources approved by city council

Strategic programme

• Clear vision (environmental policy) of the local sustainable development
• Clear priority setting based on the results of the baseline review
• 10–15 years perspective
• Quantified long and medium-term targets (target year) included
• Targets set in a participatory process
• Targets supported by meaningful key data and indicators and reference values
• Action plans for all priority aspects with measurable aims, concrete measures and assigned responsibilities established
• All stakeholders informed.
• Public information and consultation
• Strategic programme (and action plan) approved by formal council decision

Monitoring, evaluation and reporting

• Data sources, base year, time frame and periodicity for all indicators identified
• Analysis of existing monitoring and evaluation structures
• Technical solution identified: data storage and update, backup, access, etc.
• Plan for future inclusion of non-available key data/indicator
• Description /template for data reporting and evaluation
• All relevant stakeholders informed about the ongoing evaluation process
• Possibility of feedback on evaluation results for stakeholders
• Evaluation report presented to city council
• Characteristics of monitoring and evaluation system optimized