Evaluating performance of environmental health assessments

<accesscontrol>Members of projects,,Workshop2008,,beneris,,Erac,,Heimtsa,,Hiwate,,Intarese</accesscontrol>

This is a manuscript about evaluating performance of environmental health assessments. It discusses different perspectives to evaluating the goodness of assessments and proposes a new more comprehensive approach to considering performance of environmental health assessments.

Title

The properties of good assessments - a multi-perspective approach to evaluating performance of environmental health assessments

Abstract

Environmental health assessments are carried out in order to provide information about the environmental phenomena that affect human health and well-being. This information can be used by various kinds of actors in multiple societal contexts in order to create understanding about these phenomena and to decide upon possible actions of dealing with them and their effects. The quality of the information and the ways it is produced and provided for use significantly influence the effectiveness and efficiency of environmental health assessment. Understanding the factors that constitute the overall performance of environmental health assessments is thus crucial in evaluating the goodness of assessments and especially in designing and executing good assessments. Various perspectives to consider performance of assessments have been taken building on the bases of different fields of study, but a fully comprehensive approach that would sufficiently address all the aspects of this complex issue in a concise manner has not yet evolved. We propose a new approach for considering the performance of environmental health assessments. It describes the general properties of good assessments and simultaneously takes account of the assessment process, its products and their intended uses as a whole. The properties of good assessments can be used as assessment design and execution principles as well as a framework for evaluating past assessments. The new approach is capable of incorporating the relevant aspects of the previously proposed approaches to performance and related issues, but provides a more comprehensive and coherent framework by taking a trialogical perspective to the issue. It considers the assessment product as the mediating shared object of activity and scrutinizes it as the central object bridging the assessment process with the use process.

Situation

• environmental health assessment is intentional societal activity involving plural actors with diverse perspectives and needs
  • intentionality and actors
    • provide information on practical decision-making (political, industry, individuals)
    • increase awareness and level of understanding on important issues (experts, DMs, SHs, public)
    • advance scientific research (experts)
  • activity: understanding and describing real-world phenomena in order to enable rational actions accordingly
• effectiveness of assessments must be ensured → performance of assessments needs to be considered and evaluated
• also efficiency should be considered
• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...

Problem

• all perspectives provide only limited or narrow views to performance
  • use purpose of information often not considered
  • all aspects of performance not covered
  • evaluation as a separate process, often only after assessment
  • societal aspect often neglected
  • focus either on giving assessment procedure guidelines or considering product as such

Solution

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

Evaluation

• focus on the mediator of the overall intellectual process - the shared object of activity
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
  • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

Introduction

• environmental health assessment is intentional societal activity involving plural actors with diverse perspectives and needs
  • intentionality and actors
    • provide information on practical decision-making (political, industry, individuals)
    • increase awareness and level of understanding on important issues (experts, DMs, SHs, public)
    • advance scientific research (experts)
  • activity: understanding and describing real-world phenomena in order to enable rational actions accordingly
• effectiveness of assessments must be ensured → performance of assessments needs to be considered and evaluated
• also efficiency should be considered
• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...

Perspectives to performance

Assessment performance has been considered from several different perspectives building on different scientific backgrounds and approaching the issue from different environmental health relevant fields of study. Relevant research has been conducted e.g. under the titles uncertainty assessment, quality assurance and good practice guidelines within e.g. the fields of integrated assessment, risk assessment and water resource management.

Most of the perspectives fall short in taking account of the intentionality and societal aspects of assessments and they all too often consider the evaluation of performance as a process separate to the assessment itself. Furthermore, where the evaluation often focuses on the substantial parts of the assessment, such as model or its outputs and assessment reports, the guidelines are given as procedural and thus only addressing the

• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...
• all perspectives provide only limited or narrow views to performance
  • use purpose of information often not considered
  • all aspects of performance not covered
  • evaluation as a separate process, often only after assessment
  • societal aspect often neglected
  • focus either on giving assessment procedure guidelines or considering product as such

Properties of good assessments

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

• Relation of properties to information structure/content

• Evaluation process
  • a priori and/or a posteriori view
  • identification of purpose
  • evaluation of quality of content (uncertainty + relevance)
    • in principle reality, but in practice golden standard as reference point D↷
  • evaluation of applicability
  • evaluation of efficiency (effort expenditure)
  • overall performance
    • potential for effectiveness/effort given purpose
      • can be further evaluated retrospectively against realized effectiveness (possibly against redefined purpose)

Discussion

• focus on the mediator of the overall intellectual process - the shared object of activity
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
  • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

• data about hypothesis

Conclusions

• There is more to assessment performance than just statistical uncertainty and data source reliability
• Overall performance of assessment can be evaluated systematically and explicitly
  • requires consistent information structure
  • a priori evaluation should be made an inherent part of assessment process