Evaluating performance of environmental health assessments

<accesscontrol>Members of projects,,Workshop2008,,beneris,,Erac,,Heimtsa,,Hiwate,,Intarese</accesscontrol>

This is a manuscript on evaluating performance of environmental health assessments. It discusses currently prevailing perspectives to evaluating the goodness of assessments and proposes a new more comprehensive approach based on properties of good assessments.

Title

The properties of good assessments - a new approach to evaluating performance of environmental health assessments

Abstract

Environmental health assessments produce descriptions of reality as answers to questions concerning relations between environmental phenomena and human health and well-being. This information can be used by various kinds of actors in multiple societal contexts in order to create understanding about these phenomena and to decide upon possible actions of dealing with them and their effects. The quality of the descriptions and the ways they are produced and provided for use significantly influence the effectiveness of the information. Also efficiency in producing and using the information is important with regard to the goodness of environmental health assessments. Understanding the factors that constitute the overall performance of environmental health assessments is thus crucial in evaluating assessments and especially in designing and executing good assessments.

Various perspectives to consider performance of assessments have been taken building on the bases of different fields of study, but a fully comprehensive approach that would sufficiently address all aspects of this complex issue in a concise manner has not yet evolved. The two main views of how assessment performance has been considered in recent scientific literature on fields related to environmental health can be characterized as (1) the quality assurance approach, attempting to provide procedural guidelines for good assessment practice, and (2) the uncertainty assessment approach, attempting to identify and categorize types of uncertainties that tend to reside in the information produced in assessments.

We propose a new approach for considering the performance of environmental health assessments that builds on considering assessments as trialogical processes of creating collective belief systems. It describes the general properties of good assessments which can be categorized as properties related to (i) quality of information content, (ii) applicability of information and (iii) efficiency of assessment. The properties of good assessments can be used as assessment design and execution principles as well as a framework for evaluating past assessments.

The proposed new approach is capable of incorporating the main aspects of both the quality assurance approach and the uncertainty assessment aproach, but provides a more comprehensive and coherent framework that addresses the assessment process, assessment products as well as the use processes as a whole. It considers the assessment product as the mediating shared object of activity and scrutinizes it as the central object bridging the assessment process with the use process and thereby enables explication of all relevant aspects of assessment performance in a single framework.

Introduction

Environmental health assessments produce descriptions of reality as answers to questions concerning relations between environmental phenomena and human health [VIITE]. As environment can be considered to cover about the whole reality that surrounds us and health can be considered to cover a whole spectrum of different diseases as well as aspects of well-being, it can be seen that the scope of environmental health assessment is a very wide-ranging and complex one, and it can be broken down into several more detailed or strictly bound disciplines. The term environmental health assessment is used in this paper in a very broad sense as an overarching meta-discipline that can include all kinds of systematic inquiries that provide information relevant to environment and health. Such activities take place e.g. under the labels of integrated assessment and modeling, risk assessment, health impact assessment, environmental impact assessment, environmental modeling and water resources modeling and management to name a few.

The information produced in environmental health assessments can be used by various kinds of actors in multiple societal contexts in order to create understanding about these phenomena and to decide upon possible actions of dealing with them and their effects [VIITE]. It can be considered as providing input to practical decision-making in political processes, but also in e.g industrial and commercial organizations as well as on the level of everyday activities of an individual citizen [VIITE]. In addition to providing aids for finding solutions to specific decision making problems the information provided by environmental health assessments can function in increasing the general level of awareness and understanding about important issues relevant to environment and health [VIITE]. This kind of societal learning can also take place on several levels, e.g. among political and industrial decision makers, NGOs and other stakeholder organizations as well as public at large, but also within the community of scientific experts on the particular or related fields [VIITE]. The general intentionality of environmental health assessments can thus be characterized as being three-fold as:

1. Answering to specific needs of decision making
2. Contributing to societal collective learning about environment and health
3. Contributing to scientific research on environment and health

The simultaneous parallel goals of environmental health assessment and the range of potential receivers of the information produced in them bring about plural requirements for the quality of the information and the ways of producing and providing it. It appears obvious that there are several simultaneously affecting factors arising from different aspects of an environmental health assessment that can significantly influence the effectiveness of its products. For example credibility of data sources, means of data synthesis, framing of assessment problem, clarity of information representation or choice of media for disseminating assessment results can all be crucial with regard to achieving the intended outcomes of assessment [VIITE]. Also efficiency of the processes of producing and using the information is important when considering the goodness of assessments in the world with ever-present scarcity of resources [VIITE].

It is necessary to understand the factors that constitute the overall performance of environmental health assessments in order to be able to reasonably evaluate the goodness of assessments [VIITE]. Furthermore, and even more importantly, it is essential in being able to design and execute good assessments [VIITE?]. Useful conceptual tools for addressing this issue are thus needed. Various perspectives to consider the performance of assessments have been taken building on the bases of different fields of study relevant to environmental health assessment, but a fully comprehensive approach that would sufficiently address all aspects of this complex issue in a concise manner has not yet evolved [VIITE].

In the following chapter we briefly review some recently published contributions to evaluating performance of environmental health assessments and make an attempt to describe the general characteristics of the most commonly held views. Then we propose a new framework for evaluating the performance of environmental health assessments, which attempts to address the issues in a more comprehensive, yet concise, manner. In the end of the paper we discuss the differences in perspectives and the capabilities of the currently prevailing approaches and the new approach in helping to achieve the general goals of environmental health assessment as explained above.

Approaches to assessment performance

It seems that the scientific discourse regarding issues relevant to performance of environmental health assessment or related disciplines has somewhat shifted broadened the last 15 years from discussions on evaluating or reducing quantitative uncertainty within assessment (or model) outputs mainly by statistical means towards discussions that also address the more qualitative aspects of assessment performance arising not only from the products of assessments, but also from the processes of producing, and sometimes also using, the assessment outputs. This has resulted to a variety of different perspectives being presented and applied in attempts to evaluate the goodness of assessment outputs or finding the factors that tend to improve the goodness of assessments. For example Refsgaard et al. (2007) identified 14 different partly complementary methods to commonly used in uncertainty assessment and characterization in their study on uncertainty in environmental modeling.

We have identified two main views of considering assessment performance in recent scientific literature on fields related to environmental health. They can be characterized as (1) the quality assurance approach, attempting to provide procedural guidelines for good assessment (or modeling) practice (e.g. Forristal et al. 2008, Jakeman 2006, Guimarães Pereira and Funtowicz 2005, Refsgaard et al. 2005; 2004, Risbey et al. 1996), and (2) uncertainty assessment approach, attempting to identify and categorize types and sources of uncertainties that tend to reside in the information produced in assessments (or by models) (e.g. Blind and Refsgaard 2007, van der Sluijs 2005, Brown et al. 2005, Walker et al. 2003, van Asselt and Rotmans 2002, Kann and Weyant 2000). These views are however not exclusive, but rather represent alternative perspectives to basically the same issue which can generally be called assessment performance. In fact some studies have applied both approaches in order to characterize uncertainties and then draw procedural guidelines for reducing them (e.g. Refsgaard et al. 2007; 2006, van der Sluijs et al. 2008, Risbey et al. 2005, Janssen et al. 2005). Also other relevant contributions to scientific discourse on assessment performance which do not fall neatly into either of the above-mentioned categories have been given, e.g. the seven attributes of a good integrated assessment of climate change (Morgan and Dowlatadabi 1996), important features of integrated assessment modeling (Parker et al. 2002) and insights from the UPEM conference (van der Sluijs 2006).

Let us now look into the general characteristics of the two main approaches, starting with the uncertainty assessment approach.

• Two basic lines of thinking: uncertainty assessment & quality assurance
  • QA: procedural guidelines (internal, public, public interactive); process → product
  • UA: content (and wrapping) of the product and the process; product → process
• QA:
  • CEPA (Forrestal et al.) 2008, principle-based approach to QA (risk assessment)
  • Jakeman et al. 2006, 10 iterative steps in model development and evaluation (environmental modeling)
  • Risbey et al. 1996 procedural issues to address (integrated assessment)
  • Guimarães & Funtowicz 2005, QA by extended peer-review (groundwater resources, post-normal science)
  • EPA 2002: guidance for QA → guidance for modeling
  • Refsgaard et al. 2004 & 2005, modeling guidelines new QA guidelines (model based water management)
• UA:
  • van Asselt & Rotmans 2002, typology of uncertainty in IAM & pluralistic uncertainty management (integrated assessment modeling)
  • van der Sluijs et al. 2005, NUSAP (model-based environmental assessment)
  • Walker at al. 2003, conceptual basis for uncertainty management (model-based decision support)
  • Brown et al. 2005, data uncertainty recording (environmental data)
  • Blind & Refsgaard 2007, uncertainty in data and models (water resources management)
  • Kann & Weyant 2002, Decision making under uncertainty (energy/economic policy models)

QA/UA combined:

• • MNP, Risbey et al. 2005, Janssen et al. 2005, van der Sluijs 2008, guidance for QA/UA & communication (environmental management)
  • Refsgaard et al. 2006 & 2007, guidelines to dealing with uncertainty (environmental modeling)
• other:
  • Morgan & Dowlatadabi 1996, 7 attributes of good climate change IA's
  • Parker et al. 2002, Progress in IAM
  • van der Sluijs 2007, uncertainty and precaution / UPEM (environmental management)

Properties of good assessments

We propose a new approach for considering the performance of environmental health assessments that builds on considering assessments as trialogical processes of creating collective belief systems. It describes the general properties of good assessments which can be categorized as properties related to (i) quality of information content, (ii) applicability of information and (iii) efficiency of assessment. The properties of good assessments can be used as assessment design and execution principles as well as a framework for evaluating past assessments.

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency

• Evaluation process
  • a priori and/or a posteriori view
  • identification of purpose
  • evaluation of quality of content (uncertainty + relevance)
    • in principle reality, but in practice golden standard as reference point D↷
  • evaluation of applicability
  • evaluation of efficiency (effort expenditure)
  • overall performance
    • potential for effectiveness/effort given purpose
      • can be further evaluated retrospectively against realized effectiveness (possibly against redefined purpose)

Discussion

The proposed new approach is capable of incorporating the main aspects of both the quality assurance view and the uncertainty assessment view, but provides a more comprehensive and coherent framework that addresses the assessment process, assessment products as well as the use processes as a whole. It considers the assessment product as the mediating shared object of activity and scrutinizes it as the central object bridging the assessment process with the use process and thereby enables explication of all relevant aspects of assessment performance in a single framework.

• general shortcomings of QA & UA:
  • use process often not explicitly included and use purpose not considered
  • societal aspects often neglected
  • all aspects of performance not covered in any single approach
  • evaluation as a separate process, often only after assessment

• properties of good assessments:
  • focus on the mediator of the overall intellectual process - the shared object of activity
    • product|use purpose, assessment process|product
    • trialogical process, knowledge creation metaphor
      • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
  • capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
  • evaluation of performance can only be done meaningfully against purpose
    • various intentions need to be identified, explicated and prioritized in all assessments

• Assessment products as belief systems (Hilpinen)
• Trialogue: the knowledge creation metaphor, belief systems as mediators
• Assessment participation as innovatiove knowledge communities (Paavola, Hakkarainen)

• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

• data about hypothesis

Conclusions

• There is more to assessment performance than just statistical uncertainty and data source reliability
• Overall performance of assessment can be evaluated systematically and explicitly
  • requires consistent information structure
  • a priori evaluation should be made an inherent part of assessment process