Difference between revisions of "Evaluating performance of environmental health assessments"

Revision as of 11:03, 25 August 2008

<accesscontrol>Members of projects,,Workshop2008,,beneris,,Erac,,Heimtsa,,Hiwate,,Intarese</accesscontrol>

This is a manuscript on evaluating performance of environmental health assessments. It discusses currently prevailing perspectives to evaluating the goodness of assessments and proposes a new more comprehensive approach based on properties of good assessments.

Title

The properties of good assessments - a new approach to evaluating performance of environmental health assessments

Abstract

Environmental health assessments produce descriptions of reality as answers to questions concerning relations between environmental phenomena and human health and well-being. This information can be used by various kinds of actors in multiple societal contexts in order to create understanding about these phenomena and to decide upon possible actions of dealing with them and their effects. The quality of the information and the ways it is produced and provided for use significantly influence the effectiveness of the information. Also efficiency in producing and using the information is important for the goodness of environmental health assessments. Understanding the factors that constitute the overall performance of environmental health assessments is thus crucial in evaluating assessments and especially in designing and executing good assessments.

Various perspectives to consider performance of assessments have been taken building on the bases of different fields of study, but a fully comprehensive approach that would sufficiently address all aspects of this complex issue in a concise manner has not yet evolved. The two main views of how assessment performance has been considered in recent scientific literature on fields related to environmental health can be characterized as (1) quality assurance approach, attempting to provide procedural guidelines for good assessment practice, and (2) uncertainty assessment approach, attempting to identify and categorize types of uncertainties that tend to reside in the information produced in assessments.

We propose a new approach for considering the performance of environmental health assessments that builds on considering assessments as trialogical processes of creating collective belief systems. It describes the general properties of good assessments which can be categorized as properties related to (i) quality of information content, (ii) applicability of information and (iii) efficiency of assessment. The properties of good assessments can be used as assessment design and execution principles as well as a framework for evaluating past assessments.

The proposed new approach is capable of incorporating the main aspects of both the quality assurance view and the uncertainty assessment view, but provides a more comprehensive and coherent framework that addresses the assessment process, assessment products as well as the use processes as a whole. It considers the assessment product as the mediating shared object of activity and scrutinizes it as the central object bridging the assessment process with the use process and thereby enables explication of all relevant aspects of assessment performance in a single framework.

Introduction

Environmental health assessments produce descriptions of reality as answers to questions concerning relations between environmental phenomena and human health. As environment can be considered to cover about the whole reality that surrounds us and health can be considered cover a whole spectrum of different diseases as well as aspects of well-being, it can be seen that the scope of environmental health assessment is a very wide-ranging and complex one and can be broken down into several more detailed or strictly bound disciplines. The term environmental health assessment is used in this paper in a very broad sense as an overarching meta-discipline that includes all kinds of systematic assessment-type inquiries that provide information relevant to environment and health. Examples of such activities are e.g. integrated assessment and modeling, risk assessment, health impact assessment, environmental impact assessment, environmental modeling and water resources modeling and management to name a few.

The information produced in environmental health assessments can be used by various kinds of actors in multiple societal contexts in order to create understanding about these phenomena and to decide upon possible actions of dealing with them and their effects. It can be considered as providing input to practical decision-making in political processes, but also in e.g industrial and commercial organizations and on the level of everyday activities of an individual citizen. In addition to providing aids for finding solutions to specific decision making problems the information provided by environmental health assessments can function in increasing the general level of awareness and understanding about important issues relevant to environment and health. This kind of societal learning can also take place on several levels, e.g. among political and industrial decision makers, NGOs and other stakeholder organizations as well as public at large, but also within the community of scientific experts on the particular or related fields. The general intentionality of environmental health assessments can thus be characterized as being three-fold as:

1. Answering to specific needs of (usually political, but also business or individual) decision making
2. Contributing to societal collective learning about environment and health
3. Contributing to scientific research on environment and health

The quality of the information and the ways it is produced and provided for use significantly influence the effectiveness of the information.

Also efficiency in producing and using the information is important for the goodness of environmental health assessments.

Understanding the factors that constitute the overall performance of environmental health assessments is thus crucial in evaluating assessments and especially in designing and executing good assessments.

Various perspectives to consider performance of assessments have been taken building on the bases of different fields of study, but a fully comprehensive approach that would sufficiently address all aspects of this complex issue in a concise manner has not yet evolved.

Current main perspectives to performance

The two main views of how assessment performance has been considered in recent scientific literature on fields related to environmental health can be characterized as (1) quality assurance approach, attempting to provide procedural guidelines for good assessment practice, and (2) uncertainty assessment approach, attempting to identify and categorize types of uncertainties that tend to reside in the information produced in assessments.

• Two basic lines of thinking: uncertainty assessment & quality assurance
  • QA: procedural guidelines (internal, public, public interactive); process → product
  • UA: content (and wrapping) of the product and the process; product → process
• QA:
  • CEPA (Forrestal et al.) 2008, principle-based approach to QA (risk assessment)
  • Jakeman et al. 2006, 10 iterative steps in model development and evaluation (environmental modeling)
  • Risbey et al. 1996 procedural issues to address (integrated assessment)
  • Guimarães & Funtowicz 2005, QA by extended peer-review (groundwater resources, post-normal science)
  • EPA 2002: guidance for QA → guidance for modeling
  • Refsgaard et al. 2004 & 2005, modeling guidelines new QA guidelines (model based water management)
• UA:
  • van Asselt & Rotmans 2002, typology of uncertainty in IAM & pluralistic uncertainty management (integrated assessment modeling)
  • van der Sluijs et al. 2005, NUSAP (model-based environmental assessment)
  • Walker at al. 2003, conceptual basis for uncertainty management (model-based decision support)
  • Brown et al. 2005, data uncertainty recording (environmental data)
  • Blind & Refsgaard 2007, uncertainty in data and models (water resources management)
  • Kann & Weyant 2002, Decision making under uncertainty (energy/economic policy models)

QA/UA combined:

• • MNP, Risbey et al. 2005, Janssen et al. 2005, van der Sluijs 2008, guidance for QA/UA & communication (environmental management)
  • Refsgaard et al. 2006 & 2007, guidelines to dealing with uncertainty (environmental modeling)
• other:
  • Morgan & Dowlatadabi 1996, 7 attributes of good climate change IA's
  • Parker et al. 2002, Progress in IAM
  • van der Sluijs 2007, uncertainty and precaution / UPEM (environmental management)

• general shortcomings of QA & UA:
  • use process often not explicitly included and use purpose not considered
  • societal aspects often neglected
  • all aspects of performance not covered in any single approach
  • evaluation as a separate process, often only after assessment

Properties of good assessments

We propose a new approach for considering the performance of environmental health assessments that builds on considering assessments as trialogical processes of creating collective belief systems. It describes the general properties of good assessments which can be categorized as properties related to (i) quality of information content, (ii) applicability of information and (iii) efficiency of assessment. The properties of good assessments can be used as assessment design and execution principles as well as a framework for evaluating past assessments.

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

• Assessment products as belief systems (Hilpinen)
• Trialogue: the knowledge creation metaphor, belief systems as mediators
• Assessment participation as innovatiove knowledge communities (Paavola, Hakkarainen)

• Evaluation process
  • a priori and/or a posteriori view
  • identification of purpose
  • evaluation of quality of content (uncertainty + relevance)
    • in principle reality, but in practice golden standard as reference point D↷
  • evaluation of applicability
  • evaluation of efficiency (effort expenditure)
  • overall performance
    • potential for effectiveness/effort given purpose
      • can be further evaluated retrospectively against realized effectiveness (possibly against redefined purpose)

Discussion

The proposed new approach is capable of incorporating the main aspects of both the quality assurance view and the uncertainty assessment view, but provides a more comprehensive and coherent framework that addresses the assessment process, assessment products as well as the use processes as a whole. It considers the assessment product as the mediating shared object of activity and scrutinizes it as the central object bridging the assessment process with the use process and thereby enables explication of all relevant aspects of assessment performance in a single framework.

• focus on the mediator of the overall intellectual process - the shared object of activity
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
    • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

• data about hypothesis

Conclusions

• There is more to assessment performance than just statistical uncertainty and data source reliability
• Overall performance of assessment can be evaluated systematically and explicitly
  • requires consistent information structure
  • a priori evaluation should be made an inherent part of assessment process

Competing interests

Authors' contributions

Acknowledgements

References

Figures

Tables

Additional files