Evaluating performance of environmental health assessments

<accesscontrol>Members of projects,,Workshop2008,,beneris,,Erac,,Heimtsa,,Hiwate,,Intarese</accesscontrol>

This is a manuscript about evaluating performance of environmental health assessments. It discusses different perspectives to evaluating the goodness of assessments and proposes a new more comprehensive approach to considering performace of environmental health assessments.

Abstract

Situation

• environmental health assessment is intentional societal activity involving plural actors with diverse perspectives and needs
  • provide information on practical decision-making (political, industry, individuals)
  • increase awareness and level of understanding on important issues (experts, DMs, SHs, public)
  • advance scientific research (experts)
• effectiveness of assessments must be ensured → performance of assessments needs to be considered and evaluated
• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...

Problem

• all perspectives provide only limited or narrow views to performance
  • use purpose of information often not considered
  • all aspects of performance not covered
  • evaluation as a separate process, often only after assessment
  • societal aspect often neglected
  • focus either on giving assessment procedure guidelines or considering product as such

Solution

• multi-perspective approach to performance
  • quality of content
  • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

Evaluation

• focus on the mediator (conceptual artifacts) of the overall process
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
  • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

---

WARNING: CONTENT BELOW THIS TEXT OUT OF DATE - TO BE REVISED SOONISH

Background

• General assessment framework
• Societal context of assessments
• General purpose of assessments
  • to describe reality
  • fulfill specific needs
• Open Assessment
• How can performance of environmental health assessments be evaluated?

Results

• Properties of good assessments
  • quality of content
  • applicability
  • efficiency
• Relation of properties to information structure/content
• Evaluation process
  • a priori and/or a posteriori view
  • identification of purpose
  • evaluation of quality of content (uncertainty + relevance)
    • in principle reality, but in practice golden standard as reference point D↷
  • evaluation of applicability
  • evaluation of efficiency (effort expenditure)
  • overall performance
    • potential for effectiveness/effort given purpose
      • can be further evaluated retrospectively against realized effectiveness (possibly against redefined purpose)

Discussion

• Uncertainty as an aspect of performance
  • parameter uncertainty
  • model uncertainty
  • scenario uncertainty
• Data source reliability as an aspect of performance
• Performance exists only against a purpose
  • data about hypothesis

Conclusions

• There is more to assessment performance than just statistical uncertainty and data source reliability
• Overall performance of assessment can be evaluated systematically and explicitly
  • requires consistent information structure
  • a priori evaluation should be made an inherent part of assessment process