Evaluating performance of environmental health assessments

<accesscontrol>Members of projects,,Workshop2008,,beneris,,Erac,,Heimtsa,,Hiwate,,Intarese</accesscontrol>

This is a manuscript about evaluating performance of environmental health assessments. It discusses different perspectives to evaluating the goodness of assessments and proposes a new more comprehensive approach to considering performance of environmental health assessments.

Title

The properties of good assessments - a multi-perspective approach to evaluating performance of environmental health assessments

Abstract

Situation

• environmental health assessment is intentional societal activity involving plural actors with diverse perspectives and needs
  • intentionality and actors
    • provide information on practical decision-making (political, industry, individuals)
    • increase awareness and level of understanding on important issues (experts, DMs, SHs, public)
    • advance scientific research (experts)
  • activity: understanding and describing real-world phenomena in order to enable rational actions accordingly
• effectiveness of assessments must be ensured → performance of assessments needs to be considered and evaluated
• also efficiency should be considered
• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...

Problem

• all perspectives provide only limited or narrow views to performance
  • use purpose of information often not considered
  • all aspects of performance not covered
  • evaluation as a separate process, often only after assessment
  • societal aspect often neglected
  • focus either on giving assessment procedure guidelines or considering product as such

Solution

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

Evaluation

• focus on the mediator of the overall intellectual process - the shared object of activity
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
  • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

Introduction

• environmental health assessment is intentional societal activity involving plural actors with diverse perspectives and needs
  • intentionality and actors
    • provide information on practical decision-making (political, industry, individuals)
    • increase awareness and level of understanding on important issues (experts, DMs, SHs, public)
    • advance scientific research (experts)
  • activity: understanding and describing real-world phenomena in order to enable rational actions accordingly
• effectiveness of assessments must be ensured → performance of assessments needs to be considered and evaluated
• also efficiency should be considered
• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...

Perspectives to performance

• various overlapping perspectives to the issue: uncertainty assessment, quality assurance, model development guidelines, ...
• all perspectives provide only limited or narrow views to performance
  • use purpose of information often not considered
  • all aspects of performance not covered
  • evaluation as a separate process, often only after assessment
  • societal aspect often neglected
  • focus either on giving assessment procedure guidelines or considering product as such

Properties of good assessments

• multi-perspective approach to performance
  • effectiveness
    • quality of content
    • applicability
  • efficiency
• an integral part of assessment process → applicable also in design and execution (a priori), not only evaluation (a posteriori)

• Relation of properties to information structure/content

• Evaluation process
  • a priori and/or a posteriori view
  • identification of purpose
  • evaluation of quality of content (uncertainty + relevance)
    • in principle reality, but in practice golden standard as reference point D↷
  • evaluation of applicability
  • evaluation of efficiency (effort expenditure)
  • overall performance
    • potential for effectiveness/effort given purpose
      • can be further evaluated retrospectively against realized effectiveness (possibly against redefined purpose)

Discussion

• focus on the mediator of the overall intellectual process - the shared object of activity
  • product|use purpose, assessment process|product
  • trialogical process, knowledge creation metaphor
  • not only collection and use of existing information or learning to deal with new situation, but also creation of new knowledge
• capable of incorporating the goods from other perspectives into a more comprehensive and coherent approach
• evaluation of performance can only be done meaningfully against purpose
  • various intentions need to be identified, explicated and prioritized in all assessments

• data about hypothesis

Conclusions

• There is more to assessment performance than just statistical uncertainty and data source reliability
• Overall performance of assessment can be evaluated systematically and explicitly
  • requires consistent information structure
  • a priori evaluation should be made an inherent part of assessment process