Difference between revisions of "Assessments - science-based decision support"

Revision as of 08:13, 17 February 2009

Assessments - science-based decision support is a lecture about assessments as processes and their products that on the one hand are determined by the practical needs of societal decision making (policy), and on the other hand are bounded by the scientific quest for truth. The lecture, consisting of two parts, discusses how these sometimes apparently conflicting goals influence the making and outcomes of assessments.

Scope

Purpose: To describe how the practical needs of societal decision making are taken account of in assessment (design and execution), and how these needs are addressed by means of science.

Intended audience: Researchers (especially at doctoral student level) in any field of science (mainly natural, not social scientists).

Duration: 1.5 h + 1.5 h

Definition

The lecture is divided into two parts. First part discusses what assessments are about and what purpose do they serve. The second part then discusses what can be done in assessments and how, in order to serve the purpose. The second part is built on the assumption that in between parts I and II, the recipient of this lecture becomes acquainted with the concepts variable and collaboration, in the way they are defined in the context of open assessment.

In order to fully understand this lecture it is recommended to acquaint oneself also with the following lectures:

• Open assessment in research
• Variables - evolving interpretations of reality
• Science necessitates collaboration
• Evaluating assessment performance

Objectives, part I:

• learn the basics of an assessment: question/info need, and the three-part structure: decision - factors - outcome.
• learn how an information need is transformed into a research question.
• learn the importance and implications of boundaries in defining an object (assessment, variable)
• learn the roles of participants and intended users.
• learn to participate in discussions about a new assessment (Tapas) and its framing.

Objectives, part II:

• become familiar with the information structure of an assessment object
• learn how solutions to assessment problems are found (sought for)
• learn how the assessment solution addresses the need and how need influences the solution
• learn how an assessment consists of a network of individual variables (independent, given causal relations)
• learn how the information content of an assessment can be analyzed, organized and interpreted for producing the results

Result

Part I: practical need determines the assessment problem

See presentation: File:Assessments -science-based decision support.ppt

Introduction (15 min):

• Assessment is a business of creating understanding about reality in relation to a specific need
  • collection, synthesis, interpretation, organization, communication, use of information
  • driven by practical needs, constrained by the scientific quest for truth (yet also facilitated with the means and methods of science)
• Different views to assessment:
  • practical need vs. science
  • process - product - use

Example: TAPAS - Transportation, air pollution and physical activities: An integrated health risk assessment programme of climate change and urban policies. (intro 5 min, directed discussion 30 min, summary of discussion 10 min)

• urban policies (interventions)
• active transportation
• climate change
• other health related outcomes
• risks: air pollution, UV, crime, accidents
• benefits: physical activity, social interaction, mental health
• summarizing effects: DALYs
• Assessment
  • active transport policy decisions
  • Indicators: GHG emissions, ambient air quality, mental health/quality of life, attributable chronic/acute disese, net health (DALYs)

• what need is addressed?
• whose need/concern is it?
• who is interested in/affected by the issue?
• who is responsible for the state of affairs or taking action?
• who can take action?
• what actions?
• is someone affected by possible actions about them?
• what needs to be known in order to take action (right action)?
• are there conflicts of interest or differences of perception regarding the needs/concerns or actions about them?
• whose understanding about the issue is most crucial to be increased (e.g. scientists, policy-makers, industrial managers, specific stakeholder groups, common citizens)?
• what are the issues of interest that we need to know more about?

→ determination of assessment purpose, boundaries and intended users

• can the assessment problem be formulated into a form of a question or a set of questions
• TAPAS diagram

Practical need → assessment problem (15 min):

• information gap drives the assessment
  • need to know, but required knowledge/information is missing
  • need to make decision (action/no action, what action) → need to choose among decision options → need to know available decision options → need to know preferability of decision options
• Various decision makers: e.g. political leaders, industrial managers, individuals
• Various possible decisions/actions: e.g. policy, investment, consumption
• → purpose of assessment is to satisfy the information gap of intended users of the assessment
  • What is required in order to know what should be done in order to improve the state of issue of interest?
• → → the information gap needs to be indicated/identified, analyzed and understood
• → → → the information gap determines the boundaries for what needs to be assessed
• purpose, users and boundaries = assessment problem ( or = scope, see Talk:Assessment)
  • what is needed to know, why, who will use the knowledge (why will they acquire the knowledge)?
• purpose & users give practical requirements for the assessment
• boundaries define the part of reality which to describe

Looking for solution (15 min):

• What needs to done in order to address the problem?
• Decision → factor → outcome
• Indicators: issues of specific interest (can be of any type: outcomes, factors or decisions - typically outcomes)
• Assessment is about finding solutions to the assessment problem
  • probably also at least as much it is finding out what is not known in order to solve the problem (consider knowledge management vs. ignorance management)
  • → assessments are done according to need, not according to e.g. coincidental data availability

Part II: providing solutions to the assessment problem

concepts:

• information structure of assessment objects
  • scope - need → problem
    • purpose, boundaries, users, scenarios, participants
  • definition - means of providing a tentative solution to the problem
    • decision variables, indicators, other variables, indices, analyses
  • result - the tentative solution to the problem
    • results (of indicators and analyses), conclusions
• assessment is a collection of variables
  • decisions, factors and outcomes in a causal network
  • indices: operationalizations of dimensions consodered
  • analyses (examples)
    • decision optimization (which decision results in best outcome)
    • other conditioning (intentional deviations from best estimate for any variable result)
      • scenarios are conditionings that are particularly important for the use purpose (thus defined in scope)
    • value of information analysis (where is more understanding most useful?)
    • importance and sensitivity analysis (what factors / change in factors most affect the outcome?)
  • appraisal (by what means are value judgements incorporated in the assessment)
• assessments as answers to practical questions by means of science
  • the real-world phenomena defined in boundaries need to be described as truthlikely as possible to the extent and detail as relevant according to the use purpose
  • the information in the description should be organized and communicated according to the determined scope
  • practical need provides the questions, science constrains and facilitates how they answered, practical need again determines how the answered are to be conveyed/converged to use