Difference between revisions of "Open assessment workshop 2009"

From Testiwiki
Jump to: navigation, search
(Daily program: Planned contents, objectives and related material: friday exercises)
(Added category: 'Open assessment training')
 
(30 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
{{encyclopedia}}
 
{{encyclopedia}}
'''Open assessment workshop 2009''' is the third workshop about new methods in assessments (previously more narrowly on risk assessment), organised by the National Institute for Health and Welfare, (THL), Kuopio, Finland. Its main purpose is to provide the workshop participants a broad perspective to assessments, promote the idea and practice of [[Open assessment|open assessment]], as well as further develop methods to perform open assessments. The workshop has started following the needs of several EU-funded research projects, notably [http://www.intarese.org Intarese], [http://www.beneris.eu Beneris], and [http://www.heimtsa.eu Heimtsa]. The workshop is mainly targeted to postgraduate students working in the fields of environmental health or risk assessment, but it is also open to anyone interested. If you want to join the workshop, please create a user account (by following the log in/create account link in the right-top corner of the page) with your contact information and add your username to the list on the bottom of this page. There is no deadline for registration nor a registration fee for the workshop, but early registration is highly recommended in order to help both the organizers and the participants to arrange the facilities, accommodation, programs etc.  
+
 
 +
'''Open assessment workshop 2009''' is the third workshop about new methods in assessments (previously more narrowly on risk assessment), organised by the National Institute for Health and Welfare, (THL), Kuopio, Finland. Its main purpose is to provide the workshop participants a broad perspective to assessments, promote the idea and practice of [[Open assessment|open assessment]], as well as further develop methods to perform open assessments. The workshop has started following the needs of several EU-funded research projects, notably [http://www.intarese.org Intarese], [http://www.beneris.eu Beneris], and [http://www.heimtsa.eu Heimtsa]. The workshop is mainly targeted to postgraduate students working in the fields of environmental health or risk assessment, but it is also open to anyone interested. If you want to join the workshop, please create a user account (by following the log in/create account link in the right-top corner of the page) with your contact information and add your username to the list on the bottom of this page. There is no deadline for registration nor a registration fee for the workshop, but early registration is highly recommended in order to help both the organizers and the participants to arrange the facilities, accommodation, programs etc.
 +
 
 +
The workshop was successfully carried out. Read the report based on formal and informal feedback collected from the participants as well as organizers own reflections on the [[Talk:Open assessment workshop 2009|discussion page]].
  
 
== General information  ==
 
== General information  ==
Line 79: Line 82:
 
! Friday, Tietoteknia 1038
 
! Friday, Tietoteknia 1038
 
|-
 
|-
| Introduction to workshop  
+
| '''9:00-9:15''' Introduction to workshop  
| Lecture 3: Variables - interpretations of real-world phenomena, part I
+
| '''9:00-10:15''' Lecture 3: [[Variables - evolving interpretations of reality|Variables - interpretations of real-world phenomena]], part I
| Lecture 5: Science necessitates collaboration, part I  
+
| '''9:00-10:15''' Lecture 5: [[Science necessitates collaboration]], part I  
| Lecture 7: [[Assessments - science-based decision support]], part II
+
| '''8:30-10:00''' Lecture 7: [[Assessments - science-based decision support]], part II
| Lecture 8: [[Evaluating assessment performance]]
+
| '''8:30-10:00''' Lecture 8: [[Evaluating assessment performance]]
 
|-
 
|-
| Lecture 1: [[Open assessment in research]]  
+
| '''9:15-11:45''' Lecture 1: [[Open assessment in research]]  
| Lecture 4: Variables - interpretations of real-world phenomena, part II
+
| '''10:30-11:45''' Lecture 4: [[Variables - evolving interpretations of reality|Variables - interpretations of real-world phenomena]], part II
| Lecture 6: Science necessitates collaboration, part II
+
| '''10:30-11:45''' Lecture 6: [[Science necessitates collaboration]], part II
| Social activities: Skating on lake Kallavesi  
+
| '''10:00-''' Social activities: Skating on lake Kallavesi  
| Group work: Assessment performance evaluation
+
| '''10:00-11:45''' Group work: Assessment performance evaluation
 
|-
 
|-
| ''Lunch  
+
| '''11:45-12:45''' ''Lunch  
| ''Lunch  
+
| '''11:45-12:45''' ''Lunch  
| ''Lunch  
+
| '''11:45-12:45''' ''Lunch  
| Social activities: Skating on lake Kallavesi  
+
| '''-13:00''' Social activities: Skating on lake Kallavesi  
| ''Lunch
+
| '''11:45-12:30''' Lunch
 
|-
 
|-
| Lecture 2: [[Assessments - science-based decision support]], part I
+
| '''12:45-14:15''' Lecture 2: [[Assessments - science-based decision support]], part I
| Group work: Developing variables  
+
| '''12:45-''' Group work: Developing variables  
| Group work: Developing variables in collaboration  
+
| '''12:45-''' Group work: Developing variables in collaboration  
| ''Lunch  
+
| '''13:00-14:00''' ''Lunch  
| Group work presentations
+
| '''12:30-13:45''' Group work presentations
 
|-
 
|-
| Introduction to case study
+
| '''14:45-16:15''' Introduction to case study
| Group work: Developing variables  
+
| '''-15:00''' Group work: Developing variables  
| Group work: Developing variables in collaboration  
+
| '''-15:00''' Group work: Developing variables in collaboration  
| Group work: Assessment development  
+
| '''14:00-15:30''' Group work: Assessment development  
| Workshop summary and feedback
+
| '''13:45-14:00''' Workshop summary and feedback
 
|-
 
|-
| Practical arrangements  
+
| '''16:15-16:45''' Practical arrangements  
| Group work presentations  
+
| '''15:00-16:45''' Group work presentations  
| Group work presentations  
+
| '''15:00-16:45''' Group work presentations  
| Group work: Assessment development
+
| '''15:30-16:45''' Group work presentations
 
|  
 
|  
 
|-
 
|-
 
|  
 
|  
| Social activities: [http://www.rauhalahti.fi/en/ravintola_ja_tanssi/jatkankamppa_saunaillat.php dinner and smoke sauna]  
+
| '''18:00-''' Social activities: [http://www.rauhalahti.fi/en/ravintola_ja_tanssi/jatkankamppa_saunaillat.php dinner and smoke sauna]  
| Social activities: skiing, snowboarding, sledding on [[http://personal.inet.fi/yritys/puijonrinteet/ Puijo]]  
+
| '''18:00-''' Social activities: skiing, snowboarding, sledding on [[http://personal.inet.fi/yritys/puijonrinteet/ Puijo]]  
| Group work presentations
+
|  
 
|  
 
|  
 
|}
 
|}
Line 144: Line 147:
 
*understanding conveyed to other groups through group work presentations
 
*understanding conveyed to other groups through group work presentations
  
===Daily program: Planned contents, objectives and related material===
+
Follow the links to lecture pages in the schedule above also to see the specific exercise tasks.
 
 
====Monday: Introduction, motivation, and basic concepts====
 
 
 
Morning lecture: '''[[Open assessment in research]]
 
 
 
Objectives:
 
* learn how traditional articles have two distinct parts
 
* Learn how these parts can be organised in a better way
 
* Become exposed to the idea of scientific method.
 
* Identify what is the use of science in policy assessments.
 
* Learn, for a piece of information, you know what is a good object type for it.
 
* Learn to see the world as a collection of information pieces.
 
* See your own work/research as a part of a global mass collaboration project.
 
* Learn that it is possible to do the whole research process (idea - research plan - execution of a study - writing articles) in Opasnet.
 
* Learn that the impact of you research may be higher in an open system.
 
* See how Opasnet can be used in a practical case study (assessment, research).
 
 
 
Opasnet pages related to Monday morning topics:
 
*[[Study]]
 
*[[Assessment]]
 
*[[Variable]]
 
*[[Writing a scientific article in Mediawiki]]
 
*[[Acknowledgements]]
 
*[[Peer review]]
 
 
 
Afternoon lecture: '''[[Assessments - science-based decision support]], part I
 
 
 
Objectives:
 
* Learn the basics of an assessment: question/info need, and the three-part structure: decision - factors - outcome.
 
* Learn how an information need is transferred into a research question.
 
* Learn the importance and implications of boundaries in defining an object (assessment, variable)
 
* Learn the roles of participants and intended users.
 
* Learn to participate in discussions about a new assessment (Tapas) and its framing.
 
 
 
Introduction to case assessment and exercises:
 
* Societal decision making constrained by GHG emission reduction goals - optimization of health impacts and costs
 
** international climate policy → city-level decision making → individual adaptation
 
* Monday: scope of assessment - what, why, for whom? → from need to assessment problem
 
* Tuesday: variables - individual parts of assessments
 
* Wednesday: mass collaboration in practice - web-based guided formal argumentation
 
* Thursday: definition of assessment - how and why? → suggested solution to assessment problem
 
* Friday: what is a good assessment (in terms of science/practical need) - quality of content, applicability, efficiency → core product, packaging, logistics and marketing, production process
 
 
 
Objectives:
 
* learn about the outline of the workshop week
 
* know the basics of the case study
 
* know objectives of each session
 
* identify which exercise groups are most interesting for you → group formation
 
 
 
Opasnet pages related to Monday afternoon topics:
 
* [[Performing an assessment]]
 
* [[Defining the purpose of an assessment]]
 
* [[Defining the users of an assessment]]
 
* [[Issue framing]]
 
* [[Scoping an assessment]]
 
 
 
====Tuesday: variable development====
 
 
 
Lectures: '''Variables - evolving interpretations of real-world phenomena'''
 
 
 
Objectives:
 
* Learn the basic structure of an information object.
 
* Learn what variables are: independent, eternally developing descriptions of reality.
 
* Learn that the ultimate criterion of a variable is the truth.
 
* Learn the differences and similarities of a variable, assessment, study, and encyclopedia article.
 
* Learn how to use the variable definition as your information storage.
 
* Learn how to make a new synthesis when new information comes up (in Bayesian language: to update your prior).
 
* Learn how to handle variables that are partially or totally overlapping.
 
 
 
Exercises:
 
{| class="FCK__ShowTableBorders" {{prettytable}}
 
|-
 
! Variable
 
! Task
 
! Examples of available data sources
 
|-
 
| [[PM2.5 exposure-response function]]
 
| Developing the contents of the variable. Acquaint yourself with the variable concept and the structure of a variable. With the purpose of the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]] in mind, first determine the scope of the variable and then consequently definition and result based on both the provided existing variables as data sources and your own expertise. Specific things to consider: a) how to synthesize data from different sources? b) what additional data sources could be used to improve the variable? c)why could not either of the available existing variables be used instead of creating a new variable? d) what do you think about the quality of your variable (as the best estimate for the phenomenon it describes)? Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable.
 
| Existing variables [[Concentration-response to PM2.5]] and [[ERF of PM2.5 on mortality in general population]]
 
|-
 
| [[Emission factors for burning processes]]
 
| Developing the contents of the variable. Acquaint yourself with the variable concept and the structure of a variable and consider the role of this variable in the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]]. Dig out the necessary pieces of information from the provided FIRE database according to the needs of the assessment. Needed indexes and locations include:
 
* Fuel: coal (assumed to be all anthracite), natural gas (assumed to be all buthane), residual oil (assumed to be all grade 6), distillate oil (assumed to be all grade 4), process gas, landfill gas
 
* Burning processes: pulverized (coal), tangentially fired (natural gas), normal firing (oils),
 
* Emissions: CO<sub>2</sub>, PM<sub>2.5</sub>
 
* Emission control mechanism: <50 MW nothing, 50-100 MW multi-cyclone, 100-300 MW electrical precipitator + scrubber, >300 MW multi-cyclone + fiber filter
 
Having done the above, consider possible other data sources that could be used to improve the variable, e.g. in terms of making the variable more suitable also for other assessments. Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable.
 
|
 
* Emission factor data from U.S.EPA: [http://www.epa.gov/ttn/chief/software/fire/index.html FIRE database] (assume external combustion and electric generation for all fuels)
 
* List of heat production installations in Helsinki metropolitan area
 
|-
 
| [[Population of Helsinki metropolitan area]]
 
| Developing the contents of the variable. With the purpose of the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]] in mind, first determine the scope of the variable and then consequently consider definition and result. Check the available population data from different sources, consider their differences and make a synthesis out of them. Specific things to consider include:
 
* How and why include different data sources in a variable (what is the difference between a variable and data)?
 
* How to operationalize the synthesis (data &rarr; formula &rarr; result)?
 
* How can coherence be obtained/retained with other fully or partly overlapping variables (e.g. population of Finland, population of Europe).
 
Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable.
 
|
 
* [http://dataservice.eea.europa.eu/dataservice/metadetails.asp?id=1018 EEA population data]
 
* [[:en:Greater Helsinki]]
 
* Analytica node based on WHO population data (provided by organizers)
 
|-
 
| [[DALY weights]]
 
| Developing the contents of the variable. Acquaint yourself with [[DALY]] concept, in particular regarding how the weights are estimated. Consider how the existing variable could be improved by including uncertainties in the weight estimates, e.g. making a shift from point estimates to distributions. Improve the variable accordingly if possible. Prepare to present your variable to other groups and explain DALY weights, their role in estimating DALYs, and how including uncertainty in DALY weights affects DALY estimates.
 
| [http://www.who.int/topics/global_burden_of_disease/en/ WHO Global Burden of Disease website]
 
|-
 
| [[Cardiopulmonary mortality in Helsinki metropolitan area]]
 
| Developing the contents of the variable. With the purpose of the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]] in mind, first determine the scope of the variable and then consequently consider definition and result. Search for the needed information from [http://base.opasnet.org/index.php Opasnet base] and apply it in developing the variable.
 
|
 
* An existing variable [[Morbidity data for Europe]]
 
* [http://base.opasnet.org/index.php Opasnet base]
 
|}
 
 
 
Objectives:
 
* Learn to build the definition based on the scope, and compute the result based on the definition.
 
* Practical experience about topics you learned in the morning.
 
 
 
Opasnet pages related to Tuesday topics:
 
*[[Designing variables]]
 
*[[Plausibility test]]
 
*[[Applying general information]]
 
*[[Result domain]]
 
*[[Executing variables]]
 
*[[Calculating health impacts]]
 
*[[Life table]]
 
 
 
====Wednesday: variable development in collaboration====
 
 
 
Lectures: '''Science necessitates collaboration'''
 
 
 
Objectves:
 
* Learn why collaboration is better than working alone.
 
* Learn to apply the scientific method, and the importance of it in open assessment.
 
* Learn the structure of formal argumentation.
 
* Learn how to collaborate
 
 
 
Exercises (all groups):
 
# Choose (as a group) a variable that another group developed on Tuesday (see list of variables in the table above). Scrutinize the variable carefully and, according to your expertise and understanding, create statements and arguments about the variable content. Document the statements as well as the arguments for against your group comes up with on the discussion page of the variable following the formal argumentation structure. It is also possible to first collect comments informally, analyze the comments and then reorganize them in the argumentation format. Consider and argue in particular:
 
#* Is the scope unambiguously defined (what is the question the variable attempts to answer to)?
 
#* Does the definition provide the sufficient means to answer the question adequately?
 
#* Is the result an exact and explicit logical outcome of the definition?
 
#* Does the result appear reasonable and meaningful?
 
# Go back to the variable you (as a group) developed on Tuesday and scrutinize the argumentation that another group has created about the contents of your variable. React to the argumentation by counter arguing or by improving the variable contents as seen necessary.
 
# Prepare a short presentation about the argumentation about your group's variable description invoked and how it was or should be taken into account in further development and use of that variable.
 
 
 
Objectives:
 
* Learn to use formal argumentation and to turn existing discussions into this formalism.
 
 
 
Opasnet pages related Wednesday topics:
 
*[[Open participation]]
 
*[[Evaluating scientific claims]]
 
*[[Dealing with disputes]]
 
*[[Discussion]]
 
*[[Help:Argumentation]]
 
*[[Index conversion function]]
 
*[[Merging models with different grids]]
 
*[[Synthesising information about continuous variables]]
 
*[[Using non-predetermined spatial disaggregation]]
 
 
 
====Thursday: assessment development ====
 
 
 
Lecture: '''[[Assessments - science-based decision support]], part II
 
 
 
Objectives:
 
* become familiar with the information structure of an assessment object
 
* learn how solutions to assessment problems are found (sought for), the parts of assessment
 
* learn how the assessment solution addresses the need and how need influences the solution
 
* learn how an assessment consists of a network of individual (independent given causal relations) variables
 
* learn how the information content of an assessment can be analyzed, organized and interpreted for producing the results
 
 
 
Exercises:
 
 
 
{| class="FCK__ShowTableBorders" {{prettytable}}
 
|-
 
! Exercise topic
 
! Task
 
|-
 
| Meeting the needs of intended use (primary: [http://www.wwviews.org/ WWViews], [http://en.cop15.dk/ COP-15])
 
| Consider the intended use, its needs and the [[Assessment on impacts of emission trading on city-level (ET-CL)#Scope|scope of the assessment]] e.g. in light of the following questions:
 
* Is the assessment addressing right issues and asking the right questions (i.e. is the scope sufficiently defined)?
 
* Do you think it has a potential to influence the intended use process (assuming that the assessment succeeds in providing answers its questions adequately)?
 
* How should the information be wrapped, organized and communicated to the intended users in order to realize the potential effect (again, assuming that the truth-value of the content is adequate)?
 
* What aspects of the information content should be emphasized in communication?
 
Document your statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the identified need and how it is addressed.
 
|-
 
| Answering the assessment questions - variable network
 
| This task is about developing the ''definition'' of the [[Assessment on impacts of emission trading on city-level (ET-CL)#Definition|case assessment]] in terms of the variables belonging to the assessment. Explore the list of variables, their types (decision, indicator, other) and the causal diagram representation of the assessment as well as the Analytica model. Given the scope of the assessment, consider e.g. following questions:
 
* What variables (descriptions of real-world phenomena) are required to be included in the assessment in order to adequately provide answer the questions (scope) set for the assessment (assuming that the variables are well defined)?
 
* Is the given list of variables and the causal diagram/model sufficient in order to answer the questions?
 
* Are the variable list and causal diagram/model consistent with each other (variables vs. model nodes)?
 
Document statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the causal network of variables and its relation to the defined scope of the assessment.
 
|-
 
| Answering the assessment questions - BBN analysis
 
| Acquaint yourself with Bayesian belief networks and consider how they can be used in assessments. In particular look into:
 
* What is conditioning?
 
* What are the similarities and differences between causal diagrams and Bayesian nets?
 
* How can Bayesian nets be used for optimizing?
 
Analyze a simple Uninet model, provided to you by the workshop organizers, covering a part of the case assessment. By conditioning different variables try to find the optimal decision option. You can also update the model with new variable results found in Opasnet and repeat the analysis. You can document statements, arguments and comments on, or make amendments to, related pages in the Opasnet if relevant. Present your findings to other groups including a brief introductory explanation of how Bayesian nets can be used in the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]].
 
|-
 
| Answering the assessment questions - Value of information (VOI) analysis
 
| Acquaint yourself with the Value of information method (VOI). In particular look into:
 
* What does VOI measure?
 
* What is VOI used for?
 
* What information must one have (necessary conditions) in order to calculate VOI?
 
Analyze a simple Analytica model, provided to you by the workshop organizers, covering a part of the case assessment. Calculate VOI for different variables in the model. You can also update the model with new variable results found in Opasnet and repeat the analysis. Present your findings to other groups including a brief introductory explanation of how VOI can be used in the [[Assessment on impacts of emission trading on city-level (ET-CL)|case assessment]].
 
|-
 
| Scenarios
 
| Acquaint yourself with the way the concept [[Scenario]] is defined in the context of open assessment. Then consider the scenarios set for the [[Assessment on impacts of emission trading on city-level (ET-CL)#Scenarios| case assessment]] e.g. in light of the following questions:
 
* What is the conceptual difference between the base-case and the scenarios?
 
* In what ways do the decision scenarios (plausible EU-ETS alterations) differ from the IPCC "scenarios", referred to as "future pathways"?
 
* In what ways do the decision scenarios (plausible EU-ETS alterations) from the other [[Assessment on impacts of emission trading on city-level (ET-CL)#Decision variables|decision options]] assessed?
 
* Are the set EU-ETS scenarios (including the base-case) well-chosen and well-defined?
 
* How should the scenario set in the case assessment be improved?
 
Document statements, arguments and comments on the discussion page, and present your findings to other groups.
 
|}
 
 
 
Opasnet pages related to related Thursday exercise:
 
*[[Drawing a causal diagram]]
 
*[[Improving the result of a variable]]
 
*[[Bayesian network]]
 
*[[Value of information]]
 
*[[Reporting an assessment]]
 
*[[Operationalising large structural changes to existing systems]]
 
*[[Variable transfer protocol]]
 
 
 
====Friday: evaluation of assessment performance====
 
 
 
Lecture: '''[[Evaluating assessment performance]]'''
 
 
 
Exercises:
 
 
 
 
 
{|{{prettytable}}
 
! Topic
 
! Task description
 
|-----
 
| Quality of content
 
| Consider the concepts informativeness, calibration and relevance. What is their point of reference, practical need or truth? Which attributes and sub-attributes of assessments and variables they relate to ?Now choose one variable that was developed in Tuesday and Wednesday exercises. What can you say about their informativeness, calibration and relevance? Then consider the case assessment (ET-CL). What can you say about its informativeness, calibration and relevance? What implications this gives about future work regarding the variable and the assessment? Give suggestions for improving their quality of content. Prepare a presentation including a brief introduction to the concepts, evaluation of the quality of content of the variable and the assessment, and suggestions for improving the quality of content, as well as reasoning to back-up your suggestions.
 
|-----
 
| Usability and availability
 
| Consider the concepts usability and availability. What is the reference point they relate to, practical need or truth? Assuming that the information content of a variable or an assessment were complete, correct and exact, what would make it usable and available or not usable or available for someone who would need that particular chunk of information? Now choose one variable that was developed in Tuesday and Wednesday exercises. What can you say about its usability and availability given the purpose and users defined in the scope of the case assessment (ET-CL)? Then consider the case assessment. What can you say about its usability and availability given the purpose and users defined in the scope. What implications this gives about future work regarding the variable and the assessment? Give suggestions for improving their usability and availability. Prepare a presentation including a brief presentation of the concept, why is it an important aspect of assessment performance, evaluation of the usability and availability of the variable and the assessment, and suggestions for improving the usability and availability, as well as reasoning to back-up your suggestions.
 
|-----
 
| Acceptability
 
| Consider the concept acceptability and its disaggregation to acceptability of premises and acceptability of process. Which attributes and sub-attributes of variables and assessments does acceptability relate to? Now choose one variable that was developed in Tuesday and Wednesday exercises. What can you say about its acceptability given the scope of the variable? Then consider the case assessment. What can you say about its acceptability given the scope of the assessment. What aspects are most crucial regarding the acceptability of the assessment? Give suggestions for improving the acceptability of the assessment. Prepare a presentation including a brief introduction to the concept and why it is an important aspect of assessment performance, an evaluation of the acceptability of the variable and the assessment, and suggestions for improving the acceptability, as well as reasoning to back-up your suggestions.
 
|-----
 
| Efficiency
 
| Consider the concept efficiency. How is efficiency dependent on quality of content and applicability? What else influences efficiency of assessments? Now explore the case assessment, how it is designed and being executed, and think about its efficiency. If you were responsible for making the assessment and provided with scarce resources, what would you do in order to  get the best outcome? How about if you were responsible for making a somewhat related an partially overlapping assessment, e.g. [[Bioher]], [[Claih]], TAPAS, some other, and again with scarce resources, what would you tell the manager of ET-CL in order to maximize the efficiency of your assessment. How about if you were responsible for a group of overlapping assessments, say ET-CL, [[Claih]] and [[Bioher]] for example, would it change your management decisions? Prepare a presentation including a brief introduction to the concept and why it is an important aspect of assessment performance, an explanation of how you would as an assessment manager would maximize the efficiency in the above mentioned different situations, as well as reasoning to back-up your hypothetical management decisions.
 
|-----
 
| Peer review
 
| Acquaint yourself with how peer review is defined in the context of open assessment. Consider its similarities with and deviations from your prior perception of what peer review is. Which attributes of variables and assessment does peer review (in OA) address and how? Which properties of good assessments does peer review relate to? Take the position of a peer reviewer and review one variable and the case assessment. Would you accept? Prepare a presentation including a brief introduction to the method and how it is used in evaluating assessment performance
 
|}
 
 
 
Opasnet pages related to Friday exercise:
 
*[[Performance]]
 
*[[Acknowledgements]]
 
*[[Peer review]]
 
*[[Quality assurance and quality control]]
 
*[[Quality evaluation criteria]]
 
  
 
===Practicalities===
 
===Practicalities===
  
 
'''Facilities'''  
 
'''Facilities'''  
 
'''NOTE: Lecture and exercise location has changed!!!
 
  
 
The main activities throughout the workshop week take place at the seminar halls 1038 and 1039 in the Tietoteknia building (see the map in the beginning of the page to check the location). There will be a wireless internet connection available for all participants. Each participant is recommended to bring along a laptop computer. If this is not possible for you, [[User:Mikko Pohjola|please inform the organizers about this]] and we will see what we can do about it.  
 
The main activities throughout the workshop week take place at the seminar halls 1038 and 1039 in the Tietoteknia building (see the map in the beginning of the page to check the location). There will be a wireless internet connection available for all participants. Each participant is recommended to bring along a laptop computer. If this is not possible for you, [[User:Mikko Pohjola|please inform the organizers about this]] and we will see what we can do about it.  
Line 417: Line 160:
 
If there should be any people who would like to participate in the workshop, but cannot make it to Kuopio to attend the workshop in person, we try to organise a web-conference for the lecture sessions using [http://www.skype.com Skype]. The group work will be worked on using the [[Main Page|Opasnet website]], so also outsiders can relatively easily take part in that as well. It is also possible to join a group remotely and communicate with group members through Skype. This way you may also participate from wherever you are, as most major things - except the social program - will be made available via the Internet. Add your name on the list of remote ''via internet'' participants at the end of this page if you want to follow the workshop, but are unable to attend in person.
 
If there should be any people who would like to participate in the workshop, but cannot make it to Kuopio to attend the workshop in person, we try to organise a web-conference for the lecture sessions using [http://www.skype.com Skype]. The group work will be worked on using the [[Main Page|Opasnet website]], so also outsiders can relatively easily take part in that as well. It is also possible to join a group remotely and communicate with group members through Skype. This way you may also participate from wherever you are, as most major things - except the social program - will be made available via the Internet. Add your name on the list of remote ''via internet'' participants at the end of this page if you want to follow the workshop, but are unable to attend in person.
  
'''Note!''' Our Skype username is: open.risk <br>
+
Instructions for remote participation to lectures:
You can add that to your Skype contacts preferably already before the workshop
+
* '''NOTE: the lectures, including the Skype conferencing, are recorded'''
 +
* Call username open.risk on Skype (you can add this user to your Skype contacts preferably already before the workshop) a before the beginning of a lecture.
 +
* Keep your microphone on mute always when not making questions
 +
* When you want to make a question or comment, type a message to user open.risk and the lecturer will give you an opportunity to speak out your question (remember to turn on your microphone)
 +
* You can also send your question as a message and have the Skype-cast director read it out for you
 +
* The lecture videos will be put available online as soon as possible
 +
* Follow the links to lecture pages (see above) to get access to the lecture material
  
More information regarding how to participate remotely in practice will be provided here before beginning of the workshop.
+
Instructions for remote participation to exercises:
 +
* Pick a group (exercise) you want to join and indicate this to workshop organizers e.g. by calling or messaging Skype user open.risk
 +
* The organizers will find out which Skype user to contact in order to join the group discussions
 +
* Participate in the work through the Opasnet pages related to the exercises assigned
  
 
== Social activities  ==
 
== Social activities  ==
Line 509: Line 261:
 
<br>''Participation via the Internet:''  
 
<br>''Participation via the Internet:''  
  
*Katy Walker
 
 
*[[User:Alexandra Kuhn|Alex Kuhn]]  
 
*[[User:Alexandra Kuhn|Alex Kuhn]]  
*Tam T Pham
 
*Adeyanju kazeem Alade, Scientific Computing Student, University of kuopio.
 
*Aruya Omoarebun Richard, Msc Scientific Computing, University of Kuopio.
 
  
== Useful links while preparing for the workshop  ==
+
== Useful background material ==
  
 
*[[Glossary|Glossary]] is a collection of important concepts and terms and their explanations  
 
*[[Glossary|Glossary]] is a collection of important concepts and terms and their explanations  
Line 541: Line 289:
  
 
Keep an eye on this page, this is the official website of the workshop. It will be updated as new information comes available.  
 
Keep an eye on this page, this is the official website of the workshop. It will be updated as new information comes available.  
 +
 +
==Lecture videos==
 +
 +
Videos from 2009 now available. Sorry about the quality and format...
 +
 +
{{mfiles}}
  
 
[[Category:Events]] [[Category:Open_assessment]]
 
[[Category:Events]] [[Category:Open_assessment]]
 +
[[Category:Open assessment training]]

Latest revision as of 12:49, 28 March 2011


Open assessment workshop 2009 is the third workshop about new methods in assessments (previously more narrowly on risk assessment), organised by the National Institute for Health and Welfare, (THL), Kuopio, Finland. Its main purpose is to provide the workshop participants a broad perspective to assessments, promote the idea and practice of open assessment, as well as further develop methods to perform open assessments. The workshop has started following the needs of several EU-funded research projects, notably Intarese, Beneris, and Heimtsa. The workshop is mainly targeted to postgraduate students working in the fields of environmental health or risk assessment, but it is also open to anyone interested. If you want to join the workshop, please create a user account (by following the log in/create account link in the right-top corner of the page) with your contact information and add your username to the list on the bottom of this page. There is no deadline for registration nor a registration fee for the workshop, but early registration is highly recommended in order to help both the organizers and the participants to arrange the facilities, accommodation, programs etc.

The workshop was successfully carried out. Read the report based on formal and informal feedback collected from the participants as well as organizers own reflections on the discussion page.

General information

Time: 16.2. - 20.2.2009
Place: THL, Kuopio, Finland (Kuopio info)
Organizer: National Institute for Health and Welfare (THL), Department of Environmental Health

See the map below to check the locations of:

  • Hotel Savonia - accommodation
  • Tietoteknia - lectures and exercices
  • Jätkänkämppä - smoke sauna, ice swimming, traditional dinner
  • Antikkala - cross-country and downhill skiing, sledding
  • Kuopio passenger harbour - Skating tracks
  • Kuopio railway station
  • Kuopio airport
  • Kauppahalli (market hall at market square) - bus stop for line 5 from city center to Hotel Savonia
  • National Institute for Health and Welfare, (THL), Department of Environmental Health

You can zoom into the map and by clicking the red pins on the map, an information box indicating the name of the location will appear.

<googlemap width="375" zoom="11" type="map" lon="27.638855" lat="62.903976"> m62.893334, 27.679328, Kuopio, Finland 62.890603, 27.627397, THL 62.866252, 27.648468, Jätkänkämppä smoke sauna 62.890240, 27.635560, Tietoteknia - Lectures and exercises 62.901100, 27.651600, Hotel Savonia - Workshop accommodation 62.891659, 27.678895, Kauppahalli, bus line 5 to Hotel Savonia 63.014833, 27.792149, Kuopio airport 62.895882, 27.679882, Kuopio railway station 62.921720, 27.662373, Antikkala - ski slope, ski tracks and sledding slope 62.886549, 27.697393, Skating tracks </googlemap>


5-day weather forecast for Kuopio by the Finnish Meteorological Institute.

5-day weather forecast for Kuopio (in Finnish) by Foreca.

15-day forecast for Kuopio by Accuweather

Description

The aim of the workshop is to provide the participants a good understanding of how to make the practical information needs of societal action and creation of scientific knowledge meet by making use of the latest methodological developments and modern information and communication technology. This aim is pursued in the workshop lectures and exercises by discussing and considering e.g following questions:

  • How to take the practical needs of policy into account in assessments?
  • How can scientific validity of assessments be ensured?
  • How to increase the applicability and enhance the use of assessment outputs?
  • How should impacts be appraised?
  • How to make inferencing, information synthesis and communication more effective?
  • Overall, what makes a good assessment a good one?

Answers to the questions listed above are sought for by addressing them from the point of view of open assessment. After the workshop, the participants should have:

  • understanding and command of concepts, methods application areas of open assessment
  • Adequate command of tools applied in open assessment in order to efficiently make use of them in design and execution of assessments and use of assessment outputs
  • improved expertise in the environmental health relevant topics considered in the lectures and group work exercises

In addition to everything above, we wish that after the workshop at least some of the objects worked on in the exercises reach such a high quality in their information content that they can be considered to be the most up-to-date and comprehensive source of information regarding the topic in issue.

The core of the workshop program are the practical group work exercises related to a real-life assessment case called Assessment on impacts of emission trading on city-level (ET-CL), aiming to influence the COP-15 meeting to be held in Copenhagen December 2009. The exercises are complemented with lectures and guidance aiming to provide understanding of the necessary concepts and methods as well as sufficient command of the technical tools for carrying out the exercise tasks. The guidance will be provided by workshop organizers, as well as other participants suitably experienced in the question in issue, according to demand as the workshop progresses. Lectures and group work are explained in more detail below.


Schedule

Monday, Tietoteknia 1039 Tuesday, Tietoteknia 1039 Wednesday, Tietoteknia 1038 Thursday, Tietoteknia 1038 Friday, Tietoteknia 1038
9:00-9:15 Introduction to workshop 9:00-10:15 Lecture 3: Variables - interpretations of real-world phenomena, part I 9:00-10:15 Lecture 5: Science necessitates collaboration, part I 8:30-10:00 Lecture 7: Assessments - science-based decision support, part II 8:30-10:00 Lecture 8: Evaluating assessment performance
9:15-11:45 Lecture 1: Open assessment in research 10:30-11:45 Lecture 4: Variables - interpretations of real-world phenomena, part II 10:30-11:45 Lecture 6: Science necessitates collaboration, part II 10:00- Social activities: Skating on lake Kallavesi 10:00-11:45 Group work: Assessment performance evaluation
11:45-12:45 Lunch 11:45-12:45 Lunch 11:45-12:45 Lunch -13:00 Social activities: Skating on lake Kallavesi 11:45-12:30 Lunch
12:45-14:15 Lecture 2: Assessments - science-based decision support, part I 12:45- Group work: Developing variables 12:45- Group work: Developing variables in collaboration 13:00-14:00 Lunch 12:30-13:45 Group work presentations
14:45-16:15 Introduction to case study -15:00 Group work: Developing variables -15:00 Group work: Developing variables in collaboration 14:00-15:30 Group work: Assessment development 13:45-14:00 Workshop summary and feedback
16:15-16:45 Practical arrangements 15:00-16:45 Group work presentations 15:00-16:45 Group work presentations 15:30-16:45 Group work presentations
18:00- Social activities: dinner and smoke sauna 18:00- Social activities: skiing, snowboarding, sledding on [Puijo]

Morning sessions begin at 9:00. Lunch is available at Tietoteknia restaurant (at participant's own cost) 11:45 - 12:45., after which the afternoon sessions continue until approximately 16:45. Exceptions to this are Thursday when we have a later lunch time as well as delayed afternoon session due to a skating trip to lake Kallavesi in the forenoon, and Friday when the afternoon session will end already by 14:00 due to travel schedules of some workshop participants.

Tietoteknia restaurant also serves coffee and snacks. It is open 8:30 - 15:00 and lunch is served 11:00 - 13:30.

Lectures

The lectures are conducted in a form that attempt to emphasize practicality of discussed topics as well as dialogue among the lecturers and participants. The whole set of lectures comprise a discussion that continues and develops throughout the workshop week and guides the participants from considerations regarding the purpose of assessments to detailed contents of individual variables and on to a synthesis and an evaluation of overall outcomes of assessments. The lecture topics can be seen in the schedule above and the daily program description below. More detailed lecture contents can be found on a separate lecture page for each topic (follow the links to get to the lecture pages).

NOTE: The first lecture on Monday morning, Open assessment and science, is targeted also to people that do not have the possibility or interest to participate in the whole workshop. The opening lecture will consider open collaboration in scientific research, and the focus during the rest of the week will be mainly on assessments as science-based information support for societal decision making. It is thus possible to attend only the first lecture as an independent, separate part of the workshop program. It is also possible to continue to follow the rest of the workshop program after the first lecture without any prior registration.

Group work exercises

There are four exercise sessions, one every day from Tuesday to Friday, each consisting of first working in small groups on specific tasks and then presenting the outcomes of that work and lessons learned to other groups. On Tuesday and Wednesday the exercises focus on developing the contents of a chosen set of variables belonging to the Assessment on impacts of emission trading on city-level (ET-CL). On Thursday the exercises focus on certain aspects of the ET-CL case assessment. The last exercise on Friday is about evaluating the goodness of the assessment and its parts in their current form. The groups will be the same in Wednesday's and Thursday's exercises, but can be reconfigured for Thursday and Friday exercises.

The specific tasks for each group focus on one particular object, or a part of that object (a variable, an assessment attribute or group of variables located in Opasnet). The idea is to develop the information content by applying certain specified methods of open assessment. Tentative lists of objects and methods is presented below. The tasks are designed to be focused enough so that the group will be able to get a good grasp of the issue at hand, familiarize themselves adequately with the method(s) to be used, and produce some concrete output on the corresponding Opasnet pages. Methodological and tool support as well as substantial expertise will be provided according to demand by workshop organizers as well as other workshop participants already familiar with the tools, methods and issues in question.

The expected outcome of these exercise tasks consists of multiple aspects:

  • improved information content of corresponding variable, assessment, method, or other pages in Opasnet
  • increased understanding among the group members of both the phenomena that the objects describe and methods applied
  • understanding conveyed to other groups through group work presentations

Follow the links to lecture pages in the schedule above also to see the specific exercise tasks.

Practicalities

Facilities

The main activities throughout the workshop week take place at the seminar halls 1038 and 1039 in the Tietoteknia building (see the map in the beginning of the page to check the location). There will be a wireless internet connection available for all participants. Each participant is recommended to bring along a laptop computer. If this is not possible for you, please inform the organizers about this and we will see what we can do about it.


Remote participation

If there should be any people who would like to participate in the workshop, but cannot make it to Kuopio to attend the workshop in person, we try to organise a web-conference for the lecture sessions using Skype. The group work will be worked on using the Opasnet website, so also outsiders can relatively easily take part in that as well. It is also possible to join a group remotely and communicate with group members through Skype. This way you may also participate from wherever you are, as most major things - except the social program - will be made available via the Internet. Add your name on the list of remote via internet participants at the end of this page if you want to follow the workshop, but are unable to attend in person.

Instructions for remote participation to lectures:

  • NOTE: the lectures, including the Skype conferencing, are recorded
  • Call username open.risk on Skype (you can add this user to your Skype contacts preferably already before the workshop) a before the beginning of a lecture.
  • Keep your microphone on mute always when not making questions
  • When you want to make a question or comment, type a message to user open.risk and the lecturer will give you an opportunity to speak out your question (remember to turn on your microphone)
  • You can also send your question as a message and have the Skype-cast director read it out for you
  • The lecture videos will be put available online as soon as possible
  • Follow the links to lecture pages (see above) to get access to the lecture material

Instructions for remote participation to exercises:

  • Pick a group (exercise) you want to join and indicate this to workshop organizers e.g. by calling or messaging Skype user open.risk
  • The organizers will find out which Skype user to contact in order to join the group discussions
  • Participate in the work through the Opasnet pages related to the exercises assigned

Social activities

Activities arranged as part of the workshop program:

Equipment and prices for social activities:

  1. Bring your swimming suit if you want to try ice swimming on Tuesday (if you forget, the national sport of Finland, skinny-dipping, is also allowed). The price of smoke sauna is 11€ and dinner 19€, so in total the cost of Tuesday activities is 30€.
  2. On Wednesday, ski boots, skis, poles, snowboards etc. for both cross-country and downhill will be available for rental at the Antikkala lift station. Cross-country set 10€/2h, downhill ski set or snowboard 11€/2h, ski lift ticket 12€/2h. More rental price info at: Puijon Rinteet web-page. You may also bring your own equipment with you. Remember hat, gloves and warm, but comfortable clothing.
  3. Skating equipment will be available for rental on Thursday. Long-blade trekking skates cost 7€/2h and skating boots to fit in the skate bindings cost another 7€/2h. Also some blades that can be bound to your trekking/hiking shoes are available for 7€/2h. More rental info (e.g. helmets, elbow and knee pads) at: Leo's skating service web-page. Again hat, gloves and warm, but comfortable clothing is necessary.

It is still possible to adjust the social activities program. If you have wishes or suggestions, go ahead and let us know by writing them down on the discussion page

Please note that the social activities are voluntary and the costs of activities will be borne by the participants themselves. Participating in the social activities happens at each participant's own risk, the participants are not insured against any accidents by the workshop organizers. It is recommended that each participant makes sure before the workshop that they have a valid insurance that covers also the free-time activities during the workshop.

Examples of other activities, sites etc. available:

Accommodation

We have a block reservation for the workshop participants at Hotel Savonia see map. The hotel is the same as where most of the 2008 workshop participants stayed.

Room prices for workshop participants:

  • Single room 80 euros/night
  • Double room 96 euros/night

If you need accomodation, please (in addition to mentioning it in the registration below) inform the organizers about the desired room type and duration of your stay. If you wish to stay in Kuopio over the weekend before or after the workshop, the above-mentioned workshop room rates apply also for weekend nights.

Getting around in Kuopio

Arrival/departure by air

The distance from Kuopio airport to the city is about 17 km. There are two options to get from the airport to the hotel - bus and taxi. Kuopio airport transport connections

Bus:

Airport buses (5 €/trip) operate (by Linja-Karjala Oy) from all domestic flights, except for the flight AY519, arriving 00.25! The bus leaves from the airport approximately 5-15 minutes after the flight has arrived (the bus will wait for the passengers to collect their luggage before leaving). Get off the bus at Market Square in the center of Kuopio. From the Market Square it is possible to walk to the hotel (distance 2,4 km) or alternatively you can take a bus line 5 (2,70 €/trip) that goes by the hotel. Line 5 buses leave from the Market Square close to the Market Hall on the south end of the square. The timetable (unfortunately only in Finnish) for line 5 can be seen here.

Airport buses operate to all domestic flights. Bus to the airport leaves from Market Square (front of Anttila department store, east end of the square, Puijonkatu 19) 55 minutes before flight departure.

Taxi:

After each flight arrival there will be taxis waiting outside the airport. You may choose the airport taxi service (18 €/person), which means that the driver will wait until the car is full (in practice 2-3 persons/taxi) before leaving and then takes each passenger to their destination. Alternatively you may choose the normal taxi service (about 30 €/trip). If there are 2 people or more sharing a normal taxi it will be cheaper to take the it instead of the airport taxi service. If there are no taxis waiting at the airport you can order one by calling +358 17 106 400.

When going to the airport, make your airport taxi bookings (18 €/person) at least 2 hours before your scheduled departure by calling +358 17 106 400. The taxi will come to pick you up from agreed location approximately 1 hour before the scheduled departure time. Again, if there are 2 or more people sharing the taxi, it is also possible to order a normal taxi from the same taxi center number +358 17 106 400.

Arrival/departure by train

Kuopio railway station is located close to the center. It is possible to walk to the hotel or alternatively walk to Market Square and take bus line 5 going by the hotel. Line 5 buses leave from the Market Square close to the Market Hall on the south end of the square. The timetable (unfortunately only in Finnish) for line 5 can be seen here. Getting from hotel to the railway station happens similarly, but vice versa. You can also take a taxi to the hotel in front of the railway station.

Attendees

Please list your name and organisation here: (Requires login)

No accommodation needed:

Accommodation needed:

  • Eva Kunseler, PBL
  • Anne Knol, RIVM
  • Audrey de Nazelle, CREAL
  • Tek-Ang Lim, InVS
  • Danielle Vienneau, IC
  • Patrycja Gradowska, TUDelft


Participation via the Internet:

Useful background material


Keep an eye on this page, this is the official website of the workshop. It will be updated as new information comes available.

Lecture videos

Videos from 2009 now available. Sorry about the quality and format...


<mfanonymousfilelist></mfanonymousfilelist>