Open assessment workshop 2009
This page is a encyclopedia article.
The page identifier is Op_en2582 |
|---|
| Moderator:Nobody (see all) Click here to sign up. |
|
Give your opinion to the peer rating of the content of this page. |
| Upload data
|
Open assessment workshop 2009 is the third workshop about new methods in assessments (previously more narrowly on risk assessment), organised by the National Institute for Health and Welfare, (THL), Kuopio, Finland. Its main purpose is to provide the workshop participants a broad perspective to assessments, promote the idea and practice of open assessment, as well as further develop methods to perform open assessments. The workshop has started following the needs of several EU-funded research projects, notably Intarese, Beneris, and Heimtsa. The workshop is mainly targeted to postgraduate students working in the fields of environmental health or risk assessment, but it is also open to anyone interested. If you want to join the workshop, please create a user account (by following the log in/create account link in the right-top corner of the page) with your contact information and add your username to the list on the bottom of this page. There is no deadline for registration nor a registration fee for the workshop, but early registration is highly recommended in order to help both the organizers and the participants to arrange the facilities, accommodation, programs etc.
Contents
General information
| Time: | 16.2. - 20.2.2009 |
| Place: | THL, Kuopio, Finland (Kuopio info) |
| Organizer: | National Institute for Health and Welfare (THL), Department of Environmental Health |
See the map below to check the locations of:
- Hotel Savonia - accommodation
- Tietoteknia - lectures and exercices
- Jätkänkämppä - smoke sauna, ice swimming, traditional dinner
- Antikkala - cross-country and downhill skiing, sledding
- Kuopio passenger harbour - Skating tracks
- Kuopio railway station
- Kuopio airport
- Kauppahalli (market hall at market square) - bus stop for line 5 from city center to Hotel Savonia
- National Institute for Health and Welfare, (THL), Department of Environmental Health
You can zoom into the map and by clicking the red pins on the map, an information box indicating the name of the location will appear.
<googlemap width="375" zoom="11" type="map" lon="27.638855" lat="62.903976"> m62.893334, 27.679328, Kuopio, Finland 62.890603, 27.627397, THL 62.866252, 27.648468, Jätkänkämppä smoke sauna 62.890240, 27.635560, Tietoteknia - Lectures and exercises 62.901100, 27.651600, Hotel Savonia - Workshop accommodation 62.891659, 27.678895, Kauppahalli, bus line 5 to Hotel Savonia 63.014833, 27.792149, Kuopio airport 62.895882, 27.679882, Kuopio railway station 62.921720, 27.662373, Antikkala - ski slope, ski tracks and sledding slope 62.886549, 27.697393, Skating tracks </googlemap>
5-day weather forecast for Kuopio by the Finnish Meteorological Institute.
5-day weather forecast for Kuopio (in Finnish) by Foreca.
15-day forecast for Kuopio by Accuweather
Description
The aim of the workshop is to provide the participants a good understanding of how to make the practical information needs of societal action and creation of scientific knowledge meet by making use of the latest methodological developments and modern information and communication technology. This aim is pursued in the workshop lectures and exercises by discussing and considering e.g following questions:
- How to take the practical needs of policy into account in assessments?
- How can scientific validity of assessments be ensured?
- How to increase the applicability and enhance the use of assessment outputs?
- How should impacts be appraised?
- How to make inferencing, information synthesis and communication more effective?
- Overall, what makes a good assessment a good one?
Answers to the questions listed above are sought for by addressing them from the point of view of open assessment. After the workshop, the participants should have:
- understanding and command of concepts, methods application areas of open assessment
- Adequate command of tools applied in open assessment in order to efficiently make use of them in design and execution of assessments and use of assessment outputs
- improved expertise in the environmental health relevant topics considered in the lectures and group work exercises
In addition to everything above, we wish that after the workshop at least some of the objects worked on in the exercises reach such a high quality in their information content that they can be considered to be the most up-to-date and comprehensive source of information regarding the topic in issue.
The core of the workshop program are the practical group work exercises related to a real-life assessment case called Assessment on impacts of emission trading on city-level (ET-CL), aiming to influence the COP-15 meeting to be held in Copenhagen December 2009. The exercises are complemented with lectures and guidance aiming to provide understanding of the necessary concepts and methods as well as sufficient command of the technical tools for carrying out the exercise tasks. The guidance will be provided by workshop organizers, as well as other participants suitably experienced in the question in issue, according to demand as the workshop progresses. Lectures and group work are explained in more detail below.
Schedule
| Monday | Tuesday | Wednesday | Thursday | Friday |
|---|---|---|---|---|
| Introduction to workshop | Lecture 3: Variables and their information structure | Lecture 5: Scientific validity and practical need | Lecture 7: Assessments as causal networks, assessment-level analyses and conclusions | Lecture 8: Performance of assessment |
| Lecture 1: Open assessment in research | Lecture 4: Information collection and synthesis | Lecture 6: Collaboration, argumentation | Outdoor activities: Skating on lake Kallavesi | Group work: Assessment performance evaluation |
| Lunch | Lunch | Lunch | Outdoor activities: Skating lake Kallavesi | Lunch |
| Lecture 2: Decisions, actors and indicators | Group work: Variable development | Group work: Variable development in collaboration | Lunch | Group work presentations |
| Introduction to case study | Group work: Variable development | Group work: Variable development in collaboration | Group work: Assessment development | Workshop summary and feedback |
| Practical arrangements | Group work presentations | Group work presentations | Group work: Assessment development | |
| Evening program: dinner and smoke sauna | Evening program: skiing, snowboarding, sledding on [Puijo] | Group work presentations |
Morning sessions begin at 9:00. Lunch is available at Tietoteknia canteen (at participant's own cost) 11:45 - 12:45., after which the afternoon sessions continue until approximately 16:45. Exceptions to this are Thursday when we have a later lunch time as well as delayed afternoon session due to a skating trip to lake Kallavesi in the forenoon, and Friday when the afternoon session will end already by 14:00 due to travel schedules of some workshop participants.
Lectures
The lectures are conducted in a form that attempt to emphasize practicality of discussed topics as well as dialogue among the lecturers and participants. The whole set of lectures comprise a discussion that continues and develops throughout the workshop week and guides the participants from considerations regarding the purpose of assessments to detailed contents of individual variables and on to a synthesis and an evaluation of overall outcomes of assessments. The specific lecture topics can be seen in the schedule above and the daily program and contents below.
NOTE: The first lecture on Monday morning, Open assessment and science, is targeted also to people that do not have the possibility or interest to participate in the whole workshop. It emphasizes the relationship between science and open assessment and impact of OA to making science, whereas the focus in the rest of the lectures and exercises is rather on providing information support to practical decision making. It is thus possible to attend only the first lecture as an independent part of the workshop program. It is also possible to continue to follow the rest of the workshop program after the first lecture without any prior registration.
Group work exercises
There are four exercise sessions, one every day from Tuesday to Friday, each consisting of first working in small groups on specific tasks and then presenting the outcomes of that work to other groups. On Tuesday and Wednesday the exercises focus on developing the contents of a chosen set of variables belonging to the Assessment on impacts of emission trading on city-level (ET-CL). On Thursday the exercises focus on certain assessment level aspects of the ET-CL case drawing together the work on individual variables on previous days. The last exercise on Friday is about evaluating the goodness of the assessment and its parts in their current form. The groups will (probably) be reconfigured for each exercise.
The specific tasks for each group focus on one particular object, or a part of that object (a variable, an assessment attribute or group of variables located in Opasnet). The idea is to develop the information content by applying certain specified methods of open assessment. Tentative lists of objects and methods is presented below. The tasks are designed to be focused enough so that the group will be able to get a good grasp of the issue at hand, familiarize themselves adequately with the method(s) to be used, and produce some concrete output on the corresponding Opasnet pages. Methodological and tool support (also substantial expertise, if available) will be provided according to demand by workshop organizers as well as other workshop participants already familiar with the tools and methods in question.
The expected outcome of these exercise tasks consists of multiple aspects:
- improved information content variable or assessment pages in Opasnet
- improved information content of method descriptions in Opasnet
- increased understanding among the group members of both the substance that the objects describe and the methods applied
- understanding conveyed to other groups through group work presentations
Monday: Introduction, motivation, and basic concepts
Monday morning lecture: Open assessment in research
- Relations between science, assessments and policy - making "science push" and "policy pull" meet
- Open science - individual specific studies, common synthesized results
During Monday morning you should learn:
- How traditional articles have two distinct parts
- How these parts can be organised in a better way
- You are exposed to the idea of scientific method.
- What is the use of science in policy assessments.
- For a piece of information, you know what is a good object type for it.
- You learn to see the world as a collection of information pieces.
- You see your own work/research as a part of a global mass collaboration project.
- You learn that it is possible to do the whole research process (idea - research plan - execution of a study - writing articles) in Opasnet.
- You learn that the impact of you research may be higher in an open system.
- See how Opasnet can be used in a practical case study (assessment, research).
Opasnet pages related to Monday morning topics:
Monday afternoon lecture: Assessments - science-based decision support
- Assessments - providing science-based answers to practical needs
- Decisions, actors and indicators
Introduction to group work
- Societal decision making constrained by GHG emission reduction goals - optimization of health impacts and costs
During Monday afternoon you should learn:
- You learn the basics of an assessment: question/info need, and the three-part structure: decision - factors - outcome.
- You learn how an information need is transferred into a research question.
- You learn the importance and implications of boundaries in defining an object (assessment, variable)
- You learn the roles of participants and intended users.
- You learn to participate in discussions about a new assessment (Tapas) and its framing.
- You learn about the outline of the workshop week, the basics of the cse study, and objectives of each session.
a Opasnet pages related to Monday afternoon topics:
- Performing an assessment
- Defining the purpose of an assessment
- Defining the users of an assessment
- Issue framing
- Scoping an assessment
Tuesday: variable development
Tuesday morning lectures: Variables - evolving interpretations of real-world phenomena and Variables - best estimates of reality
During Tuesday morning you should learn:
- You learn the basic structure of an information object.
- You learn what variables are: independent, eternally developing descriptions of reality.
- You learn that the ultimate criterion of a variable is the truth.
- You learn the differences and similarities of a variable, assessment, study, and encyclopedia article.
- You learn how to use the variable definition as your information storage.
- You learn how to make a new synthesis when new information comes up (in Bayesian language: to update your prior).
- You learn how to handle variables that are partially or totally overlapping.
Tuesday afternoon exercises:
| Variable | Task | Examples of available data sources |
|---|---|---|
| PM2.5 exposure-response function | Developing the contents of the variable. Acquaint yourself with the variable concept and the structure of a variable. With the purpose of the case assessment in mind, first determine the scope of the variable and then consequently definition and result based on both the provided existing variables as data sources and your own expertise. Specific things to consider: a) how to synthesize data from different sources? b) what additional data sources could be used to improve the variable? c)why could not either of the available existing variables be used instead of creating a new variable? d) what do you think about the quality of your variable (as the best estimate for the phenomenon it describes)? Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable. | Existing variables Concentration-response to PM2.5 and ERF of PM2.5 on mortality in general population |
| Emission factors for burning processes | Developing the contents of the variable. Acquaint yourself with the variable concept and the structure of a variable and consider the role of this variable in the case assessment. Dig out the necessary pieces of information from the provided FIRE database according to the needs of the assessment. Needed indexes and locations include:
Having done the above, consider possible other data sources that could be used to improve the variable, e.g. in terms of making the variable more suitable also for other assessments. Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable. |
|
| Population of Helsinki metropolitan area | Developing the contents of the variable. With the purpose of the case assessment in mind, first determine the scope of the variable and then consequently consider definition and result. Check the available population data from different sources, consider their differences and make a synthesis out of them. Specific things to consider include:
Prepare to present your variable to other groups and explain how your thinking evolved in the process of developing the variable. |
|
| DALY weights | Developing the contents of the variable. Acquaint yourself with DALY concept, in particular regarding how the weights are estimated. Consider how the existing variable could be improved by including uncertainties in the weight estimates, e.g. making a shift from point estimates to distributions. Improve the variable accordingly if possible. Prepare to present your variable to other groups and explain DALY weights, their role in estimating DALYs, and how including uncertainty in DALY weights affects DALY estimates. | WHO Global Burden of Disease |
| Cardiopulmonary mortality in Helsinki metropolitan area | Developing the contents of the variable. With the purpose of the case assessment in mind, first determine the scope of the variable and then consequently consider definition and result. Search for the needed information from Opasnet base and apply it in developing the variable. | A general variable Background incidence rates in Europe, [1] |
During Tuesday afternoon you should learn:
- You learn to build the definition based on the scope, and compute the result based on the definition.
- You get practical experience about topics you learned in the morning.
Opasnet pages related to Tuesday topics:
- Designing variables
- Plausibility test
- Applying general information
- Result domain
- Executing variables
- Calculating health impacts
- Life table
Wednesday: variable development in collaboration
Wednesday morning lectures: Scientific method necessitates collaboration and Argumentation coordinates collaboration
During Wednesday morning you should learn:
- You learn why collaboration is better than working alone.
- You learn to apply the scientific method, and the importance of it in open assessment.
- You learn the structure of formal argumentation.
- You learn how to collaborate
Wednesday afternoon exercises:
- Choose (as a group) a variable that another group developed on Tuesday (see list of variables in the table above). Scrutinize the variable carefully and, according to your expertise and understanding, create statements and arguments about the variable content. Document the statements as well as the arguments for against your group comes up with on the discussion page of the variable following the formal argumentation structure. It is also possible to first collect comments informally, analyze the comments and then reorganize them in the argumentation format. Consider and argue in particular:
- Is the scope unambiguously defined (what is the question the variable attempts to answer to)?
- Does the definition provide the sufficient means to answer the question adequately?
- Is the result an exact and explicit logical outcome of the definition?
- Does the result appear reasonable and meaningful?
- Go back to the variable you (as a group) developed on Tuesday and scrutinize the argumentation that another group has created about the contents of your variable. React to the argumentation by counter arguing or by improving the variable contents as seen necessary.
- Prepare a short presentation about the argumentation about your group's variable description invoked and how it was or should be taken into account in further development and use of that variable.
During Wednesday afternoon you should learn:
- You learn to use formal argumentation and to turn existing discussions into this formalism.
Opasnet pages related Wednesday topics:
- Open participation
- Evaluating scientific claims
- Dealing with disputes
- Discussion
- Help:Argumentation
- Index conversion function
- Merging models with different grids
- Synthesising information about continuous variables
- Using non-predetermined spatial disaggregation
Thursday: assessment development
Thursday morning lecture: Assessments - providing answers to practical questions
- Assessments as collections of variables
- Assessments as answers to practical questions
- Assessments as causal networks
- Bayes belief nets
- Value of information
- Scenarios - conditioning the causal network
| Exercise topic | Task |
|---|---|
| Meeting the needs of intended use (primary: WWViews, COP-15) | Consider the intended use, its needs and the scope of the assessment e.g. in light of the following questions:
Document your statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the identified need and how it is addressed. |
| Answering the assessment questions - variable network | This task is about developing the definition of the case assessment in terms of the variables belonging to the assessment. Explore the list of variables, their types (decision, indicator, other) and the causal diagram representation of the assessment as well as the Analytica model. Given the scope of the assessment, consider e.g. following questions:
Document statements, arguments and comments on the discussion page, and make amendments directly to the assessment page as necessary. Also present your findings to other groups including a brief explanation of the causal network of variables and its relation to the defined scope of the assessment. |
| Answering the assessment questions - BBN analysis | Acquaint yourself with Bayesian belief networks and consider how they can be used in assessments. In particular look into:
Analyze a simple Uninet model, provided to you by the workshop organizers, covering a part of the case assessment. By conditioning different variables try to find the optimal decision option. You can also update the model with new variable results found in Opasnet and repeat the analysis. You can document statements, arguments and comments on , or make amendments to, related pages in the Opasnet if relevant. Present your findings to other groups including a brief introductory explanation of how Bayesian nets can be used in the case assessment. |
| Answering the assessment questions - Value of information (VOI) analysis | Acquaint yourself with the Value of information method (VOI). In particular look into:
Analyze a simple Analytica model, provided to you by the workshop organizers, covering a part of the case assessment. Calculate VOI for different variables in the model. You can also update the model with new variable results found in Opasnet and repeat the analysis. Present your findings to other groups including a brief introductory explanation of how VOI can be used in the case assessment. |
| Scenarios | Acquaint yourself with the way the concept Scenario is defined in the context of open assessment. Then consider the scenarios set for the case assessment e.g. in light of the following questions:
Document statements, arguments and comments on the discussion page, and present your findings to other groups. |
Opasnet pages related to related Thursday exercise:
- Drawing a causal diagram
- Improving the result of a variable
- Bayesian network
- Value of information
- Reporting an assessment
- Operationalising large structural changes to existing systems
- Variable transfer protocol
Friday: evaluation of assessment performance
Friday lecture: Properties of good assessment - can practical need and scientific truth be served simultaneously?
- Properties of good assessments
- Quality of content
- Applicability
- Efficiency
- Technical quality
- Peer review
- Contributor rating
Methods related to Friday exercise:
- Performance
- Acknowledgements
- Peer review
- Quality assurance and quality control
- Quality evaluation criteria
Practicalities
Facilities
NOTE: Lecture and exercise location has changed!!!
The main activities throughout the workshop week take place at the seminar halls 1038 and 1039 in the Tietoteknia building (see the map in the beginning of the page to check the location). There will be a wireless internet connection available for all participants. Each participant is recommended to bring along a laptop computer. If this is not possible for you, please inform the organizers about this and we will see what we can do about it.
Remote participation
If there should be any people who would like to participate in the workshop, but cannot make it to Kuopio to attend the workshop in person, we try to organise a web-conference for the lecture sessions using Skype. The group work will be worked on using the Opasnet website, so also outsiders can relatively easily take part in that as well. It is also possible to join a group remotely and communicate with group members through Skype. This way you may also participate from wherever you are, as most major things - except the social program - will be made available via the Internet. Add your name on the list of remote via internet participants at the end of this page if you want to follow the workshop, but are unable to attend in person.
Note! Our Skype username is: open.risk
You can add that to your Skype contacts preferably already before the workshop
More information regarding how to participate remotely in practice will be provided here before beginning of the workshop.
Social activities
Activities arranged as part of the workshop program:
- Tuesday evening: smoke sauna, ice swimming and dinner at Jätkänkämppä
- Wednesday evening: skiing, snowboarding, sledding etc. at Puijo (Antikkala)
- Thursday forenoon: skating on lake Kallavesi exploiting the Finland Ice Marathon skating tracks
Equipment and prices for social activities:
- Bring your swimming suit if you want to try ice swimming on Tuesday (if you forget, the national sport of Finland, skinny-dipping, is also allowed). The price of smoke sauna is 11€ and dinner 19€, so in total the cost of Tuesday activities is 30€.
- On Wednesday, ski boots, skis, poles, snowboards etc. for both cross-country and downhill will be available for rental at the Antikkala lift station. Cross-country set 10€/2h, downhill ski set or snowboard 11€/2h, ski lift ticket 12€/2h. More rental price info at: Puijon Rinteet web-page. You may also bring your own equipment with you. Remember hat, gloves and warm, but comfortable clothing.
- Skating equipment will be available for rental on Thursday. Long-blade trekking skates cost 7€/2h and skating boots to fit in the skate bindings cost another 7€/2h. Also some blades that can be bound to your trekking/hiking shoes are available for 7€/2h. More rental info (e.g. helmets, elbow and knee pads) at: Leo's skating service web-page. Again hat, gloves and warm, but comfortable clothing is necessary.
It is still possible to adjust the social activities program. If you have wishes or suggestions, go ahead and let us know by writing them down on the discussion page
Please note that the social activities are voluntary and the costs of activities will be borne by the participants themselves. Participating in the social activities happens at each participant's own risk, the participants are not insured against any accidents by the workshop organizers. It is recommended that each participant makes sure before the workshop that they have a valid insurance that covers also the free-time activities during the workshop.
Examples of other activities, sites etc. available:
- Finland Ice marathon, a skating event on natural ice of lake Kallavesi 18-22 February, 2009
- Puijon laturetki - a cross country skiing event
- Tahko, a ski resort approximately 1 hour drive northeast from Kuopio
- Puijo tower, the most famous attraction of Kuopio
- Rauhalahti Spa, a spa hotel in Kuopio
- Live music e.g. at Henkka or Kuopio Music Centre
Accommodation
We have a block reservation for the workshop participants at Hotel Savonia see map. The hotel is the same as where most of the 2008 workshop participants stayed.
Room prices for workshop participants:
- Single room 80 euros/night
- Double room 96 euros/night
If you need accomodation, please (in addition to mentioning it in the registration below) inform the organizers about the desired room type and duration of your stay. If you wish to stay in Kuopio over the weekend before or after the workshop, the above-mentioned workshop room rates apply also for weekend nights.
Getting around in Kuopio
Arrival/departure by air
The distance from Kuopio airport to the city is about 17 km. There are two options to get from the airport to the hotel - bus and taxi. Kuopio airport transport connections
Bus:
Airport buses (5 €/trip) operate (by Linja-Karjala Oy) from all domestic flights, except for the flight AY519, arriving 00.25! The bus leaves from the airport approximately 5-15 minutes after the flight has arrived (the bus will wait for the passengers to collect their luggage before leaving). Get off the bus at Market Square in the center of Kuopio. From the Market Square it is possible to walk to the hotel (distance 2,4 km) or alternatively you can take a bus line 5 (2,70 €/trip) that goes by the hotel. Line 5 buses leave from the Market Square close to the Market Hall on the south end of the square. The timetable (unfortunately only in Finnish) for line 5 can be seen here.
Airport buses operate to all domestic flights. Bus to the airport leaves from Market Square (front of Anttila department store, east end of the square, Puijonkatu 19) 55 minutes before flight departure.
Taxi:
After each flight arrival there will be taxis waiting outside the airport. You may choose the airport taxi service (18 €/person), which means that the driver will wait until the car is full (in practice 2-3 persons/taxi) before leaving and then takes each passenger to their destination. Alternatively you may choose the normal taxi service (about 30 €/trip). If there are 2 people or more sharing a normal taxi it will be cheaper to take the it instead of the airport taxi service. If there are no taxis waiting at the airport you can order one by calling +358 17 106 400.
When going to the airport, make your airport taxi bookings (18 €/person) at least 2 hours before your scheduled departure by calling +358 17 106 400. The taxi will come to pick you up from agreed location approximately 1 hour before the scheduled departure time. Again, if there are 2 or more people sharing the taxi, it is also possible to order a normal taxi from the same taxi center number +358 17 106 400.
Arrival/departure by train
Kuopio railway station is located close to the center. It is possible to walk to the hotel or alternatively walk to Market Square and take bus line 5 going by the hotel. Line 5 buses leave from the Market Square close to the Market Hall on the south end of the square. The timetable (unfortunately only in Finnish) for line 5 can be seen here. Getting from hotel to the railway station happens similarly, but vice versa. You can also take a taxi to the hotel in front of the railway station.
Attendees
Please list your name and organisation here: (Requires login)
No accommodation needed:
- Jouni Tuomisto, THL, Risk research group
- Juha Villman, THL
- Päivi Meriläinen, THL
- Mikko Pohjola, THL
- Juuso Uski, Toxicology student, University of Kuopio
- Virpi Kollanus, THL
- Pauliina Ahtoniemi, THL
- Olli Leino, THL
- Marko Tainio, THL
- Pasi Sorsa, THL
Accommodation needed:
- Eva Kunseler, PBL
- Anne Knol, RIVM
- Audrey de Nazelle, CREAL
- Tek-Ang Lim, InVS
- Danielle Vienneau, IC
- Patrycja Gradowska, TUDelft
Participation via the Internet:
- Katy Walker
- Alex Kuhn
- Tam T Pham
- Adeyanju kazeem Alade, Scientific Computing Student, University of kuopio.
- Aruya Omoarebun Richard, Msc Scientific Computing, University of Kuopio.
Useful links while preparing for the workshop
- Glossary is a collection of important concepts and terms and their explanations
- See Help:Contents for guidance on using Opasnet (this site, running on Mediawiki software platform).
- We recommend participants to familiarise themselves with the following concepts:
- A free 15-day trial version of Analytica or an Analytica player for viewing the models from Lumina Decision Systems website on this site.
- Guidebook is a collection of links to theory and method pages related to the workshop.
- Lectures from workshop 2008:
- Lecture 1: Introduction to Open Risk Assessment (ORA) and the workshop (also in Finnish!)
- Lecture 2: General assessment framework
- Lecture 3: Information structure of ORA: assessment products and their relations
- Lecture 4: Information structure of ORA: defining variables
- Lecture 5: participating in assessments: argumentation
- Lecture 6: participating in assessments: moderation and quality control
- Lecture 7: evaluating assessment performance
- Lecture 8: assessing uncertainties
Keep an eye on this page, this is the official website of the workshop. It will be updated as new information comes available.
