Difference between revisions of "Open assessment"

Revision as of 08:55, 30 September 2009

<section begin=glossary />

Open assessment (previously also known as pyrkilo) is a method that attempts to answer the following research question and to apply the answer in practical assessments:

How can scientific information and value judgements be organised for improving societal decision-making in a situation where open participation is allowed?

Open assessment can also refer to the actual making of such an assessment (precisely: open assessment process), or the end product of the process (precisely: open assessment product or report). Usually, the use of the term open assessment is clear, but if there is a danger of confusion, the precise term (open assessment method, process, or product) should be used. In practice, the assessment processes are performed using Internet tools (notably Opasnet) among traditional tools. Stakeholders and other interested people are able to participate, comment, and edit its contents already since an early phase of the process. Open assessment is based on a clear information structure and scientific method as the ultimate rule for dealing with disputes.

<section end=glossary />

For a brief description about open assessment and the related workspace, see Opasnet.

Basics of Open assessment

Open assessment is not very difficult as such. Usually the problem is that people don't believe how simple it actually is.

Open assessment typically starts with a need for an informed decision. Someone must decide about something, and the decision should be based on the best information about the topic and the objectives that are pursued. The decision can be anything, and the decision-maker can be anyone. Open assessment does not perform too well, if the decision is very small and does not warrant much effort (where do we go for lunch today?), or if a large part of the related information is private or cannot be released to open scrutiny due to other reasons (which girl should I marry?). However, both of these decisions can also be assessed from a general point of view, and then they are good topics for open assessment (What are good places for lunch such that serve good food, are not expensive, and do not take too much time from our workplace? Which properties of a relationship or a potential spouse predict a successful and happy marriage, and which predict an unsuccessful and unhappy marriage?)

As the examples showed, the decision was formulated as a question. This is the first, critical step in open assessment. All information gathering starts with an identification of a question that needs to be answered. If you don't know your question, you actually don't know what you are trying to decide. Therefore it is essential that a question is formulated.

I should mention here that this is the reason why open assessment works with basic science as well. Science asks research questions and attempts to answer them based on scientific information. When a question has been defined, it is treated in the exactly same way, be it a policy question or a science question. Actually, the difficult question of a science-policy interface practically disappears, because politicians and scientists are

The question must be clear enough that it can be answered unambiguously. It must also be possible to see which answers are better than others. In the case of a decision question, some performance criteria must be defined. This means that we must define what a good decision is, in other words, which are outcomes to be pursued and which are to be avoided. Although "successful and happy marriage" is still rather fuzzy, we can assume that if a large group of people ranked a large number of marriages based on successfulness and happiness, the results would show clear patterns. In addition, the research question rules out a lot of objectives such as the amount of money that the potential farther-in-law is going to give to the newly-wed couple. Or, to be precise, this money comes into the assessment only through its potential to facilitate a happy marriage, but it does not have any intrinsic value in the comparison of options in the assessment.

When the question has been defined, the rest of the work is to answer the question as precisely as it is possible with the information available. The scientific method is applied here. It means that anyone can suggest potential answers, and other people try to falsify them, i.e. use data and argumentation to show that an answer is wrong or irrelevant, given the question.

An answer to the main question may depend on a lot of other issues. These, again, are framed as questions, and their answers are used to answer the original question. Thus, the whole assessment consists of pieces of information, which themselves consist of pairs of questions and answers.

Why is open assessment a revolutionary method?

There are several things that are done differently, and arguably better, in open assessment compared with traditional ways of collecting information. These are briefly listed here and then described in more detail.

1. Open assessment can be applied to most decision-making situations.
2. Open assessment helps to focus on relevant issues.
3. Important issues are explicated.
4. The expression of values is encouraged.
5. It becomes more difficult to promote non-explicated values, i.e. hidden agendas.
6. Open assessment focuses on primary issues and thus gives little emphasis on secondary issues.
7. Open assessment separates the policy-making (developing and evaluating potential decision options) and the actual decision-making (making of the decision by the authoritative body).
8. Open assessment breaks the information monopoly of the authoritative body and motivates participation.
9. Open assessment makes the information collection quicker and easier.
10. Open assessment is based on the scientific method.
11. Open assessment does not prevent the use of any previous methods.

Open assessment can be applied to most decision-making situations.

Open assessment helps to focus on relevant issues.

The assessment work boils down to answering the assessment question. Whatever helps in answering is useful, and whatever does not help is useless. You can always ask a practical question from a person who suggests additional tasks: "How would this task help us in answering the question?" It is a simple question to ask, but a difficult one to answer by a bureaucrat.

Important issues are explicated.

The key part of an assessment product is an answer to the assessment question. The question can easily be falsified, unless it is explicitly defended by relevant arguments found in the assessment. This simple rule forces the assessment participants to explicate all issues that might be relevant for the end users when they evaluate the acceptability of the answer.

The expression of values is encouraged.

Valuations are used to optimise decisions. Implicit values are not used to make conclusions. Therefore, if you don't like Chinese food, you must express this value in the lunch place assessment, otherwise the value is ignored. Any values can be included, and these will be taken into account.

It becomes more difficult to promote non-explicated values, i.e. hidden agendas.

Open assessment focuses on primary issues and thus gives little emphasis on secondary issues.

Secondary issues include preparation committees, meeting minutes and so on. After all, all these secondary issues are only needed to get answers to the questions. Instead of trying to get into the committee, participate in the discussions, and write the minutes, you just go to your assessment page and write down your suggestions, and you are done. Of course you can still participate in any meetings to stimulate your thinking, but many people have experience on meetings that take a lot of working time without giving any stimulation.

Open assessment separates the policy-making (developing and evaluating potential decision options) and the actual decision-making (making of the decision by the authoritative body).

Open assessment opens the policy-making so that anyone can participate and bring in their information.

Open assessment breaks the information monopoly of the authoritative body and motivates participation.

The authoritative body can still make the decision just like before. But this body has no say over issues that will be included in an open assessment. The inclusion or exclusion of issues depends only on relevance, which ultimately depends only on the assessment question.

Open assessment makes the information collection quicker and easier.

Open assessment is based on the scientific method.

Open assessment does not prevent the use of any previous methods.

Open assessment does not preclude any methods. It only determines the end product (a hierarchical thread of questions and answers, starting from the main question of the assessment), and how individual pieces of information are evaluated (based on relevance, logic, and coherence with observations). Of course, some traditional methods will perform poorly in these conditions, but those who insist to use them are free to do so.

See also

• Opasnet
• Open Assessors' Network
• Open assessment method
• Opasnet base