Difference between revisions of "Peer rating"
(idea of non-linear exchange rate for onor-money) |
(old onor rating table removed, instructions added) |
||
| Line 1: | Line 1: | ||
[[Category:Quality control]] | [[Category:Quality control]] | ||
{{method|moderator=Jouni}} | {{method|moderator=Jouni}} | ||
| − | '''Peer rating''' is a [[method]] to give merit ratings or [[onor]]s to a page in [[Opasnet]]. Peer rating has a scale from 0 to 100, but it | + | '''Peer rating''' is a [[method]] to give merit ratings or [[onor]]s to a page in [[Opasnet]]. Peer rating has a scale from 0 (very poor) to 100 (outstanding). Previously this number was converted to onors using a specific conversion function, but later it was understood that it is not clear what the function should be. The rating has two parts: scientific quality, and usefulness. Both properties are opinions of the person who makes the rating. Both evaluate the actual content of the page at the time of evaluation. Note that the average rating shown on the rating bar may be largely based on the historical quality of the page. Methods to compensate for page improvements is under development. |
| + | |||
| + | '''Practical hints: | ||
| + | * A person needs to login to be able to make peer ratings. | ||
| + | * Internet Explorer contains deliberate bugs (i.e., deviations from the standard html; the bugs are installed by Microsoft) that may cause trouble using and viewing the rating bar. Improvements are under way to overcome the problem, but in the meantime, we recommend that you complain directly to Microsoft. | ||
==Scope== | ==Scope== | ||
| Line 11: | Line 15: | ||
==Definition== | ==Definition== | ||
| − | The [[rating bar]] functionality only accepts integers between 0 and 100. Therefore, it is practical to make the first attempt of peer rating reflect this interval | + | The [[rating bar]] functionality only accepts integers between 0 and 100. Therefore, it is practical to make the first attempt of peer rating reflect this interval. |
| − | |||
| − | |||
| − | |||
| − | |||
{{discussion | {{discussion | ||
| Line 23: | Line 23: | ||
|Argumentation = | |Argumentation = | ||
| − | {{attack|1 | | + | {{attack|1 |Instead, we could do so that if you want to give negative rating, you actually should make a peer review instead. The peer review result may be negative as well as positive. |
| − | |||
| − | |||
| − | |||
| − | |||
|--[[User:Jouni|Jouni]] 12:29, 22 March 2010 (UTC)}} | |--[[User:Jouni|Jouni]] 12:29, 22 March 2010 (UTC)}} | ||
}} | }} | ||
==Result== | ==Result== | ||
| + | |||
| + | [http://en.opasnet.org/en-opwiki/index.php?title=Peer_rating&oldid=15158 A previously used onor rating table.] | ||
| + | |||
| + | A special table will be developed to rank pages based on their scientific quality and usefulness. Then the users can compare their estimates to other people's estimates of other pages. | ||
The total [[onor]]s earned by an individual i are calculated in the following way: | The total [[onor]]s earned by an individual i are calculated in the following way: | ||
| − | O<sub>i</sub> = Σ<sub>p</sub> O<sub>p</sub> | + | O<sub>i</sub> = Σ<sub>p</sub> E(O<sub>p</sub>) f<sub>i,p</sub>, |
| − | where O<sub>i</sub> is the amount of onors earned by individual i, O<sub>p</sub> is the amount of onors | + | where O<sub>i</sub> is the amount of onors earned by individual i, E(O<sub>p</sub>) is the expectation of the amount of onors given to a particular page p, and f<sub>i,p</sub> if the fraction of merit that the contributor i earns for his/her work on page p. |
Onors can be exchanged to money, but in a non-linear manner. There is a threshold for exchange, and onors below the threshold are not exchanged. The actual level of threshold is determined by the members of the society. The amount of exchangeable onors is calculated in the following way: | Onors can be exchanged to money, but in a non-linear manner. There is a threshold for exchange, and onors below the threshold are not exchanged. The actual level of threshold is determined by the members of the society. The amount of exchangeable onors is calculated in the following way: | ||
| − | O(E)<sub>i</sub> = Σ<sub>p</sub> O | + | O(E)<sub>i</sub> = Σ<sub>p</sub> O<sub>p</sub> f<sub>i,p</sub> (if O<sub>p</sub> f<sub>i,p</sub> >= T then 1 else 0), |
where T is the threshold. A major reason for this non-linearity is that anyone can do a large number of low-onor edits without specific skills. With a linear exchange rate, most of the money would be exchanged to such low-value contributions. It is more motivating and cost-effective to give higher prizes for hard and skilled work and none for others. This motivates people (who want to do this for living) to work especially on high-value things. | where T is the threshold. A major reason for this non-linearity is that anyone can do a large number of low-onor edits without specific skills. With a linear exchange rate, most of the money would be exchanged to such low-value contributions. It is more motivating and cost-effective to give higher prizes for hard and skilled work and none for others. This motivates people (who want to do this for living) to work especially on high-value things. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
Revision as of 15:09, 30 August 2010
This page is a method.
The page identifier is Op_en4123 |
|---|
| Moderator:Jouni (see all) |
|
Give your opinion to the peer rating of the content of this page. |
| Upload data
|
Peer rating is a method to give merit ratings or onors to a page in Opasnet. Peer rating has a scale from 0 (very poor) to 100 (outstanding). Previously this number was converted to onors using a specific conversion function, but later it was understood that it is not clear what the function should be. The rating has two parts: scientific quality, and usefulness. Both properties are opinions of the person who makes the rating. Both evaluate the actual content of the page at the time of evaluation. Note that the average rating shown on the rating bar may be largely based on the historical quality of the page. Methods to compensate for page improvements is under development.
Practical hints:
- A person needs to login to be able to make peer ratings.
- Internet Explorer contains deliberate bugs (i.e., deviations from the standard html; the bugs are installed by Microsoft) that may cause trouble using and viewing the rating bar. Improvements are under way to overcome the problem, but in the meantime, we recommend that you complain directly to Microsoft.
Scope
What is such a method to give respect or merit to contributions (Opasnet pages) such that the method
- reflects the true social respect on an absolute scale,
- can be learned and used easily.
Definition
The rating bar functionality only accepts integers between 0 and 100. Therefore, it is practical to make the first attempt of peer rating reflect this interval.
| Statements: Negative respect should also be possible with peer rating.
Resolution: Not accepted. Negative respect must be given through peer review. (Resolved, i.e., the resolution has been updated to the main page.) |
| Argumentation:
⇤1 : Instead, we could do so that if you want to give negative rating, you actually should make a peer review instead. The peer review result may be negative as well as positive. --Jouni 12:29, 22 March 2010 (UTC) |
Result
A previously used onor rating table.
A special table will be developed to rank pages based on their scientific quality and usefulness. Then the users can compare their estimates to other people's estimates of other pages.
The total onors earned by an individual i are calculated in the following way:
Oi = Σp E(Op) fi,p,
where Oi is the amount of onors earned by individual i, E(Op) is the expectation of the amount of onors given to a particular page p, and fi,p if the fraction of merit that the contributor i earns for his/her work on page p.
Onors can be exchanged to money, but in a non-linear manner. There is a threshold for exchange, and onors below the threshold are not exchanged. The actual level of threshold is determined by the members of the society. The amount of exchangeable onors is calculated in the following way:
O(E)i = Σp Op fi,p (if Op fi,p >= T then 1 else 0),
where T is the threshold. A major reason for this non-linearity is that anyone can do a large number of low-onor edits without specific skills. With a linear exchange rate, most of the money would be exchanged to such low-value contributions. It is more motivating and cost-effective to give higher prizes for hard and skilled work and none for others. This motivates people (who want to do this for living) to work especially on high-value things.
