Difference between revisions of "Talk:Biofuel assessments"
m (→to do list) |
(some updates and edits) |
||
| Line 3: | Line 3: | ||
Things to be presented on this summary page in order to be in line with what is said in the [[:heande:Evaluating effectiveness of open assessments on alternative biofuel sources|biodiesel assessments -article]]: | Things to be presented on this summary page in order to be in line with what is said in the [[:heande:Evaluating effectiveness of open assessments on alternative biofuel sources|biodiesel assessments -article]]: | ||
| − | + | * example of the discussions (formalized presentations of participant comments) (Vilma) | |
| − | * example of the discussions (formalized presentations of participant comments) | + | * (anonymized) numerical questionnaire results (Teemu) |
| − | + | * R-code to run the analyses of numerical questionnaire data + results ((Teemu) | |
| − | + | * summary of the overall evaluation (Mikko) | |
| − | + | * checking/proofreading of the summary of the assessments and their results as well as discussion contents (Mikko) | |
| − | |||
| − | |||
| − | |||
| − | * | ||
| − | - | + | |
| + | == Invited stakeholders == | ||
| + | |||
| + | '''Environmental non-governmental organizations | ||
| + | * Greenpeace | ||
| + | * WWF Finland | ||
| + | * The Finnish Association for Nature Conservation | ||
| + | * Dodo ry | ||
| + | * Friends of the Earth Finland | ||
| + | |||
| + | '''Expert organisations | ||
| + | * National Institute for health and welfare (THL) | ||
| + | * Finnish Environment Institute (SYKE) | ||
| + | * Bioteknologia.info | ||
| + | * Motiva Ltd | ||
| + | * MTT Agrifood Research Finland | ||
| + | * Finnish Game and Fisheries Research Institute | ||
| + | * Ministry of the environment | ||
| + | * Finnish energy industries | ||
| + | * VTT Technical Research Centre of Finland | ||
| + | * Paula Tommila (Researcher, specialist in Jatropha) | ||
| + | |||
| + | '''Human rights organisations | ||
| + | * Amnesty International Finland | ||
| + | |||
| + | '''Energy corporations | ||
| + | * Neste Oil | ||
| + | * St1 Biofuels Oy | ||
| + | * Vapo biofuels | ||
| + | |||
| + | |||
| + | == Examples of formalized discussions == | ||
| + | |||
| + | 'Tähän pari keskustelusivulle laitettavaa edustavaa esimerkkiä, esim. yksi kommentti + siihen liittyvät argumentit ([[op_fi:Keskustelu:Jatropan käyttö bioenergian lähteenä]] tai [[:op:fi:Keskustelu:Kalanjalostuksen öljyjäte bioenergian lähteenä]]) sekä yksi jonkin muuttujan keskustelusivulle kirjattu keskustelulaatikko (väittämät, ratkaisu, argumentaatio). | ||
| + | |||
| + | == Evaluation of assessments == | ||
| + | |||
| + | === Questionnaire === | ||
| + | |||
| + | After finishing the assessment, participants were contacted again and they were asked to evaluate the performance of the two biofuel assessments. All of the invited stakeholder groups, primary users of the assessments (Neste Oil), summer trainees who worked with the assessment as well as the coordinators at THL were contacted by e-mail and asked to give numerical evaluations of certain questions of a questionnaire on a scale 1 - 5 (1 meaning bad and 5 meaning good). They were also asked to consider and argue the positive and negative aspects of the assessments and provide textual comment to accompany the numerical values. Both assessments (Jatropha and fish waste) were analysed together but the participants were told to specify if the answers differed in the two assessments. The questions were: | ||
| + | |||
| + | Impact of the participation (Q1 only for the stakeholders) | ||
| + | |||
| + | :1. Do You feel that Your contribution has been included in the assessment fairly? | ||
| + | |||
| + | The performance of the assessments (Q2-Q6 for all) | ||
| + | |||
| + | :2. Does the assessment content correspond to the research question accurately, truthfully and comprehensively? | ||
| + | :3. How well does the information provided by the assessment serve Your needs (or the needs of Your organization)? | ||
| + | :4. Has the information provided by the assessment reached You and Your organisation? | ||
| + | :5. Did Your understanding increase about the issue along with the assessment? | ||
| + | :6. Is the assessment result (output), and the way it is obtained and delivered for use acceptable? | ||
| + | |||
| + | Efficiency (Q7 only for the primary users, here Neste Oil) | ||
| + | |||
| + | :7. How good is the assessment output in relation to the resources used? | ||
| + | |||
| + | === Questionnaire results and statistical analysis === | ||
| + | |||
| + | === Results === | ||
| + | |||
| + | === Analyses === | ||
| + | |||
| + | In the two instances (users) where different numerical values were given to different assessments on the same question by one respondent, the average was calculated and rounded up to the nearest integer in order to retain the ordinality of data. The medians for all questions were tested for deviation from value 3 with Wilcoxon signed-rank test and the differences between respondent groups for each question were analyzed with an ordered logit model. In addition, the respondent averages were calculated for questions 3-6 (applicability) and questions 1-7 (effectiveness) using all existing values and omitting missing values. These averages were interpreted as continuous and normally distributed variables, and for them the differences between groups were tested by one-way ANOVA and Tukey post-hoc test. The significance level in all tests was set to p < 0.05. | ||
| + | |||
| + | add code + analysis results | ||
| + | |||
| + | === Summary of evaluation results === | ||
| + | |||
| + | Finally, the textual comments provided with the evaluation, the informal communications during the assessments and the subjective experiences among the assessment coordinators were scrutinized in light of the questionnaire analysis results and compiled as an overall evaluation of the assessment effectiveness in these two cases. | ||
| + | |||
| + | summary to be added | ||
Revision as of 07:17, 19 June 2012
Contents
to do list
Things to be presented on this summary page in order to be in line with what is said in the biodiesel assessments -article:
- example of the discussions (formalized presentations of participant comments) (Vilma)
- (anonymized) numerical questionnaire results (Teemu)
- R-code to run the analyses of numerical questionnaire data + results ((Teemu)
- summary of the overall evaluation (Mikko)
- checking/proofreading of the summary of the assessments and their results as well as discussion contents (Mikko)
Invited stakeholders
Environmental non-governmental organizations
- Greenpeace
- WWF Finland
- The Finnish Association for Nature Conservation
- Dodo ry
- Friends of the Earth Finland
Expert organisations
- National Institute for health and welfare (THL)
- Finnish Environment Institute (SYKE)
- Bioteknologia.info
- Motiva Ltd
- MTT Agrifood Research Finland
- Finnish Game and Fisheries Research Institute
- Ministry of the environment
- Finnish energy industries
- VTT Technical Research Centre of Finland
- Paula Tommila (Researcher, specialist in Jatropha)
Human rights organisations
- Amnesty International Finland
Energy corporations
- Neste Oil
- St1 Biofuels Oy
- Vapo biofuels
Examples of formalized discussions
'Tähän pari keskustelusivulle laitettavaa edustavaa esimerkkiä, esim. yksi kommentti + siihen liittyvät argumentit (op_fi:Keskustelu:Jatropan käyttö bioenergian lähteenä tai op:fi:Keskustelu:Kalanjalostuksen öljyjäte bioenergian lähteenä) sekä yksi jonkin muuttujan keskustelusivulle kirjattu keskustelulaatikko (väittämät, ratkaisu, argumentaatio).
Evaluation of assessments
Questionnaire
After finishing the assessment, participants were contacted again and they were asked to evaluate the performance of the two biofuel assessments. All of the invited stakeholder groups, primary users of the assessments (Neste Oil), summer trainees who worked with the assessment as well as the coordinators at THL were contacted by e-mail and asked to give numerical evaluations of certain questions of a questionnaire on a scale 1 - 5 (1 meaning bad and 5 meaning good). They were also asked to consider and argue the positive and negative aspects of the assessments and provide textual comment to accompany the numerical values. Both assessments (Jatropha and fish waste) were analysed together but the participants were told to specify if the answers differed in the two assessments. The questions were:
Impact of the participation (Q1 only for the stakeholders)
- 1. Do You feel that Your contribution has been included in the assessment fairly?
The performance of the assessments (Q2-Q6 for all)
- 2. Does the assessment content correspond to the research question accurately, truthfully and comprehensively?
- 3. How well does the information provided by the assessment serve Your needs (or the needs of Your organization)?
- 4. Has the information provided by the assessment reached You and Your organisation?
- 5. Did Your understanding increase about the issue along with the assessment?
- 6. Is the assessment result (output), and the way it is obtained and delivered for use acceptable?
Efficiency (Q7 only for the primary users, here Neste Oil)
- 7. How good is the assessment output in relation to the resources used?
Questionnaire results and statistical analysis
Results
Analyses
In the two instances (users) where different numerical values were given to different assessments on the same question by one respondent, the average was calculated and rounded up to the nearest integer in order to retain the ordinality of data. The medians for all questions were tested for deviation from value 3 with Wilcoxon signed-rank test and the differences between respondent groups for each question were analyzed with an ordered logit model. In addition, the respondent averages were calculated for questions 3-6 (applicability) and questions 1-7 (effectiveness) using all existing values and omitting missing values. These averages were interpreted as continuous and normally distributed variables, and for them the differences between groups were tested by one-way ANOVA and Tukey post-hoc test. The significance level in all tests was set to p < 0.05.
add code + analysis results
Summary of evaluation results
Finally, the textual comments provided with the evaluation, the informal communications during the assessments and the subjective experiences among the assessment coordinators were scrutinized in light of the questionnaire analysis results and compiled as an overall evaluation of the assessment effectiveness in these two cases.
summary to be added
