Difference between revisions of "Evaluating performance of environmental health assessments"

Latest revision as of 16:20, 4 March 2015

This page is the final manuscript of Pohjola, M.V.; Pohjola, P.; Tainio, M.; Tuomisto, J.T. Perspectives to Performance of Environment and Health Assessments and Models—From Outputs to Outcomes? Int. J. Environ. Res. Public Health 2013, 10:2621-2642. http://dx.doi.org/10.3390/ijerph10072621. Originally, the text was written on page heande:Evaluating performance of environmental health assessments.

Keywords

Evaluation, EMS, Effectiveness, Open Assessment, REA, Outcome

Highlights

• Conventional evaluation of models and assessments focuses on processes and outputs
• Recent evaluation approaches also emphasize societal model/assessment outcomes
• Model/assessment effectiveness is the likelihood of delivering intended outcomes
• An outcome-oriented turn is taking place in modelling, assessment and evaluation
• New views combine design, making and evaluation of models, assessments and practices

Abstract

This paper reviews different perspectives to evaluation and management of model and assessment performance. While there is an increasing need to evaluate the success of environment and health relevant models and assessments according to their societal outcomes, most perspectives are primarily focused on the outputs of modelling and assessment as well as the procedures of their making. The practical application of outputs and the consequential societal outcomes, are most often left without explicit consideration. Only quite recently, the approaches to evaluation and management of model and assessment performance have recognized the outcomes as an essential performance criterion.

The perspectives to model and assessment performance can be categorized according to their foci as i) quality assurance/control, ii) uncertainty analysis, iii) technical assessment of models, iv) effectiveness and v) other perspectives. In practice many approaches found in the literature combine characteristics of different perspectives, but the categorization, however, illustrates which aspects are primarily emphasized.

It seems that the existing approaches and perspectives do not sufficiently serve all needs of evaluation and management of model and assessment performance. We need a more comprehensive approach that covers a) the making of models and assessments, b) their outputs, c) their use and other interaction with the societal context, as well as d) the decisions and actions influenced by the models and assessments, and e) the consequential outcomes. Achieving this necessitates a thorough account of the mechanisms of collective knowledge creation and the relations between knowledge and action - aspects that are often not recognized as relevant to evaluation and management of model and assessment performance. Some attempts to combine certain collaborative and pragmatic methods, frameworks and tools into a comprehensive methodology for design, making and evaluation of models, assessments as well as practices are being made, and the early results look promising.

Introduction

In this paper we review different perspectives to evaluation and management of model and assessment performance. While there is an increasing need to evaluate the success of models, modelling, and assessment upon issues of environment and health according to their societal outcomes, most evaluation approaches do not seem to support this need. In a recent thematic issue on the assessment and evaluation of environmental models and software (McIntosh et al., 2011a), Matthews et al. (2011) opened up a debate on how to best respond to the increasing desire to evaluate the success of environmental modelling and software projects in terms of their outcomes, i.e changes to values, attitudes, and behaviour outside the walls of the research organization, rather than mere outputs. Until now, there has been limited appreciation within the environmental modelling and software community regarding the challenges of shifting the focus of evaluation from outputs to outcomes (Matthews et al., 2011).

The current situation in the different overlapping communities of environment and health related assessment, such as integrated assessment (e.g. van der Sluijs, 2002), health impact assessment (e.g. WHO, 1999), risk assessment (e.g. NRC, 1983, 1996, 2009), chemical safety assessment (ECHA, 2008), environmental impact assessment (e.g. Wood, 1995), and integrated environmental health impact assessment (Briggs, 2008), appears to be somewhat similar. For example, a recent study on the state of the art in environmental health assessment revealed that despite the fact that most assessment approaches explicitly state influencing of society as their aim, this is rarely manifested in the principles and practices of evaluating assessment performance (Pohjola et al., 2012).

The emphasis in the scientific discourses on evaluating models and assessments has been on rather scientific and technical aspects of evaluation within the research domain, and perspectives that also duly address the impacts of modelling and assessment in broader societal contexts have emerged only quite recently and are still relatively rare (cf. McIntosh et al., 2011b). Such evaluations are qualitatively different (Matthews et al., 2011), which invokes a need for reconsidering the criteria for evaluating modelling and assessment endeavours, as well as the frameworks within which they are applied.

Evaluation of models and assessments is not only a matter of judging the goodness of a model or assessment, but has a more profound influence on their use as well as making. On the one hand, evaluation informs users e.g. regarding the possible and appropriate uses of models and assessment or model outputs. On the other hand, the criteria for evaluations also tend to direct the design and execution of modelling and assessment endeavours to meet those criteria, particularly if they are explicitly known in advance (cf. What you measure is what you get (WYMIWYG) by Hummel and Huitt, 1994).

This paper contributes to the debate by first describing a framework within which model and assessment performance can be considered, then reviewing different perspectives to model and assessment performance, and finally discussing the capability of existing approaches to address the challenges of what could be called the "outcome-oriented turn in evaluation and management of models and assessments". Some current attempts to develop a comprehensive approach to design, making and evaluation of models, assessments as well as practices are also briefly presented.

Models, assessments and performance

For several years, the authors have been engaged with developing methods and tools for environmental health modelling and assessment. Most notably this work has resulted in the open assessment method and Opasnet web-workspace for science-based collaborative knowledge creation and policy support upon virtually any topic (Pohjola et al. 2011, 2012, Tuomisto and Pohjola 2007, http://en.opasnet.org). These developments have required thorough familiarization of and several inquiries into the kinds of existing modelling and assessment means and practices, and particularly the factors that determine how well they perform. This review is based on the extensive information and knowledge base on model and assessment performance that has been obtained by these inquiries.

Here modelling and assessment are considered as instances of fundamentally the same issue of science-based support to decision making regarding environment and health. In fact, assessments virtually always involve modelling of some kind, at least implicit conceptual models. Conversely, modelling is also often identified with assessment (Jakeman and Letcher, 2003). In addition, decision support systems, information support tools, integrated modelling frameworks and other software tools and information systems to assist in developing, running, and analyzing models and assessments are perceived as integral parts of modelling and assessment (e.g. Rizzoli et al., 2008).

Models and assessments can be considered e.g. as diagnostic, prognostic, or summative according to the kinds of questions they address (Briggs, 2008), ex-ante or ex-post according to their timing in relation to the activities being assessed (Pope et al., 2004), and regulatory or academic according to the contexts of their development and application (Pohjola et al., 2012). They can also be developed, executed, and applied by many kinds of actors, e.g. consultants, federal agencies or academic researchers. Modelling and assessment should be, however, clearly distinguished from purely curiosity-driven research, ad hoc assessments, and models or assessments made only to justify predetermined decisions. In addition, it should be noted that "assessment of models", often interchangeable with "evaluation of models", is also excluded from the concept of assessment adopted here. Instead, technical assessment of models is considered in the review below as one of the common perspectives to evaluating model and assessment performance. Altogether, modelling and assessment can be considered as fundamentally having two purposes: i) describing reality, and ii) serving the needs of practical decision-making.

Inspired by the conceptual framework presented by Matthews et al. (2011), and influenced by the works of e.g. Blackstock et al. (2007), Bina (2008), Leviton (2003), and Patton (2002), Figure 1 describes the essential aspects in the interaction of modelling and assessment with their broader societal context.

The endeavours of modelling and assessment are here broken down into 1) process, the procedures and practices of developing and executing models and assessments, 2) output, the products or results that come out of these modelling and assessment processes, and 3) use, the practical application of the outputs. The surrounding context provides enablers for modelling and assessment endeavours to take place e.g. in the form of funding, facilities and education, but also as acceptance or demand for certain kinds of modelling and assessment endeavours. On the other hand, the context also poses constraints e.g. in the form of prioritization of scarce resources, non-acceptance of some modelling and assessment endeavours, or lacking of capability for applying their outputs. The societal context is also the medium where the outcomes of modelling and assessment, i.e. the changes in values, attitudes and behaviour among others than assessors and modellers themselves are (or are not) realized. It is important to distinguish the outcomes from what Matthews et al. (2011) termed process effects, i.e. the changes in the capacity of those engaged in the modelling and assessment endeavours, which does not directly result in impacts in the broader societal context.

In Figure 1, use is intentionally located on the boundary between the modelling and assessment domain and the context in order to emphasize that the practical application of outputs should be considered as a shared responsibility between the modellers or assessors and the intended users outside the modelling and assessment domain. This implies that the responsibility of modellers and assessors should not end in completion of the output. On the other hand, the users of the outputs should not expect realization of the outcomes without having to invest any effort themselves in their delivery. The boundary between the modelling and assessment domain and its context is marked with a dashed line, indicating that the domain should not be perceived as a strictly separate entity, but in continuous interaction with its context.

The case-specific uses of modelling and assessment outputs in varying contexts are critical for achieving the intended outcomes of models and assessments. Therefore, in order for models or assessments to perform well, it is not sufficient to only provide good descriptions of reality. They must also fulfil their instrumental purposes of serving the needs of their intended uses, and thereby resulting in the desired outcomes. However, this aspect of model and assessment performance is often given little attention, as will turn out when the existing perspectives to model and assessment performance are reviewed and discussed in light of the framework described in Figure 1.

Perspectives to model and assessment performance

Within the abundance of contributions in the scientific literature addressing model and assessment performance, certain commonalities between contributions can be identified. These contributions can be categorized as representing different perspectives according to their primary focus: i) quality assurance/control, ii) uncertainty analysis, iii) technical assessment of models, iv) effectiveness, or v) other perspectives. These perspectives provide the structure for the review. As the number of different relevant contributions is so large that it is not be possible to include them all in the review, we have instead attempted to compile representative and comprehensive collections of examples, grouped according to certain subcategories, for each perspective. Recent contributions are emphasized, but some important or illustrative examples that were published before 2000 have been included as well. Also the relevant contributions stemming from the method and tool development by the authors are also included in the review.

Quality assurance/control

One of the major themes in modelling and assessment performance related literature is what can be referred to as quality assurance/control (QA/QC) perspective. The focus in this perspective is primarily on determining how the processes of modelling or assessment are to be conducted in order to assure the quality of the output. The QA/QC perspective appears commonly in both modelling and assessment literature, sometimes extending to also consider the processes of decision-making.

There are multiple alternative definitions for quality (see e.g. Reeves and Bednar, 1994). However, as regards modelling and assessment, the interpretation is mostly analogous with the perception in the ISO-9000 framework, i.e. as the organisational structures, responsibilities, procedures, processes, and resources to assure and improve quality (Harteloh, 2003). Also the hierarchy of evidence, commonly applied in the field of medicine, is a quality assurance/control perspective, which ranks types of evidence strictly according to the procedure by which they were obtained (Guyatt et al., 1995). However, as pointed out by Cartwright (2007) with regard to randomized controlled trials, the procedure alone cannot guarantee delivery of useful information in practical contexts.

One common variation of this perspective is stepwise procedural guidance. Such guidance provides often relatively strict and detailed descriptions of the steps or phases of a modelling or assessment process that are to be executed in a more or less defined order. Faithful execution of the procedure is assumed to lead to good outputs. A somewhat similar, but perhaps less rigorous, variation of the QA/QC perspective is checklist-type guidance emphasizing issues that need to be taken account of in the modelling or assessment process or their evaluation. The checklists can be more or less detailed and they usually do not strictly define the order or sequence of execution.

Also the accounts that address evaluation of input quality can be considered as manifestations of the QA/QC perspective. However, the primary focus in QA/QC is often on the outputs, and the input quality evaluations typically complement uncertainty analyses or technical assessments of models. For example, model parameter uncertainty analysis can be considered as an example of evaluation of input quality, but in practice it is most often considered as an aspect of either uncertainty analysis or technical assessment of models.

Characteristic for stepwise guidance is that it attempts to predetermine a procedure in order to guarantee good quality of outputs. As such, it takes a proactive approach to managing performance. Checklist-type guidance and evaluation of input quality can also be applied proactively, but the examples found in literature mostly represent a reactive approach of evaluating already executed modelling and assessment processes.

Uncertainty analysis

Another major theme in the modelling and assessment performance literature can be referred to as the uncertainty analysis perspective. The contributions within this theme vary significantly, ranging from descriptions of single methods to overarching frameworks, but the common main idea is characterization of certain properties of the outputs of models and assessments. Fundamentally the perspective builds on quantitative statistical methods based on probability calculus (O'Hagan, 2011), but also other than probability-based approaches to uncertainty have been presented (e.g. Colyvan, 2008). Many manifestations of this perspective in the context of environment and health modelling and assessment also extend to consider qualitative properties of the outputs. The uncertainty analysis perspective appears to be more commonly represented in the domain of assessment than of modelling. In addition to model and assessment outputs, models themselves could be considered as outputs whose properties are considered and evaluated in a similar fashion. However, this issue is addressed as a separate perspective in the next section, technical assessment of models.

One variation of the uncertainty analysis perspective is identification and typifying of the kinds and sources of uncertainty that characterize the modelling or assessment outputs and their quality. Some uncertainties are often considered as being primarily expressible in quantitative, while others in qualitative terms. The sources of uncertainty can also extend to include aspects of the modelling and assessment processes. In some cases also intended or possible uses and use contexts of the outputs are acknowledged.

Also different types of guidance on how to assess or deal with different kinds of uncertainties exist. Such frameworks usually combine both qualitative and quantitative aspects of uncertainty deriving from various sources. Consequently, aspects of the processes of producing the outputs, e.g. input quality, and acknowledgment of the intended or possible uses and use contexts of the outputs, e.g. in terms of acceptance, are often also included in the frameworks. The primary focus is, however, on the characteristics of the model and assessment outputs.

In addition to the broad characterizations and frameworks, also numerous more or less explicit methods, means and practices to analyze uncertainties of model and assessment outputs exist. For example, sensitivity, importance, and value of information analysis and Bayesian modelling, in addition to the standard statistical characterization, are essential in the context of environment and health modelling and assessment. Such methods are dominantly quantitative. The standard means for statistical analysis are considered common knowledge and not discussed here in any more detail.

For the examples of uncertainty analysis perspective, it appears characteristic that the issue of uncertainty is typically approached from an external observer’s point of view. The evaluation of performance is thus mainly considered as a separate, typically ex-post, activity taking place in addition to the actual modelling or assessment process, not as its integral proactive part.

Technical assessment of models

If the uncertainty analysis perspective is more commonly adopted in assessment literature, the technical assessment of models is a grand theme particularly in the modelling literature. In contrast to the focus on model or assessment outputs in the uncertainty analysis approach, the focus in this perspective is on the models and their characteristics. In addition to models, the perspective also considers different kinds of software tools that are applied in developing, running, and analyzing the models.

Particularly the object of interest in technical assessment of models is development and application of formal methods for testing and evaluating models within defined domains of application. Generally, model evaluation and performance is considered to cover structural features of models, representativeness of model results in relation to a certain part of reality, as well as usefulness with regard to a designated task (cf. Beck, 2002). However, if usefulness is considered at all, it mainly addresses expert use of models, which corresponds mostly to the so-called process effects rather than outcomes. Most commonly technical assessment of models takes place in terms of validation and verification by comparing models and their results against each other or measurement data.

A variation of this perspective, more common for the discourses in assessment literature, is analysis of model uncertainty. Here the aim typically is to characterize the properties of a model in order to be able to correctly interpret or evaluate its outputs. Model uncertainty is often considered as one aspect of a broader uncertainty concept.

The technical assessment of models is predominantly a reactive perspective to evaluate models, as it requires an existing model or software system that can be tested and analyzed. The evaluation, however, is usually considered as being an integral part of the model development, not a separate entity, which also enables application of technical assessment of models in different developmental stages within the modelling or assessment process. On the other hand, the common practice of self evaluation of models may also lead e.g. to limited usability, credibility and acceptability unless interaction with the context is otherwise realized.

Effectiveness

Whereas the three former perspectives have been subjects of discussion regarding model and assessment performance for longer times, emphasizing of model and assessment effectiveness can be considered to have become a major topic only quite recently. Contributions addressing effectiveness appear more common in the assessment literature, but the topic has recently been addressed also in the context of modelling and software.

In the effectiveness perspective, the aim of modelling and assessment is generally seen as to promote changes in values, attitudes, and behaviour outside the walls of the research community (Matthews et al., 2011) by maximizing the likelihood of an assessment process to achieve the desired results and the goals set for it (Hokkanen and Kojo, 2003). In principle, performance of models and assessments is thus fundamentally characterized in terms of the impacts delivered into the broader societal context. However, due to the complexity of reality, evaluation of outcomes is often perceived as very difficult, if not impossible (Kauppinen et al., 2006), and possibly even leading to incorrect conclusions regarding effectiveness (cf. Ekboir, 2003). Consequently, the effectiveness criteria and frameworks often address various aspects of process and output, as well as contextual enablers and constraints, rather than outcomes, as factors of effectiveness. Some contributions also make a distinction between (immediate) impacts and (indirect) outcomes. As a result, although the aim is to address outcomes, some approaches to effectiveness in the end turn out somewhat similar to those considered as checklist guidance in quality assurance/control (in Table 1).

In addition, the approaches emphasizing the use of models, tools and their outputs can be considered as a manifestation of the effectiveness perspective. They can generally be characterized as attempts to operationalise the interaction of modelling and assessment with the practical uses of their outputs. Most of the contributions are, however, relatively tool-centred (cf. evaluation of decision support systems as models), and most often little attention is given to the cognitive processes involved in the delivery and reception of information produced by models and assessments.

The approaches to effectiveness range from external ex-post evaluations to support for development and management of modelling, assessment as well as decision making practices. All approaches, however, explicitly acknowledge the role of use in delivering the effects of knowledge provided by models and assessment, despite that the criteria against which effectiveness is considered may vary.

Other perspectives

Although many contributions to modelling and assessment performance in relevant literature can be quite comfortably located within the four perspectives above, there are also some other aspects that deserve to be mentioned. Those to be brought up here address credibility and acceptability, information quality, and communication.

Credibility is often considered necessary for acceptance of modelling and assessment endeavours and their outputs. It can be obtained more or less formally or informally e.g. through peer review, extended peer-review (Funtowicz and Ravetz, 1990) or reputation. Credibility and acceptability are often considered as aspects of broader performance concepts.

Modelling and assessment are essentially processes of producing information. Therefore, the contributions regarding information quality also outside the domains of modelling and assessment are of relevance here. This perspective resembles the uncertainty analysis perspective as they both focus into certain properties of an information product. Similarly, the variation among contributions addressing information quality is big.

Also communication of results, e.g. in terms of communicating uncertainties and risk information, is linked to performance of models and assessments. However, the issues of communication are not necessarily considered as integral parts of modelling and assessments endeavours. For example, risk assessment, risk management and risk communication are traditionally considered as separate, yet interrelated, entities, each having their own aims, practices, and practitioners (e.g. WHO/FAO, 2006).

Discussion

By considering the reviewed contributions in light of the framework described in Figure 1, it seems that none of the perspectives nor any individual contributions alone sufficiently serve all the needs of evaluation and management of model and assessment performance. In most of the contributions, the main emphasis is on the processes and outputs of modelling and assessment while contextual aspects, outcomes, as well as use are addressed to a lesser extent, although more frequently in recent literature. The accounts of interaction of modelling and assessment with their societal context also appear to be vaguer in comparison to the commonly applied perspectives focusing on processes and outputs. No fundamental differences in perspectives to performance between domains of modelling and assessment can be seen.

Many approaches to performance seem to perceive evaluation as a separate entity that most often takes place only after the development of a model or assessment, and often is considered a responsibility of others than modellers and assessors themselves. Some major exceptions to this are the essentially proactive stepwise guidance in the quality assurance/control perspective, and the technical assessment of models perspective, in which the evaluation is often integrated in the model development. In addition, some of the effectiveness frameworks are explicitly intended as means to support design and execution, not only evaluation, of models and assessments.

The emphasis on processes and outputs in evaluation and management of model and assessment performance is in line with the fact that e.g. the issues of effectiveness and policy-relevance have become major topics also in modelling and assessment only during the last decades. As modellers, assessors, and researchers more generally, have been lacking requirements and incentives for effectiveness and policy-relevance (cf. Harris, 2002), correspondingly the practices, principles and methods of performance management and evaluation have not developed to address these issues. Instead, the impacts of modelling and assessment have mostly been considered mainly in terms of their process effects (cf. Matthews et al., 2011) within the communities of modellers and assessors, rather than outcomes in the broader societal context. Virtually all modelling and assessment endeavors in the fields of environment and health are, however, at least nominally, motivated by the aim to influence societal decision-making. The societal outcomes should thus be considered as the ultimate criterion for model and assessment performance. This has also been recognized in many new approaches to modeling, assessment as well as their evaluation (e.g. Tijhuis et al., 2012; Matthews et al., 2011). However, the complexity of addressing the outcomes remains a challenge. In the eyes of the evaluators, the relative simplicity of considering only processes, outputs or direct impacts in tightly bound settings of expert activities may appear inviting in comparison to attempting to account for complex indirect impacts within the broader social context. Unfortunately, this may not be adequate for serving the purposes of modelling, assessment and their evaluation.

It appears that a more comprehensive approach that covers all aspects of modelling and assessment in their societal context, as described in Figure 1, is needed to support model and assessment evaluation and management. In practice, this requires taking account of the making of models and assessments, their outputs, their use and other interaction with the societal context, as well as the decisions and actions influenced by the models and assessments, and the consequential outcomes. Such an approach would need to combine the essential characteristics of the different perspectives into one methodology, framework, or tool. However, a mere compilation of features taken from different perspectives would probably not be sufficient. A more thorough account of the mechanisms of collective knowledge creation and the relations between knowledge and action in a societal context is needed in order to truly bridge models and assessments with their outcomes (Pohjola et al., 2011). Unfortunately these aspects are barely even recognized in most current approaches to model and assessment performance.

Despite that the contributions within the effectiveness perspective were above characterized as often being somewhat vague, it seems likely that the effectiveness perspective provides the best basis for a comprehensive approach as the concept brings together the making, content, use as well as outcomes of models and assessments. Attempts to combine methods, frameworks and tools into a comprehensive methodology for design, making and evaluation of models, assessments as well as practices are being made, and the early results look promising. For example, some of the above mentioned effectiveness frameworks have been jointly applied for evaluating the effectiveness of two open assessments on alternative biofuel sources (Sandström et al. manuscript, http://en.opasnet.org/w/Biofuel_assessments) as well as analysing how different health, safety and environment assessment approaches serve the needs of decision making in public policy as well as manufacturing industry (Pohjola manuscript). A more noteworthy effort in this aspect is the ongoing TEKAISU-project (http://en.opasnet.org/w/Tekaisu) in which methods and tools for evaluation and management of knowledge-based city-level decision making are developed and applied. This development combines four above mentioned effectiveness frameworks by Kauppinen et al. (2006), Tuomisto and Pohjola (2007), Pohjola and Tuomisto (2011), and Pohjola et al. (2012) with a theoretical framework and practical tool for evaluating and managing practices in the social and health field by Koivisto and Pohjola (2012, https://pilotointi.innokyla.fi). All these combined frameworks and tools essentially build on collaboration and pragmatism, and thus emphasize the interrelations between knowledge and action in a social context. Consequently, the approach does not focus primarily on any separate aspect, process, output, use or outcome, of model and assessment performance, but considers and addresses the knowledge that is created, transferred and applied within the intertwined processes of modelling, assessment and decision making. This, we believe, is the key to making the whole chain from design of models and assessments to societal outcomes of decisions and actions both evaluable and manageable.

The emphasis in evaluation and management of models and assessments has been on rather scientific and technical aspects within the research domain, and perspectives addressing the impacts of models and assessments in terms of societal outcomes have emerged only quite recently and are still relatively rare. Still, it can be said that an outcome-oriented turn is taking place in evaluation and management of models and assessments, at least in development of theories and related academic discourses. However, a lot of work still remains in developing the frameworks, methods, tools and ultimately the common practices of modelling and assessment to sufficiently serve the needs of practical decision making for sustainable and healthy future. Succeeding in this will require better recognition of the mechanisms how models and assessments influence the knowledge, decisions and actions that deliver the consequences.

Acknowledgements

This review builds on research done in several projects receiving funding from various sources. Most importantly the authors would like to mention the EU projects INTARESE (Integrated Assessment of Health Risks of Environmental Stressors in Europe, 2005-2011, GOCE-CT-2005-018385), and BENERIS (Benefit–Risk Asessment of Food: An iterative Value-of-Information approach, 2006–2009, FOOD-CT-2006-022936), Safefoodera project BEPRARIBEAN (Best Practices for Risk-Benefit Analysis of Foods, project ID 08192), Academy of Finland (Grants 218114 and 126532), and SYTYKE doctoral programme in environmental health at the University of Eastern Finland.

References

Aertsen, W., Kint, V., van Orshoven, J., Muys, B., 2011. Evaluation of modelling techniques for forest site productivity prediction in contrasting ecoregions using stochastic multicriteria acceptability analysis (SMAA). Environmental Modelling & Software 26, 929-937.

Alexandrov, G.A., Ames, D., Bellochi, G., Bruen, M., Crout, N., Erechtchoukova, M., Hildebrandt, A., Hoffman, F., Jackisch, C., Khaiter, P., Mannina, G., Matsunaga, T., Purucker, S.T., Rivington, M., Samaniego, L., 2011. Technical assessment and evaluation of environmental models and software: Letter to the editor. Environmental Modelling & Software 26, 328-336.

van Asselt, M.B.A., Rotmans, J., 2002. Uncertainty in integrated assessment modelling: From positivism to Pluralism. Climatic Change 54, 75-105.

Aumann, C.A., 2011. Constructing model credibility in the context of policy appraisal. Environmental Modelling & Software 26, 258-265.

Bai, Y., Wagener, T., Reed, P., 2009. A top-down framework for watershed model evaluation and selection under uncertainty. Environmental Modelling & Software 24, 901-916.

Baker, D.C., McLelland, J.N., 2003. Evaluating the effectiveness of British Columbia's environmental assessment process for first nations' participation in mining development. Environmental Impact Assessment Review 23, 581-603.

Beck, B., 2002. Model evaluation and performance, in: El-Shaarawi, A.H., Piegorsch, W.W. (Eds.), Encyclopedia of Environmetrics. John Wiley & Sons Ltd, Chichester, Volume 3, pp. 1275-1279.

Basson, L., Petrie, J.G., 2007. An integrated approach for the consideration of uncertainty in decision making supported by Life Cycle Assessment. Environmental Modelling & Software 22, 167-176.

Bina, O., 2008. Context and systems: Thinking more broadly about effectiveness in strategic environmental assessments in China. Environmental Management 42, 717-733.

Bischof, N., Eppler, M.J., 2011. Caring for clarity in knowledge communication. Journal of Universal Computer Science 17, 1455-1473.

Blackstock, K.L., Kelly, G.J., Horsey, B.L., 2007. Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics 60, 726-742.

Blind, M.W., Refsgaard, J.C., 2007. Operationalising uncertainty in data and models for integrated water resource management. Water, Science & Technology 56, 1-12.

Borgonovo, E., 2008. Sensitivuty analysis of model output with input constraints: A generalized rationale for local methods. Risk Analysis 28, 667-680.

Briggs, D.J., 2008. A framework for integrated environmental health impact assessment of systemic risks. Environmental Health, 7:61. doi:10.1186/1476-069X-7-61

Briggs, D.J., Sable, C.E., Lee, K., 2009. Uncertainty in epidemiology and health risk and impact assessment. Environmental Geochemistry and Health 31, 189-203.

Brouwer, R., De Blois, C., 2008. Integrated modelling of risk and uncertainty underlying the cost and effectiveness of water quality measures. Environmental Modelling & Software 23, 922-937.

Brown, J.D., Heuvelink, G.B., Refsgaard, J.C., 2005. An integrated methodology for recording uncertainties about environmental data. Water, Science & Technology 52, 153-160.

Cartwright, N., 2007. Are RCTs the gold standard?. BioSocieties 2, 11-20. doi:10.1017/S1745855207005029

Clark, W.C., Majone, G., 1985. The critical appraisal of scientific inquiries with policy implications. Science, Technology & Human Values 10, 6-19.

Colyvan, M., 2008. Is probability the only coherent approach to uncertainty?. Risk Analysis 28, 645652

Covello, V.T., Peters, R.G., Wojtecki, J.G., Hyde, R.C., 2001. Risk communication, the west nile virus epidemic, and bioterrorism: Responding to the communication challenges posed by the intentional or unintentional release of a pathogen in an urban setting. Journal of Urban Health: Bulletin of the New York Academy of Medicine 78, 382-391.

van Delden, H., Seppelt, R., White, R., Jakeman, A.J., 2011. A methodology for the design and development of integrated models for policy support. Environmental Modelling & Software 26, 266-279.

Dewar, J.A., Bankes, S.C., Hodges, J.S., Lucas, T., Saunders-Newton, D.K., Vye, P., 1996. Credible uses of the distributed interactive simulation (DIS) system. RAND, Santa Monica, CA.

Diez, E., McIntosh, B.S., 2011. Organisational drivers for, constraints on and impacts of decision and information support tool use in desertification policy and management. Environmental Modelling & Software 26, 317-327.

ECHA, 2008. Guidance on Information Requirements and Chemical Safety Assessment. Guidance for the implementation of REACH. European Chemicals Agency.

Ekboir, J., 2003. Why impact analysis should not be used for research evaluation and what the alternatives are. Agricultural Systems 78, 166-184.

Fischer, T.B., Gazzola, P., 2006. SEA effectiveness criteria - equally valid in all countries? The case of Italy. Environmental Impact Assessment Review 26, 396-409.

Forristal, P.M., Wilke, D.L., McCarthy, L.S., 2008. Improving the quality of risk assessments in Canada using a principle-based apporach. Regulatory Toxicology and Pharmacology 50, 336-344.

Funtowicz, S.O. and J.R. Ravetz 1990. Uncertainty and Quality in Science for Policy. Kluwer Academic Publishers, the Netherlands.

Granger Morgan, M., Dowlatadabi, H., 1996. Learning from integrated assessment of climate change. Climatic change 34, 337-368.

Guyatt, G.H., Sackett, D.L., Sinclair, J.C., Hayward, R., Cook, D.J., Cook, R.J., 1995. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA 274, 1800–1804.

Harris, G., 2002. Integrated assessment and modelling: an essential way of doing science. Environmental Modelling & Software 17, 201-207.

Harteloh, P.P.M., 2002. Quality systems in health care: a sociotechnical approach. Health Policy 64, 391-398.

Hildén, M., Furman, E., Kaljonen, M., 2004. Views on planning and expectations of SEA: the case of transport planning. Environmental Impact Assessment Review 24, 519-536.

Hoekstra, J., Hart, A., Boobis, A., Claupein, E., Cockburn, A., Hunt, A., Knudsen, I., Richardson, D., Schilter, B., Schütte, K., Torgerson, P.R., Verhagen, H., Watzl, B., Chiodini, A., 2010. BRAFO tiered approach for benefit-risk assessment of foods. Food and Chemical Toxicology, In press. doi:10.1016/j.fct.2010.05.049

Hokkanen, P., Kojo, M., 2003. How environmental impact assessment influences decision-making [in Finnish]. Ympäristöministeriö, Helsinki.

Hummel, J., Huitt, W., 1994. What you measure is what you get. GaASCD Newsletter: The Reporter, 10-11. Retrieved 13.10.2011 from http://www.edpsycinteractive.org/papers/wymiwyg.html

Inman, D., Blind, M., Ribarova, I., Krause, A., Roosenschoon, O., Kassahun, A., Scholten, H., Arampatzis, G., Abrami, G., McIntosh, B., Jeffrey, P., 2011. Perceived effectiveness of environmental decision support systems in participatory planing: Evidence from small groups of end-users. Environmental Modelling & Software 26, 302-309.

Jakeman, A.J., Letcher, R.A., 2003. Integrated assessment and modelling: features, principles and examples for catchment management. Environmental Modelling & Software 18, 491-501.

Jakeman, A.J., Letcher, R.A., Norton, J.P., 2006. Ten iterative steps in development and evaluation of environmental models. Environmental Modelling & Software 21, 602-614.

Janssen, P.H.M., Petersen, A.C., van der Sluijs, J.P., Risbey, J.S., Ravetz, J.R., 2005. A guidance for assessing and communicating uncertainties. Water Science & Technology 52, 125-131.

Kann, A., Weyant, J.P., 2000. Approaches for performing uncertainty analysis in large-scale energy/economic policy models. Environmental Modeling and Assessment 5, 29-46.

Kauppinen, T., Nelimarkka, K., Perttilä, K., 2006. The effectiveness of human impact assessment in the Finnish Health Cities Network. Public Health 120, 1033-1041.

Kloprogge, P., van der Sluijs, J.P., Petersen, A.C., 2011. A method for the analysis of assumptions in model-based environmental assessments. Environmental Modelling & Software 26, 289-301.

Koivisto, J., Pohjola, P., 2012. Practices, modifications and generativity - REA: a practical tool for managing the innovation processes of practices. Systems, Signs & Actions 5(1), 100-116. http://www.sysiac.org/uploads/SySiAc2011-Koivisto-Pohjola.pdf

Larocque, G.R., Bhatti, J.S., Ascough II, J.C., Liu, J., Luckai, N., Mailly, D., Archambault, L., Gordon, A.M., 2011. An analytical framework to assist decision makers in the use of forest ecosystem model predictions. Environmental Modelling & Software 26, 280-288.

Leu, W., Williams, W.P., Bark, A.W., 1996. Development of an environmental impact assessment evaluation model and its application: Taiwan case study. Environmental Impact Assessment Review 16, 115-133.

Leviton, L.C., 2003. Evaluation use: Advances, challenges and applications. American Journal of Evaluation 24, 525-535.

Liu, Y., Gupta, H., Springer, E., Wagener, T., 2008. Linking science with environmental decision making: Experiences from an integrated modelling approach to supporting sustainable water resources management. Environmental Modeling & Software 23, 846-858.

Mahmoud, M., Liu, Y., Hartmann, H., Stewart, S., Wagener, T., Semmens, D., Stewart, R., Gupta, H., Dominguez, D., Dominguez, F., Hulse, D., Letcher, R., Rashleigh, B., Smith, C., Street, R., Ticehurst, J., Twery, M., van Delden, H., Waldick, R., White, D., Winter, L., 2009. A formal framework for scenario development in support of environmental decision-making. Environmental Modelling & Software 24, 798-808.

Matthews, K.B., Rivington, M., Blackstock, K.L., McCrum, G., Buchan, K., Miller, D.G., 2011. Raising the bar? - The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling & Software 26 (3), 247-257.

McIntosh, B.S., Giupponi, C., Voinov, A.A., Smith, C., Matthews, K.B., Monticino, M., Kolkman, M.J., Crossman, N., van Ittersum, M., Haase, D., Haase, A., Mysiak, J., Groot, J.C.J., Sieber, S., Verweij, P., Quinn, N., Waeger, P., Gaber, N., Hepting, D., Scholten, H., Sulis, A., van Delden, H., Gaddis, E., Assaf, H., 2008. Bridging the gaps between design and use: Developing tools to support environmental management and policy, in: Jakeman, A.J., Voinov, A.A., Rizzoli, A.E., Chen, S.H. (Eds.): Environmental Modelling, Software and Decision Support. Elsevier, Amsterdam, pp. 33-48.

McIntosh, B.S., Alexandrov, G., Matthews, K., Mysiak, J., van Ittersum, M. (Eds.), 2011a. Thematic issue on the assessment and evaluation of environmental models and software. Environmental Modelling & Software 26, 245-336.

McIntosh, B.S., Alexandrov, G., Matthews, K., Mysiak, J., van Ittersum, M., 2011b. Preface: Thematic issue on the assessment and evaluation of environmental models and software. Environmental Modelling & Software 26, 245-246.

Mo, X., Liu, S., Lin, Z., 2011. Evaluation of an ecosystem model for a wheat-maize double cropping system over the North China Plain. Environmental Modelling & Software, In press. doi:10.1016/j.envsoft.2011.07.002

Moody, D., Walsh, P., 1999. Measuring the value of information: An asset valuation approach. Seventh European Conference on Information System (ECIS’99), Copenhagen Business School, Frederiksberg, Denmark, 23-25 June, 1999. Available: http://wwwinfo.deis.unical.it/zumpano/2004-2005/PSI/lezione2/ValueOfInformation.pdf Accessed 13.10.2011.

Moschandreas, D.J., Karuchit, S., 2002. Scenario-model-parameter: a new method of cumulative risk uncertainty analysis. Environment International 28, 247-261.

Nilsen, T., Aven, T., 2003. Models and model uncertainty in the context of risk analysis. Reliability Engineering & System Safety 79, 309-317.

NRC, 1983. Risk Assessment in the Federal Government: Managing the Progress. The National Research Council. National Academy Press, Washington D.C.

NRC, 1996. Understanding Risk: Informing Decisions in a Democratic Society. The National Research Council. National Academy Press, Washington D.C.

NRC, 2009. Science and Decisions: Advancing Risk Assessment. National Research Council. National Academy Press, Washington D.C.

O'Hagan, A., 2011. Probabilistic uncertainty specification: Overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environmental Modelling & Software, In press. doi:10.1016/j.envsoft.2011.03.003

Patton, M.Q., 2008: Utilization-focused evaluation, fourth edition. SAGE Publications, Inc. MN.

Pohjola, M.V., Leino, O., Kollanus, V., Tuomisto, J.T., Gunnlaugsdόttir, H., Holm, F., Kalogeras, N., Luteijn, J.M., Magnusson, S.H., Odekerken, G., Tijhuis, M.J., Ueland, Ø., White, B.C., Verhagen, H., 2012. State of the art in benefit–risk analysis: environmental health. Food Chem. Toxicol. 2012, 50:40-55. http://dx.doi.org/10.1016/j.fct.2011.06.004

Pohjola, M.V., Pohjola, P., Paavola, S., Bauters, M., Tuomisto, J.T., 2011. Pragmatic Knowledge Services. Journal of Universal Computer Science 17:472-497. http://dx.doi.org/10.3217/jucs-017-03-0472

Pohjola, M.V., Tuomisto, J.T., 2011. Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results. Environmental Health 10, 58. doi:10.1186/1476-069X-10-58

Pohjola, M.V. Assessment of impacts to health, safety, and environment in the context of materials processing and related public policy. Manuscript.

Pollino, C.A., Woodberry, O., Nicholson, A., Korb, K., Hart, B.T., 2007. Parameterisation and evaluation of a Bayesian network for use in an ecological risk assessment.

Pope, J., Annandale, D., Morrison-Saunders, A., 2004. Conceptualising sustainability assessment. Environmental Impact Assessment Review 24, 595-616.

Pölönen, I., Hokkanen, P., Jalava, K., 2011. The effectiveness of the Finnish EIA system - What works, what doesn't, and what could be improved? Environmental Impact Assessment Review 31, 120-128.

Quigley, R.J., Taylor, L.C., 2004. Evaluating health impact assessment. Public Health 118, 544-552.

Reeves, C.A., Bednar, D.A., 1994. Defining quality: Alternatives and implications. Academy of Management Review 19, 419-445.

Refsgaard, J.C., Henriksen, H.J., 2004. Modelling guidelines - terminology and guiding principles. Advances in Water Resources 27, 71-82.

Refsgaard, J.C., Henriksen, H.J., Harrar, W.G., Scholten, H., Kassahun, A., 2005. Quality assurance in model-based water management - review of existing practice and outline of new approaches. Environmental Modelling & Software 20, 1201-1215.

Refsgaard, J.C., van der Sluijs, J.P., Brown, J., van der Keur, P., 2006. A framework for dealing with uncertainty due to model structure error. Advances in Water Resources 29, 1586-1597.

Refsgaard, J.C., van der Sluijs, J.P., Lajer Højberg, A., Vanrolleghem, P.A., 2007. Uncertainty in the environmental modelling process - A framework and guidance. Environmental Modelling & Software 22, 1543-1556.

Risbey, J., Kandlikar, M., Patwardhan, A., 1996. Assessing integrated assessment. Climatic Change 34, 369-395.

Risbey, J. van der Sluijs, J.P., Kloprogge, P., Ravetz, J., Funtowicz, S., Corral Quintana, S., 2005. Application of a checklist for quality assistance in environmental modelling to an energy model. Environmental Modeling and assessment 10, 63-79.

Rizzoli, A.E., Leavesley, G., Ascough II, J.C., Argent, R.M., Athanasiadis, I.N., Brilhante, V., Claeys, F.H.A., David, O., Donatelli, M., Gijsbergs, P., Havlik, D., Kassahun, A., Krause, P., Quinn, N.W.T., Scholten, H., Sojda, R.S., Villa, F., 2008. Integrated modelling frameworks for environmental assessment decision support, in: Jakeman, A.J., Voinov, A.A., Rizzoli, A.E., Chen, S.H. (Eds.): Environmental Modelling, Software and Decision Support. Elsevier, Amsterdam, pp. 101-118.

Sandström, V., Tuomisto, J.T., Majaniemi, S., Rintala, T., Pohjola, M.V. Evaluating effectiveness of open assessments on alternative biofuel sources. Manuscript.

Siebenhüner, B., Barth, V., 2005. The role of computer modelling in participatory integrated assessments. Environmental Impact Assessment Review 25, 367-389.

Skyrme, D.J., 1994. Ten ways to add value to your business. Managing Information 1, 20-25. Available: http://www.skyrme.com/pubs/tenways.htm (accessed 13.10.2011)

Sterk, B., van Ittersum, M.K., Leeuwis, C., 2011. How, when, and for what reasons does land use modelling contribute to societal problem solving?. Environmental Modelling & Software 26, 310-316.

van der Sluijs, J.P., 2002. Integrated Assessment, in: Tolba, M.K. (Ed.), Volume 4, Responding to Global Environmental Change, pp. 250-253, in: Munn, T. (Ed.), Encyclopedia of Global Environmental Change. John Wiley & Sons, Ltd, Chichester.

van der Sluijs, J.P., Craye, M., Funtowicz, S., Kloprogge, P., Ravetz, J., Risbey, J., 2005. Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: The NUSAP system. Risk Analysis 25, 481-492.

van der Sluijs, J.P., Petersen, A.C., Janssen, P.H.M., Risbey, J.S., Ravetz, J.R., 2008: Exploring the quality of evidence for complex and contested policy decisions. Environmental Research Letters 3, 024008. doi:10.1088/1748-9326/3/2/024008

Sojda, R.S., 2007. Empirical evaluation of decision support systems: Needs, definitions, potential methods, and an example pertaining to waterfowl management. Environmental Modelling & Software 22, 269-277.

Sonneveld, B.G.J.S., Keyzer, M.A., Stroosnijder, L., 2011. Evaluating quantitative and qualitative models: An application for nationwide water erosion assessment in Ethiopia. Environmental Modelling & Software 26, 1161-1170.

Tijhuis, M.J., de Jong, N., Pohjola, M.V., Gunnlaugsdόttir, H., Hendriksen, M., Hoekstra, J., Holm, F., Kalogeras, N. Leino, O., van Leeuwen, F.X., Luteijn, J.M., Magnússon, S.H., Odekerken, G., Rompelberg, C., Tuomisto, J.T., Ueland, Ø., White, B.C., Verhagen, H., 2012. State of the art in benefit–risk analysis: food and nutrition. Food Chem. Toxicol. 2012, 50:5-25. http://dx.doi.org/10.1016/j.fct.2011.06.010

Tongchuay, C., Praneetpolgrang, P., 2010. Knowledge quality and quality metrics in knowledge management systems. Proceedings of the Fifth International Conference on eLearning for Knowledge-Based Society. December 11-12, 2008. Bangkok Metro, Thailand.

Tuomisto, J.T., Pohjola, M.V., 2007. Open Risk Assessment - A new way of providing information for decision-making. Publications of the National Public Health Institute B18/2007. KTL - National Public Health Institute, Kuopio.

Wagener, T., Kollat, J., 2007. Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox. Environmental Modelling & Software 22, 1021-1033.

Walker, W.E., Harremoës, P., Rotmans, J., van der Sluijs, J.P., van Asselt, M.B.A., Janssen, P., Krayer von Krauss, M.P., 2003. Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support. Integrated Assessment 4, 5-17.

Wang, R.Y., Strong, D.M., 1996. Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems 12, 5-34.

Wardekker, J.A., van der Sluijs, J.P., Janssen, P.H.M., Kloprogge, P., Petersen, A.C., 2008. Uncertainty communication in environmental assessments: views from the Dutch science-policy interface. Environmental Science & Policy 11, 627-641.

Wismar, M., Blau, J., Ernst, K., Figueras, J. (Eds.), 2007. The Effectiveness of Health Impact Assessment.: Scope and limitations of supporting decision-making in Europe. WHO, Copenhagen.

WHO, 1999. Health Impact Assessment (HIA), main concepts and suggested approach. Gothenburg consensus paper. World Health Organization, Brussels.

WHO/FAO, 2006. Food safety risk analysis: A guide for national food safety authorities. World Health Organization, Food and Agriculture Organization of the United Nations, Rome.

Wood, C., 1995. Environmental impact assessment: a comparative review. Longman Scientific & Technical, New York.

Wyat Appel, K., Gilliam, R.C., Davis, N., Howard, S.C., 2011. Overview of the atmospheric model evaluation tool (AMET) v1.1 for evaluating meteorological and air quality models. Environmental Modelling & Software 26, 434-443.

Xu, Y., Booij, M.J., Mynett, A.E., 2007. An appropriateness framework for the Dutch Meuse decision support system. Environmental Modelling & Software 22, 1667-1678.