Evaluating scientific claims

From Testiwiki
Jump to: navigation, search

Evaluating scientific claims is a method for estimating the truthlikeness of statements that are claimed to have scientific basis. Originally the need for this method came from food area, where there are a lot of functional and other foods or food items that have health claims about their usage.

Scope

How should health claims be evaluated in a way that can be scientifically defended?

Definition

Rationale

The Passclaim approach

The Passclaim project developed consensus criteria for evaluating scientific claims. They concluded that the following criteria should be used:

  1. The food or food component to which the claimed effect is attributed should be characterised.
  2. Substantitation of a claim should be based on human data, primarily from intervention studies the design of which should include the following considerations:
    1. Study groups that are representative of the target group.
    2. Appropriate controls.
    3. An adequate duration of exposure and follow up to demonstrate the intended effect.
    4. Characteristaion of the study groups' background diet and other relevant aspects of lifestyle.
    5. An amount of the food or food component consistent with its intended pattern of consumption.
    6. The influence of the food matric and dietary context on the functional effect of the component.
    7. Monitoring of subjects' compliance concerning intake of food or food component under test.
    8. The statistical power to test the hypothesis.
  3. When the true endpoint of a claimed benefit cannot be measured directly, studies should use markers.
  4. Markers should be:
    1. Biologically valid in that they have a known relationship to the final outcome and their variability within the target population is known;
    2. Methodologically valid with respect to their analytical characteristics.
  5. Within a study the target variable should change in a statistically significant way and the change should be biologically meaningful for the target group consistent with the claim to be supported.
  6. A claim should be scientifically substantiated by taking into account the totality of the available data and by weighing of the evidence.

The open assessment approach

The open assessment requires everything in an assessment to be described as a formal object, mainly a variable or an assessment. In the case of a single health claim, the assessment approach seems to work well. The purpose of the assessment is to find whether the claim fulfils the criteria that are set. The criteria will be defined as a part of the scope of the assessment. The research question should be explicit and give a quantitative answer. It should also pass the clairvoyant test.

The health claims are typically in the format: "The product may help in reducing cholesterol." In this case, the scope of the assessment could be: "What is the impact of the product X on serum cholesterol in the population Y?" The assessment would have two scenarios: one where the consumers do not buy the product, and the other one where they do buy it. The evaluation of the health claim is based on the comparison of there two scenarios.

The assessment consists of the following variables:

  • the exposure to the product X by a member of the population Y given that he has/has not bought it;
  • the exposure-response function of product X (or the active compound in it) on serum cholesterol in the population Y;
  • the actual cholesterol level given the exposure and the exposure-response;
  • and the value judgements about what probability corresponds the word "may" and about what level of cholesterol reduction is considered significant by the consumer.

The value judgements can also be determined in the scope of the assessment, and then there is not a need to try to evaluate how these value judgements occur in the population Y. In an even simpler case, also the exposure can be determined in the scope of the assessment so that a nominal, not actual, exposure is used. In this case, the health claim can be evaluated in a model with a single variable, which is the exposure-response function.

The results of the exposure-response function (and other variables, if needed) are assessed just like in any other assessment: by bringing in scientifically valid information that is used to falsify possible values from the domain of the variable, and having structured argumentation about issues when need be.

The actual evaluation of the claim is made by looking at the result of the outcome variable. We read the probability that the cholesterol difference between the bought and not-bought scenarios is higher than the level of "significant change". If this probability is higher than the probability for "may", the claim is accepted.

Discussion

The major problem with the Passclaim approach is that although it is based on scientific expertise and understanding of strengths and weaknesses of data, it is itself not a scientific method. In other words, the criteria are based on consensus rather than explicit research questions. In addition, the criteria used are not falsifiable in the sense that they could be proved wrong based on observations. However, they could be developed into falsifiable criteria, if a set of meta-criteria was developed to evaluate the performance of these criteria in evaluating the quality of evidence for the health claims.

Therefore, the Passclaim approach cannot be used as the basis for a scientific evaluation of health claims. (It can be used to evaluate scientific support, as the name says, but the evaluation itself is not scientific.)

Instead, the open assessment approach fulfils the falsifiability criterion and thus ban be used as basis for scientific evaluation. The Passclaim criteria can then be used within an assessment to defend or attack individual inferences from data in a formal argumentation.

Other problems of Passclaim, compared with open assessment, include the following.

  • It does not require the health claim to be quantified or pass the clairvoyant test, which makes it very difficult to evaluate its acceptability.
  • It does not define the domain of the claim, i.e. other possible values the claim could have. For example, if the claim is "product X may reduce cholesterol", is the claim untrue if and only if the following is true? "It is impossible that product X could reduce cholesterol at all." If yes, there must be very strong evidence to prove the claim wrong. If no, what does it mean?

Procedure

  1. An open assessment is launched.
  2. A research question is set so that the answer will give a quantitative answer about the magnitude of the health impact of the product.
  3. Quantitative criteria are set for determining whether the claim is fulfilled or not. The answer to the research question will be evaluated against the criteria.
  4. A model is created for estimating the answer to the research question.
  5. Data is collected and critically evaluated (using argumentation when necessary) to quantitate the model. The PASSCLAIM criteria may be used in defending or attacking inferences based on a particular piece of data.
  6. The model is run and the result compared with the criteria.

See also

  • Aggett et al: Passclaim: Process for the Assessment of Scientific Support for Claims of Foods. Consensus on Criteria. European Journal of Nutrition (2005): 44: 1 (suppl).
  • Asp et al: Passclaim: Process for the Assessment of Scientific Support for Claims on Foods. Phase Two: Moving Forward. European Journal on Nutrition (2003): 43: 1 (suppl).
  • Asp et al: Passclaim: Process for the Assessment of Scientific Support for Claims on Foods. Phase One: Preparing the Way. European Journal on Nutrition (2003): 42: 1 (suppl).
Retrieved from "http://www.opasnet.org/testiwiki/index.php?title=Evaluating_scientific_claims&oldid=6508"