Assessing likelihood ratio of clinical symptoms: handling vagueness

A.L.B. Rutten, C.F. Stolper, RFG Lugten, RWJM Barthels
Commissie Methode en Validering VHAN (Dutch Association of Homeopathic Physicians), The Netherlands

Clinical symptoms and especially homeopathic symptoms are often vague. So far we see some reluctance to assess clinical symptoms as diagnostic instruments because they are hard to define. Still, clinical symptoms appear effective in daily practice. Expert systems and in neural networks analyse vague data successfully.
Theoretical considerations predict the kind of problems we may expect. There is a difference between quantitative and qualitative vagueness. Vague data cause problems if we try to prove a hypothesis because of expectation bias. We assess likelihood ratio (LR) of homeopathic symptoms only to improve the method. Homeopathy (2003) 92, 182-186

Introduction

Likelihood ratio (LR) is the modern Bayesian translation of homeopathic expressions like ‘keynote’ or ‘characteristic symptom’. In a previous paper(1) we stated that, while researching likelihood ratio, we must investigate the same instrument (symptom) as that used in daily pratice. This involves the vagueness of clinical judgement for many symptoms. In trying to avoid this vagueness by the use of questionaires we are not assessing the right instrument. The complexity of many clinical symptoms makes vagueness unavoidable. But unnecessary vagueness causes unreliable LR results so we must deal with it conscientiously.

Assessing LR of homeopathic symptoms requires prospective study of the prevalence of a symptom, comparing the target (remedy) population with the rest of the population visiting the homeopathic doctor. To perform such a study successfully we should be prepared for methodological problems and bias in order to obtain sufficient data and reliable results. This article present some theoretical considerations; and the result of a pilot study on three different kinds of symptoms that are presented in an accompanying paper.(2)

Vagueness and ambiguity

According to classical logic a term is vague if it has borderline cases.(3) This is described by the ‘Sorites paradox’:(4)

Borderline cases exist for many clinical symptoms, like baldness and loquacity. It is hard to define how many words per day a person must utter before we must call him or her loquacious.

The clinical notion of vagueness is broader than in classical logic, it comprises also generality and ambiguity. The concept of generality can be observed in the repertory. The rubric ‘Loquacity’ contains several sub-rubrics like ‘Loquacity, during which answers no questions’, or ‘Loquacity, changing quickly from one subject to another’, or ‘Loquacity, makes speeches’. Kent struggled with this problem, we see many rubrics where not all the remedies of the sub-rubrics are present, e.g. the medicine chamomilla is present in the sub-rubric ‘Loquacity, makes speeches’, but not in the main-rubric. One could argue that ‘making speeches’ is not some sort of ‘loquacity’, which indicates that that ‘loquacity’ is an ambiguous notion. The Sorites vagueness is quantitative, whereas generality and ambiguity are qualitative.

Some logicians have struggled to find solutions for Sorites-vagueness,(5) for instance by defining a margin for error (interval of acceptable data). Others, like William James, regard logic as fundamentally insufficient to describe reality:(6) "… reality is not only broader than the known; it is broader than the knowable". It appears that James defines vagueness in a broader sense, including ambiguity and generality.

In medicine the border between Sorites vagueness and generality-ambiguity is not always clear. One could imagine that extreme thirst caused by diabetes insipidus is semantically different from extreme thirst based on other causes, although at first sight it appears to be Sorites vagueness. The clinical consequences are however significant. The same could be the case for extreme headache, with causes different from ‘normal’ headache.

Confounding and modalities

Covariates may confound the outcome of a test.(7) Age is such a confounder: many symptoms only occur above a certain age or are bound to a certain age interval. Profession might also be a confounder: a barker may appear talkative to his public, but quiet among his family. In the assessment of clinical tests it is essential to be aware of confounders and to evaluate them. The influence of confounders can be measured by regression techniques. The assessment of clinical and homeopathic symptoms is, however, more complex. In general there are more confounders influencing a clinical or homeopathic symptom, eg it is hard to imagine that the profession of the patient will influence the outcome of ultrasonography. In homeopathy many confounders are known as ‘modalities’. This leads us back to the concept of generality, if we research the symptom ‘loquacity’ we will have to take all possible modalities into account. But it will be impossible to formalise all possible confounders influencing homeopathic symptoms. Accounting for possible confounders must be based on clinical judgement. Another reason for this is that many confounders/modalities are also vague, eg weather and activities.

Nevertheless there is a margin for error based on Sorites vagueness in judging clinical symptoms with a lower and an upper margin. In this article we take ambiguity and generality together as qualitative vagueness and discriminate it form ‘Sorites’ or quantitative vagueness. We will show the practical consequences of this distinction.

Pragmatics and science

Is vagueness inimical to science? In medicine scoring systems are often used to avoid vagueness. These scoring systems are applied to research, and to quality assurance programs. There is also a need for scoring systems for judicial reasons where vagueness cannot be tolerated. However, scoring systems rarely benefit the clinician. Behind the same abstract figures very different diseases mat be hidden.(8) This is comparable to what we stated about modalities.

There is a difference in attitude between a scientific investigation and a therapeutic function. The scientific process demands to have a hypothesis first and then the gathering of data to prove it or otherwise. One is tempted to interpret vague data in a convenient way (= expectation bias). Expert systems and using neural networks is built on the observation and analysis of clinicians that are just doing their job without any scientific goal. Thanks to information technology, and especially fuzzy logic, we are now able to handle vague data. Fuzzy logic is the mathematical translation of margin for error and makes it possible to include data within a certain range and a certain weight. Fuzzy logic is used in clinical decision-making, like the diagnosis of acute abdominal pain(9) and an increasing number of applications is developing, eg modelling of nursing intuition.(10) One of the techniques used in this field is Case Based Learning, which is in fact a statistical analysis of successful cases, about the same procedure as defining LR.

In the examples mentioned above a third party assesses the interaction between doctor and patient. When the same person (doctor) wants to do his therapeutic function and investigate this activity at the same time, this creates a problem. Expectation bias will cause clinical data to be judged in a convenient way, the more so if these data are vague. Vagueness is awkward if we want to prove things, but it is unavoidable if we want to improve clinical practice. The problem of inaccuracy in medical research is well known. One of the solutions is aggregation of multiple raters and there are epidemiological techniques for assessing rater performance.(11) ‘Regression modelling’ can be used to adjust LR for covariates (confounding) and also be applied to evaluate ambiguity.(12) This requires further study. Here we concentrate on some practical consequences in conducting prospective research on LR.

In the following paragraphs we will show that vague diagnostics can be very effective, then we indicate the problems that may arise when we are not careful when vagueness is involved.

Intuition

Probably the worst case of vagueness is intuition, which is a major component of clinical judgement. We know little about the diagnostic value of intuition but we are convinced it is indispensable.(13) Greenhalgh states: ‘There has been little formal research into how (and to what extent) intuition contributes to decision making in the clinical setting’ and: ‘The educational research literature suggests that we can improve our intuitive powers through systematic reflection about intuitive judgements’.(14)

Let us look at the diagnostic process of appendicitis. The diagnostic value of ultrasonography in appendicitis(15) is known (see figure 1). In this graph we can see the increase or decrease in certainty of the diagnosis appendicitis after ultrasonography. If, say, we estimate the chance that a patient has appendicitis before ultrasonography to be 33% the positive ultrasound test will increase this chance to nearly 80%. The clinical diagnostic process that leads to the suspicion of appendicitis however depends largely on intuition and has scarcely been investigated. The prevalence of appendicitis in patients where appendicitis was suspected on clinical grounds after referral,(16-20) varies from 22 to 88%. In Dutch general practice appendicitis occurs once in every 1000 patients a year.(21) In the US the prevalence of appendicitis with patients send home after attending with suspected appendicitis is 2 %.(17) In a population where the prior-chance of appendicitis is 2%, the posterior-chance for appendicitis after positive ultrasonography is 13% (Figure 1). Posterior chance after referral by GP’s is over 22%, ie clinical judgement performs better than ultrasonography, the LR+ being more than 14. There is some evidence that ‘… on a population level, diagnosis of appendicitis has not improved with the availability of advanced diagnostic testing’.(22) Part of the clinical judgement is described in the Alvarado-score (migration of pain, anorexia, nausea-vomiting, tenderness of abdomen, rebound pain, elevated temperature, leucocytosis and shift in leucocytes).(20) Leukocyte counts are often not performed in general practice when appendicitis is suspected. The diagnostic properties of the other clinical signs are poor, so intuition still plays an important role.

Figure 1 Ultrasonography and appendicitis: LR+ = 7.6; LR- = 0.27

There is wide variety in prevalence of appendicitis after referral, but it is unlikely that there is the same variety in prevalence of appendicitis in general practice. This could be further investigated. The diagnostic properties of ultrasonography for appendicitis have been well assessed, but the far more effective clinical judgement leading to referral is poorly investigated. In this case we deduced the LR of intuition and clinical judgement, we feel that it demonstrates the need to investigate the strength of clinical judgement despite all the possible pitfalls. We need not to be too afraid of vagueness. If we can assess the LR of intuition we can certainly assess the LR of a homeopathic symptom. But of course it helps if we know what the symptom (like intuition) consists of and what pitfalls we may expect. One of the main pitfalls we expect is expectation bias, as demonstrated below.

A model for vagueness

What does vagueness imply for homeopathy? The problem is the complexity of the choice of a homeopathic medicine; it depends on the co-occurrence of several symptoms. One symptom becomes more important (or more desirable) if certain other symptoms are also present. There is a difference between choosing a medicine during a homeopathic consultation and justifying this choice afterwards. If we prescribed Lachesis it is possible that we consider a patient as loquacious where we would not have done this if we prescribed the same patient another remedy. The margin for error based on quantitative vagueness can easily lead to different interpretations, caused by expectation bias. If the lower margin for error is low we can expect the prevalence of a characteristic symptom for a remedy to be high.

It is important to distinguish between quantitative and qualitative vagueness. Take the symptom ‘loquacity’ in relation to the remedy Lachesis. If one doctor is stricter in (quantitatively) limiting loquacity than another doctor it will not influence the likelihood ratio. This is because the quantitative limitation is equal in the ‘Lachesis-group’ to that of the rest of the population. If, on the other hand, there is qualitative vagueness involved the likelihood ratio is indeed influenced. If only ‘loquacity, changing the subject frequently’ is perceived the collection of patients with this symptom will be relatively more confined to the collection ‘Lachesis-patients’ (Figure 2).

We can also express this in a 2x2 table (see Tables 1-4). We examine the less strict interpretation (on ‘Sorites’ principle) of loquacity as ‘Loquacity(+)’ in Table 1 and the more strict interpretation as ‘Loquacity(+++) in Table 2. Here we see that we can allow a margin for error as long as we assess the Lachesis-group and the rest the same way; the LR+ remains 4. If, however, we use the more strict interpretation for loquacity in the rest and the less strict interpretation for the Lachesis-group (expectation bias) we favour the Lachesis-group incorrectly and the LR+ becomes 8 (Table 3). If we are not aware of ambiguity and assess a special kind of loquacity, ie ‘Loquacity, changing the subject quickly’, we also set a higher LR+, =10. Based on a combination of expectation bias and ambiguity we can get an even greater distortion of the results.

Figure 2 Loquacity with different kinds of vagueness

Table 1 2x2 table (hypothetical); loquacity and Lachesis; LR+=4

	Lachesis	rest
loquacity (+)	4(a)	100(b)	104
no loquacity	6(c)	900(d)	906
	10	1000	1010

Table 2 2x2 table (hypothetical), quantitative shift; loquacity(+++) and Lachesis (sorites shift); LR+=4

	Lachesis	rest
loquacity(+++)	2(a)	50(b)	52
no loquacity	8(c)	950(d)	958
	10	1000	1010

Table 3 2x2 table (hypothetical); loquacity and Lachesis (quantitative expectation biased shift); LR+=8

	Lachesis	rest
loquacity (+)	4(a)	50(b)	54
no loquacity	6(c)	950(d)	956
	10	1000	1010

Table 4 2x2 table (hypothetical), qualitative shift; loquacity (changing the subject quickly) and Lachesis; LR+=10

	Lachesis	rest
loquacity (changing subject)	3(a)	30(b)	33
no loquacity	7(c)	970(d)	977
	10	1000	1010

Conclusion

Expectation bias and qualitative vagueness mainly influence the outcome of LR investigation of homeopathic symptoms while quantitative vagueness has no influence. Therefore the main concern in the assessment of homeopathic symptoms is not the quantitative definition of a symptom as long as we assess the target and the control group equally.

It is all a matter of attitude. There is no scientific proof that the LR of vague homeopathic symptoms is correct. Nevertheless, using LR instead of the present construction of repertory symptoms could improve our method. Conditions should be optimised by proper preparation of the researchers:

• Researchers must be convinced that they fool themselves by polishing the data; the aim is to improve, not to prove.

• The research should be carried out by several independent and experienced homeopathic physicians.
• The message to the participants is: ‘Do what you always did, but more consciously’.
• The choice of the remedy must not influence the scoring of symptoms.
• The participants must be familiar with the materia medica concerning the investigated symptom.
• The consensus about the symptom must be described.
• The symptoms must be assessed during the first consultation and this assessment may not be altered afterwards unless well motivated.

Vagueness renders proving of hypotheses impossible, but we still believe that we can use vague clinical symptoms to detect successful diagnostic strategies.

References

Lex Rutten, MD

Aard 10 - 4813 NN Breda, Netherlands