The Lone Dog has been mulling over the topic of lies, damned lies, and statistics as they are utilized by the refractive surgery industry (RSI). How to make this article both witty and comprehensible? Serendipitously, late last night the peer-reviewed journal Ophthalmology delivered unto the LD the perfect platform for discussion:

“U.S. Food and Drug Administration Clinical Trial of the Implantable Contact Lens for Moderate to High Myopia”

Opthamology*
Volume 110 Number 2 (February 2003)
Pages 255 — 266 (a fairly long article)

Brought to you by The Implantable Contact Lens in the Treatment of Myopia (ITM) Study Group.

*The official journal of the American Academy of Ophthalmology.

This article talks about implantable contact lenses (ICL) in individuals with between 2 and 20 diopters of myopia (nearsightedness) but the points made about the study design flaws and statistical shenanigans apply to all articles reporting all types of LVC procedures.

Lies . . .

In the Results section the authors close with the statement, “For halos, there was a less than 3.0% difference between the percent of patients reporting an improvement versus a worsening in this symptom” (emphasis added). Hey, that sounds safe enough.

How about if 51% of patients reported worsening, and 49% reported improvement? The ITM Study Group didn’t say anything statistically untrue in their closing sentence.

How about if the 51% of patients who didn’t have any halos before surgery got worse, and the 49% who had bad halos because their contacts were always dirty got better? The ITM Study Group still isn’t fibbing. [Which kind of patient are you?]

How about if 51% had severe worsening and 49% had slight improvement? Still the truth, in statistic-speak. But maybe this procedure isn’t so safe after all, from the perspective of an ordinary person like you.

You get my drift.

Let’s pony up the actual data on the Subjective Patient Symptoms analysis from the report (Table 9, page 262):

N = number of patients
% = percent of total N

If we add up the colored boxes we get 10.0% of subjects who felt that halos were better, and 12.7% who felt they were worse. And the difference is . . . 2.7%. This is less than 3%, as advertised.

A digression onto hidden errors in study design . . . or, garbage in garbage out.

The data for subjective vision quality was gathered from a self-administered patient questionnaire. If this questionnaire is typical RSI output, patients are asked to rate a symptom on some type of point scale, such as 1 = no glare and 5 = ‘bad’ glare. If your pre-op number was 3 and your post-op number is 1, your data point takes up residence in the "improved 2 categories" cell in the Table. In the trial, the two questionnaires (before and after) were administered about 12 months apart. Unless these patients are a lot smarter than the LD, how many of them are really going to remember what personal degree of glare they were rating as a 3 on that first questionnaire? Just exactly how bad is a 5? Is my 5 worse than your 5? Did the study use standardized drawings or photographs that demonstrated various degrees of “quality” (whatever that is), glare, halos, diplopia, and “night driving,” with a corresponding severity score? The article doesn’t say.

In the absence of evidence to the contrary, we are free to assume the worst: no visual aids to help subjects rate their symptoms consistently, broadly stated questions, and a deliberate statistical massage to report "changes" rather than absolute values. Suppose all the people who had halos “2 categories worse” had substantial halos pre-op? These folks might be in really bad shape after surgery — they might have halos off the objective scale, if there was one, which there isn’t. Or what if there was a correlation between worse glare and blue eyes? Pre-operative halos and blue eyes might be contra-indications to this type of procedure in some patients. This would be a darned important finding at this stage of ICL development. Reporting it now would save a lot of patients a lot of visual misery, plus possibly a second procedure to dig the contact lens back out of the eye. So why didn’t they look into it? Not enough RAM in the laptop?

Near the end of the Discussion (page 263, right column, paragraph 4) the ITM Study Group smugly concludes that “The incidence of subjective patient symptoms showed a slight improvement (glare, double vision, quality of vision, night driving difficulties) compared with before surgery in contrast to other refractive surgery procedures in which large increases in symptoms are commonly reported.” We have the little problem of survey-in-garbage-out, but let’s suspend that complaint for a moment. So, you have a large proportion of content patients (no change), a small proportion of really happy patients (improved categories), and a small but non-trivial proportion of patients with surgically-induced vision degradation (worsened categories) who may or may not still be “happy”. By focusing on the average change — which gives a non-frightening result of “less than 3% difference”, for example — the ITM guys can skip the sweaty stuff, which is figuring out why the ‘worsened’ patients are worse, and how they might be identified before surgery.

Notice the ITM Study Group hasn’t told any lies yet.

Damned lies . . .

Remember Lou “Big Pupil” Schmo, the LVC hopeful who has no glare, halos, ghosting, or trouble driving at night despite dark pupil diameters approximately the size of the Channel Tunnel? Consulting Table 9, we see that if Lou wants to learn anything about his personal risk of vision quality loss, we must discard any patients who reported improvement in symptoms, because Lou doesn’t have any symptoms. That is to say, Lou is not like 41 subjects in the study who had pre-operative halos that got better, and he is not interested in being compared to them vis-a-vis his halo category change, because the only way he can go is ‘worsened’. What might exclusion of these patients do to Lou’s calculated risks? A little button punching gives us risk of worsened glare = 9.2%, risk of worsened halos = 14%, risk of worsened night driving = 9.1%. Let’s further assume the worst of the worsened, that these risks are independent (this isn’t likely, by the way), and therefore additive: his total risk of feeling ‘worsened’ in some way after the surgery then becomes 32%. In reality, patients with worse glare and/or halos are probably the same subjects recording worse night driving. But glare and halos do not seem to be the same optical phenomenon, so we should not assume that the 9% worse glare group is encompassed within the 14% worse halos group. Starting with worse halos (14%) and throwing some in for worse glare without worse halos, we can reasonably get a corrected risk of ‘worsened’ for Lou of > 15%.

[The above calculations have their own statistical shakiness because what we would really like to know is what happened to just the no-halo patients. The “no change” group contains patients with no halos before or after surgery, and patients who had halos before surgery that didn’t change. The argument still illustrates the LD’s point.]

Back to the infamous Table 9 — “quality of vision” data (left hand columns). Hey ITM Study Group, can we have a definition of “quality” please? Maybe it’s like pornography, you can’t describe it but you know it when you see it. Might 291 patients have 291 non-congruent definitions of pornography and vision quality? How does Lou translate “worsened 2 categories night driving” into his own life? How does anyone? Got a photograph of worsened headlights? Got milk?

[The LD gets grumpy after reading these sleight-of-statistical-hand report-of-FDA-study articles late at night.]

Starting from the premise that someone with no glare-halos-ghosting-night driving problems has terrific vision quality, once again the only way Lou can go is down. We then estimate he has a > 15% chance of having decreased vision quality, presumably due to glare-halos-ghosting-night driving problems, plus possibly some issues with contrast sensitivity and binocular summation.

If you were Lou, which statistical presentation of this data would you feel more honestly informs you about your risk of decreased vision quality as a result of implantable contact lenses:

1. ITM Study Group version: “For haloes, there was a less than 3.0% difference between the percent of patients reporting an improvement versus a worsening in this symptom.” Etc. etc. for the other symptoms.
   
   
2. LD version: “You have at least a 15% chance of experiencing a decrease in your vision quality in some fashion that may disturb you and possibly interfere with your life.”

And Statistics . . .

The LD will acknowledge that people who are more than 10 diopters myopic have messy lives with thick glasses, tedious contacts, and very large bedside alarm clocks. Highly nearsighted people wearing glasses cannot avoid optical aberrations and have zippo peripheral vision. Anyone wearing toric soft contact lenses (that correct astigmatism) long ago made a quality trade-off to avoid glasses. People like this are not like Lou.

As it happens, “only 21.2%” of eyes in this study had a Lou-like level of myopia between 2.1 and 7.0 diopters (page 258, left column, paragraph 3). The other 79% were strung out between 7 and 20 diopters (Table 3, page 258).

Ahem . . . there are major physiologic and optical differences between a 3 - 6 diopter myope (garden variety), and a 15 - 20 diopter myope (prone to disaster). Lumping them together in one trial is like saying that a kiwi and a tomato are the same thing because they are both fruit, and then reaching an “average” conclusion about whether they taste good in spaghetti sauce. The lower the myopia, the lower the likelihood of pre-operative quality symptoms unless the person is a total slob. So here we have almost a quarter of the subjects in one little cluster at the shallow end of the pool, a group that is at risk to experience worsening of vision quality because they generally start out with fine vision quality. A group that typically does very well with properly conducted laser vision correction. A group that is representative of most people considering LVC. Is this group analyzed separately by the ITM Study crew? No.

So, despite its 11 pages of text, 12 Tables, and 42 references, the LD concludes that the report on the U.S. Food and Drug Administration Clinical Trial of the Implantable Contact Lens for Moderate to High Myopia is scary and can’t be trusted.

Sniffing the Small Print . . .

At the bottom of the first page of the article, we find that to request a reprint (a glossy offprint rather than a photocopy), you contact STAAR, the ICL manufacturer, as opposed to the senior author, a guy called Vukich. The LD, on the inside of the medical publishing gig, knows that this is markedly atypical and finds it inordinately creepy. What do you want to bet that anyone making a request gets put in the STAAR database of potential future doc ICL users? But that’s not all.

Just above the References, in something like 5 point type, we encounter the following enlightening facts:

1. Four members of the ITM Study Group, including the senior author (John Vukich MD), are paid consultants for STAAR.
   
   
2. Of the four people who actually wrote the manuscript, Monica Gaston is employed by STAAR, Vukich and David Brown MD are paid consultants, and Kimberly Doney works for Donald Sanders MD PhD (also a paid consultant).

Monica Gaston works for STAAR? She what? She works for STAAR? And the other 3 authors take money directly or indirectly from STAAR (probably lots of it, too)?

Remember Enron . . . and MCI . . . and Ahold. Time for an unbiased external audit by some entity with clout. Isn’t that the Food and Drug Administration? For more on the FDA read the forthcoming Rick Kwiecinski . . . and weep.