“Lies, damned lies, and statistics.” Benjamin Disraeli

20 01 2009

This variously attributed quote was most probably made originally by Benjamin Disraeli. His point was to highlight the use of numbers in an argument, the further use of statistics to bolster weak arguments and then the ultimate disparagement of statistics themselves, particularly if those statistics don’t fit the correct side of the argument.

The debate over the MMR vaccine sprung from one single scientific paper that spawned a million column inches, months of debating hours and possibly even the death of some children. Everyone has an opinion. Yet eight years later the study has been totally discreditied and Dr Wakefield, the first author has been struck off for research “issues”.

Have you actually read the paper? Did you know there were only 12 apparently random children in the study? And that only 8 had the histological change that allegedly was the “cause” of autism itself becoming clinically apparent within 6 days of the vaccination. Now, what was your opinion on MMR vaccination again?

Currently, the hot topic on the lips and legal pads of medical managers everywhere is “The Surgical Checklist.” Everyone has heard about it particularly how a simple list is hugely reducing surgical complication and even death rates across the world.

Have you actually read the paper?

What do you think about the potential for bias in the data collection (see results, not method)?

How many patients actually had the checklist applied?

What does the “checklist” actually comprise of?

Save for the list, are there any other particular changes to care that may have affected the outcome?

How do you think a simple surgical checklist on its own actually reduces the rate of complications such as myocardial infarction, pulmonary embolus and ventilator requirement? Or is it perhaps something else affecting results?

Importantly for hospitals trying to prevent “wrong side surgery”, what evidence does the paper provide for reduction in this problem?

Is the reduction in death rate statistically significant?

What is the Hawthorne effect?

(I’m not sure it’s clear but what economic area does it suggest UK, Canada and New Zealand are in?)

The newspapers are particularly hot for this topic not the least because of the highlighted resistance of surgeons to this change. The fire of this debate is fuelled additionally by the age old antagonisms and prejudices particularly of managers and patient groups.

A small sample of that debate can be found in the comment section here. It is not supportive of the medical profession. Perhaps we could hand out some ladders of inference to help calm things?

Now read just about any news article or pronouncement from management on the issue. Are the changes to be instituted  justified by the vigour and results of the study? Can you understand why there may be some antagonism between the two groups? If you were (are) a manager, how would you approach the situation?

Now I’m not against improvements in clinical care or checklists or debate, I’d just like it a bit better informed.

Lies, damned lies or statistics, you decide.

Advertisements

Actions

Information

13 responses

21 01 2009
Anne Marie

OK . I’m not a surgeon and I haven’t discussed the paper with any surgeons. I have read the paper a few times. And I have read all the comments on the running-a-hospital blog you linked to. I’m not sure why you think that that discussion is unsupportive of medics. This research was carried out by medics and published in a medical journal- not a management journal.
Yes, there was no significant reduction in mortality in wealthy countries. But this is a much rarer event in these countries and maybe reflects limited power of the study.
I’m surprised you compare this to the Wakefield study. But I look forward to further discussion.

21 01 2009
inseiffolliet

I was rather hoping others might have picked up on the various issues.

My main points are that the paper itself has a poor scientific basis- poor data from varied sites with strong potential for bias due to more than half the data being collected by the clinical teams themselves, rather than the trumpeted “independent observers”. The “checklist” wasn’t even used appropriately in nearly half the patients. Most importantly, there were MAJOR changes instituted in clinical practice in various centres; use of prophylactic antibiotics, appropriate major haemorrhage preparation, use of pulse oximetry and even anaesthetic pre-assessment which themselves are well recognised as bringing about MAJOR changes in outcome. Each one could have simply been the cause rather than a checklist to assess whether they were in place or not. Then the results themselves are very difficult to interpret due to the vast difference in clinical settings, most importantly NON SIGNIFICANT for mortality in developed countries. To spell that out in english; “it made no difference.”

So what has brought about “dramatic reduction in death rate”? Who knows. Maybe even chance itself. Poor science.

And yet most discussions, virtually all the headlines and most management edicts are now focusing on the institution of a checklist being read out in theatre, “because it will save lives”. (L,dl and s)

the reason I mentioned the MMR study is that once again, a poor science paper caused an amazing furore principally because of ill-informed debate brought into the public domanin fuelled by preconceptions and old antagonisms.

The comments on the other blog site interestingly, have been editted but the guy is CEO of a big hospital.

22 01 2009
Anne Marie Cunningham

Hello
Sorry I didn’t get round to replying earlier. I really would love to hear comments from more people about this paper. To my mind, the introduction of the checklist is part of a process of quality improvement. Thus I would expect that a team sitting down and negotiating how to use this checklist in their practice may decide to introduce pulse oximetry as a result. That is the point of the process.

You mention that the checklist wasn’t even properly administered in 50% of cases. This is why the study is described as being pragmatic. It is about what happens in real life when you start using a tool like this. You are not sure how everyone is going to use it but it looks like it still leads to improvements in quality of care.

I had already addressed your comments about death rates in developed countries by suggesting that this may be a Type 1 error. The study was not powered to find a reduction in death rates in the subgroup of developed countries, but to find a reduction in all complications in all countries. The fact that there have been significant reductions for most complications, if anything makes the case for the use of the checklist stronger.

I am still unhappy about your comparison of this to the Wakefield study. Decreased uptake of MMR has undountably caused harm. How do you think that uptake of the surgical checlist will cause harm to populations?

At the end of the day this trial can not answer why death rates were reduced. But I can see a logic for how use of this checklist may have lead to improved quality of care.

22 01 2009
inseiffolliet

maybe i shouldn’t have been encouraged into blogging about this. 😉

my point of contrasting and comparing the MMR study was purely that again a paper which i don’t believe was good science ended up bringing about significant changes probably on the basis of press exposure rather than those many good papers, with excellent science and statistics that don’t get this sort of attention.

attention to good pre-op anaesthesia practice, use of pulse ox, prophylactic antibiotics and preparation for major blood loss have ALL been shown to make a difference but everyone is getting excited over a list, which hasn’t.

at least we are all working towards improving care eh?

hey ho.

22 01 2009
Anne Marie Cunningham

It is the risk of blogging! To be honest, I thought that this paper, and its approach to quality improvement, and the fact that implementing use of the clinical checklist is about clinical leadership, would have been right up your street. But then I haven’t been following your blog too long, so I’m not quite sure what your take on clinical management/leadership is.

The problem with finding out that something is good and helps save lives like prophylactic antibiotics or monitoring O2 saturation is that it does not necessaruly get implemented. Surely the point of the checklist is to get people talking about their protocols and, more importantly, practice and start changing things for the better.

The authors clearly state that they “hypothesized that implementation of this checklist and the associated culture changes it signified” was what would bring about improvements in care. It isn’t just about standing together reading out the checklist.

Thank you. I asked you to blog about it because you were so forceful in your criticisms of the paper and I didn’t see that we could have a meaningful conversation through Twitter.

You are right that seeing how others interpret and comment on this will be interesting.

22 01 2009
inseiffolliet

your point:

The authors clearly state that they “hypothesized that implementation of this checklist and the associated culture changes it signified” was what would bring about improvements in care. It isn’t just about standing together reading out the checklist.

is exactly what I am getting at. What IS happening is only the checklist. and there will be expectation of significant improvements because of that. I think it is unlikely.

the point about Clinical Leadership is interesting. I hadn’t consider it from that angle. There of course is a huge area for debate.

22 01 2009
B

Now, as far as I can see, we are all agreeing that this is a poorly paper.
I can however see, how they might have thought this was a good idea in the first place. A recent death of a pilots wife showed that is was a) completely unnecessary and b) very easily avoidable if there would have been good communication amongst the team. The pilot argued that in his line of work there are various checklists to go through to eliminate what they call “the human error”. And we all agree that you can’t do anything about a hurricane, but it helps to make sure the doors are locked in a situation like that.
However I fail to see how this particular checklist will make a huge amount of difference. First of all (maybe apart from introducing yourself and your role) were I work all of the things on the list get done. And we don’t need yet another piece of paper to fill in! And I can’t believe that a major hospital like Toronto doesn’t use a pulse ox on their patients.
Another site mentioned where a lot of the “safety measures” apparently are not present was London. I think it would be more interesting to compare sites were these things are in place against sites where they aren’t, rather than introducing a list. Which brings us nicely over to the Hawthorne effect. Dear Ann Marie it wasn’t inseiffoliett’s idea, they even mention it themselves in the paper.
Two things make this paper even harder to believe. They say “the list was adjusted to fit into flow of care on each site”. What does that mean? Did they even use the list?
And secondly they mention 6 safety indicators (all on the list) which process was measured. They increased from 34% to nearly 57%. But that would mean that they were not used in about half of the cases wouldn’t it? So what exactly did they do with that checklist? Read it and forget about it?
I can however clearly see how the list improved the teamwork. A common enemy/hatred has never failed to bond a team together.

22 01 2009
Anne Marie Cunningham

@B

I don’t have any problem in acknowleging the Hawthorne effect at all. I’m not sure why you think that I do but I’ll let things rest there.

@inseiffolliet
To be honest I understood the process of introducing the checklist a lot more from reading the comments in the other post, that from this paper.
It sounds like in your experience the discussions which are meant to be part of the process of introducing this checklist- local adaptation to increase ownership etc, aren’t happening and that is why you feel so angry about it.

22 01 2009
inseiffolliet

thanks for the input guys- blogging as a conversation rather than monoogue is better.

can honestly say i’m not angry about this. am not aware of any discussions about the checklist, merely its introduction. that speaks more of the management issue than of team working and perhaps is where the disquiet rises.

are we agreed on the poor science, pragmatic review with probable hawthorne effect BUT good overall idea? teamwork and shared responsibilities.

23 01 2009
Anne Marie

Hello again
I guess I am still having a problem seeing how else a study of this type could have been carried out in the centres involved. As the authors point out it is hard to disentangle the Hawthorn effect. How would the better science study have looked and would it have been realistically achievable?

Thank you for letting me learn more about all of this.
AM

23 01 2009
inseiffolliet

The study design was good but the problem was in the methodology.

They should have instituted the appropriate changes(1), measured the outcomes(2) THEN instituted the 6 point checklist (3) Then compared 2 and 3, not 1 and 3.

Independent data collection was clearly a priority, so why then did it fall apart with only 32% of the information? That should have been maintained and hugely weakens the findings that were made.

Then interpretation of “results” needs to be made explicitly in the face of such problems.

Just because “a study” is done or has big numbers doesn’t make it good. Interpretation of poor results, pragmatic or even hawthornian only leads to further building on poor foundations. Such is the basis of medical science rather than true science and a constant disappointment, particularly when the statistics are then entered into debate as I made in the origin post; “Lies, damned lies and statistics.”

The loss of rigour when looking for the H nought hypothesis in medical literature is a concern to me. We should aspire to good, simple, straighforward studies if we are loking for progress on that basis. Otherwise the literature and practice of medicine will remain bedevilled and confused by arguement about the value and implications of the findings.

I appreciate that that can be difficult and even costly. Moreover I do appreciate that there is some value in the results obtained in this study but as long as we use pragmatic studies and apply them dogmatically we will make much slower progress.

A perfect example of this is the vast difference between oncology management in paediatrics and adult practice. In the former, virtually every child is in a trial, protocol driven treatment and rapid increase in knowledge. The same is not true for adults.

Lies, damned lies and statistics. I don’t know which this is, what I do know is that it isn’t evidence.

3 07 2011
Week Five: Discussion points for Safer Surgery Checklist « Twitter Journal Club

[…] settings and there have been many criticisms made of the methodology of the paper (see this blogpost & this letters page for examples of the criticisms). Is this adequate enough to support the […]

12 08 2011
Week 5 – Discussion points: Safer Surgery Checklist | Twitter Journal Club

[…] settings and there have been many criticisms made of the methodology of the paper (see this blogpost & this letters page for examples of the criticisms). Is this adequate enough to support the […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: