Beware of Samples

nqican

7 years ago

Thank you for the positive feedback and support on our first blog. We received almost unanimous support which is really encouraging, therefore we’ll continue and I’ll update you as time goes by on my own QI project to reduce my cholesterol levels.

As promised I wanted this blog to focus on the pitfalls of analysing and interpreting data based on samples as I feel this is an important skill that is often overlooked.

Resources within clinical audit are stretched like other areas of the NHS – at a time when there is an increased need to provide assurance or improvement. This combination sometimes creates pressure on project teams to “make good” of what information is collected and report to meet deadlines, even if it isn’t in-line with the project plan.

As analysts its easy to switch on the autopilot and calculate percentages, create pie charts etc and send back the analysis as requested and present the results as if they were matter of fact regardless of how the data was obtained. However it is our duty to critique the data and present it accordingly detailing limitations. It is really important for all involved (clinicians, patients, the organisation and your own reputation) that we get it right. Acting on sub-standard data can be worse than doing nothing at all, as it can provide false assurance or initiate a change in practice when there wasn’t actually a problem.

So how do we know that the dataset we are analysing is fit for purpose? I’m sure answers to this question would vary greatly, as I’m not entirely sure there is enough support and training provided on this at present.

The Health Foundation recently highlighted that there is a current lack of skilled analysts within the NHS so this presents a problem which will take time to address.

So to help in the interim – I wish to share my knowledge on this matter via a set of 5 tips to highlight key aspects of analysing data based on samples and where to highlight possible limitations. This has been gleaned from over 15 years analysing healthcare data since graduating (with a joint statistics and business BSc). My wife is currently reading Davina McCall’s autobiography – which is entitled ‘Lessons I’ve Learned – I’ve made mistakes so you don’t have to’ so this is something along those lines (Other autobiographies are available but not as aptly entitled :-)).

Tip 1 Make sure you are clear what the aims and plan of the project are – Is the project for assurance, improvement or research?

Experience over time has taught me the importance of understanding the reasoning behind the project, so keep your analysis focused around the aims. Assurance and improvement data analysis require different approaches so its essential to understand the differences between the two. Sometimes a pragmatic approach is required to find a balance around what is needed and what is possible but this needs to be explained.

Tip 2 – Understand what sampling techniques have been used?

A) Is it clear what the population is that you’ree trying to make conclusions about?

B) is it clear how the sample was selected? Is the reasoning explained?

C) How confident do we need to be in this data? What margin of error can you accept?

I have provided links to sampling websites below that can help with this so that you can identify how representative a sample is and the margins of error – both planning and critiquing retrospectively. There are also links to guides on sampling for clinical audit so this is worth referring to especially around the techniques you can use to select your sample. The most important thing here to assess is, have we introduced a bias from how the sample was drawn and do we have enough data for us to be confident that our sample contains the true result for our population?

Tip 3 – Have SMART questions been used?

Its very important to use questions that are SMART (Specific, measurable, achievable, realistic & timely) when collecting data to minimise individual interpretation especially if more than one person is collecting data / completing survey. None compliance or levels of satisfaction can sometimes be lower if your response options have no opt out (NA / contra-indication or no opinion). If you collect information that is not linked to your aims this is potentially an information governance concern (we plan to write future blogs on this matter).

Tip 4 – What checks / validation has been undertaken to ensure the data has been collected accurately and consistently?

Who has collected the data? If it was more than one person check for consistencies – usually best to collect the first few cases together and discuss the interpretation of questions/answers to resolve any misunderstandings before going alone. Pilots also highlight variations that you may have missed in your planning. Inter reliability checks are ideal to test this but are often not used due to limited time and resources. Also it is best practice to carry out a senior review for any cases that are non-compliant to ensure that there wasn’t a valid reason that had not been previously included as a valid option.

Tip 5 – Use confidence intervals

A sample only provides us with a point estimate about the true result for the population. So by adding confidence interval to our sample value it provides us with a range of compliance / satisfaction that we can be confident (often 95%) that it contains the true population result. In clinical audit we usually look at proportions (%) of compliance so make sure you use the right forumula – useful link here if you want to miss out the calculation part.

In Summary

It’s obviously limited in a blog as to how much detail I can provide but hopefully this stimulates some further thinking and review of your own data analysis / interpretation skills. I have added a few references below for additional reading if you’re interested and where training is available. HQIP have produced a very good guide to ensuring data quality in clinical audits and I recommend reading this as a starting point. Please add a comment below if you have anything relevant to add / share.

These views are my own and do not reflect any NQICAN discussion other than general experience obtained. I hope this will start the conversation across our clinical audit networks (and beyond) & perhaps provide a basis for future network training sessions.

The clinical audit support centre recently shared some results from their annual ‘state of clinical audit’ survey. I tweeted that the survey had limitations and promised to explain why so hopefully I have now indirectly done this. It would be great to see some of these tips applied to final analysis to give the report & future surveys increased value for the clinical audit community & beyond.

Thanks for taking the time to read this blog – well done if you’ve made it to the end 🙂

Feedback as always very welcome.