OBJECTIVE: To assess the robustness of patient responses to a new national survey of patient experience as a basis for providing financial incentives to doctors. DESIGN: Analysis of the representativeness of the respondents to the GP Patient Survey compared with those who were sampled (5.5 million patients registered with 8273
general practices in England in January 2009) and with the general population. Analysis of
non-response bias looked at the relation between practice
response rates and scores on the survey. Analysis of the reliability of the survey estimated the proportion of the
variance of practice scores attributable to true differences between practices. RESULTS: The overall
response rate was 38.2% (2.2 million responses), which is comparable to that in surveys using similar methodology in the UK. Men, young adults, and people living in deprived areas were under-represented among respondents. However, for questions related to pay for performance, there was no systematic association between
response rates and questionnaire scores. Two questions which triggered payments to
general practitioners were reliable measures of practice performance, with
average practice-level reliability coefficients of 93.2% and 95.0%. Less than 3% and 0.5% of practices had fewer than the number of responses required to achieve conventional reliability levels of 90% and 70%. A change to the payment formula in 2009 resulted in an increase in the
average impact of
random variation in patient scores on payments to
general practitioners compared with payments made in 2007 and 2008. CONCLUSIONS: There is little evidence to support the concern of some
general practitioners that low
response rates and
selective non-response bias have
led to systematic unfairness in payments attached to questionnaire scores. The study raises issues relating to the validity and reliability of payments based on patient surveys and provides lessons for the UK and for other countries considering the use of patient experience as part of pay for performance schemes.
