http://www.ambpeds.org/imm.html

 

Evidence-based Immunization Delivery
	The University of North Carolina at Chapel Hill   University of Rochester School of Medicine and Dentistry
Part 1: Audit and Feedback
	Table of Contents

 

visitors since 5/1/98
FastCounter by LinkExchange

Table of Contents

·         Abstract

·         Background

·         Objectives

·         Criteria considering studies for this review

·         Search strategy for identification of studies

·         Methods of the review

·         Description of studies

·         Methodological qualities of included trials

·         Results

·         Findings from excluded studies

·         Conclusions

·         Implications for Practice

·         Implication for research

·         Reference

·         Related web sites

 

Abstract

 

Objective: To review the effectiveness of audit and feedback in improving immunization delivery by health care professionals.

 

Data Sources: We searched Medline from 1966-present, using the search terms: quality of health care, physician’s practice patterns, audit, reaudit, assessment, outcome assessment (health care), feedback, feed-back, fed back, immunization, immuniz$, preventive health services, vaccines, immunization programs, vaccin$, vaccination, inoculat$, measles, rubella, hepatitis B, poliomyelitis, influenza, mumps, pneumococcal infections, and diphtheria-tetanus-pertusis vaccine. Additional studies were obtained from bibliographies of relevant articles obtained and files of study collaborators.

 

Study Selection:

Population: Physicians or other health care personnel who deliver immunizations.

Interventions: Audit and feedback.

Outcomes: Provider’s delivery of immunizations to patients.

Study Design: Randomized controlled trials (RCT), and controlled before-and-after (CBA) and interrupted time series (ITS) studies.

Of initially identified studies, 10.9% met selection criteria.

 

Data Extraction: Studies were read independently by two reviewers, and abstracted using a validated checklist developed by the Cochrane Collaboration Effective Professional Practice. We collected information on study design, blinding, participants, setting and nature of intervention(s), and main outcomes.

 

Data Synthesis: Studies were summarized in tables. Three of the five included studies found that audit and feedback was associated with improvements in immunization rates (-17 to +27 percentage point changes).

 

Conclusions: Audit and feedback is often effective at increasing immunization rates. The effect is modest. Additional studies are needed to identify the optimal format and frequency of audit and feedback, and to examine the long term effect of feedback on provider immunization practices.

[Table of Contents]

 

Glossary of Methodologic Terms:

[Table of Contents]

Background

Immunization is a cost-effective and widely accepted means of preventing disease. However, the percentage of two-year old children up-to-date for immunizations in the U.S. is well below the U.S. Public Health Service's goal of 90%. Adult immunizations, such as influenza vaccination for the elderly, also fall below targeted rates. One of the numerous strategies devised to improve the provision of immunizations and other preventive care services is audit and feedback. Audit in medical practice usually refers to the collection of data regarding clinical performance by review of medical charts, laboratory orders or billing records; while feedback refers to the presentation of such data to the relevant clinician, with or without recommendations for practice. After the implementation of routine audit and feedback to providers in public health clinics in the state of Georgia, the percentage of 2-year olds up to date for immunizations increased from 40% to 89% over eight years. Though the intervention in Georgia was multi-factorial, the apparent positive effect of audit and feedback in that state led the U.S. Congress in 1995 to require that all states receiving federal money for immunizations perform routine audit and feedback of vaccination rates in public clinics. Similarly, the Advisory Committee on Immunization Practices (ACIP) has recommended that private providers, as well as those in public settings, implement routine measurement of vaccination coverage. Despite these recommendations, evidence from the medical literature provides mixed support for audit and feedback.

A review by Buntinx of the effect of audit and feedback on a variety of preventive services found that feedback does increase adherence to clinical practice guidelines, particularly when clinicians are involved in the development of those guidelines. Another review by Thomson concluded that the effect on feedback on various clinical services is generally small and that current evidence regarding optimum frequency and format is too varied to support any recommendations for practice.

It is possible that the effect of audit and feedback differs according to the specific preventive service for which it is used. Because immunization is a service that is generally well-accepted and cared about by both providers and patients (or parents), the effect of feedback may be greater than that seen with other preventive services which are not as universally accepted. Other features unique to immunization delivery, such as the number and frequency of immunizations recommended for children, may reduce the impact of audit and feedback compared to other preventive services. Audit and feedback addresses one barrier to immunization delivery (lack of provider knowledge) by giving providers an objective estimate of their vaccination coverage. Even if audit and feedback has no direct effect, it may motivate providers to examine other potential barriers to immunization delivery in their practice settings (i.e., missed opportunities) and ways to address them. However, because audit and feedback can be time-intensive and expensive for providers to implement, a systematic review of the literature is needed to better define the effectiveness of audit and feedback on immunization delivery and implications for future research.

[Table of Contents]

Objectives

To review the effectiveness of audit and feedback in improving immunization delivery by health care professionals.

 

Criteria for considering studies for this review

Types of participants

Physicians or other health care personnel, including resident physicians, who deliver immunizations to adults or children in academic or non-academic settings.

Types of interventions

Audit and feedback: any summary of clinical performance gathered over a defined period of time which is presented to the health care provider after the information is gathered. Other interventions and recommendations for clinical action may or may not be included with the feedback.

Types of outcome measures

Objective measures of a provider's delivery of immunizations to patients, typically the percent of eligible patients immunized. All results are expressed as absolute percent changes in immunization rates rather than relative rates to allow valid comparisons between studies.

Types of studies

All randomized controlled trials (RCT) examining the effect of audit and feedback on immunization delivery by clinicians treating adults or children were included. We also included controlled before-and-after (CBA) and interrupted time series (ITS) studies if they had appropriate controls and met methodological quality standards (see Methods).

Studies examining the effect of audit and feedback on other preventive services (i.e., cervical cancer screening) without including immunizations were excluded. For studies which included immunizations as well as other preventive services as outcomes, results are reported for the effect on immunization delivery only.

[Table of Contents]

 

Search strategy for identification of studies

We searched Medline from 1966-present using the following search terms: quality of health care (mh), physician's practice patterns (mh), audit (tw), reaudit (tw), assessment (tw), outcome assessment (health care) (mh), feedback (mh), feedback (tw), feed-back (tw), fed back (tw), immunization (mh), immuniz$ (tw), preventive health services (mh), vaccines (mh), immunization programs (mh), vaccin$ (tw), vaccination (mh), inoculat$ (tw), as well as mesh headings for individual vaccines including measles, rubella, hepatitis B, poliomyelitis, influenza, mumps, pneumococcal infections, and diphtheria-tetanus-pertusis vaccine. The bibliographies of all relevant articles obtained and published reviews were reviewed for additional studies. Relevant files of study collaborators were also searched for references.

[Table of Contents]

 

Methods of the review

For each included trial we collected information on the method of randomization or assembly of control groups, blinding, trial participants and setting and nature of the intervention(s). The main outcome extracted was the percentage of a provider's eligible patients who were fully immunized. Data abstraction was performed using a validated checklist developed by the Cochrane Collaboration Effective Professional Practice (CCEPP). All identified studies were read independently by two reviewers. Reviewers were blinded to the study's author and institution. Disagreements were resolved by discussion.

[Table of Contents]

 

Description of studies

This review includes the following studies: Hillman (1996), Buffington (1991), Belcher (1990), Korn (1988) and Tierney (1986). Only one of the five studies meeting inclusion criteria considered childhood immunizations. Immunization delivery was the only outcome measured in the Buffington (1991) study; immunizations were one of several preventive services measures in the other four.

Hillman (1996) analyzed the effect of audit and feedback alone, and audit and feedback combined with financial incentives on provider compliance with pediatric preventive care guidelines, including immunizations. The settings were 53 primary care practice sites in one city, each site having at least 25 pediatric patients.

Buffington (1991) examined the effect of weekly feedback in a group of 45 physicians in 13 private practices over the course of one influenza season. Rates were calculated as the cumulative percentages of each physicians' targeted populations and were displayed prominently in the practices on specially prepared charts. The intervention was studied alone and in combination with postcard reminders sent to patients.

Belcher (1990) compared the impact of a "physician-focused" intervention which combined annual audit and feedback with provider education and generic preventive services chart flowsheets to two "patient focused" interventions (postcard reminders or an invitation to come to a separate clinic devoted to health promotion) in a large multi-specialty VA outpatient facility. Influenza vaccination rates were examined over five years.

In another VA outpatient residency clinic setting, Korn (1988) studied the effect of a combination of interventions which included performance feedback (via a chart review conference), didactic sessions on preventive care and reminders in the form of preventive service flow sheets placed in patients' charts on influenza and pneumococcal vaccine delivery.

Tierney (1986) used a two-by-two factorial design in a nine month study to examine the impact of audit and feedback alone, patient reminders alone and the combination of the two on pneumococcal vaccine administration in a Veterans Administration (VA) clinic staffed by internal medicine residents. Feedback was provided monthly in the form of computer-generated reports that allowed residents to (1) have patients who missed indicated services be rescheduled, or (2) have the encounter form at the patient's next visit marked to prompt the resident to provide the service.

[Table of Contents]

 

Methodological qualities of included trials

Following previously validated methods, we included all randomized controlled trials (RCT’s) in this review and considered controlled before-and-after (CBA) trials and interrupted time series (ITS) that met methodological standards. We assessed numerous quality criteria (depending on study design) as listed in Table 1.

Forty-four studies were identified by the literature search. Thirteen studies were potentially relevant; five studies satisfied the inclusion criteria and the remaining eight were excluded from the final analysis. Hillman (1996), Buffington (1991), Belcher (1990) and Tierney (1986) are RTCs, and Korn is a CBA. Many of the identified studies were excluded because of study designs that failed to meet quality criteria (Table 2, Excluded Studies) or the failure to present interpretable data (i.e. reviews, ecological studies).

Studies used different approaches to randomization; however, none explicitly described the randomization technique used. In Tierney (1986), the authors state simply that resident physicians were randomly allocated to one of four clinic sites; while in Buffington (1991) private practitioners were stratified based on their practice size before randomization. In Belcher (1990), outpatients at a VA hospital were randomized to control or intervention groups. In Hillman (1996) primary care sites were stratified by practice type (pediatrics, family practice and general practice) then randomly assigned to one of three study groups. If unreliable randomization techniques (i.e. by day of week or open random number lists) were used in any of these trials there is the risk that intervention and control groups may differ in significant ways. If such differences are present the intervention's effect may not be accurately estimated.

Follow-up data were collected on at least 80 percent of practice sites in the Hillman (1996) trial, but the extent of follow up on patients was not clearly described. Belcher (1990) obtained follow-up data on less than 80% of patients. The extent of follow-up of providers and patients was not described clearly in Buffington (1991) or Tierney (1986). If patients or providers who were lost to follow-up differ from those who remained in the study with respect to outcomes, the risk of selection bias and distortion in the effect measured exists.

It was unclear in the Belcher (1990) study whether chart auditors were blinded with respect to patient's exposure to the intervention. And, the Hillman study (1996) does not specify whether chart auditors were blinded to study group assignments of the practice sites. Many of the problems of the included studies described above may reflect inadequate reporting rather than faulty methods.

Contamination of the control group was possible in the Korn (1988) study. Residents in the study group received didactic lectures on preventive care, performance feedback during chart review conference and chart reminders during a rotation prior to the study period. The preventive care performance of these residents was compared to that of residents not yet exposed to the intervention during weekly outpatient clinics. Physicians from the control and intervention groups are likely to have worked in close proximity during clinic days (i.e. sharing a workroom, discussing cases with faculty), so it is possible that the control group was influenced by the residents who received the intervention. This potential contamination could have decreased differences between the two groups' performance and led to an underestimate of the true effect of the intervention.

[Table of Contents]

 

Results

Three of the five studies found that audit and feedback was associated with improvements in immunization rates. The magnitude of the effect was -17 to +27 percentage points (Table 3, Included Studies).

 

Comparison 1: Audit and feedback vs. no intervention

Hillman (1996), Buffington (1991) and Tierney (1986) compared audit and feedback alone to no intervention. Buffington (1991) and Tierney (1986) found that audit and feedback increased immunization rates. Over the course of one flu season Buffington (1991) found that influenza vaccination rates increased from 50% in the control group to 66% in the intervention group (p<0.001). Tierney (1986) found that pneumococcal immunization rates increased from 5% in the control group to 20% in the study group (p<0.01). In Hillman (1996), immunization rates for DTP, OPV, Hib and MMR increased 16-37 percentage points over the 1.5 year follow up period for both the audit and feedback group and the control groups, but there was no significant difference between the intervention and control groups.

 

Comparison 2: Audit and feedback combined with other interventions vs. no intervention

Audit and feedback was combined with other complimentary interventions in all of the included studies. Hillman (1996) compared immunization rates for OPV, DPT, MMR and Hib vaccines for three study groups, audit and feedback only, audit and feedback combined with financial incentives, and a control group. Audits occurred at baseline and every six months for three follow up periods. The feedback or feedback plus financial bonuses were distributed to the respective practice sites after the three follow up audits. The immunization rates in the two intervention groups increased over the 1.5 year follow up period. For example, MMR rates increased 33% (52% --> 85%) in the group that received audit and feedback alone and 44% (41% --> 85%) in the group that received audit and feedback plus financial incentives. However, increased 37 percentage points (56% --> 93%) in the control group as well. There were no statistically significant differences in the rates of improvement between the three study groups for any of the immunizations measured. It is possible that the financial incentive was not great enough to facilitate greater increases in compliance with immunization delivery than were already occurring with pediatric preventive care at the time. And, the methods for informing or educating the providers about the immunization compliance may have been insufficient. Forty-three percent of sites responding to a survey indicated they were not aware of the study.

Buffington (1991) included a study group that received weekly feedback and their patients were sent postcards reminding them to come in for their influenza vaccine. Influenza immunization rates of physicians in the combined group increased by 17 percentage points compared to 16 percentage points for physicians who received feedback only (combined group, 67%; audit and feedback only, 66%; control group 50%; p<0.001).

Belcher (1990) found that over five years, physicians who received annual audit and feedback, attended training sessions, and whose patients' charts had flowsheets had 4% lower influenza immunization rates compared to physicians in the control group (63% vs. 67%). The lack of effect in this study may have occurred because the investigators had to combine data from two intervention groups because of poor physician turnout for the annual group feedback sessions.

Korn (1988) found that immunization rates for physicians receiving bi-weekly feedback combined with didactic seminars on adult health maintenance screening, and preventive service flow sheets improved 21 percentage points for influenza vaccinations (28% vs. 7% in controls; p=0.03) and 12 percentage points for pneumococcal vaccinations (28% vs. 16% in controls; p value not significant).

Tierney (1986) found that the pneumococcal immunization rate for the physicians in the combined intervention group (audit and feedback combined with patient-specific reminders to the providers) increased by 27 percentage points (32% vs. 5%; p<0.01), compared to 15 percentage points (20% vs. 5%; p<0.01) in the audit and feedback only group.

[Table of Contents]

 

Findings from excluded studies

Eight additional ITS studies were reviewed, but were excluded from this analysis. Although many showed a positive effect, they were excluded for methodological reasons (see Excluded Studies Table). ITS studies are a weaker design than RCT's or CBA studies due to their lack of a concurrent control group. In the absence of a concurrent control group, such studies cannot exclude the possibility that secular trends accounted for their results. This issue is particularly important for studies of immunization delivery performed between the late-1980s and the present. During this period the United States experienced a major measles epidemic (1989-1991), and since then a considerable amount of attention has been given to improving immunization rates by governmental, professional and lay organizations. Childhood immunization rates have increased nationally during this period.

ITS studies that include single measures of an outcome before and after an intervention are further weakened by the potential instability of pre- and post-outcome measures. Including two or more pre- and post-outcome measures increases the stability of both estimates. The eight ITS studies excluded from this analysis all failed to include two or more measures. The excluded studies are abstracted in the Appendix.

[Table of Contents]

 

Conclusions

We identified numerous studies that examined the effect of audit and feedback on immunization delivery. However, only four of these studies were randomized control trials. The remainder were a mix of controlled before and after and interrupted time series studies. All of the ITS were excluded due to their inability to exclude the role of secular trends in their findings.

Only three studies examined the role of audit and feedback in isolation. Others combined audit and feedback with a variety of co-interventions (e.g. flow sheets, patient reminders, educational programs, financial incentives, and comprehensive quality improvement programs). Thus there is little evidence upon which to assess the independent effectiveness of audit and feedback of immunization delivery. Similarly, in the trials where audit and feedback was combined with other interventions, it is impossible to determine if the multiple interventions had a simple additive effect or some more synergistic effect.

Only one of the included studies, and two of the excluded studies, examined the effect of audit and feedback on childhood immunizations; the remainder examined influenza and pneumococcal vaccine delivery. To the extent that audit and feedback addresses physician and practice barriers to immunization, it is reasonable to assume that similar interventions should also have a beneficial effect on childhood immunization delivery. However, there are important differences between childhood and adult immunization strategies. Influenza and pneumococcal vaccines are designed for targeted populations, and involve either annual or one-time injection schedules. In contrast, all infants must receive 16 separate immunizations in the first 18 months of life.

[Table of Contents]

 

Implications for Practice

The studies reviewed indicate that audit and feedback is often effective at increasing immunization rates. The effect is modest. This finding is consistent with the review performed by Thomson which reviewed the effectiveness of audit and feedback in improving a wide range of health care practices and outcomes. Both reviews identify several unanswered questions. No studies attempt to identify the optimal format (i.e., face-to-face, written or computerized) and frequency of audit and feedback. In addition, there is no data on the long term effect of feedback on provider immunization practices. Finally, it is unclear if studies of the effect of audit and feedback on influenza and pneumococcal immunization practices are generalizable to childhood immunization efforts.

[Table of Contents]

 

Implication for research

Because of the importance of secular trends in immunization practices, interrupted time series studies are not suited to measuring the impact of audit and feedback or other interventions on childhood immunization practices. The use of concurrent control groups is imperative, utilizing either through controlled before and after or randomization approaches. Because data for audit and feedback are most easily obtained from practice-wide data systems, randomization of audit and feedback within practices may be difficult. True concealment of randomization and prevention of contamination is difficult within provider groups such as residency programs or small to moderate sized group practices. Future studies should randomize providers at the practice level. Finally, because of the limited number of studies of audit and feedback in pediatric populations, additional research is needed to determine if this intervention is effective at improving the delivery of childhood immunizations.

[Table of Contents]

 

Table 1. Quality criteria assessed for each study design.9

[Table of Contents]

Table 2. Excluded Studies

 

[Table of Contents]

Table 3. Included Studies

 [Table of Contents]

 [Table of Contents]

 [Table of Contents]

[Table of Contents]

 

References

Appendix: Abstracts of excluded studies and major reviews

LeBaron C., Chaney M., Baughman A., et al. Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994. JAMA 1997; 277(8): 631-635.

Design: ITS

Setting: Non-academic/U.S. public health clinics

Intervention(s): Annual audit of a sample of charts at each clinic with feedback to providers and incentives (peer comparison, awarding of plaques, annual meetings with presentation of successful programs) for clinics to improve coverage. Individual clinics developed strategies to improve their vaccination coverage, e.g. reminder-recall systems and linkage to WIC benefits.

Outcomes: percentage of eligible children receiving 4 doses of DTP vaccine, 3 doses of OPV, and 1 dose of MMR.

Results: 37% increase in primary series completion rates over 6 years. In reviewing the data, the authors ask whether changes in other areas (improved documentation of vaccination, reduced numbers of children to vaccinate, epidemics of vaccine-preventable diseases or national trends) can explain the increase. While other improvements did occur, they cannot account for the magnitude of the effect. Yet since clinics developed varied additional strategies to improve immunization delivery, the effect due to the measurement and feedback process itself is still unquantifiable.

Reason(s) for exclusion: pre- and post-intervention data points insufficient to rule out secular trend.

[Table of Contents]

Ghodes D., Rith-Najarian S., Acton K., and Shields R. Improving diabetes care in the primary health setting. The Indian Health Service Experience. Annals of Internal Medicine 1996;124 (1 part 2): 149-152.

Design: ITS

Setting: Non-academic/U.S. Indian Health Service Clinics

Intervention(s): annual chart audits with immediate feedback to providers.

Outcomes: percentage of eligible patients receiving pneumococcal immunization.

Results: increase from 24% in first annual audit to 59% over 7 years.

Reason(s) for exclusion: Data insufficient to rule out secular trend.

[Table of Contents]

Morrow R., Gooding A., and Clark C. Improving physicians’ preventive health care behavior through peer review and financial incentives. Archives of Family Medicine 1995;4 (2): 165-169.

Design: ITS

Setting: Non-academic/U.S. 418 physician practices in an IPA HMO.

Intervention(s): 1) Annual audit of 20 patient charts with feedback to providers by mail or interpersonally. 2) Financial incentive (physician reimbursement linked to audit results).

Outcomes: percentage of eligible patients receiving MMR immunization.

Results: Reported as mean percentage of practices in compliance with standard for MMR immunization coverage--increase from 78.1% in year one to 95.6% in year three of study.

Reason(s) for exclusion: Pre- and post-intervention data points insufficient to rule out secular trend.

[Table of Contents]

Carey T., Levis D., Pickard C. Development of a model quality-of-care assessment program for adult preventive care in rural medical practices. Quality Review Bulletin 1991;17 (2): 54-59.

Design: ITS.

Setting: rural health care practices

Intervention(s): Two annual chart audits with feedback to practices including comparison to other practices.

Outcomes: percentage of eligible patients receiving recommended preventive care services including influenza immunization. Pneumococcal immunization was included in the second audit only.

Results: no improvement in influenza immunization rates.

Reason(s) for exclusion: Pre- and post-intervention data points insufficient to rule out secular trend.

[Table of Contents]

Chodroff C. Cancer screening and immunization quality assurance using a personal computer. Quality Review Bulletin 1990; August 1990: 279-287.

Design: ITS

Setting: internal medicine residency outpatient clinic of a community teaching hospital.

Intervention(s): 1) computer-generated monthly audit and feedback with peer group comparison. 2) computer-generated tailored provider reminders to perform recommended preventive services.

Outcomes: percentage of eligible patients receiving recommended preventive care services including influenza, pneumococcal and tetanus immunizations.

Results: 43% increase in tetanus immunization over control; +38% for pneumococcal immunization and +49% for influenza. Possible threshold effect for tetanus and pneumococcal vaccines.

Reason(s) for exclusion: Pre- and post-intervention data points insufficient to rule out secular trend.

[Table of Contents]

Barton M. and Schoenbaum S. Improving influenza vaccination performance in an HMO setting: The use of computer-generated reminders and peer comparison feedback. American Journal of Public Health 1990;80 (5): 534-536.

Design: interrupted time series (ITS).

Setting: metropolitan area HMO (university affiliated).

Intervention(s):

Year 1: 1) patient reminder (computer-generated mailed postcard). 2) patient education materials. 3) provider reminder (computer-generated chart prompt to remind MD to immunize high-risk patients < age 65).

Year 2: same as year one with the addition of provider reminders to immunize patients age 65+ and feedback to chiefs of service.

Year 3: previous interventions plus feedback to individual physicians with "periodic" distribution of lists of eligible patients not yet immunized.

Outcomes: percentage of eligible patients receiving influenza immunization.

Results: 42% of patients > age 65 immunized in year 1 increasing to 60% in year 3.

38% of high-risk patients > age 65 immunized in year 1 increasing to 55% in year 3.

Reason(s) for exclusion: Unable to rule out secular trend because study lacks two post-intervention data points. Also unable to estimate effect of audit and feedback because of numerous concurrent interventions targeted at multiple patient groups.

[Table of Contents]

Colver A. Health surveillance of preschool children: four years’ experience. British Medical Journal 1990;300: 1246-1248.

Design: ITS

Setting: Non-academic/U.K. public health clinics (one health district)

Intervention(s): Feedback of information obtained from an immunization database to primary health teams with peer comparison.

Outcomes: percentage of eligible children receiving measles, DTP, and polio vaccines.

Results: 25% increase in measles immunization coverage over five years. Health districts not receiving feedback of information did not see a comparable increase. The number of practices achieving > 90% DTP and polio immunization coverage increased from 22/57 to 54/57.

Reason(s) for exclusion: Lack of data regarding methods of surveillance and data extraction and format of audit and feedback.

[Table of Contents]

Shank J., Powell T., Llewelyn J. A five-year demonstration project associated with improvement in physician health maintenance behavior. Family Medicine 1989;21 (4): 273-278.

Design: ITS

Setting: family medicine residency outpatient clinic.

Intervention(s): 1) provider reminder--health maintenance guidelines flowsheet placed on all adult patient charts. 2) audit and feedback--ongoing audit of residents’ compliance with the flowsheet recommendations by faculty with monthly or bi-monthly individual feedback. Annual audit of 100 randomly selected adult patient charts.

Outcomes: percentage of physicians in compliance with health maintenance guidelines including tetanus, influenza and pneumococcal immunizations.

Results: While the effect of the intervention was positive for all immunization outcomes, only the increase in tetanus immunization was statistically significant (5% compliance in 1983 to 26% in 1986). The authors note that a decrease in influenza and pneumococcal immunization rates in year four of the study may have been due to an interruption in the normal office routine of sending reminder cards to at-risk patients for these immunizations, thus the effect of audit and feedback on these immunizations is less clear.

Reason(s) for exclusion: Pre- and post-intervention data points insufficient to rule out secular trend.

[Table of Contents]

 

Major Reviews

 

Buntinx F., Winkens R., Grol R., Knottnerus J.A. Influencing Diagnostic and Preventive Performance in Ambulatory Care by Feedback and Reminders. A Review. Family Practice 1993;10 (2): 219-228.

This review of 26 studies focuses on the effect of two interventions--audit and feedback and reminders--on ambulatory care services. A reminder is considered any information provided to the clinician before a patient encounter. Of the 26 studies identified, 10 are RCTs; five of the remaining 16 use a concurrent control group and 11 are uncontrolled before-and-after analyses. Targeted activities include reducing diagnostic tests and procedures and adherence to clinical practice guidelines. Of the 10 randomized studies, two examine audit and feedback combined with reminders. The authors found the heterogeneity of studies too great for a quantitative analysis. The conclude that feedback seems to reduce diagnostic test usage and improve adherence to clinical practice guidelines, particularly when clinicians themselves participate in the development of those guidelines. While the majority of studies reviewed took place in academic settings, a positive effect was seen in those in non-academic settings as well. In the randomized controlled trials, immediate reminders appear to have a greater effect than feedback. One study showed a greater effect when peer comparison was added to feedback. Only one of the randomized studies tested audit and feedback as a means to improve vaccination rates (Tierney, 1986); while a positive effect was seen with feedback, the effect of reminders was greater in terms of percent compliance with a preventive care protocol.

[Table of Contents]

Gyorkos T.W., Tannenbaum T.N., Abrahamowicz M., et al. Evaluation of the Effectiveness of Immunization Delivery Methods. Canadian Journal of Public Health 1994;85, Suppl. 1: S14-S30.

This review is organized by type of immunization (DTP, MMR, etc.) rather than by type of practice intervention. Two studies of audit and feedback are included. One (Barton, 1990) tested the effect of combinations of a patient-oriented intervention, a physician reminder and feedback to chiefs of service and individual MD’s on influenza immunization delivery. The effect of audit and feedback could not be isolated from that of the other interventions in this study. Another study examining the effect of audit and feedback on delivery of DPT and polio vaccines to preschool children in public health clinics (Colver, 1990) found a large effect of feedback (94.7% intervention coverage vs. 38.6% control coverage). While feedback of information had a positive effect on childhood immunizations in one study, other interventions such as standing orders to vaccinate and reminder-recall systems had the most effect in the included studies of adult influenza immunization rates.

[Table of Contents]

Mugford M., Banfield P., O’Hanlon M. Effects of feedback of information on clinical practice: a review. BMJ 1991;303: 398-402.

This review of 36 studies focuses on the effect of audit and feedback on a range of clinical activities from number of surgical procedures to ordering of cancer screening tests. Only one study focuses on preventive measures other than cancer screening. In general the authors conclude that feedback is most effective when it is directed at those who have expressed a desire to review their practice and when it is given close to the time of clinical decision making. No conclusions regarding the optimum format for feedback could be drawn.

[Table of Contents]

Oxman A.D., Thomson M.A., Davis D.A., Haynes R.B. No Magic Bullets: A Systematic Review of 102 Trials of Interventions to Improve Professional Practice. Canadian Medical Association Journal, 1995;153 (10): 1423-1431.

This review of various healthcare practice interventions identified 31 studies of audit and feedback, including 21 randomized controlled trials. Fourteen of the controlled studies combine audit and feedback with another intervention. As in another review, feedback and reminders were analyzed together and the effect ranged from zero to "moderate." Results for individual studies are not reported.

[Table of Contents]

Thomson M.A., Oxman A.D., Haynes R.B., Davis D.A., Freemantle N., Harvey E.L. The effectiveness of audit and feedback in improving health care professional practice and health care outcomes. In: Bero L, Grilli R, Grimshaw J, Oxman A (eds.) Cochrane

Collaboration on Effective Professional Practice Module of The Cochrane Database of Systematic Reviews, [database on disk, CDROM and online; updated 03 June 1997]. The Cochrane Collaboration; Issue 3. Oxford: Update Software; 1997(updated quarterly).

·  This review includes 32 randomized controlled trails (RCTs) of the effect of audit and feedback on a range of clinical activities including laboratory test utilization, drug prescribing, hypertension management, and preventive care. At least three trials include immunization delivery as a health care outcome (Buffington 199l, Tierney 1986, Winickoff 1984). One trial includes recording of immunization status in the medical record as an outcome (Mayefsky 1993). Problems in the methodological quality of trials include unclear concealment of randomization, inadequate follow-up of health care professionals, and unblinded assessment of outcomes. Feedback given is most often provided as a computer-generated report rather than interpe