http://bmj.com/cgi/content/full/324/7337/598
| Home | Help | Search/Archive | Feedback | Table of Contents |
|
||||||||||
 Collections under
which this article appears: Other
Medical Informatics Other
Pediatrics World
Wide Web |
How
to find the good and avoid the bad or ugly: a short guide to tools for rating
quality of health information on the internet
Commentary:
On the way to quality
Petra Wilson
Directorate General for the Information Society (Applications relating to Health), European Commission, 1049 Brussels, Belgium
Health related websites are frequently accessed on the internet. A poll in August 2001 concluded that almost 100 million American adults regularly go on line for information about health care.1 As over 100 000 sites offer health related information, "trying to get information from the internet is like drinking from a fire hose, you don't even know what the source of the water is." 2 3
To help users discriminate between sites, a wide range of organisations have developed methods and tools for evaluating and rating the quality of websites. These tools aim to guide the site developers, filter content, and help consumers become discerning users of information.
A range of tools for rating quality exists, and their number has continued to
grow since 1996 when the first initiatives produced codes of conduct
for health information on the internet. 4
5 Some approaches focus on setting ethical
standards and promoting the "good" whereas other more pragmatic
approaches concentrate on sifting huge amounts of information into
manageable chunks. Some approaches address general ethical principles
about the nature of health related content whereas others focus on
the mode of delivery and the integrity of the use of the web as a
medium for the dissemination of information. I describe a
classification of five types of approaches for rating the quality of
English language websites (table). All start from the basic concept
of an agreed set of criteria for good practice in the provision of
health related information through websites.
| Summary points
|
|
 |
Codes of conduct |
|---|
Codes of conduct are defined as sets of quality criteria that provide a list
of recommendations for the development and content of websites. These
codes inform a process of self assessment by providers of websites
and educate both providers and consumers of websites about "good"
practice so that providers adhere to the codes and consumers grow
wary of sites that do not. Several organisations are developing a set
of quality criteria for health related websites (box), but the extent
to which such codes are implemented varies. Where the code is adopted
by an umbrella organisation such as national or specialty based
medical associations, the associations ensure that members comply
with the code and may discipline members who are not compliant (box).
However, some codes have been adopted by a group of individuals whose
sole purpose is to draft the code rather than to oversee its
implementation (box).
| Organisations
responsible for codes of conduct
eHealth Code of Ethics of the Internet Health Coalition (www.ihealthcoalition.org/ethics/ethics.html) is one organisation developing a set of quality criteria for health related websites The American Medical Association (www.ama-assn.org/about/guidelines) oversees the quality of several websites and disciplines providers that do not comply with its criteria Health Summit Working Group (www.mitretek.com) from north America created a code but did not oversee its implementation eEurope Draft Good Practice Guidelines for the Health Internet (europa.eu.int/information_society/eeurope/ehealth/quality/draft_guidelines/) seeks to stimulate the development and implementation of codes of conduct in member states of the European Union |
Costs and benefits
Creating codes of conduct has few costs, only an outlay for meetings
to draw up the code. But low costs can affect consumers because the
absence of any enforcement mechanisms may mean that the code has a
limited life.
|
| (Credit: SUE SHARPLES) |
| |
Self applied code of conduct or quality label |
|---|
A quality label (logo or symbol) is displayed on screen and represents a
commitment by a provider to implement or adhere to a code of conduct.
A site can display the label only after submission of a formal
application and acknowledgement of a commitment to the principles.
The site may be checked by the label provider, and users may report
misuse of the label.
| Quality labels
Health On the Net Foundation (www.hon.ch) produces the oldest, and perhaps best known, quality label (currently used by more than 3000 websites) Hi-Ethics code (www.hiethics.com/Principles/index.asp) produces a quality label, mainly for commercial sites |
Costs and benefits
Self applied labels are comparatively cheap for both the site
provider and the label provider. The label provider supports a small
team that processes applications, maintains random checks of sites
displaying its label, and responds to any reports of misuse. The site
provider ensures compliance with a simple set of criteria in the
design and implementation of the site. Consumers may benefit because
their attention is drawn to the importance of the principles inherent
in the label. Such benefits must be weighed against the requirement
of consumers to understand the nature of the label and, perhaps more
importantly, to care about its aims and objectives.
| |
User guidance systems |
|---|
A user guidance system enables users to check if a site and its contents
comply with certain standards by accessing a series of questions from
a displayed logo. Tools may be specific, general, or targeted at
particular categories of users (box).
| User guidance systems
DISCERN (www.discern.org.uk) is a brief questionnaire for users to validate information on treatment choices NETSCORING (www.chu-rouen.fr/dsii/publi/critqualv2.html) gives guidance on all health related information QUICK (www.quick.org.uk) provides children with a step by step guide to assessing health related information on the internet |
Costs and benefits
The costs to the provider are none, and the costs to the developer of
the guide are low, often not extending beyond the initial development
costs. However, since the burden of the use of the tool falls
entirely on the consumer, the extent to which it is used, and thus
its real benefit, may be small.
| |
Filtering tools |
|---|
Filters, applied manually or automatically, accept or reject whole sites of
information based on preset criteria. These tools are based on the
"gateway" approach to organising access to the internet
that
is, resources are selected for their quality and relevance to a
particular audience. The resources are reviewed and classified and
the descriptions stored in a database. These tools improve the recall
and precision of internet searches for a particular group of
consumersfor example,
OMNI is aimed at students, researchers, academics, and practitioners
in the health and medical sciences (box).
| Filtering tools
OMNI (www.biome.ac.uk/guidelines/eval/factors) provides a gateway to evaluated, quality resources in health and medicine |
Costs and benefits
The costs of creating a filtering tool are relatively high because
trained experts are needed to review and classify the information.
Filtering tools provide a valuable shortcut to searches using
non-specific search engines.
| |
Quality and accreditation labels awarded by third parties |
|---|
Quality and accreditation labels are logos or symbols awarded by a third
party, usually for a fee, to inform consumers that a site provides
information meeting current standards for content and form. This is
the most advanced approach for quality rating as a third party
provides a label as a result of its own investigation and certifies
that the site complies with quality criteria. No third party
accreditation bodies are fully operational in Europe yet, although
two pilots are running (box).
| Third party quality
and accreditation labels
MEDCERTAIN (www.medcertain.org/) and TNO QMIC (www.health.tno.nl/en/news/qmic_uk.pdf) are running pilot schemes for third party accreditation bodies in Europe URAC (www.urac.org/) has started a health website accreditation programme; it recently processed 20 applications by US websites for formal accreditation lasting a year |
Costs and benefits
Third parties range from intra-organisation bodies offering their
services at low cost, similar to those responsible for the CE mark on
electrical goods sold in the European Union, to high cost external
independent assessors who perform audits and grant
accreditation.
| |
Discussion |
|---|
So, what is the value of this wide range of tools and applications? No organisation or label has the capacity to identify objectively what is good or bad information. Quality remains an inherently subjective assessment, which depends on the type of information needed, the type of information searched for, and the particular qualities and prejudices of the consumer.
Delamothe questioned the value of codes of conduct, rating instruments, and user guides that have proliferated over the past few years, and urged legislators and policymakers not to add to their number. He argued that consumers will cope with the content of websites as they have coped with other media "unassisted by kitemarks," despite the reality that "much of their content contains medical information that is wrong, incomplete, and unbalanced from the point of view of anybody except its originators."6 Yet to argue thus is to misunderstand the objective of most quality rating tools, which is not to inhibit publication, but to provide a system by which consumers can assess the nature of the information they are accessing.
As consumers of traditional media we have learnt to use a wide range of assessment tools. We have learnt to judge the nature of the outlet providing the information (mainstream bookshop or provided by the author), the look and feel of the publication (magazine or one page pamphlet), and we know who to contact for further information (librarian, bookshop assistant, publisher). For the internet, however, we still have to learn to read the signs of quality relevant to our needs. It is for this reason that quality marks and user guides have proliferated. Just as selling a magazine through the right retailer attracts a particular market, so a label such as HON or MedCertain may help consumers assess the information and its provider. It may also allow the provider to gain a foothold in an already crowded market.
It can be argued therefore that labels, codes, and guidance tools that assist
consumers to identify information that meets their subjective
understanding of quality are useful. However, to argue thus makes one
large and fundamentally flawed assumption: that consumers have the
time, energy, and inclination to use the tools appropriatelythat
is, to apply the scoring chart, to check the currency and validity of
a label, to access the filtering site, and so on. As such, tools
place a burden on consumers, which represents "a serious threat to
the sustainability and maintenance of the quality standards."7
The greatest challenge is not to develop yet more rating tools, but to
encourage consumers to seek out information critically, and to
encourage them to see time invested in critical searching as
beneficial. It may be argued that the only way to do this is to have
a centrally controlled system that would offer quality labels on a
par with the CE mark or through the adoption of a gold standard code.
8 9 It can be argued
that no single tool or enforcement body can meet this need. Rather,
that consumers will become proficient in accessing health on the
internet with time, just as we have become critical consumers of
advertising. It can only be hoped that on the road to such savvyness
users of the internet for health information will not fall foul of
too many ugly sites nor consume too much information that turns out
to be bad for them.
| |
Acknowledgments |
|---|
The opinions expressed are those of the author and do not necessarily reflect the position of the European Commission.
| |
Footnotes |
|---|
Funding: None.
Competing interests: None declared.
| |
References |
|---|
| 1. | Harris Poll. www.harrisinteractive.com/harris_poll/index.asp (accessed 14 Nov 2001). |
| 2. | Eysenbach G, Eun Ryoung Sa ER, Diepgen TL. Shopping around
the internet today and tomorrow: towards the millennium of cybermedicine.
BMJ 1999; 319: 1294 |
| 3. | McLellen F. "Like hunger, like thirst": patients, journals
and the internet. Lancet 1998; 352 (suppl II): 39-43S |
| 4. | Jadad A, Gagliardi A. Rating health information on the
internet: navigation to knowledge or to Babel. JAMA 1998; 279(8):
611-614 |
| 5. | Gagliardi A, Jadad A. Examination of instruments used to
rate quality of health information on the internet: chronicle of a voyage
with an unclear destination. BMJ 2002; 324: 569-573 |
| 6. | Delamothe T. Quality of websites: kitemarking the west
wind. BMJ 2000; 321: 843-844 |
| 7. | Risk A, Dzenowagis J. Review of internet health information
quality initiatives. J Med Internet Res 2001; 3(4): e288 |
| 8. | Rigby M, Forsström J, Roberts R, Wyatt J. Verifying quality
and safety in health informatics services. BMJ 2001; 323: 552-556 |
| 9. | Darmoni SJ, Haugh MC, Lukas B, Boissel JP. Level of
evidence should be gold standard [letters]. BMJ 2001; 322: 1366 |
(Accepted 21 January 2002)
Ahmad Risk
3 Adelaide Crescent, Brighton BN3 2JD
Is that it? Encouragement and hope? Encouragement for citizens to assess critically health information on the internet, combined with the hope that they will grow wise and seek only the beautiful?
Good quality health information can help fulfil the promise of better health
for all citizens of the world. There's more that we could and should
do to ensure that it's available.
| |
What doesn't work |
|---|
All the major quality initiatives listed in Wilson's paper have one or more of the following limitations1:
We expect a lot of the seekers of health information. They must learn how to critically appraise information, determine its relevance and validity of context, compare various sources of information for cross validation, and really care about quality. Some citizens manage this, but they are probably a tiny minority. So far, no vision has emerged of how the great majority might be converted. Their indifference and ambivalence will continue to be formidable obstacles. A better strategy may be to educate providers to put high ethical and quality standards at the heart of their organisations.
For providers, the cost of implementing quality programmes varies from little (self certification) to enormous (third party accreditation). The important cost, however, is in the long term maintenance of these efforts.
A serious limitation of the current quality initiatives is that they have
originated in the developed world, most of them from English speaking
countries. Though the internet's potential to improve the health of
people in the developing world is enormous, so too is its potential
for doing harm, particularly where regulatory systems are
weak.
| |
What might work |
|---|
The challenge in quality assurance on the internet is to make it work "out of the box" in a transparent way that fulfils two objectives:
|
For this to happen, we need advances in the design of intelligent web
browsers and search engines, development of systems of making
information understandable by machines, andcruciallybetter
understanding of the principles of the semantic web2
and their widespread implementation by providers of health
information on the internet. On the other hand, quality assurance
could be achieved through a system of certification by a trusted and
credible organisation, which has a well known brand that is
recognisable throughout the world.
| |
Quality: the personal practice |
|---|
What decides the quality of a website for me? This depends on my particular information needs at the time and how much trade-off I am prepared to accept. The quicker and easier the website fulfils my information needs, the higher I rate its quality (box 1). However, I also have a set of demands that a website must fulfil before I look any further (box 2). Ethical standards loom large in these generic criteria.
Reputation matters more to me than any other factor in assessing the quality of websites. Hernández-Borges et al have found a positive correlation between the number of inbound links to a health website (reputation) and the likelihood of that site conforming to the quality criteria of the Health on the Net Code.4 This supports my intuitive decision to make the search engine Google5 the starting point for finding good health information: Google ranks websites partly by the number of inbound links to a given site.
Perhaps here lies the answer to the question of how to get good health
information on the internet: do what we do in the rest of our lives,
and rely on reputation, sometimes.
| |
References |
|---|
| 1. | Risk A, Dzenowagis J. Review of internet health information
quality initiatives. J Med Internet Res 2001; 3(4): e28 |
| 2. | Berners-Lee T, Hendler J, Lassila O. The semantic web. A new form of web content that is meaningful to computers will unleash a revolution of new possibilities. Sci Am 2001 May:35-43 www.scientificamerican.com/2001/0501issue/0501berners-lee.html (accessed 23 Jan 2002). |
| 3. | Rippen H, Risk A, for the e-Health Ethics Initiative.
e-Health Code of Ethics. J Med Internet Res 2000; 2(2): e9 |
| 4. | Hernández Borges AA, Macías Cervi P, Torres Álvarez de Arcaya ML, Gaspar Guardado MA, Ruíz Rabaza A, Jiménez Sosa A. Rate of compliance with the HON code of conduct versus number of inbound links as quality markers of pediatric web sites. Proceedings of the 6th world congress on the internet in medicine, Udine, Italy, Nov 29-2 Dec 2001. http://mednet2001.drmm.uniud.it/proceedings/paper.php?id=75 (accessed 2002 Jan 23). |
| 5. | Brin S, Page L. The anatomy of a large-scale hypertextual
Web search engine. Computer Networks and ISDN Systems 1998; 30:
107-117 |
|
||||||||||
|
Collections under
which this article appears:Other
Medical Informatics Other
Pediatrics World
Wide Web |
Read all Rapid Response responses
| Home | Help | Search/Archive | Feedback | Table of Contents |
ALL INFORMATION, DATA, AND
MATERIAL CONTAINED, PRESENTED, OR PROVIDED HERE IS FOR GENERAL INFORMATION
PURPOSES ONLY AND IS NOT TO BE CONSTRUED AS REFLECTING THE KNOWLEDGE OR OPINIONS
OF THE PUBLISHER, AND IS NOT TO BE CONSTRUED OR INTENDED AS PROVIDING MEDICAL OR
LEGAL ADVICE. THE DECISION WHETHER OR NOT TO VACCINATE IS AN IMPORTANT AND
COMPLEX ISSUE AND SHOULD BE MADE BY YOU, AND YOU ALONE, IN CONSULTATION WITH
YOUR HEALTH CARE PROVIDER.