Download hearing - Plymouth University

WHAT IS THE EVIDENCE FOR THE BENEFITS AND OUTCOMES
FOR DIGITAL HEALTH SERVICES?
FINAL REPORT
NOVEMBER 2009
Prepared by: Professor Ray Jones and Lesley Goldsmith (University of Plymouth) with
assistance from Daniela Cassola (University of Plymouth), Mark Duman and Sarah Smith
(Patient Information Forum) and contributors to a ‘project website’ from the ‘e-health
community’.
Funded by, and submitted to, NHS Choices.
University of Plymouth, 2009.
1
CONTENTS
Executive Summary
1. Introduction
2. Other reviews
3. Extent of RCTs and systematic reviews on digital health services
3.1 Accessing Care and Patient Choice.
3.2 Healthy Living
3.3 Screening Services
3.4 Quality of primary care consultations
3.5 Long Term Conditions: cancer, diabetes, stroke, mental health
4. Information Prescriptions
5. Digital Divide
6. Similar services to NHS Choices
7. Methodological difficulties in assessing benefits
8. Discussion
9. Appendix: Call for proposals from NHS Choices
10. References
2
EXECUTIVE SUMMARY
Accessing Care and Patient Choice
Are digital health services enabling informed patient choice of treatment option?
There is weak evidence that patients may use information from the Internet to avoid
health service contact or may discontinue treatment. Decision aids have enabled
informed patient choice and there is potential to make these available via an Internet
repository.
A systematic review of good quality research studies shows that decision aids performed
better than usual care (normally patient-clinician discussion) interventions in psychological
effects such as greater knowledge, lower decisional conflict, and greater participation in
decision making. We do not know how many of these have been adopted as routine
practice. Furthermore, they were not necessarily delivered online, but clearly could be.
However, another study has shown that it is not easy to find decision aids online. There does
not appear to be any central digital repository or guide and awareness, and services to help
patients with treatment choice, or choices of screening, could benefit from wider digital
accessibility The further work needed in this area is the development of proven decision aids
as online resources in a central repository and raising awareness of that resource. NHS
Choices could play a major role in developing greater awareness both in clinical staff and
patients. We have not explored how this relates to patient accessible information produced
by NICE but discussions with NICE on this topic would seem worthwhile.
Empowering patients with information also means giving them the option not to consult, or
not to follow the doctor’s advice. A 2009 US study found that 11% (of 562) had used
information obtained via the internet to refuse or discontinue treatment recommended by a
doctor or dentist. A 2001 UK study found over one-third (of 1068) said that their condition
had improved after having visited a website and more than a quarter said that Web
information had resulted in a deferred or avoided visit to the doctor. Of course, both these
studies are limited by being self reporting by self-selected online participants who may have
taken those actions with a placebo or no intervention. Studies of the impact on raising
awareness in defined populations (e.g. through cluster randomised trials) on safe self-care
Vs. health service consultation, would be worthwhile.
Are digital health services enabling patient choice in finding and selecting
providers?
Few people use NHS Choices in choosing a hospital, but choosing between hospitals or
primary care providers is not currently a high priority for most people.
A 2008 US systematic review of the impact of publishing performance data found that it had
only a modest impact on health plan selection. Services to help patients choose providers
are evaluated by NHS commissioned work; a large Ipsos Mori survey in 2008 for the
Department of Health found that only 5% used NHS Choices in choosing a hospital. This is
supported by a very recent Kings Fund survey which found that only half the patients had
been offered a choice and of those only 4% had consulted NHS Choices in choosing a
hospital for referral. However, a recent review found that choosing between hospitals or
primary care providers is not currently a high priority for the public except where local
3
services are poor. Other literature suggests that patients may struggle to use some
comparative data and that they prefer approaches using ‘people like me’.
Is scorecard functionality effective at influencing a patient’s choice of providers?
We only found one US publication. In that, the scorecard was not effective.
How effective are digital health services at providing an end to end service, e.g.
all the way through to accessing Choose & Book?
We do not know. We only found one paper reviewing the end of this process, i.e.
Choose and Book from a GP’s point of view. There was considerable local variation.
The only relevant literature that we found was a review by the BMA on Choose and Book
itself, and from the clinician’s point of view. Potential benefits had been identified by GPs
and there was a need to learn from regions where it worked well.
Are digital health services helping patients achieve care in their preferred setting
(home/self care vs visiting a GP)?
Telehealthcare and assistive technologies may well help patients achieve care in their
preferred settings but was not in the scope of the review. Decision aids (see above) may
help.
Clearly, there is a large body of work on telehealth, telecare, telemedicine etc., in which
digital health services help patients to be treated either in the home or at local health
services rather than more remote specialist care, but NHS Choices had said that this did not
need to be included in this report. We are not aware of other types of digital health services
that help patients in this way, other than the decision aids (above), or the weak evidence on
deferred visits to the GP (see below under primary care consultations).
Healthy Living
Do people using digital health services have an improved understanding of
important health determinants such as calorific intake, fat, alcohol units, smoking,
five-a-day?
Yes, there is evidence that different interventions can result in improved understanding.
Are digital health services bringing about healthy lifestyle changes?
Yes, there is evidence that at least in the short term people have improved diet, tobacco
and alcohol abuse, sexual behaviour.
Are there improved health outcomes as a result of use of digital health services?
Most studies are on too short a timescale to show improved health outcomes, and there is
less evidence on maintenance of changes in lifestyle. However, there is enough
epidemiological evidence about lifestyle and health that would allow modelling of health
impact.
A 2008 meta-analysis of 75 studies concluded that computer-delivered interventions can
lead to “immediate post-intervention improvements in health-related knowledge, attitudes,
and intentions as well as modifying health behaviours such as dietary intake, tobacco use,
4
substance use, safer sexual behavior, binge/purging behaviors, and general health
maintenance.” However, they concluded that computer-delivered interventions “do not
provide benefits in all contexts; the evidence does not support the use of CDIs to improve
physical activity, weight loss, or diabetes self-management”. An earlier (2004) and smaller
review (22 studies) had concluded that Web-based interventions compared to non-Webbased interventions showed an improvement in outcomes such as increased exercise time,
increased knowledge of nutritional status, increased knowledge of asthma treatment,
increased participation in healthcare, slower health decline, improved body shape
perception, and 18-month weight loss maintenance. The more recent review probably is the
more accurate picture but, given the heterogeneity of studies, there may be room for
alternative interpretations.
Digital health services which seem to have had positive effect on knowledge, attitudes,
behaviour, or health include:
Smoking: almost half of the ‘first generation’ interventions (in which participants were mailed
computer-generated feedback reports) published between 1995 and 2004 reported
significant reduction in smoking relative to a comparison group. Two of the 4 studies
involving adolescents reported a significant reduction in smoking initiation and prevalence as
a result of computer-tailored material. More recent studies also show success in helping
people reduce or quite smoking. Conclusions from a 2004 study were that ‘smoking
cessation intervention can be delivered via the internet with some degree of short-term
efficacy’. A 2008 RCT of 5 five different web-assisted tobacco interventions (WATIs) found
quit rates of 10-13% at 13 months for the different interventions and concluded that tailored,
interactive websites may help smokers to quit and maintain cessation. There are various
caveats and lessons to be learned and contextual issues to consider. For example, another
2008 review found that web-assisted tobacco interventions (WATIs) were improving in their
quality including more interactive features but could still be improved by more
personalisation and efforts to sustain behaviour change. In 2007, Canadian smokers were
marginally less likely to be internet-users than non-smokers, but some 40% of them were
interested in a web-assisted tobacco intervention. Not surprisingly, greater adherence to the
intervention results in greater quit rates and the source of the message, the degree of
tailoring and the timing of the exposure to the intervention are important.
Alcohol: there are fewer published reports of primary research or reviews for reduction in
alcohol use than for smoking cessation and the evidence is weaker. Nevertheless, there is
accumulating evidence of successful outcomes for at least some groups. For example, one
continuing UK website (DownYourDrink) found measures of dependency, alcohol-related
problems, and mental health symptoms all reduced for the 17% of registrants who
completed the whole 6 weeks, but the comparison was with baseline and not with controls.
The problem with all sensible drinking interventions, as with all web-based healthy lifestyle
interventions, is in attrition both from the point of view of the success of the intervention but
also in the methodological problems it raises.
Weight loss, healthy diet and physical activity: There are large number of both primary
studies and systematic reviews, with a great deal of variability in both. This makes synthesis
and a conclusion difficult. Nevertheless, the most recent reviews conclude that evidence for
second generation computer-tailored interventions for dietary behaviour change is fairly
strong, and that weight reduction interventions are ‘promising’. However, we were not able to
find the websites used in the various RCTs easily. By contrast various commercial websites
were easily found yet apparently with little evidence of their effectiveness. A 2005 review of
US commercial weight loss sites concluded that they were associated with high costs, high
attrition rates, and a high probability of regaining 50% or more of lost weight in 1 to 2 years.
5
Sexual health: A recent meta-analysis of e-health interventions to modify HIV-related sexual
risk behaviour suggested that these were effective but, as with other health lifestyle
interventions suffered from problems of attrition. The authors concluded that digital health
services (including Internet, mobile phone and other) were lower cost and more flexible than
human-delivered interventions. The importance of getting the intervention right is illustrated
by a recent study of a single Internet session promoting condom use amongst use amongst
18-24 year olds. The impact was minimal and they concluded that Internet-based
interventions need to be more intensive to see greater effects.
There is evidence (above) that online support for healthy living can have benefit. However,
methodological criticisms mean the evidence is weak and the effect size relatively small.
The marginal cost of providing online support is likely to be very low; more research is
needed which models longer term costs and benefits. There may be a role for NHS Choices
in making available the best features of evaluated health living interventions.
Are digital health services successful in reaching the C2DE population and
reducing the health inequality gap between socioeconomic groups?
We did not find papers addressing the first part of this question, although we would
assume that most interventions have participants from C2DE. However, national statistics
suggest that it is most unlikely that digital health services will reduce the health inequality
gap; it is much more likely they will increase it. Steps should be taken therefore to make
digital health services as accessible as possible.
There is still a major difference in levels of access to the Internet by socio-economic group
(35% in DE compared to 82%) and by age (10% of those aged 75+ compared to 82% of
those aged 16-19). There is also an urban-rural split with many rural areas having very low
Internet connection speeds.
Quality of primary care consultations
What effect do digital health services have on the efficiency of GP consultations?
We do not have too much evidence on the efficiency of GP consultations. A minority of
patients take information to the consultation and GPs with good communication skills can
make use of that. Computer patient interviewing prior to the consultation could make
consultations more efficient and is used widely in the USA. There is a need for studies of
its use in the UK.
In 2000, in the USA, 91% of online health seekers reported they had learned something
new, 55% said it improved how they get medical and health information, 48% said the
online advice had improved the way they take care of themselves, and 47% who had looked
for health information for themselves during their last Internet search indicated the
information affected their decisions about care and treatments. However, in 2003 only 8%
had taken information to their physician; in 2004 amongst people with multiple sclerosis it
was a third. The effect of taking information to the physician on the physician-patient
relationship was likely to be positive as long as the physician had adequate communication
skills, and did not appear challenged by the patient bringing in information. Although, many
patients appear to see the Internet as an additional resource to support existing and valued
relationships with their doctors, the change from patient-clinician communication to the
‘triangulation’ of patient-Web-clinician can be expected to significantly impact dynamics of
the physician-patient relationship. A 2007 review identified potential advantages of Webacquired information include helping patients make informed health care choices, and more
6
efficient use of clinical time. On the other hand, factors such as misinformation due to highly
variable quality of Web information, possible exacerbation of socioeconomic health
disparities, and shifting of conventional notions of the physician-patient relationship present
their own set of challenges for the health care provider. Clinicians should therefore be
educated in use of the Web, should understand the opportunities open to patients, and be
able to incorporate this ‘triangulation’ into the consultation.
Do digital health services impact on the appropriateness of GP consultations and
the avoidance of inappropriate consultations?
There is weak evidence that GP visits may be deferred but we do not know if that was
appropriate.
There is weak evidence that GP visits may be deferred as a result of patients using the
Internet. In a 2009 US study, 1 in 10 said that had used information obtained via the internet
to refuse or discontinue treatment recommended by a doctor or dentist. In a 2001 UK study,
one in four said that Web information had resulted in a deferred visit or had actually replaced
a visit to the doctor. We are not aware of evidence from consultation rates to support
reductions in visits, nor do we know what impact avoided consultations or discontinued
treatments may have had on the patients’ health.
Computer patient interviewing (CPI), in which patients complete an online interview about
their presenting medical history, and a summary is available to patient and clinician for the
consultation, is used widely in US general practice but not in the UK. There is evidence
(mostly from secondary care) of the benefits of CPI in giving patients more time to think
about questions, allowing patients to more easily disclose information about embarrassing
topics, ensuring that lines of investigation are not forgotten. In the US, CPI is now widely
used in primary care. Giving patients the possibility of using an Internet based CPI before a
consultation with their GP potentially helps to ‘enable’ the patient, ensuring the consultation
is more focused on their needs and may result in patients being better informed and more
able to efficiently find information or support on the Internet. It offers the potential of reorganising care, with some consultations becoming ‘virtual’.
What is the uptake and effectiveness of Information Prescriptions?
We have not found information about the uptake of Information Prescriptions. A
Department of Health report showed that patients and professionals were positive about
information prescriptions.
The Department of Health commissioned evaluation of the Information Prescriptions pilot
project. The final report was published in August 2008. A high number of patients and
carers found the information useful and professionals involved in the pilots were positive
about information prescriptions. In qualitative interviews with pilots, there was strong
support for the concept and potential of information prescriptions, with most of the pilots
continuing to implement information prescriptions beyond the pilot funding allocation.
7
Long term conditions
Do people with LTCs use digital health services to achieve better health outcomes
than those who do not?
Given the way the question is worded (patients self-selecting), yes, those who use digital
health services have achieved better health outcomes. Even, those allocated to a digital
health intervention have achieved better health outcomes than controls in childhood
asthma and diabetes.
Do digital health services help users with LTCs better manage their care?
Yes.
Cancer: A 2007 review of 9 RCTs found that IT based methods had improved patient
knowledge (effect sizes ranging from 0.12 to 1.03) and that satisfaction was improved
(although the effect was more equivocal).
Chronic conditions for children (particularly asthma): A 2009 systematic review identified 27
relevant articles, 96% of which reported improved outcomes. Outcomes included reduced
number of emergency room visits, improved knowledge on disease management, and
enhanced satisfaction. In particular, eight asthma studies included in the review have
individually shown improvement in quality of life, reduction in emergency department and
hospital visits, infrequent rescue therapy and a high level of satisfaction with home
telemonitoring, improvement in knowledge and asthma self management. However, studies
that have looked at health outcomes have had more mixed results some showing no
difference on asthma symptoms such as coughing, wheezing, shortness of breath, night time
awakenings, lung-function measures, but some showing decrease in asthma symptom days,
decrease in number of emergency department visits, significantly lower dose of inhaled
corticosteroids, decrease in school days missed, and in days of activity limitation.
Adult asthma: A recent RCT has shown Internet-based self-management resulted in
improvements in asthma control and lung function but did not reduce exacerbations, and
improvement in asthma-related quality of life was slightly less than clinically significant.
Chronic Obstructive Pulmonary Disease (COPD): Although a recent RCT reported numerous
technical challenges, a low recruitment and high drop-out rate, amongst those who remained
in the trial there were positive outcomes.
Diabetes: Diabetes (as well as renal disease) has been at the forefront of digital health
services for many years, not just offering ‘information’ on the web but complete
comprehensive services. For example showing: (i) benefit of patient access to their records
with better control of their fasting blood glucose, HbA1c and total cholesterol levels than
those in the control group; (ii) benefit of a shared electronic medical record with improved
HbA1c levels (by 0.7%) compared with usual-care patients; (iii) benefit of the use of webbased glucose uploads with significant improvements in HbA1c for type 2 diabetics. A review
of 26 studies to the end of 2003 showed overall positive results and found that IT-based
interventions improved health care utilization, behaviours, attitudes, knowledge, and skills.
Stroke: There is evidence from non-computer based interventions that information
interventions can improve patient and carer knowledge, patient satisfaction and depression.
We only identified two e-health studies of which only one had a positive result for the
computer intervention. Stroke would appear to be an area where evaluation of digital health
services that build upon evidence from successful non-computer-based interventions would
be worthwhile.
Mental Health: Mental health is one of the most advanced areas both in terms of the
development, availability, and evaluation of digital health services. A meta-analysis of
WebCBT showed moderate to large effect size for anxiety and depression. Two webCBT
systems are available for depression via the Internet and widely used. As well as the CBT
approach, the Internet is successfully being used for professional-patient approaches. A
review of Internet-based psychotherapeutic interventions and found a medium effect size
8
similar to the average effect size of traditional, face-to-face therapy, and a recent RCT of
therapist-delivered internet psychotherapy for depression in primary care concluded that
personally delivered CBT seems to be effective when delivered online in real time by a
therapist, with benefits maintained over 8 months.
Computer-mediated support groups (CMSG): CMSGs are distinct from the informal self-help
groups found on sites such Yahoo in that formal CMSGs have both educational and group
communication components, closed membership enrolment, fixed duration, and may include
some form of expert leadership. A review of 28 studies covering 12 conditions concluded
that participating in a CMSG led to increased social support, decreased depression,
increased quality of life, and increased self-efficacy to manage one's health condition.
Changes in health outcomes were moderated by group size, the nature of the
communication channels available, and the duration of the CMSG intervention.
Preventative services
What effect do digital health services have on take up of screening services?
What effect do digital health services have on take up of vaccine services?
Although there is evidence that decision aids can help people reach informed decisions
about screening, these are not necessarily Internet based. We found no evidence of the
effect of digital health services on take up of screening or vaccine services.
The limited literature available on the impact of digital health services on screening and
vaccine services is mostly related to decision aids, i.e. where the aim is to reduce anxiety in
making the decision or decisional conflict, rather than the aim of increasing uptake of
screening. A systematic review of cancer-related decision aids concluded that they are
effective in increasing patient knowledge (effect size 0.5) compared with usual practice
without increasing anxiety particularly in the area of cancer screening. Another systematic
review of prostate cancer screening decisions aids concluded that these decision aids
enhance patient knowledge, decrease decisional conflict, and promote greater involvement
in decision making but that the absence of outcome measures that reflect all elements of
informed decision making limits the field. Such decision aids can be web-based but may also
be delivered using other media, e.g. print, DVD.
The use of digital health services to try to increase screening or uptake of vaccinations
suffers from the problem that only those who are sufficiently interested will see the
information, i.e.participants have to be seeking information. The groups most requiring
information and help in making decisions about screening or preventative behaviour are
often either not interested or do not have access to the Internet. For example, an Australian
study found that not many women used the Internet to obtain information about breast
screening. Although, 80% (of 415) women accessed the Internet only 7% of the total women
accessed breast health and screening information. In the USA, Kreuter et al (2006) had used
community placed kiosks aiming to bring information about breast cancer screening to
populations with limited access to the Internet. Their kiosks were well used but, as far as we
are aware the kiosks have not continued.
Patient experiences of screening are important in getting others to accept screening, and
innovative ways of ‘information push’ rather than ‘information pull’ are needed probably
through links to mass media campaigns.
9
Similar services to NHS Choices
What similar services similar to NHS Choices exist and have they been evaluated?
Other sites with some similarities to NHS Choices include Health Canada and
Passeportsante in Canada, Healthfinder in USA, Healthinsite in Australia, everybody in
New Zealand, and Sanidad in Spain. The only studies we identified were some rather
preliminary studies of the impact of Passeportsante on empowerment.
Methodologies
What methodologies are used in assessing the impact of digital health services and
what are their strengths and weaknesses?
In primary research, the RCT, as the ‘gold standard’ method, can be used to assess the
impact of digital health services. However, there are problems in recruitment and
identification of the target population, choice of control group, possible inability to stop
access in control groups, attrition and how this is dealt with in the analysis, and problems
in choice of outcome measures.
In secondary research, there are problems in locating publications, publication bias, and
‘shelf life’ of an intervention.
RCTs are use in assessing the impact of digital health services but carrying these out in ehealth or digital health services is difficult. The inclusion and exclusion criteria, recruitment
method, and choice of placebo or control group in an e-health trial are important, often
subject to difficulties, and may be a limitation in the interpretation of results of any study.
For example, there will be self-selection of those who agree to take part. The use of intention
to treat analysis is supposed to cope with these problems but many studies, do not deal with
this well. Recruitment and inclusion and exclusion criteria are particularly relevant in
assessment of computerised methods where those who do not have access to the Internet
or those who are less IT literate may be excluded from trials. Choice of control group or
placebo is also difficult.
Comparison of an e-health intervention (additional to normal care) with no intervention
(additional to normal care) is likely to have a positive result. Head-to-head comparisons of ehealth interventions or studies which explore costs and benefits of different approaches are
needed. The marginal cost of an extra participant in a web-based intervention for the health
service is virtually zero. Analysis of trials therefore should include modelling of the impact on
routine delivery. Although intention to treat analysis is the correct approach for a given
population, studies which identify who might benefit and consider implementation for that
sub-population are worthwhile.
Finally, studies of the impact of digital health services are becoming increasingly available
but they are published in a great variety of journals. The need to monitor such evidence is
recognised by other governments such as the US; international collaboration to maintain a
body of evidence on these (often global) services is recommended.
10
1. Introduction and method
In June 2009, Bob Gann, Head of Strategy & Engagement for NHS Choices, invited tenders
for consultancy services to perform a rapid literature review of benefits and patient outcomes
of digital health services (Appendix 1). The University of Plymouth submitted a bid by the
closing date of 10th July in collaboration with the Patient Information Forum (PIF) and was
awarded the contract. We set up a discussion forum on a University of Plymouth website
and invited current PIF members and others around the world to be ‘members’ of the
Benefits of Digital Health Services Review Project. The core team from University of
Plymouth (RJ + LG) also posted on the website until mid-September. Since then we have
been reviewing and synthesising the literature and continuing to identify further literature.
Many bibliographic databases have been used as well as emails to authors and Google
Scholar. References have been managed using Endnote.
2. Other reviews
Before trying to synthesise the evidence from systematic reviews, RCTs and other studies it
is worth noting other narrative or more general reviews of use of the Internet in health care.
Nguyen (2004)1 says ‘Despite the proliferation of many online health portals and
communities and anecdotal reports of benefits, formal evaluations of their impact on
participants’ health outcomes, level of resource utilization, and user satisfaction have lagged
far behind’ and cites a 1999 report by Gustafson et al2. Nguyen et al identified 333 citations
up to 2002 using terms including computer-mediated communication, internet, online, web,
world wide web, email, electronic mail, discussion groups, patient education, health
education, chronic illness, self-help, and support groups. Studies were limited to those that
were classified as evaluation, outcomes, intervention, control, comparative, or pilot studies.
From these they selected 17 articles dividing them into two groups, 7 that provided computer
equipment to subjects, and 10 that included only subjects who had ready access to the
Internet. They noted two of the key researchers in the 1990s namely:
 David Gustafson and colleagues who produced a series of studies on variants of CHESS
(Comprehensive Health Enhancement Support System ) and its impact3-7.
 Patricia Flatley Brennan and colleagues who worked on home care after coronary artery
bypass graft surgery and the Computer-link project8-14.
Car et al15 carried out a literature review on the impact of eHealth on the quality and safety of
healthcare, and although they specifically excluded consumer health informatics (patientoriented eHealth), they reviewed computer-patient interviewing which is relevant. As argued
in a review of kiosk use for NHS Choices (since published as a paper at 16), there is synergy
between the development of computer-patient interviewing (where patients access either a
kiosk in the practice or the Internet at home, before a consultation to complete a branching
interview that gather’s information about their presenting history) and other e-health
applications for patients. This of course is recognised by the NHS with its development of
Healthspace and attempts to use that to start to tailor health information for patients.
Therefore it is worth citing the findings of Car et al with respect to computer-patient
interviewing. They say:
• Computer history taking systems can be used in a variety of clinical settings and
have, when eliciting data directly from patients, proven particularly useful in
11
identifying potentially sensitive information such as alcohol consumption, sexual
health and psychiatric illnesses, e.g. suicidal thoughts.
• Computer-based questionnaires are particularly useful for gathering important
background data prior to the consultation, which can then allow more time for
focusing on key aspects of the health problems in the actual consultation. These
systems can also save money by reducing administrative costs.
• Speech software and speech completed response computer history taking
systems allow adaptability for those with particular needs such as non-English
speaking patients, patients with hearing impediments and those who are illiterate.
• There is moderate evidence that data collected electronically tend to be more
accurate and contain fewer errors than data captured manually with traditional pen
and paper techniques; such data are also more legible.
• The current generation of computers is however not adept at detecting nonverbal behaviour; these systems should therefore be seen as not a substitute but
rather an adjunct to the clinical history.
• There have as yet been no comparative studies that have formally assessed the
effectiveness and cost-effectiveness of different computer history taking systems.
• It is important for NHS CFH to carefully consider the considerable potential
efficiency gains to be made from incorporating computer history taking systems
functionality—particularly if this involves direct entry of data by patients—into
future iterations of the NHS CRS. HealthSpace could facilitate this as could a
number of other modalities such as touch-screen or voice-recognition equipped
computers available in waiting rooms. This will, however, need to be introduced
within a clear evaluative context.
Goldzweig et al carried out a review of ‘Health Information Technology’ but we did not
locate a copy17
3. Extent of RCTs and systematic reviews on digital health services
Despite the methodological problems discussed in section 7, using the traditional approach
of hierarchy of evidence18-20 (systematic review, individual RCTs, case-control and cohort
studies, qualitative studies and case reports) is the ‘best that we have’ to assess evidence
and so is the general approach we have taken. There have been a number of RCTs of
different e-health interventions and indeed in some areas a substantial number of systematic
reviews. To give an idea of the scope of work carried out these are listed by the subject
headings requested by NHS Choices.
To set the scene, it is worth thinking of the differences between some of these e-health
services. Some are fulfilling the needs of health consumers who seek information either to
help them make a choice or to give peace of mind about their condition. Others are
promoting healthy lifestyles or behaviour that leads to more successful treatment. In this
latter case, people using a digital service have, by definition, already engaged with the idea
of changing behaviour, but maintaining their motivation is important21.
3.1 Accessing Care and Patient Choice.
Patient choice (clinical): The most rigorous evidence on patient choice comes from the
systematic reviews carried out by Annette O’Connor et al in 199922, updated in 200923, on
decision aids for people facing health treatment or screening decisions. Decision aids, such
as pamphlets and videos that describe options, are designed to help people understand the
options, consider the personal importance of possible benefits and harms, and participate in
12
decision making. They are used when there is more than one medically reasonable option no option has a clear advantage in terms of health outcomes, each has benefits and harms
that people value differently. In total the 2009 review included 55 RCTs. Ten examples of the
studies are shown in Table 1.
Author, year
Auvinen 2004
Comparison made in study
Pamphlet vs usual care
Target audience
Men newly diagnosed with prostate cancer
Barry 1997
Interactive video disc vs general information
Patients considering benign prostatic
hyperplasia treatment
Bekker 2004
Decision analysis + routine consultation vs
routine consultation
Bernstein 1998
Video vs usual care
Clancy 1988
Deschamps
2004
Deyo 2000
Pamphlet + personal decision analysis vs
usual care
Written + audiotaped consultation vs
pamphlets
Audiotape booklet vs pharmacist
consultation
Interactive videodisc vs pamphlet
Dodin 2001
Audiotape booklet vs pamphlet
Dunn 1998
Video + pamphlet vs pamphlet
Pregnant women who have received a
maternal serum screening positive test
result for Down syndrome
Patients with coronary artery disease
considering revascularization surgery
Health physicians considering Hep B
vaccine
Men with prostate cancer considering
treatment
Women considering hormone replacement
therapy
Adults with herniated disc or spinal
stenosis considering back surgery
Women considering hormone replacement
therapy
Parents of infants considering polio vaccine
schedules
Davison 1997
Table 1. Ten examples of studies included in the O’Connor Cochrane Review of patient
decision aids.
Although not specified as being web-based, almost by default the decision aids reviewed
were, or could be delivered as a 'digital health service'. O'Connor et al concluded that
decision aids performed better than usual care interventions in psychological effects, i.e. in
terms of:
a) greater knowledge
b) lower decisional conflict related to feeling uninformed
c) lower decisional conflict related to feeling unclear about personal values,
d) reduced proportion of people who were passive in decision making
e) reduced proportion of people who remained undecided post-intervention.
It also resulted in different actions in some situations, e.g. reduced rates of elective invasive
surgery in favour of conservative options, and reduced PSA screening for prostate cancer.
Studies of this sort (patient decision aids) typically recruit people and allocate to either the
new intervention or some placebo or ‘treatment as usual’. However, where participants are
targeted from a defined population to be recruited to a study the results may be more
positive that is achieved in routine practice because (a) the researchers ‘find’ the participants
and raise awareness of the study, (b) participants in such studies are perhaps less likely to
drop out of the intervention than they are in routine practice. So, for example, to know more
about the effectiveness of web-based decision support we would also need to know that
people were aware or were referred to the relevant website and were able to find them.
Morris et al (2008) looked at five medical conditions and found that relatively few patient
decision aids were identified using a number of different Google searches24. This assumes
that patients are even aware that such things exist.
13
Making decision aids more available online, or at least via computer, may help overcome the
major problems of literacy and health literacy. A method for designing interactive soap
operas to explain patient decision aids has been described25.
Patient empowerment and the doctor-patient relationship: The literature on ‘patient choice’,
of course, overlaps with that of the ‘doctor-patient relationship’ (e.g.26-35), ‘patient
empowerment’ (e.g.36-43) and ‘coaching’ (e.g.44-46). Use of the Internet clearly impacts on
empowerment and the doctor-patient relationship. For example, Murray et al found that in
the USA in 200326 8% had taken information from the Internet to their physician. The impact
was likely to be positive as long as the physician had adequate communication skills, and
did not appear challenged by the patient bringing in information31 (see section 3.4 below on
primary care).
Empowering patients with information means giving them the option not to consult, or not to
follow the doctor’s advice. Sometimes this may be misguided because the information they
have is of poor quality or they have misinterpreted it. At other times it means they have been
truly empowered to make a choice based on their own values rather than those assumed for
them by their doctor. For example, Weaver et al (2009)47 invited people from a population
panel from Seattle. The 17.8% response in 48 hours comprised 562 individuals aged 19-90
who completed an online questionnaire. Eleven percent had used information obtained via
the internet to refuse or discontinue treatment recommended by a doctor or dentist.
In the UK, Nicholas (2001)48 et al carried out an online survey of visitors to SurgeryDoor
http://www.surgerydoor.co.uk/ in November 2000. In total, 1,068 users answered the
questionnaire, 5% of the 20,611 unique IP addresses that were recorded as visiting the site
that month. Two-thirds said that the information found had 'helped a lot' in being better
informed. Just under half felt that the information they found had helped in their dealings with
the doctor, while just over half felt that information found had changed the way they felt
about their condition. Over one-third of respondents said that their condition had improved
after having visited the site and more than one in four said that Web information had resulted
in a deferred visit or had actually replaced a visit to the doctor. These types of surveys
(Weaver et al, Nicolas et al) of course suffer from the usual limitations of online feedback –
the self selection of those people who chose to answer the questionnaire and the lack of any
comparison group. For example, would the users of SurgeryDoor perhaps have deferred a
visit if they had been given ‘placebo’ information?
Patient choice (provider): one of the main strands of NHS policy over the last few years has
been in the publication of information about providers (a) to allow patient choice of provider,
and (b) with the aim of improving quality of care.
In the USA Fung et al49 carried out a systematic review to synthesize the evidence for using
publicly reported performance data to improve quality. Forty-five articles published since
1986 (27 of which were published since 1999) evaluated the impact of public reporting on
quality in the USA. For these US studies they found a modest association between public
reporting and ‘health plan’ selection, at hospital level stimulation of quality improvement
activity, inconsistent association between public reporting and selection of hospitals and
individual providers, and inconsistent association between public reporting and improved
effectiveness. They concluded, despite 45 published papers, that evidence was scant,
particularly about individual providers and practices and rigorous evaluation of many major
US public reporting systems is lacking. Another recent US review50 (2009) included 14
studies, examining quality information and its impact on the consumer's choice of health
plans. We only obtained the abstract for this study and found it difficult to understand. It
seems they did not review the impact on decision making.
14
Finally, for the US experience, the HSC 2007 Health Tracking Household Survey which is a
US nationally representative survey of 18,000 people found that, despite the longer history of
producing such data in the US than UK, when selecting new primary care physicians, half
relied on word-of-mouth recommendations from friends and relatives, but many also used
doctor recommendations (38%) and health plan information (35%), and nearly two in five
used multiple information sources when choosing a primary care physician. When choosing
specialists for medical procedures, most people relied exclusively on doctors’ referrals.
People using online provider information were in the minority, ranging from 3% for those
undergoing procedures to 7% for those choosing new specialists to 11% for people choosing
new primary care physicians51.
One of the problems in using performance data, and probably one of the reasons why it has
so far had little impact is that it may require a higher level of literacy and numeracy than is
widespread. As Hibbard and Peters have described we are only able to ‘process’ a limited
number of variables in any one choice and that more information does not always improve
decision making52-55. Furthermore, many patients find information about other people like
them56 easier to understand and use. Levels of awareness and how people are invited to
use comparative information are also important57.
NHS Choices provide services aiming to enable choice, for example, ‘Find nearby services’
allowing users to find (e.g.) nearby GP practices with details of opening hours etc. Is there
any evidence that directories of this sort give users more choice? Most evidence comes from
the Department of Health itself. Dixon (2009)58 reported for the Department of Health on the
fifteenth National Patient Choice Survey in September 2008, the first was in February 2006.
In the 2008 survey carried out by Ipsos Mori , patients who had been referred by a GP for a
first outpatient appointment in any of 142 major acute NHS trusts or 16 Independent Sector
(IS) organisations during the two-week period 15 to 28 September 2008 were invited to take
part in the survey. Around 248,000 questionnaires were issued. Patients were invited to
complete the questionnaire and return it using the pre-paid envelope to Ipsos MORI. They
got 93,003 (38%) valid responses. Key findings from the study include (i) 46% of patients
recalling being offered a choice of hospital for their first outpatient appointment, (ii) 49% of
patients who were offered choice said they used the GP as a source of information to
choose their hospital, 34% used their own experience or that of friends and family; a booklet
about choice was used by 8% and 5% used the NHS Choices website. This report will be
well known to NHS Choices so will not be described further.
The GP patient survey for 2008-2009 was published in June 200959 but although it
addressed many aspects of care in general practice (on the whole finding that patients are
satisfied with their care), use of the Internet and how this related to patient care and patient
choice was not part of the survey. (Interestingly, the word Internet is not used once in the
350 page report). Nevertheless, these annual surveys by Mori for the Department of Health
provide a useful baseline and context for future study of the impact of digital health services.
It would seem worthwhile for NHS Choices to discuss the inclusion of relevant questions in
future surveys.
A small interview study by Henderson, Robertson, and Dixon(2009)60 of 18 patients
attending outpatient clinics in England found that none had consulted the Internet and only
half felt they had been offered a choice of provider. However, five of those not offered a
choice said that they would not have wanted one. Robertson and Dixon have very recently
(4/11/09) published a Kings Fund report61 on patient choice at the point of referral. A postal
questionnaire was sent to 5,997 NHS patients who booked their first outpatient appointment
in January 2009 at eight NHS trusts and two independent sector treatment centres (ISTCs)
15
across four case study sites in England. The response rate was 36%. Half said they were
offered a choice of hospital; most of these were offered a choice by their GP (60 per cent), in
a letter outlining the options (21 per cent), or by a telephone-booking adviser (20 per cent).
Patients drew on various information sources to help them choose, including their own past
experience (41 per cent), and advice from their GP (36 per cent) and from friends and family
members (18 per cent) but only 4% had looked at the NHS Choices website and 1% other
websites. Of patients who were offered a choice, 60 per cent were satisfied with the amount
of information they were given, 22 per cent did not want any information and 14 per cent
would have liked more. Cleanliness, quality of care, and the standard of facilities were the
three most important factors that patients said had influenced their choice of hospital. It
would appear that NHS Choices so far has little impact on choice of referral.
There are of course other ratings of NHS services such as www.drfosterhealth.co.uk ,
www.patientopinion.org.uk and www.iwantgreatcare.org. Magee et al62 undertook a focus
group study (6 groups with a total of 50 participants) in six different locations in England,
published in 2003. Participants felt that independent monitoring of healthcare performance
was necessary, but they were ambivalent about the value of performance indicators and
hospital rankings. They tended to distrust government information and felt that 'Dr Foster'
gave more detailed locally relevant information. They concluded that if public access to
performance information is to succeed in informing referral decisions and raising quality
standards, the public and general practitioners will need education on how to interpret and
use the data.
Finally, it is also worth noting that a recent review63 found that choosing between hospitals or
primary care providers is not currently a high priority for the public, except where local
services are poor, e.g. they have long waiting times and where individual patients'
circumstances do not limit their ability to travel. They also found that better educated
populations make greater use of information and are more likely to exercise choice in health
care, and finally that there was little evidence in the literature that providing greater choice
will in itself improve efficiency or quality of care.
Is scorecard functionality effective at influencing a patient’s choice of providers?
We have found not much literature addressing the use of computer based scorecards by
patients. One (US 2008) publication64 described a case study in which use of the balanced
scorecard (BSC) framework by a financially struggling US hospital did not succeed in using
patient satisfaction monitoring to change culture and achieve financial stability.
How effective are digital health services at providing an end to end service, e.g. all the
way through to accessing Choose & Book?
There is evidence about the ‘end’ of this process, i.e. Choose and Book, although it is from
the GP’s rather than the patient’s point of view. The BMA65 published in January 2009 a
review of local variation in the experience of using Choose and Book. They had received
both praise and extreme criticism of the system so carried out a study involving interviews
with clinicians and staff at the PCT and Trust in one locality. The majority of those
interviewed felt that Choose and Book offers potential and the paper based system it has
replaced is far from ideal. Potential benefits identified by these professionals include “the
ability to provide patients with an appointment on the spot, control over their appointment
and an indication of waiting times. Other benefits include the ability to track referrals on the
system and allow the confidential exchange of information between clinicians. The Directory
of Services also offers the potential for consultants to define their clinics and can offer GPs
the opportunity to be confident that they are referring to the correct clinic.” The benefit to
patients is touched upon in some of the interviews, for example, (p14) “The GPs liked the
16
patients to leave the practice with the assurance that the booking had been made and
patients were positive about leaving with an appointment.”
Are digital health services helping patients achieve care in their preferred setting
(home/self care vs visiting a GP)?
Clearly, there is a large body of work on telehealth, telecare, telemedicine etc in which
digital health services help patients to be treated either in the home or at local health
services rather than more remote specialist care66, but NHS Choices had said that this did
not need to be included in this report. We are not aware of research exploring how
‘information’ from digital health services have helped patients achieve care in their preferred
setting, other than the weak findings on deferred GP visits reported above by Weaver 47 and
Nicholas48.
3.2 Healthy Living
This is the area where there is probably the most evidence for the impact of digital health
services, not only randomised trials but a number of reviews, systematic reviews, and metaanalyses. The evidence from primary studies has been steadily growing over the last
decade.
For example, in 2002 Bessel et al carried out a systematic review of ‘Internet interventions
for consumers’ which only identified 10 papers (4 deemed to be poor quality) including
smoking cessation, interventions on eating habits, body image and physical activity in young
women and psychosocial benefits prior to cardiac surgery. They concluded ‘All studies
showed some positive effect on health outcome, although the methodological quality of
many studies was poor’67.
However six years later in 2008, Portnoy et al68 conducted a meta-analysis of 75 studies that
investigated the efficacy of computer-delivered interventions (CDIs) in a range of contexts
(nutrition, weight loss, physical activity, sexual health, tobacco use, substance use, bingeing
and general health maintenance). All included health information (eg diet, weight, alcohol),
88% included a motivational component, 89% included a skills training component.
Examples of the interventions are shown in Table 2.
First author, year
Intervention
Kypri (2004)
10-15 minutes of web-based assessment and personalized feedback on their drinking.
Swartz (2006)
Video based internet site that presented current strategies for smoking cessation and
motivational materials tailored to the user’s race/ethnicity, sex, and age.
Structured 8-week intervention delivered through the Internet. Included explanation
and definition of eating disorders. The interactive software featured text, audio and
video components, on-line self-monitoring journals, and behavior change exercises.
There was also a linked discussion forum.
Winzelberg (2000)
Table 2. Examples of studies of computer-delivered interventions reviewed by Portnoy
(2008).
Portnoy et el concluded that CDIs can lead to “immediate post-intervention improvements in
health-related knowledge, attitudes, and intentions as well as modifying health behaviours
such as dietary intake, tobacco use, substance use, safer sexual behavior, binge/purging
behaviors, and general health maintenance. CDIs do not provide benefits in all contexts; the
17
evidence does not support the use of CDIs to improve physical activity, weight loss, or
diabetes self-management”.
Adding to this view that the number of primary studies has increased considerably in the last
ten years, are other reviews including:




Weinstein (2006) reviewed 3 web-based weight loss programs69 (results equivocal).
A complete issue from JMIR in 2008 on web-assisted tobacco interventions70
containing 13 studies71-83. Norman et al in their editorial conclude that the webassisted tobacco interventions described illustrate “how new technologies can
support health promotion and population health overall, empowering change and
ushering in a new era of public eHealth”.
Lustria et al (2009)84 reviewed and analysed the key components of computertailored health interventions delivered over the web based on 30 studies. The 30
studies that used tailoring focussed on diet (n = 10), physical activity (n = 7),
alcoholism (n = 3), smoking cessation (n = 7) and one study each for encopresis,
eating disorders, and general risk behaviors. However, their study did not focus on
outcomes, rather on how computers were used to tailor responses.
Wantland et al (2004)85 reviewed 22 studies published from 1996-2003 on webbased behavioural change compared to non web-based interventions. The study
areas were: HIV AIDS (5), Weight control (2), Asthma (2), Eating Disorders (2), and
PSD, headache, heart disease, MOS, physical activity, paediatric encopresis,
tinnitus, and nutrition. They concluded “The effect size comparisons in the use of
Web-based interventions compared to non-Web-based interventions showed an
improvement in outcomes for individuals using Web-based interventions to achieve
the specified knowledge and/or behaviour change for the studied outcome variables.
These outcomes included increased exercise time, increased knowledge of
nutritional status, increased knowledge of asthma treatment, increased participation
in healthcare, slower health decline, improved body shape perception, and 18-month
weight loss maintenance.”
The more recent Portnoy review seems to contradict the Wantland review so we have
explored this further. Portnoy does reference Wantland in his introduction but interestingly
does not return to discuss the apparent differences in his discussion. Table 3 summarises
the two studies.
Wantland et al (2004)
Meta-analysis(studies 1996-2003)
Included only studies comparing web-based
interventions and non-web-based interventions
Portnoy et al (2008)
Meta-analysis (studies 1988-2007)
Studies assessing web-based interventions
compared with controls (which may include ‘do
nothing’)
75 studies
Similar, related to health domains in “Healthy People
2010) (US Dept of Health publication)
N=35,685
Retention rate 86%
22 studies
Outcomes – knowledge and behaviour change
N=11,754
Dropout rate 21%
Table 3. Summaries of the Wantland and Portnoy systematic reviews of web-based health
living interventions.
We think the differences have arisen because Portnoy is more rigorous (perhaps verging on
the side being too critical) while Wantland’s review is not as rigorous as might be expected.
In fact, although the topic areas for their reviews are the same, there are only 5 papers
which appear in both meta-analyses.
18
For example, both include the paper by Harvey-Berino86 on weight-loss maintenance.
Wantland refers to improved outcomes in 18 month weight loss maintenance based on
rather weak evidence from this paper, whereas Portnoy points out that there is no evidence
of the validity or reliability of the assessment instruments used.
Both reviewers include a paper by Marshall et al 2003 ‘Print Versus Website Physical
Activity Programs: A Randomized Trial’ 87, yet Portnoy states that the evidence does not
support the use of CDIs to improve physical activity. The paper that Wantland based his
evidence on states that there is no significant difference between outcomes of web-based or
print interventions. Marshall gives no evidence of the validity or reliability of the instruments,
so, Portnoy’s conclusions may be more valid.
Both reviewers also include Southard et al 2003 ‘Clinical trial of an internet based case
management system for secondary prevention of heart disease.’ But we have not been able
to extract how their different interpretations affected their conclusion.
.
Both reviews had the problems that the studies they identified were very heterogeneous both
in their use of controls, their interventions and their quality. Both authors admit that their
reviews are limited by these factors. Given this variability there is scope for variability in
interpretation, Portnoy has perhaps been more rigorous and erred on the side of caution
compared to Wantland.
For the remainder of this section we have divided the evidence into disease or topic areas.
As mentioned above, evidence ranges from primary research, including RCTs or quasiexperiments, through qualitative studies, to systematic reviews of the literature.
Smoking cessation: Although there is a body of evidence concerning internet or web-based
smoking cessation interventions, much of it conducted online, there are several associated
methodological problems. Firstly, outcomes such as smoking status are self-reported and
therefore unvalidated. Secondly, as outlined by Eysenbach88 and seen above, attrition in
internet based research is high. Eysenbach also describes the problem of ‘controlling’ the
participants’ behaviour – it is not only difficult to ensure that the intervention is being used,
but also to eliminate use of any other similar interventions to be found on the internet.
Despite these methodological issues, however, some useful evidence has been identified.

Walters et al89 reviewed 19 papers published between 1995 and 2004 which studied the
outcomes of automated smoking cessation interventions in terms of smoking behaviour.
This review contained mixed results. However, in studies with adults, almost half of the
studies reported significant reduction in smoking relative to a comparison group. Two of
the 4 studies involving adolescents reported a significant reduction in smoking initiation
and prevalence as a result of computer-tailored material. The majority of the studies
utilised what is known as ‘first generation’ interventions in which participants were mailed
computer-generated feedback reports.

In a randomized control study, Swartz et al (2006)90 found that their ‘treatment group’,
using a tailored motivational intervention, was significantly more likely to be abstinent
after 90 days than a control group (no intervention). The authors conclude that a
smoking cessation intervention can be delivered via the internet with some degree of
short-term efficacy. Attrition was high and smoking status was not substantiated but the
authors did use a conservative intention to treat analysis assuming that those who were
lost to follow up were still smoking. The cessation rate among all treatment group
subjects (n = 171) was 12.3% (n = 21) and among control condition subjects (n = 180)
was 5.0% (n = 9).
19

In 2008, the Journal of Medical Internet Research published a special theme issue on
web-assisted tobacco interventions. Five of these are shown in Table 4.
Publication
Methods/findings
Conclusions and comments
Bock et al
(2008)73
Review. Compared with a similar review in
2004, web-assisted tobacco interventions
(WATIs) were of higher quality. More sites
included at least one interactive feature (39%
increased to 56%) but there was underutilisation of interactive capabilities to
personalise treatment.
The authors stress the need for further
research to identify the optimal level of
information and interactivity for these
interventions to be successful and sustain
population-based health behaviour
change.
Rabius et al
(2008)81
Compared five different WATIs in a six-arm
RCT. Tailored, interactive websites may help
smokers to quit and maintain cessation; 1013% of those enrolled had quit at 13 months.
Smokers were suffering from depression, the
extra effort needed in using an interactive tool
produced poorer results.
Although a large study (n=6451), only
38% provided smoking status information
thirteen months after randomisation.
Cunningham
(2008)75
Telephone survey of 8467 adult respondents,
18 years and older, in Ontario, Canada 200607. Smokers were marginally less likely to be
internet-users, but overall, 40% of smokers
were interested in a web-assisted tobacco
intervention.
Strecher et al
(2008)83
Randomized trial of 1866 smokers to examine
the efficacy of 5 different treatment
components of a Web-based smoking
cessation intervention.
Level of engagement influences the
impact of a web-assisted tobacco
intervention, and this is influenced by the
source of the message, the degree of
tailoring and the timing of the exposure to
the intervention.
Zbikowski et
al (2008)91
‘Real world evaluation’: tracked program
utilization (calls completed, Web log-ins), quit
status, satisfaction, and demographics of
11,143 participants who enrolled in the Free &
Clear Quit For Life Program between May
2006 and October 2007
phone interventions were used more than
the web. Women used the phone/web
interventions significantly more than men.
However, 30-day quit rates at six months
were only 21% (intent to treat analysis). It
appeared that greater adherence to
web/phone interventions led to higher quit
rates.
Table 4. Five studies from JMIR special 2008 issue on web-based tobacco interventions
Reduction in alcohol use: Although there appear to be fewer published reports of primary
research or reviews for reduction in alcohol use than for smoking cessation, there is still a
substantial body of work. Although the similar problems of high attrition rates and selfreported data occur, there seems to be accumulating evidence of successful outcome for at
least some groups.

Bewick et al92 reported a systematic review in 2008 of the effectiveness of web-based
interventions designed to decrease alcohol consumption. This included papers up to
2006. The authors thought the quality of the papers was low and in their summary said
there was only one RCT. This was rather misleading as the body of the paper showed
that five studies93-97 were RCTs but that one94 had, what they called a ‘pure control’, i.e.
no comparison group (see section 7 on methodological problems).They concluded that
although web-based interventions were well received the evidence was inconclusive. It is
20
perhaps worth looking in a bit more detail the studies that Bewick et al reported and what
has happened to those ‘streams of work’ since (see below). As is often the case, their
systematic review was work they did in preparation for their own trial of a web-based
alcohol intervention98.

Riper et al’s pragmatic randomised trial in the Netherlands reported in 200799 involved
problem drinkers (as defined by the country’s safe drinking guidelines) recruited from the
general public. An interactive intervention based on cognitive behavioural and self
control principles included access to a peer-to-peer discussion forum and was compared
to the control group which simply had access to web-based psycho-educational material.
Despite the limitations of high attrition and self-reporting, the authors concluded that selfhelp interventions without therapeutic guidance can effectively reduce problem drinking
in self-referred adults. It was suggested that future research should identify which
groups would benefit from this type of intervention.

Linke, Murray et al have been conducting an ongoing randomised trial of a web-based
promotion of sensible drinking100 101. Down Your Drink is a web-based 6 week
intervention aimed at heavy drinkers100. The intervention is an on-line psychologically
enhanced interactive computer-based intervention and the control a flat, text-based
information web-site102. Linke et al (2007) analyzed records of 10000 users (intervention
group). The mean age was 37, 51% were female, 38% were single, and 42% lived with
children. Most were White British, lived in the UK, and reported occupations from the
higher socioeconomic strata. Only 17% of registrants completed the whole 6 weeks but
for those who did the final outcome measures, measures of dependency, alcohol-related
problems, and mental health symptoms were all reduced (compared to baseline not to
controls).

Bewick et al, who are based in Leeds, have developed (as reported above) and trialled
(2008) a web-based intervention in 506 University students98. Participants were
randomly allocated to a control (i.e. assessment only) or an intervention condition
(immediate personalised feedback and social norms information plus an additional
invitation to visit the intervention website at week 6). Interestingly, for a group who had
conducted a systematic review they do not specifically mention ‘intention to treat’
analysis or deal specifically with attrition from the two arms (calculated as 34% from
control and 41% in intervention despite higher levels of reward (printer credits) offered to
the intervention group) in their own primary research. They concluded, however, that
those in the intervention reported a larger decrease in alcohol than controls.

Riper et al (2009) reported a systematic review and meta-analysis of personalized
feedback interventions for problem drinking103. Six studies delivered the personalizedfeedback interventions by mail, and the other eight did so via the Internet. The metaanalysis showed that about eight people needed to be recipients of the intervention in
order to generate one good clinical outcome.
Some continuing systems in drink awareness or sensible drinking are shown in Table 5.
Website
Main authors and related papers
Country
www.downyourdrink.org.uk
Linke, Murray100 102 104 105
UK
www.mystudentbody.com
Chiauzzi95 106-109
USA
www.otago.ac.nz/studenthealth/thrive
Kypri93 94 110 111
New Zealand
21
www.minderdrinken.nl
Riper99 112 113
Netherlands
Table 5. Examples of drink awareness and sensible drinking websites with continuing use
and research.
Of course the Internet is not the only medium that can be used to encourage safe drinking
and perhaps more cross platform comparisons are needed? In a study with 181 participants,
Kramer et al114 found that the television supported self-help was more successful than the
waitlist group in achieving low-risk drinking post-intervention and at 3-month follow-up.
Weight loss, healthy diet and physical activity: There are large number of both primary
studies and systematic reviews, with a great deal of variability in both making synthesis and
a conclusion difficult. Table 6 shows systematic reviews; Table 7 shows recent primary
studies not included in the reviews.
Some reviews have only included RCTs or quasi-experiments, some with objective,
measurable outcomes115; others have included several interventions, such as physical
activity and dietary behaviour change116. It is worth noting the conclusions of the most
recent reviews by Neville and colleagues that “future studies should use objective outcome
measures, sampling approaches that increase external validity, and follow up long term”
(Neville, Milat & O’Hara, 2009)115.
Reviews
Weinstein
(2006)69
Eight studies evaluating weight loss via
internet programs
Results equivocal, although it was suggested
that the internet may be an alternative to
traditional face-to-face programs. They
questioned the long term efficacy of internet
weight loss programs.
Kroeze
(2006)117
Thirty evaluations of computer-tailored
interventions physical activity and dietary
behaviours, 1965 and 2004.
Evidence for effectiveness of computer-tailored
nutrition interventions was quite strong, but not
for physical activity
Norman
(2007)118
Forty nine publications defined as ‘second
generation’ interventions 2000-2005. There
were only five overlapping papers with
Kroeze.
Mixed findings. Comparing their review with
that of Kroeze et al, said “both reviews indicate
that more rigorous research is needed to
evaluate eHealth intervention technologies and
understand the program mechanisms that
promote physical activity and dietary behaviour
changes”
Vandelanotte
(2007)119
Web-delivered physical activity
interventions
A little over half of the trials had reported
positive behavioural outcomes. But effects
were short-lived and there was limited evidence
of maintenance of physical activity changes.
van den Berg
(2007) 120
Similar to Vandelanotte study, with many of
the same studies identified
Both van den Berg and Vandelanotte stated the
need for further research with increased
emphasis on the methodological problems
identified – type of outcome measure, sample
size, maintenance of any effects
Eyles
Meta-analysis of 15 trials and narrative
review for 5 trials. Long-term effectiveness
of tailored nutrition education for adults,
includes non-web based tailored
interventions such as print, email or
workbooks as well as internet and kiosks.
Tailored nutrition interventions are a promising
strategy over the longer term
(2009) 121
22
Marcus
(2009)122
Policy review, similar body of literature as
Vandelanotte and van den Berg
Proposes referral to internet based physical
activity interventions by primary care physicians
Neville
Weight reduction interventions, only six
studies were included in the final analysis.
Four of these had used objective weight
reduction measurements such as height,
weight and/or body fat, and/or waist
circumference (conducted in a clinic
setting); two others used self-reported
data.
Evidence for second generation computertailored interventions for dietary behaviour
change is fairly strong, and that weight
reduction interventions are ‘promising’. Further
research is needed (i) to establish whether the
effects can be sustained in the long term and
also whether they are generalisable, (ii) to
investigate different methods of tailoring, and
the relative success of different components of
the interventions
(2009)113
Neville
(2009)123
Neville
Dietary behaviour change interventions,
the main outcome measure had to be
dietary behaviour change and studies used
self-reported data.
Computer-tailored physical activity
interventions.
Despite stating that this review was different
from previous reviews such as those by Kroeze
and Norman, the conclusions were similar in
finding that the evidence is inconclusive. Again
questions are raised about the generalisability
and sustainability of any beneficial effects of
these interventions.
Eighteen studies (1995-2008) met their
inclusion criteria of: participants aged
≥18 years with a body mass index ≥25, at
least one study arm involving a web-based
intervention with the primary aim of weight
loss or maintenance, and reported weightrelated outcomes. Thirteen studies aimed
to achieve weight loss, and five focused on
weight maintenance
Concluded that although the four metaanalyses suggest meaningful weight change, it
is not possible to determine the effectiveness of
web-based interventions in achieving weight
loss or maintenance due to heterogeneity of
designs and thus the small number of
comparable studies. Higher usage of website
features may be associated with positive weight
change, but we do not know what features
improve this effect or reduce attrition.
(2009) 124
Neve
(2009)123
Table 6. Reviews of web-based weight loss interventions
23
Recent primary studies
Marcus (2007)124
RCT comparing a tailored internet-based
physical activity intervention, a tailored printbased intervention and a standard internetbased (not tailored) physical activity
intervention
No significant differences between three
arms of the study. Outcomes were not
only physical activity counts, but also
functional assessments such as
treadmill tests. Despite positive results,
authors concluded that internet-based
interventions may provide an opportunity
to reach more sedentary adults in a
more cost-effective way.
Hunter (2008)125
446 overweight military personnel (222M,
224F), mean age 34, mean BMI 29. Study
participation 2003-2005. Participants randomly
assigned to 6-month behavioral Internet
treatment (BIT, n=227) or usual care (n=224).
Change in body weight, BMI, percent body fat,
and waist circumference; presented as group
by time interactions, measured.
Van Wier (2009)126
1386 employees to an RCT comparing three
treatments: (i) intervention materials with
phone counselling (phone group); (ii) a webbased intervention with e-mail counselling
(internet group); and (iii) usual care, i.e.
lifestyle brochures (control group). The
interventions used lifestyle modification and
lasted a maximum of six months. They used
multiple imputation for missing values.
Completers who received BIT lost 1.3 kg
while those assigned to usual care
gained 0.6 kg. Results similar for the
intention-to-treat model. BIT participants
also had significant changes in BMI,
percent body fat, and waist
circumference.
Concluded: Internet-based weightmanagement interventions result in
small amounts of weight loss, prevent
weight gain, and have potential for
widespread dissemination as a
population health approach.
Body weight reduced 1.5 kg in the
phone group and 0.6 kg in the internet
group, compared with controls.
Concluded that lifestyle counselling by
phone and e-mail is effective for weight
management in overweight employees
and shows potential for use in the work
setting
Ferney et al 2009127
Brisbane, Australia. Compared a local
neighbourhood environment-focused physical
activity website with a motivational-information
website and their effects on walking and
overall physical activity. 106 participants,
conducted between August 2005 and February
2006
Significant increases in walking and total
physical activity were observed in both
groups but the local neighbourhoodenvironment focused physical activity
website seemed more effective at
engaging participants than the
motivational-information website.
Bennett (2009)128
12-week RCT to evaluate the short-term
efficacy of a web-based weight loss
intervention among 101 primary care patients
with obesity and hypertension. Patients had
access to a website and participated in 4 (2 inperson & 2 phone) counseling sessions with a
health coach
Intent-to-treat analysis showed greater
weight loss at 3 months (−2.56 kg; 95%
CI −3.60, −1.53) among intervention
participants (−2.28 ± 3.21 kg), relative to
usual care (0.28 ± 1.87 kg).
Table 7. Recent RCTs on weight loss or sensible eating.
We searched for the websites used in the above studies but it seems that few are currently
openly available. On the other hand there are numerous sites offering such support, many of
them with significant charges for the service. Table 8 shows a list of some of these openly
available sites and any evidence we have been able to find on their effectiveness
24
Some British sites
Boots
£1.55/week
No evaluation studies identified. We
tried to contact the site owners but
got no reply.
www.bootsdiets.com/
Tesco
£2.99/week
www.tescodiets.com
Slimming World
www.bodyoptimise.com
£4.65/week
Some US sites
Ediets
Womble (2004), RCT, 47 women
randomly assigned to eDiets.com
or weight loss manual. Based on
‘completers’ weight loss manual
was better, but on ITT using carry
forward, no significant difference.
www.ediets.com
Table 8 commercial weight loss sites
One UK publication129 for a study of a ‘commercial website’, despite having an extensive
press release (http://reporter.leeds.ac.uk/press_releases/current/weightloss.htm),seems not
to have completed and the website no longer exists.
Tsai et al (2005)130 systematically reviewed US commercial sites finding studies of
eDiets.com, Health Management Resources, Take Off Pounds Sensibly, OPTIFAST, and
Weight Watchers. They found that the programs were associated with high costs, high
attrition rates, and a high probability of regaining 50% or more of lost weight in 1 to 2 years.
They concluded that commercial interventions available over the Internet and organized selfhelp programs produced minimal weight loss.
Other studies of media or commercial interest in weight loss include a campaign by a North
Carolina newspaper, the Herald-Sun, that advertised its 15-week Lose to Win weight loss
challenge, both in its print version and on its Web site took place from January through May
2005. Carter-Edwards et al131 obtained the data after the campaign and analysed it and
published it with the misleading title of ‘An Internet-based weight loss intervention initiated by
a newspaper’, as the intervention was only ‘Internet-based’ in the sense that the newspaper
was available on paper on and the web. The intervention had been to publish motivational
articles. Given that the newspaper is described as the most-read newspaper in the Durham,
North Carolina, the results published in a journal were disappointing; of 705 people who
signed up only 154 (22%) participants remained to the end of the 15 week campaign. These
‘completers’ lost an average of 5.9 lb, but given the attrition and no comparison group, this is
inconclusive. Reference to the newspaper website now finds no obvious link to any weight
loss program, suggesting that (as is likely) media interest in weight loss only lasts as long as
there is a ‘story’.
Sexual health
Noar et al132 conducted a meta-analysis of intervention trials of computer-based
interventions designed to modify HIV-related sexual risk behaviour. The aims were to
establish whether these interventions changed sexual risk behaviour; to compare the mean
effect size with that of traditional face-to-face clinical encounters, and thirdly to identify what
factors moderated the efficacy of these interventions. Twelve studies met the inclusion
25
criteria. Their results indicated that computer-based interventions are as effective as
human-based interventions. It should be noted, however, that only three of these studies
involved internet-delivered interventions, and these showed poorer retention. The authors
concluded that computer technology based interventions have many advantages compared
to human-delivered interventions – lower cost and greater flexibility, for example. Their
comments are not restricted to internet delivery, however – and include such channels as
mail, internet or mobile phones.
A recent study (Bull et al, 2009133) tested a single Internet session promoting condom use
amongst use amongst 18-24 year olds, one sample recruited online and one in clinics.
Among sexually active youths in the Internet sample, persons exposed to the intervention
had very slight increases in condom norms. There were no intervention effects in the clinic
sample. They concluded that Internet-based interventions need to be more intensive to
see greater effects.
3.3 Screening Services
We have found mostly literature related to decision aids in screening, i.e. where the aim is to
reduce anxiety in making the decision or decisional conflict, rather than any studies where
the aim is to increase uptake of screening. Investigation of the literature on information for
genetic screening would be worthwhile but has not been included in this report.
Cancer screening aids (in general)
O’Brien et al (2009)134 identified 34 RCTs of decision aids in screening (n = 22 trials) or
preventive/treatment (n = 12 trials) context. Meta-analysis showed improved knowledge
about screening options when compared to usual practice (effect size, 0.50), general anxiety
was not increased in most trials and was significantly reduced in a screening context.
Decisional conflict was reduced overall. They concluded that “Cancer-related DAs are
effective in increasing patient knowledge compared with usual practice without increasing
anxiety particularly in the area of cancer screening”.
A recent qualitative study135 on what affects screening for bowel cancer suggested that
“people might feel more inclined to accept screening if they had current information about
patients' experiences of colonoscopy and treatment for early bowel cancer.” The widely
known DIPEX system (now renamed as Healthtalkonline (www.healthtalkonline.org) covers
the experiences of almost 50 different conditions and uses systematic qualitative research
for each condition to gather the experiences of a broad range of people. The site is used
extensively by patients and professionals and is linked to NHS Choices.
Prostate cancer screening
Volk et al (2007)136 identified 18 trials in a systematic review of decisions aids for prostate
cancer screening concluding that these decision aids enhance patient knowledge, decrease
decisional conflict, and promote greater involvement in decision making but that the absence
of outcome measures that reflect all elements of informed decision making limits the field.
Such decision aids can be web-based but may also be delivered using other media, e.g.
print, DVD.
Ellison et al (2008)137 in a small trial with 87 African-Americans found evidence that the
enhanced web decision aid was significantly more effective than the usual care decision aid
in promoting knowledge of the benefits, limitations and risks of prostate cancer screening.
26
Web-based sites may be effective in facilitating discussions about screening between
patients and health care providers.
Ilic et al (2006)138 compared Internet with video and written materials in informing men about
patient decision making, attitudes, knowledge, and screening interest in prostate cancer in a
randomised trial with 161 men who had never been screened. They found no difference in
the three methods and concluded that “Health professionals should provide patient health
education materials via a method that is most convenient to the patient and their preferred
learning style.”
Access to information about screening
RCTs in prevention and health promotion all suffer from the problem that only those who are
sufficiently interested will take part, whereas the groups most requiring information and help
in making decisions about screening or preventative behaviour are often either not interested
or do not have access to the Internet. It is therefore worth mentioning two studies:
 Dey et al (2008)139 in an Australian study found that not many women used the Internet
to obtain information about breast screening. Of 415 women, 80% accessed the Internet
but only 7% of the total women accessed breast health and screening information.
 Kreuter et al (2006)140 carried out a study (not an RCT) of community placed kiosks
aiming to bring information about breast cancer screening to populations with limited
access to the Internet. Their kiosks were well used but, as far as we are aware the
kiosks have not continued – i.e. they were only there for the duration of the trial.
3.4 Quality of primary care consultations
There are numerous studies showing that patients perceive that they benefit from being able
to access Internet information (e.g. 141 142). For example, in the US, Pew has conducted a
series of surveys over the last decade asking respondents if and how they use the Internet
and if it has any impact. In the August 2000 Pew survey143, 91% of online health seekers
reported they had learned something new, 55% said it improved how they get medical and
health information, 48% said the online advice had improved the way they take care of
themselves, and 47% who had looked for health information for themselves during their last
Internet search indicated the information affected their decisions about care and treatments.
In June 2001, 16% of online health information seekers said it had a major impact, and 52%
said a minor impact, on their own health care routine or the way they helped care for
someone else. Of the online health information seekers in the December 2002 survey144,
73% reported that the Internet had improved the health and medical information and
services they received, and 14% said it had not improved.
Wise et al145 in a study of the impact of online narrative, concluded that clinicians should
provide lists of web high quality resources that provide both narrative and didactic
information. Although patients may seek information, only a minority then take that
information to the consultation. Murray et al in 200326 carried out a US telephone survey
(n=3209) and found a third had looked for health information on the Internet in the past 12
months, 16% had found health information relevant to themselves and 8% had taken
information from the Internet to their physician. Of the 8%, most (71%) wanted the
physician’s opinion, rather than a specific intervention. The effect of taking information to the
physician on the physician-patient relationship was likely to be positive as long as the
physician had adequate communication skills, and did not appear challenged by the patient
bringing in information. Barnoy similarly found with Israeli nurses in 2008146 that generally
nurses were positive about patients bringing information from the Internet but that
experienced nurses were better able to handle it than less experienced colleagues. In a
27
small 2004 study of MS patients Hay et al35 found that 82% of MS patients are informed by
online information. However, like Murray, they found that only about a third of those
presented this to their physician, many having concerns about the appearance of nonadherence on the doctor-patient relationship.
Stevenson et al32 carried a focus group study with adult patients with diabetes mellitus,
ischaemic heart disease or hepatitis C. This suggested that despite evidence of increasing
patient activism in seeking information and the potential to challenge the position of the
doctor, patients appear to see the Internet as an additional resource to support existing and
valued relationships with their doctors. Their accounts did not suggest a desire to disrupt the
existing balance of power, or roles, in the consultation.32 On the other hand we reported in
section 3.1 that Weaver et al (2009)47 invited people from a population panel from Seattle.
The 18% response in 48 hours comprised 562 individuals aged 19-90 who completed an
online questionnaire. Eleven percent had used information obtained via the internet to refuse
or discontinue treatment recommended by a doctor or dentist.
Wald et al29 who reviewed the literature on the effects of the Web in regard to health care
delivery and the physician-patient relationship found that “the "triangulation" of patient-Webphysician can be expected to significantly impact dynamics of the physician-patient
relationship. Potential advantages of Web-acquired information include helping patients
make informed health care choices (with potential to decrease health care disparities),
shared decision-making with a collaborative, teamwork approach, more efficient use of
clinical time, augmenting of physician-provided information, online support groups, and/or
access to patients' own health information. Alternatively, factors such as misinformation due
to highly variable quality of Web information, possible exacerbation of socioeconomic health
disparities, and shifting of conventional notions of the physician-patient relationship
("traditional" medical authority) present their own set of challenges for the health care
provider.” The need for clinicians to take account of this ‘triangulation’ had been described
before by Jones et al (2001) in specifying the learning needs of clinicians147.
The way Internet searching affects the consultation perhaps needs to be placed in the
context of other influences. A recent review (Edwards 2009)34 tried to identify external
influences on information exchange and shared decision-making in healthcare consultations
and conceptualise how information is used both outside and within consultations. The
literature suggested that “practitioner influences were: receptiveness to informed patients
and patient choice, lack of knowledge of cultural difference, patient centredness vs.
stereotyping. Patient influences were: motivation to seek and engage with information; the
appraisal of information before a consultation, expression of cultural identity, and ways of
managing the risk of poor information. Shared influences were: differing illness notions, role
expectations and language. Empowerment, disempowerment and non-empowerment were
outcomes of information exchange and health literacy was a mediator of external influences
and empowerment.”
The other type of digital health service which can affect the quality of primary care
consultations is the use of computer-patient interviewing. Computers have been used for
patient interviews since 1966148. A branching series of questions is developed to assist the
medical history taking of the clinician. Standard, carefully worded questions are used to
collect a history, with systems having hundreds if not thousands of questions, but patients
only answering those relevant. Some 200 studies of computer-patient interviewing (reviewed
by Jones149, Bachman150, and Slack151) demonstrate benefits for patients such as:
 giving patients more time to think about questions150
 allowing patients to more easily disclose information about embarrassing topics, for
example an English study152 of computer interviewing for pelvic floor symptoms in both
28

primary care and hospital found ‘Despite the taboo nature of many of the items, the
questionnaire was well received by women in both settings.’
ensuring that lines of investigation are not forgotten, leading to more complete data and
fewer errors in diagnosis and better agreement between patient and doctor. For
example, a recent German hospital study153 found that computer histories reported an
additional average of 3.5 problems per patient which were not recorded in corresponding
physician histories. The authors recommended a combination of computer and physician
histories as the best method. A further study in Canadian emergency care, reported that
the computer history asked 90%, and the emergency physician 55%, of important
historical elements154.
In addition to these benefits, CPI could ‘enable’ or ‘empower’ patients but we are not aware
of any studies that have focussed on this as a primary outcome. By preparing patients better
for their general practice consultations, with the use of a CPI, patients may be ‘enabled’ or
‘empowered’, have more time in the consultation and for the consultation to be more patient
centred, thus potentially increasing patient satisfaction with the consultation. The summary
information may also help patients to find more relevant information on the Internet by
ensuring that they search with the correct keywords.
CPI can also be of benefit to the doctor or nurse in general practice, in part through the
benefits it brings patients and by allowing clinicians to focus more quickly on areas of
concern to the patient. If a summary of data gathered by the CPI is sent directly to the
electronic medical record it leads to more complete records, and offers the potential for more
effective use of personnel by reorganising care, for example, through e-consultations
(personal communication, John Bachman, Mayo Clinic).
One of the problems of CPI research in the past is that recommendations of successful
research have failed to become embedded in routine practice149. In the USA, the commercial
development of a system called Instant Medical History (IMH) has changed the situation by
embedding and supporting routine use of CPI. Primetime Software is successfully marketing
a wide ranging CPI and continuing to incorporate further interviews developed in research
projects. IMH has been under development in the US since the early 1990s155 and is now
embedded in 40 US Electronic Medical Record systems, has been sold directly to hundreds
of additional practices and has been used on the Internet for several million patient visits in
the USA156. Patients are able to use IMH and respond favourably to its use157 158. To our
knowledge, no such system has been implemented in the UK general practice, although we
believe that EMIS is developing a similar system. A recorded webinar by Wenner describes
how IMH works159; patients complete online interviews before consultations, history
summaries are added to medical records and are available to patients and doctors before
the consultation.
At the moment there is no direct evidence that CPI has benefit on the primary care
consultation in the UK, but the evidence from US primary care and studies on secondary
care CPI suggest this is a development worthy of further study.
3.5 Long Term Conditions
Cancer
We can see how the nature of reviews has changed over the last decade going from looking
at various paper based information provision to videotapes and computer technologies. We
might assume however that the interventions reviewed in the earlier studies, if now
29
presented as a web application and compared in the same way, may well have similar
outcomes, although we have no evidence to support that.
 McPherson (2001)160 reviewed 10 studies of interventions ranging from written
information to audiotapes, audiovisual aids and interactive medium. Individually tailored
methods such as patient care records and patient educational programmes were also
reviewed. They concluded that the interventions had positive effects on a number of
patient outcomes, such as knowledge and recall, symptom management, satisfaction,
preferences, health care utilization and affective states.

Eysenbach (2003)161 reviewed the impact of the Internet on cancer outcomes. He
concluded that provision of information to persons with cancer had been shown to help
patients gain control, reduce anxiety, improve compliance, create realistic expectations,
promote self-care and participation, and generate feelings of safety and security (Mills &
Sullivan 1999, Mossman et al 1999), and that satisfaction with information had been
shown to correlate with quality of life (Annunziata 1998) and patients who feel satisfied
with the adequacy of information given are more likely to feel happy with their level of
participation in the overall process of decision making (Turner Br J Cancer 1996). He
stated that more than 15 randomized trials had evaluated interventions to provide
information to persons with cancer, but cited (Mohide 1996) which was an RCT (?). He
said that most had focussed on evaluating the effect of providing printed patient
education pamphlets or computer-based personalized (Jones 1999) information to
patients. He claimed that in 2003, relatively little was known about the effects of general
undirected “chaotic” Internet information on persons with cancer.

Gysels (2007)162 reviewed nine randomised trials (three on the use of videotapes and six
on computer technologies). Agre et al.(1994) (Colonoscopy video), Brown et al.(2004)
(Breast cancer video), Thomas et al.(2000) (Video for various cancers), Gustafson et
al.(2001) (Breast cancer computer), Jones et al.(1999) (Breast, prostate, cervical,
laryngeal cancer computer), Maslin et al. (1998) (Breast cancer computer), Shaw et al.
(2001) (Colonoscopy Computer), Street et al.(1995) (Breast cancer computer), Whelan
et al.(2003) (Breast cancer computer). These showed improved patient knowledge
(effect sizes ranging from 0.12 to 1.03). Satisfaction was improved in some studies, but
the overall effect was more equivocal-effect sizes ranged (0.05 to 0.40) of benefit for
knowledge and from 0 to 0.40 for satisfaction.
Paediatrics (particularly asthma)
Moenedin et al163 carried out a systematic review of the application of ‘biomedical
informatics’ to chronic conditions in children and adolescents 18 years of age or younger.
They focused on studies evaluating applications of any web or computer-based information
and communication technology designed to aid the clinical care of chronic illnesses in
paediatric settings, including computer-based interventions to support clinical care of
physical or mental chronic conditions. Their inclusion criteria was wider than is of interest in
this review, including applications or systems used to diagnose or detect symptoms;
applications that prevent or monitor symptoms; decision support systems, alert and reminder
systems; as well as patient-centred education applications. They identified 27 relevant
articles (predominantly paediatric asthma), 96% of which reported improved outcomes.
Outcomes included but were not limited to, reduced number of emergency room visits,
improved knowledge on disease management, and enhanced satisfaction. The papers dealt
with: asthma 8, autism, 9, cognitive disability 7, paediatric oncology 3, diabetes 2, and other
3. (Some dealt with more than one condition). Paediatric asthma had been the subject of a
previous systematic review164. Given the size and potential of benefit of digital health
services for asthma, it is worthwhile to look at the 8 asthma studies in more detail. These are
shown in Table 8 together with subsequent from the same authors.
30
First author
Chan 2003165.
Shegog
2001166
Bartholomew
2000167
Huss 2003168
McPherson
2002169
McPherson
2006170
Porter 2004171
Krishna
2003172
Krishna
2006173
Homer
2000174
Intervention
Therapeutic monitoring included digital
videos of patients using their controller
medication inhaler. Diagnostic monitoring
included an asthma symptom diary and a
video of peak flow meter use. Videos were
submitted electronically twice a week by
using in-home telemonitoring with store-andforward technology. Feedback was provided
electronically to each patient.
Watch, Discover, Think and Act (WDTA), a
theory-based application of CD-ROM
educational technology for pediatric asthma
self-management education.
Watch, Discover, Think, and Act
Computer-assisted instructional game on
asthma symptoms that focused on reducing
environmental irritants and allergens and the
correct use of prescribed medications to
prevent asthma symptom.
Pilot Study. The Asthma Files, an interactive
CD-ROM for children with asthma
Full RCT. The Asthma Files, an interactive
CD-ROM for children with asthma
Asthma kiosk: (i) front-end computer-patient
interview (2) output to both patients and
clinical providers regarding best practice.
Self-management education through the
Interactive Multimedia Program for Asthma
Control and Tracking
Self-management education through the
Interactive Multimedia Program for Asthma
Control and Tracking
An interactive educational computer
program, Asthma Control, designed to teach
children about asthma and its management,
daily events, the game emphasizes: 1)
monitoring; 2) allergen identification; 3) use
of medications; 4) use of health services;
and 5) maintenance of normal activity.
Outcome
Improvement in quality of life, reduction in
emergency department and hospital visits,
infrequent rescue therapy and a high level of
satisfaction with home telemonitoring
Improvement in knowledge and asthma self
management
Increase in knowledge of asthma selfmanagement, reduction in emergency room
visits and fewer hospitalizations, greater self
efficacy
No difference on asthma symptoms such as
coughing, wheezing, shortness of breath, night
time awakenings
Increased knowledge about triggers of asthma
Improved knowledge and a more internal locus
of control but no differences in objective lungfunction measures, hospitalizations, or oral
steroid use at one month. Fewer children in the
intervention group had required oral steroids
and an indication of possibly fewer oral steroids
and less time off school at 6 months but this
was not significant in intention to treat analysis
Improved quality of asthma care and patient
satisfaction through use of a kiosk by parents
Increase in asthma knowledge of children and
their caregivers, decrease in asthma symptom
days, decrease in number of emergency
department visits, significantly lower dose of
inhaled corticosteroids.
Improvement in days of asthma symptoms, in
emergency room visits, in school days missed,
and in days of activity limitation
Reduced emergency department and office
visits and improved asthma-related knowledge
Table 9. Child asthma interventions and outcomes
Adult asthma
Van der Meer (2009)175 reported an RCT with 200 adults who were treated with inhaled
corticosteroids for 3 months or more during the previous year and had access to the Internet,
in 37 general practices and 1 academic outpatient department in the Netherlands. They
compared Internet-based self-management (weekly asthma control monitoring and
treatment advice, online and group education, and remote Web communications) with usual
31
care. Internet-based self-management resulted in improvements in asthma control and lung
function but did not reduce exacerbations, and improvement in asthma-related quality of life
was slightly less than clinically significant.
Chronic Obstructive Pulmonary Disease (COPD)
Nguyen et al176 carried out an RCT comparing internet-based versus face-face selfmanagement for patients with COPD. They had numerous technical challenges, a low
recruitment and high drop-out rate, but amongst those who remained in the trial there were
positive outcomes.
Diabetes
As has been the situation for many years diabetes (and renal disease) are at the forefront of
digital health services, not just offering ‘information’ on the web but complete comprehensive
services.
 Patient access to their records. Lee et al177 used a quasi- experimental method to
evaluate patient access for 274 patients with diabetes, to their records. We need further
review of the paper but it seems they claim that the patients in the intervention group had
better control of their fasting blood glucose, HbA1c and total cholesterol levels than
those in the control group due to the assistance of the system.

Ralston et al (2009)178 report a RCT to assess the impact of web-based care
management of glycemic control using a shared electronic medical record with patients
who have type 2 diabetes. They recruited 83 adults with type 2 diabetes and randomised
them to receive usual care plus web-based care management or usual care alone
between August 2002 and May 2004. Intervention patients received 12 months of webbased care management. The Web-based program included patient access to electronic
medical records (My Health Record), secure e-mail with providers, feedback on blood
glucose readings (My Upload Meter), an educational Web site, and an interactive online
diary for entering information about exercise, diet, and medication. Haemoglobin a1
levels declined by 0.7% among intervention patients compared with usual-care patients.
Systolic blood pressure, diastolic blood pressure, total cholesterol levels, and use of inperson health care services did not differ between the two groups.

Azar and Gabbay have recently (2009)179 reviewed the use of web-based glucose
uploads. They found that studies in type 1 diabetes tended to show equivalent HbA1c
improvements in both intervention and control groups without statistically significant
difference, while type 2 patients seemed to do better than controls with significant
differences in HbA1c. Patients were the beneficiaries of web-based diabetes
management both through savings in time and cost. Major obstacles to wider
implementation are patient computer skills, adherence to the technology, architectural
and technical design, and the need to reimburse providers for their care.

Bond et al180 carried out an RCT with 62 adults aged 60 and over with diabetes
comparing a 6-month web-based intervention plus usual care, compared with usual care
alone. They concluded that the intervention (need more details) was effective in
improving HbA1c, weight, cholesterol, and HDL levels at a 6-month follow-up.

Jackson et al181 reviewed publications to the end of 2003 identifying 26 studies (27
reports): internet (n=6; 3 RCTs), telephone (n=7; 4 RCTs), and computer-assisted
integration of clinical information (n=13, 7 RCTs). Most studies reported overall positive
results and found that IT-based interventions improved health care utilization, behaviors,
attitudes, knowledge, and skills.
32
Rheumatoid Arthritis (RA)
People with rheumatoid arthritis (RA) should use DMARDs (disease-modifying antirheumatic
drugs) within the first three months of symptoms in order to prevent irreversible joint damage
but patients often delay in using them. Li et al (2009)182 report the development (work in
progress) of a web-based decision aid for patients in RA in using DMARDs. Their evaluation
results are likely to be available in 2011.
Stroke
Smith et al183 completed a Cochrane review of RCTs involving patients or carers of patients
with a clinical diagnosis of stroke or transient ischaemic attack ( TIA) where an information
intervention was compared with standard care, or where information and another therapy
were compared with the other therapy alone. They identified 17 trials, however, despite
nearly all including some form of booklet, leaflet, or face to face information and education,
only two were computer-based (Hoffman 2007184 and Maasland 2007185). Meta- analyses of
the 17 trials showed a significant effect in favour of the intervention on patient and carer
knowledge, one aspect of patient satisfaction, and patient depression scores. There was no
significant effect on number of cases of anxiety or depression in patients, carer mood or
satisfaction, or death. Smith et al concluded that information improves patient and carer
knowledge of stroke, aspects of patient satisfaction, and reduces patient depression scores.
However, the reduction in depression scores was small and probably clinically insignificant.
Smith et al allocated studies to one of two categories - passive information or active
information - according to the nature of the intervention. An intervention was classified as
passive if the information was provided on a single occasion and there was no subsequent
systematic follow up or reinforcement procedure. An intervention was classified as active if,
following the provision of the information, there was a purposeful attempt to allow the
participant to assimilate the information and a subsequent agreed plan for clarification and
consolidation or reinforcement. They classified both Hoffman and Maasland as passive
interventions. Smith concluded that although the best way to provide information is still
unclear there is some evidence that strategies that actively involve patients and carers and
include planned follow up for clarification and reinforcement have a greater effect on patient
mood.
In Hoffman’s study, in an Australian acute stroke unit, 138 stroke patients were randomised
to receive either computer-generated tailored written information about stroke or generic
written information while in hospital. Three months following discharge patients with tailored
information were more satisfied with the content and presentation and less likely to want
additional information. However, anxiety scores improved slightly more in favour of the
control group. There was no difference for effect on knowledge about stroke, self-efficacy,
depression, or perceived health status.
In Maasland’s Dutch study, 65 transient ischemic attack or minor stroke patients were
randomised to health education by a physician (n = 32) or to a combination of education by a
physician and a multimedia program that tailored the information to the individual (n=33).
Overall knowledge amongst patients was low but the intervention group had slightly better
scores at 1 week after using the computer program. However, after 12 weeks, the score in
the intervention group dropped significantly and was no longer different from the standard
group. They concluded that there was no lasting effect of health education by an
individualized computer program on the knowledge of TIA and minor stroke patients.
33
Rochette et al (2008)186 carried out an extensive systematic search to identify and appraise
existing stroke rehabilitation websites. They identified 17 but although some addressed
stroke rehabilitation interventions in layperson language, none discussed the numerous
treatment options based on scientifically based information. As a result they have started to
develop a site to meet this need.
In conclusion, stroke patients can benefit from information interventions and so appear to be
one group who might benefit from digital health services but as yet there is only limited
evidence of improved outcomes. It would appear to be an area where evaluation of digital
health services that build upon evidence from successful non-computer-based interventions
would be worthwhile.
Mental Health
Mental health is one of the most advanced areas both in terms of the development and
availability of digital health services but also in their evaluation. For example, computerised
cognitive behavioural therapy (CCBT) offers effective treatment for depression. In 2004
NICE reviewed ‘stand-alone’ CCBT packages recommending ‘Beating the Blues’ for
depression187 but an update to this guidance has recommended any CCBT with given
characteristics. Since 2004 various free to use packages available via the Internet, including
MoodGym188 and Living Life To The Full189 (LLTTF) have become available. ‘Stand-alone’
CCBT requires licence fees and booking of a practice computer whereas webCBT is free to
the user and can be accessed anywhere. A meta-analysis of WebCBT showed moderate to
large effect size for anxiety and depression190 .
Work continues developing, using, and evaluating webCBT sites and some of the
groups/researchers are shown in Table 9. (As far as we know Beating The Blues, the CCBT
first recommended by NICE is only available via licenced NHS sites, whereas MoodGym and
LLTTF are openly available on the Internet).
Website
http://moodgym.anu.edu.au/welcome
http://www.livinglifetothefull.org.uk/
http://www.beatingtheblues.co.uk/
Research group
Christensen, Griffiths et al188 191-199
Williams et al200 201
Proudfoot, Cavanagh, Marks et al202-205
Location
Australia (web)
Scotland (web)
England (GenPrct)
Table 10. Web CBT for depression
As well as the cognitive behavioural approach, as with depression (above) and other
conditions such as panic disorder206, use of the Internet for information seeking in mental
health207, we are now seeing use of the Internet for professional-patient approaches.
Barak208 reviewed Internet-based psychotherapeutic interventions and found a medium
effect size similar to the average effect size of traditional, face-to-face therapy. More
recently, Kessler recently reported an RCT of therapist-delivered internet psychotherapy for
depression in primary care209 concluding that personally delivered CBT seems to be effective
when delivered online in real time by a therapist, with benefits maintained over 8 months.
Computer-mediated support groups
It seems appropriate to review here, under long term conditions, the literature on computermediated support groups (CMSG). The most well-known work is that of David Gustafson et
al with the Comprehensive Health Enhancement Support System (CHESS) and this has
been referred to above (section 2). Rains recently reviewed CMSGs focusing on the role of
group communication and social support210. They described CMSGs as “distinct from the
informal and loosely structured self-help groups found on Websites such as WebMD.com
and Yahoo.com in that formal CMSGs have both educational and group communication
34
components, closed membership enrolment, fixed duration, and may include some form of
expert leadership”. These would seem of relevance to NHS Choices as Choices could
provide a portal for such support. They included 28 studies covering 12 conditions
(depression, diabetes, disordered eating, breast cancer, weight loss, chronic illness,
cardiovascular disease, Parkinson’s disease, back pain, smoking cessation, HIV/AIDS, heart
transplant) in their meta-analysis. They concluded that “The results show that participating in
a CMSG intervention-comprised of educational and group communication components-led to
increased social support, decreased depression, increased quality of life, and increased selfefficacy to manage one's health condition. Changes in health outcomes were moderated by
group size, the nature of the communication channels available, and the duration of the
CMSG intervention.”
4. Information Prescriptions
The Department of Health commissioned evaluation of the Information Prescriptions pilot
project. The final report was published in August 2008211. A high number of patients and
carers found the information useful and professionals involved in the pilots were positive
about information prescriptions. In qualitative interviews with pilots, there was strong
support for the concept and potential of information prescriptions, with most of the pilots
continuing to implement information prescriptions beyond the pilot funding allocation.
5. Digital Divide
NHS Choices asked “Are digital health services successful in reaching the C2DE population
and reducing the health inequality gap between socioeconomic groups?”
Although home Internet access for the UK had increased from just over 30% in 2000 to 55%
in 2005 and to 70% by August 2009 (www.statistics.gov.uk/cci/nugget.asp?ID=8 ) there is
still variation by income, age and region212. For example, in 2007 60% of households in
London compared to 40% in Northern Ireland had a broadband connection. In 2006, 87% of
16-30 year olds had used a computer in the previous three months compared with 45% of
those aged 50 and over213. In 2003/04, 12% of those in the worst paid compared to 85% in
the best paid decile, had access to the Internet. There is a considerable overlap in different
types of information and communication technologies ( ICT); in 2006 212 45% of households
had a digital television service, owned one or more mobile phones, and had access to the
Internet, while 8% did not have access to any of these ICT capabilities.
35
120
100
16-24
80
25-44
60
45-54
55-64
40
65+
20
Jul-05
Jan-06
Jan-05
Jul-04
Jan-04
Jul-03
Jan-03
Jul-02
Jan-02
Jul-01
Jan-01
Jul-00
0
Percentage of different age groups in Great Britain who have
ever used the Internet. (Constructed from NOS statistics) 213
The above series of bar charts (extracted from Ofcom) show that in 2007 there were marked
differences in access to the Internet by socio-economic group, with 35% of those in DE
having access to the Internet compared to 82% of those in AB. The digital divide is also very
marked by age (10% of those aged 75+ compared to 82% of those aged 16-19 having
access to the Internet in 2007.
Access to the Internet in rural areas is likely to be affected by low access speeds214.
36
6. Similar services to NHS Choices
NHS Choices wanted to know ‘how similar digital services to the NHS Choices programme
around the world have been successful or not in achieving the benefits NHS Choices was
set up to achieve’. Other services that we are aware of include:
 www.hc-sc.gc.ca Health Canada is provided by the Ministry if Health for Canada in
English and French.
 http://www.passeportsante.net/Fr/Accueil/Accueil/Accueil.aspx. Passeportsante is a
French Canadian site provided by a non governmental organisation (Fondation Lucie et
André Chagnon) in Montreal.
 http://www.healthfinder.gov/. Healthfinder is provided by the US Department of Health
and Human Services in English and Spanish.
 http://www.healthinsite.gov.au/ is provided by the Australian government.
 http://www.everybody.co.nz/ everybody is a consumer health information resource that
is owned and published by CMPMedica (NZ) Ltd. The product is distributed in print (free
to New Zealand GPs) and on the internet.
 http://www.sanidad.es/index.php for Spain (though it is not clear if this is provided by a
commercial organisation or the Ministerio de Sanidad. (It has a strange disclaimer).
The only studies of these sites that we identified were two papers by Lemire et al on
Passeportsante38 215. One of these was a simple survey of visitors attitudes to using the
Internet and why they were visiting the site215 while the other tried to assess empowerment
(mainly considering the elements of what is empowerment’)38. Both papers were based on a
rather selective sample of 2-3000 users of the site who completed an online questionnaire
made available on the site.
7. Methodological difficulties in assessing benefits
There are several problems in trying to review the literature on the benefits of e-health
interventions. The major problems are in the difficulty of primary research to evaluate a
particular intervention but there are also difficulties in identifying and assessing the
relevance of the literature.
7.1 Primary Research
Applying the ‘gold standard’
The ‘gold standard’ approach to assess whether an intervention is beneficial is to carry out a
randomised controlled trial (RCT) comparing the new method against ‘treatment as usual’
and then, when there are sufficient RCTs, to carry out a systematic review and possibly a
meta-analysis. This approach is well defined in the assessment of new medications and
relies on design features such as:
 both new intervention and control (treatment as usual) looking the same so that the
participants can be blinded to the treatment they are taking (single blind) and preferably
also the people administering the treatment (double blind)
 randomisation and lack of other confounding factors so that any differential effect can be
attributed to the treatments
 knowing how the treatment works
Carrying this out in e-health or digital health services is difficult. The inclusion and exclusion
criteria, recruitment method, and choice of placebo or control group in an e-health trial are
very important and we need to discuss these more before reviewing the evidence about
digital health services.
37
Recruitment
In any pragmatic RCT where the aim is to see how a new treatment will work in routine
practice, there are problems of the representativeness of the sample which may be limited
by entry criteria and the self-selection of those who agree to take part. The use of intention
to treat analysis is supposed to cope with these problems but many studies, whether
medication or other intervention, do not deal with this well. Recruitment and inclusion and
exclusion criteria are particularly relevant in assessment of computerised methods where
those who do not have access to the Internet or those who are less IT literate may be
excluded from trials. On the one hand there is perhaps little point trying to recruit people for
an Internet intervention if they do not have access to the Internet and have never used a
computer. On the other hand, if recruitment to an e-health trial results in the participants
being highly IT literate people who spend a lot of time online, the results are likely to be more
favourable than if recruitment was made ‘in real life’ from (e.g.) patients attending an
outpatient clinic. Internet access is increasing all the time, but it is not clear if it is possible to
generalise from Internet users in (say) 2001 to Internet users in 2009. The first were mainly
‘early adopters’ and will have other characteristics associated with that, while the second will
include many later adopters. So it is important to know more about the sampling methods
used and the recruitment rate of any trial. Unfortunately this is not always reported well and
systematic reviewers often seem to be more concerned with trying to calculate effect sizes
(which may be meaningless unless more is known about recruitment).
Comparison groups
Another major problem with e-health research is the choice of comparison group or control.
The same difficulty is found in e-learning research. Cook recently argued216 that systematic
reviews (e.g.217) have shown that e-learning is better than no intervention, but that this
should be no great surprise. He argues that e-learning and traditional instruction are similar
in effectiveness (but of course it depends on how good the particular e-learning package is
and how good the particular traditional instruction was), and that we do not need more nointervention-controlled studies or comparisons with traditional instructional methods, but that
studies should seek to clarify when to use e-learning (studies exploring strengths and
weaknesses) and how to use it effectively (head-to-head comparisons of e-learning
interventions). How does this argument transfer to e-health? In most case the same applies.
If you compare an e-health intervention against no intervention, the e-health intervention is
likely to have a positive result. Unfortunately, ‘treatment as usual’ is often ‘do nothing’, or at
least the e-health intervention is additional to ‘treatment as usual’. For example, an e-health
intervention to improve diet can not remove all the other influences there may be on a
person with respect to their diet. If the comparison group therefore is do nothing, as an
unblended trial the participants are fully aware that they are ‘getting nothing’ while those in
the intervention group are ‘getting something’. The experimental (Hawthorne) effect makes
the prediction easy that trials of that sort may just show that ‘doing something’ is better than
‘doing nothing’. E-health trials, like drug trials, require a placebo arm and like Cook’s appeal
for research in e-learning, the trials we need should seek to clarify when and how to use ehealth effectively.
Research and discussion on placebos in complementary medicine218-220 is relevant.
Placebos can have an effect on patient outcomes to the extent that some placebos are
better than others, response to placebo will depend on individuals and the context218 219, and
the different elements of placebos (e.g. assessment and observation, therapeutic ritual
(placebo treatment), and a supportive patient-practitioner relationship) can be combined to
give a graded placebo220.
Outcome measures
Having appropriate outcome measures that can measure all the benefits of an e-health
intervention is another problem and illustrated by the growth of peer to peer (i.e. patient to
38
patient, public to public) support and information in health. A well conducted, rigorous,
systematic review of peer-peer support by Eysenbach, Powell et al in 2004221 was criticised
by Jadad et al222 because it “.. attempted to measure the value of virtual communities in the
health sector through the traditional lens of quantitative research methods, missing the fact
that such value transcends conventional measures of clinical outcome, resource use, or
social support.” It may be argued that peer-peer support is outwith the scope of ‘digital
health services’ in the style of NHS Choices. However, many ‘information’ websites are
increasingly including elements of peer-peer support. This is also the direction of travel for
Choices, with blogs, online support groups, and expert patients.
Finding appropriate outcome measures to assess the ‘benefit’ of e-health interventions is
difficult. For example, empowerment or enablement is often cited as a general aim of
providing digital health services but is a difficult concept to ‘measure’ and often seems to be
confused with patient satisfaction (e.g. an editorial entitled Evidence Based Patient
Empowerment42 is all about patient satisfaction). Furthermore the benefits of empowerment
are disputed40. Salmon et al40 argue that patient empowerment sits uncomfortably with
evidence-based medicine and is in any case constrained by organizational, clinical or
economic factors. They also argue that although controlled studies of empowerment—for
example, arranging for patients to choose the nature or timing of treatment, or teaching them
‘coping skills’—do often favour intervention groups, the effects are variable, and (they argue)
sometimes favour lack of choice. They argue that there has been little evidence that most
patients seek control or choice223 and some evidence that they do not224. Being ill reduces
the importance that people attach to control225.
Paterson et al226 argue that the interpretation of 'outcomes' that may be appropriate for
clinical trials of pharmaceutical products, is problematic when used in evaluations of complex
interventions in areas such as complementary medicine, palliative care, rehabilitation, and
health promotion. Nevertheless, we would argue that we should not ‘throw the baby out with
the bath water’. Although outcome measurement is difficult, the randomised trial format still
has applicability in evaluating e-health but it does need to be used more carefully and
innovatively, and we should be wary of unthinking application of ‘medication’ trial guidelines
to e-health.
Problems with analysis and interpretation
Many papers of course can be criticised for some aspect of their methods. For example,
most RCTs claim to analyse their data on an intention to treat basis, but some referees and
journals have more flexible interpretations of this than others. Attrition from online RCTs is a
particular problem and can be high, for example Riper et al (2007) in a trial of self-help for
drinking reported 46.2% attrition in the intervention group and 38.2% in the control over a 6
month follow up99. Swartz et al (2006) in a trial of smoking cessation reported similar levels
of attrition (49% in intervention and 39% in control) at 90 day follow up. With such high
attrition, and particularly if there is differential attrition, there must always be some doubt
about the validity of the results. This can be magnified by different methods of undertaking
‘intention to treat’ analysis. Most authors tend to just analyse those completing follow up but
there is argument that a more conservative approach should be taken. Swartz et al took the
conservative approach of assuming that anyone lost to follow up has not changed, ie you
assume that people lost to follow up are the same as at baseline: if they smoked at baseline
we assume they still smoke, if they had a particular anxiety score at baseline you ‘carry
forward’ that score. This has the effect of ‘damping down’ the calculated effect of any
intervention. Riper et al on the other hand used a multiple imputed values approach. They
replaced missing values (where participants had not completed follow up) by values drawn
randomly from ‘donor’ cases with complete data based on ‘donor’ cases stratified by gender,
age group, and other baseline scores. They ran this simulation ten times (hence ‘multiple’
39
imputed values). One could argue however that this actually increases the ‘power’ of the
study and that the assumption (that people who did not complete follow up are ‘like’ those
who did) is wrong. (This is similar to the difficulty of non response in surveys where there is
evidence that non responders are not like responders). There is a good argument for all
three types of analysis (i.e. completers, carry forward, and imputed values) to be presented.
On the other hand, there is a good argument that the most appropriate result to present for
web-based interventions is ‘completers’ because, unlike a trial of medication or face-face
therapy, the marginal cost of an extra participant in a web-based intervention for the health
service is virtually zero. Therefore if the intervention helps some people, that is what is
important and we should just report the take up of the ‘service’ amongst the affected
population and how many (and if possible the characteristics) of those for whom a webbased intervention is successful. Murray et al101 note that “exposure to an Internet
intervention is important for the interpretation of trial results. Since participants who never
used the intervention are likely to differ systematically from those who did, they must be
included in the analysis in their randomized group (the intention-to-treat principle227).
However, there may be interest in understanding the benefit of the intervention in those who
did use it. This should be explored by methods such as estimating the “complier average
causal effect” (CACE), which effectively deduces the benefit of the intervention in those who
did use it from the intention-to-treat results and the proportion of intervention users228”.
7.2 Secondary research
Locating publications
At the time of writing this paragraph we had 232 publications in our bibliographic database of
which 13 were reports, 13 conference proceedings, and 206 were journal articles. As can be
seen from Table 11 these are scattered through 113 or more journals; 52 were in 2009 or in
press, 57 in 2008, 25 in 2007, 26 in 2006. Although the increase in numbers of publications
in recent years may be partly because of our emphasis on getting up to date information it
also illustrates the growth in impact assessment of e-health.
Journal Of Medical Internet Research
Patient Education and Counseling
British Medical Journal
Journal of the American Medical Informatics Association
American Journal of Preventive Medicine
Preventive Medicine
Addictive Behaviors
Annals of Internal Medicine
Archives of Internal Medicine
Health Education Research
International Journal of Medical Informatics
Journal of Health Communication
Pediatrics
Acta Neurologica Scandinavica, Addiction, Addiction Research & Theory, Age and Ageing, Aids, Aids and
Behavior, Alcohol and Alcoholism, Alcoholism-Clinical and Experimental Research, American Behavioral
Scientist, American Journal of Community Psychology, American Journal of Health System Pharmacy,
Annals of Allergy Asthma & Immunology, Annals of Behavioral Medicine, Annual Review of Public Health,
Australian and New Zealand Journal of Psychiatry, BMC Complementary and Alternative Medicine, BMC
Family Practice, Bmc Medical Informatics and Decision Making, BMC Medical Informatics and Decision
Making, BMC Psychiatry, BMC Public Health, British Journal of General Practice, etc..............(100 journal
titles)
Table 11. Source of publications used in this review (at time of writing this paragraph)
40
29
10
8
6
5
5
4
4
4
4
4
4
4
3 or
less
Keeping track of this evidence in a timely manner, is going to be quite difficult and
international collaboration seems worthwhile. As Bates et al argue229 “... enormous savings
might be realized rapidly if international eHealth collaborations become more frequent, and
more knowledge generation and even data interchange begin to occur.”.
‘Shelf life’ of benefits research
Another problem is the rapid progression of technology such that new technology becomes
available during the course of an RCT which makes the RCT obsolete if it is too focussed on
a particular technology rather than human behaviour. This is also a problem for systematic
reviewers who need to choose a cut-off date for their reviews, but that may also exclude
papers which are still relevant. Taking into account the discussion about recruitment above,
we need to be wary of the timescale used in systematic reviews to combine RCTs; the
individual trial design needs to be considered to make a judgement on its current relevance.
Related to the possibility that the technology used may become obsolete are population
changes in attitude and the context in which a system is used. People’s expectations change
and a computer system that was effective in one decade may no longer be so one year later,
as part of the Hawthorne (novelty) effect is no longer present. (It would be difficult to imagine
patients now taking part in a study of computer-patient interviewing, using a ‘teletype’ and
attributing the computer with ‘understanding’ (personal communication from Robin KnillJones concerning his research in the 1980s230). Finally, effective systems may ‘die’ because
of the lack of infrastructure or funding, for example, projects funded as research not adopted
(despite good evidence) as routines by health service organisations. We have shown above
that many of the sites used for studies of impact are not publicly available. There may be a
role for a national organisation such as NHS Choices in ‘adopting’ and making publicly
available ‘research developed’ web services.
Publication bias
The problems of publication bias (where positive results are easier to publish than negative
results or ‘no significant difference’) are well known in all fields of research but may be
particularly severe in e-health. In drug trials the tendency is to blame the researchers – with
the implication that authors do not want their results published. In e-health, my personal
experience is that journals are often not interested in equivocal results (e.g. my BMJ paper231
reporting an RCT in personalised education for patients with schizophrenia (relatively weak
effect but from a ‘strong’ study) was initially not going to be published by the BMJ and was
only published on appeal.
8. Discussion
Although there are methodological difficulties which are important to consider, there seems
to be some evidence of patient benefit from digital health services. Quite understandably,
there are far more published evaluations in the promotion of healthy lifestyles, in particular
smoking cessation, sensible drinking, exercise, and healthy eating, than education or
support for long term conditions other than mental health. Setting up and recruiting to a
website that promotes behaviour change amongst the anonymous public is operationally far
simpler than recruiting NHS patients for follow up. Digital mental health interventions are
also well developed and a number of outcome studies published or underway indicating that
they are effective.
There seems to be some evidence that well designed sites aiming to change behaviour have
some effect, taking into account that people registering are mostly those who are already
41
motivated to take some action, that many studies have high attrition which is not always
dealt with in the analysis and interpretation, and that outcomes are for obvious reasons self
reported with little chance of external validation. For example, there seems to be some
evidence that web-based attempts to encourage sensible drinking have some effect, though
there are numerous methodological difficulties in individual studies and even more in trying
to synthesise them. We only found one cost-effectiveness study of a healthy living
intervention and more work on cost effectiveness would be worthwhile, as although the
effect size may be small, the marginal cost of extra users of such web-based services is
virtually zero and the impact in terms of longer term may be substantial. In particular, there is
very little evidence of the cost effectiveness of digital approaches to behaviour change or
mental health compared to more traditional General Practice or mass media approaches.
Although more work is needed to follow up this review it seemed that a number of streams of
work and web sites are still underway or available for sensible drinking and mental health
and it would seem sensible to make use of that work rather than re-invent it. Unlike the
sensible drinking sites most of those used for trials of healthy eating, quit smoking etc seem
to have used ‘project’ sites. Nevertheless, contact with the main groups across the world
would seem worthwhile. One of the most productive lines of further work would seem to be
in producing a repository of resources on decision support.
Evidence on patient choice is of two types:
1. Information about providers. The evidence of the effect on patient choices here
comes mainly from NHS surveys which will be well known to NHS Choices. We have
not found much further evidence.
2. Information about treatment choices. There is good evidence from systematic
reviews but these are not necessarily available or easily found online. This would
seem to be an area where NHS Choices could further the ‘patient choice agenda’ by
(a) collecting, collating and making available all such materials digitally and (b)
raising awareness of these for both clinical staff and patients. Although in the body of
this report we have included headed it ‘screening’, the literature on screening is
mainly concerned with decision support. Like treatment choices, the literature is not
necessarily about online support.
Many patients will access the Internet, before, after, or instead of seeing a clinician and there
are numerous studies showing the benefit to the patient of that access, but what is the
impact on the quality of the consultation? Does it make for more productive use of time or
does it lead to conflict or time wasting in dealing with misinformation? Those patients who
take the information to the doctor tend to be in the minority and some patients are worried
that it will be seen as non-adherence. On the whole experienced clinicians and those with
good communication skills are able to deal with patients who do bring information, and they
welcome the patients taking this initiative. However, a recent Seattle survey found that 11%
used information from the Internet to discontinue or change a doctor or dentists
recommended treatment.
‘Stand-alone’ e-health services, i.e. those which provide support for anyone with a condition
without any link to medical records, or patient-clinicians communication, have proved of
benefit to people with long term conditions. However, some of the most interesting and
advanced uses of digital health services have taken a more integrated approach are in
supporting people with long term conditions, integrating (for example) patient education,
support, along with access to medical records, and the ability to upload their own monitoring
data.
42
Further international collaboration. The literature and evidence on digital health services,
benefits and outcomes and how consumers use the Internet for information is being
developed rapidly (approximately half the 200 papers we have in our database are from
2008 or later) and it will become increasingly difficult to keep up to date. International
collaboration seems obligatory. For example, in response to President Obama's call for
increased transparency, public participation and collaboration in federal decision making, the
US National Institute of Health232 published a call for information in January 2009 on (i)
Health information-seeking behaviors and trends; (ii) Consumers' health information
interests; (iii) How the public accesses and uses health information; and (iv) Barriers that
might hinder NIH's ability to communicate with health consumers. Responses to the RFI are
due Dec. 31 2009. Collaboration between NHS Choices and other national bodies (not just
US) should be pursued.
43
Appendix
NHS Choices ICF
Department of Health
Skipton House
London SE1 6LH
19 June 2009
Dear Sir/ Madam,
RE: INVITATION TO OFFER FOR CONSULTANCY SERVICES TO PERFORM A RAPID LITERATURE
REVIEW OF BENEFITS AND PATIENT OUTCOMES OF DIGITAL HEALTH SERVICES
The Department of Health Intelligent Customer Function invites competitively tendered Offers to
participate in the above tender.
The documents must be returned by no later than 9am on the 10th July 2009, by email to
[email protected]. Hard copies will not be accepted as an offer and will result in offers
being rejected. Late responses will not be accepted under any circumstances.
Yours sincerely
Bob Gann
Head of Strategy & Engagement
44
1. Purpose of this document
NHS Choices (www.nhs.uk) is a provider of digital services supporting health,
wellbeing and patient choices. In order to stay at the forefront of delivering high
quality services, it wishes to undertake a rapid review of the evidence on the
effectiveness of digital services similar to, and including, NHS Choices. We are
interested in wider overviews of the impact of digital health information services,
but are seeking a particular focus on five key areas: improvement of access to
care and supporting choice; improving health literacy and enabling healthy living;
increasing the efficiency of GP consultations; helping those with Long Term
Conditions better manage their care; and increasing the uptake of preventative
services such as screening and vaccination.
2. Background
NHS Choices must provide benefits to its users, otherwise people will not engage
with the service and investment cannot be justified. NHS Choices has established a
benefits realisation and management function with the objective of identifying,
measuring, and reporting on the delivery of the service’s benefits.
Having identified and prioritised a list of 21 benefit opportunities that are of
strategic importance to the Department of Health (DH) and other key
stakeholders, in November 2008 the NHS Choices Intelligent Client Function (part
of the DH and the commissioner of the service) benefits team began the exercise
of measuring and reporting on the delivery of the identified benefits. This was
achieved by establishing a list of over 90 metrics for the shortlist of 21 benefits.
Many of these metrics can be captured from routine data gathering and existing or
scheduled research; however it was found that where gaps existed, further
evidence was required in order to ensure reporting would be comprehensive for all
benefits of the service.
As per Section 1, the intention of this literature review is to commission research
for a broad sector scan of how similar digital services to the NHS Choices
programme around the world have been successful or not in achieving the benefits
NHS Choices was set up to achieve.
3. Areas of study to be covered in the literature review
This section sets out the areas of study NHS Choices seeks greater understanding
of the existing evidence and literature.
45
In the majority of instances the missing data described in Section 2 above
pertained to understanding behaviour, and the ability of digital services and
interventions to affect genuine behavioural change. This takes the analysis past
simple transactional information, i.e. how many people visit an area on the site,
through to the deeper understanding of how they felt about their experience and
most critically whether it lead to an alternative outcome, i.e. a person visiting their
GP instead of A&E or adopting a lifestyle change.
The benefit areas that this deeper and broader understanding can be grouped into
five broad categories and are summarised as follows:

Accessing care: understanding if people are better able to find the
appropriate provider, whether the user enabled to make a holistic choice from
treatment options through to finding their preferred provider and then being
supported in making the necessary appointments, and understanding if this
results in people receiving their care in the preferred setting.

Health living: are users of digital health information services becoming more
health literate, do they understand risks to health and healthy lifestyle
options, and do they use this information to inform healthy living choices.

Quality of primary care consultations: are people going to the GP for the
right reasons, are they more prepared when they go to the GP.

Long term conditions: do people who use digital health information services
and live with a long term condition have better understanding of care options
available and ultimately better outcomes.

Preventative services: exploring the impact of digital health information
services on uptake of screening and vaccine services.
4. Evaluation Objectives
The literature review should be a comprehensive, evidence based, horizon scan
focusing particularly on the 5 strategic areas. We are not seeking a formal
systematic review, but rather a rapid literature review. This should focus primarily
on published literature and websites but where possible it would be helpful to
include relevant unpublished sources and work in progress.
It is hoped that the literature review will provide evidence to show how services
such as NHS Choices have or have not achieved the following objectives, and to
enable NHS Choices to make informed research decisions on the design and
methodology of future studies. The research should primarily focus on answering
the objectives below, but any areas of related or significant benefit pertaining to
the use of digital health services which are similar to NHS Choices should also be
explored.
Accessing care and Patient Choice

To evaluate the extent digital health services are enabling informed patient
choice of treatment option.

To evaluate the extent digital health services are enabling patient choice in
46
finding and selecting providers.

To evaluate if scorecard functionality is effective at influencing a patient’s
choice of providers.

How effective digital health services are at providing an end to end service,
e.g. all the way through to accessing Choose & Book.

To determine if digital health services are helping patients achieve care in
their preferred setting (home/self care vs visiting a GP).
Healthy Living

To understand if people using digital health services have an improved
understanding of important health determinants such as calorific intake, fat,
alcohol units, smoking, five-a-day, etc. is improved.

To investigate if improved health outcomes are occurring as a result of digital
health services usage.

To evaluate if digital health services are bringing about healthy lifestyle
changes.

To examine if digital health services are successful in reaching the C2DE
population and reducing the health inequality gap between socioeconomic
groups.
Quality of primary care consultations

To determine the effect of digital health services on the efficiency of GP
consultations.

To determine the effect of digital health services on appropriateness of GP
consultations, and whether inappropriate consultations are being avoided.
 To evaluate the uptake and effectiveness of Information Prescriptions. (A GP or
carer assembled package of digital information pertaining to the best treatment
and management of a specific condition)
Long term conditions
 To understand if users with LTCs using digital health services achieve better
health outcomes that those who do not.
 To evaluate if digital health services are helping users with LTCs better manage
their care.
Preventative services

To explore the effect digital health services have in take up of screening
services.

To explore the effect digital health services have in take up of vaccine
services.
Additional objectives: review of methodologies
This literature review is intended to form the basis and background to inform the
design and methodology of further research to evaluate the benefits of NHS
Choices. In addition to a review of evidence supporting the above 5 strategic areas
47
in similar digital health services around the world, an additional section is
requested to give an overview of methodologies which have been used to evaluate
digital health services.
5. Time frame and evaluation criteria
The DH requests that responses to this ITT be returned no later than 9am on the
10th of July, by email to [email protected].
Hard copies will not be accepted and will result in offers being rejected. Late
responses will not be accepted under any circumstances.
The DH requests that the review be completed within 4 weeks of contract award.
The literature review and associated analysis should be presented as a written
report.
Responses will be evaluated on the basis of cost, time, proposed research
methodology, prior academic research experience and reputation.
Contract award, and execution will be subject to DH terms and conditions.
Responses are not to exceed 10,000 GBP.
48
References
1. Nguyen HQ, Carrieri-Kohlman V, Rankin SH, Slaughter R, Stulbarg MS. Internet-based patient
education and support interventions: a review of evaluation studies and directions for future
research. Computers in Biology and Medicine 2004;34:95-112.
2. Gustafson DH, Robinson TN, Ansley D, Adler L, Brennan PF. Consumers and evaluation of
interactive health communication applications. American Journal of Preventive Medicine
1999;16(1):23-29.
3. Smaglik P, Hawkins RP, Pingree S, Gustafson DH, Boberg E, Bricker E. The quality of interactive
computer use among HIV-infected individuals. Journal of Health Communication
1998;3(1):53-68.
4. Chess - an Interactive Computer-System for Women with Breast-Cancer Piloted with an underServed Population. 18th Annual Symposium on Computer Applications in Medical Care Transforming Information, Changing Health Care; 1994 Nov 05-09; Washington, Dc.
5. The Use and Impact of a Computer-Based Support System for People Living with Aids and HivInfection. 18th Annual Symposium on Computer Applications in Medical Care - Transforming
Information, Changing Health Care; 1994 Nov 05-09; Washington, Dc.
6. Lu HP, Gustafson DH. An Empirical-Study of Perceived Usefulness and Perceived Ease of Use on
Computerized Support System Use over Time. International Journal of Information
Management 1994;14(5):317-329.
7. Gustafson DH. What research tells us about the acceptance, use and impact of interactive health
communication systems. Psychosomatic Medicine 2000;62(1):1482.
8. Moore SM, Brennan PF, O'Brien RG, Visovsky C, Bjournsdottir G. Customized computer home
support improves recovery of CABG patients. Circulation 2001;104(17):2523.
9. Brennan PF, Moore SM, Bjornsdottir G, Jones J, Visovsky C, Rogers M. HeartCare: an Internetbased information and support system for patient home recovery after coronary artery
bypass graft (CABG) surgery. Journal of Advanced Nursing 2001;35(5):699-708.
10. Brennan PF, Caldwell B, Moore SM, Sreenath N, Jones J. Designing HeartCare: Custom
computerized home care for patients recovering from CABG surgery. Journal of the
American Medical Informatics Association 1998:381-385.
11. Gerdin U, Tallberg M, Wainwright P, editors. The ComputerLink projects: a decade of experience.
6th Triennial International Congress of IMIA-NI on Nursing Informatics - The Impact of
Nursing Knowledge on Health Care Informatics (NI 97); 1997 1997; Stockholm, Sweden.
12. Brennan PF, Moore SM, Smyth KA. The Effects of a Special Computer Network on Caregivers of
Persons with Alzheimers-Disease. Nursing Research 1995;44(3):166-172.
13. Brennan PF. Characterizing the Use of Health-Care Services Delivered Via Computer-Networks.
Journal of the American Medical Informatics Association 1995;2(3):160-168.
14. Frisse ME, editor. Computer-Networks Promote Caregiving Collaboration - the Computerlink
Project. 16th Annual Symp on Computer Applications in Medical Care; 1992 Nov 08-11;
Baltimore, Md.
15. Car J, Black A, Anandan C, Cresswell K, Pagliari C, McKinstry B, et al. The Impact of eHealth on the
Quality & Safety of Healthcare: A systematic overview & synthesis of the literature. .
Birmingham: NHS Connecting for Health Evaluation Programme, 2008.
16. Jones R. The Role of Health Kiosks in 2009: Literature and Informant Review. International
Journal of Environmental Research and Public Health 2009;6(6):1818-1855.
17. Costs And Benefits Of Health Information Technology: New Trends From The Literature. 15th
Princeton Conference on Can Payment and Other Innovations Improve the Quality and Value
of Health Care; 2008 May 27-28; Princeton, NJ. Project Hope.
18. Tang KC, Choi BCK, Beaglehole R. Grading of evidence of the effectiveness of health promotion
interventions. Journal of Epidemiology and Community Health 2008;62(9):832-834.
49
19. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence
and strength of recommendations. British Medical Journal 2004;328(7454):1490-1494.
20. Brownson RC, Fielding JE, Maylahn CA. Evidence-Based Public Health: A Fundamental Concept
for Public Health Practice. Annual Review of Public Health 2009;30:175-201.
21. Jones RB, Gann B, Murphy J, Duman M. Consumer Health Informatics. In: Hayes G, Barnett D,
editors. BCS History of Health Informatics British Computer Society, 2008.
22. O'Connor AM, Rostom A, Fiset V, Tetroe J, Entwistle V, Llewellyn-Thomas H, et al. Decision aids
for patients facing health treatment or screening decisions: systematic review. British
Medical Journal 1999;319(7212):731-734.
23. O'Connor AM, Bennett CL, Stacey D, Barry M, Col NF, Eden KB, et al. Decision aids for people
facing health treatment or screening decisions. Cochrane Database of Systematic Reviews
2009(3):119.
24. Morris D, Drakea E, Saarimakia A, Bennetta C, O'Connora A. Can people find patient decision aids
on the Internet? Patient Education and Counseling 2008;73(3):557-560.
25. Jibaja-Weiss ML, Volk RJ. Utilizing computerized entertainment education in the development of
decision aids for lower literate and naive computer users. Journal of Health Communication
2007;12(7):681-697.
26. Murray E, Lo B, Pollack L, Donelan K, Catania J, Lee K, et al. The impact of health information on
the Internet on health care and the physician-patient relationship: national U.S. survey
among 1.050 U.S. physicians. Journal Of Medical Internet Research 2003;5(3):e17-e17.
27. Ahluwahlia S, Murray E, Stevenson F, Kerr C, Burns J. “A heartbeat moment”: Qualitative study of
GP views of patients bringing health information from the internet to a consultation. British
Journal of General Practice 2010.
28. Knaevelsrud C, Maercker A. Internet-based treatment for PTSD reduces distress and facilitates
the development of a strong therapeutic alliance: a randomized controlled clinical trial. BMC
Psychiatry 2007;7(1):13.
29. Wald HS, Dube CE, Anthony DC. Untangling the Web--The impact of Internet use on health care
and the physician-patient relationship. Patient Education and Counseling 2007;68(3):218224.
30. Hardey M. Doctor in the house: the Internet as a source of lay health knowledge and the
challenge to expertise. Sociology of Health and Illness 1999;21(6):820-835.
31. Murray E, Lo B, Pollack L, Donelan K, Catania J, White M, et al. The Impact of Health Information
on the Internet on the Physician-Patient Relationship. Patient perceptions. Archives of
Internal Medicine 2003;163:1727-1734.
32. Stevenson FA, Kerr C, Murray E, Nazareth I. Information from the Internet and the doctor-patient
relationship: the patient perspective - a qualitative study. BMC Family Practice 2007;8(47).
33. Broom A. Virtually He@lthy: The Impact of Internet Use on Disease Experience and the DoctorPatient Relationship. Qualitative Health Research 2005;15:325-344.
34. Edwards M, Davies M, Edwards A. What are the external influences on information exchange and
shared decision-making in healthcare consultations: A meta-synthesis of the literature.
Patient Education and Counseling 2009;75(1):37-52.
35. Hay MC, Strathmann C, Lieber E, Wick K, Giesser B. Why Patients Go Online Multiple Sclerosis,
the Internet, and Physician-Patient Communication. Neurologist 2008;14(6):374-381.
36. van Uden-Kraan CF, Drossaert CHC, Taal E, Seydel ER, van de Laar M. Self-Reported Differences in
Empowerment Between Lurkers and Posters in Online Patient Support Groups. Journal of
Medical Internet Research 2008;10(2):9.
37. van Uden-Kraan CF, Drossaert CHC, Taal E, Shaw B, Seydel ER, van de Laar MAFJ. Empowering
processes and outcomes of participation in online support groups for patients with breast
cancer, arthritis, or fibromyalgia. Qualitative Health Research 2008;18:405-417.
50
38. Lemire M, Sicotte C, Pare G. Internet use and the logics of personal empowerment in health.
Health Policy 2008;88(1):130-140.
39. van Uden-Kraan CF, Drossaert CHC, Taal E, Seydel ER, van de Laar MAFJ. Participation in online
patient support groups endorses patients' empowerment. Patient Education and Counseling
2009;74:61-69.
40. Salmon P, Hall G. Patient empowerment or the emperor's new clothes. Journal of the Royal
Society of Medicine 2004;97(2):53-56.
41. Harris R, Veinot T. The Empowerment Model and Using e-Health to Distribute Information.:
University of Western Ontario, 2004.
42. Wensing M. Evidence-based patient empowerment. Quality Health Care 2000;9(4):200-201.
43. Howie JG, Heaney DJ, Maxwell M, Walker JJ. A comparison of a Patient Enablement Instrument
(PEI) against two established satisfaction scales as an outcome measure of primary care
consultations, 1998:165-171.
44. Leveille SG, Huang A, Tsai SB, Allen M, Weingart SN, Iezzoni LT. Health Coaching via an Internet
Portal for Primary Care Patients With Chronic Conditions A Randomized Controlled Trial.
Medical Care 2009;47(1):41-47.
45. Goessens BMB, Visseren FLJ, de Nooijer J, van den Borne HW, Algra A, Wierdsma J, et al. A pilotstudy to identify the feasibility of an Internet-based coaching programme for changing the
vascular risk profile of high-risk patients. Patient Education and Counseling 2008;73(1):6772.
46. O'Connor AM, Stacey D, Legare F. Coaching to support patients in making decisions. British
Medical Journal 2008;336(7638):228-229.
47. Weaver JB, Thompson NJ, Weaver SS, Hopkins GL. Healthcare non-adherence decisions and
internet health information. Computers in Human Behaviour 2009;In press.
48. Nicholas D, Huntington P, Williams P, Blackburn P. Digital health information provision and
health outcomes. Journal of Information Science 2001;27(4):265-276.
49. Fung CH, Lim Y-W, Mattke S, Damberg C, Shekelle PG. Systematic Review: The Evidence That
Publishing Patient Care Performance Data Improves Quality of Care. Annals of Internal
Medicine 2008;148(2):111-123.
50. Faber M, Bosch M, Wollersheim H, Leatherman S, Grol R. Public Reporting in Health Care: How
Do Consumers Use Quality-of-Care Information? A Systematic Review. Medical Care
2009;47(1):1-8.
51. Tu HT, Lauer J. Word of Mouth and Physician Referrals Still Drive Health Care Provider Choice,
2008.
52. Greene J, Peters E, Mertz CK, Hibbard JH. Comprehension and choice of a consumer-directed
health plan: An experimental study. American Journal of Managed Care 2008;14(6):369-376.
53. Hibbard JH, Peters E, Dixon A, Tusler M. Consumer competencies and the use of comparative
quality information - It isn't just about literacy. Medical Care Research and Review
2007;64(4):379-394.
54. Peters E, Dieckmann N, Dixon A, Hibbard JH, Mertz CK. Less is more in presenting quality
information to consumers. Medical Care Research and Review 2007;64(2):169-190.
55. Peters E, Hibbard J, Slovic P, Dieckmann N. Numeracy skill and the communication,
comprehension, and use of risk-benefit information. Health Affairs 2007;26(3):741-748.
56. Lubalin JS, Harris-Kojetin LD. What do consumers want and need to know in making health care
choices? Medical Care Research and Review 1999;56:67-102.
57. Ranganathan M, Hibbard J, Rodday AMC, de Brantes F, Conroy K, Rogers WH, et al. Motivating
Public Use of Physician-Level Performance Data An Experiment on the Effects of Message
and Mode. Medical Care Research and Review 2009;66(1):68-81.
58. Dixon S. Report on the National Patient Choice Survey - September 2008 England. In: Health Do,
editor: Department of Health, 2009.
51
59. Department of Health and Ipsos MORI. Social Research Institute. The GP patient survey 2008/09 :
national report for Department of Health. London, 2009.
60. Henderson S, Robertson R, Dixon A. Are patients choosing? British Journal of Healthcare
Management 2009;15(2):77-80.
61. Robertson R, Dixon S. Choice at the point of referral: Early result of a patient survey. London:
King's Fund, 2009.
62. Magee H, Davis LJ, Coulter A. Public views on healthcare performance indicators and patient
choice. Journal of the Royal Society of Medicine 2003;96(7):338-342.
63. Fotaki M, Roland M, Boyd A, McDonald R, Scheaff R, Smith L. What benefits will choice bring to
patients? Literature review and assessment of implications. Journal of Health Services
Research & Policy 2008;13(3):178-184.
64. Lorden A, Coustasse A, Singh KP. The balanced scorecard framework - A case study of patient and
employee satisfaction: What happens when it does not work as planned? Health Care
Management Review 2008;33(2):145-155.
65. British Medical Association. Choose and book : learning lessons from local experience.: British
Medical Association, 2009.
66. DelliFraine JL, Dansky K. Home-based telehealth: a review and meta-analysis. Journal of
Telemedicine and Telecare 2008;14:62-66.
67. Bessell TL, McDonald S, Silagy CA, Anderson JN, Hiller JE, Sansom LN. Do Internet interventions
for consumers cause more harm than good? A systematic review. Health Expectations
2002;5:28-37.
68. Portnoy DB, Scott-Sheldon LAJ, Johnson BT, Carey MP. Computer-delivered interventions for
health promotion and behavioral risk reduction: a meta-analysis of 75 randomized
controlled trials, 1988-2007. Preventive Medicine 2008;47(1):3-16.
69. Weinstein PK. A review of weight loss programs delivered via the internet. Journal of
Cardiovascular Nursing 2006;21(4):251-258.
70. Eysenbach G. Journal of Medical Internet Research. Volume 10, Issue. Theme issue on Web
Assisted Tobacco Interventions. In: Eysenbach G, editor. Toronto, 2008.
71. An LC, Schillo BA, Saul JE, Wendling AH, Klatt CM, Berg CJ, et al. Utilization of smoking cessation
informational, interactive, and online community resources as predictors of abstinence:
cohort study. Journal Of Medical Internet Research 2008;10(5):e55-e55.
72. Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized
coaching program for smoking cessation. Journal Of Medical Internet Research
2008;10(5):e54-e54.
73. Bock BC, Graham AL, Whiteley JA, Stoddard JL. A review of web-assisted tobacco interventions
(WATIs). Journal Of Medical Internet Research 2008;10(5):e39-e39.
74. Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through internet
and cell phone without nicotine replacement (happy ending): randomized controlled trial.
Journal Of Medical Internet Research 2008;10(5):e51-e51.
75. Cunningham JA. Access and interest: two important issues in considering the feasibility of webassisted tobacco interventions. Journal Of Medical Internet Research 2008;10(5):e37-e37.
76. Finkelstein J, Lapshin O, Cha E. Feasibility of promoting smoking cessation among methadone
users using multimedia computer-assisted education. Journal Of Medical Internet Research
2008;10(5):e33-e33.
77. Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool:
comparison of different media campaigns to increase demand for smoking cessation
interventions. Journal Of Medical Internet Research 2008;10(5):e50-e50.
78. Houston TK, Richman JS, Ray MN, Allison JJ, Gilbert GH, Shewchuk RM, et al. Internet delivered
support for tobacco control in dental practice: randomized controlled trial. Journal Of
Medical Internet Research 2008;10(5):e38-e38.
52
79. McKay HG, Danaher BG, Seeley JR, Lichtenstein E, Gau JM. Comparing two web-based smoking
cessation programs: randomized controlled trial. Journal Of Medical Internet Research
2008;10(5):e40-e40.
80. Norman CD, McIntosh S, Selby P, Eysenbach G. Web-assisted tobacco interventions: empowering
change in the global fight for the public's (e)Health. Journal Of Medical Internet Research
2008;10(5):e48-e48.
81. Rabius V, Pike KJ, Wiatrek D, McAlister AL. Comparing internet assistance for smoking cessation:
13-month follow-up of a six-arm randomized controlled trial. Journal Of Medical Internet
Research 2008;10(5):e45-e45.
82. Stoddard JL, Augustson EM, Moser RP. Effect of adding a virtual community (bulletin board) to
smokefree.gov: randomized controlled trial. Journal Of Medical Internet Research
2008;10(5):e53-e53.
83. Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, et al. The role of
engagement in a tailored web-based smoking cessation program: randomized controlled
trial. Journal Of Medical Internet Research 2008;10(5):e36-e36.
84. Lustria MLA, Cortese J, Noar SM, Glueckauf RL. Computer-tailored health interventions delivered
over the web: Review and analysis of key components. Patient Education and Counseling
2009(74):156-173.
85. Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Webbased vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes.
Journal Of Medical Internet Research 2004;6(4):e40-e40.
86. Harvey-Berino J, Pintauro S, Buzzell P, DiGiulio M, Gold BC, Moldovan C, et al. Does using the
Internet facilitate the maintenance of weight loss? International Journal of Obesity
2002;26(9):1254-1260.
87. Marshall AL, Leslie ER, Bauman AE, Marcus BH, Owen N. Print versus website physical activity
programs - A randomized trial. American Journal of Preventive Medicine 2003;25(2):88-94.
88. Eysenbach G. Issues in evaluating health websites in an Internet-based randomized controlled
trial. Journal of Medical Internet Research, 2002:e17.
89. Walters ST, Wright JA, Shegog R. A review of computer and internet-based interventions for
smoking behavior. . Addictive Behaviors 2006;31(2):264-277.
90. Swartz LHG, Noell JW, Schroeder SW, Ary DV. A randomised control study of a fully automated
internet based smoking cessation programme. Tobacco Control 2006;15:7-12.
91. Zbikowski SM, Hapgood J, Smucker Barnwell S, McAfee T. Phone and web-based tobacco
cessation treatment: real-world utilization patterns and outcomes for 11,000 tobacco users.
Journal Of Medical Internet Research 2008;10(5):e41-e41.
92. Bewick BM, Trusler K, Barkham M, Hill JA, Cahill J, Mulhern B. The effectiveness of web-based
interventions designed to decrease alcohol consumption - A systematic review. Preventive
Medicine 2008;47(1):17-26.
93. Kypri K, Saunders JB, Williams SM, McGee RO, Langley JD, Cashell-Smith ML, et al. Web-based
screening and brief intervention for hazardous drinking: a double-blind randomized
controlled trial. Addiction 2004;99(11):1410-1417.
94. Kypri K, McAnally HM. Randomized controlled trial of a web-based primary care intervention for
multiple health risk behaviors. Preventive Medicine 2005;41(3-4):761-766.
95. Chiauzzi E, Green TC, Lord S, Thum C, Goldstein M. My student body: A high-risk drinking
prevention Web site for college students. Journal of American College Health
2005;53(6):263-274.
96. Moore MJ, Soderquist J, Werch C. Feasibility and efficacy of a binge drinking prevention
intervention for college students delivered via the Internet versus postal mail. Journal of
American College Health 2005;54(1):38-44.
53
97. Cunningham JA, Blomqvist J, Koski-Jannes A, Cordingley J. Current heavy drinkers' reasons for
considering change: Results from a natural history general population survey. Addictive
Behaviors 2005;30(3):581-584.
98. Bewick BM, Trusler K, Mulhern B, Barkham M, Hill AJ. The feasibility and effectiveness of a webbased personalised feedback and social norms alcohol intervention in UK university
students: A randomised control trial. Addictive Behaviors 2008;33(9):1192-1198.
99. Riper H, Kramer J, Smit F, Conijn B, Schippers G, Cuijpers P. Web-based self-help for problem
drinkers: a pragmatic randomized trial. Addiction 2007;103:218-227.
100. Linke S, Murray E, Butler C, Wallace P. Internet-Based Interactive Health Intervention for the
Promotion of Sensible Drinking: Patterns of Use and Potential Impact on Members of the
General Public. Journal of Medical Internet Research 2007;9(2):e10.
101. Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, et al. Methodological
Challenges in Online Trials. J Med Internet Res 2009;11(1):e9.
102. Murray E, McCambridge J, Khadjesari Z, White IR, Thompson SG, Godfrey C, et al. The DYD-RCT
protocol: an on-line randomised controlled trial of an interactive computer-based
intervention compared with a standard information website to reduce alcohol consumption
among hazardous drinkers. Bmc Public Health 2007;7:9.
103. Riper H, van Straten A, Keuken M, Smit F, Schippers G, Cuijpers P. Curbing Problem Drinking
with Personalized-Feedback Interventions A Meta-Analysis. American Journal of Preventive
Medicine 2009;36(3):247-255.
104. Linke S, McCambridge J, Khadjesari Z, Wallace P, Murray E. Development of a Psychologically
Enhanced Interactive Online Intervention for Hazardous Drinking. Alcohol and Alcoholism
2008;43(6):669-674.
105. Wallace P, Linke S, Murray E, McCambridge J, Thompson S. A randomized controlled trial of an
interactive Web-based intervention for reducing alcohol consumption. Journal of
Telemedicine and Telecare 2006;12:52-54.
106. Walters ST, Miller E, Chiauzzi E. Wired for wellness: e-interventions for addressing college
drinking. Journal of Substance Abuse Treatment 2005;29(2):139-145.
107. Chiauzzi E, Brevard J, Thum C, Decembrele S, Lord S. My StudentBody Stress: An online stress
management intervention for college students (vol 13, pg 555, 2008). Journal of Health
Communication 2008;13(8):827-827.
108. Chiauzzi E, Brevard J, Thurn C, Decembrele S, Lord S. MyStudentBody-Stress: An online stress
management intervention for college students. Journal of Health Communication
2008;13(6):555-572.
109. Chiauzzi E, Green TC, Lord S, Thum C, Goldstein M. My student body: A high-risk drinking
prevention Web site for college students (vol 53, pg 263, 2005). Journal of American College
Health 2006;54(6):371-371.
110. Kypri K, Hallett J, Howat P, McManus A, Maycock B, Bowe S, et al. Randomized Controlled Trial
of Proactive Web-Based Alcohol Screening and Brief Intervention for University Students.
Archives of Internal Medicine 2009;169(16):1508-1514.
111. Kypri K, Langley JD, Saunders JB, Cashell-Smith ML, Herbison P. Randomized controlled trial of
Web-based alcohol screening and brief intervention in primary care. Archives of Internal
Medicine 2008;168(5):530-536.
112. Riper H, Kramer J, Conijn B, Smit F, Schippers G, Cuijpers P. Translating Effective Web-Based
Self-Help for Problem Drinking Into the Real World. Alcoholism-Clinical and Experimental
Research 2009;33(8):1401-1408.
113. Riper H, Kramer J, Keuken M, Smit F, Schippers G, Cuijpers P. Predicting Successful Treatment
Outcome of Web-Based Self-help for Problem Drinkers: Secondary Analysis From a
Randomized Controlled Trial. Journal of Medical Internet Research 2008;10(4).
54
114. Kramer J, Riper H, Lemmers L, Conijn B, van Straten A, Smit F. Television-supported self-help for
problem drinkers: A randomized pragmatic trial. Addictive Behaviors 2009;34(5):451-457.
115. Neville LM, Milat AJ, O'Hara B. Computer-tailored weight reduction interventions targeting
adults: a narrative systematic review. Health Promotion Journal of Australia 2009;20(1):4857.
116. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A Review of eHealth
Interventions for Physical Activity and Dietary Behavior Change. American Journal of
Preventive Medicine 2007;33(4):336-345.e16.
117. Kroeze W, Werkman AM, Brug J. A systematic review of randomized trials on the effectiveness
of computer-tailored education on physical activity and dietary behaviors. Annals of
Behavioral Medicine 2006;31(3):205-223.
118. Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A Review of eHealth
Interventions for Physical Activity and Dietary Behavior Change. American Journal of
Preventive Medicine 2007;33(4):336-345.
119. Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Web-site delivered physical activity
interventions: A review of the literature. American Journal of Preventive Medicine
2007;33(1):54-64.
120. van den Berg MH, Schoones JW, Vlieland TPM. Internet-based physical activity interventions: A
systematic review of the literature. Journal of Medical Internet Research 2007;9(3):e26.
121. Eyles HC, Mhurchu CN. Does tailoring make a difference? A systematic review of the long-term
effectiveness of tailored nutrition education for adults. Nutrition Reviews 2009;67(8):464480.
122. Marcus BH, Ciccolo JT, Sciamanna CN. Using electronic/computer interventions to promote
physical activity. British Journal of Sports Medicine 2009;43:102-105.
123. Neve M, Morgan PJ, Jones PR, Collins CE. Effectiveness of web-based interventions in achieving
weight loss and weight loss maintenance in overweight and obese adults: a systematic
review with meta-analysis. Obesity Reviews 2009;9999(9999).
124. Marcus BH, Lewis BA, Williams DM, Dunsiger S, Jakicic JM, Whiteley JA, et al. A Comparison of
Internet and Print-Based Physical Activity Interventions. Archives of Internal Medicine
2007;167:944-949.
125. Hunter CM, Peterson AL, Alvarez LM, Poston WC, Brundige AR, Haddock K, et al. Weight
management using the Internet - A randomized controlled trial. American Journal of
Preventive Medicine 2008;34(2):119-126.
126. van Wier MF, Ariens GAM, Dekkers JC, Hendriksen IJM, Smid T, van Mechelen W. Phone and email counselling are effective for weight management in an overweight working population:
a randomized controlled trial. Bmc Public Health 2009;9:12.
127. Ferney SL, Marshall AL, Eakin EG, Owen N. Randomized trial of a neighborhood environmentfocused physical activity website intervention. Preventive Medicine 2009;48(2):144-150.
128. Bennett GG, Herring SJ, Puleo E, Stein EK, Emmons KM, Gillman MW. Web-based Weight Loss in
Primary Care: A Randomized Controlled Trial Obesity 2009.
129. Kirk S, Harvey E, McConnon A, Pollard J, Greenwood D, Thomas J, et al. A randomised trial of an
internet weight control resource: The UK Weight Control Trial [ISRCTN58621669], 2003:19.
130. Tsai AG, Wadden TA. Systematic review: an evaluation of major commercial weight loss
programs in the United States. Annals Of Internal Medicine 2005;142(1):56-66.
131. Carter-Edwards L, Bastian LA, Schultz M, Amamoo MA, Østbye T. An Internet-based weight loss
intervention initiated by a newspaper. Prev Chronic Dis 2009;6(3).
132. Noar SM, Black HG, Pierce LB. Efficacy of computer technology-based HIV prevention
interventions: a meta-analysis. AIDS 2009;23(1):107-115.
133. Bull S, Pratte K, Whitesell N, Rietmeijer C, McFarlane M. Effects of an Internet-Based
Intervention for HIV Prevention: The Youthnet Trials. Aids and Behavior 2009;13(3):474-487.
55
134. O'Brien MA, Whelan TJ, Villasis-Keever M, Gafni A, Charles C, Roberts R, et al. Are CancerRelated Decision Aids Effective? A Systematic Review and Meta-Analysis. Journal of Clinical
Oncology 2009;27(6):974-985.
135. Chapple A, Ziebland S, Hewitson P, McPherson A. What affects the uptake of screening for
bowel cancer using a faecal occult blood test (FOBt): A qualitative study. Social Science &
Medicine 2008;66(12):2425-2435.
136. Volk RJ, Hawley ST, Kneuper S, Holden W, Stroud LA, Cooper CP, et al. Trials of decision aids for
prostate cancer screening - A systematic review. American Journal of Preventive Medicine
2007;33(5):428-434.
137. Ellison GL, Weinrich SP, Lou M, Xu H, Powell IJ, Baquet CR. A Randomized Trial Comparing WebBased Decision Aids on Prostate Cancer Knowledge for African-American Men. Journal of the
National Medical Association 2008;100(10):1139-1145.
138. Ilic D, Egberts K, McKenzie JE, Risbridger G, Green S. Informing men about prostate cancer
screening: A randomized controlled trial of patient education materials. 2nd Asia/Pacific
Forum on Andrology. Shanghai, PEOPLES R CHINA: Springer, 2006:466-471.
139. Dey A, Reid B, Godding R, Campbell A. Perceptions and behaviour of access of the Internet: A
study of women attending a breast screening service in Sydney, Australia. International
Journal of Medical Informatics 2008;77(1):24-32.
140. Kreuter MW, Black WJ, Friend L, Booker AC, Klump P, Bobra S, et al. Use of computer kiosks for
breast cancer education in five community settings. Health Education & Behavior
2006;33(5):625-642.
141. Rice RE. Influences, usage, and outcomes of Internet health information searching: Multivariate
results from the Pew surveys. International Journal of Medical Informatics 2006;75:8-28.
142. Dolan G, Iredale R, Williams R, Ameen J. Consumer use of the internet for health information: a
survey of primary care patients. International Journal of Consumer Studies 2004;28(2):147153.
143. Fox S, Rainie L. The Online Health Care Revolution, Pew Internet and American Life Project.
Washington, 2002.
144. Fox S, Fallows D. Internet Health Resources, Pew Internet and American Life Project.
Washington DC, 2003.
145. Wise M, Han JY, Shaw B, McTavish F, Gustafson DH. Effects of using online narrative and
didactic information on healthcare participation for breast cancer patients. Patient
Education and Counseling 2008;70(3):348-356.
146. Barnoy S, Volfin-Pruss D, Ehrenfeld M, Kushnir T. Factors affecting nurses' attitudes in Israel
toward patients who present them with Internet medical Information. Nursing Outlook
2008;56(6):314-321.
147. Jones RB, Hampshire AJ, Tweddle S, Moult B, Hill A. The clinician's role in meeting patient
information needs: suggested learning outcomes. Medical Education 2001;35(6):565-571.
148. Slack WV, Hicks GP, Reed CE, Van Cura LJ. A computer-based medical history system. N. Engl. J.
Med. 1966;274:194-198.
149. Jones RB, Knill-Jones RP. Electronic Patient Record Project: Direct Patient Input to the Record.
Report for the Strategy Division of the Information Management Group of the NHS
Management Executive: University of Glasgow, 1994.
150. Bachman JW. The patient-computer interview: A neglected tool that can aid the clinician. Mayo
Clinic Proceedings 2003;78(1):67-78.
151. Slack WV. Cybermedicine for the patient. American Journal of Preventive Medicine
2007;32(5):S135-S136.
152. Radley SC, Jones GL, Tanguy EA, Stevens VG, Nelson C, Mathers NJ. Computer interviewing in
urogynaecology: concept, development and psychometric testing of an electronic pelvic
56
floor assessment questionnaire in primary and secondary care. Bjog-an International Journal
of Obstetrics and Gynaecology 2006;113(2):231-238.
153. Zakim D, Braun N, Fritz P, Alscher MD. Underutilization of information and knowledge in
everyday medical practice: Evaluation of a computer-based solution. Bmc Medical
Informatics and Decision Making 2008;8:12.
154. Benaroia M, Elinson R, Zarnke K. Patient-directed intelligent an interactive computer medical
history-gathering systems: A utility and feasibility study in the emergency department.
International Journal of Medical Informatics 2007;76(4):283-288.
155. Wenner AR, Ferrante M, Belser D. Instant Medical History. Journal of the American Medical
Informatics Association 1994:1036-1036.
156. Wenner AR. Numbers of centres using IMH. In: Jones RB, editor. Plymouth, 2009.
157. Dugaw JE, Civello K, Chuinard C, Jones GN. Will patients use a computer to give a medical
history? Journal of Family Practice 2000;49(10):921-923.
158. Pierce B. The use of instant medical history in a rural clinic: case study of the use of computers
in an Arkansas physician’s office. J Ark Med Soc. 2000;96:444-447.
159. Wenner AR. Patient Entered Data Using Instant Medical History: University of Plymouth,
2008:Recorded webinar by Allen Webinar.
160. McPherson CJ, Higginson IJ, Hearn J. Effective methods of giving information in cancer: a
systematic literature review of randomized controlled trials. Journal of Public Health
Medicine 2001;23(3):227-234.
161. Eysenbach G. The Impact of the Internet on Cancer Outcomes. CA Cancer J Clin 2003;53(6):356371.
162. Gysels M, Higginson IJ. Interactive technologies and videotapes for patient education in cancer
care: systematic review and meta-analysis of randomised trials. Supportive Care in Cancer
2007;15(1):7-20.
163. Moeinedin F, Moineddin R, Jadad AR, Hamid JS, To T, Beyene J. Application of biomedical
informatics to chronic pediatric diseases: a systematic review. Bmc Medical Informatics and
Decision Making 2009;9:7.
164. Sanders DL, Aronsky D. Biomedical informatics applications for asthma care: A systematic
review. Journal of the American Medical Informatics Association 2006;13(4):418-427.
165. Chan DS, Callahan CW, Sheets SJ, Moreno CN, Malone FJ. An Internet-based store-and-forward
video home telehealth system for improving asthma outcomes in children. American Journal
of Health System Pharmacy 2003;60(19):1976-1981.
166. Shegog R, Bartholomew LK, Parcel GS, Sockrider MM, Masse L, Abramson SL. Impact of a
computer-assisted education program on factors related to asthma self-management
behavior. Journal of the American Medical Informatics Association 2001;8(1):49-61.
167. Bartholomew LK, Gold RS, Parcel GS, Czyzewski DI, Sockrider MM, Fernandez M, et al. Watch,
Discover, Think, and Act: evaluation of computer-assisted instruction to improve asthma
self-management in inner-city children. Patient Education and Counseling 2000;39(2-3):269280.
168. Huss K, Winkelstein M, Nanda J, Naumann PL, Sloand ED, Huss RW. Computer game for innercity children does not improve asthma outcomes. Journal of Pediatric Health Care
2003;17(2):72-78.
169. McPherson A, Forster D, Glazebrook C, Smyth A. The asthma files: evaluation of a multimedia
package for children's asthma education. Paediatric Nursing 2002;14:32-35.
170. McPherson AC, Glazebrook C, Forster D, James C, Smyth A. A randomized, controlled trial of an
interactive educational computer package for children with asthma. Pediatrics
2006;117(4):1046-1054.
57
171. Porter SC, Cai ZH, Gzibbons W, Goldmann DA, Kohane IS. The asthma kiosk: A patient-centered
technology for collaborative decision support in the emergency department. Journal of the
American Medical Informatics Association 2004;11(6):458-467.
172. Krishna S, Francisco BD, Balas EA, Konig P, Graff GR, Madsen RW. Internet-enabled interactive
multimedia asthma education program: A randomized trial. Pediatrics 2003;111(3):503-510.
173. Krishna S, Balas EA, Francisco BD, Konig P. Effective and sustainable multimedia education for
children with asthma: A randomized controlled trial. Childrens Health Care 2006;35(1):75-90.
174. Homer C, Susskind O, Alpert HR, Owusu C, Schneider L, Rappaport LA, et al. An evaluation of an
innovative multimedia educational software program for asthma management: Report of a
randomized, controlled trial. Pediatrics 2000;106(1):210-215.
175. van der Meer V, Bakker MJ, van den Hout WB, Rabe KF, Sterk PJ, Kievit J, et al. Internet-Based
Self-management Plus Education Compared With Usual Care in Asthma A Randomized Trial.
Annals of Internal Medicine 2009;151(2):110-W31.
176. Nguyen HQ, Donesky-Cuenco D, Wolpin S, Reinke LF, Benditt JO, Paul SM, et al. Randomized
Controlled Trial of an Internet-Based Versus Face-to-Face Dyspnea Self-Management
Program for Patients With Chronic Obstructive Pulmonary Disease: Pilot Study. Journal of
Medical Internet Research 2008;10(2):19.
177. Lee TI, Yeh YT, Liu CT, Chen PL. Development and evaluation of a patient-oriented education
system for diabetes management. International Journal of Medical Informatics
2007;76(9):655-663.
178. Ralston JD, Hirsch IB, Hoath J, Mullen M, Cheadle A, Goldberg HI. Web-Based Collaborative Care
for Type 2 Diabetes A pilot randomized trial. Diabetes Care 2009;32(2):234-239.
179. Azar M, Gabbay R. Web-based management of diabetes through glucose uploads: Has the time
come for telemedicine? Diabetes Research and Clinical Practice 2009;83(1):9-17.
180. Bond GE, Burr R, Wolf FM, Price M, McCurry SM, Teri L. The effects of a web-based intervention
on the physical outcomes associated with diabetes among adults age 60 and older: A
randomized trial. Diabetes Technology & Therapeutics 2007;9(1):52-59.
181. A systematic review of interactive computer-assisted technology in diabetes care. 64th Annual
Meeting of the American-Diabetes-Association; 2004 Jun 04-08; Orlando, FL. Blackwell
Publishing.
182. Li LC, Adam P, Townsend AF, Stacey D, Lacaille D, Cox S, et al. Improving healthcare consumer
effectiveness: An Animated, Self-serve, Web-based Research Tool (ANSWER) for people with
early rheumatoid arthritis. Bmc Medical Informatics and Decision Making 2009;9:6.
183. Smith J, Forster A, House A, Knapp P, Wright J, Young J. Information provision for stroke
patients and their caregivers. Cochrane Database of Systematic Reviews 2008(2):79.
184. Hoffmann T, McKenna K, Worrall L, Read SJ. Randomised trial of a computer-generated tailored
written education package for patients following stroke. Age and Ageing 2007;36:280-286.
185. Maasland E, Koudstaal PJ, Habbema JDF, Dippel DWJ. Effects of an individualized multimedia
computer program for health education in patients with a recent minor stroke or transient
ischemic attack - a randomized controlled trial. Acta Neurologica Scandinavica
2007;115(1):41-48.
186. Rochette A, Korner-Bitensky N, Tremblay V, Kloda L. Stroke rehabilitation information for clients
and families: Assessing the quality of the StrokEngine-Family website. Disability and
Rehabilitation 2008;30(19):1506-1512.
187. NICE. Computerised cognitive behaviour therapy for depression and anxiety: Technology
Appraisal 97
In: NICE, editor. London: Department of Health, 2004.
188. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the
internet: randomised controlled trial, 2004:265-.
189. Williams C. Living Life To The Full, 2009.
58
190. Spek V, Cuijpers PIM, Nykl, Iacute, Cek I, Riper H, et al. Internet-based cognitive behaviour
therapy for symptoms of depression and anxiety: a meta-analysis. Psychosomatic Medicine
2007;37(03):319-328.
191. Griffiths KM, Christensen H, Jorm AF, Evans K, Groves C. Effect of web-based depression literacy
and cognitive-behavioural therapy interventions on stigmatising attitudes to depression Randomised controlled trial. British Journal of Psychiatry 2004;185:342-349.
192. Christensen H. The prevention of depression using the Internet. MJA 2002;177:S122-S125.
193. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the
internet: randomised controlled trial. British Medical Journal 2004;328(7434):265-268A.
194. Christensen H, Griffiths MK, Farrer L. Adherence in Internet Interventions for Anxiety and
Depression: Systematic Review. J Med Internet Res 2009;11(2):e13.
195. Christensen H, Griffiths KM, MacKinnon AJ, Brittliffe K. Online randomized controlled trial of
brief and full cognitive behaviour therapy for depression. Psychological Medicine
2006;36(12):1737-1746.
196. Christensen H, Griffiths K. The prevention of depression using the Internet. Medical Journal of
Australia 2002.
197. Christensen H, Griffiths KM, Farrer L. Adherence in Internet Interventions for Anxiety and
Depression: Systematic Review. J Med Internet Res 2009;11(2):e13.
198. Griffiths K, Christensen H. Griffiths, K. M., & Christensen, H. (2006). Review of randomised
controlled trials of
Internet interventions for mental disorders and related conditions. Clinical
Psychologist, 10, 16–29. Clinical Psychologist 2006;10.:16-29.
199. Griffiths KM, Christensen H, Jorm AF, Evans K, Groves C. Effect of web-based depression literacy
and cognitive-behavioural therapy interventions on stigmatising attitudes to depression:
Randomised controlled trial, 2004:342-349.
200. Jones RB, Martinez R, Maramba I, Prestwich J, McCauley M, Farrand P, et al. Exploratory study
of chat room motivational support for users of an Internet-based cognitive behavioural
therapy for the treatment of depression. In: Kay S, editor. Healthcare computing 2008.
Weybridge, Surrey: BJHC Books, 2008.
201. Williams C, Wilson P, Walker A, Wallace I, Morrison J, Whitfield G, et al. An evaluation of the
effectiveness of structured cognitive behaviour therapy self-help materials delivered by a
self-help support worker within primary care.Report submitted to the Chief Scientific Officer
in Scotland (contract number CZH/4/61). Glasgow: University of Glasgow, 2007.
202. Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. The effectiveness of
computerized cognitive behavioural therapy in routine care. The British Journal Of Clinical
Psychology / The British Psychological Society 2006;45(Pt 4):499-514.
203. van den Berg S, Shapiro DA, Bickerstaffe D, Cavanagh K. Computerized cognitive-behaviour
therapy for anxiety and depression: a practical solution to the shortage of trained therapists.
Journal Of Psychiatric And Mental Health Nursing 2004;11(5):508-513.
204. Grime PR. Computerized cognitive behavioural therapy at work: a randomized controlled trial in
employees with recent stress-related absenteeism. Occupational Medicine (Oxford, England)
2004;54(5):353-359.
205. Proudfoot J, Goldberg D, Mann A, Everitt B, Marks I, Gray JA. Computerized, interactive,
multimedia cognitive-behavioural program for anxiety and depression in general practice.
Psychological Medicine 2003;33(2):217-227.
206. Farvolden P, Denisoff E, Selby P, Bagby MR, Rudy L. Usage and Longitudinal Effectiveness of a
Web-Based Self-Help Cognitive Behavioral Therapy Program for Panic Disorder. J Med
Internet Res 2005;7(1):e7.
207. Powell J, Clarke A. Internet information-seeking in mental health: Population survey, 2006:273277.
59
208. Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the
effectiveness of Internet-based psychotherapeutic interventions. Journal of Technology in
Human Services 2008;26:109-160.
209. Kessler D, Lewis G, Kaur S, Wiles N, King M, Weich S, et al. Therapist-delivered internet
psychotherapy for depression in primary care: a randomised controlled trial. Lancet
2009;374(9690):628-634.
210. Rains SA, Young V. A Meta-Analysis of Research on Formal Computer-Mediated Support
Groups: Examining Group Characteristics and Health Outcomes. Human Communication
Research 2009;35(3):309-U5.
211. OPM, GFK, University of York. Evaluation of Information Prescriptions. Final Summary Report.
March 2008, 2008.
212. NOS. Households with access to the Internet, GB: National Office of Statistics, 2007.
213. NOS. Adults who have ever used the Internet by sex/age (Great Britain): Individual Internet
Access: National Statistics, 2006.
214. Communities CfR. Mind the Gap: Digital England - a rural perspective, 2009.
215. Lemire M, Pare G, Sicotte C, Harvey C. Determinants of Internet use as a preferred source of
information on personal health. International Journal of Medical Informatics
2008;77(11):723-734.
216. Cook DA. The failure of e-learning research to inform educational practice, and what we can do
about it. Medical Teacher 2009;31(2):158-162.
217. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in
the health professions - A meta-analysis. Jama-Journal of the American Medical Association
2008;300(10):1181-1196.
218. Kaptchuk TJ, Kelley JM, Deykin A, Wayne PM, Lasagna LC, Epstein IO, et al. Do "placebo
responders" exist? Contemporary Clinical Trials 2008;29(4):587-595.
219. Whalley B, Hyland ME, Kirsch I. Consistency of the placebo effect. Journal of Psychosomatic
Research 2008;64(5):537-541.
220. Kaptchuk TJ, Kelley JM, Conboy LA, Davis RB, Kerr CE, Jacobson EE, et al. Components of
placebo effect: randomised controlled trial inpatients with irritable bowel syndrome. British
Medical Journal 2008;336(7651):999-1003.
221. Eysenbach G, Powell J, Englesakis M, Rizo C, Stern A. Health related virtual communities and
electronic support groups: systematic review of the effects of online peer to peer
interactions. British Medical Journal 2004;328(7449):1166-1170A.
222. Jadad AR, Enkin MW, Glouberman S, Groff P, Stern A. Are virtual communities good for our
health? They seem to be good at managing chaotic information - and may have other virtues
too. British Medical Journal 2006;332(7547):925-926.
223. Auerbach SM. Do patients want control over their own health care? A review of measures,
findings and research issues. J Health Psychol 2001;6:191-203.
224. Waterworth S, Luker K. Waterworth S, Luker K. Reluctant collaborators: do patients want to
be involved in decisions concerning care? J Adv Nurs 1990;15:971–6. J Adv Nurs 1990;15:971-976.
225. Beaver K, Luker KA, Owens RG, Leinster SJ, Degner L, Sloan JA. Treatment decision making in
women newly diagnosed with breast cancer. Cancer Nurs 1996;19:8-19.
226. Paterson C, Baarts C, Launso L, Verhoef MJ. Evaluating complex health interventions: a critical
analysis of the 'outcomes' concept. Bmc Complementary and Alternative Medicine
2009;9:11.
227. Heritier SR, Gebski VJ, Keech AC. Inclusion of patients in clinical trial analysis: the intention-totreat principle. Medical Journal of Australia 2003;179(8):438-440.
228. White IR. Uses and limitations of randomization-based efficacy estimators. Statistical Methods
in Medical Research 2005;14(4):327-347.
60
229. Bates DW, Wright A. Evaluating eHealth: Undertaking Robust International Cross-Cultural
eHealth Research. PLoS Med 2009;6(9):e1000105.
230. Knill-Jones RP, Crean GP, Holden R, Marjoribanks F. Practical aspects of expert systems and
computer-taken clinical histories in dyspepsia. In: Hedley AJ, Chun Por W, editors. MIC90.
Proceedings of the first Hong Kong (Asia-Pacific) Medical Informatics Conference. Hong Kong:
Hong Kong Society of Medical Informatics Ltd and Hong Kong Computer Society, 1991:192199.
231. Jones RB, Atkinson JM, Coia DA, Paterson L, Morton AR, McKenna K, et al. Randomised trial of
personalised computer based information for patients with schizophrenia. British Medical
Journal 2001;322(7290):835-840.
232. National Institute of Health. Request for Information (RFI) on Consumer Health Information
Interests and Behaviors for Seeking and Using Health Information, 2009.
61