Download Validation of a short food frequency questionnaire to assess

European Journal of Clinical Nutrition (1998) 52, 557±564
ß 1998 Stockton Press. All rights reserved 0954±3007/98 $12.00
http://www.stockton-press.co.uk/ejcn
Validation of a short food frequency questionnaire to assess
consumption of cereal foods, fruit and vegetables in Chinese
Singaporeans
AMC Ling, C Horwath and W Parnell
Department of Human Nutrition, University of Otago, PO Box 56, Dunedin, New Zealand
Objective: To assess the ability of a 16-item food frequency questionnaire (FFQ) to measure consumption of
cereal foods, fruit and vegetables in Chinese Singaporeans.
Design: Subjects completed the questionnaire twice, at the beginning and end of a six-week period during which
they also provided three 24 h diet recalls. Estimates of intake from the questionnaire were compared with those
from diet recalls.
Subjects: Subjects were recruited from a range of occupational groups through random sampling across
divisions in the headquarters of the Singapore Ministry of Health. Of the 81 subjects initially recruited, three
failed to complete the diet recalls, one was excluded due to changes in diet, and seven did not return the second
questionnaire.
Results: Mean difference in food group consumption estimated by the two methods did not differ signi®cantly
from zero for fruit (0.00 serving, s.d. ˆ 0.54, 95% CI ˆ 70.13, ‡ 0.12, P ˆ 0.95) or vegetables (70.05,
s.d. ˆ 0.29, 95% CI ˆ 70.12, ‡ 0.02, P ˆ 0.13). For cereal foods, the mean difference was small, but
signi®cantly different from zero only in women (70.32 servings, s.d. ˆ 0.92, 95% CI ˆ 70.59, 70.06,
P ˆ 0.02). At an individual level, cereal food intake as measured by the FFQ may be 37% below or 59%
above the diet recall values; and values for total fruit and vegetables may be half or double the recall values.
Among subjects whose intake was classi®ed into the lowest quartiles by diet recalls, 78% and 94% respectively,
fell into the lowest two questionnaire quartiles for cereal foods, and total fruit and vegetables. The ability of the
questionnaire to predict those having inadequate intake based on recall data was more than 90% for the three
food groups.
Conclusion: The short questionnaire cannot replace the three-day recalls in intake assessment for individuals,
but it could be used to screen for low consumers in intervention programmes, to assess mean food group intake in
population groups, and to rank individuals into broad categories of food group intake.
Sponsorship: International Life Sciences Institute, South East Asia.
Descriptors: cereals; fruit; food frequency questionnaire
Introduction
The dietary guidelines of most countries emphasize the
importance of increasing the consumption of fruit and
vegetables and cereal foods. In Singapore, ®ve to six
servings (for women) and six to seven servings (for men)
of cereal foods, and two servings each of fruit and vegetables are encouraged (Ministry of Health, 1989). It is
increasingly recognised that these foods provide not only
nutrients, but also a combination of other dietary components that appear to be protective against chronic disease.
This has led to recent interest in monitoring population
intakes of foods and food groups, not just nutrients (Cleveland et al, 1997; Krebs-Smith et al, 1997).
However, short instruments to assess adequacy of food
group intake are sparse. Most food frequency questionnaires (FFQs) have been developed and validated as measures of nutrient intake (Jain et al, 1980; Willet et al, 1985;
Block et al, 1986; Willett et al, 1987; Pietinen et al, 1988;
Hankin et al, 1991; Block et al, 1992; Eck et al, 1996).
Correspondence: Ms AMC Ling; Department of Nutrition, Institute of
Health, 3 Second Hospital Avenue, Singapore 168937 (after August 1998).
Received 30 June 1997; revised 12 April 1998; accepted 17 April 1998
Several studies (Jain et al, 1980; Mullen et al, 1984;
Pietinen et al, 1988; Salvini et al, 1989; Hankin et al,
1991; Serdula et al, 1993) have evaluated FFQs at the foodbased level, but with the exception of one (Serdula et al,
1993), the FFQs were relatively long (55±278 items).
Serdula et al (1993) evaluated a six-item fruit and vegetable questionnaire for telephone administration in the US
Behavioural Risk Factor Surveillance System, and generally found the mean number of daily fruit and vegetable
servings to be similar to those estimated by more detailed
methods. The authors concluded that the short FFQ is likely
to be useful for surveillance purposes. However, Serdula et
al (1993) and others investigating validity at the food group
level (Jain et al, 1980; Mullen et al, 1984; Pietinen et al,
1988; Hankin et al, 1991) did not evaluate the FFQs in
terms of their ability to identify low and high consumers,
nor did these studies adequately assess agreement between
methods at an individual level. Correlation coef®cients in
the range of 0.23±0.91 have been interpreted as indicating
moderate agreement. However, Bland & Altman (1986,
1995a) have stressed that correlation coef®cients do not
measure the extent to which methods produce the same
results, but merely quantify the extent to which they are
linearly related.
Validation of a short food frequency questionnaire
AM Chuan Ling et al
558
In this present study, we developed a short FFQ capable
of self-administration as part of a larger study investigating
consumption of cereal foods and total fruit and vegetables
in Chinese Singaporeans. Both food groups and servings
were de®ned as in the Daily Food Guide (Ministry of
Health, 1995) to enable assessment of compliance with
dietary recommendations. Therefore, foods such as cakes
and pastries, french fries and pickled vegetables, or fried
fruit fritters were excluded from the respective food groups.
However, for consistency with the aims of our larger study,
the FFQ assessed only fresh fruit and excluded dried fruits
and juices. The suitability of the FFQ was evaluated not
only for monitoring purposes, but also for screening for low
intakes and ranking individuals into broad categories of
intake. We compared estimated food group intake with that
measured by three 24 h diet recalls. Between-method
agreement was assessed by examination of group mean
intakes, mean difference and standard deviation of the
differences, classi®cation into quartiles of intake, ability
to identify individuals not meeting recommended intakes,
and Willett's actual values for surrogate categories (Willett,
1990).
Methods
Subjects
Eighty-one Chinese subjects were recruited from the headquarters of the Ministry of Health through random sampling within divisional status. As there were more female
than male employees, the latter were over-sampled. Subjects came from a wide range of occupational groups: from
secretaries, clerks, typists, telephone operators, of®ce attendants and drivers to statisticians, ®nance and accounts
executives, personnel and public affairs executives, and a
few nurses and doctors.
Study design
Subjects ®lled in the FFQ twice, at the beginning and end
of a six-week period. During this time, trained dietitians
administered 24 h diet recalls on three occasions. The
recalls included two week-days and one week-end day,
and were spaced over two to three weeks. The ®rst FFQ
was mailed to subjects together with a schedule for the
three interviews. Subjects were asked to complete the FFQ
and to bring it to the ®rst interview, during which the
dietitians reviewed and clari®ed any ambiguity in
responses. Interviews were conducted during of®ce hours.
The second FFQ was mailed two weeks after completion of
all diet recalls and consequently was not reviewed by the
dietitians.
The food frequency questionnaire (FFQ)
Questions and response formats. The 16-item FFQ (available from AMC Ling) was designed to assess usual intake
of cereal foods, fruit and vegetables over the previous three
months. The FFQ was administered during the ®st two
months of 1997. As a variety of fruit and vegetables is
readily available throughout the year in Singapore, seasonal
variation in intake is not likely to be signi®cant. The aim
was to provide estimates in mean daily numbers of servings. The FFQ took an average of ®ve minutes to
complete.
Subjects were asked to indicate the number of times per
day, per week or per month the following types of food
were eaten: porridge; plain rice; rice-based dishes (for
example chicken rice, fried rice); noodle dishes; breakfast
cereals; stir-fried vegetables; meat or seafood dishes with
added vegetables; soups with vegetables; vegetable salads;
and local vegetable salads (for example rojak and gadogado). For fresh fruit, bread, bread-based items (for example roti prata and chapati) and plain salted biscuits, portion
size estimates were requested as pieces or slices.
We included black and white photographs of various
portion sizes for plain rice and stir-fried vegetables. Five
portion sizes were shown as fractions or multiples of
amounts which make up one reference serving of rice or
vegetables. Photographs were used for plain rice and stirfried vegetables because these are probably the single
greatest contributors to the respective food groups, and
amounts consumed tend to vary widely.
The FFQ was pre-tested among a total of twenty
Singapore adults for clarity, ease of completion and interpretation of questions and preferred combinations of photographed portion sizes. None of these subjects were included
in the validation study. Special attention was paid to
aggregation of foods into single questions. For example,
the original question on rice and rice-based items was
separated into three questions with porridge preceding
more general items like rice and rice-based dishes. Feedback suggested that patterns of consumption differed for
these three types of food.
Coding. Servings were calculated using as a guide the
reference portions in the Daily Food Guide (Ministry of
Health, 1995). These were derived with consideration both
for nutrient content and quantities of food reported to be
used by respondents in a recent food consumption survey
(Ministry of Health, 1994). One serving of cereal foods is
de®ned as the amount likely to provide the carbohydrateequivalent of half a bowl (100 g) of rice. A serving of fruit
refers to units commonly eaten (for example a small
orange, a 130 g wedge papaya); and a serving of vegetables
is one cup (150 g) of cooked vegetables. Legumes, rather
than being included in the vegetable group, are classi®ed as
alternatives to meat.
For questions on frequency, reported use was converted
to daily frequency and multiplied by a serving value. Rice,
noodles and vegetables are often eaten as mixed dishes, but
there is little information on the contribution of individual
components to composite dishes. Therefore, serving values
used for several questions on mixed dishes were derived
based on our experience, or assessment of typical servings
in popular eating outlets for foods commonly eaten away
from home. For example, consumption of a meat dish with
added vegetables was credited as providing a quarter of the
reference vegetable serving; and a rice-based dish was
counted as contributing two cereal food servings (namely
two half bowls of rice).
Amounts reported to be eaten were often fractions of a
serving (for example three pieces of fruit per week is
equivalent to three-sevenths of a serving per day). A
daily average was computed by summing serving estimates
across all questions.
The reference method
24 h diet recalls. Although diet recording is generally
considered the preferred method for validating a FFQ
designed to estimate food or nutrient intakes (Willett,
1990), it is probably suboptimal within the context of the
local dietary pattern. Families usually share communal
dishes, and the practice of portioning out and weighing
food would be likely to interfere with usual intake. Moreover, foods are usually eaten as mixed dishes. The need to
separately weigh individual ingredients of interest over a
period of several days would demand highly-motivated
Validation of a short food frequency questionnaire
AM Chuan Ling et al
subjects. Therefore, non-consecutive 24 h recalls were
chosen as the reference method. The FFQ was intended
to determined consumption of broad food groups rather
than individual foods or nutrients. However, data documenting the number of days required to assess food group
intake is lacking. A three-day survey was chosen in order to
avoid overburdening the subjects, and because three to
seven days are normally adequate to assess macronutrients
such as energy and total carbohydrate (Willett, 1990), and
thus presumably food group intakes.
All interviews were conducted by the ®rst investigator
with the help of two dietitians who attended two half-day
training sessions in order to ensure consistency of interview
approach and coding method. As foods are often eaten as
composite dishes, we asked subjects to estimate amounts of
individual components eaten by pouring dried beans into
standard plates, bowls or cups. Following the interview, the
dietitians coded the number of cereal, fruit and vegetable
servings on the interview forms, and these were checked by
the ®rst investigator to ensure uniform coding decisions.
Daily intake was estimated by averaging number of
servings over the three days.
Coding. Gram weights were converted into servings
using reference portions in the Daily Food Guide (Ministry
of Health, 1995). These were supplemented by serving
values determined for foods not listed in the Guide.
In estimating total food group intake, several investigators (Kant et al, 1991; Patterson et al, 1990) excluded
foods consumed in amounts below a minimum cut-off (for
example 30 g for fruit and vegetables, and 15 g for cereal
foods) to avoid giving credit for consumption when
amounts eaten were small. Rather than assigning fractions
of a serving, Kant et al (1991) credited one serving to
individuals who consumed amounts between the minimum
cut-off and median serving size; and Patterson et al
(1990) counted 30 g±75 g of vegetables and 30 g±120 g
of fruit as contributing one serving. As the authors
acknowledged, this latter procedure may lead to overestimation. Discounting small amounts on the other hand,
would bias towards under-estimation especially considering the way vegetables and some cereals are consumed in
the Singaporean diet. Spoonfuls of cereals are often added
to milk drinks, and multiple small amounts of vegetables
eaten as part of noodle or meat dishes can add up to
signi®cant intakes. Therefore, servings were assigned to
all recorded foods belonging to the three food groups,
using fractions of servings to represent small amounts.
Diet recalls thus, re¯ect comparable levels of measurements to FFQ.
Statistical analysis
The ability of the FFQ to estimate group means was
investigated using paired t-tests. As recommended by
Bland & Altman (1986), mean and standard deviation of
the differences were calculated to assess agreement
between methods at the individual level. For the analysis
to be meaningful, it must be assumed that the mean
difference and variability of the differences remain reasonably constant throughout the range of measurements (Bland
& Altman, 1995b). This was checked graphically by plotting for each food group the difference between methods
against their mean. As some plots showed a tendency for
differences to increase with measurements, we calculated
the 95% limits of agreement (de®ned as mean difference
1.96 s.d. of differences) for both log-transformed and
actual values.
To assess the ability of the FFQ to rank individuals into
broad categories of intake, we divided serving estimates
from both FFQs and diet recalls into quartiles to examine
their cross-classi®cation. Cut-off points for quartiles were
determined separately for the two methods. The percentage
correctly classi®ed within one quartile and into extreme
quartiles was calculated. To quantify intake de®ned by the
FFQ quartiles, we calculated mean servings from the three
diet recalls for subjects in each quartile. This gives the
actual values for surrogate categories as described by
Willett (1990). Differences between quartiles are a function
of both the true variation in serving intake and the measurement error associated with the FFQ. For comparison,
mean servings were also computed for each quartile of the
diet recall distribution. Comparing the `actual' means for
quartiles as de®ned by diet recalls with means from
surrogate categories as de®ned by FFQ permits a simultaneous evaluation of misclassi®cation and error in estimation of intake.
As an instrument may perform with different accuracy
in those meeting or not meeting the serving recommendations, we evaluated the FFQ for its sensitivity, speci®city
and predictive values. Speci®city was de®ned as the proportion of subjects having less than the recommended
number of servings by diet recalls who also fell short on
the FFQ. Sensitivity was the proportion of those having
adequate servings by diet recalls, and who also reported
similar intake on the FFQ. Negative predictive value was
the proportion of those falling short of recommended
servings on the FFQ who were `truly negatives' on the
diet recalls. Positive predictive value was the proportion of
those meeting serving targets on the FFQ and whose diet
recalls showed likewise.
Actual values for surrogate categories and sensitivity,
speci®city and predictive values were presented for the
second rather than ®rst FFQ based on two considerations.
As the FFQ asked about food intake during the previous
three months, the repeat FFQ referred to a period encompassing the diet recalls. Furthermore, the repeat FFQs were
returned by mail as is planned in the main study, unlike the
®rst FFQ which was personally collected from subjects and
brie¯y reviewed with them.
Since several distributions were skewed towards higher
serving values, we looked at both Spearman rank and
Pearson (after log transformation of values) correlations.
For fruit, some subjects reported no use, so 1.0 was added
to all values before transformation. As variation in intake
among men and women of varying age arti®cially increases
observed correlations (Willett et al, 1987), serving estimates were adjusted for sex and age.
FFQ repeatability was assessed by determining the mean
differences and standard deviation of the differences
between repeat administrations. All analyses were performed using the SPSS statistical programme.
Results
Seventy-seven subjects completed the ®rst FFQ and three
diet recalls, and 70 (21 men and 49 women) returned the
second FFQ. One subject failed to complete the diet recalls
because of extended sick leave, and two because of an
inability to participate. Recalls from one subject were
excluded due to changes in diet. Those who completed
the study were approximately equally divided between 24±
45 and 46±59 y age groups (mean age 42.6, s.d. 10.2), and
came from diverse socio-economic backgrounds: 38%
professional, 26% administrative support, 27% clerical
559
Validation of a short food frequency questionnaire
AM Chuan Ling et al
560
and 9% blue collar employees. There were no signi®cant
differences in demographic characteristics between those
who completed the study (n ˆ 70) and those who failed to
do so (n ˆ 11).
Repeatability
At the group level, no signi®cant mean difference was
found between repeat FFQ measures for cereal foods, fruit
or vegetables. When fruit and vegetables were combined,
we found the difference to be statistically signi®cant from
zero (P ˆ 0.02), but small (mean difference ˆ 70.14 serving). Between repeat administrations, individual FFQ
measurements for total fruit and vegetables may be lower
by 1.12 servings or higher by 0.84 servings. For cereal
foods, the differences between repeated measurements may
vary from being 1.34 servings lower to 1.18 servings
higher.
Validity
Mean and median servings from the diet recalls and two
FFQ administrations are presented in Table 1. No signi®cant differences were found in mean intakes between the
methods for fruit and vegetables, separately and combined.
For cereal foods, differences were signi®cant only in
women (P ˆ 0.01 for FFQ1; P ˆ 0.02 or FFQ2). Differences were however, small with the FFQ overestimating
cereal intake by an average of 0.28 servings.
For the vegetable servings from diet recalls and FFQ,
Figure 1 shows that the scatter of the differences between
measurements increases with serving valuesÐan indication
that the methods did not agree equally across the entire
range of measurements. We would expect the limits of
agreement to be wider apart than necessary for small
serving values and narrower than they should be for
larger values. The pattern of variability is improved when
the difference between log transformed values was plotted
against their mean. Figure 2 shows reduced tendency for
systematic variation.
Similar agreement was observed between diet recalls
and the ®rst or second FFQ (Table 2). Limits obtained
using log-transformed values allow assessment of agreement between methods in terms of a ratio; and limits
computed from the actual data give an idea of the
magnitude of differences between methods. As a reference point, central points have also been included for all
intervals. Thus, for cereal foods, FFQ measurements may
be about 37% below to 59% above recall values, and
95% of total fruit and vegetable servings estimated by the
FFQ would be expected to lie between half or double the
diet recall values. Looking at the untransformed values,
the FFQ may over-estimate or under-estimate cereal
intake by about 1.5 servings, and total fruit and vegetable
intake by about 1.25 servings, when compared with
estimates from diet recalls. Intervals were similarly
wide for fruit and vegetables analysed separately. These
levels of between-method differences would mean that
the FFQ is unacceptable for intake assessment of individuals.
The proportion of subjects in diet recall quartiles correctly classi®ed by the FFQ into the same quartile and into
Table 1 Daily number of food group servings based on the average of three 24h diet recalls, and on the food frequency questionnaires administered at the
beginning (FFQ1) and end of study (FFQ2)
24 h diet recalls (n ˆ 77)
FFQ1 (n ˆ 77)
FFQ2 (n ˆ 70)
Food group
Mean
Median
Standard deviation
Mean
Median
Standard deviation
Mean
Median
Standard deviation
Cereal foods
Men
Women
Fruit
Vegetables
Fruit and vegetables
4.61
3.34
1.52
0.91
2.43
4.61
3.48
1.33
0.85
2.03
1.37
0.95
1.15
0.45
1.34
4.63
3.58
1.45
0.92
2.37
4.44
3.63
1.00
0.85
1.92
1.08
0.99
1.22
0.47
1.35
4.74
3.62
1.51
0.93
2.43
4.49
3.46
1.00
0.88
2.09
1.45
1.15
1.22
0.45
1.36
Figure 1 Plot of difference between vegetable servings estimated by three 24h diet recalls and the second-administered FFQ against mean servings by the
two methods, showing 95% limits of agreement and central point.
Validation of a short food frequency questionnaire
AM Chuan Ling et al
561
Figure 2 Plot of difference between log-transformed data on vegetable servings from three 24 h diet recalls and the second-administered FFQ against the
mean of the log-transformed data by the two methods, showing 95% limits of agreement and central point.
Table 2 95% limits of agreement (& central pointa) between the number of food group servings as assessed by three 24 h diet recalls and the ®rst (FFQ1)
and second (FFQ2) administrations of food frequency questionnaire
Log-transformed valuesb
Food group
Cereal foods
Fruit
Vegetables
Fruit and vegetables
24h diet recalls vs FFQ1
0.63±1.45
0.64±1.66
0.41±2.38
0.51±1.99
a
(0.95 )
(1.03a)
(0.99a)
(1.01a)
Actual valuesc
24h diet recalls vs FFQ2
0.57±1.55
0.62±1.61
0.40±2.25
0.48±1.93
a
(0.94 )
(1.00a)
(0.95a)
(0.96a)
24h diet recalls vs FFQ1
71.59±1.24
71.12±0.98
70.67±0.66
71.16±1.27
a
(70.18 )
(0.07a)
(70.01a)
(0.06a)
24h diet recalls vs FFQ2
72.09±1.60
71.08±1.07
70.84±0.70
71.33±1.19
(70.24a)
(0a)
(70.07a)
(70.07a)
a
For the log-transformed values, the central point should give a ratio close to 1. For the non-transformed data, the central point should give a mean
difference close to 0.
b
95% limits of agreement were based on subtraction of the log-transformed FFQ values from those of diet recalls. 95% of diet recall to FFQ ratios would lie
between the limits (mean 1.96).
c
95% of actual differences between methods would lie between these limits (mean 1.96).
Table 3 Cross-classi®cation by quartiles of food group serving distributions derived from three 24h diet recalls and the ®rst (FFQ1) and second (FFQ2)
administrations of the food frequency questionnaire
Lowest quartiles on diet recalls
Food group
Cereal foods
Fruit
Vegetables
Fruit and vegetables
a
Quartiles on diet recalls
Lowest two
quartiles
on FFQ1
(%)
Lowest two
quartiles
on FFQ2
(%)
Same
quartiles
on FFQ1
(%)
Within-one
quartiles
on FFQ1
(%)
Extremea
quartiles
on FFQ1
(%)
Same
quartiles
on FFQ2
(%)
Within-one
quartiles
on FFQ2
(%)
Extreme
quartiles
on FFQ2
(%)
78.9
100
84.2
100
76.5
100
88.9
88.9
46.8
62.3
36.4
54.5
90.9
98.7
83.1
96.1
2.6
1.3
0
0
47.1
64.3
35.7
62.8
85.7
97.1
81.4
94.3
1.4
0
1.4
0
Quartile 1 on diet recalls versus quartile 4 on FFQ or vice versa.
within-one quartiles ranged respectively, from an average
of 36% and 82% for vegetables to 63% and 98% for fruit
(Table 3). Due to chance alone, 25% or 62.5% of subjects
would be expected to be correctly classi®ed into the same
or within-one quartiles. Of those subjects belonging to the
lowest diet recall quartile, an average of 78% fell into the
lowest two quartiles for cereal foods, and an average of
94% were in the lowest two quartiles for total fruit and
vegetables, when classi®ed by the FFQ. Gross misclassi®-
cation was rare (0±2.6%). Better results were observed at
the high end of the distributions (data not shown).
For all food groups, mean servings as measured by diet
recalls increased across the FFQ quartiles (Table 4). Actual
values for surrogate categories were presented only for the
second FFQ. Stepwise increase across quartiles was slightly
better, especially for vegetables with the ®rst FFQ. As a
result of misclassi®cation and regression towards the mean
(Willett et al, 1985), the ratio of mean servings for highest
Validation of a short food frequency questionnaire
AM Chuan Ling et al
562
Table 4 Mean daily number of food group servings derived from three 24h diet recalls for quartiles de®ned by diet recalls,
and for quartiles derived by the second food frequency questionnaire administration (n ˆ 70)
Mean 24h diet recall values for FFQ2 and diet recall quartiles
Quartiles
1 FFQ2
24 h diet
2 FFQ2
24 h diet
3 FFQ2
24 h diet
4 FFQ2
24 h diet
recall
recall
recall
recall
Cereal foods
Fruit
Vegetables
Fruit and vegetables
2.65 0.77
2.31 0.46
3.45 0.69
3.30 0.28
3.92 0.86
3.97 0.18
4.86 1.44
5.31 1.02
0.64 0.63
0.39 0.27
1.10 0.53
1.10 0.18
1.69 0.43
1.73 0.20
3.98 1.17
3.18 1.15
0.63 0.30
0.45 0.14
0.81 0.31
0.72 0.08
0.96 0.34
1.01 0.10
1.07 0.32
1.51 0.50
1.29 0.61
1.10 0.37
1.78 0.43
1.81 0.13
2.34 0.58
2.53 0.31
4.06 1.39
4.31 1.13
Table 5 Sensitivity, speci®city and predictive values of the second-administered food frequency questionnaire for assessing adequacy of intake of food
group servings
Food group
No. of subjects
Cereal foods
21 men
49 women
70
70
70
Fruit
Vegetables
Fruit and vegetables
Recommended
no. of servings
% meeting
recommendation
by diet recalls
% meeting
recommendation
by FFQ
Sensitivity
(%)
Speci®city
(%)
Negative predictive
value (%)a
6
5
2
2
4
19.0
2.0
30.0
1.4
10.0
14.3
14.3
38.6
5.7
14.3
75
100
86
100
100
100
88
82
96
95
94
100
93
100
100
a
For cereal foods, negative predictive value for men ˆ 17/18 or 94%; negative predictive value for women ˆ 42/42 or 100%. For fruit and vegetables,
negative predictive value ˆ 60/60 or 100%; positive predictive value ˆ 7/10 or 70%. Too few individuals met the recommended number of servings to
allow computation of other positive predictive values.
to lowest quartiles was lower for FFQ. Comparison
between FFQ and diet recall across quartiles suggests that
the FFQ may have dif®culty in distinguishing between
middle levels of intake for cereal foods, and for vegetables.
Sensitivity, speci®city and negative predictive values
are presented for the second FFQ (Table 5). Similar values
were observed for the ®rst FFQ. Too few subjects met the
recommended intakes of cereal foods to allow meaningful
computation of the positive predictive value. Of ten
subjects predicted by the FFQ as having adequate servings
of total fruit and vegetables, three were falsely positive
(data not shown). The FFQ was suf®ciently sensitive for
example, to identify all seven subjects who had four or
more servings of fruit and vegetables by diet recalls.
Speci®city and negative predictive values were high. For
example, 42 out of 48 women who had less than ®ve
servings of cereal foods by diet recalls also reported this
level of intake in the FFQ; and all women who had
inadequate intakes on the FFQ were `truly negatives' by
diet recalls.
Very similar values were observed for Spearman rank
and Pearson correlations. The unadjusted Pearson correlation coef®cients ranged from a mean of 0.63 for vegetables
to 0.79 for cereal foods and 0.84 for fruit with only slight
reduction following adjustment (data not shown). When
fruit and vegetables were combined, correlation coef®cient
averaged 0.80. Correlations were generally higher between
intakes measured by diet recalls and the ®rst FFQ.
Discussion
Although a high response rate was achieved from random
sampling across the divisions in the headquarters of the
Ministry of Health, with more than a third of the sample
comprising of clerical and blue collar workers, it must be
acknowledged that the sampling frame for this small
validation study did not encompass the entire Singapore
population. As such, it is likely that the study population is
of somewhat higher educational level and socio-economic
status than the general population. Nevertheless, it must be
noted that none of the subjects were nutritionists or dietitians and none had any prior experience of dietary interviews and were unaware that they would be asked to recall
speci®c day's intake.
Acceptable repeatability was established. For all food
groups, there were no signi®cant differences between
repeated measurements except for a signi®cant but small
difference between measurements when fruit and vegetables were combined. Most previous studies have assessed
repeatability for reported frequencies of individual foods at
intervals of between three months and two years, mainly
using correlation coef®cients (Nomura et al, 1976; Colditz
et al, 1987; Byers et al, 1987; Salvini et al, 1989). For
reasons explained earlier, correlation coef®cients for
repeated FFQ administrations are presented here only for
purposes of comparisonÐPearson's correlations were 0.85
for cereal foods and 0.90 for total fruit and vegetables.
These were substantially higher than those observed in
other studies for individual cereal (0.38±0.71) and fruit
and vegetable items (0.19±0.73). To our knowledge, this is
the only published report examining the repeatability of
questions on aggregated food groups. The shorter time
interval in our study could have overestimated repeatability
as subjects may remember earlier responses. However, long
intervals mean that true dietary changes are more likely,
thus reducing apparent repeatability.
Despite the relatively high correlation coef®cients and
little variation in mean differences between repeated measures, it is of interest to note individual FFQ measurements
from one administration to another may vary by about one
serving both for cereal foods ( ‡ 1.18, 7 1.34) and for total
fruit and vegetables ( ‡ 0.84, 7 1.12).
Mean servings measured by the FFQ and the diet
recalls were remarkably similar. More detailed FFQs
Validation of a short food frequency questionnaire
AM Chuan Ling et al
have generally been found to over-estimate mean intakes
for both nutrients and foods (Jain et al, 1980; Mullen et
al, 1984; Pietinen et al, 1988; Salvini et al, 1989; Hankin
et al, 1991). Krebs-Smith et al (1995) found that estimates
of total fruit and vegetables were positively associated
with the number of food items included in food frequency
questionnaires. They suggested that summing across individual foods may not be a valid way to assess intake of
total fruit and vegetables, and that summary food group
questions and/or adjustment factors may increase precision of estimates. Therefore, this short FFQ may have an
advantage over longer questionnaires for example, when
used for dietary surveillance. It is of interest to note that,
consistent with other reports (Cleveland et al, 1997),
mean intakes for all food groups in this study were
below recommendations.
Clearly, the FFQ cannot be used interchangeably with
three 24 h diet recalls if estimates of individual intake are
required. For the FFQ to do this, 95% of differences would
have to lie within limits that would not be of practical
importance. Tables 3 and 4 however, show that exact
agreement between measurements by the two methods is
not critical to the ability of the FFQ to rank individuals, an
observation similar to that by Wilson & Horwath (1996)
using a calcium FFQ. Jain et al (1980) also found that the
higher mean intakes given by their diet history questionnaire had little effect on its ability to classify individuals
due to the constancy of overestimation. The observation
that a FFQ can produce good group means but have
unsatisfactory validity at the individual level (or vice
versa) points to the usefulness of examining validity at
varying levels of precision at both group and individual
levels using different approaches.
Mean intake of cereal foods from the FFQ was signi®cantly higher than that from the diet recalls for women, but
not men. This may suggest that men performed better in
estimating portion sizes, which is contrary to what has been
observed by other investigators (Yuhas et al, 1989). However, the direction of bias is not clear. Whereas detailed
FFQs have produced higher intake estimates, there is some
evidence of underreporting with 24 h recalls (Gibson,
1990). To further explore this question, we compared in
®ve men and seven women, weighed portions of rice with
estimates using dried beans. Ten subjects (six women)
underestimated portions by the bean method. Underestimation ranged from 710% to 750% and was on average,
greater in women. Women tended to estimate amounts by
spreading the beans thinly. Therefore, the use of beans to
aid in serving size estimation may have contributed to
underestimation of cereal foods by the 24 h recall method
in women. Young & Marion (1995) in their recent review,
noted that dif®culties with portion estimations may be
related to perceptual problems that have `barely been
explored'. As Block (1982) pointed out, in the absence of
a `gold standard', we have to accept the approach of
relative validity.
There was little difference in all measures of agreement
for the two FFQs, except perhaps for the slightly better
correlations observed with the ®rst FFQ. Other investigators (Willett et al, 1985; Pietinen et al, 1988; Salvini et al,
1989; Eck et al, 1996) have found better agreement with
the second rather than ®rst FFQ administration. In the
present study, differences between correlations for the
®rst and second FFQ were small, suggesting any bias was
probably not great.
Precision in terms of absolute intake at the individual
level is often not necessary for epidemiologic studies. It
may be suf®cient for an instrument to discriminate
among groups of individuals (Block, 1982). The mean
intakes of individuals in the highest FFQ quartiles for
cereal foods and fruit and vegetables are respectively,
83% and 215% higher than in the lowest quartiles. The
corresponding percentages are 130% and 292% for quartiles de®ned by diet recalls. Although the between-quartile contrasts are more modest for the FFQ, the variation
in intake from the lowest to highest quartiles suggests
that the FFQ could readily distinguish between the high
and low consumers. Such categorisation of food group
intake is valuable in population studies investigating dietdisease relationships.
At the public health level, the high negative predictive
values of the FFQ would mean that almost all individuals
identi®ed in screening programmes as requiring an
increased intake of cereal foods and fruit and vegetables
would be given appropriate advice. As the proportion of
individuals not meeting the guidelines for fruit and vegetables is relatively high, the accuracy of the FFQ in
identifying `negatives' is desirable. The ability to identify
those consuming less than the recommended amount of
fruit and vegetables is consistent with the work of Cox et al
(1997), in which responses to a self-monitoring `portion'
measurement of fruit and vegetable consumption were
compared with weighed intakes. If the FFQ is used to
guide intervention efforts, the low error in speci®city
(0±12%) would mean that relatively small proportion of
individuals would lose the possibility of receiving a
required intervention. The sensitivity error is comparably
low (less than 5% for all food groups). However, receiving
an unneeded intervention is probably of secondary importance to not getting a required intervention. Being relatively
short, the FFQ can be easily included as part of overall
health surveys. In Australia and the United States, similar
short questions have been used as indicators to monitor
fruit and vegetable intake (Coles-Rutishauser, 1996;
Serdula et al, 1995).
Rather than intended as a measure of agreement, Pearson correlation coef®cients were included here simply for
comparison with previous studies. Such comparisons are
however, limited by differences in the composition of the
study populations and reference methods. Generally, correlations fell in the range of 0.08±0.70 for individual fruit
and vegetable items (Salvini et al, 1989). At the food group
level, correlations ranged from 0.60±0.77 for cereals, and
0.29±0.71 for fruit and vegetables (Jain et al, 1980;
Pietinen et al, 1988; Hankin et al, 1991; Serdula et al,
1993). Comparable or higher correlations were observed in
the present study.
Despite the much shorter FFQ, the validity of food
group intake observed in the present study was comparable to or better than that reported for longer FFQs listing
individual foods (Jain et al, 1980; Mullen et al, 1984;
Pietinen et al, 1988; Hankin et al, 1991). Perhaps, subjects are better able to describe frequency of use for more
generalised categories than for speci®c foods (Horwath,
1990). It is possible that the validity of the FFQ in the
present study may be over-estimated as a result of
reporting errors shared with diet recalls. Under-estimated
validity is however, just as likely. Salvini et al (1989)
found the week-to-week within- to between-person variability in mean intake to be greater than one for most
foods. Although we would expect less within-person
variation at the food-group level, it is plausible that
three 24 h recalls may have been insuf®cient to characterize the usual diet.
563
Validation of a short food frequency questionnaire
AM Chuan Ling et al
564
Conclusions
This short FFQ produced repeatable and valid estimates of
mean food group intake in a population of Chinese Singaporeans. The FFQ was able to classify individuals into
broad categories of intake, and had high predictive value in
identifying those not consuming the recommended number
of food group servings. Clearly, the FFQ cannot replace the
three-day diet recalls in estimating intake for individuals.
The relevance of this FFQ to other populations would need
to be established by speci®c validation studies.
AcknowledgementsÐGrateful thanks to the Ministry of Health, Singapore
for their support and assistance in co-ordination of the study; and Pauline
Wilson, Sheila Williams and Caryn Thompson for their assistance in data
analysis.
References
Bland JM & Altman DG (1986): Statistical methods for assessing the
agreement between two methods of clinical measurement. Lancet 1,
307±310.
Bland JM & Altman DG (1995): Comparing two methods of clinical
measurement: A personal history. Int. J. Epidemiol. 24, Suppl. 1,
S7±S14.
Bland JM & Altman DG (1995): Comparing methods of measurement:
why plotting difference against standard method is misleading. Lancet
346, 1085±1087.
Block G (1982): A review of validations of dietary assessment methods.
Am. J. Epidemiol. 115, 492±505.
Block G, Hartman AM, Dresser CM, Carroll MD, Gannon J & Gardner L
(1986): A data-based approach to diet questionnaire design and testing.
Am. J. Epidemiol. 124, 453±469.
Block G, Thompson FE, Hartman AM, Larkin FA & Guire KE (1992):
Comparison of two dietary questionnaires validated against multiple
dietary records collected during a 1-year period. J. Am. Diet. Assoc. 92,
686±693.
British Standards Institution (1979): Precision of Test Methods I. Guide for
the Determination and Reproducibility for a Standard Test Method. (BS
5497, part 1). London: BSI.
Byers T, Marshall J, Anthony E, Fiedler R & Zielezny M (1987): The
reliability of dietary history from the distant past. Am. J. Epidemiol.
125, 999±1011.
Cleveland LE, Cook DA, Krebs-Smith SM & Friday J (1997): Method for
assessing food intakes in terms of servings based on food guidance. Am.
J. Clin. Nutr. 65, Suppl, 1254S±1263S.
Colditz GA, Willett WC, Stampfer MJ, Sampson L, Rosner B, Hennekens
CH & Speizer FE (1987): The in¯uence of age, relative weight,
smoking and alcohol intake on the reproducibility of a dietary questionnaire. Int. J. Epidemiol. 16, 392±398.
Coles-Rutishauser IHE (1996): A Guide to Instruments for Monitoring
Food Intake, Food Habits and Dietary Change. Canberra: Australian
Institute of Health and Welfare (Food and Nutrition Monitoring Unit
Working paper No. 96.2).
Cox DN, Anderson AS, Reynolds J, McKellar S, Mela DJ & Lean MEJ
(1997): Measuring fruit and vegetable intake: is ®ve-a-day enough?
Eur. J. Clin. Nutr. 51, 177±180.
Eck L, Klesges LM & Klesges RC (1996): Precision and estimated
accuracy of two short-term food frequency questionnaires compared
with recalls and records. J. Clin. Epidemiol 49, 1195±1200.
Gibson R (1990): Validity in dietary assessment: a review. J. Can. Diet.
Assoc. 51, 275±280.
Hankin JH, Wilkens LR, Kolonel LN & Yoshizawa CN (1991): Validation
of a quantitative diet history method in Hawaii. Am. J. Epidemiol. 133,
616±628.
Horwath CC (1990): Food frequency questionnaires: a review. Aust. J.
Diet. Assoc. 47, 71±76.
Jain M, Howe GR, Johnson KC & Miller AB (1980): Evaluation of a diet
history questionnaire for epidemiologic studies. Am. J. Epidemiol. 111,
212±219.
Kant AK, Block G, Schatzkin A, Ziegler RG & Nestle M (1991): Dietary
diversity in the US population, NHANES II, 1976±1980. J. Am. Diet.
Assoc. 91, 1526±1531.
Krebs-Smith SM, Heimendinger J, Subar AF, Patterson BH & Pivonka E
(1995): Using food frequency questionnaires to estimate fruit and
vegetable intake: association between the number of questions and
total intakes. J. Nutr. Edu. 27, 80±85.
Krebs-Smith SM, Cleveland LE, Ballard-Barbash R, Cook DA & Kahle
LL (1997): Characterising food intake patterns of American adults. Am.
J. Clin. Nutr. 65, Suppl, 1264S±1268S.
Ministry of Health, Singapore, Training and Health Education Department
(1989): Guidelines for a Healthy DietÐRecommendations of the
National Advisory Committee on Food and Nutrition.
Ministry of Health, Singapore, Food and Nutrition Department (1994):
Food Consumption Study 1993.
Ministry of Health, Singapore, Food and Nutrition Department (1995): A
Balanced DietÐThe Pyramid Way.
Mullen BJ, Krantzler NJ, Grivetti LE, Schutz HG & Meiselman HL
(1984): Validity of a food frequency questionnaire for the determination
of individual food intake. Am. J. Clin. Nutr. 39, 136±143.
Nomura A, Hankin JH & Rhoads GG (1976): The reproducibility of
dietary intake data in a prospective study of gastrointestinal cancer. Am.
J. Clin. Nutr. 29, 1432±1436.
Patterson BH, Block G, Rosenberger WF, Pee D & Kahle LL (1990): Fruit
and vegetables in the American diet: data from the NHANES II survey.
Am. J. Public Health. 80, 1443±1449.
Pietinen P, Hartman AM, Haapa E, Rasanen L, Haapakoski J, Palmgren J,
Albanes D, Virtamo J & Huttunen JK (1988): Reproducibility and
validity of dietary assessment instruments: a self-administered food-use
questionnaire with a portion size picture booklet. Am. J. Epidemiol. 128,
655±666.
Salvini S, Hunter DJ, Sampson L, Stampfer MJ, Colditz GA, Rosner B &
Willett WC (1989): Food-based validation of a dietary questionnaire:
the effects of week-to-week variation in food consumption. Int. J.
Epidemiol. 18, 858±867.
Serdula M, Coates R, Byers T, Mokdad A, Jewell S, Chavez N, MaresPerlman J, Newcomb P, Ritenbaugh C, Treiber F & Block G (1993):
Evaluation of a brief telephone questionnaire to estimate fruit and
vegetable consumption in diverse study populations. Epidemiol. 4,
455±463.
Serdula M, Coates R, Byers T, Simoes E, Mokdad A & Subar A (1995):
Fruit and vegetable intake among adults in 16 states: results of a brief
telephone survey. Am. J. Public Health 85, 236±239.
Willett WC, Sampson L, Stampfer MJ, Rosner B, Bain C, Witschi J,
Hennekens CH & Speizer FE (1985): Reproducibility and validity of a
semiquantitative food frequency questionnaire. Am. J. Epidemiol. 122,
51±65.
Willett WC, Reynolds RD, Cottrell-Hoehner S, Sampson L & Browne
ML (1987): Validation of a semiquantitative food frequency questionnaire: comparison with a 1 year diet record. J. Am. Diet. Assoc. 87,
43±47.
Willett WC (1990): Nutritional Epidemiology. New York: Oxford University Press.
Wilson P & Horwath C (1996): Validation of a short food frequency
questionnaire for assessment of dietary calcium in women. Eur. J. Clin.
Nutr. 50, 220±228.
Young LR & Marion M (1995): Portion sizes in dietary assessment: issues
and policy implications. Nutr. Rev. 53, 149±158.
Yuhas JA, Bolland JE & Bolland TW (1989): The impact of training, food
type, gender, and container size on the estimation of food portion sizes.
J. Am. Diet. Assoc. 89, 1473±1477.