Download VALIDITY AND RELIABILITY OF PEER ASSESSMENT RATING

VALIDITY AND RELIABILITY OF PEER ASSESSMENT RATING INDEX
SCORES OF DIGITAL AND PLASTER MODELS
A THESIS
Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in
the Graduate School of The Ohio State University
By
Curtis Kyo-shin Andrews, D.D.S.
*****
The Ohio State University
2008
Master’s Examination Committee:
Approved by
Dr. Allen Firestone, Advisor
Dr. Michael Beck
Dr. Robert Rashid
Advisor
Graduate Program in Dentistry
ABSTRACT
Introduction: The object of this research was to evaluate the reliability and validity of
Peer Assessment Rating (PAR) index scores derived from computerized digital models.
Computer based digital orthodontic models have been shown to be valid and reliable
when making measurements of tooth size and simple diagnoses of dental malocclusions.
The validity and reliability of PAR scores from digital models and plaster models that
have been previously scored and standardized for a calibration exercise for the PAR
index have not been established. Methods: Ten examiners were trained to score digital
and plaster models using the PAR index. Thirty standardized and previously scored
plaster PAR models used in PAR calibration were digitized using digital study model
technology. The plaster and digital calibration models were randomized and scored by
the ten examiners. All measurements were repeated by all examiners at a second session.
Results: Plaster and digital scores were compared to the gold standard PAR scores for
validity. The Intraclass Correlation Coefficients (ICC) for plaster models ranged from
0.808-0.926. The mean ICC for plaster models was 0.891 with a 95% confidence interval
(CI) of 0.829-0.940. The ICC’s for digital models ranged from 0.846-0.922. The mean
ICC for digital models was 0.878 with CI of 0.811-0.931. Intra-rater reliability was
excellent for plaster (ICC =0.919) and for digital models (ICC= 0.905). Inter-rater
reliability was also good for plaster (ICC=0.912) and digital models (ICC=0.883).
ii
Conclusions: The PAR index scores obtained from the digital and plaster calibration
models were shown to be valid and reliable.
iii
Dedicated to my beautiful wife Laurie and our children
iv
ACKNOWLEDGMENTS
I wish to thank my advisor, Allen Firestone, for his hard work and support of me
in my orthodontic education. I also wish to thank Michael Beck and Robert Rashid for
their help in making this project a success.
I thank the faculty and staff of The Ohio State University Section of Orthodontics
for the sacrifices of time and effort that contribute to an outstanding education.
I wish to thank my fellow residents for their tireless support in gathering data and
making this project successful. I am grateful to Spencer Johnson for his assistance in all
aspects of my research project.
I also wish to thank my wife, Laurie, who’s love and support have been a source
of strength through everything.
v
VITA
May 1, 1977…………………………Born- Honolulu, Hawaii
2001………………………………....B.S. Family Science, Brigham Young University
2005…………………………………D.D.S., University of Iowa
2005-present…………………………Resident, Graduate Orthodontic Program, The Ohio
State University
FIELDS OF STUDY
Major Field: Dentistry
Specialty: Orthodontics
vi
TABLE OF CONTENTS
Page
Abstract………………………………………………………………………..…
ii
Dedication………………………………………………………………..………
iv
Acknowledgements………………………………………………………..……..
v
Vita…………………………………………………………………………..…… vi
List of Tables…………………………………………………………………….. viii
List of Figures…………………………………………………………….……… ix
Chapters:
1.
Introduction…………………………………………………………..…..
1
2.
Materials and Methods…………………………………………..……….. 4
3.
Manuscript…………………………………………………….………….. 6
4.
Conclusion………………………………………………………….…….. 18
List of References……………………………………………………….………… 19
Appendix A: Additional Figures………………..………………………………… 22
vii
LIST OF TABLES
Page
Table
3.1: Validity of Overall Weighted PAR Scores ………………………....………. 5
3.2: Intra-rater Reliability of Weighted PAR Scores…………………………….. 6
3.3: Inter-rater Reliability of Weighted PAR Scores…………….………………. 7
3.4: Time Difference of Digital and Plaster PAR Scores………………………... 8
3.5 : Calibration status of each rater……………………………………………... 10
3.6 : Intra-rater reliability of PAR components………………………………….. 12
3.7 : Inter-rater reliability of PAR components………………………………….. 14
viii
LIST OF FIGURES
Page
Figure
Figure 1:Anterior segment contact point displacement (Richmond et al.,
1992) 1…………………………………………………………………………… 23
Figure 2: Mixed dentition crowding assessment (Richmond et al.,
1992)1 ……………………………………………………………………………. 23
Figure 3: Buccal occlusion assessment (Richmond et al., 1992)1 ……….……… 24
Figure 4: Overjet assessment (Richmond et al., 1992) 1 ………………………...
24
Figure 5: Overbite assessment (Richmond et al., 1992) 1 ……………………….. 25
Figure 6: Midline assessment (Richmond et al., 1992) 1 ………………………... 25
Figure 7: PAR index conventions (Richmond et al., 1992)1 …………………… 26
Figure 8: PAR ruler (Richmond et al., 1992)1 ……………………………….… 27
Figure 9: PAR index component weightings (Richmond et al., 1992)1 …….….
27
Figure 10: Digital PAR Methodology …………..…………………………….....
28
Figure 11: Digital PAR Methodology …………..……………………………….
30
Figure 12: Digital PAR Methodology ………….………………………………... 32
Figure 13: Digital PAR Methodology …………..………………...……………..
33
Figure 14: Digital PAR Methodology …………..……………………………….. 35
ix
CHAPTER 1
INTRODUCTION
The modern orthodontic office has become dependent on computer technology.
There is a trend toward a paperless office, including patient charts and records. In
addition to digital photography and radiography, there has been growing adoption of
digital study models. Digital study models replace plaster models as a representation of
the patient’s occlusal and dental relationships. Computer software creates a pseudo 3dimensional image that can be manipulated to simulate plaster models. The digital
models can be magnified, rotated, and measured using proprietary software provided by
the manufacturer.
Several previous studies have investigated the accuracy and reliability of digital
models compared with plaster models.2-10 Santoro et al. (2003)2 found measurements on
digital models for tooth size and overbite to be slightly smaller (0.16-0.49mm) than
measurements on plaster models, but the difference was not deemed clinically significant.
Mullen et al. (2007)3 found no significant difference in performing Bolton analyses on
digital models compared with plaster models. Tomassetti et al. (2001)4 also found no
difference in Bolton analyses performed on digital models compared with those on
plaster models.
1
Zilberman et al. (2003)5 compared tooth size and arch width measurements
derived from plaster models and those derived from digital models. Measurements from
digital models were found to be valid and reproducible. Quimby et al. (2003)6 found
small differences in measurements of tooth size, arch length and width, overjet and
overbite between digital models and plaster models. These investigators concluded that
these differences were not clinically significant.
To fully utilize digital study models it is also necessary to visually assess spatial
relationships, in addition to making linear measurements of casts. Rheude et al. (2004)7
showed that practitioners produced identical diagnoses and treatment plans using digital
and plaster models almost 90% of the time. Whetten et al. (2005)8 found no statistical
difference in treatment planning patients with Class II malocclusions using digital or
plaster models.
Costalos et al. (2005)9 and Okunami et al. (2005)10 investigated how accurately
the American Board of Orthodontics objective grading system (ABO OGS) could be
applied to digital models. They reported that the alignment and buccolingual inclination
could not be accurately determined and concluded that the technology was not adequate
for scoring all parameters of the ABO OGS.
The Peer Assessment Rating (PAR) index is an occlusal index that was developed
to record the dental malocclusion at any developmental stage.11 Researchers, institutions
and clinicians use the PAR index to objectively measure pre-treatment and post-treatment
severity of malocclusion and the degree of improvement through treatment.12-17 The total
PAR score is derived from scoring the individual components of occlusion. These consist
of overbite, overjet, midline discrepancy, anterior segment alignment, and buccal
2
occlusion. The PAR index has been shown to be valid and reliable using plaster study
models.1, 18-23
Past studies by Mayers et al. (2005)24 and Stevens et al. (2006)25 have
investigated the reliability of PAR scores derived from digital models. The PAR scores
were found to be reliable and the possibility of developing a computer-based calibration
exercise was discussed.
The purpose of this study was to determine the validity and reliability of digital
and plaster PAR index scores using current standardized PAR calibration models.
The null hypotheses for this study are:
•
Ho(i): Intra-examiner scores from digital and plaster models are not reliable.
•
Ho(ii): Inter-examiner scores from digital and plaster models are not reliable.
•
Ho(iii): Weighted PAR scores derived from digital models and plaster models are
not valid when compared to the “gold standard scores” of the plaster models.
3
CHAPTER 2
MATERIALS AND METHODS
Sample Collection
The Ohio State University Institutional Review Board approved this experimental
protocol. Ten examiners were trained to score digital and plaster models using the PAR
index. The examiners included five first-year, and five second-year orthodontic residents
in graduate training at The Ohio State University. The five second-year residents had
been previously calibrated in the PAR index using plaster models eighteen months prior
to the study. The five first-year residents had not been previously calibrated in the PAR
index. There were seven male and three female examiners ranging in age from twenty-six
to thirty-one years old. The mean age of examiners was twenty-nine years old. Thirty
standardized plaster models used for training and calibrating examiners in the PAR index
were digitized using digital study model technology. Each model was scanned in
accordance with company procedure and a digital study model was produced from the
original plaster model (OrthoCAD, Cadent, Inc., Carlstadt, NJ). A training session was
held two weeks prior to data collection. PAR scoring methods were reviewed using
plaster models. PAR scoring methods using digital models were introduced and thirty
digital training models were scored during the training session.
4
Data Collection
Each examiner scored the thirty digital and thirty plaster models in one session.
The models were randomized and each examiner started on a different number. Five
plaster models were scored, followed by five digital models, continuing until all models
had been scored. The method of scoring the models using the PAR index followed the
guidelines (Figures 1-7) laid out by Richmond et al. (1992).1 Plaster casts were measured
with plastic PAR rulers (Figure 8). Digital models were scored using proprietary software
(OrthoCAD, Version 2.9, Cadent, Inc., Carlstadt, NJ). Total PAR score was determined
using the European weightings of the component scores (Figure 9). The amount of time
required to score each model was recorded by the examiners using a digital stopwatch.
Three weeks after the initial scoring session, all measurements on all models were
repeated by all examiners at a second scoring session to establish reliability. The models
were randomized again and scored in sets of five as in the first session. Only
measurements from the first scoring session were used to assess validity.
Statistical Methods
Plaster and digital scores were compared to the gold standard PAR scores for
validity using the intra-class correlation coefficient (ICC). Reliability also was evaluated
using the intra-class correlation coefficient. The time required to score each model was
analyzed using student’s t test.
5
CHAPTER 3
MANUSCRIPT:
Validity and Reliability of Peer Assessment Rating Index Scores of Digital
and Plaster Models
ABSTRACT
Introduction: The object of this research was to evaluate the reliability and validity of
Peer Assessment Rating (PAR) index scores derived from computer-based digital
models. Computer-based digital orthodontic models have been shown to be valid and
reliable when making measurements of tooth size and simple diagnoses of dental
malocclusions. The validity and reliability of PAR scores from digital models and plaster
models that have been previously scored and standardized for a calibration exercise for
the PAR index have not been established. Methods: Ten examiners were trained to score
digital and plaster models using the PAR index. Thirty standardized and previously
scored plaster PAR models used in PAR calibration were digitized using digital study
model technology. The plaster and digital calibration models were randomized and
scored by the ten examiners. All measurements were repeated by all examiners at a
second session. Results: Plaster and digital scores were compared to the gold standard
6
PAR scores for validity. The Intraclass Correlation Coefficients (ICC) for plaster models
ranged from
0.808-0.926. The mean ICC for plaster models was 0.891 with a 95% confidence interval
(CI) of 0.829-0.940. The ICC’s for digital models ranged from 0.846-0.922. The mean
ICC for digital models was 0.878 (CI 0.811-0.931). Intra-rater reliability was excellent
for plaster (ICC =0.919) and for digital models (ICC= 0.905). Inter-rater reliability was
also excellent for plaster (ICC=0.912) and digital models (ICC=0.883). Conclusions:
The PAR index scores derived from the digital models of calibration casts were valid and
reliable.
INTRODUCTION
The current orthodontic office increasingly incorporates computer technology.
The trend is toward a paperless office, including patient charts and records. In addition to
digital photography and radiography, there has been growing adoption of digital study
models. Digital study models replace plaster models as a representation of the patient’s
occlusal and dental relationships. Computer software creates a pseudo 3-dimensional
image that can be manipulated to simulate plaster models. The digital models can be
magnified, rotated, and measured using proprietary software provided by the
manufacturer.
Several previous studies have investigated the accuracy and reliability of digital
models compared with plaster models.2-10 Santoro et al. (2003)2 found measurements on
7
digital models for tooth size and overbite to be slightly smaller (0.16-0.49mm) than
measurements on plaster models, but the difference was not deemed clinically significant.
Mullen et al. (2007)3 found no significant difference in performing Bolton analyses on
digital models compared with plaster models. Tomassetti et al. (2001)4 also found no
difference in Bolton analyses performed on digital models compared with those on
plaster models.
Zilberman et al. (2003)5 compared tooth size and arch width measurements
performed on plaster models and those performed on digital models. Measurements on
digital models were found to be valid and reproducible. Quimby et al. (2003)6 found
small differences in measurements of tooth size, arch length and width, overjet and
overbite between digital models and plaster models. These investigators concluded that
these differences were not clinically significant.
To fully utilize digital study models it is also necessary to assess spatial
relationships, in addition to making linear measurements. Rheude et al. (2004)7 reported
that practitioners produced identical diagnoses and treatment plans using digital and
plaster models almost 90% of the time. Whetten et al. (2005)8 found no statistical
difference in treatment plans for patients with Class II malocclusions using digital or
plaster models.
Costalos et al. (2004)9 and Okunami et al. (2005)10 investigated how accurately
the American Board of Orthodontics objective grading system (ABO OGS) could be
applied to digital models. They reported that the alignment and buccolingual inclination
could not be accurately determined and concluded that the technology was not adequate
for scoring all parameters of the ABO OGS.
8
The Peer Assessment Rating (PAR) index is an occlusal index that was developed
to record the dental malocclusion at any developmental stage.11 Researchers, institutions
and clinicians use the PAR index to objectively measure pre-treatment and post-treatment
severity of malocclusion and the degree of improvement through treatment.12-17 The total
PAR score is derived from scoring the individual components of occlusion. These consist
of overbite, overjet, midline discrepancy, anterior segment alignment, and buccal
occlusion. The PAR index has been shown to be valid and reliable using plaster study
models.1, 18-23
Mayers et al. (2005)24 and Stevens et al. (2006)25 have reported the reliability of
PAR scores derived from digital models. The PAR scores were found to be reliable and
the possibility of developing a computer-based calibration exercise was discussed.
The purpose of this study was to determine the validity and reliability of digital
PAR index scores derived from PAR calibration models.
MATERIALS AND METHODS
The Ohio State University Institutional Review Board approved this experimental
protocol. Ten examiners were trained to score digital and plaster models using the PAR
index. The examiners included five first-year and five second-year orthodontic residents
in graduate training at The Ohio State University. The five second-year residents had
been previously calibrated in the PAR index using plaster models eighteen months prior
to the study. The five first-year residents had not been previously calibrated in the PAR
9
index. There were seven male and three female examiners ranging in age from twenty-six
to thirty-one years old. The mean age of examiners was twenty-nine years old. Thirty
standardized plaster models used for training and calibrating examiners in the PAR index
were digitized using digital study model technology. Each model was scanned in
accordance with company procedure and a digital study model was produced from the
original plaster model (OrthoCAD, Cadent, Inc., Carlstadt, NJ). A training session for the
examiners was held two weeks prior to data collection. PAR scoring methods were
reviewed using plaster models. PAR scoring methods using digital models were
introduced and thirty digital training models were scored during the training session.
Data Collection
Each examiner scored the thirty digital and thirty plaster models in one session.
The models were randomized and each examiner started on a different number. Five
plaster models were scored, followed by five digital models, continuing until all models
had been scored. The method of scoring the models using the PAR index followed the
guidelines (Figures 1-7) laid out by Richmond et al. (1992).1 Plaster casts were measured
with plastic PAR rulers (Figure 8). Digital models were scored using proprietary software
(OrthoCAD, Version 2.9, Cadent, Inc., Carlstadt, NJ). Total PAR score was determined
using the European weightings of the component scores (Figure 9). The amount of time
required to score each model was recorded by the examiners using a digital stopwatch.
Three weeks after the initial scoring session, all measurements on all models were
repeated by all examiners at a second scoring session to establish reliability. The models
were randomized again and scored in sets of five as in the first session. Only
measurements from the first scoring session were used to assess validity.
10
Statistical Methods
Plaster and digital scores were compared to the gold standard PAR scores for
validity using the intra-class correlation coefficient (ICC). Reliability also was evaluated
using the intra-class correlation coefficient. The time required to score each model was
analyzed using student’s t test.
RESULTS
Validity of Overall Weighted PAR Scores
The results of the overall weighted PAR scores for digital and plaster models
compared with the gold standard scores are summarized in Table 3.1. There was a high
correlation between PAR scores derived from plaster (ICC=0.891) and digital models
(ICC=0.878) with the gold standard PAR scores derived from plaster models.
ICC
LCB
UCB
Plaster Models
0.891
0.829
0.940
Digital Models
0.878
0.811
0.931
Table 3.1: Mean intraclass correlation coefficients (ICC), upper (UCB) and lower (LCB)
95% confidence limits for the total PAR scores for digital and plaster models compared
with the gold standard.
11
Intra-rater Reliability of Weighted PAR Scores
Table 3.2 summarizes the intra-rater reliability of PAR scores derived from digital
and plaster models using the intraclass correlation coefficient with 95% confidence
intervals. Mean intra-rater reliability was excellent for both plaster (ICC=0.919) and
digital (ICC=0.905) measurements.
ICC
LCB
UCB
Plaster Models
0.919
0.897
0.934
Digital Models
0.905
0.882
0.924
Table 3.2: Intraclass correlation coefficients (ICC), upper (UCB) and lower (LCB) 95%
confidence limits for the weighted PAR scores for digital and plaster models for intrarater reliability.
12
Inter-rater Reliability of Weighted PAR Scores
Table 3.3 summarizes the inter-rater reliability of PAR scores derived from digital
and plaster models using the intraclass correlation coefficient with 95% confidence
intervals. Mean inter-rater reliability was good for both plaster (ICC=0.912) and digital
(ICC=0.883) measurements.
ICC
LCB
UCB
Plaster Models
0.912
0.842
0.942
Digital Models
0.883
0.802
0.925
Table 3.3: Intraclass correlation coefficients (ICC), upper (UCB) and lower (LCB) 95%
confidence limits for the weighted PAR scores derived from digital and plaster models
for inter-rater reliability.
13
Time Difference of Digital and Plaster PAR Scores
Table 3.4 displays the difference in time required to score plaster and digital models.
PAR scoring on the plaster models (mean = 97.7s) was significantly faster than PAR
scoring on the digital models (mean = 133.7s).
Mean (s.d.)
(s)
p value
Plaster
97.7
(15.3)
Digital
133.7 (18.7)
Difference
-36.3 (18.75)
p<0.0001
Table 3.4: Mean time in seconds (s) and standard deviation (s.d.) required to PAR score
plaster and digital models.
14
PAR Calibration Pass Rates
Criteria for calibration in PAR index scoring are: root mean square error of less
than 5, mean difference between the rater and gold standard of less than 2 PAR points,
confidence interval of the difference and the 95% limits of agreement within +/- 12, and a
non-significant bias. For the purposes of this study only those raters meeting all criteria
passed, those that passed half of the criteria were classified as borderline, and those that
met less than half of the criteria failed. Table 3.5 displays the PAR calibration pass rates
of the examiners. On plaster models, one rater successfully calibrated, six were
borderline, and three failed. Using digital models, no raters successfully passed, three
were borderline, and seven failed.
15
Plaster models
Calibration Status
Digital models
Calibration Status
Previously
un-calibrated raters
1
Borderline
Fail
2
Borderline
Fail
3
Borderline
Fail
4
Borderline
Borderline
5
Borderline
Borderline
6
Fail
Fail
7
Pass
Fail
8
Fail
Fail
9
Fail
Fail
10
Borderline
Borderline
Previously
calibrated raters
Table 3.5 : Calibration status of each rater for plaster and digital models.
16
Intra-rater Reliability of PAR components
Table 3.6 summarizes the intra-rater reliability of PAR components derived from
digital and plaster models using the intraclass correlation coefficient with 95%
confidence intervals. All components had ICC greater than 0.7 except for buccal anteroposterior relationship of the posterior teeth for digital (ICC = 0.655) and plaster (ICC =
0.674) models and buccal-vertical relationship (posterior open-bite) for digital models
(ICC = 0.666).
17
Variable
Maxillary Anterior
Maxillary Anterior
Mandibular Anterior
Mandibular Anterior
Buccal A-P
Buccal A-P
Buccal Transverse
Buccal Transverse
Buccal Vertical
Buccal Vertical
Overjet
Overjet
Overbite
Overbite
Midline
Midline
Weighted Total
Weighted Total
Unweighted Total
Unweighted Total
Method
ICC (CI95 )
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
0.918 (0.934, 0.899)
0.915 (0.932, 0.894)
0.931 (0.944, 0.913)
0.908 (0.925, 0.884)
0.674 (0.731, 0.607)
0.655 (0.712, 0.581)
0.896 (0.916, 0.871)
0.856 (0.883, 0.821)
unity
0.666 (0.724, 0.598)
0.839 (0.868, 0.800)
0.790 (0.829, 0.743)
0.785 (0.824, 0.736)
0.821 (0.855, 0.781)
0.728 (0.778, 0.671)
0.749 (0.795, 0.695)
0.919 (0.934, 0.897)
0.905 (0.924, 0.882)
0.941 (0.952, 0.926)
0.899 (0.919, 0.875)
Table 3.6 : Intraclass correlation coefficients (ICC), upper (UCB) and lower (LCB) 95%
confidence limits for PAR components derived from digital and plaster models for intrarater reliability.
18
Inter-rater Reliability of PAR components
Table 3.7 summarizes the inter-rater reliability of PAR components derived from
digital and plaster models using the intraclass correlation coefficient with 95%
confidence intervals. All components had ICC greater than 0.7 except for buccal A-P for
digital (ICC = 0.575) and plaster (ICC = 0.613) models, buccal-Vertical for digital
models (ICC = -0.004), and midline deviations for both digital (ICC = 0.690) and plaster
models (ICC = 0.664).
19
Variable
Maxillary Anterior
Maxillary Anterior
Mandibular Anterior
Mandibular Anterior
Buccal A-P
Buccal A-P
Buccal Transverse
Buccal Transverse
Buccal Vertical
Buccal Vertical
Overjet
Overjet
Overbite
Overbite
Midline
Midline
Weighted Total
Weighted Total
Unweighted Total
Unweighted Total
Method
ICC (CI95 )
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
Plaster
Digital
0.816 (0.883, 0.709)
0.716 (0.810, 0.574)
0.909 (0.935, 0.825)
0.878 (0.911, 0.770)
0.613 (0.717, 0.436)
0.575 (0.663, 0.370)
0.869 (0.920, 0.789)
0.888 (0.933, 0.819)
unity
-0.004 (0.072, -0.043)
0.838 (0.901, 0.748)
0.775 (0.859, 0.661)
0.736 (0.833, 0.615)
0.723 (0.822, 0.595)
0.664 (0.787, 0.536)
0.690 (0.798, 0.555)
0.912 (0.942, 0.842)
0.883 (0.925, 0.802)
0.915 (0.941, 0.840)
0.812 (0.882, 0.706)
Table 3.7 : Intraclass correlation coefficients (ICC), upper (UCB) and lower (LCB) 95%
confidence limits for PAR components derived from digital and plaster models for interrater reliability.
20
DISCUSSION
PAR index scores derived from digital models are valid and reliable compared with
gold standard PAR scores derived from plaster calibration models. Intra- and Inter-rater
reliability for plaster and digital models was high.
The results of this study support earlier reports by Mayers et al. (2005)24 and Stevens et
al. (2006)25 who showed no significant differences between overall PAR scores derived
from digital and plaster models.
While reliability and validity of scores derived from digital and plaster models were
high, only one examiner was able to successfully calibrate on plaster models and no
examiners were able to calibrate using digital models. This reflects the stringent
requirements of PAR calibration.
The examiners who had previously calibrated in the PAR index did not score better
than examiners who had never successfully calibrated. A total of eighteen months had
elapsed from the time they had previously calibrated, with limited experience utilizing
the PAR index in the intervening months. It is possible that their skills in scoring models
for the PAR index had lapsed through infrequent use in this time period. This emphasizes
the need for continued re-calibration after the initial training is complete.
Overall trends suggested more difficulty meeting calibration criteria using scores
derived from digital models than plaster models. Examiners indicated that they felt it was
difficult to manipulate the virtual models, compared to physically handling the plaster
models. This may have contributed to the pattern of lower success using digital models.
The components that had the lowest reliability were buccal occlusion in the anteroposterior and vertical directions, as well as midline discrepancies. This is similar to
21
previous findings by Mayers et al. (2005)24 who also found buccal occlusion to be less
reliable than other components.
Examiners spent about one-half minute less time scoring plaster models than they did
scoring the digital models. It is arguable how much of a difference this amount of time
would make in everyday application of the PAR index. Further, the effective time
difference in scoring the models might well be significantly reduced by the increased
time associated with retrieving and replacing the plaster models.
Currently an examiner must be trained to score a standardized set of plaster models
in order to become calibrated in the PAR index. This involves physically travelling to a
location where such training can be obtained. The location must possess a set of gold
standard calibration models. The requirement for such firsthand training potentially limits
the numbers of examiners who can be trained, calibrated and re-calibrated.
This study used digital representations of those same standardized PAR calibration
models. It has been demonstrated that scores derived from those digital models are valid
and reliable. However, additional training may be required to successfully meet the strict
requirements of calibration using scores derived from digital models.
Limitations of this study include the relatively small sample size. Also, the
inexperience of the examiners and their inability to meet the requirements for calibration
using either plaster or digital models make it difficult to draw conclusions on the efficacy
and efficiency of calibration using digital models.
The present study was performed using examiners with limited orthodontic
experience and PAR scoring experience. Future studies may benefit from using more
experienced examiners who had recently calibrated or re-calibrated in the PAR index.
22
It is important to continue development of a computer based protocol that would
allow training, calibration, and recalibration in the PAR index using digital models and a
computer-based calibration exercise. With the use of the internet, this training could be
easily applied anywhere in the world, for both individuals and institutions. Candidates
could become initially calibrated in the PAR index, and previously calibrated examiners
could recalibrate using the computerized exercise.
CONCLUSIONS
The PAR index scores derived from digitized PAR calibration models are valid
and reliable. However, additional training may be required to help candidates meet the
requirements of PAR calibration.
23
CHAPTER 4
CONCLUSIONS
This study examined the validity and reliability of digital and plaster PAR index
scores using current standardized PAR calibration models. Scores derived from plaster
and digital models were valid and reliable. However, successful PAR calibration using
digital models was not achieved by any of the subjects.
Conclusions:
•
Intra-examiner reliability of scores from digital and plaster models is high.
•
Inter-examiner reliability of scores from digital and plaster models is high.
•
Validity of scores from digital and plaster models is high.
24
REFERENCES
1. Richmond S, Shaw W, O'Brien K, Buchanan I, Jones R, Stephens C, et al. The
development of the PAR Index (Peer Assessment Rating): reliability and validity. Eur J
Orthod. 1992;14(2):125.
2. Santoro M, Galkin S, Teredesai M, Nicolay O, Cangialosi T. Comparison of
measurements made on digital and plaster models. Am J Orthod. 2003;124(1):101.
3. Mullen SR, Martin CA, Ngan P, Gladwin M. Accuracy of space analysis with emodels
and plaster models. Am J Orthod Dentofacial Orthop. 2007;132(3):346-52.
4. Tomassetti J, Taloumis L, Denny J, Fischer J,Jr. A comparison of 3 computerized
Bolton tooth-size analyses with a commonly used method. Angle Orthod.
2001;71(5):351.
5. Zilberman O, Huggare J, Parikakis K. Evaluation of the validity of tooth size and arch
width measurements using conventional and three-dimensional virtual orthodontic
models. Angle Orthod. 2003;73(3):301.
6. Quimby M, Vig K, Rashid R, Firestone A. The accuracy and reliability of
measurements made on computer-based digital models. Angle Orthod. 2004;74(3):298.
7. Rheude B, Sadowsky P, Ferriera A, Jacobson A. An evaluation of the use of digital
study models in orthodontic diagnosis and treatment planning. Angle Orthod.
2005;75(3):300.
8. Whetten J, Williamson P, Heo G, Varnhagen C, Major P. Variations in orthodontic
treatment planning decisions of Class II patients between virtual 3-dimensional models
and traditional plaster study models. Am J Orthod. 2006;130(4):485.
9. Costalos P, Sarraf K, Cangialosi T, Efstratiadis S. Evaluation of the accuracy of digital
model analysis for the American Board of Orthodontics objective grading system for
dental casts. Am J Orthod. 2005;128(5):624.
25
10. Okunami TR, Kusnoto B, BeGole E, Evans CA, Sadowsky C, Fadavi S. Assessing
the American Board of Orthodontics objective grading system: Digital vs plaster dental
casts. American Journal of Orthodontics & Dentofacial Orthopedics. 2007;131(1):51-6.
11. Richmond S, O'Brien K, Buchanan I, Burden D, editors. An introduction to Occlusal
Indices. ; 1992.
12. Onyeaso CO, Begole EA. Orthodontic treatment--improvement and standards using
the peer assessment rating index. Angle Orthod. 2006 Mar;76(2):260-4.
13. Deguchi T, Honjo T, Fukunaga T, Miyawaki S, Roberts WE, Takano-Yamamoto T.
Clinical assessment of orthodontic outcomes with the peer assessment rating, discrepancy
index, objective grading system, and comprehensive clinical assessment. Am J Orthod
Dentofacial Orthop. 2005 Apr;127(4):434-43.
14. Dyken RA, Sadowsky PL, Hurst D. Orthodontic outcomes assessment using the peer
assessment rating index. Angle Orthod. 2001 Jun;71(3):164-9.
15. King GJ, McGorray SP, Wheeler TT, Dolce C, Taylor M. Comparison of peer
assessment ratings (PAR) from 1-phase and 2-phase treatment protocols for Class II
malocclusions. Am J Orthod Dentofacial Orthop. 2003 May;123(5):489-96.
16. Cassinelli AG, Firestone AR, Beck FM, Vig KW. Factors associated with
orthodontists' assessment of difficulty. Am J Orthod Dentofacial Orthop. 2003
May;123(5):497-502.
17. Hamdan AM, Rock WP. An appraisal of the Peer Assessment Rating (PAR) Index
and a suggested new weighting system. Eur J Orthod. 1999 Apr;21(2):181-92.
18. DeGuzman L, Bahiraei D, Vig KW, Vig PS, Weyant RJ, O'Brien K. The validation of
the Peer Assessment Rating index for malocclusion severity and treatment difficulty. Am
J Orthod Dentofacial Orthop. 1995 Feb;107(2):172-6.
19. Firestone AR, Beck FM, Beglin FM, Vig KW. Evaluation of the peer assessment
rating (PAR) index as an index of orthodontic treatment need. Am J Orthod Dentofacial
Orthop. 2002 Nov;122(5):463-9.
20. McGorray SP, Wheeler TT, Keeling SD, Yurkiewicz L, Taylor MG, King GJ.
Evaluation of orthodontists' perception of treatment need and the peer assessment rating
(PAR) index. Angle Orthod. 1999 Aug;69(4):325-33.
21. Brown R, Richmond S. An update on the analysis of agreement for orthodontic
indices. Eur J Orthod. 2005 Jun;27(3):286-91.
26
22. Shaw WC, Richmond S, O'Brien KD. The use of occlusal indices: a European
perspective. Am J Orthod Dentofacial Orthop. 1995 Jan;107(1):1-10.
23. Richmond S, Shaw WC, Roberts CT, Andrews M. The PAR Index (Peer Assessment
Rating): methods to determine outcome of orthodontic treatment in terms of
improvement and standards. Eur J Orthod. 1992 Jun;14(3):180-7.
24. Mayers M, Firestone AR, Rashid R, Vig KW. Comparison of peer assessment rating
(PAR) index scores of plaster and computer-based digital models. Am J Orthod
Dentofacial Orthop. 2005 Oct;128(4):431-4.
25. Stevens DR, Flores-Mir C, Nebbe B, Raboud DW, Heo G, Major PW. Validity,
reliability, and reproducibility of plaster vs digital study models: comparison of peer
assessment rating and Bolton analysis and their constituent measurements. Am J Orthod
Dentofacial Orthop. 2006 Jun;129(6):794-803.
27
APPENDIX A
ADDITIONAL FIGURES
28
Anterior Segment Contact Point Displacement
Score
Displacement
0
1
2
3
4
5
0mm – 1mm
1.1mm-2mm
2.1mm-4mm
4.1mm-8mm
Greater than 8mm
Impacted teeth (if space between teeth
is 4mm or less)
Figure 1: Anterior segment contact point displacement (Richmond et al., 1992)1
Mixed Dentition Crowding assessment
Upper
Distal of Canine- Mesial of 1st molar should total 22mm
(impaction <= 18mm)
Lower
Distal of Canine- Mesial of 1st molar should total 21mm
(impaction <= 17mm)
Figure 2: Mixed dentition crowding assessment as (Richmond et al., 1992) 1
29
Buccal Occlusion Assessments
Antero-Posterior
Vertical
Transverse
0
Good interdigitation,
Class I, II or III
0 No lateral open bite
1
Less than half unit from
full interdigitation
2
Half a unit(cusp to cusp)
1 Lateral open bite on at
least two teeth greater
than 2mm
0 No crossbite
1 Crossbite tendency
2 Single tooth in
crossbite
3 > 1 tooth in
crossbite
4 > 1 tooth in scissors
bite
** Temporary developmental stages and submerging deciduous teeth are excluded.
Figure 3: Buccal occlusion assessment (Richmond et al., 1992) 1
Overjet Assessment
Overjet
0 0-3mm
1 3.1-5mm
2 5.1-7mm
3 7.1-9mm
4 greater than 9mm
Anterior Crossbites
0 No crossbite
1 One or more teeth edge to edge
2 One single tooth in crossbite
3 Two teeth in crossbite
4 More than two teeth in crossbite
Figure 4: Overjet assessment (Richmond et al., 1992) 1
30
Overbite Assessment
Overbite
0
<= 1/3 coverage of lower incisor
1
1/3-2/3 coverage of lower incisor
2
> 2/3 coverage of lower incisor
3
>= full tooth coverage
Open bite
0 No open bite
1 Open bite <=1mm
2 1.1-2mm
3 2.1-3mm
4 >= 4mm
Figure 5: Overbite assessment (Richmond et al., 1992) 1
Midline Assessment
0 Coincident and up to 1/4 lower incisor width
1 1/4- 1/2 lower incisor width
2 > 1/2 lower incisor width
Figure 6: Midline assessment (Richmond et al., 1992) 1
31
PAR Index Conventions
General:
1. All scoring is accumulative.
2. There is no maximum cut off level.
3. Increased overjets, contact point displacements etc. associated with poor
restorative work are not recorded.
4. Contact point displacements between deciduous teeth and between deciduous
teeth and permanent teeth are not recorded.
5. Spaces are not recorded if the patient is to receive a prosthetic replacement.
Canines:
1. Ectopic canines which have erupted in the palate should be recorded as an
anterior crossbite in the overjet section.
Impactions:
1. If a tooth is unerupted due to insufficient space or is ectopic it is recorded as
impacted.
Incisors:
1. Spacing in the anterior segment resulting from extraction, agenesis or
avulsion of incisors or canines is recorded using the following protocol:
a) If orthodontic space closure is appropriate then the space is recorded.
b) If increasing the space is appropriate(for prosthetic replacement) then
the space is only recorded if it is <= 4mm.
2. When recording an overjet, if the tooth falls on the line the lower score is
recorded.
3. If a lower incisor has been extracted or is missing an estimate of the lower
dental midline is made.
Figure 7: PAR index conventions (Richmond et al., 1992) 1
32
Figure 8: PAR ruler (Richmond et al., 1992) 1
Components
Upper and lower anterior segment
Left and right buccal occlusions
Overjet
Overbite
Centreline
Weightings
x1
x1
x6
x2
x4
Figure 9: PAR index component weightings (Richmond et al., 1992) 1
33
DIGITAL PAR Scoring Methodology
I.Anterior Segment Contact Point Displacement
- Select the “Diagnostics” icon from the OrthoCAD toolbar
a.
↓
-Use the “Teeth Width” function to measure the shortest distance
between contact points of the anterior teeth. Record the PAR score.
-Use the planar adjustments to ensure that the measurement is
made parallel to the occlusal plane.
Figure 10: Digital PAR Methodology
34
b.
HELPFUL HINTS
-Magnify the image.
-Turn the model to the sides to
confirm the measurement is parallel
to the occlusal plane
Figure 10: Digital PAR Methodology
35
II. Buccal Occlusion
-Select the full screen “Frontal View” from the toolbar.
-Examine the A-P, vertical and transverse
occlusion and record the PAR scores.
a.
Figure 11: Digital PAR Methodology
36
b.
HELPFUL HINTS
-Use the “View Control” box to manipulate the models as
needed.
-Magnify the image if needed.
-The sides and back of the model will “disappear” when
the models are parallel to the viewer
-Crossbites are best detected by tilting the casts both up
and down
Figure 11: Digital PAR Methodology
37
III. Overjet
-Select the “Diagnostics” icon from the toolbar
-Use the “OB/OJ” function to measure overbite and overjet.
-Select the incisor with the largest overjet. Measure and record the PAR score.
HELPFUL HINTS
-If there are incisors in crossbite as well as in
positive overjet both conditions are measured and
added together.
Example:
A lateral incisor in crossbite=2
Central incisor with overjet of 4mm=1
Total PAR score=3
Figure 12: Digital PAR Methodology
38
IV. Overbite
-Use the “OB/OJ” function to measure overbite and overjet.
-Select the incisor with the largest overbite or openbite.
-Measure and record the PAR score.
a.
Figure 13: Digital PAR Methodology
39
b.
HELPFUL
HINTS
- Clicking on the
Maxillary cast in
the “View
Control” box will
cause it to
disappear. Click
it several times to
best visualize
where the
overbite is the
deepest.
Figure 13: Digital PAR Methodology for OrthoCAD
40
V. Midline Assessment
-Use the “OB/OJ” function to assess the midline.
-Measure and record the PAR score.
HELPFUL HINTS
-Place the green line through the maxillary midline.
-Click on the maxillary cast in the “View Control” box, causing
it to disappear.
Figure 14: Digital PAR Methodology for OrthoCAD
41