Download An Approach Finding the Right Tolerance Level for Clinical Data Acceptance

Paper P024
An Approach Finding the Right Tolerance Level for Clinical Data
Acceptance
Karen Walker, Walker Consulting LLC, Chandler Arizona
ABSTRACT
Highly broadcasted zero tolerance initiatives for database error involving clinical trials are discussed ad
infinitum among pharmaceutical research executives. Their cause is construed as 100% audit of efficacy
variables and serious adverse events to prove the quality of the data used in the clinical trial is good. For
some clinical trials studies databases are too large to perform an audit of all the efficacy and safety
variables. Therefore, an audit proving a large database acceptable without a structured approach may be
dubious at best. For instances such as this, it is reasonable to employ a systematic approach for finding the
right level of tolerance for clinical data acceptance.
This approach must test the following markers:
tolerance limit, error rate, edit checks, manual review, sample sizes, and rank of sampling schemes; given
the tolerance limit is the amount of meaningful data communicated properly. In this paper we will explore
the use of a statistical inference to determine how good the error rate is with regard to specific tolerance
limits. We will set an acceptable level of quality or tolerance limit that certifies the elements converted and
stored in the database match all sources of information.
INTRODUCTION
The purpose of this presentation is to add rhyme or reason to the task of validating a clinical study database.
For the scope of this discussion we will examine a typical collection of case report forms converted to the
database by doing manual review. From a random sample of records; we will discover how many records
match with the original source data, and how many records are flawed. With further evaluation of the data
elements using a computerized edit check, we will learn more information about the data. Thus errors can
be corrected without compromising data integrity. A test of hypothesis will be employed as a systematic way
to tell us weather or not to accept the database. Finally I will demonstrate how to arrive at the same
conclusion using as SAS procedure.
TOLERANCE LIMIT
The tolerance level is a value prespecified by clinical research management. They usually impose a zero
tolerance level being under constraints of executive management. Clinical data management is lucky if the
tolerance level is within the bounds of the rate of errors in the database. The error rate is calculated from
the amount of meaningful data communicated properly. This covers all obvious mismatches with source
data as well as data incongruencies. Consider a database having 100,000 records, after checking 1000, if
only 10 errors are found the estimated error rate is 10 for every 1000 records checked. This error rate is
called “P hat” (ˆp =0.01).
The tolerance level is compared to the error rate (ˆp ) because to satisfy our objective the errors must be
less. If the tolerance level is zero, it can be difficult to prove any error rate is lower.
Therefore, it is
reasonable to consider the tolerance level as the limit as the true error rate (p) approaches zero. Thus,
tolerance level is better understood as the “Tolerance Limit” and will be subsequently denoted by “P knot”
(p0≈0). My undergraduate calculus professor would express the tolerance limit in these terms:
p0 = LIM p
PÆ 0
ERROR RATE
The number of data items converted to the database is checked against the original source information. The
error rate is the ratio of mismatched observations over the total observations checked. In our paper the true
error rate is denoted as (p). As a subject undergoes treatment during a clinical study, case report
questionnaires are filled out. The number of discrepancies from what was written on the form and that
converted to the database adds up as mismatches and increases the true error rate. For example, case
report form “INFORMED CONSENT” contains information about a subject who has agreed to participate in a
study. We should see the subject’s initials, a unique identifier, and date the consent was given. If not all
this data is converted to the database correctly it counts as an error.
ERROR RATE
Also note there are other types of errors that arise from situations conducting the clinical trial. For instance,
consider a randomized double blind clinical trial for a drug that generates stem cells. This study measures
growth of certain cells based on dose group. We have a subject on this study that has experienced
logarithmic growth of cells for that group. When an investigator mistakenly reported an adverse event a
dose modification is done. Now on a different time line, and with a different dose the logarithmic growth is
stopped, and dates on the CRF pages are inconsistent for that group. This is another type error, which will
be addressed in my next paper.
Case Report Book
01
02
03
04
05
06
07
08
09
10
INFORMED CONSENT
INCLUSION /EXCLUSION CRITERIA
MEDICAL HISTORY
PE / VITAL SIGNS
DOSE ADMINISTRATION
LABS
COLONY STIMULATING FACTOR
ADVERSE EVENTS
CONCOMITANT MEDICATIONS
STUDY COMPLETION/TERMINATION
Clinical Database
01111100
010010100
110011001
000001111
001111000
111000111
MANUAL REVIEW
To insure the correctness of the data that has been converted to the database from the source, the data is
audited. Every item from the source is compared to every item in the database manually.
EDIT CHECKS
Queries are small programs created to perform comparisons of source data to that which is in the database.
This kind of edit checking serves well as it counts the number of errors no matter the size of the database. It
will systematically alert clinical data management of errors.
A simple edit check query routine may
resemble the following:
Proc SQL;
Create table informedyes as
Select subjects, 1 as errorflg
From INFCNST
Where INFCNDT ^= .;
Quit;
This query will execute SQL running on a UNIX server, under WINDOWS, or even MVS to name just a few.
The routine asks the system to create a table called “INFORMEDYES” and write to the table all SUBJECTS
from the INFCNST database table having no informed consent date. You will recall in a previous section of
this paper, having no informed consent date represents a discrepancy and counts as an error. In order to
trigger an alert we tag the records by setting the “errorflg” variable to 1. This table can be used to generate
“Alerts” for clinical data management. A list of alerts is used to ascertain the information necessary to
correct the error. This will reduce the true error rate p.
EDIT CHECKS
A routine is created here using SAS™ to count the number of errors or alerts in the database from a table
ALERTS. The SAS routine computes the sum of the errors, the number of records, and the standard error
for a normal distribution. These variables are stored in a SAS table QUALITY for later use.
proc means data=db.alerts noprint;
var errorflg;
output out=quality n=totalrec
max=maxerror;
run;
stderr=se
sum=error
min=minerror
SAMPLE SIZES
If we checked each and every record in our clinical database we would know the absolute number of
differences from the original source data in the converted database. The error rate (p) calculated is
undisputable. Bear in mind that key clinical data for efficacy or safety should be checked in this manner,
however for large databases it is not always feasible to do this. It’s fair to make statistical inference based
upon a carefully selected portion of observations, in other words from a sampling of the database. The error
rate becomes the number of incorrect items over not the total possibilities but over the number checked (ˆp).
APPROACH
Our objective is to prove to management that the clinical database is of the highest quality possible. We will
show this by presenting a test of hypothesis that supports acceptance of the database.
Consider a
database audited 100% of data elements with no other issues the test is proven if the following is true:
H0:
^p < p0 -- Error rate is less than the Tolerance Limit
^p ≥ p0 -- Error rate is greater than the Tolerance Limit (no good!)
RANK OF SAMPLING SCHEMES
Nothing so far has been determined about the distribution of errors for our clinical database. We do know
that it is very likely that for a large database there will be errors. If not in exact data elements there will be
errors in CRF casebooks. Since our errors are base upon the number of incorrect items over the number
checked, close attention must be paid to the number checked. We know that to make a fair and unbiased
statistical inference about the database as a whole we must give careful consideration to how we select
samples from the database. Therefore, we have choices to make about the sampling distribution which will
get us where we want to be. From the following sampling schemes:
random samples,
casebook samples
There little influence upon the outcome of the hypothesis test. So let’s move on to the statistical inference
test.
MODIFIED APPROACH
The strength of our approach depends upon how sure we are about the errors. For a complete 100% audit,
the hypothesis test gives the strongest assurance the statistical inference is good. However, we need a
systematic approach where it isn’t required to do 100% audit to prove the statistical inference is good. We
must now consider how likely it is for the inference to breakdown when using an estimated error rate.
Factors affecting the precision of the hypothesis test other than the error rate and prespecified tolerance limit
are: alpha (α) type I errors, or beta (β) type II errors. The alpha error is the chance that we could falsely
reject the null hypothesis H0 . The beta error is the chance we could falsely accept the null hypothesis H0 .
We know we can control type I and type II errors because the number of errors in a sample from a finite
population is hypogenometirc, especially for situations where the sample size is large. Furthermore, if we
choose a large enough sample size, and let the tolerance limit approach zero but not be exactly zero we can
use a standard normal distribution which yields the following standard error :
SE=
p(1-p) N-n
N
N-1
Using a standard normal distribution we have full control of type I & type II errors:
Type I = α = 0.5
Type II = β = 0.5
Using a zscore of (1- α)100% we have…
With a standard normal distribution we gain better control over the issues affecting our statistical inference.
We have definitions for the standard error(SE), alpha (α) , beta (β) , true error rate (p), calculated error rate
(ˆp) and the tolerance limit (p0). Therefore the null hypothesis that the true error rate is greater than or equal
to the tolerance limit can be tested against the alternative hypothesis that the true error rate is less than the
tolerance limit.
H0:
p≥ p0 vs.
H1: p< p0
Where p0 has been redefined as:
p0 = p0 ± SE*zscore
SAS PROCEDURE
***************************************************************************
*
Purpose :
DATABASE ACCEPTANCE
*
*
*
*
*
*
Description: This program pulls in database discrepancy counts and *
*
compares them to Acceptable Quality Standards per SOP *
*
*
*
inputs:
db.quality.sas7bdat
*
*
error
= error flag
*
*
totalrec
= number of observations in DB
*
*
phat
= error rate
*
*
se
= Standard Error
*
*
l_se
= log of the standard error
*
*
pknot
= tolerance limit
*
*
*
options ls=132 ps=60 nodate nonumber
libname db “~”;
*** input quality SAS dataset;
%let aplha = 0.05
%let beta = 0.05
data pknot;
*** P knot approaches zero;
input pknot ;
cards;
.005
.004
.003
.002
.001
.0005
.0004
.0002
.0001
run;
data test;
merge db.quality pknot;
l_se = log(se);
phat = error/totalrec;
zscore= probit(1-&alpha);
lower = pknot-se*zscore;
loglw = exp(log(pknot)-1_se*zscore);
if (phat < lower) or (phat < loglw) then
h0 = “Accept”;
else if phat => pknot then
h0 = “Reject”;
else
h0 = “Please take next observation”;
run;
*** Print out Error Rate and Decision ***;
proc print data=test noobs label;
var phat lower loglw ho;
label phat
= ‘Error Rate for the Database’
lower = ‘Lower Bound’
1oglw = ‘Log of Lower Bound’
ho
= ‘Result of Hypothesis Test’
;
run;
CONCLUSION
We’ve learned that validating a large database can be proven by using a statistical test of hypothesis if we
consider that the chance of making an error or falsely reporting a clinical study database acceptable
depends upon the true error rate, the tolerance limit, and the selection of the sample size. Make fully use of
manual review, and edit checks to ascertain the true error rate p. However extreme caution must be taken
and research must be done to understand the exact sampling size and for a large database the distribution
must be normal.
REFERENCES
1.
Rondel RD, Varley SA, Webb C F. Clinical Data Management. 2nd edition New York, NY: John
Wiley and Sons;2000
2.
Kirk Paul Lafler, PROC: SQL Beyond the Basics Using SAS SAS Books by Users, Cary NC:2004
3.
Zhang, Drug Information Journal, Volume 38 Number 4, 2004 ISSN 0092-8615 pages 371-386
ACKNOWLEDGMENTS
I wish to thank God above who keeps me safe. Thanks for giving us Christ
CONTACT INFORMATION
Karen Walker
Walker Consulting LLC
6481 West Gary Drive
Chandler Arizona 85226
480.227.7047
[email protected]
SAS and all other SAS Institute Inc. product or services names are registered trademarks or trademarks of
SAS Institute Inc. in the USA and other countries. ® indicates USA registration.
Other brand and product names are trademarks of their respective companies.