Download Lab 2

Lab 2 – Data Preprocessing with Genetic Data (source: KDnuggets)
The process of data mining frequently has many small steps that all need to be done correctly to get good
results. However tedious these steps may seem, the goal is often a worthy one. With this particular data the
goal is to help make an early diagnosis for leukemia, a common form of cancer, and one should take care to
process it carefully.
We will be using Excel and R to explore and preprocess a genetic data set, though you should feel free to use
software that you are familiar with. Though R is not useful for certain pieces, it’s benefit lies in the ability to
explore the data during and after preprocessing.
The data for this lab comes from the pioneering work by Todd Golub et al at MIT Whitehead Institute (now
MIT Broad Institute). The analysis done by the MIT biologists is described in their paper Molecular
Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring (pdf).
The data is contained in a zip file here:
http://www.kdnuggets.com/data_mining_course/data/ALL_AML_original_data.zip
It contains 3 files –
table_ALL_AML_samples.txt
data_set_ALL_AML_train.txt
data_set_ALL_AML_independent.txt
The “samples” file has sample collection, treatment, and diagnosis information. The “train” and
“independent” text files contain gene information for each of the samples. Both train and test (“independent”)
datasets are tab-delimited files with 7130 records. The train file should have 78 fields and test 70 fields. The
first two fields are Gene Description (a long description like GB DEF = PDGFRalpha protein) and Gene
Accession Number (a short name like X95095_at). The remaining fields are pairs of a sample number (e.g.
1,2,..38) and an Affymetix "call" (P is gene is present, A if absent, M if marginal).
You can think of the training data as a very tall and narrow table with 7130 rows and 78 columns, but you
should note that it is "sideways" from machine learning point of view - the attributes (genes) are in rows, and
observations (samples) are in columns. This is the standard format for microarray data, but to use with
machine learning algorithms like WEKA, we will need to do "matrix transpose" (flip) the matrix to make files
with genes in columns and samples in rows.
Here is a small extract :
Gene Description
Gene Accession Number
1
call
2
call
...
GB DEF = GABAa receptor alpha-3
subunit
A28102_at
151
A
263
P
...
...
AB000114_at
72
A
21
A
...
...
AB000115_at
281
A
250
P
...
...
AB000220_at
36
A
43
A
...
Since we will use the training set to create the class predictor and the test set to check it, both sets need to be
cleaned. In general, it is good practice to create separate files for each step.
After each step, report the number of fields and records. When in R, you can do this with nrow and ncol
commands. Additionally include what the first few rows look like using the “head” R command (when
appropriate).
You may also wish to export the files to text or csv files between steps.
1. Obtain the zip file containing the data. Unzip the data and rename the train file to
ALL_AML_grow.train.orig.txt and test file to ALL_AML_grow.test.orig.txt
Convention: use the same file root for files of similar type and use different extensions for different versions
of these files. Here "orig" stands for original input files and "grow" stands for genes in rows. We will use
extension .tmp for temporary files that are typically used for just one step in the process and can be deleted
later.
Data Reduction in Excel: (Note, you can do this in R using commands such as grep.)
Open the training set txt file to see what the data looks like. Select all of the data and copy it into Excel.
Excel should recognize that the data is tab delimited and properly place the data columns into separate
columns in Excel. If not, you may need to “paste special” and choose delimited.
2. Classify the attributes as binary, discrete, and continuous. Additionally classify them as qualitative
(nominal or ordinal) or quantitative (interval or ratio).
3. Initial Microarray Cleaning Steps
a. Remove the initial records with Gene Description containing "control". (Those are Affymetrix
controls, not human genes). Call the resulting files ALL_AML_grow.train.noaffy.tmp
How many control records did you delete?
b. Remove the first field (long description) and the "call" fields, i.e. keep fields numbered 2,3,5,7,9,...
c. Transpose the data. (Note this could be done in R using transposed.frame <- t(old.frame), though
care has to be taken due to changes in variable names.)
Use extensions gcol.test.tmp and gcol.train.tmp ("gcol" stands for genes in columns). These files
should each have 7071 fields, and 39 records in "train", 35 records in "test" datasets.
d. Export the file to a tab delimited txt file.
Preprocessing in R:
To read a text file into R, use the following command:
framename <- read.table(“file_name.txt”, header=T, sep=”\t”)
Note: You may need to change your working directory.
If it is imported without a problem, R will not say anything after the command. To check the data and see
what the beginning looks like type into the command prompt head(framename).
Unless you take any special action, read.table reads all the columns as character vectors and then tries to select
a suitable class for each variable in the data frame. It tries in turn logical, integer, numeric and complex,
moving on if any entry is not missing and cannot be converted. If all of these fail, the variable is converted to
a factor.
4. Import the data. Notice that R renamed the attributes. Describe what happened.
5. Inspect the data you just imported with the following commands:
nrow(framename) = number of rows (excluding the header)
ncol(framename) = number of columns
attributes(framename) gives the column and row names and the data class.
summary(framename) gives the count by value for qualitative attributes and five-number summary
for quantitative attributes
6. Microarray Cleaning Continued
a. Change "Gene.Accession.Number" to "ID" in the first record. There are a couple of ways to do this.
One is to create a variable called “ID” with the same values and then delete
“Gene.Accession.Number”
Another way is to install the package “plyr” and then use the following command:
framename <- rename(framename, c(oldname="newname"))
To download and install a package, use the following commands:
install.packages(‘plyr’)
library(plyr)
b. Normalize the data: for each value, set the minimum field value to 20 and the maximum to 16,000.
(Note: The expression values less than 20 or over 16,000 were considered by biologists unreliable for
this experiment.)
Indicate that you have done this with the extensions grow.train.norm.tmp and grow.test.norm.tmp
Sample R code: normal <- df(df$ID, replace(df[,2:7071],df[,2:7071]<20,20), df$ID.class)
c. Import the file table_ALL_AML_samples.txt (you may want to split it into 2 text files – one for
“train” and one for “independent”). You will need to rename the first column to be ID, and you
probably want to rename the second as well (for example ALL.AML or ID.class). We will only be
using the first 2 columns/attributes from this file, so you have the option of altering the text file before
importing or only keeping the first 2 columns once imported (using something like new.df <subset(df, select = c(``ID’’,``ID_class’’)) ). Indicate with idclass.train.txt and idclass.test.txt.
d. Merge the files you just created with the appropriate gene data files by ID so that the ALL/AML class
information is now available. To merge data frames in R use:
New.train <- merge(df1 , df2 , by="ID", all=TRUE)
How many of the individuals are classifies as ALL and how many as AML in each of the training and
test files?
e. Export the file as a csv file.
Here is R code for exporting data:
write.table(dataframe.name, file = "filename.txt", append = FALSE, quote = FALSE, sep = " ",
eol = "\n", na = "NA", dec = ".", row.names = TRUE,
col.names = TRUE)
write.csv(...)
The data is now clean enough to build a model. We will do this later.
7. (Bonus) Should you make it through all of the cleaning, here are some questions asking you to examine
the gene variation. For the following, just use the “train” data set.
a. Compute a fold difference for each gene. Fold difference is the maximum value across samples
divided by minimum value. This value is frequently used by biologists to assess gene variability.
b. What is the largest fold difference and how many genes have it?
c. What is the lowest fold difference and how many genes have it?
d. Count how many genes have fold ratio in the following ranges
Range
Count
Val <= 2
..
2 <Val <= 4
..
4 <Val <= 8
..
8 <Val <= 16
..
16 <Val <= 32
..
32 <Val <= 64
..
64 <Val <= 128
..
128 <Val <= 256
..
256 <Val <= 512
..
512 <Val
..
e. Graph fold ratio distribution appropriately.