Download Data Mining and Data Warehousing – Clustering

Data Mining and Data Warehousing –
Clustering-Outlier Analysis
Insights
After clustering gained importance and wide spread use in data mining, many
data scientists started using clustering in various data sets. They observed some
interesting information about points in data sets which were not part of any
clusters. Such points were of importance for study as they explained more about
the irregularities of the data sets. Such points were called outliers.
Imagination
Given a set of all mails of an organization, can you perform outlier analysis on
them? If yes, what type of outliers will you get? What is the importance of such
outliers? Can you detect fraudulent mails and spams from the mails using outlier
detection? If so, does it help in classification of mails as normal, fraudulent and
spams?
Resources



Outliers introduction PPT (For your convenience you can get them inside Learn
More Quadrant)
Outlier detection methods PDF (For your convenience you can get them inside
Learn More Quadrant)
JIT lecture on LOF algorithm PDF (For your convenience you can get them
inside Learn More Quadrant)
References

Grubbs test for
outliers: http://itl.nist.gov/div898/handbook/eda/section3/eda35h.htm
Glossary




Outlier: A data point/observation which is quite different from the remainder
of the data.
Hawkins outlier: An outlier is an observation that deviates so much from other
observations as to arouse suspicion that it is generated by a different
mechanism.
Distance based outlier: A DB(p,D) outlier is an object O in a dataset T such that
atleast a fraction p of the object in T lies at a distance greater than D from O.
LOF: The local outlier factor computed of a point. It is used in density based
outlier mining algorithms.