Download Document

Statistical Data Mining - 3
Edward J. Wegman
A Short Course for Interface ‘01
Visual Data Mining
Outline of Lecture


Visual Complexity
Description of Basic Techniques




Parallel Coordinates
Grand Tour
Saturation Brushing
Illustrations of Basic Techniques






Rapid Data Editing, Density Estimation (Pollen Data)
Inverse Regression, Tree Structured Decision Rules (Bank Data)
Classification & Clustering (SALAD Data & Artificial Nose )
Structural Inference (PRIM 7 Data)
Data Mining (BLS Cereal Scanner Data)
Cluster Trees (Oronsay Sand Particle Size Data)
Visual Complexity
Scenarios
Typical high resolution workstations,
1280x1024 = 1.31x106 pixels
Realistic using Wegman, immersion, 4:5 aspect ratio,
2333x1866 = 4.35x106 pixels
Very optimistic using 1 minute arc, immersion, 4:5
aspect ratio, 8400x6720 = 5.65x107 pixels
Wildly optimistic using Maar(2), immersion, 4:5
aspect ratio, 17,284x13,828 = 2.39x108 pixels
Visual Complexity
Visualization for Data Mining can realistically hope
to deal with somewhere on the order of 106 to 107
observations. This coincides with the approximate
limits for interactive computing of O(n2) algorithms
and for data transfer. This also roughly corresponds
to the number of foveal cones in the eye.
Methodologies for Visual Data Mining

Parallel Coordinates
 Effective
Method for High Dimensional Data
 High Dimensions = Multiple Attributes

Grand Tour
 Generalized
Rotation in High Dimensions
 In Depth Study of High Dimensional Data

Saturation Brushing
 Effective
Method for Large Data Sets
Visual Data Mining Techniques
Multidimensional Data Visualization
Scatterplot matrix
Parallel coordinate plots
3-D stereoscopic scatterplots
Grand tour on all plot devices
Density plots
Linked views
Saturation brushing
Pruning and cropping
Crystal Vision
Crystal Vision
Crystal Vision
Crystal Vision
Data Editing and Density Estimation
Pollen Data
3848 points
5 dimensions
C
Pollen Data
Pollen Data
Pollen Data
Pollen Data
Pollen Data
Pollen Data
Inverse Regression and Tree Structured
Decision Rules with Financial Data

Bank Demographic Data in 8 Dimensions
with 12,000+ points
Inverse Regression and Tree Structured
Decision Rules with Financial Data
Inverse Regression and Tree Structured
Decision Rules with Financial Data
Inverse Regression and Tree Structured
Decision Rules with Financial Data
Classification and Clustering Using
SALAD Data

Chemical Agent Detection Data in 13
Dimensions with 10,000+ points
Classification and Clustering Using
SALAD Data
Classification and Clustering Using
SALAD Data
Artificial Dog Nose
19 dimensional time series in 2 spectral bands
60 time steps for 300 chemical species
c
Artificial Dog Nose
Time series in two spectral bands for same chemical species
Artificial Dog Nose
Phase loop
Artificial Dog Nose
Orthogonal components
Artificial Dog Nose
After grand tour, orthogonal variables x2*, x9*, x15*, x16*, x18* separate the two spectral bands
Artificial Dog Nose
Four chemical species, target highlighted in red
Artificial Dog Nose
Target species separated by x1*, x3*, x5*, x6*, x11*, x15*
PRIM-7
7 dimensional high energy physics data
500 data points
pi-meson proton interaction
Structural Inference Using
PRIM 7 Data
Structural Inference Using
PRIM 7 Data
Structural Inference Using
PRIM 7 Data
Structural Inference Using
PRIM 7 Data
Structural Inference Using
PRIM 7 Data
Scanner Data for
Breakfast Cereals
5.5 gigabytes of scanner data in relational database
Price, sales volume, promotion, store, chain, PSU, UPC
Work done at BLS
Phase 1 – Basic Data Analysis – Single Month
Phase 2 – Price Relative Effects – 1 Year
Phase 3 – Churning Effects – 5 Years
Scanner Data for
Breakfast Cereals
Promotion has huge impact on sales volume
Scanner Data for
Breakfast Cereals
Stores not randomized
Scanner Data for
Breakfast Cereals
Aggressive promotion pays
Scanner Data for
Breakfast Cereals
Scanner Data for
Breakfast Cereals
Scanner Data for
Breakfast Cereals
Phase 2
Scanner Data for
Breakfast Cereals
Scanner Data for
Breakfast Cereals
Outliers belong to same chain
Scanner Data for
Breakfast Cereals
Promotion both years
Scanner Data for
Breakfast Cereals
Range of items with no promotion
Scanner Data for
Breakfast Cereals
One chain ceased promotions
Scanner Data for
Breakfast Cereals
Phase 3
Scanner Data for
Breakfast Cereals
Churning comes from both new items and new stores
Scanner Data for
Breakfast Cereals
Churning Effects: Red: PR=0, Blue: PR>0, Green PR=infinity
Scanner Data for
Breakfast Cereals
New items tend to have higher prices
Scanner Data for
Breakfast Cereals
Many discontinued items have high expenditures
Scanner Data for
Breakfast Cereals
Effect of item churning
Scanner Data for
Breakfast Cereals
Removing Store Birth-Death Effects
Scanner Data for
Breakfast Cereals
Outlier due to price coding error
Scanner Data for
Breakfast Cereals
Effects of Cereal Types
Scanner Data for
Breakfast Cereals
Quantity Effects
Sands of Time Data
300 Samples of Sand Data from Oronsay Island in the
Scotch Hebrides
Sands of Time - Objective
“The mesolithic shell middens on the island
of Oronsay are one of the most important
archeological sites in Britain. It is of
considerable interest to determine their
position with respect to the mesolithic
coastline. If the sand below the midden were
beach sand and the sand from the upper
layers dune sand, this would indicate a
seaward shift of the beach-dune interface.”
Flenley and Olbricht, 1993
Sands of Time - Objective
Cluster samples of modern sand into “beach-like” or
“dune-like” sand.
Classify archeological sand samples as to whether they
are beach sand or dune sand.
Sands of Time –
Parametric Analysis
Historical strategy is to fit parametric distributions and
compare modern and archeological sands based on
parameters.
Weibull, 1933; lognormal (breakage models), loghyperbolic, log-skew-Laplace, 1937, BarndorffNielsen, 1977.
Models 2 to 4 parameters, theory developed, practice
problematic.
Sands of Time - Graphical Analysis
Multidimensional Parallel Coordinate Display
Combined with Grand Tour.
BRUSH-TOUR strategy
Clusters recognized by gaps in any horizontal axis.
Brush existing clusters with colors.
Execute grand tour until new clusters appear, brush again.
Continue until clusters are exhausted.
Mining the Sands of Time
Mining the Sands of Time
Mining the Sands of Time
Mining the Sands of Time
Mining the Sands of Time
Mining the Sands of Time
Mining the Sands of Time
Sands of Time - Conclusions
Sands from the CC site and the CNG site have
considerably different particle size distributions and
cannot be effectively aggregated.
Data at small and at large particle dimensions is too
quantized to be used effectively.
The visual based BRUSH-TOUR strategy is extremely
effective at clustering.
Sands of Time - Conclusions Continued
Midden sands are neither modern beach sands nor
modern dune sands.
Midden sands are more similar to modern dune sands.
This result does not support the seaward-shift-of-thebeach-dune-interface hypothesis, but suggests the
middens were always in the dunes