Download Lab worksheet 5 – Retrieval and Classification

Lab worksheet 5 – Retrieval and Classification
Objectives



To better understand histograms
To use histogram as one of the visual features to represent image content
To develop image classification and retrieval methods
The Histogram Algorithm
To refresh your memory, here are some highlights from the lecture notes on the
definition of a histogram and a general algorithm for histogram calculation.
For colour image (3 bands, for R, G, B respectively)

The basic idea is to quantize each of the RGB values into m intervals resulting
in a total number of m3 (m=256) colour combinations (or bins). You can also
use 3 x 256 to calculate the RGB bins separately to improve the efficiency.

A colour histogram H(I) is then constructed. This colour histogram is a vector
{h1, h2, …, hm3} where element hx represents the number of pixels in image I
falling within bin x.
The following is a sample algorithm to calculate a grey level histogram (band = 1).
For details on calculating colour histogram, please read the program discussed below
from step 1.
Create an array histogram with 2b elements
for all grey levels, i, do
histogram[i] = 0
end for
for all pixel coordinates, x and y, do
Increment histogram [f(x,y)] by 1
end for
//b representing number of byte
Overview of Deliverables
For this lab sheet the deliverable application you produce must be capable or
performing two tasks.
Classification:
For the first deliverable you will be required to collect two sets of image data,
ideally each set should have certain similarity in terms of their colour content. For
example, you can collect 5 images with oranges and 5 images with bananas (or 5
images with grass, 5 images with cloud and sky). Your application should be
capable of producing histogram data for each image (including the query image),
determining the average histogram data for each of the two classifications, and
printing to the command line which classification the query image belongs to.
Retrieval
For the second deliverable you should modify the class you have created above
(or create a new class which is capable of reading in 11 images as previously), but
instead of classifying the query image, your new class should retrieve and display
the image most similar from the entire set of images (i.e. the whole dataset.)
Lab Instructions
The first section of the lab is concerned with understanding what the code in
“Histogram.java” class does. You do not have to run the Histogram.java file. The
steps below highlight how each sentence in this algorithm above is implemented in
Java.
Step1 In NetBeans, open “Histogram.java” which is located in the directory “javaimNick\Classes\Source\com\pearsoneduc\ip\op”.
The easiest way to do this is by selecting “Open File” from the “File” menu in
NetBeans, and navigating to the “Histogram.java” file. There is no need to
incorporate this class into the source of your project like you may have done
with previous labs.
Step 2 from the “view” menu, choose “show line numbers”, so that you can see the
line numbers of the code, which are used for reading reference for the rest of this
document. If the “show line numbers” option is greyed-out please click once in the
editor window (anywhere in the code) and then try to select the option again.
Step 3. Please try to read through the whole class “Histogram” as much as you can.
The following section provides some help for you by explaining some important parts
in the class. Further explanation on the important lines are given as below:
In class “Histogram.java”
Line 44: public final class Histogram implements Cloneable{….
The whole Histogram class is implemented very sophisticatedly, although the only
key and indispensable part is the class is the accumulateFrequencies( ) between line
584 and 608. The author has added a lots more functionality into this class apart from
just calculating histogram in accumulateFrequencies( ).
From line 60 to line 112 defines the member variables in the Histogram class.
Line 60 private int bands;
This to be used for representing the number of bands in the source image. If band=1,
the source image to dealt with is grey level image (only deal with grey level intensity
range from 0-255); if band ==3, means image to be dealt with is colour image, with R
G B colour bands. Each band will have colour range from 0-255.
Line 66 private int samples;
This to store the total number of samples processed. Typically it will be the total
number of the pixels in the image, samples = image.getWidth()*image.getHeight();
However, if you wish to calculate the histogram for a smaller region of image, the
sample is the pixel number in that region.
Line 72
private int[][]freq;
2D array holding frequency data for a grey scale or colour image. This is where the
histogram information is held. If it is a grey image to be processed, the band=1, then
freq[0][] will represent the histogram for grey image. If it is a colour image, the
band=3, then freq[0][] freq[1][] freq[2][] respectively will represent the histogram
(frequency) for bands R, G B.
From line 78-112, the class also defines some other variables to represent statistical
data about the image, would be useful for advanced application of histogram. If you
like, you can use them in your coursework as extra criteria for possible improvement
of similarity measure (you are encouraged to do so if you have more time if you wish
to provide more analysis in your report, but it is beyond the minimum requirement).
Line 219 public void computeHistogram(BufferedImage image)
This function is to call the calculation function (accumulateFrequenccies() ) to
compute histogram.
Line 224,
if (image.getType()==BufferedImage.TYPE_BYTE_GREY) is to judge whether the
input image is a grey level image or a colour image. If it is a grey image, then assign
bands ==1, otherwise it is a colour image, then bands = 3.
NB. BufferedImage is a built-in class in Java to represent an image. In this module we
mostly will use this class for representing an image.
Line public void write (Writer destination){
This function is to write histogram data to the specified destination. You can use this
function to write the histogram data to a file. Please note in this function, for grey and
colour images, the formats of writing the histogram is slightly different. You will see
this in step 4.
Line 338 to line 540 include methods to return some useful statistical data generated
from histogram.
Line 371-390, define methods:
public int getFrequency(int value), and public int getFrequency(int band, int value)
These two functions are very important as they are the methods you may want to use
for the coursework to obtain histogram value of the analysed image. The reason for
using these methods is because they are public methods, and can be accessed by any
class you create. The variable associated with this method (freq[band][value]) is
declared as a private member and cannot be accessed outside the Histogram class.
Line 549 and line 564 contains code for allocating storage and initialisation, you may
find in here a bit more information on how the data structure has been designed for
the class Histogram.
Line 584 private void accumulateFrequencies (BufferedImage image)
This is the key function that actually calculates the histogram. The histogram is stored
in private variable freq[ ][ ].
Line 592 to 596 calculate the grey level histogram. You can see 2D array freq here is
only used as 1D array as freq[0] [], where freq[0][i] represents the pixel frequency in
the image whose grey intensity is i.
Line 597 to 606 calculate the colour histogram, where freq[0][], freq[1][], freq[2][]
respectively represent frequency of each band (RGB).
Line 615 - 660 private void computeStatistics()
This is the method to calculate other useful statistical information related to
histogram. It will help your understanding on histogram if you look through this code
This concludes the tour of the Histogram class. Please try and understand what has
been described above.
Step 4 Now we will use a main program to call the functionality in the histogram
class. To do this we are going to examine another class – the “CalcHist.java” class.
This class can be found in the “javaim-Nick\Apps\Chap06” directory and you should
include this directory into your project using the same method as you have in the
previous labs.
IMPORTANT NOTE
At this step you should make sure your project has both the “iplib.jar” and
“ipapps.jar” included in the libraries section of project properties.
Both libraries
included
The following section highlights the key functionality in the “CalcHist.java” class
Line 40 ImageDecoder input = ImageFile.createImageDecoder(argv[0]);
Line 41 BufferedImage image = input.decodeAsBufferedImage();
These two lines are to take input from the main(), the first argument will be the image
file (argv[0]) that you would like to process. The two lines convert the input image
file into an image object (image), an instance of class BufferedImage.
Line 42
Histogram histogram = new Histogram(image);
This passes the image variable as a parameter to the Histogram( ) constructor, which
will construct all the statistical data related to histogram, in particular, freq[][].
Line 43
FileWriter histFile = new FileWriter(argv[1]);
Line 44
histogram.write(histFile);
The second argument from the command line or the NetBeans arguments specifies a
text file where the histogram data will be written to. Histogram.write() enable writing
information in freq[][] to a text file. (This is how you get the text file for histogram in
your first assignment. In this final assignment, we do not have to write the histogram
to files so we do not need this.)
Line 45-47 performs processing that produces the cumulative histogram which is
another important statistical data about image. It is useful when calculate the
histogram equalisation. (This is not needed for this assignment.)
Step 5 Set the appropriate arguments to run the CalcHist class. Provide the input the
image you would like to process, and indicate a filename for storing the output result.
For example, you can set the NetBeans arguments to:
matthew1.jpg histo-out.txt
This will process a colour image (matthew1.jpg) and save the result in histo-out.txt.
You can later open histo-out.txt to check the result. For grey level image, you can
input a grey image such as “mattgrey.jpg” to see the difference. (We have done this in
the first assignment.)
Step 6 The above process is to help you identify the flow of data when calculating
and using the histogram functionality provided in the Histogram.java class. For your
coursework, you can create a new class, for example called “Classification.java” and
copy the code from “CalcHist.java” to use as a template.
Coursework Overview:
Image classification and retrieval based on Histogram
A key issue is what representation or encoding of the object is used in the recognition
process? Alternatively, what features are important for recognition? We often talk
about colour, shape, and texture information being important visual cues for
recognition. Here we focus on using colour information to represent image content.
Feature vector representation: Objects may be compared for similarity based on their
representation as a vector of measurements. Suppose each object is represented by
exactly d measurements. The ith coordinate of such a feature vector has the same
meaning for each object; In our case, we can use a colour histogram to form such
vector. (We recommend using 3x256 bins instead of 2563 because processing a large
number of bins can be processor intensive You can use some other design, e.g., partial
accumulative histogram, first 10 bins as one vector element, second 10 bins as the
second vector element and so on).
The similarity, or closeness, between the feature vector representations of two objects
can then be described using Euclidean distance between the vectors defined in
Equation 1. Sometimes the Euclidean distance between an observed vector and a
stored class prototype can provide a useful classification function.
Definition: the Euclidean distance between two d-dimensional feature vectors h1
and h2 is
||h1-h2|| =
 h1[i]h2[i]
2
Equation 1
i 1, d
h1 and h2 can be seen as two colour histograms, where hi[i] represents the frequency
value at bin i.
Classification
Such a method can be used for classification. Assume that we have 10 images
belonging to class “grass images”, another 10 images belonging to “cloud images”.
We can use the same method of calculating the centroid in the segmentation task
(which was a two dimensional problem), to calculate the centre mean of the class:
d
 h[i]
hmean[i] =
0
d
d is the number of samples, in this case, 10. h[i] is the histogram for image i.
o class mean
oo o o
o. o
ooooo o
o ox x x x x
xx x x
xxxx x
x class mean
Figure 1 two compact classes: classification using nearest mean
After having obtained the centre mean, we can then use above distance calculation
method (Euclidean or Absolute distance that is mentioned in the lecture note on
Retrieval based on Colour Histogram, also see below) to calculate the distance
between the unknown image and the two centre means, the closer value means that it
should be more possible for this unknown image belong to that class.
Retrieval:
A similar method can be used for retrieval. If hq means a colour histogram for query
image q, the distance between this query image and every image e in the database –he
||hq-he|| =


 hq [i] he[i]
i 1, d
2
Equation 2
the one in the database with the shortest distance to the query image should be
retrieved.
(we have practised Euclidean distance in 2-dimensional problem when we tried to
calculated the distance between two centroids, ||D|| =
( x1  x 2 ) 2  ( y1  y 2 ) 2
)
Alternative method for colour histogram based retrieval: Absolute distance
To retrieve image from the database, the user supplies either a sample image or a
specification for the system to construct a colour histrogram h(Q).
A distance metric is used to measure the similarity between h(Q) and h(I). Where I
represents each of the images in the database. And example distance metric is shown
as follows:
x tbin
D(Q, I ) =
| q
x 0
z
 iz |
Where qx and ix are the numbers of pixels in the image Q and I, respectively, falling
within bin x. tbin can be 3x256 bins or of 2563
Coursework Deliverable
This section contains step-by-step instructions on how to approach the coursework.
Classification:
1. Collect two sets of image data, ideally each set should have certain
similarity in terms of their colour content. For example, you can collect 5
images with oranges and 5 images with bananas (or 5 images with grass, 5
images with cloud and sky).
2. You should create a new class (possibly copying the code in CalcHist.java
or the sample code given below to use as a reference) which is capable of
reading in these images (where the filenames are specified as arguments in
NetBeans) and storing them in an array of type BufferedImage
3. Your class should also be capable of reading in another image (specified
using the NetBeans arguments) which will be a query image (In total you
should be reading 11 image filenames from the arguments argv[] array).
4. Your class should be able to produce histogram data for each image
(including the query image). Your class should also be capable of printing
to the command line which classification the query image belongs to, for
example orange or banana / grass or sky.
5. You will need to calculate the mean of the histograms for each of the
sample images for each class and then measure the distance between the
histogram of the query image and each classification’s mean. The shorter
the distance, the more similar the input image to the class.
Retrieval
1. You should modify the class you have created above or create a new class
which is capable of reading in 11 images as previously described (1 query
image and 10 other images), but instead of classifying the image your new
class should retrieve a similar image from all the images (i.e. the whole
dataset.) In this case you can have a mixture of various kinds of images in
your dataset.
2. The similarity measure will be the calculation of the distance between the
query image and every single image in the dataset based on their
histograms. The closer the distance, the more similar. This is an example
of Content-based Image Retrieval.
Hints and Example Code
I have provided an example outline of the code you will be required to produce for the
first deliverable. Please note that if you cut and paste this code into NetBeans it
will NOT work without modification! You will need to add extra code (as specified
in the comments) to provide all of the required functionality.
When this code is complete you should specify appropriate arguments in NetBeans:
e.g. g01.jpg s1.jpg s2.jpg s3.jpg s4.jpg s5.jpg g1.jpg g2.jpg g3.jpg g4.jpg g5.jpg
You can also adapt this code for the Retrieval deliverable.
import
import
import
import
import
import
import
import
import
import
java.awt.image.BufferedImage;
java.io.FileWriter;
com.pearsoneduc.ip.io.*;
com.pearsoneduc.ip.op.Histogram;
java.io.*;
java.awt.image.*;
java.awt.color.ColorSpace;
javax.swing.*;
com.pearsoneduc.ip.io.*;
com.pearsoneduc.ip.gui.*;
public class ClassificationAndRetrieval extends JFrame {
private ImageView[] views;
// image display components
//constructor which creates a window and displays the 11 images
//specified as filename arguments
public ClassificationAndRetrieval(BufferedImage[] imageSequence)
throws IOException, ImageDecoderException {
super("Classification and Retrieval");
views = new ImageView[11];
for (int i=0;i<11;i++) {
views[i] = new ImageView(imageSequence[i]);
}
JTabbedPane tabbedPane = new JTabbedPane();
tabbedPane.add(new JScrollPane(views[0]),"Query Image");
for (int i=1;i<11;i++) {
tabbedPane.add(new JScrollPane(views[i]),"Sample"+i);
}
getContentPane().add(tabbedPane);
addWindowListener(new WindowMonitor());
}
public static void main(String[] argv) {
if (argv.length > 1) {
//create an array of BufferedImages and Histograms here
Histogram[] histogram = new Histogram[11];
BufferedImage[] imageSequence= new BufferedImage[11];
try {
for(int i=0;i<imageSequence.length;i++) {
ImageDecoder input =
ImageFile.createImageDecoder(argv[i]);
BufferedImage image =
input.decodeAsBufferedImage();
//create histogram instance for each input image
histogram[i]=new Histogram(image);
//add the image to the BufferedImage array
imageSequence[i] = image;
}
} catch (Exception ex) {
ex.printStackTrace();
}
//here we create 4 integer arrays that will contain our histogram
//data for each classification type. Notice we are using 256*3 bins
//instead of pow(256,3) to limit the amount of computation
int
int
int
int
hisSum1[] = new int[256*3];
hisSum2[] = new int[256*3];
average1[] = new int [256*3];
average2[] = new int [256*3];
//try and understand how the two-dimensional histogram data for each
//image is extracted into the one-dimensional arrays specified above
for (int b=0;b<3;b++) {
for(int j=0;j<256;j++){
for (int i=1;i<11;i++){
int n=256*b;
if(i<6) {
hisSum1[j+n]+=histogram[i].getFrequency(b,j);
} else {
hisSum2[j+n]+=histogram[i].getFrequency(b,j);
}
}
}
}
//here we calculate the histogram average for each of the two
//classifications and store the result in the
//appropriate place in the average1 or average2 array
for (int i=0;i<256*3;i++) {
average1[i]=Math.round(hisSum1[i]/5);
average2[i]=Math.round(hisSum2[i]/5);
}
int queryHistogram[] = new int[256*3];
for (int b=0;b<3;b++) {
for(int j=0;j<256;j++){
int n=256*b;
queryHistogram[j+n]=histogram[0].getFrequency(b,
j);
}
}
//here you should implement the calculations that determine which
//classification your query image belongs to
//the following code creates the window and displays it to the user
JFrame frame = null;
try {
frame = new
ClassificationAndRetrieval(imageSequence);
} catch (ImageDecoderException ex) {
System.out.println("Problem processing images");
ex.printStackTrace();
System.exit(0);
} catch (IOException ex) {
System.out.println("Problem loading images");
ex.printStackTrace();
System.exit(0);
}
frame.setSize(900,700);
frame.setVisible(true);
} else {
System.err.println(
"usage: java ClassificationAndRetrieval <Query
image> <sample1> <sample2> <sample3> ........");
System.exit(1);
}
}
}