Download Document

Refining –
Finding Words/expanding
Taly Sharon
[email protected]
[email protected]
1
www.sharon-it.com
Contents
•
•
•
•
•
2
Expanding/Learning terms
Categorization/Clustering engines
Google Suggest
SurfWax FocusWords
When you don’t know where to start
www.sharon-it.com
Make Longer Queries
Average Search Terms per Query
Overall
Experienced
•Yahoo
3
•Harvest Digital
www.sharon-it.com
Adding Words
•
•
•
•
•
4
Holocaust
23,200,000
holocaust memorial
836,000
holocaust memorial budapest
42,300
holocaust memorial budapest danube
4,910
holocaust memorial budapest danube promenade
692
www.sharon-it.com
Classification/Categorization
• Classification: the process of deciding the
appropriate category for a given document.
• Examples:
– deciding what newsgroup an article belongs to.
– what folder an email message
should be directed to.
– what is the general topic of an essay.
5
www.sharon-it.com
Clustering
• The process of automatically grouping
documents.
6
www.sharon-it.com
Search Categorization/Clustering
• The result documents are ordered
according to categories.
• The searcher can select the relevant
category to display the related documents.
• Examples:
– Vivisimo/Clusty
– Excite
– Teoma
– Exalead
7
www.sharon-it.com
Clusty
8
www.sharon-it.com
Excite
9
www.sharon-it.com
Ask
10
www.sharon-it.com
Exalead
11
www.sharon-it.com
Google Suggest
• As you type – you get query suggestions
and number of results per query.
12
www.sharon-it.com
Google Suggest (2)
13
www.sharon-it.com
Google Suggest
14
www.sharon-it.com
15
www.sharon-it.com
Yahoo Search Assist
16
www.sharon-it.com
SurfWax FocusWords
• SurfWax has an option “Focus”
• This option invokes the FocusWords
mechanism
• You get suggestions to make your query:
– Broader
– Similar
– Narrower
• http://www.surfwax.com
17
www.sharon-it.com
SurfWax FocusWords
18
www.sharon-it.com
When you don’t know how to start
• Reverse Dictionary
• Glossaries and
Dictionaries
• Taxonomy/Folksonomy
• Pearl Culturing
• Analyzing pages
• Finding similar pages
– Google’s related:
– Alexa www.alexa.com
19
www.sharon-it.com
Reverse Dictionary
• OneLook reverse dictionary:
http://www.onelook.com/reversedictionary.shtml
• Example: “bird of prey” => raptor
• Example: economical measure of a
nation’s wealth => Gross Domestic
Product
20
www.sharon-it.com
21
www.sharon-it.com
Glossaries and Dictionaries
• Google search:
– <topic>
– glossary OR thesaurus OR dictionary OR taxonomy
• Example 1: agriculture glossary
– http://www.cnie.org/nle/AgGlossary/AgGlossary.htm
– http://www.cahe.nmsu.edu/news/aggloss.html
– http://agriculture.house.gov/info/glossary.html
• Example 2: agriculture thesaurus
– http://agclass.nal.usda.gov/agt/agt.shtml
– http://www.fao.org/aims/ag_intro.htm (multilingual)
• http://www.glossarist.com
– http://www.glossarist.com/glossaries/business/primaryindustry/agriculture.asp
22
www.sharon-it.com
23
www.sharon-it.com
Taxonomy/Folksonomy
• Taxonomies
– Found via directory search (example DMOZ):
http://search.dmoz.org/cgibin/search?search=taxonomy
– www.taxonomywarehouse.com (paid)
• Folksonomy
– Use tags in
• Technorati www.technorati.com
• Delicious www.del.icio.us
24
www.sharon-it.com
del.icio.us
25
www.sharon-it.com
del.icio.us
26
www.sharon-it.com
Pearl Culturing
• What to do when you don’t have the
category nor the right keywords?
• Find one good relevant website
• Look it up in directories
• You will find:
– the category/main keywords
– authoritative websites
• Useful search engine: Exalead
27
www.sharon-it.com
Analyze Pages
• Distilling: what is problematic in a bad page?
– what is wrong? Is there an interfering keyword/term
appearing.
– Remove interfering terms (using “-”).
• Identifying clues and patterns in a good page.
– Read the document, what are the clues?
– Look for new keywords, word combinations and other
things differentiating between it to non-authoritative
documents.
– Use frequency counter:
• http://www.wordcounter.com/
• http://www.georgetown.edu/faculty/ballc/webtools/web_freq
s.html
28
www.sharon-it.com
Frequency
Counter
29
www.sharon-it.com
30
www.sharon-it.com
Wordcounter
31
www.sharon-it.com
32
www.sharon-it.com
References
• http://hacks.oreilly.com/pub/ht/2
• www.batesinfo.com
• http://www.searchtools.com/info/classifier
s.html
33
www.sharon-it.com
Exercises
1.
2.
3.
4.
5.
6.
7.
8.
34
How are bad user interfaces called (hint: try Google suggest)
Reverse dictionary
a.
b.
c.
Find relevant keywords for chemistry
What is the terminology for when menstruation stops?
How was the separation between the west and the soviet union called?
What are the related terms to Competitive Intelligence?
Check suggestions from Google Suggest for a query starting with biofuel.
Using SurfWax, learn options to focus or broaden the query: biodiesel.
Identify the most relevant terms in the website: www.uspto.gov
Identify the most relevant terms in the Biofuel Wikipedia entry
http://en.wikipedia.org/wiki/Biofuel.
Search in Onelook reverse dictionary and in other glossaries terms: fuel,
natural energy, geothermal, and other terms. Look at the results.
www.sharon-it.com