Download SearchIndexer for 1_8

VIVO 1.8
SearchIndexer
http://gist.github.com/j2blake/388cbc50efb611481698
 Efficient
 Configurable
 Visible
 Maintainable
 Theory
 Flow
 Modularity
 Configuration
 Performance
 Testing
 Questions
 Task Force
 Adjust automatically to changes in the model.
 User experience does not require that updates are
synchronous.
 When a triple is changed, several search records may
require updating.
 Several altered triples may affect the same search
record.
 Each search record can be updated independently.
Theory
 Respond to search Indexer admin page: request a
rebuild.
 Respond to search indexing service API: specify a list
of URIs to be re-indexed.
 Respond to changes in the Abox
 From the list of altered triples, build a list of URIs that
require re-indexing.
 Respond to request for rebuild
 Build a list of all URIs in the model and re-index.
 Respond to the search indexing service
 Accept a list of URIs for re-indexing.
Flow – asynchronous
Task Queue
Search
Indexing
Service
Search Indexer
Admin page
Change to
ABox model
Update URIs
Task
Rebuild Index
Task
Update
Statements
Task
Thread Pool
Work Unit
Work Unit
Work Unit
Flow – update URIs
 Accept a list of URIs to be indexed.
 Determine the eligible URIs from the list.
 Must be an existing individual with at least one VClass
(type).
 Must pass the list of SearchIndexExcluders.
 Remove the ineligible URIs from the index.
 Build a search document for each eligible URI.
 Execute the list of DocumentModifiers.
 Add the completed document to the index.
Flow – update URIs
Flow – update URIs
Flow – rebuild index
 Get a list of all individuals in the ABox.
 Update these URIs.
 (use the existing logic to update URIs)
 Remove obsolete records from the index.
 Anything that was indexed prior to the rebuild.
Flow – rebuild index
Flow – rebuild index
Flow – change to ABox
 Accumulate changes into a batch
 Delimited by a quiescent interval.
 Optional delimiting calls to pause() and unpause()
 Produce a set of URIs to be updated
 Each statement is examined by the list of
IndexingUriFinders.
 Update these URIs.
 (use the existing logic to update URIs)
Flow – change to ABox
Flow – change to ABox
 Application.java
 SearchIndexer.java
 ConfigurationBeanLoader.java
 [vivo-home]/config/applicationSetup.n3
 Read at runtime.
Application.java
public interface Application {
ServletContext getServletContext();
VitroHomeDirectory getHomeDirectory();
SearchEngine getSearchEngine();
SearchIndexer getSearchIndexer();
ImageProcessor getImageProcessor();
FileStorage getFileStorage();
ContentTripleSource getContentTripleSource();
ConfigurationTripleSource getConfigurationTripleSource();
TBoxReasonerModule getTBoxReasonerModule();
void shutdown();
}

https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/c25ef7c063
8f46473e935fec86545c689ffd4fc9/Application.java
SearchIndexer.java
public interface SearchIndexer extends Application.Module {
void startup(Application app, ComponentStartupStatus ss);
void shutdown(Application app);
void pause();
void unpause();
void addListener(Listener listener);
void removeListener(Listener listener);
void scheduleUpdatesForStatements(List<Statement> changes);
void scheduleUpdatesForUris(Collection<String> uris);
void rebuildIndex();
SearchIndexerStatus getStatus();
}

https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/a1ffb6330d
ec0010cdff0ce283178a57b1fb1048/SearchIndexer.java
ConfigurationBeanLoader
public ConfigurationBeanLoader(Model model);
/**
* Load the instance with this URI,
* if it is assignable to this class.
*/
public <T> T loadInstance(String uri, Class<T> resultClass)
throws ConfigurationBeanLoaderException;
/**
* Find all of the resources with the specified class,
* and instantiate them.
*/
public <T> Set<T> loadAll(Class<T> resultClass)
throws ConfigurationBeanLoaderException;
ApplicationImpl.java
@Override
public SearchIndexer getSearchIndexer() {
return searchIndexer;
}
@Property(uri =
"http://vitro.mannlib.cornell.edu/ns/vitro/ApplicationSetup#hasS
earchIndexer")
public void setSearchIndexer(SearchIndexer si) {
searchIndexer = si;
}
@Validation
public void validate() throws Exception {
if (searchIndexer == null) {
throw new IllegalStateException(
"Configuration did not include a
SearchIndexer.");
}
}
applicationSetup.n3
@prefix : <http://vitro.mannlib.cornell.edu/ns/vitro/ApplicationSetup#> .
:application
a
<java:edu.cornell.mannlib.vitro.webapp.application.ApplicationImpl> ,
<java:edu.cornell.mannlib.vitro.webapp.modules.Application> ;
:hasSearchEngine
:instrumentedSearchEngineWrapper ;
:hasSearchIndexer
:basicSearchIndexer ;
:hasImageProcessor
:jaiImageProcessor ;
:hasFileStorage
:ptiFileStorage ;
:hasContentTripleSource
:sdbContentTripleSource ;
:hasConfigurationTripleSource :tdbConfigurationTripleSource ;
:hasTBoxReasonerModule
:jfactTBoxReasonerModule .
# ...
:basicSearchIndexer
a
<java:edu.cornell.mannlib.vitro.webapp.searchindex.SearchIndexerImpl> ,
<java:edu.cornell.mannlib.vitro.webapp.modules.searchIndexer.SearchIndexer> ;
:threadPoolSize "10" .

https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/7b009909ee8f812366cde14f3c1253ded
514f85e/applicationSetup.n3
 UriFinders
 What URIs are affected by an altered triple?
 Excluders
 What URIs should not have documents in the index?
 DocumentModifiers
 What data belongs in the search document?
Configuration
 RDF in the display model
 [vitro]/webapp/rdf/display/everytime/searchIndexerConfig
urationVitro.n3
 https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/4990dc99f6a68
6ac501dc8f8b68808b20759657e/searchIndexerConfigurationVitro.n3
 [vivo]/rdf/display/everytime/searchIndexerConfigurationVi
vo.n3
 https://gist.githubusercontent.com/j2blake/388cbc50efb611481698/raw/904d3457aeeaa
de03cb5515ef7856c010ff774f4/searchIndexerConfigurationVivo.n3
 Read at startup
Configuration - examples
:searchExcluder_typeExcluder
a
<java:edu.cornell.mannlib.vitro.webapp.searchindex.exclusions.ExcludeBasedOnType> ,
<java:edu.cornell.mannlib.vitro.webapp.searchindex.exclusions.SearchIndexExcluder> ;
:excludes
"http://www.w3.org/2002/07/owl#AnnotationProperty" ,
"http://www.w3.org/2002/07/owl#DatatypeProperty" ,
"http://www.w3.org/2002/07/owl#ObjectProperty" .
:uriFinder_forDataProperties
a
<java:edu.cornell.mannlib.vitro.webapp.searchindex.indexing.IndexingUriFinder> ,
<java:edu.cornell.mannlib.vitro.webapp.searchindex.AdditionalURIsForDataProperties> .
:documentModifier_NameFieldBooster
a
<java:edu.cornell.mannlib.vitro.webapp.searchindex.document.FieldBooster> ,
<java:edu.cornell.mannlib.vitro.webapp.searchindex.document.DocumentModifier> ;
:hasTargetField "nameRaw" ;
:hasTargetField "nameLowercase" ;
:hasTargetField "nameUnstemmed" ;
:hasTargetField "nameStemmed" ;
:hasBoost "1.2"^^xsd:float .
 Simple efficiency:
 Don’t ask for more information than we need.
 Don’t discard information we have obtained.
 Multi-threading:
 Finding URIs for a collection of altered statements.
 Need to remove duplicate URIs.
 Building the search records for a collection of URIs.
Performance – memory
 Keeping the memory footprint low:
 A list of URIs
 One URI is likely to be ~100 bytes.
 100,000 URIs is likely to be ~10 megabytes.
 A list of Individuals.
 One Individual may be ~100 kilobytes.
 50,000 Individuals may be ~5 gigabytes.
Performance - timing
 Continuous improvement: timings from the developer
panel:
Performance - timing
 https://gist.githubusercontent.com/j2blake/388cbc50efb
611481698/raw/26000d6793bef1bdb0d2ff9a885000caf
175a619/vivo.all.log
 Add to the battery of Selenium tests
 For example:
 Create a person, and a book written by that person.
 Search for the person’s name: both the person and the book
should be returned.
 Search for the title of the book: both the person and the book
should be returned.
 A continuing effort
 So far, not very successful.
 Other configurable document modifier classes?
 Other configurable excluder classes?
 What about configurable URI finder classes?
Questions
 Boost – multiplicative would make it order-independent.
Do we care?
 Private data – the SPARQL-based DocumentModifiers
do not use a filtered connection. Is that a problem?
 At what point does it make sense to do a full rebuild
instead of responding to model changes?
 Finding URIs can be more expensive than building search
documents.
Questions
 What should be the default configuration?
 Don Ellsborg and I have talked about creating a group
from the community.
 Short time-frame, specific deliverable:
 The default configuration for the SearchIndexer, postrelease 1.8.
Questions?