Download Word - Kuali: Jira

[KULCFG-406] Cannot Create Full Text Index in Contour Created: 22/Oct/09
Updated:
24/Mar/10 Resolved: 23/Oct/09
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Closed
Z Archived: Foundation CM
KC
None
Type:
Reporter:
Resolution:
Labels:
Remaining
Estimate:
Time Spent:
Original
Estimate:
Task
Kenton Hensley (Inactive)
Fixed
None
0 minutes
None
Priority:
Assignee:
Votes:
Critical
Farooq Sadiq (Inactive)
0
2 hours
Not Specified
Description
Contour Re-indexing facility for creating its full-text indices is failing. There currently is data in
the Contour DB that is not accessible and the index condition is suspected.
An issue has been logged with Contour, but they have asked us to see if we can see any other
processes that may be interfering with the memory TomCat and MySQL during the reindexing
process. In the logs we are seeing the exception below
While running the indexer, I need some assistance to watch the state of the processes on the box
while tailing the Contour log file to see if we can detect anything abnormal. My suspicion is
that this is not going to reveal anything but we have to do our due dilligence.
EXCEPTION:
2009-10-20 20:26:52,338 WARN [org.directwebremoting.impl.DefaultRemoter] - Method
execution failed:
java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518)
at java.lang.StringBuffer.append(StringBuffer.java:307)
at org.hibernate.type.TextType.get(TextType.java:64)
at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184)
at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173)
at org.hibernate.type.AbstractType.hydrate(AbstractType.java:105)
at
org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:2114)
at org.hibernate.loader.Loader.loadFromResultSet(Loader.java:1404)
at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1332)
at org.hibernate.loader.Loader.getRow(Loader.java:1230)
at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:603)
at org.hibernate.loader.Loader.doQuery(Loader.java:724)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2232)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
at org.hibernate.loader.Loader.list(Loader.java:2124)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl.index(Unknown Source)
at com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl.createDocumentIndex(Unknown
Source)
at com.jamasoftware.contour.service.impl.IndexServiceImpl.indexDocuments(Unknown
Source)
at com.jamasoftware.contour.dwr.impl.DwrIndexServiceImpl.indexDocuments(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.directwebremoting.impl.ExecuteAjaxFilter.doFilter(ExecuteAjaxFilter.java:34)
at org.directwebremoting.impl.DefaultRemoter$1.doFilter(DefaultRemoter.java:428)
2009-10-20 20:26:52,340 WARN [org.directwebremoting.dwrp.BaseCallMarshaller] - -Erroring: batchId[11] message[java.lang.OutOfMemoryError: Java heap
COMMENTS FROM JAMA:
2009-10-20 20:15:36,811 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Max memory :1065484288
2009-10-20 20:15:36,811 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Total memory :1065484288
2009-10-20 20:15:36,811 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Free memory before gc :990811336
2009-10-20 20:15:36,812 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Used memory before gc :74672952
2009-10-20 20:15:37,903 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Free memory after gc :999361352
2009-10-20 20:15:37,903 INFO [com.jamasoftware.contour.dao.hibernate.HibIndexDaoImpl] Used memory after gc :66122936
2009-10-20 20:15:45,699 INFO [org.directwebremoting.impl.DefaultRemoter] - Exec:
lookupSvc.getLookupTypeList()
2009-10-20 20:15:48,082 INFO [org.directwebremoting.impl.DefaultRemoter] - Exec:
reportSvc.getReportList()
2009-10-20 20:15:49,623 INFO [org.directwebremoting.impl.DefaultRemoter] - Exec:
adminSvc.getOrganizationList()
2009-10-20 20:16:11,187 INFO [org.directwebremoting.impl.DefaultRemoter] - Exec:
adminSvc.getSystemLicense()
2009-10-20 20:16:15,609 WARN [org.directwebremoting.impl.DefaultRemoter] - Method
execution failed:
java.lang.OutOfMemoryError: Java heap space
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518)
at java.lang.StringBuffer.append(StringBuffer.java:307)
at org.hibernate.type.TextType.get(TextType.java:64)
You can see that Tomcat only used 66.1M memory when it ran out of memory. This means the
system didn't have any memory left at that time though the Java could use up to 1G memory.
You can monitor the memory usage by using %top or %free during indexing.
Another thing you can try it to set the max memory for Tomcat to be 512M.
Thanks,
Sean
_____________________________________________________________________
We were able to index about 180k items on a small instance of EC2 Linux with CentOS.
It looks like the system was out of memory during index when it only uses about 75M memory.
I highly suspect that other processes have used all the memory leaving no memory left for
Tomcat. It could be another Tomcat is running.
To find out the memory usage and what processes use the most memory:
%top
then shift-m (to sort by memory usage)
To find out how many Java processes are running:
%ps -axl | grep java
Other Commands regarding memory usage:
%free
%vmstat -S M
Are you running MySql on the same box?
Please let me know what you find out.
Thanks,
Sean
Comments
Comment by Kenton Hensley (Inactive) [ 23/Oct/09 ]
My latest post to the Jama version of this issue:
___________________________________________
Hello Sean,
I got Contour setup on my local box which has 4GB RAM (Windows Vista X64) and where (for
me) I could see more easily what was going on. Whaterver I tried, I could not get it to index:
keep running out of memory - the Tomcat log is full of Java Heap exceptions from my attempts.
Tomcat would not start with -Xms set to anything greater than 512M and even though I had
more than a 2G free on my machine, it never used more than 600M (was monitoring this with
MS SysInternals Process Explorer).
Second, I attempted running the indexer again on the EC2 machine while watching the process
utilization with the same result. Farooq (our sysadmin) was going to make another attempt
tonight.
I'm not sure what else to do at this point except to provide you with the files and see if you can
replicate the issue in your environment. In case you want to, I've attached a copy of our
database, and the two log files from my session on my windows box in case they may shed any
light. If there was a way to see where the indexer was failiing for each attempt - what database
record it was on each time, that might prove useful.
Finally, this is causing a some problems for us, because we have data that is not accessible to us
right now in the database. We can get back to some items by navigating the relationships we
have setup, but they are not retrievable directly.
Comment by Farooq Sadiq (Inactive) [ 23/Oct/09 ]
Kenton Wrote :
Jama has taken a look at it and found that larger items, over 100K, cause the indexer to run out
of memory.
Generated at Wed May 03 18:13:01 EDT 2017 using JIRA 6.1.5#6160sha1:a61a0fc278117a0da0ec9b89167b8f29b6afdab2.