How Solr uses RefCounted
RefCounted is an important concept in Solr, it keeps track of a reference count on a resource and close it when the count hits zero.
For example, a Solr core reuses SolrIndexSearcher, it uses RefCounted to keep track of its count and close it when the count hits zero.
when Solr initializes a Solr core, it will create one SolrIndexSearcher, and put it into searcherList, but keep 2 reference to this searcher: one is in the searcherList(_realtimeSearchers or _searchers), one is the variable realtimeSearcher.
from its searchList, thus there will be no pointer to the searcher, GC will reclaim it.
Code: SolrCore.newHolder(SolrIndexSearcher, List>)
When we send a request to Solr for a request handler, SolrDispatchFilter will create a SolrQueryRequest which has a RefCounted searcherHolder. In Solr request handler, it can call SolrQueryRequest.getSearcher() to get SolrIndexSearcher, this will increase the reference count to SolrIndexSearcher. At the end of the request, SolrDispatchFilter will call SolrQueryRequest.close() which will decrease the reference count.
How we should write our own code
In our own code, if we create a SoelrQueryRequest, then we have to call close after we don't need it.
If we call req.getCore().getSearcher() - this will increase the reference count, after we are done with the searcher, we have to call decref to decrease the reference count.
Otherwise it will cause memory leak, as the reference count of this searcher would never be zero, thus will not be close, and caches such as filterCache, queryResultCache, documentCache, fieldValueCache will not be cleaned, it will be kept in the
Code example
To demonstrate, I write a simple request handler which create a SolrRequestHandler to run facet query. But we forgot to close the SolrRequestHandler.
Lesson Learned
1. Read the documentation/javadoc when we use some API.
For example in javadoc of SolrQueryRequest, we know it's not thread safe, so we shouldn't share it in multi thread environment, we know we should call its close method explicitly when we no need it any more.
RefCounted is an important concept in Solr, it keeps track of a reference count on a resource and close it when the count hits zero.
For example, a Solr core reuses SolrIndexSearcher, it uses RefCounted
when Solr initializes a Solr core, it will create one SolrIndexSearcher, and put it into searcherList, but keep 2 reference to this searcher: one is in the searcherList(_realtimeSearchers or _searchers), one is the variable realtimeSearcher.
org.apache.solr.core.SolrCore.openNewSearcher(boolean, boolean)
RefCounted<SolrIndexSearcher> newSearcher = newHolder(tmp, searcherList); // refcount now at 1
// Increment reference again for "realtimeSearcher" variable. It should be at 2 after.
// When it's decremented by both the caller of this method, and by realtimeSearcher being replaced,
// it will be closed.
newSearcher.incref();
realtimeSearcher = newSearcher;
searcherList.add(realtimeSearcher);
Solr core always keeps 2 reference to the SolrIndexSearcher instance until the solr core is closed/ unloaded which will call SolrCore.closeSearcher(), this will decrease the count to 0, then SolrCore will release related resource, then remove its reference variable RefCountedCode: SolrCore.newHolder(SolrIndexSearcher, List
When we send a request to Solr for a request handler, SolrDispatchFilter will create a SolrQueryRequest which has a RefCounted
How we should write our own code
In our own code, if we create a SoelrQueryRequest, then we have to call close after we don't need it.
If we call req.getCore().getSearcher() - this will increase the reference count, after we are done with the searcher, we have to call decref to decrease the reference count.
Otherwise it will cause memory leak, as the reference count of this searcher would never be zero, thus will not be close, and caches such as filterCache, queryResultCache, documentCache, fieldValueCache will not be cleaned, it will be kept in the
Code example
To demonstrate, I write a simple request handler which create a SolrRequestHandler to run facet query. But we forgot to close the SolrRequestHandler.
public class DemoUnClosedSolrRequest extends RequestHandlerBase {
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)
throws Exception {
SolrCore core = req.getCore();
SolrQuery query = new SolrQuery();
query.setQuery("datatype:4").setFacet(true).addFacetField("szkb")
.setFacetMinCount(2);
SolrQueryRequest facetReq = new LocalSolrQueryRequest(core, query);
try {
SolrRequestHandler handler = core.getRequestHandler("/select");
handler.handleRequest(facetReq, new SolrQueryResponse());
} finally {
// Don't forget to close SolrQueryRequest
//facetReq.close();
}
}
}
Test Codepublic void unclosedSearcher() throws Exception {
long startTime = System.currentTimeMillis();
int i = 0;
try {
for (; i < 100000; i++) {
// in the server, this request handler will increase the reference
// count, but forget to decrease it.
HttpSolrServer server = new HttpSolrServer(
"http://mailsearchsolr:7766/solr");
AbstractUpdateRequest request = new UpdateRequest("/demo1");
server.request(request);
// Nomally commit will close the old search, and open a new searcher,
// but in this case because the reference of SolrIndexSearcher is not 0,
// so the old search will not be cleaned.
request = new UpdateRequest("/update");
request.setParam("commit", "true");
server.request(request);
}
} catch (Exception e) {
e.printStackTrace();
throw e;
} finally {
System.out.println("run " + i + " times");
System.out.println("Took " + (System.currentTimeMillis() - startTime)
+ " mills");
}
}
If we run this DemoUnClosedSolrRequest, then run a commit, in normal case, this will close the old searcher, and open a new searcher. But in this case, as the reference count in the old searcher is not zero, so the old searcher will not be closed, and it will be kept in memory forever - including its caches, as noone will remove it from SolrCore.searchList. Thus GC can't reclaim it.Test Result
After run 6817 times, 58 minutes, client will fail.
org.apache.solr.common.SolrException: Server at http://host:port/solr returned non ok status:500, message:Server Error
run 6817 times
Took 3513530 mills
In the server, it throws OutOfMemoryError:
SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:733)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.OpenBitSet.(OpenBitSet.java:88)
at org.apache.solr.search.DocSetDelegateCollector.collect(DocSetDelegateCollector.java:56)
at org.apache.lucene.search.Scorer.score(Scorer.java:64)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:605)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1553)
at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1300)
at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:395)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:412)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at com.commvault.solr.handler.DemoUnClosedSearcher.handleRequestBody(DemoUnClosedSearcher.java:24)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
The memory usage looks like below: increase constantly.
Solr Admin Console
In Solr Admin console, we can see there are a lot of SolrIndexSearcher instances.
CPU usage
After Java throws OutOfMemoryError, JAVA tries hard to do GC, but no impact at all. As these searchers are kept in SolrCore.searchList forever.
Another example
This example demonstrates what will happen if we forgot to decref Core().getSearcher(). Same thing will happen - the old searcher will not be cleaned instead it will be kept in core.searchList forever.public class DemoUnClosedSearcher extends RequestHandlerBase {
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)
throws Exception {
RefCounted<SolrIndexSearcher> refCounted = req.getCore().getSearcher();
try {
SolrIndexSearcher searcher = refCounted.get();
String qstr = "datatype:4";
QParser qparser = QParser.getParser(qstr, "lucene", req);
Query query = qparser.getQuery();
int topn = 1;
TopDocs topDocs = searcher.search(query, topn);
for (int i = 0; i < topDocs.totalHits; i++) {
ScoreDoc match = topDocs.scoreDocs[i];
Document doc = searcher.doc(match.doc);
System.out.println(doc.get("contentid"));
}
} finally {
// Dont' forget to decref efCounted<SolrIndexSearcher>
//refCounted.decref();
}
}
}
Lesson Learned
1. Read the documentation/javadoc when we use some API.
For example in javadoc of SolrQueryRequest, we know it's not thread safe, so we shouldn't share it in multi thread environment, we know we should call its close method explicitly when we no need it any more.
Also for SolrCore.getSearcher(): It must be decremented when no longer needed. SolrCoreState.getIndexWriter: It must be decremented when no longer needed.
2. Use tools like VisualVM to monitor threads and memory usage.
3. Solr uses JMX to monitor its resource. We can check values in SOlr Admin or visualVM.
4. Generate and analyze Heap dump - visualvm or eclipse mat. Use DQL to find the instances, and check values.
