Using Object Pool Design Pattern to Reduce Garbage Collection in Solr

Object Pool is a common used design pattern in Android.  The most famous example is android.os.Message class.

The same pattern can be used in Solr. When Solr is running in machine with limited resource or running at client machine, and it's important to reduce resource usage as possible as we can.

In Solr application, we usually have to create thousands or millions of SolrQueryRequest, SolrQueryResponse, UpdateCommand.

As described at Solr: Export Large(Millions) Data to a CSV File, it may create a lot of SolrQueryRequest objects in short time, also as described at Solr RefCounted: Don't forget to close SolrQueryRequest or decref solrCore.getSearcherIt's important to close SolrQueryRequest otherwise it will cause resource(SolrIndexSearcher) leak.

So this is a good place to use the object pool pattern to reuse SolrQueryRequest and encapsulate its creation and close operations.

PooledLocalSolrQueryRequest
How to Use PooledLocalSolrQueryRequest
1. Call one of the obtain methods in PooledLocalSolrQueryRequest: obtain(solrCore), obtain(solrCore, solrParams), obtain(solrCore, solrParams, streams) to get a SolrQueryRequest object: it will reuse existing one if exists or create a new if no available free object.

2. Call pooledRequest.recycle() when no need to use the pooledRequest instance any more.

3. Call PooledLocalSolrQueryRequest.closeAll() to clean all cached free PooledLocalSolrQueryRequest instances.
The Code
package org.codeexample.lifelongprogrammer.solr.util;

public class PooledLocalSolrQueryRequest extends LocalSolrQueryRequest {
  protected static final Logger logger = LoggerFactory
      .getLogger(CVCSVExportToFileHandler.class);
  
  /**
   * Private constructor, only allow new objects from obtain()
   */
  private PooledLocalSolrQueryRequest(SolrCore core,
      ModifiableSolrParams modifiableSolrParams) {
    super(core, modifiableSolrParams);
  }
  
  // Reference to next object in the pool
  private PooledLocalSolrQueryRequest next;
  
  private static final Object sPoolSync = new Object();
  private static PooledLocalSolrQueryRequest firstInstance;
  
  // private static int sPoolSize = 0;
  // private static final int MAX_POOL_SIZE = 50;
  
  /**
   * Return a new LocalSolrQueryRequest, caller has to set SolrParmas, and
   * ContentStream if needed.
   */
  public static PooledLocalSolrQueryRequest obtain(SolrCore core) {
    synchronized (sPoolSync) {
      if (firstInstance != null) {
        PooledLocalSolrQueryRequest m = firstInstance;
        firstInstance = m.next;
        m.next = null;
        // sPoolSize--;
        return m;
      }
    }
    return new PooledLocalSolrQueryRequest(core, new ModifiableSolrParams());
  }
  
  public static PooledLocalSolrQueryRequest obtain(SolrCore core,
      ModifiableSolrParams newParams) {
    PooledLocalSolrQueryRequest request = obtain(core);
    request.setParams(newParams);
    return request;
  }
  
  public static PooledLocalSolrQueryRequest obtain(SolrCore core,
      ModifiableSolrParams newParams, Iterable<ContentStream> newStream) {
    PooledLocalSolrQueryRequest request = obtain(core);
    request.setParams(newParams);
    request.setContentStreams(newStream);
    return request;
  }
  
  /**
   * Recycle this object. You must release all references to this instance after
   * calling this method.
   */
  public void recycle() {
    
    this.close();
    synchronized (sPoolSync) {
      // if (sPoolSize < MAX_POOL_SIZE) {
      next = firstInstance;
      firstInstance = this;
      // sPoolSize++;
      // }
    }
  }
  
  /**
   * Close All SolrQueryRequest
   */
  public static void closeAll() {
    synchronized (sPoolSync) {
      
      while (firstInstance != null) {
        firstInstance.close();
        firstInstance= null;
        firstInstance = firstInstance.next;
      }
    }
  }
}
Resources
Recycling objects in Android with an Object Pool to avoid garbage collection.
Object Pool
android.os.Message class
Solr: Export Large(Millions) Data to a CSV File
Solr RefCounted: Don't forget to close SolrQueryRequest or decref solrCore.getSearcher
Post a Comment

Labels

Java (159) Lucene-Solr (112) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (38) Eclipse (33) Code Example (31) Linux (25) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) Problem Solving (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) Shell (7) ANT (6) Coding Skills (6) Database (6) Lesson Learned (6) Programmer Skills (6) Scala (6) Tips (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) System Design (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts