Solr: Using LengthFilterFactory to Reduce Index Size and Memory Usage

Our Goal: Use less Memory
As our Solr application runs in client machine, so it's important for us to use less memory as possible as we can.

In Solr, we have fields which are used only for search and one way to reduce the index size and memory usage is to remove the term that is larger than the threshold: for example: 50. 

The reason is that the user is very unlikely to search on these large terms, so why keep them in the index.

The Definition for content field

 <fieldType name="text_rev_trucated" class="solr.TextField" positionIncrementGap="100" >
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.EnglishMinimalStemFilterFactory"/>
    <filter class="solr.LengthFilterFactory" min="1" max="50"/>
    <filter class="solr.ReversedWildcardFilterFactory" withOriginal="true" maxPosAsterisk="3" maxPosQuestion="2" maxFractionAsterisk="0.33"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>       
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="0" preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.EnglishMinimalStemFilterFactory"/>
  </analyzer>
</fieldType>

Other Tricks
Reduce fetch size when retrieve data from remote server.
Commit frequently.
Reduce the cache size
Monitor memory usage and run gc when needed.

Solr: Use STAX Parser to Read XML Response to Reduce Memory Usage

My Solr client talks with a proxy application which communicates with remote Solr Server to get data. 
In previous post, Solr: Use JSON(GSon) Streaming to Reduce Memory Usage, I described the problem we faced, how to use JSON(GSon) Streaming, and also some other approaches to reduce memory usage. 

In post Solr: Use SAX Parser to Read XML Response to Reduce Memory Usage
I also described how to use SAX to parse response for better performance.

In this post, I will introduce how to use Stax Parser to parse XML response.
Implementation
The code to use Stax to read document one by one from http stream:
-- Use Stax parser and Java Executors Future to wait all thread finished: all docs imported.

private static ImportedResult handleXMLResponseViaStax(
      SolrQueryRequest request, InputStream in, int fetchSize)
      throws XMLStreamException {
    XMLInputFactory factory = XMLInputFactory.newInstance();
    XMLStreamReader reader = null;
    
    ImportedResult importedResult = new ImportedResult();
    List<Future<Void>> futures = new ArrayList<Future<Void>>();
    
    try {
      reader = factory.createXMLStreamReader(in);
      int fetchedSize = 0;
      int numFound = -1, start = -1;
      
      while (reader.hasNext()) {
        int event = reader.next();
        switch (event) {
          case XMLStreamConstants.START_ELEMENT: {
            if ("result".equals(reader.getLocalName())) {
              numFound = Integer.valueOf(reader.getAttributeValue("",
                  "numFound"));
            } else if ("start".equals(reader.getLocalName())) {
              start = Integer.valueOf(reader.getAttributeValue("", "start"));
            } else if ("doc".equals(reader.getLocalName())) {
              ++fetchedSize;
              futures.add(readOneDoc(request, reader));
            }
            break;
          }
          default:
            break;
        }
        
      }
      importedResult.setFetched(fetchedSize);
      importedResult.setHasMore((fetchedSize + start) < numFound);
      importedResult.setImportedData((fetchedSize != 0));
      return importedResult;
    } finally {
      if (reader != null) {
        reader.close();
      }
    }
  }
  
  private static Future<Void> readOneDoc(SolrQueryRequest request,
      XMLStreamReader reader) throws XMLStreamException {
    String contentid = null, bindoc = null;
    OUTER: while (reader.hasNext()) {
      int event = reader.next();
      INNER: switch (event) {
        case XMLStreamConstants.START_ELEMENT: {
          if ("str".equals(reader.getLocalName())) {
            
            String fieldName = reader.getAttributeValue(0);
            if ("contentid".equals(fieldName)) {
              contentid = reader.getElementText();
            } else if ("bindoc".equals(fieldName)) {
              bindoc = reader.getElementText();
            }
          }
          break INNER;
        }
        case XMLStreamReader.END_ELEMENT: {
          if ("doc".equals(reader.getLocalName())) {
            break OUTER;
          }
        }
        default:
          break;
      }
    }
    return CVSyncDataImporter.getInstance().importData(request, contentid,
        bindoc);
  }
    

Resources
Parsing XML using DOM, SAX and StAX Parser in Java
Java SAX vs. StAX

Solr: Use SAX Parser to Read XML Response to Reduce Memory Usage

My Solr client talks with a proxy application which talks with remote Solr Server to get data. 
In previous post, Solr: Use JSON(GSon) Streaming to Reduce Memory Usage

I described the problem we faced, how to use JSON(GSon) Streaming, and also some other approaches to reduce memory usage. In this post I will use XML SAX Parser to iterative xml response stream. In next post I will introduce how to use Stax Parser to parse XML response.
Implementation
The code to use SAX to read document one by one from http stream:
-- Use SAX parser and Java Executors Future to wait all thread finished: all docs imported.

private static ImportedResult handleXMLResponseViaSax(
      SolrQueryRequest request, InputStream in, int fetchSize)
      throws IOException, ParserConfigurationException, SAXException {
 
    ImportedResult importedResult = new ImportedResult();
    SAXParserFactory parserFactor = SAXParserFactory.newInstance();
    SAXParser parser = parserFactor.newSAXParser();
    SolrResponseHandler handler = new SolrResponseHandler(request);
    parser.parse(in, handler);
    
    importedResult.setFetched(handler.fetchedSize);
    importedResult
        .setHasMore((handler.fetchedSize + handler.start) < handler.numFound);
    importedResult.setImportedData((handler.fetchedSize != 0));    
    return importedResult;
  }
  
  private static class SolrResponseHandler extends DefaultHandler {
    protected int fetchedSize = 0;
    protected int numFound = -1, start = -1;
    protected String contentid, bindoc = null;
    protected List<Future<Void>> futures = new ArrayList<Future<Void>>();
    
    String curName, curValue;
    private SolrQueryRequest request;
    
    public SolrResponseHandler(SolrQueryRequest request) {
      this.request = request;
    }
    
    @Override
    public void startElement(String uri, String localName, String qName,
        Attributes attributes) throws SAXException {
      
      switch (qName) {
        case "result": {
          numFound = Integer.valueOf(attributes.getValue("numFound"));
          start = Integer.valueOf(attributes.getValue("start"));
          break;
        }
        case "str": {
          String name = attributes.getValue("name");
          if ("contentid".equals(name)) {
            curName = "contentid";
          } else if ("bindoc".equals(name)) {
            curName = "bindoc";
          }
          break;
        }
        default:
          break;
      }
    }
    
    @Override
    public void endElement(String uri, String localName, String qName)
        throws SAXException {
      switch (qName) {
        case "str": {
          if ("contentid".equals(curName)) {
            contentid = curValue;
          } else if ("bindoc".equals(curName)) {
            bindoc = curValue;
          }
          break;
        }
        case "doc": {
          ++fetchedSize;
          futures.add(CVSyncDataImporter.getInstance().importData(request,
              contentid, bindoc));
          break;
        }
        default:
          break;
      }
    }
    @Override
    public void characters(char[] ch, int start, int length)
        throws SAXException {
      curValue = String.copyValueOf(ch, start, length).trim();
    }
  }
    

Resources
Parsing XML using DOM, SAX and StAX Parser in Java
Java SAX vs. StAX

Solr: Use JSON(GSon) Streaming to Reduce Memory Usage

My Solr application runs at user's laptop, the max memory is set to 512mb. It pull JSON data from remote proxy which talks with remote Solr Server: 100m hundred at a time, commit after 20 times.

Our code gets the whole json and put it into memory, and use UpdateRequestProcessor.processAdd(AddUpdateCommand) to add it into local solr.

Recently it throws OutOfMemoryError, after use Eclipse Memory Analyzer (MAT) to analyze the heapdump file.

I found out that it is because the data returned from remote proxy is too large:, one data is 50-60kb on average. But some data is huge, 100 data would be 60 mb: this is rare case, but when this happens it will cause the application throws OutOfMemoryError and stops to work.

To fix this and reduce memory usage at client side, I take several measures:

1. Reboot the application when OutOfMemoryError happens.
2. Run a thread to monitor free memory, at a certain threshold(40%), run gc. If less than 30%, decrease fetch size(100 to 50, to 25) and decrease commit interval( 20 times, 10 times). If less than 50 mb memory, restart the application.
3. Enable Auto SoftCommit and AutoCommit, reduce Solr cache size.
3. Use Streaming JSON. - This is the topic of this article.
Read document one by one from http input stream, put it to queue, instead read the whole big document in to memory. Another thread is responsible to write the document to local solr.

Same approach apples if we use XML: we can use StAX or SAX to read document one by one.

I use GSON, about how to use Gson Streaming to read and write JSON, please read Gson Streaming 

The code to read document one by one from http stream:
-- Use GSon Stream API and Java Executors Future to wait all thread finished: all docs imported.

/**
 * Use Gson Streaming API to read docuemts one by one to reduce memory usage 
 */
private static ImportedResult handleResponse(SolrQueryRequest request,
    InputStream in, int fetchSize) throws UnsupportedEncodingException,
    IOException {
  ImportedResult importedResult = new ImportedResult();
  JsonReader reader = null;
  List<Future<Void>> futures = new ArrayList<Future<Void>>();
  
  try {
    reader = new JsonReader(new InputStreamReader(in, "UTF-8"));
    reader.beginObject();
    String str = reader.nextName();
    reader.beginObject();
    int fetchedSize = 0;
    int numFound = -1, start = -1;
    while (reader.hasNext()) {
      str = reader.nextName();
      if ("numFound".equals(str)) {
        numFound = Integer.valueOf(reader.nextString());
      } else if ("start".equals(str)) {
        start = Integer.valueOf(reader.nextString());
      } else if ("docs".equals(str)) {
        reader.beginArray();
        // read documents
        while (reader.hasNext()) {
          fetchedSize++;
          readOneDoc(request, reader);
        }
        
        reader.endArray();
      }
    }
    
    reader.endObject();
    waitComplete(futures);
    importedResult.setFetched(fetchedSize);
    importedResult.setHasMore((fetchedSize + start) < numFound);
    importedResult.setImportedData((fetchedSize != 0));
    return importedResult;
  } finally {
    if (reader != null) {
      reader.close();
    }
  }
}

private static java.util.concurrent.Future<Void> readOneDoc(
    SolrQueryRequest request, JsonReader reader) throws IOException {
  String str;
  reader.beginObject();
  String id = null, binaryDoc = null;
  while (reader.hasNext()) {
    str = reader.nextName();
    
    if ("id".equals(str)) {
      id = reader.nextString();
    } else if ("binaryDoc".equals(str)) {
      binaryDoc = reader.nextString();
    }
  }
  reader.endObject();
  return CVSyncDataImporter.getInstance().importData(request, id,
      binaryDoc);
}

The code to write document to local solr:

public Future<Void> importData(SolrQueryRequest request, String id,
    String binaryDoc) {
  if (id == null) {
    throw new IllegalArgumentException("id can't be null.");
  }
  if (binaryDoc == null) {
    throw new IllegalArgumentException("binaryDoc can't be null.");
  }
  SolrDataImporter task = new SolrDataImporter(request, id, binaryDoc);
  return executor.submit(task);
}
private static SolrInputDocument convertToSolrDoc(String id,
    String binaryDoc) throws IOException {
  byte[] bindata = Base64.base64ToByteArray(binaryDoc);
  SolrInputDocument resultDoc = (SolrInputDocument) readZippedFile(bindata);
  resultDoc.setField("id", id);
  return resultDoc;
}

private class SolrDataImporter implements Callable<Void> {
  private SolrQueryRequest request;
  private String id, binaryDoc;
  @Override
  public Void call(){
    try {
      UpdateRequestProcessorChain updateChian = request.getCore()
          .getUpdateProcessingChain("mychain");
      SolrInputDocument toSolrServerSolrDoc = convertToSolrDoc(id,
          binaryDoc);
      binaryDoc = null;
      AddUpdateCommand command = new AddUpdateCommand(request);
      command.solrDoc = toSolrServerSolrDoc;
      SolrQueryResponse response = new SolrQueryResponse();
      UpdateRequestProcessor processor = updateChian.createProcessor(request,
          response);
      processor.processAdd(command);
    } catch (Exception e) {
      logger.error("Exception happened when importdata, id: "
          + id, e);
    }
    return null;
  }
}