Part1: Run Time-Consuming Solr Query Faster: Auto Run Queries X Minutes after Startup and Commit

The Problem
In our web application, the very first request to solr server is a stats query. When there are more than 50 millions data, the first stats query may take 1, 2 or more minutes. As it need load millions of documents, terms into Solr.
For subsequent stats queries, it will run faster as Solr load them into its caches, but it still takes 5 to 10 or more seconds as the stats query is a compute-intensive task, and there is too many data.

We want these stats queries run faster to make the web GUI more responsive.
Main Steps
1. Make the first stats query run faster
This is described in this article: auto run quries X minutes after no update after startup or commit.
2. Make subsequent stats qury run faster.
Task: Make the first stats query run faster
The first stats query is like this: q=*&stats=true&stats.field=szkb&stats.pagination=true&f.szkb.stats.query=*&f.szkb.stats.facet=file_type.
Solr firstSearcher and newSearcher

From Solr wiki:
A firstSearcher event is fired whenever a new searcher is being prepared but there is no current registered searcher to handle requests or to gain autowarming data from (ie: on Solr startup). A newSearcher event is fired whenever a new searcher is being prepared and there is a current searcher handling requests (aka registered).

In our application, we can't use firstSearcher. As there are too many data, and multiple cores in one solr server, the startup would be very slow, it may take 3 to 5 minutes, 
It also may take 1 to 2 minutes to run commit. Also during push date phrase, client will push many data and commit multiple times, we don't want to slow down the commit, or run the queries every time after commit.
Expected Solution
We want run defined queries after no update in last 5 minutes after server startup; run defined queries after no update in last 10 minutes after a commit.
In this way, we will not run these queries too often: we only run them when the data is kind of stable. No update in 10 minutes.
The Implementation
QueryAutoRunner
This singleton classes maintains the mapping between the SolrCore and the queries, and will auto run them X minutes after no update after startup or commit.

public class QueryAutoRunner {
  protected static final Logger logger = LoggerFactory
      .getLogger(QueryAutoRunner.class);
  
  public static final long DEFAULT_RUN_AUTO_QUERIES_AFTER_COMMIT = 1000 * 60 * 10;
  public static final long DEFAULT_RUN_AUTO_QUERIES_AFTER_STARTUP = 1000 * 60 * 2;
  
  public static long RUN_AUTO_QUERIES_AFTER_COMMIT = DEFAULT_RUN_AUTO_QUERIES_AFTER_COMMIT;
  public static long RUN_AUTO_QUERIES_AFTER_STARTUP = DEFAULT_RUN_AUTO_QUERIES_AFTER_STARTUP;
  private ConcurrentHashMap<SolrCore,CoreAutoRunnerState> autoRunQueries = new ConcurrentHashMap<SolrCore,CoreAutoRunnerState>();
  
  private static QueryAutoRunner instance = null;  
  public static QueryAutoRunner getInstance() {
    if (instance == null) {
      synchronized (QueryAutoRunner.class) {
        if (instance == null) {
          instance = new QueryAutoRunner();
        }
      }
    }
    return instance;
  }

  public void scheduleAutoRunnerAfterCommit(SolrCore core) {
    CoreAutoRunnerState autoQueriesState = autoRunQueries.get(core);
    autoQueriesState.setLastUpdateTime(new Date().getTime());
    autoQueriesState.schedule(RUN_AUTO_QUERIES_AFTER_COMMIT,
        RUN_AUTO_QUERIES_AFTER_COMMIT);
  }  
  public void updateLastUpdateTime(SolrCore core) {
    autoRunQueries.get(core).setLastUpdateTime(new Date().getTime());
  }
  
  public synchronized void initQueries(SolrCore core, Set<NamedList> queries) {
    CoreAutoRunnerState autoQueriesState = new CoreAutoRunnerState(core,
        queries);
    autoRunQueries.put(core, autoQueriesState);
    // always run auto queries for first start
    autoQueriesState.schedule(RUN_AUTO_QUERIES_AFTER_STARTUP, -1);
  }
  private QueryAutoRunner() {
    String str = System.getProperty("RUN_AUTO_QUERIES_AFTER_COMMIT");
    if (StringUtils.isNotBlank(str)) {
      try {
        RUN_AUTO_QUERIES_AFTER_COMMIT = Long.parseLong(str);
      } catch (Exception e) {
        logger
            .error("RUN_AUTO_QUERIES_AFTER_COMMIT should be a positive number");
      }
    }
    str = System.getProperty("RUN_AUTO_QUERIES_AFTER_STARTUP");
    if (StringUtils.isNotBlank(str)) {
      try {
        RUN_AUTO_QUERIES_AFTER_STARTUP = Long.parseLong(str);
      } catch (Exception e) {
        logger
            .error("RUN_AUTO_QUERIES_AFTER_STARTUP should be a positive number");
      }
    }
  }
  
  private static class CoreAutoRunnerState {
    protected static final Logger logger = LoggerFactory
        .getLogger(CoreAutoRunnerState.class);
    
    private SolrCore core;
    private AtomicLong lastUpdateTime = new AtomicLong();
    private Set<NamedList> paramsSet = new LinkedHashSet<NamedList>();

    private ScheduledFuture pending;
    private final ScheduledExecutorService scheduler = Executors
        .newScheduledThreadPool(1);

        public CoreAutoRunnerState(SolrCore core, Set<NamedList> queries) {
      this.core = core;
      this.paramsSet = queries;
    }
    
    public void schedule(long withIn, long minTimeNoUpdate) {
      // if there is already one scheduled runner whose remaining time less
      // than withIn (almost always), cancel the old one.
      if (pending != null && pending.getDelay(TimeUnit.MILLISECONDS) < withIn) {
        pending.cancel(false);
        pending = null;
      }
      if (pending == null) {
        pending = scheduler.schedule(new AutoQueriesRunner(minTimeNoUpdate),
            withIn, TimeUnit.MILLISECONDS);
        logger.info("Scheduled to run queries in " + withIn);
      }
    }
    
    private class AutoQueriesRunner implements Runnable {
      private long minTimeNoUpdate;
      
      public AutoQueriesRunner(long minTimeNoUpdate) {
        this.minTimeNoUpdate = minTimeNoUpdate;
      }      
      @Override
      public void run() {
        if (minTimeNoUpdate > 0
            && (new Date().getTime() - lastUpdateTime.get()) < minTimeNoUpdate) {
          long remaingTime = minTimeNoUpdate
              - (new Date().getTime() - lastUpdateTime.get());
          if (remaingTime > 1000) {
            // reschedule auto runner
            pending = scheduler.schedule(
                new AutoQueriesRunner(minTimeNoUpdate), remaingTime,
                TimeUnit.MILLISECONDS);
            return;
          }
        }
        logger.info("Started to execute auto runner for " + core.getName());
        // if there is no update in less than X minutes,
        for (NamedList params : paramsSet) {
          SolrQueryRequest request = null;
          try {
            request = new LocalSolrQueryRequest(core, params);
            
            String qt = request.getParams().get(CommonParams.QT);
            if (StringUtils.isBlank(qt)) {
              qt = "/select";
            }
            request.getContext().put("url", qt);
            core.execute(core.getRequestHandler(request.getParams().get(
                CommonParams.QT)), request, new SolrQueryResponse());
          } catch (Exception e) {
            logger.error("Error happened when run for " + core.getName()
                + " auro query: " + params, e);
          } finally {
            if (request != null) {
              request.close();
            }
          }
        }
        logger.info("Excuted auto runner for " + core.getName());
      }
    }
    public CoreAutoRunnerState setLastUpdateTime(long lastUpdateTime) {
      this.lastUpdateTime.set(lastUpdateTime);
      return this;
    }
  }
}

AutoRunQueriesRequestHandler
This request handler is a abstract handler, not meant to be called via http. It's used to define the query list which will be run automatically at some point, also it will shcedule a AutoRunner in 2 minutes.
Its definition in solrConfig.xml looks like this:

<requestHandler name="/abstracthandler_autorunqueries" class="AutoRunQueriesRequestHandler" >
  <lst name="defaults">
    <arr name="autoRunQueries">
      <lst> 
        <str name="q">*</str>
        <str name="rows">0</str>                 
        <str name="stats">true</str>
        <str name="stats.pagination">true</str>
        <str name="f.szkbround1.stats.query">*</str>
        <str name="stats.field">szkbround1</str>
        <str name="f.szkbround1.stats.facet">ext_name</str>
      </lst>
    </arr>
  </lst>
</requestHandler>

public class AutoRunQueriesRequestHandler extends RequestHandlerBase
    implements SolrCoreAware {  
  private Set<NamedList> paramsSet = new LinkedHashSet<NamedList>();
  private static final String PARAM_AUTO_RUN_QUERIES = "autoRunQueries";
  public void init(NamedList args) {
    super.init(args);
    if (args != null) {
      NamedList nl = (NamedList) args.get("defaults");
      List<NamedList> allLists = (List<NamedList>) nl
          .get(PARAM_AUTO_RUN_QUERIES);
      if (allLists == null) return;
      for (NamedList nlst : allLists) {
        if (nlst.get("distrib") == null) {
          nlst.add("distrib", false);
        }
        paramsSet.add(nlst);
      }
    }
  }
  public void inform(SolrCore core) {
    if (!paramsSet.isEmpty()) {
      QueryAutoRunner.getInstance().initQueries(core, paramsSet);
    }
  }
  public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp)
      throws Exception {
    throw new SolrServerException("Abstract Hanlder, not meant to be called.");
  }
}

AutoRunQueriesProcessorFactory
This processor factory needed to be added in the default processor chain, and all updateRequestProcessorChain. The InvalidateCacheProcessorFactory is used to invalidate the Solr response cache. It's described at a later post.

<updateRequestProcessorChain name="defaultChain" default="true">
  <processor class="solr.LogUpdateProcessorFactory" />
  <processor class="solr.RunUpdateProcessorFactory" />
  <processor class="InvalidateCacheProcessorFactory" />
  <processor
   class="AutoRunQueriesProcessorFactory"/>      
</updateRequestProcessorChain>

It's processAdd, processDelete will update lastUpdateTime of CoreAutoRunnerState, its processCommit method will schedule a AutoRunner in 10 minutes. 

public class AutoRunQueriesProcessorFactory extends
    UpdateRequestProcessorFactory {
  public UpdateRequestProcessor getInstance(SolrQueryRequest req,
      SolrQueryResponse rsp, UpdateRequestProcessor next) {
    return new AutoRunQueriesProcessor(next);
  }
  
  private static class AutoRunQueriesProcessor extends UpdateRequestProcessor {
    public AutoRunQueriesProcessor(UpdateRequestProcessor next) {
      super(next);
    }
    public void processAdd(AddUpdateCommand cmd) throws IOException {
      updateLastUpdateTime(cmd);
      super.processAdd(cmd);
    }
    public void processDelete(DeleteUpdateCommand cmd) throws IOException {
      updateLastUpdateTime(cmd);
      super.processDelete(cmd);
    }
    public void processCommit(CommitUpdateCommand cmd) throws IOException {
      super.processCommit(cmd);
      QueryAutoRunner.getInstance().scheduleAutoRunnerAfterCommit(
          cmd.getReq().getCore());
    }
    public void updateLastUpdateTime(UpdateCommand cmd) {
      QueryAutoRunner.getInstance().updateLastUpdateTime(
          cmd.getReq().getCore());
    }
  }
}