Java Lambda-Enabled Concurrency - Don't Use Mutable Variables

What's wrong in following code?
The following code tries to submit all query task to threadpool, but it fails to do that - It only reads part of data.
private List<Future<List<T>>> findAllAsyncImpl(final SolrParams params, final int readSize,
        final int totalCount) {
    final ModifiableSolrParams query = new ModifiableSolrParams(params);
    query.set(CommonParams.ROWS, readSize);
    final List<Future<List<T>>> futures = new ArrayList<>(totalCount);
    int start = 0;
    while (start < totalCount) {
        query.set(CommonParams.START, start);
        futures.add(executor.submit(() -> querySolr(new ModifiableSolrParams(query))));
        start += readSize;
    }
    return futures;
}

I checked the log - what I expected is that it runs query like start=0, 1000, 2000, until totalCount-1000 (readSize is 1000)-  but it's not, it runs query with some start value (here start=45000) multiple times, some start value 0 time.
AbstractSolrRepository querySolr: q=omitted...&rows=1000&start=45000
AbstractSolrRepository querySolr: q=omitted...&rows=1000&start=45000
AbstractSolrRepository querySolr: q=omitted...&rows=1000&start=45000
AbstractSolrRepository querySolr: q=omitted...&rows=1000&start=45000

What the code is doing
Now let's change the lambda expression to the anonymous class() - Eclipse quick fix(Command +1) then select "convert to anonymous class creation".
The second part code creates explicit callable.


while (start < totalCount) {
    query.set(CommonParams.START, start);
    futures.add(executor.submit(new Callable<List<T>>() {
        @Override
        public List<T> call() throws Exception {
            return querySolr(new ModifiableSolrParams(query));
        }
    }));
    start += readSize;
}


while (start < totalCount) {
    query.set(CommonParams.START, start);
    futures.add(executor.submit(new SolrQueryTask(query)));
    start += readSize;
}

private class SolrQueryTask implements Callable<List<T>> {
    private final ModifiableSolrParams query;
    public SolrQueryTask(final ModifiableSolrParams query) {
        this.query = query;
    }
    public List<T> call() throws Exception {
        return querySolr(new ModifiableSolrParams(query));
    }
}
Now the root cause is kind of clear: the task is running in a different thread(for example at that time start=0), it refers variable whose state is changed later,  when the task is actually running, the query variable is already changed.

The solution
private List<Future<List<T>>> findAllAsync
    (final SolrParams params, final int readSize, final int totalCount) {
    final List<Future<List<T>>> futures = new ArrayList<>(totalCount);
    int start = 0;
    while (start < totalCount) {
        final ModifiableSolrParams targetParams =
        new ModifiableSolrParams(params).set(CommonParams.ROWS, readSize).set(CommonParams.START, start);

        futures.add(executor.submit(() -> querySolr(targetParams)));
        start += readSize;
    }
    return futures;
}

Post a Comment

Labels

Java (159) Lucene-Solr (110) All (58) Interview (58) J2SE (53) Algorithm (41) Soft Skills (36) Eclipse (34) Code Example (31) Linux (25) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Continuous Integration (8) Dynamic Languages (8) Http Client (8) Maven (8) Security (8) Trouble Shooting (8) bat (8) blogger (8) Big Data (7) Google (7) Guava (7) JSON (7) Problem Solving (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) IDE (5) Lesson Learned (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Python (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Troubleshooting (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts