Windows BAT: Using Apache Procrun to Install Java Application As Windows Service

I have a java application which combines embedded jetty and Solr in one application, and wrap it as a windows service.

This embedded jetty and Solr application should be able to be installed at different locations, and run as a different services.

To do this: I use Apache procrun, put prunsrv.exe and prunmgr.exe in bin folder, create several windowshowAdsenses bat files to install it as a widows service, to uninstall it, start and stop the services.
installService.bat
We can install the application with specified service name, service display name, service description, we can use start_params to specify parameters to the application like port number.

The script will first use %PRUNSRV% //IS, %PRUNSRV% //US  to install it as a service, then copy prunmgr.exe to %service_name%.exe, so we can use %service_name%.exe to monitor, edit and manage the service.

installService.bat -service_name service1 -service_display_name display_name1 -service_description service_description1 -start_params "start;-port;9753" -stop_params shutdown

@echo off
@echo off
@setlocal enableextensions enabledelayedexpansion
goto install

:parseParams
set key=%~1
set value=%~2
if "%key%" == "" goto :eof
if "%key%" == "-PR_JAVA_OPTIONS" (   
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -PR_JAVA_OPTIONS
        goto end
    )
 SET PR_JAVA_OPTIONS=%value%
) else if "%key%" == "-remotedebug" (  
 shift
 SET remotedebug=true
) else if "%key%" == "-action" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -action
        goto end
    )
 SET ACTION=%value%
) else if "%key%" == "-service_name" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_name
        goto end
    )
 SET SERVICE_JAVA=%value%
) else if "%key%" == "-service_display_name" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_display_name
        goto end
    )
 SET PR_DESPLAYNAME=%value%
) else if "%key%" == "-service_description" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_description
        goto end
    )
 SET PR_DESCRIPTION=%value%
) else if "%key%" == "-start_params" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -start_params
        goto end
    )
 SET START_PARAMS=%value%
) else if "%key%" == "-stop_params" (  
 shift
 shift
 if "%value%" == "" (
        echo Empty value for -stop_params
        goto end
    )
 SET STOP_PARAMS=%value%
) else if "%key%" == "-pr_jvm" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -pr_jvm
        goto end
    )
 SET PR_JVM=%value%
) else if "%key%" == "-uninstallIfExsit" (  
 shift
 SET UNINSTALL_IF_EXSIT=true
) else (
 shift
)
goto parseParams %*
goto :eof

:install
set SERVICE_JAVA=MyApp
set PR_DESPLAYNAME=MyApp
set PR_DESCRIPTION="MyApp Service"
set START_PARAMS=start
set STOP_PARAMS=shutdown
set PR_JVM=auto
SET PR_JAVA_OPTIONS="-Xms512m;"
SET CMD_OPTS=%*
set APP_RUNNING_HOME=
set "APP_HOME=%~dp0/../"
set "PATH_PRUNSRV=%~dp0"
set "PR_LOGPATH=%APP_HOME%/logs"
set UNINSTALL_IF_EXSIT=false
SET MYCLASSPATH=
rem set PR_INSTALL=%PATH_PRUNSRV%prunsrv.exe

call :parseParams %*
set PRUNSRV=%PATH_PRUNSRV%%SERVICE_JAVA%.exe
if "%PR_DESPLAYNAME%" == "" (
 set PR_DESPLAYNAME=%SERVICE_JAVA%
)

REM check whether service already exists
echo Installing %SERVICE_JAVA%
SC QUERY %SERVICE_JAVA% > NUL
REM if ERRORLEVEL 1060 means it doesn't exists
IF not ERRORLEVEL 1060 (
 if "%UNINSTALL_IF_EXSIT%" == "true" (
  "%PRUNSRV%" //DS//%SERVICE_JAVA%
 ) else  (
  echo %SERVICE_JAVA% already exists, and uninstallIfExsit is false. Do nothing
  goto end
 )
)

"%PRUNSRV%" //IS//%SERVICE_JAVA% --Install="%PRUNSRV%"

if not errorlevel 1 goto installed
echo Failed to install '%SERVICE_JAVA%' service
goto end
:installed
echo The service '%SERVICE_JAVA%' has been installed.
set libDir=%APP_HOME%\lib
set MYCLASSPATH="%libDir%\*;%libDir%\ext\*;%libDir%\shared-libs\*;%APP_HOME%\etc"
if "%remotedebug%" == "true" (
 SET PR_JAVA_OPTIONS="%PR_JAVA_OPTIONS%;-agentlib:jdwp=transport=dt_socket,address=9999,server=y,suspend=n"
)

echo Setting the parameters for %SERVICE_JAVA%
echo "%PRUNSRV%" //US//%SERVICE_JAVA% --DisplayName "%PR_DESPLAYNAME%" --Description "%PR_DESCRIPTION%" --StdOutput auto --StdError auto ^
--Classpath=%MYCLASSPATH% --Jvm="%PR_JVM%" --JvmOptions="%PR_JAVA_OPTIONS%" --Startup=auto ^
--StartMode=jvm --StartClass=MainClass --StartParams="%START_PARAMS%" ^
--StopMode=jvm  --StopClass=MainClass  --StopParams="%STOP_PARAMS%"

"%PRUNSRV%" //US//%SERVICE_JAVA% --DisplayName "%PR_DESPLAYNAME%" --Description "%PR_DESCRIPTION%" --StdOutput auto --StdError auto ^
--Classpath=%MYCLASSPATH% --Jvm="%PR_JVM%" --JvmOptions="%PR_JAVA_OPTIONS%" --StartPath %APP_HOME% --Startup=auto ^
--StartMode=jvm --StartClass=MainClass --StartParams="%START_PARAMS%" ^
--StopMode=jvm  --StopClass=MainClass  --StopParams="%STOP_PARAMS%"

if not errorlevel 1 goto updated
echo Failed updating '%SERVICE_JAVA%' service
goto end
:updated

If NOT exist %SERVICE_JAVA%w.exe (
 copy /B prunmgr.exe %SERVICE_JAVA%w.exe
)
echo The service '%SERVICE_JAVA%' has been updated.
echo Installation of %SERVICE_JAVA% is complete
goto end

:end
endlocal 
@echo on

uninstallService.bat
To uninstall the service, we just need specify service name:
uninstallService.bat -service_name service1
It will use "%PRUNSRV%" //DS to uninstall the service.

@echo off
setlocal
goto uninstall

:parseParams
set key=%~1
set value=%~2
if "%key%" == "" goto :eof
if "%key%" == "-service_name" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_name
        goto end
    )
 SET SERVICE_JAVA=%value%
) else if "%key%" == "-delete_exe" (  
 shift
 SET DELETE_EXE=true
) else (
 shift
)
goto parseParams %*
goto :eof

:uninstall
set MYPATH=%~dp0
set PATH_PRUNSRV=%MYPATH%
set "PR_LOGPATH=%MYPATH%/../logs"
set SERVICE_JAVA=MyApp
set DELETE_EXE=false

call :parseParams %*
set PRUNSRV=%PATH_PRUNSRV%%SERVICE_JAVA%.exe
echo Removing %SERVICE_JAVA%

cmd /c stopService.bat -service_name %SERVICE_JAVA%
"%PRUNSRV%" //DS//%SERVICE_JAVA%

if "%DELETE_EXE%" == "true" (
  del /Q %SERVICE_JAVA%.exe
 echo %SERVICE_JAVA%.exe deleted
)
if not errorlevel 1 goto removed
echo.
echo Failed uninstalling '%SERVICE_JAVA%' service
goto :eof
:removed
echo The service '%SERVICE_JAVA%' has been removed

endlocal
@echo on

startService.bat/stopService.bat
It's very easy to implement: just get service name, then use %PRUNSRV% //ES to start the service, use %PRUNSRV% //SS to stop the service.
startService.bat/stopService.bat -service_name service1
startService.bat

@echo off
setlocal

goto run
:parseParams
set key=%~1
set value=%~2
echo %key% %value%
if "%key%" == "" goto :eof
if "%key%" == "-service_name" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_name
        goto end
    )
 SET SERVICE_JAVA=%value%
) else if "%key%" == "-install_if_not_exist" (  
  shift
 SET INSTALL_IF_NOT_EXIST=true
) else if "%key%" == "-service_description" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_description
        goto end
    )
 SET service_description=%value%
) else if "%key%" == "-service_display_name" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_display_name
        goto end
    )
 SET service_display_name=%value%
) else if "%key%" == "-start_params" (  
  shift
 shift
 if "%value%" == "" (
        echo Empty value for -start_params
        goto end
    )
 SET start_params=%value%
) else (
 shift
)
goto parseParams %*
goto :eof

:run
set "PATH_PRUNSRV=%~dp0"
set SERVICE_JAVA=MyApp
set INSTALL_IF_NOT_EXIST=true
set service_display_name=MyApp
set service_description="MyApp Service"
set start_params=""
call :parseParams %*

set PRUNSRV=%PATH_PRUNSRV%%SERVICE_JAVA%.exe
REM if INSTALL_IF_NOT_EXIST is true, check whether the service exists, if false, install the service if needed.
SC QUERY %SERVICE_JAVA% > NUL 2>&1
REM if ERRORLEVEL 1060 means it doesn't exists
IF ERRORLEVEL 1060 (
  echo %SERVICE_JAVA% doesn't exists
 if "%INSTALL_IF_NOT_EXIST%" == "true" (
   echo install_if_not_exist is true, install the service %SERVICE_JAVA%.
  cmd /c installEmbededJettyService.bat -service_name %SERVICE_JAVA% -service_display_name "%service_display_name%" -service_description "MyApp Service" -start_params "%start_params%"
  echo service %SERVICE_JAVA% is installed.
 ) else (
  echo install_if_not_exist is false, exit with error.
  exit -1
 )
)

%PRUNSRV% //ES//%SERVICE_JAVA%
if %ERRORLEVEL% == 0 (
 echo %SERVICE_JAVA% is started.
) else (
 echo Failed starting '%SERVICE_JAVA%' service
)

:end
endlocal 
@echo on

StopService.bat

@echo off
setlocal

goto run
:parseParams
set key=%~1
set value=%~2
echo %key% %value%
if "%key%" == "" goto :eof
if "%key%" == "-service_name" (  
    shift
 shift
 if "%value%" == "" (
        echo Empty value for -service_name
        goto end
    )
 SET SERVICE_JAVA=%value%
) else (
 shift
)
goto parseParams %*
goto :eof

:run

set PATH_PRUNSRV=%~dp0
set SERVICE_JAVA=MyApp
call :parseParams %*

set PRUNSRV=%PATH_PRUNSRV%%SERVICE_JAVA%.exe
"%PRUNSRV%"  //SS//%SERVICE_JAVA%
rem force kill the service as a safety bet
Taskkill /F /FI "SERVICES eq %SERVICE_JAVA%"
rem recheck service exists
Tasklist /FI "SERVICES eq %SERVICE_JAVA%" 2>NUL | find /I /N "%SERVICE_JAVA%">NUL 2>&1
if %ERRORLEVEL% == 0 (
 echo Failed stopping '%SERVICE_JAVA%' service
) else (
  echo %SERVICE_JAVA% is stopped.
)
:end
endlocal 
@echo on

Resource
http://commons.apache.org/proper/commons-daemon/procrun.html
Commons Daemon Procrun as a Java Service Wrapper for ActiveMQ

Remove SyntaxHighlighter Vertical Scrollbar

In previous post, I put java code in Syntax Highlighter pre section, noticed that there is an unnecessary vertical scollbar in Chrome(Works fine in Firefox) which only scrolls for one or two pixels. 

To fix this issue, edit Blogspot template, add the following code after stylesheet shCore.css:

.syntaxhighlighter {
 overflow-y: hidden !important; 
}

If you add previous code before loading shCore.css, the overflow definition .syntaxhighlighter {overflow: auto !important;} in shCore.css will overwrite previous definition.

If you want to hide horizontal scrollbar, add following code snippet after stylesheet shCore.css:

.syntaxhighlighter {
 overflow-y: hidden !important; 
}

Solr: Use STAX Parser to Read XML Response to Reduce Memory Usage

My Solr client talks with a proxy application which communicates with remote Solr Server to get data. 
In previous post, Solr: Use JSON(GSon) Streaming to Reduce Memory Usage, I described the problem we faced, how to use JSON(GSon) Streaming, and also some other approaches to reduce memory usage. 

In post Solr: Use SAX Parser to Read XML Response to Reduce Memory Usage
I also described how to use SAX to parse response for better performance.

In this post, I will introduce how to use Stax Parser to parse XML response.
Implementation
The code to use Stax to read document one by one from http stream:
-- Use Stax parser and Java Executors Future to wait all thread finished: all docs imported.

private static ImportedResult handleXMLResponseViaStax(
      SolrQueryRequest request, InputStream in, int fetchSize)
      throws XMLStreamException {
    XMLInputFactory factory = XMLInputFactory.newInstance();
    XMLStreamReader reader = null;
    
    ImportedResult importedResult = new ImportedResult();
    List<Future<Void>> futures = new ArrayList<Future<Void>>();
    
    try {
      reader = factory.createXMLStreamReader(in);
      int fetchedSize = 0;
      int numFound = -1, start = -1;
      
      while (reader.hasNext()) {
        int event = reader.next();
        switch (event) {
          case XMLStreamConstants.START_ELEMENT: {
            if ("result".equals(reader.getLocalName())) {
              numFound = Integer.valueOf(reader.getAttributeValue("",
                  "numFound"));
            } else if ("start".equals(reader.getLocalName())) {
              start = Integer.valueOf(reader.getAttributeValue("", "start"));
            } else if ("doc".equals(reader.getLocalName())) {
              ++fetchedSize;
              futures.add(readOneDoc(request, reader));
            }
            break;
          }
          default:
            break;
        }
        
      }
      importedResult.setFetched(fetchedSize);
      importedResult.setHasMore((fetchedSize + start) < numFound);
      importedResult.setImportedData((fetchedSize != 0));
      return importedResult;
    } finally {
      if (reader != null) {
        reader.close();
      }
    }
  }
  
  private static Future<Void> readOneDoc(SolrQueryRequest request,
      XMLStreamReader reader) throws XMLStreamException {
    String contentid = null, bindoc = null;
    OUTER: while (reader.hasNext()) {
      int event = reader.next();
      INNER: switch (event) {
        case XMLStreamConstants.START_ELEMENT: {
          if ("str".equals(reader.getLocalName())) {
            
            String fieldName = reader.getAttributeValue(0);
            if ("contentid".equals(fieldName)) {
              contentid = reader.getElementText();
            } else if ("bindoc".equals(fieldName)) {
              bindoc = reader.getElementText();
            }
          }
          break INNER;
        }
        case XMLStreamReader.END_ELEMENT: {
          if ("doc".equals(reader.getLocalName())) {
            break OUTER;
          }
        }
        default:
          break;
      }
    }
    return CVSyncDataImporter.getInstance().importData(request, contentid,
        bindoc);
  }
    

Resources
Parsing XML using DOM, SAX and StAX Parser in Java
Java SAX vs. StAX

Solr: Use SAX Parser to Read XML Response to Reduce Memory Usage

My Solr client talks with a proxy application which talks with remote Solr Server to get data. 
In previous post, Solr: Use JSON(GSon) Streaming to Reduce Memory Usage

I described the problem we faced, how to use JSON(GSon) Streaming, and also some other approaches to reduce memory usage. In this post I will use XML SAX Parser to iterative xml response stream. In next post I will introduce how to use Stax Parser to parse XML response.
Implementation
The code to use SAX to read document one by one from http stream:
-- Use SAX parser and Java Executors Future to wait all thread finished: all docs imported.

private static ImportedResult handleXMLResponseViaSax(
      SolrQueryRequest request, InputStream in, int fetchSize)
      throws IOException, ParserConfigurationException, SAXException {
 
    ImportedResult importedResult = new ImportedResult();
    SAXParserFactory parserFactor = SAXParserFactory.newInstance();
    SAXParser parser = parserFactor.newSAXParser();
    SolrResponseHandler handler = new SolrResponseHandler(request);
    parser.parse(in, handler);
    
    importedResult.setFetched(handler.fetchedSize);
    importedResult
        .setHasMore((handler.fetchedSize + handler.start) < handler.numFound);
    importedResult.setImportedData((handler.fetchedSize != 0));    
    return importedResult;
  }
  
  private static class SolrResponseHandler extends DefaultHandler {
    protected int fetchedSize = 0;
    protected int numFound = -1, start = -1;
    protected String contentid, bindoc = null;
    protected List<Future<Void>> futures = new ArrayList<Future<Void>>();
    
    String curName, curValue;
    private SolrQueryRequest request;
    
    public SolrResponseHandler(SolrQueryRequest request) {
      this.request = request;
    }
    
    @Override
    public void startElement(String uri, String localName, String qName,
        Attributes attributes) throws SAXException {
      
      switch (qName) {
        case "result": {
          numFound = Integer.valueOf(attributes.getValue("numFound"));
          start = Integer.valueOf(attributes.getValue("start"));
          break;
        }
        case "str": {
          String name = attributes.getValue("name");
          if ("contentid".equals(name)) {
            curName = "contentid";
          } else if ("bindoc".equals(name)) {
            curName = "bindoc";
          }
          break;
        }
        default:
          break;
      }
    }
    
    @Override
    public void endElement(String uri, String localName, String qName)
        throws SAXException {
      switch (qName) {
        case "str": {
          if ("contentid".equals(curName)) {
            contentid = curValue;
          } else if ("bindoc".equals(curName)) {
            bindoc = curValue;
          }
          break;
        }
        case "doc": {
          ++fetchedSize;
          futures.add(CVSyncDataImporter.getInstance().importData(request,
              contentid, bindoc));
          break;
        }
        default:
          break;
      }
    }
    @Override
    public void characters(char[] ch, int start, int length)
        throws SAXException {
      curValue = String.copyValueOf(ch, start, length).trim();
    }
  }
    

Resources
Parsing XML using DOM, SAX and StAX Parser in Java
Java SAX vs. StAX

Conditional Display of AdSense/Alternative Ads to Maximize Revenue

[Updated]Please read Display AdSense Ads Efficiently and Conditionally Show Alternative Ads, which introduce a simpler way to implement same function.

AdSense allows 3 standard ad units and 3 link units in one page. 
This is enough for us in most cases. But sometimes we want more flexibility: to be able to conditional display of AdSense Ads.

For example, in lifelongprogrammer.blogspot.com, we display 3 standard ad units in post: one at header, one at left-sidebar, and one inside the post body. 

But when view the homepage or archive page, label page, it's kind of messed up: it does display 3 standard ads, but one is at header as expected, the other two is inside the first two posts. The ads slot at left side bar is left blank. This makes the site looks ugly.

This is because AdSense fills up to 3 ads in the order of ads in html page. In blogspot template, the html order is: the header, then center post, left sidebar, then right sidebar.

I want to make sure the ad in left side bar always display and always display 3 AdSense Ads. So I make the following changes: 
At home, archive and label page, display alternative ads(I choose chitika) in post body, display 2 AdSense ads in left sidebar.
At post/item page, display AdSense ads in post, display one AdSense ads, one chitika ads in left sidebar..

Next, how to do it?
Conditional Display Ads in Post Body
Put the following code to the location where we want to display ads when write a post.
This checks whether the pathname of url matches /[0-9]{4}/[0-9]{2}/.* If so, means this is a post/item page, display AdSense ads, otherwise display chitika ads.

<script>
//<![CDATA[ 
var isItemPage = (document.location.pathname.match(new RegExp("/[0-9]{4}/[0-9]{2}/.*")) !=null);
if(isItemPage){
  document.write("<ins class='adsbygoogle' ... data-ad-client='' data-ad-slot=''></ins>
  <script>(adsbygoogle = window.adsbygoogle || []).push({});<\/script>")
} else {
    chitika-ads-code
}
//]]>  
</script>

Conditional Display Ads for the Second Ads in Left Sidebar
Add a HTML/JavaScript gadget at Left Sidebar in Layout page, put the following code in content text box.
It is not a post/item page, display AdSense ads, otherwise display chitika ad.

<script>
var isItemPage = (document.location.pathname.match(new RegExp("/[0-9]{4}/[0-9]{2}/.*")) !=null);
if(!isItemPage){
  document.write("Google-AdSense-Ads-Code");
}  else {
  chitika-ads-code
}
</script>

Now it looks much better, no blank ad, and always display 3 AdSense ads. Display alternative(chitika) ads for ads slot in post when at home, archive and label page.

For better performance, I will put the code which detect page type  before the start body tag, so it will be just executed one time.

<script>var isItemPage = (document.location.pathname.match(new RegExp("/[0-9]{4}/[0-9]{2}/.*")) !=null)</script>

Utilize Horizontal Link Units
dSense also allow us to put 3 links units, so I choose one above the header ad - this is easy, just add one HTML/JavaScript gadget, one below the title, one below post body, and I want these two only display in item/post page.

How to do it? Basically, it is to edit Blogspot template, put the following code below

and .

<script>
if(isItemPage){
  document.write("<ins class='adsbygoogle' ... data-ad-client='' data-ad-slot=''></ins>
  <script>(adsbygoogle = window.adsbygoogle || []).push({});<\/script>")
}</script>

Read more from
How To Insert Adsense Ads In Blogger Post

Use Google App Engine to Hold AdSense Backup Ads

Google AdSense is great, but at some time, it may not show ads in the ads slot. Instead it will leave blank space. This will make the site looks ugly and unprofessional. 
This is rare but it may happen. Anyway we don't want to miss out on any potential income.

This is why we want to set backup ads. There are some AdSense Alternatives: please read Top 10 Highest Paying Google Adsense Alternatives in 2013

But when we set up ad, most of ads network just gave us some javascript code which we can paste to our website. 
But as described at Backup ads, AdSense backup ads should be one url.

To fix this, we can create a html to hold the JavaScript: just put an empty page with the JavaScript:
AdSense Backup Ads: No Ad Space Left Behind

Next, how/where to serve these ads pages?
Luckily Google App Engine comes to rescue. Google App Engine allow us to create free application, and gave us generous free quotas: 1 GB daily request limit. This is engough for my small blog.
This saves me the cost of renting a web server.

I created a GAE application: http://placeholdads.appspot.com/, and add some ads pages:
Chitika 160X600 Ads
Chitika 728X90 Ads

Now, I can put these urls as Adsense backup ads in Adsense website.

Solr: Use JSON(GSon) Streaming to Reduce Memory Usage

My Solr application runs at user's laptop, the max memory is set to 512mb. It pull JSON data from remote proxy which talks with remote Solr Server: 100m hundred at a time, commit after 20 times.

Our code gets the whole json and put it into memory, and use UpdateRequestProcessor.processAdd(AddUpdateCommand) to add it into local solr.

Recently it throws OutOfMemoryError, after use Eclipse Memory Analyzer (MAT) to analyze the heapdump file.

I found out that it is because the data returned from remote proxy is too large:, one data is 50-60kb on average. But some data is huge, 100 data would be 60 mb: this is rare case, but when this happens it will cause the application throws OutOfMemoryError and stops to work.

To fix this and reduce memory usage at client side, I take several measures:

1. Reboot the application when OutOfMemoryError happens.
2. Run a thread to monitor free memory, at a certain threshold(40%), run gc. If less than 30%, decrease fetch size(100 to 50, to 25) and decrease commit interval( 20 times, 10 times). If less than 50 mb memory, restart the application.
3. Enable Auto SoftCommit and AutoCommit, reduce Solr cache size.
3. Use Streaming JSON. - This is the topic of this article.
Read document one by one from http input stream, put it to queue, instead read the whole big document in to memory. Another thread is responsible to write the document to local solr.

Same approach apples if we use XML: we can use StAX or SAX to read document one by one.

I use GSON, about how to use Gson Streaming to read and write JSON, please read Gson Streaming 

The code to read document one by one from http stream:
-- Use GSon Stream API and Java Executors Future to wait all thread finished: all docs imported.

/**
 * Use Gson Streaming API to read docuemts one by one to reduce memory usage 
 */
private static ImportedResult handleResponse(SolrQueryRequest request,
    InputStream in, int fetchSize) throws UnsupportedEncodingException,
    IOException {
  ImportedResult importedResult = new ImportedResult();
  JsonReader reader = null;
  List<Future<Void>> futures = new ArrayList<Future<Void>>();
  
  try {
    reader = new JsonReader(new InputStreamReader(in, "UTF-8"));
    reader.beginObject();
    String str = reader.nextName();
    reader.beginObject();
    int fetchedSize = 0;
    int numFound = -1, start = -1;
    while (reader.hasNext()) {
      str = reader.nextName();
      if ("numFound".equals(str)) {
        numFound = Integer.valueOf(reader.nextString());
      } else if ("start".equals(str)) {
        start = Integer.valueOf(reader.nextString());
      } else if ("docs".equals(str)) {
        reader.beginArray();
        // read documents
        while (reader.hasNext()) {
          fetchedSize++;
          readOneDoc(request, reader);
        }
        
        reader.endArray();
      }
    }
    
    reader.endObject();
    waitComplete(futures);
    importedResult.setFetched(fetchedSize);
    importedResult.setHasMore((fetchedSize + start) < numFound);
    importedResult.setImportedData((fetchedSize != 0));
    return importedResult;
  } finally {
    if (reader != null) {
      reader.close();
    }
  }
}

private static java.util.concurrent.Future<Void> readOneDoc(
    SolrQueryRequest request, JsonReader reader) throws IOException {
  String str;
  reader.beginObject();
  String id = null, binaryDoc = null;
  while (reader.hasNext()) {
    str = reader.nextName();
    
    if ("id".equals(str)) {
      id = reader.nextString();
    } else if ("binaryDoc".equals(str)) {
      binaryDoc = reader.nextString();
    }
  }
  reader.endObject();
  return CVSyncDataImporter.getInstance().importData(request, id,
      binaryDoc);
}

The code to write document to local solr:

public Future<Void> importData(SolrQueryRequest request, String id,
    String binaryDoc) {
  if (id == null) {
    throw new IllegalArgumentException("id can't be null.");
  }
  if (binaryDoc == null) {
    throw new IllegalArgumentException("binaryDoc can't be null.");
  }
  SolrDataImporter task = new SolrDataImporter(request, id, binaryDoc);
  return executor.submit(task);
}
private static SolrInputDocument convertToSolrDoc(String id,
    String binaryDoc) throws IOException {
  byte[] bindata = Base64.base64ToByteArray(binaryDoc);
  SolrInputDocument resultDoc = (SolrInputDocument) readZippedFile(bindata);
  resultDoc.setField("id", id);
  return resultDoc;
}

private class SolrDataImporter implements Callable<Void> {
  private SolrQueryRequest request;
  private String id, binaryDoc;
  @Override
  public Void call(){
    try {
      UpdateRequestProcessorChain updateChian = request.getCore()
          .getUpdateProcessingChain("mychain");
      SolrInputDocument toSolrServerSolrDoc = convertToSolrDoc(id,
          binaryDoc);
      binaryDoc = null;
      AddUpdateCommand command = new AddUpdateCommand(request);
      command.solrDoc = toSolrServerSolrDoc;
      SolrQueryResponse response = new SolrQueryResponse();
      UpdateRequestProcessor processor = updateChian.createProcessor(request,
          response);
      processor.processAdd(command);
    } catch (Exception e) {
      logger.error("Exception happened when importdata, id: "
          + id, e);
    }
    return null;
  }
}

Improve Solr CSVParser to Log Invalid Characters

Recently, one colleague in another team told me that when import one csv file to Solr server, it failed with following exception:
SEVERE: org.apache.solr.common.SolrException: CSVLoader: input=file:/sample.txt, line=1,can't read line: 12450
        values={NO LINES AVAILABLE}
        at rg.apache.solr.handler.loader.CSVLoaderBase.input_err(CSVLoaderBase.java:320)
        at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:359)
        at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
  ...
Caused by: java.io.IOException: (line 12450) invalid char between encapsulated token end delimiter.
        at org.apache.solr.internal.csv.CSVParser.encapsulatedTokenLexer(CSVParser.java:481)
        at org.apache.solr.internal.csv.CSVParser.nextToken(CSVParser.java:359)
        at org.apache.solr.internal.csv.CSVParser.getLine(CSVParser.java:231)
        at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:356)
At that time, I enabled remote debug, and used Eclipse Display view to find the invalid character, and 2 more characters, then searched in the CSV file to find the reason: it is because there is " in the value of the from field: |  | "an,xxxx"
For more information, please read: 
Use Eclipse Display View While Debugging to Fix Real Problem
Import CSV that Contains Double-Quotes into Solr

This makes me change Solr's code so if similar problem happens next time, we can find the problem directly from the log, not have to do remote debug again.
The code looks like below:

private Token encapsulatedTokenLexer(Token tkn, int c) throws IOException {
 for (;;) {
   c = in.read();
   else if (c == strategy.getEncapsulator()) {
   if (in.lookAhead() == strategy.getEncapsulator()) {
   } else {
  for (;;) {
   c = in.read();
   else if (!isWhitespace(c)) {
    // error invalid char between token and next delimiter
    throw new IOException(
     "(line "
      + getLineNumber()
      + ") invalid char between encapsulated token end delimiter, invalid char: "
      + String.valueOf((char)c) + ", context " + getContextChars(c));
    }
  }
  }
   } 
 }
}
// new method: read more 3 characters
private String getContextChars(int c) {
    int count =0;
    String moreChars=String.valueOf((char)c);
    while (count < 3) {
      try {
        int tmpc = in.read();
        moreChars +=String.valueOf((char)tmpc);
        ++count;
      } catch (Exception e) {
        break;
      }
    }
    return moreChars;
}

Chrome Download Dialog Hidden Behind the Browser Window and the Workaround

The problem
At workplace, my desktop machine is connected to 2 monitors, I mainly use the main display, use the second monitor as remote connection to other machines. - I think this's very common at workplace.

I am bothered by this problem of Chrome for a long time: 
1. When I try to download a file(Eclipse in this case) from Chrome, it opens the  Download Dialog. See below:

2. At that time, I may go to some other application: I may check explorer to check folder, or some message application may pop up, or I may just go to other application: notepad++ or anything.

The I found out I can never find the Download Dialog again: I go to that Chrome window, there is no the Download Dialog; ctrl-Alt can't find it either. See below:

The Workaround
Today, I found a workaround: Click "Close Window" on the Chrome window, because there is an open Download Dialog, Chrome will bring the dialog front. Now we can save the file or cancel download. After that Chrome will NOT close the Chrome.
This is a problem Chrome team should fix it, but anyway I have a workaround, so not bother me that much any more.
Resources
https://plus.google.com/101157854606139706613/posts/5Y32TuS4LzH
Issue 110807:dialog boxes are hidden behind the browser window

Learning Java Integer Code and Puzzles

1. Question - What the program would output? 

int i = 127;

boolean b = (Integer.valueOf(i) == Integer.valueOf(i));

System.err.println(b);

i = 128;

b = (Integer.valueOf(i) == Integer.valueOf(i));

System.err.println(b);

We can immediately know the answer after we look at the source code.

public static Integer valueOf(int i) {

    if(i >= -128 && i <= IntegerCache.high)

        return IntegerCache.cache[i + 128];

    else

        return new Integer(i);

}

We can know it doesn't cache all Integer values, as this may consume too much memory, so it just caches the numbers between [-128, 127] in the static IntegerCache class, for other int numbers, each time it would return a new Integer. Class Long also only caches the numbers between [-128, 127].

The output of the previous program would be obvious now: true and false.

2. Autobox and Auto-unbox

2. 1 How is it implemented in JDK?

Simply put, when we call "Integer wrapper = 2;", the java compile would translate it to "Integer wrapper = Integer.valueOf(wrapper);".

When we call "int i = wrapper;", the java compile would translate it to "int i = wrapper.intValue();".

You can verify this by using javap to look. at the compiled java class: javap -c IntegerTester.

Long.valueOf(0L).equals(0)?

2.2 What would be the output of the following program?

Long l = 0L;

Integer i = 0;

System.out.println(l.equals(i));

System.out.println(i.equals(l));

So let's look at the JDK source code again:

public final class Integer extends Number

  implements Comparable {

    public boolean equals(Object obj) {

    if (obj instanceof Integer) {

        return value == ((Integer)obj).intValue();

    }

    return false;

    }

}

public final class Long extends Number

  implements Comparable {

    public boolean equals(Object obj) {

      if (obj instanceof Long) {

          return value == ((Long)obj).longValue();

      }

      return false;

    }

}

From the code, we can see if the type of parameter is not same, these method would return false.

So the output of previous program would be false, false.

Autobox and auto-unbox are good features, as it will convert the primitive type or wrapper type to the needed one, and we can write less code, but we should use it carefully, as usually, it may create object under the hood, if we are unaware of this, it may cause big performance penalty.

public void test(Integer i) {

    while (i < 0) {

        --i;

    }

}

In the previous program, in each step, JVM actually does this: it calls i.intValue(), subtracts it by one, and then create a new Integer value.

For JVM, it would be same as:

So in each step, we would unnecessary create one Integer value and call intValue methods twice.

Try to use javap to look at the compiled class file.

If we know this, we can change our code like the below, it would be faster.

public void test(Integer i) {

      int j = i;

      while (j < 0) {

           --j;

      }

      i = j;

}

In our code, it's to better use primitive type as often as possible. 

In Integer class there are many other interesting and useful methods, some methods are listed below, are they cool and a little confusing? Try to figure it out.

public final class Integer extends Number

  implements Comparable {

    public static int reverseBytes(int i) {

        return ((i >>> 24)           ) |

               ((i >>   8) &   0xFF00) |

               ((i <<   8) & 0xFF0000) |

               ((i << 24));

    }

    public static int reverse(int i) {

        // HD, Figure 7-1

    i = (i & 0x55555555) << 1 | (i >>> 1) & 0x55555555;

    i = (i & 0x33333333) << 2 | (i >>> 2) & 0x33333333;

    i = (i & 0x0f0f0f0f) << 4 | (i >>> 4) & 0x0f0f0f0f;

    i = (i << 24) | ((i & 0xff00) << 8) |

        ((i >>> 8) & 0xff00) | (i >>> 24);

    return i;

    }

    public static int bitCount(int i) {

        // HD, Figure 5-2

    i = i - ((i >>> 1) & 0x55555555);

    i = (i & 0x33333333) + ((i >>> 2) & 0x33333333);

    i = (i + (i >>> 4)) & 0x0f0f0f0f;

    i = i + (i >>> 8);

    i = i + (i >>> 16);

    return i & 0x3f;

    }

}




Reference:

The Advantages and Traps of Autoboxing

a