Solr ThreadLocal DateFormat - Open Source Code Snippet

The good part of open source project is that you can read the code and learn from it. I write down the good code I learned recently here, so I refer this later.
ThreadLocal DateFormat
private static class org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat
   extends ThreadLocal<DateFormat> {
    DateFormat proto;
    public ThreadLocalDateFormat() {
      SimpleDateFormat tmp = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT);
      proto = tmp;
    protected DateFormat initialValue() {
      return (DateFormat) proto.clone();
  public static DateFormat getThreadLocalDateFormat() {
    return fmtThreadLocal.get();
  private static ThreadLocalDateFormat fmtThreadLocal = new ThreadLocalDateFormat();
Support multiple date formats

static {
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd hh:mm:ss");
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd HH:mm:ss");
 DEFAULT_DATE_FORMATS.add("EEE MMM d hh:mm:ss z yyyy");
public static Date parseDate(String d, Collection<String> fmts) throws ParseException {
    if (d.endsWith("Z") && d.length() > 20) {
      return getThreadLocalDateFormat().parse(d);
    return parseDate(d, fmts, null);
public static Date parseDate(String dateValue, Collection dateFormats, Date startDate) 
throws ParseException {
    Iterator formatIter = dateFormats.iterator();
    while (formatIter.hasNext()) {
      String format = (String);
      if (dateParser == null) {
        dateParser = new SimpleDateFormat(format, Locale.ROOT);
      } else {
      try {
        return dateParser.parse(dateValue);
      } catch (ParseException pe) {
        // ignore this exception, we will try the next format

Eclipse Tips Tricks

List collected eclipse tricks recently for future references.
Linking an external folder to Eclipse / Flex Builder Project - Great Feature

Use Display view to execute (any) code  in the context of the current stack frame while debug

Always run/debug the last launched class instead of the selected one in Eclipse
Go to Window > Preferences > Run/Debug > Launching.
Select the option "Always launch the previously launched application".

Open folder in windows explorer from eclipse
Create a new program by clicking 
Location: C:\Windows\explorer.exe
Arguments: /n,/e, ${container_loc}
Automatically run FindBugs
You may customize how FindBugs runs by opening the Properties dialog for a Java project, and choosing the "Findbugs" property page.
Enable or disable the "Run FindBugs Automatically" checkbox. When enabled, FindBugs will run every time you modify a Java class within the project.

Skip over certain classes when using Step Into in Eclipse’s debugger

Go to Window > Preferences > Java > Debug > Step Filtering.
Select the option Use Step Filters.
Click Select All to enable all the filters

Where is the workspace local history stored?
grep -R "word"  .metadata/.plugins/org.eclipse.core.resources/.history/

Hide Closed Projects in Eclipse Project Explorer View

What if Copy/Paste intermittently doesn't work
Disabling "Java->Editor->Typing->Update imports" seems to be an effective work around.

What is a bookmark and how do I use it?

Java Http Authentication

Scenario: Want to access protected resource in remote machine. There are several ways to do this.

1. If credential of the current logged-on user can be used to access the remote protected resource, then there is no need to add user/password info explicitly: Java URLConnection can automatically do this for me.

Apache HttpClient is a great tool to execute http requests, and add authentication, but it doesn't support to automatically authentication using current logged-on user credential. So sometimes, we have to use Java UrlConnection instead of Apache http client libaray.
Please refer to:
2. Use Authenticator.setDefault
Authenticator.setDefault(new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication (username, password.toCharArray());
This sets default Authenticator which is called whenever authentication is required for any URLConnection.

This works for both basic form authentication. 
If you want to use a domain user/passwword to do login(NTLM widows integrated authentication), just use:
return new PasswordAuthentication(domain + "\\" + userName, password.toCharArray());

The problem in java is that Authenticator.setDefault() setups an authenticator for all HttpURLConnection, there is no such a method setAuthenticator on URLConnection.
3. Sending Basic authentication using URLConnection
String authorizationString = “Basic “ + Base64.encode(username:password);
urlConnection.setRequestProperty ("Authorization", authorizationString)
Http Negotiate (SPNEGO) Example
SPNEGO is used to negotiate one of a number of possible real mechanisms. SPNEGO is used when a client application wants to authenticate to a remote server, but neither end is sure what authentication protocols the other supports. The pseudo-mechanism uses a protocol to determine what common GSSAPI mechanisms are available, selects one and then dispatches all further security operations to it. This can help organizations deploy new security mechanisms in a phased manner.

Security in Server Side
Http Debug
1 Use Fiddler to log traffic between client and sever.
2 Change Java Class Log level
For this, we want to change the log level of

java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter = ALL = ALL
Main Classes[])

Other Resources
Authentication scheme
Basic, Digest, NTLM, Http Negotiate (SPNEGO)
Scheme Preference
GSS/SPNEGO -> Digest -> NTLM -> Basic

How Solr Join Works

Exaple query: http://localhost:8080/solr/select?q={!join from=classid to=classid}classname:"Grade 1"
QueryComponent.prepare call QParser.getParser which will 
QParser parser = QParser.getParser(rb.getQueryString(), defType, req);

default parser is lucene. It will parse local parameter to get the parser type, in this case it is join: {!join from=classid to=classid}. Put the query classname:"Grade 1" into localMap, key=QueryParsing.V.

if (qstr != null && qstr.startsWith(QueryParsing.LOCALPARAM_START)) {
 localParamsEnd = QueryParsing.parseLocalParams(qstr, 0, localMap, globalParams);
 //It first tries to get query string from local param
 String val = localMap.get(QueryParsing.V);
 if (val == null) {
  val = qstr.substring(localParamsEnd);
  localMap.put(QueryParsing.V, val);
For the join query, this will get JoinQParserPlugin.
QParserPlugin qplug = req.getCore().getQueryPlugin(parserName);
QParser parser =  qplug.createParser(qstr, localParams, req.getParams(), req);

QueryComponent.prepare then calls the parser to get the query. 
 Query q = (QParser)parser.getQuery();, 
This will call the abstract parse method to pear the query string -the parse is only called once.
In this case, it is the anonymous parser class returned by JoinQParserPlugin.createParser.

In the anonymous parser class, 
if fromIndex parameter is not null, 
Then it creates subQuery from the query string:classname:"Grade 1". This will again determine/ create the parser for the query string via Parser.getParser, and get the query via LuceneQParser.parse.
QParser fromQueryParser = subQuery(v, null);
fromQuery = fromQueryParser.getQuery();

Then create one join query, and return it.
JoinQuery jq = new JoinQuery(fromField, toField, fromIndex, fromQuery);
jq.fromCoreOpenTime = fromCoreOpenTime;

createParser creates LuceneQParser.
QParser is an abstract class, main methods include:
static getParser like a factory which creats concrete parser class based on local parameter. 
abstract parse method: create and return the Query object represented by qstr.
LuceneQParser extends QParser and implements parse method. In the parse method, LuceneQParser creates a QueryParser, then call QueryParser.parse to parse the qstr and return a Query.
QueryParser extends SolrQueryParserBase
SolrQueryParser extends QueryParser
JoinQuery, TermQuery,BooleanQuery.

QueryComponent.prepare then parse fq(filer query), save it to ResponseBuilder.
QParser fqp = QParser.getParser(fq, null, req);
rb.setFilters( filters )

this will create a QueryCommand: Query:JoinQuery.
SolrIndexSearcher.QueryCommand cmd = rb.getQueryCommand();,cmd);
filterList:[studentname:*], query:JoinQuery{!join from=classid to=classid}classname:Grade 1

Main task is done in
SolrIndexSearcher extends

SolrIndexSearcher.getDocListC(QueryResult, QueryCommand)
Query q = cmd.getQuery() --> JoinQuery
It will first try to get it from cache, the key is comprised of the Query object, sort, nflags. 
key = new QueryResultKey(q, cmd.getFilterList(), cmd.getSort(), flags);
superset = (SolrCache<QueryResultKey,DocList> queryResultCache)queryResultCache.get(key);
if (maxDocRequested < queryResultWindowSize) {
At the first time, superset is null, then it will call getDocListNC(qr,cmd);

ProcessedFilter pf = getProcessedFilter(cmd.getFilter(), cmd.getFilterList());
This will search the fq, and save the returned docset in pf.
cmd.getFilterList() is List<Query>: [studentname:*]
sets[end] = getPositiveDocSet(posQuery);
answer is DocSet, in this case BitDocSet which includs OpenBitSet.
answer = getDocSetNC(q,null);
if (filterCache != null) filterCache.put(q,answer);
pf.answer = answer;

getDocSetNC calls (,null,collector);

topCollector = TopScoreDocCollector.create(len, true);
Query is {!join from=classid to=classid}classname:Grade 1. luceneFilter is the the result which fq search returns
(, luceneFilter, collector);

(, createNormalizedWeight(wrapFilter(query, filter)), results);

createNormalizedWeight will call Weight weight = query.createWeight(this); This will go to JoinQuery.createWeight.

SolrRequestInfo info = SolrRequestInfo.getRequestInfo();

rb.setResult( result );
doFieldSortValues(rb, searcher);

Classes<AtomicReaderContext>, Weight, Collector)


Java (159) Lucene-Solr (110) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (36) Eclipse (34) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Troubleshooting (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Http Client (8) Maven (8) Problem Solving (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) Lesson Learned (5) Programmer Skills (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) System Design (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts