Solr ThreadLocal DateFormat - Open Source Code Snippet

The good part of open source project is that you can read the code and learn from it. I write down the good code I learned recently here, so I refer this later.
ThreadLocal DateFormat
private static class org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat
   extends ThreadLocal<DateFormat> {
    DateFormat proto;
    public ThreadLocalDateFormat() {
      super();
      SimpleDateFormat tmp = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT);
      tmp.setTimeZone(UTC);
      proto = tmp;
    }
    protected DateFormat initialValue() {
      return (DateFormat) proto.clone();
    }
  }
  public static DateFormat getThreadLocalDateFormat() {
    return fmtThreadLocal.get();
  }
  private static ThreadLocalDateFormat fmtThreadLocal = new ThreadLocalDateFormat();
Support multiple date formats
org.apache.solr.common.util.DateUtil


static {
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd'T'HH:mm:ss'Z'");
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd'T'HH:mm:ss");
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd");
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd hh:mm:ss");
 DEFAULT_DATE_FORMATS.add("yyyy-MM-dd HH:mm:ss");
 DEFAULT_DATE_FORMATS.add("EEE MMM d hh:mm:ss z yyyy");
 DEFAULT_DATE_FORMATS.addAll(DEFAULT_HTTP_CLIENT_PATTERNS);
}
  
public static Date parseDate(String d, Collection<String> fmts) throws ParseException {
    if (d.endsWith("Z") && d.length() > 20) {
      return getThreadLocalDateFormat().parse(d);
    }
    return parseDate(d, fmts, null);
}
public static Date parseDate(String dateValue, Collection dateFormats, Date startDate) 
throws ParseException {
    Iterator formatIter = dateFormats.iterator();
    while (formatIter.hasNext()) {
      String format = (String) formatIter.next();
      if (dateParser == null) {
        dateParser = new SimpleDateFormat(format, Locale.ROOT);
        dateParser.setTimeZone(GMT);
        dateParser.set2DigitYearStart(startDate);
      } else {
        dateParser.applyPattern(format);
      }
      try {
        return dateParser.parse(dateValue);
      } catch (ParseException pe) {
        // ignore this exception, we will try the next format
      }
    }
}

Eclipse Tips Tricks


List collected eclipse tricks recently for future references.
Linking an external folder to Eclipse / Flex Builder Project - Great Feature

Use Display view to execute (any) code  in the context of the current stack frame while debug
http://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2Freference%2Fviews%2Fdisplay%2Fref-display_view.htm

Always run/debug the last launched class instead of the selected one in Eclipse
Go to Window > Preferences > Run/Debug > Launching.
Select the option "Always launch the previously launched application".

Open folder in windows explorer from eclipse
http://stackoverflow.com/questions/1161240/in-eclipse-reveal-current-file-in-filesystem
Create a new program by clicking 
Location: C:\Windows\explorer.exe
Arguments: /n,/e, ${container_loc}
Automatically run FindBugs
You may customize how FindBugs runs by opening the Properties dialog for a Java project, and choosing the "Findbugs" property page.
Enable or disable the "Run FindBugs Automatically" checkbox. When enabled, FindBugs will run every time you modify a Java class within the project.

Skip over certain classes when using Step Into in Eclipse’s debugger

Go to Window > Preferences > Java > Debug > Step Filtering.
Select the option Use Step Filters.
Click Select All to enable all the filters


Where is the workspace local history stored?
grep -R "word"  .metadata/.plugins/org.eclipse.core.resources/.history/

Hide Closed Projects in Eclipse Project Explorer View

What if Copy/Paste intermittently doesn't work
Disabling "Java->Editor->Typing->Update imports" seems to be an effective work around.

What is a bookmark and how do I use it?

Java Http Authentication

Scenario: Want to access protected resource in remote machine. There are several ways to do this.

1. If credential of the current logged-on user can be used to access the remote protected resource, then there is no need to add user/password info explicitly: Java URLConnection can automatically do this for me.

Apache HttpClient is a great tool to execute http requests, and add authentication, but it doesn't support to automatically authentication using current logged-on user credential. So sometimes, we have to use Java UrlConnection instead of Apache http client libaray.
Please refer to: http://httpcomponents.10934.n7.nabble.com/Authenticate-Proxy-using-currently-logged-on-domain-user-s-credentials-td11338.html
2. Use Authenticator.setDefault
Authenticator.setDefault(new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication (username, password.toCharArray());
    }
});
This sets default Authenticator which is called whenever authentication is required for any URLConnection.

This works for both basic form authentication. 
If you want to use a domain user/passwword to do login(NTLM widows integrated authentication), just use:
return new PasswordAuthentication(domain + "\\" + userName, password.toCharArray());

The problem in java is that Authenticator.setDefault() setups an authenticator for all HttpURLConnection, there is no such a method setAuthenticator on URLConnection.
3. Sending Basic authentication using URLConnection

http://blogs.deepal.org/2008/01/sending-basic-authentication-using-url.html
String authorizationString = “Basic “ + Base64.encode(username:password);
urlConnection.setRequestProperty ("Authorization", authorizationString)
Http Negotiate (SPNEGO) Example
SPNEGO is used to negotiate one of a number of possible real mechanisms. SPNEGO is used when a client application wants to authenticate to a remote server, but neither end is sure what authentication protocols the other supports. The pseudo-mechanism uses a protocol to determine what common GSSAPI mechanisms are available, selects one and then dispatches all further security operations to it. This can help organizations deploy new security mechanisms in a phased manner.

Security in Server Side
Http Debug
1 Use Fiddler to log traffic between client and sever.
http://blog.alner.net/archive/2008/10/06/fiddler-ndash-put-a-breakpoint-in-your-network-traffichellip.aspx
http://blog.alner.net/archive/2008/10/03/use-fiddler-to-view-traffic-when-running-locallyhellip.aspx
2 Change Java Class Log level
For this, we want to change the log level of 
-Djava.util.logging.config.file=logging.properties

In logging.properties
handlers=java.util.logging.ConsoleHandler
.level=ALL
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
sun.net.www.protocol.http.HttpURLConnection.level = ALL
java.net.URLConnection.level = ALL
Main Classes
sun.net.www.protocol.http.HttpURLConnection.getInputStream()
sun.net.www.protocol.http.AuthenticationHeader.parse()
java.net.Authenticator.requestPasswordAuthentication
sun.net.www.protocol.http.spnego.NegotiateCallbackHandler.handle(Callback[])

Other Resources
Authentication scheme
Basic, Digest, NTLM, Http Negotiate (SPNEGO)
Scheme Preference
GSS/SPNEGO -> Digest -> NTLM -> Basic

How Solr Join Works

QParserPlugin
Exaple query: http://localhost:8080/solr/select?q={!join from=classid to=classid}classname:"Grade 1"
QueryComponent.prepare call QParser.getParser which will 
QParser parser = QParser.getParser(rb.getQueryString(), defType, req);

QParser.getParser
default parser is lucene. It will parse local parameter to get the parser type, in this case it is join: {!join from=classid to=classid}. Put the query classname:"Grade 1" into localMap, key=QueryParsing.V.

if (qstr != null && qstr.startsWith(QueryParsing.LOCALPARAM_START)) {
{
 localParamsEnd = QueryParsing.parseLocalParams(qstr, 0, localMap, globalParams);
 //It first tries to get query string from local param
 String val = localMap.get(QueryParsing.V);
 if (val == null) {
  val = qstr.substring(localParamsEnd);
  localMap.put(QueryParsing.V, val);
 }
}
For the join query, this will get JoinQParserPlugin.
QParserPlugin qplug = req.getCore().getQueryPlugin(parserName);
QParser parser =  qplug.createParser(qstr, localParams, req.getParams(), req);

QueryComponent.prepare then calls the parser to get the query. 
 Query q = (QParser)parser.getQuery();, 
This will call the abstract parse method to pear the query string -the parse is only called once.
In this case, it is the anonymous parser class returned by JoinQParserPlugin.createParser.

In the anonymous parser class, 
if fromIndex parameter is not null, 
Then it creates subQuery from the query string:classname:"Grade 1". This will again determine/ create the parser for the query string via Parser.getParser, and get the query via LuceneQParser.parse.
QParser fromQueryParser = subQuery(v, null);
fromQuery = fromQueryParser.getQuery();

Then create one join query, and return it.
JoinQuery jq = new JoinQuery(fromField, toField, fromIndex, fromQuery);
jq.fromCoreOpenTime = fromCoreOpenTime;

Classes:
LuceneQParserPlugin
createParser creates LuceneQParser.
QParser
QParser is an abstract class, main methods include:
static getParser like a factory which creats concrete parser class based on local parameter. 
abstract parse method: create and return the Query object represented by qstr.
LuceneQParser extends QParser and implements parse method. In the parse method, LuceneQParser creates a QueryParser, then call QueryParser.parse to parse the qstr and return a Query.
SolrQueryParserBase
QueryParser extends SolrQueryParserBase
SolrQueryParser extends QueryParser
Query
JoinQuery, TermQuery,BooleanQuery.

QueryComponent.prepare then parse fq(filer query), save it to ResponseBuilder.
QParser fqp = QParser.getParser(fq, null, req);
filters.add(fqp.getQuery());
rb.setFilters( filters )

QueryComponent.process
this will create a QueryCommand: Query:JoinQuery.
SolrIndexSearcher.QueryCommand cmd = rb.getQueryCommand();
searcher.search(result,cmd);
filterList:[studentname:*], query:JoinQuery{!join from=classid to=classid}classname:Grade 1

Main task is done in SolrIndexSearcher.search
SolrIndexSearcher extends lucene.search.IndexSearcher.

SolrIndexSearcher.getDocListC(QueryResult, QueryCommand)
Query q = cmd.getQuery() --> JoinQuery
It will first try to get it from cache, the key is comprised of the Query object, sort, nflags. 
key = new QueryResultKey(q, cmd.getFilterList(), cmd.getSort(), flags);
superset = (SolrCache<QueryResultKey,DocList> queryResultCache)queryResultCache.get(key);
if (maxDocRequested < queryResultWindowSize) {
  supersetMaxDoc=queryResultWindowSize;
}
cmd.setSupersetMaxDoc(supersetMaxDoc);
At the first time, superset is null, then it will call getDocListNC(qr,cmd);

[getDocListNC]
ProcessedFilter pf = getProcessedFilter(cmd.getFilter(), cmd.getFilterList());
This will search the fq, and save the returned docset in pf.
cmd.getFilterList() is List<Query>: [studentname:*]
sets[end] = getPositiveDocSet(posQuery);
answer is DocSet, in this case BitDocSet which includs OpenBitSet.
answer = getDocSetNC(q,null);
if (filterCache != null) filterCache.put(q,answer);
pf.answer = answer;

getDocSetNC calls (lucene.search.IndexSearcher)super.search(query,null,collector);

[getDocListNC]
topCollector = TopScoreDocCollector.create(len, true);
Query is {!join from=classid to=classid}classname:Grade 1. luceneFilter is the the result which fq search returns
(lucene.search.IndexSearcher)super.search(query, luceneFilter, collector);

Here
(lucene.search.IndexSearcher)search(leafContexts, createNormalizedWeight(wrapFilter(query, filter)), results);

createNormalizedWeight will call Weight weight = query.createWeight(this); This will go to JoinQuery.createWeight.

[JoinQueryWeight]
Threadlocal:
SolrRequestInfo info = SolrRequestInfo.getRequestInfo();

[QueryComponent.process]
searcher.search(result,cmd);
rb.setResult( result );
doFieldSortValues(rb, searcher);
doPrefetch(rb);

Classes
lucene.search.IndexSearcher.search(List<AtomicReaderContext>, Weight, Collector)

Labels

Java (159) Lucene-Solr (112) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (39) Eclipse (33) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Problem Solving (10) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) Shell (7) ANT (6) Coding Skills (6) Database (6) Lesson Learned (6) Programmer Skills (6) Scala (6) Tips (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) System Design (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Life (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) Invest (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts