Boost Search Relevancy Using boilerpipe, Nutch and Solr

The Problem
We use Nutch to crawl web sites and save the content into Solr for search.

A website usally applies a template which defines header, footer, navigation menu. Take one faked storage related documentation site as an example. The word storage appears multiple times in header, footer and menus.
The website has some very simple contact-us or login page. Because there is only a few content in these pages, it's very likely if user search storage, these 2 pages would be ranked highly and listed in first page.

We want to avoid this. We would like to save main content in one field in Solr, and boost that field.
The Solution
Change Nutch to Send Raw Content to Solr
By default, Nutch sends Solr the html tag stripped content to solr, not the rawl html page content.
To send raw content to Solr, we have to create one extra nutch plugin:

public class ExtraIndexingFilter implements IndexingFilter {
  public static final String FL_RAWCONTENT = "rawcontent";
  private Configuration conf;
  private boolean indexRawContent;
  private static final Collection<WebPage.Field> FIELDS = new HashSet<WebPage.Field>();

  static {
    FIELDS.add(WebPage.Field.CONTENT);
  }
  public NutchDocument filter(NutchDocument doc, String url, WebPage page)
      throws IndexingException {
    if (indexRawContent) {
      ByteBuffer bb = page.getContent();
      if (bb != null) {
        doc.add(FL_RAWCONTENT, new String(bb.array()));
      }
    }
    return doc;
  }
  public void setConf(Configuration conf) {
    this.conf = conf;
    indexRawContent = conf.getBoolean("index-extra.rawcontent", false);
  }
}

Then change nutch-site.xml, add the plugin(index-extra) in plugin.includes. 
Set extra-index.rawcontent to true, and set http.content.limit to -1, so Nutch will crawl whole page.

<property>
  <name>extra-index.rawcontent</name>
  <value>true</value>
</property>
<property>
 <name>http.content.limit</name>
 <value>-1</value>
</property>

In solrindex-mapping.xml, add:

<field dest="rawcontent" source="rawcontent" />

Using boilerpipe to Remove Boilerplate Content in Solr
Next, we will define one Solr update processor which will use boilerpipe to remove the surplus "clutter" (boilerplate, templates).
BoilerpipeProcessor will use boilerpipe to remove boilerplate from originalfield and save stripped main content into strippedField. 

import de.l3s.boilerpipe.BoilerpipeProcessingException;
import de.l3s.boilerpipe.extractors.ArticleExtractor;

public class BoilerpipeProcessorFactory extends UpdateRequestProcessorFactory {
  private static final Logger logger = LoggerFactory
      .getLogger(BoilerpipeProcessorFactory.class);
  private boolean enabled = true;
  private String originfield, strippedField;
  private boolean removeOriginfield = true;

  public void init(NamedList args) {
    super.init(args);
    if (args != null) {
      SolrParams params = SolrParams.toSolrParams(args);
      enabled = params.getBool("enabled", true);
      if (!enabled) return;
      removeOriginfield = params.getBool("removeOriginfield", true);
      originfield = Preconditions.checkNotNull(params.get("originfield"),
          "Must set originfield.");
      
      strippedField = Preconditions.checkNotNull(params.get("strippedField"),
          "Must set strippedField.");
    }
  }
  public UpdateRequestProcessor getInstance(SolrQueryRequest req,
      SolrQueryResponse rsp, UpdateRequestProcessor next) {
    if (!enabled) return null;
    return new BoilerpipeProcessor(next);
  }
  
  private class BoilerpipeProcessor extends UpdateRequestProcessor {
    public BoilerpipeProcessor(UpdateRequestProcessor next) {
      super(next);
    }
    public void processAdd(AddUpdateCommand cmd) throws IOException {
      SolrInputDocument doc = cmd.solrDoc;
      Collection<Object> colls = doc.getFieldValues(originfield);
      if (colls != null) {
        for (Object obj : colls) {
          if (obj != null) {
            String str = obj.toString();
            try {
              String strippedText = ArticleExtractor.getInstance().getText(str);
              doc.addField(strippedField, strippedText);
              if (removeOriginfield) {
                doc.removeField(originfield);
              }
            } catch (BoilerpipeProcessingException e) {
              logger.error("Error happened when use boilerpipe to strip text.",
                  e);
            }
          }
        }
      }
      super.processAdd(cmd);
    }
  }
}

Add the processor into the default chain in the solrconfig.xml:

<updateRequestProcessorChain name="defaultChain" default="true`">
  <processor
   class="org.lifelongprogrammer.BoilerpipeProcessorFactory">
    <bool name="enabled">true</bool>
    <str name="originfield">rawcontent</str>
    <str name="strippedField">main_content</str>
    <bool name="removeOriginfield">true</bool>
  </processor>
  <processor class="solr.LogUpdateProcessorFactory" />
  <processor class="solr.RunUpdateProcessorFactory" /> 
</updateRequestProcessorChain>

Add main_content field in schema.xml:

<field name="main_content" type="text_rev" indexed="true" stored="true"  omitNorms="false" />

After all this, we can change our search handler to boost on main_content field:

<requestHandler name="/select" class="solr.SearchHandler" default="true">
  <lst name="defaults">
    <!-- Omitted -->
    <str name="qf">main_content^10 body_stored</str>
  </lst>
</requestHandler>

Resources
boilerpipe library
Filtering Source Code Using boilerpipe

Nutch2 Http Form Authentication-Part3: Integrate Http Form Post Authentication in Nutch2

The Problem
Http Form-based Authentication is a very common used authentication mechanism to protect web resources.
When crawl, Nutch supports NTLM, Basic or Digest authentication to authenticate itself to websites. But It doesn't support Http Post Form Authentication.

This series of articles talks about how to extend Nutch2 to support Http Post Form Authentication.
Main Steps
Use Apache Http Client to do http post form authentication.
Make http post form authentication work.
Integrate http form authentication in Nutch2.

After previous two steps, now we can integrate http form authentication in Nutch2.
Define Http Form Post Authentication Properties in httpclient-auth.xml
First, in nutch-site.xml change plugin.includes to use protocol-httpclient plugin: not the default protocol-http.

Nutch uses http.auth.file to locate the xml file that defines credentials info, default value is httpclient-auth.xml. We extend httpclient-auth.xml to include information about http form authentication properties. The httpclient-auth.xml for the asp.net web application in last post is like below:

<?xml version="1.0"?>
<auth-configuration>
  <credentials authMethod="formAuth" loginUrl="http://localhost:44444/Account/Login.aspx" loginFormId="ctl01" loginRedirect="true">
    <loginPostData>
      <field name="ctl00$MainContent$LoginUser$UserName" value="admin"/>
      <field name="ctl00$MainContent$LoginUser$Password" value="admin123"/>
    </loginPostData>
    <removedFormFields>
      <field name="ctl00$MainContent$LoginUser$RememberMe"/>
    </removedFormFields>
  </credentials>
</auth-configuration>

Read Http Form Post Authentication from Configuration XML File

In Nutch's http-client plugin, change org.apache.nutch.protocol.httpclient.Http.setCredentials() method to read authentication info into variable formConfigurer from configuration file.

Then change Http.resolveCredentials() method: if formConfigurer is not null, use HttpFormAuthentication to do form post login.

package org.apache.nutch.protocol.httpclient;
public class Http extends HttpBase {
 private void resolveCredentials(URL url) {
  if (formConfigurer != null) {
   HttpFormAuthentication formAuther = new HttpFormAuthentication(
     formConfigurer, client, this);
   try {
    formAuther.login();
   } catch (Exception e) {
    throw new RuntimeException(e);
   }
   return;
  }
  }
 private static synchronized void setCredentials()
   throws ParserConfigurationException, SAXException, IOException {

  if (authRulesRead)
   return;

  authRulesRead = true; // Avoid re-attempting to read
  InputStream is = conf.getConfResourceAsInputStream(authFile);
  if (is != null) {
   Document doc = DocumentBuilderFactory.newInstance()
     .newDocumentBuilder().parse(is);

   Element rootElement = doc.getDocumentElement();
   if (!"auth-configuration".equals(rootElement.getTagName())) {
    if (LOG.isWarnEnabled())
     LOG.warn("Bad auth conf file: root element <"
       + rootElement.getTagName() + "> found in "
       + authFile + " - must be <auth-configuration>");
   }

   // For each set of credentials
   NodeList credList = rootElement.getChildNodes();
   for (int i = 0; i < credList.getLength(); i++) {
    Node credNode = credList.item(i);
    if (!(credNode instanceof Element))
     continue;

    Element credElement = (Element) credNode;
    if (!"credentials".equals(credElement.getTagName())) {
     if (LOG.isWarnEnabled())
      LOG.warn("Bad auth conf file: Element <"
        + credElement.getTagName()
        + "> not recognized in " + authFile
        + " - expected <credentials>");
     continue;
    }
        // read http form post auth info
    String authMethod = credElement.getAttribute("authMethod");
    if (StringUtils.isNotBlank(authMethod)) {
     formConfigurer = readFormAuthConfigurer(credElement,
       authMethod);
     continue;
    }
      }
    }
  }
 private static HttpFormAuthConfigurer readFormAuthConfigurer(
   Element credElement, String authMethod) {
  if ("formAuth".equals(authMethod)) {
   HttpFormAuthConfigurer formConfigurer = new HttpFormAuthConfigurer();

   String str = credElement.getAttribute("loginUrl");
   if (StringUtils.isNotBlank(str)) {
    formConfigurer.setLoginUrl(str.trim());
   } else {
    throw new IllegalArgumentException("Must set loginUrl.");
   }
   str = credElement.getAttribute("loginFormId");
   if (StringUtils.isNotBlank(str)) {
    formConfigurer.setLoginFormId(str.trim());
   } else {
    throw new IllegalArgumentException("Must set loginFormId.");
   }
   str = credElement.getAttribute("loginRedirect");
   if (StringUtils.isNotBlank(str)) {
    formConfigurer.setLoginRedirect(Boolean.parseBoolean(str));
   }

   NodeList nodeList = credElement.getChildNodes();
   for (int j = 0; j < nodeList.getLength(); j++) {
    Node node = nodeList.item(j);
    if (!(node instanceof Element))
     continue;

    Element element = (Element) node;
    if ("loginPostData".equals(element.getTagName())) {
     Map<String, String> loginPostData = new HashMap<String, String>();
     NodeList childNodes = element.getChildNodes();
     for (int k = 0; k < childNodes.getLength(); k++) {
      Node fieldNode = childNodes.item(k);
      if (!(fieldNode instanceof Element))
       continue;

      Element fieldElement = (Element) fieldNode;
      String name = fieldElement.getAttribute("name");
      String value = fieldElement.getAttribute("value");
      loginPostData.put(name, value);
     }
     formConfigurer.setLoginPostData(loginPostData);
    } else if ("additionalPostHeaders".equals(element.getTagName())) {
     Map<String, String> additionalPostHeaders = new HashMap<String, String>();
     NodeList childNodes = element.getChildNodes();
     for (int k = 0; k < childNodes.getLength(); k++) {
      Node fieldNode = childNodes.item(k);
      if (!(fieldNode instanceof Element))
       continue;

      Element fieldElement = (Element) fieldNode;
      String name = fieldElement.getAttribute("name");
      String value = fieldElement.getAttribute("value");
      additionalPostHeaders.put(name, value);
     }
     formConfigurer
       .setAdditionalPostHeaders(additionalPostHeaders);
    } else if ("removedFormFields".equals(element.getTagName())) {
     Set<String> removedFormFields = new HashSet<String>();
     NodeList childNodes = element.getChildNodes();
     for (int k = 0; k < childNodes.getLength(); k++) {
      Node fieldNode = childNodes.item(k);
      if (!(fieldNode instanceof Element))
       continue;

      Element fieldElement = (Element) fieldNode;
      String name = fieldElement.getAttribute("name");
      removedFormFields.add(name);
     }
     formConfigurer.setRemovedFormFields(removedFormFields);
    }
   }
   return formConfigurer;
  } else {
   throw new IllegalArgumentException("Unsupported authMethod: "
     + authMethod);
  }
 }  
}  

Resources

Nutch Jira 827 HTTP POST Authentication

Nutch HttpAuthenticationSchemes

Nutch2 Http Form Authentication-Part2: Make Http Post Form Authentication Work

The Problem
Http Form-based Authentication is a very common used authentication mechanism to protect web resources.
When crawl, Nutch supports NTLM, Basic or Digest authentication to authenticate itself to websites. But It doesn't support Http Post Form Authentication.

This series of articles talks about how to extend Nutch2 to support Http Post Form Authentication.
Main Steps
Use Apache Http Client to do http post form authentication.
Make http post form authentication work.
Integrate form authentication in Nutch2.

This article will focus on how to make http post form authentication work via a practical example.
Create and Run ASP.NET Web Application
In visual studio, create a ASP.NET (MVC2) web application, the default created web application supports form authentication. It's good to test our http form login.

Write Test Code
To use HttpFormAuthentication to do http post form authentication, we have to figure out the loginFormId: this can be done by searching "<form" in page source. Also use Chrom Devtools's "Inspect element" function, we can easily find out the name of username and password fields. Be sure to use name field, not id field of input element.

Now we can write test code:

private static void authTestAspWebApp() throws Exception, IOException {
  HttpFormAuthConfigurer authConfigurer = new HttpFormAuthConfigurer();
  authConfigurer.setLoginUrl("http://localhost:44444/Account/Login.aspx")
    .setLoginFormId("ctl01").setLoginRedirect(true);
  Map<String, String> loginPostData = new HashMap<String, String>();
  loginPostData.put("ctl00$MainContent$LoginUser$UserName", "admin");
  loginPostData.put("ctl00$MainContent$LoginUser$Password", "admin123");
  authConfigurer.setLoginPostData(loginPostData);

  Set<String> removedFormFields = new HashSet<String>();
  removedFormFields.add("ctl00$MainContent$LoginUser$RememberMe");
  authConfigurer.setRemovedFormFields(removedFormFields);

  HttpFormAuthentication example = new HttpFormAuthentication(
    authConfigurer);

  // example.client.getHostConfiguration().setProxy("127.0.0.1", 8888);

  String proxyHost = System.getProperty("http.proxyHost");
  String proxyPort = System.getProperty("http.proxyPort");
  if (StringUtils.isNotBlank(proxyHost)
    && StringUtils.isNotBlank(proxyPort)) {
   example.client.getHostConfiguration().setProxy(proxyHost,
     Integer.parseInt(proxyPort));
  }

  example.login();
  String result = example
    .httpGetPageContent("http://localhost:44444/secret/needlogin.aspx");
  System.out.println(result);
 }

Run the previous test code, check Response Code, Response headers and response body. We can copy the whole response body to jsbin, there we can view the html much easily.

What to Do if it doesn't Work?
But sometimes things are not that simple, the previous code may still not work: that user is not logined, and we can't access protected resource.

When this happens, we need compare the request Apache http client sends with the request Chrome sends, including headers and request body. 

We can use Chrome DevTools to get request headers and post body, we can even copy the request as a cURL request and execute in command line.

We can start fiddler as a proxy, add example.client.getHostConfiguration().setProxy("127.0.0.1", 8888); in test code, then monitor request and response Apache http client sends and receives in fiddler.

Compare them and check whether some headers a missing, if so add them into additionalPostHeaders. Check whether we need remove some fields, if so add them into removedFormFields. Check whether we need add more fields, if so add them into loginPostData.

After all this, we should be able to make it work.
We can get request headers and post body via Chrome DevTools like below, we can even copy the request as a cURL request and execute in command line.

Nutch2 Http Form Authentication-Part1: Using Apache Http Client to Do Http Post Form Authentication

The Problem
Http Form-based Authentication is a very common used authentication mechanism to protect web resources.
When crawl, Nutch supports NTLM, Basic or Digest authentication to authenticate itself to websites. But It doesn't support Http Post Form Authentication.

This series of articles talks about how to extend Nutch2 to support Http Post Form Authentication.
Main Steps
Use Apache Http Client to do http post form authentication.
Make http post form authentication work.
Integrate post from authentication in Nutch2.

Use Apache Http Client to Do Http Post Form Authentication
HttpFormAuthConfigurer
First let's check the HttpFormAuthConfigurer class. No need to explain loginUrl and loginFormId. loginPostData stores the field name and value for login fields, such as username:user1, passowrd:password1. removedFormFields told us input field we want to remove, additionalPostHeaders is uesed when we have to add addtional header name and value when do post form login. if loginRedirect is true, and http post login returns redirect code: 301 or 302, Http Client will automatically follow the redirect.

package org.apache.nutch.protocol.httpclient;
public class HttpFormAuthConfigurer {
 private String loginUrl;
 private String loginFormId;
 private Map<String, String> loginPostData;
 private Set<String> removedFormFields; 
 private Map<String, String> additionalPostHeaders;
 private boolean loginRedirect;
} 

HttpFormAuthentication 
In login method, it first calls CookieHandler.setDefault(new CookieManager()); so if login succeeds, subsequent request would not require login again.

Then it sends a http get request to the loginUrl, uses Jsoup.parse(pageContent) to parse the response, iterates all input fields in the login form, adds all field names and values into List params, sets values for username and password fields which are stored in loginPostData, we may also have to remove some form fields(in removedFormFields). Then send a post request to the loginUrl with data: List params.

The following code uses Apache Http Client 3.x, as Nutch2 still uses the pretty old http client library.

package org.apache.nutch.protocol.httpclient;

public class HttpFormAuthentication {
 private static final Logger LOGGER = LoggerFactory
   .getLogger(HttpFormAuthentication.class);
 private static Map<String, String> defaultLoginHeaders = new HashMap<String, String>();
 static {
  defaultLoginHeaders.put("User-Agent", "Mozilla/5.0");
  defaultLoginHeaders
    .put("Accept",
      "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
  defaultLoginHeaders.put("Accept-Language", "en-US,en;q=0.5");
  defaultLoginHeaders.put("Connection", "keep-alive");
  defaultLoginHeaders.put("Content-Type",
    "application/x-www-form-urlencoded");
 }

 private HttpClient client;
 private HttpFormAuthConfigurer authConfigurer = new HttpFormAuthConfigurer();
 private String cookies;

 public HttpFormAuthentication(HttpFormAuthConfigurer authConfigurer) {
  this.authConfigurer = authConfigurer;
  this.client = new HttpClient();
 }
 public HttpFormAuthentication(HttpFormAuthConfigurer authConfigurer,
   HttpClient client, Http http) {
  this.authConfigurer = authConfigurer;
  this.client = client;
  defaultLoginHeaders.put("Accept", http.getAccept());
  defaultLoginHeaders.put("Accept-Language", http.getAcceptLanguage());
  defaultLoginHeaders.put("User-Agent", http.getUserAgent());
 }
 public void login() throws Exception {
  // make sure cookies is turn on
  CookieHandler.setDefault(new CookieManager());
  String pageContent = httpGetPageContent(authConfigurer.getLoginUrl());
  List<NameValuePair> params = getLoginFormParams(pageContent);
  sendPost(authConfigurer.getLoginUrl(), params);
 }

 private void sendPost(String url, List<NameValuePair> params)
   throws Exception {
  PostMethod post = null;
  try {
   if (authConfigurer.isLoginRedirect()) {
    post = new PostMethod(url) {
     @Override
     public boolean getFollowRedirects() {
      return true;
     }
    };
   } else {
    post = new PostMethod(url);
   }
   // we can't use post.setFollowRedirects(true) as it will throw
   // IllegalArgumentException:
   // Entity enclosing requests cannot be redirected without user
   // intervention
   setLoginHeader(post);
   post.addParameters(params.toArray(new NameValuePair[0]));
   // post.setEntity(new UrlEncodedFormEntity(postParams));

   int rspCode = client.executeMethod(post);
   if (LOGGER.isDebugEnabled()) {
    LOGGER.info("rspCode: " + rspCode);
    LOGGER.info("\nSending 'POST' request to URL : " + url);

    LOGGER.info("Post parameters : " + params);
    LOGGER.info("Response Code : " + rspCode);

    for (Header header : post.getRequestHeaders()) {
     LOGGER.info("Response headers : " + header);
    }
   }
   String rst = IOUtils.toString(post.getResponseBodyAsStream());
   LOGGER.debug("login post result: " + rst);
  } finally {
   if (post != null) {
    post.releaseConnection();
   }
  }
 }

 private void setLoginHeader(PostMethod post) {
  Map<String, String> headers = new HashMap<String, String>();
  headers.putAll(defaultLoginHeaders);
  // additionalPostHeaders can overwrite value in defaultLoginHeaders
  headers.putAll(authConfigurer.getAdditionalPostHeaders());
  for (Entry<String, String> entry : headers.entrySet()) {
   post.addRequestHeader(entry.getKey(), entry.getValue());
  }
  post.addRequestHeader("Cookie", getCookies());
 }

 private String httpGetPageContent(String url) throws IOException {

  GetMethod get = new GetMethod(url);
  try {
   for (Entry<String, String> entry : authConfigurer
     .getAdditionalPostHeaders().entrySet()) {
    get.addRequestHeader(entry.getKey(), entry.getValue());
   }
   client.executeMethod(get);
      
   Header cookieHeader = get.getResponseHeader("Set-Cookie");
   if (cookieHeader != null) {
    setCookies(cookieHeader.getValue());
   }
   return IOUtils.toString(get.getResponseBodyAsStream());
  } finally {
   get.releaseConnection();
  }
 }

 private List<NameValuePair> getLoginFormParams(String pageContent)
   throws UnsupportedEncodingException {
  List<NameValuePair> params = new ArrayList<NameValuePair>();
  Document doc = Jsoup.parse(pageContent);
  Element loginform = doc.getElementById(authConfigurer.getLoginFormId());
  if (loginform == null) {
   throw new IllegalArgumentException("No form exists: "
     + authConfigurer.getLoginFormId());
  }
  Elements inputElements = loginform.getElementsByTag("input");

  // skip fields in removedFormFields or loginPostData
  for (Element inputElement : inputElements) {
   String key = inputElement.attr("name");
   String value = inputElement.attr("value");
   if (authConfigurer.getLoginPostData().containsKey(key)
     || authConfigurer.getRemovedFormFields().contains(key)) {
    continue;
   }
   params.add(new NameValuePair(key, value));
  }
  // add key and value in loginPostData
  for (Entry<String, String> entry : authConfigurer.getLoginPostData()
    .entrySet()) {
   params.add(new NameValuePair(entry.getKey(), entry.getValue()));
  }
  return params;
 }
}

Http Form Authentication in Apache Http Client 4.x

public class HttpCilentFormLoginExample {
  private static final Logger LOGGER = LoggerFactory
      .getLogger(HttpCilentFormLoginExample.class);
  private DefaultHttpClient client = new DefaultHttpClient();
  private String loginUrl, loginForm;  
  private static Map<String,String> defaultLoginHeaders = new HashMap<String,String>();  
  static {
    defaultLoginHeaders.put("User-Agent", "Mozilla/5.0");
    defaultLoginHeaders.put("Accept",
        "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    defaultLoginHeaders.put("Accept-Language", "en-US,en;q=0.5");
    defaultLoginHeaders.put("Connection", "keep-alive");
    // defaultLoginHeaders.put("Referer",
    // "https://accounts.google.com/ServiceLoginAuth");
    defaultLoginHeaders
        .put("Content-Type", "application/x-www-form-urlencoded");
  }
  private Map<String,String> loginPostData;
  private Map<String,String> additionalPostHeaders;
  private Set<String> removedFormFields;
  private String cookies;
  
  public HttpCilentFormLoginExample(String loginUrl, String loginForm,
      Map<String,String> loginPostData,
      Map<String,String> additionalPostHeaders, Set<String> removedFormFields) {
    this.loginUrl = loginUrl;
    this.loginForm = loginForm;
    this.loginPostData = loginPostData == null ? new HashMap<String,String>()
        : loginPostData;
    this.additionalPostHeaders = additionalPostHeaders == null ? new HashMap<String,String>()
        : additionalPostHeaders;
    this.removedFormFields = removedFormFields == null ? new HashSet<String>()
        : removedFormFields;
  }
    
  public void login() throws Exception, UnsupportedEncodingException {
    client.setRedirectStrategy(new LaxRedirectStrategy());
    // make sure cookies is turn on
    CookieHandler.setDefault(new CookieManager());
    String pageContent = httpGetPageContent(loginUrl);
    List<NameValuePair> postParams = getLoginFormParams(pageContent);
    sendPost(loginUrl, postParams);
  }
  
  private void sendPost(String url, List<NameValuePair> postParams)
      throws Exception {
    HttpPost post = new HttpPost(url);
    try {
      setLoginHeader(post);
      post.setEntity(new UrlEncodedFormEntity(postParams));      
      HttpResponse response = client.execute(post);      
      int responseCode = response.getStatusLine().getStatusCode();
      if (LOGGER.isDebugEnabled()) {
        LOGGER.info("rspCode: " + responseCode);
        LOGGER.info("\nSending 'POST' request to URL : " + url);
        LOGGER.info("Post parameters : " + postParams);
        for (Header header : response.getAllHeaders()) {
          LOGGER.info("Response headers : " + header);
        }
      }
      String rst = IOUtils.toString(response.getEntity().getContent());
      LOGGER.debug("login post result: " + rst);
    } finally {
      post.releaseConnection();
    }
  }
  
  private void setLoginHeader(HttpPost post) {
    Map<String,String> headers = new HashMap<String,String>();
    headers.putAll(defaultLoginHeaders);
    // additionalPostHeaders can overwrite value in defaultLoginHeaders
    headers.putAll(additionalPostHeaders);
    for (Entry<String,String> entry : headers.entrySet()) {
      post.setHeader(entry.getKey(), entry.getValue());
    }
    post.setHeader("Cookie", getCookies());
  }
  
  private String httpGetPageContent(String url) throws IOException {    
    HttpGet get = new HttpGet(url);
    try {
      for (Entry<String,String> entry : additionalPostHeaders.entrySet()) {
        get.setHeader(entry.getKey(), entry.getValue());
      }
      HttpResponse response = client.execute(get);
      setCookies(response.getFirstHeader("Set-Cookie") == null ? "" : response
          .getFirstHeader("Set-Cookie").toString());
      return IOUtils.toString(response.getEntity().getContent());
    } finally {
      get.releaseConnection();
    }    
  }
  
  private List<NameValuePair> getLoginFormParams(String pageContent)
      throws UnsupportedEncodingException {
    Document doc = Jsoup.parse(pageContent);
    List<NameValuePair> paramList = new ArrayList<NameValuePair>();
    Element loginform = doc.getElementById(loginForm);
    if (loginform == null) {
      throw new IllegalArgumentException("No form exists: " + loginForm);
    }
    Elements inputElements = loginform.getElementsByTag("input");
    // skip fields in removedFormFields or loginPostData
    for (Element inputElement : inputElements) {
      String key = inputElement.attr("name");
      String value = inputElement.attr("value");
      if (loginPostData.containsKey(key) || removedFormFields.contains(key)) {
        continue;
      }
      paramList.add(new BasicNameValuePair(key, value));
    }
    // add key and value in loginPostData
    for (Entry<String,String> entry : loginPostData.entrySet()) {
      paramList.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));
    }
    return paramList;
  }
}
Resources
Cookie Handling in Java SE 6
Apache HttpClient – Automate login Google

Using Solr DocTransformer to Add Anchor Tag and Text into Response

This series talks about how to use Nutch and Solr to implement Google Search's "Jump to" and Anchor links features. This article introduces how to use Nutch, HTML Parser Jsoup and Regular Expression to Extract Anchor Tag and Content
The Problem
In the search result, to help users easily jump to the section uses may be interested, we want to add anchor link below page description. Just like Google Search's "Jump to" and Anchor links features.
Main Steps
1. Extract anchor tag, text and content in Nutch
Please refer to
Using Nutch to Extract Anchor Tag and Content
Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression
2. Using UpdateRequestProcessor to Store Anchor Tag and Content into Solr
3. Using Solr DocTransformer to Add Anchor Tag and Content into Response
This is described in current article.

Task: Using Solr DocTransformer to Add Anchor Tag and Content into Response
In previous article, we have used Nutch to extract anchor tag, text and content from web page, and saved content into Solr as separate docs with docType 1.

The first thought was to use Solr group feature: &q=keyword&fl=anchorTag,anchorText,anchorContent&group=true&group.field=url_sort&group.limit=6. 

Then we ignore the url of the main page in group, and convert the anchors in group response to anchors map: the key is the anchorTag, the value is the anchorText.

But there is one critical issue in this approach: the groups are sorted by the score of the top document within each group. 

In a webpage, there maybe a anchor section: its content is small and matches the keyword: for example, the score is 0.9, but the whole webpage is not really related with the keyword: the whole webpage's score is 0.01. But solr sorts groups by the score of the top document within each group. So this group's score would be 0.9, and would be listed first. This is unacceptable.

To return tag information for the web page that matches the query, we decide to use Solr DocTransformer to add fields into response.

[future thought]
We can change solr's code to make solr run DocTransformer in parallel to improve performance.

AnchorTransformerFactory
DocTransformer is very powerful and useful, allows us to add/remove or update fields before returning. But it has one limit: it can only add one field, and the field name must be [transformer_name].

AnchorTransformer adds tow fields anchorTag, anchorText into SolrDocument. If we just use fl=[anchors], the response would not contains these fields. We have to use fl=[anchors],anchorTag,anchorText. The anchorTag,anchorText would tell Solr to add them into SolrReturnFields. Please refer the code at SolrReturnFields.add(String, NamedList<String>, DocTransformers, SolrQueryRequest).

public class AnchorTransformerFactory extends TransformerFactory {
  
  private String defaultSort;
  private int defaultAnchorRows = 5;
  private static final String SORT_BY_ORDER = "order";
  protected static Logger logger = LoggerFactory
      .getLogger(AnchorTransformerFactory.class);
  public void init(NamedList args) {
    super.init(args);
    Object obj = args.get("sort");
    if (obj != null) {
      defaultSort = (String) obj;
    }
    obj = args.get("anchorRows");
    if (obj != null) {
      defaultAnchorRows = Integer.parseInt(obj.toString());
    }
  }
  @Override
  public DocTransformer create(String field, SolrParams params,
      SolrQueryRequest req) {
    String sort = defaultSort;
    if (!StringUtils.isBlank(params.get("sort"))) {
      sort = params.get("sort");
    }
    int anchorRows = defaultAnchorRows;
    if (StringUtils.isNotBlank(params.get("anchorRows"))) {
      anchorRows = Integer.parseInt(params.get("anchorRows"));
    }
    return new AnchorTransformer(field, req, sort, anchorRows);
  }
  
  private static class AnchorTransformer extends DocTransformer {
    private SolrQueryRequest req;
    private String sort;
    private int anchorRows;
    
    public AnchorTransformer(String field, SolrQueryRequest req, String sort,
        int anchorRows) {
      this.req = req;
      this.sort = sort;
      this.anchorRows = anchorRows;
    }
    
    @Override
    public void transform(SolrDocument doc, int docid) throws IOException {
      String oldQuery = req.getParams().get(CommonParams.Q);
      Object idObj = doc.getFieldValue("contentid");
      
      // java.lang.RuntimeException: When this is called? obj.type:class
      // org.apache.lucene.document.LazyDocument$LazyField at
      String id;
      if (idObj instanceof org.apache.lucene.document.Field) {
        org.apache.lucene.document.Field field = (Field) idObj;
        id = field.stringValue();
      } else if (idObj instanceof IndexableField) {
        IndexableField field = (IndexableField) idObj;
        id = field.stringValue();
      } else {
        throw new RuntimeException("When this is called? obj.type:"
            + idObj.getClass());
      }
      SolrQuery query = new SolrQuery();
      query
          .setQuery(
              "anchorContent:" + ClientUtils.escapeQueryChars(oldQuery)
                  + " AND url: " + ClientUtils.escapeQueryChars(id))
          .addFilterQuery("docType:1").setRows(anchorRows)
          .setFields("anchorTag", "anchorText");
      if (SORT_BY_ORDER.equals(sort)) {
        query.setSort("anchorOrder", ORDER.asc);
      }
      // else default, sort by score
      List<Map<String,String>> anchorMap = extractSingleFieldValues(
          req.getCore(), "/select", query, "anchorTag", "anchorText");
      for (Map<String,String> map : anchorMap) {
        doc.addField("anchorTag", map.get("anchorTag"));
        doc.addField("anchorText", map.get("anchorText"));
      }
    }
    
  public static List<Map<String,String>> extractSingleFieldValues(
      SolrCore core, String handlerName, SolrQuery query, String... fls)
      throws IOException {
    SolrRequestHandler requestHandler = core.getRequestHandler(handlerName);
    query.setFields(fls);
    SolrQueryRequest newReq = new LocalSolrQueryRequest(core, query);
    try {
      SolrQueryResponse queryRsp = new SolrQueryResponse();
      requestHandler.handleRequest(newReq, queryRsp);
      return extractSingleFieldValues(newReq, queryRsp, fls);
    } finally {
      newReq.close();
    }
  }
  
  @SuppressWarnings("rawtypes")
  public static List<Map<String,String>> extractSingleFieldValues(
      SolrQueryRequest newReq, SolrQueryResponse newRsp, String[] fls)
      throws IOException {
    List<Map<String,String>> rst = new ArrayList<Map<String,String>>();
    NamedList contentIdNL = newRsp.getValues();
    
    Object rspObj = contentIdNL.get("response");
    SolrIndexSearcher searcher = newReq.getSearcher();    
    if (rspObj instanceof ResultContext) {
      ResultContext resultContext = (ResultContext) rspObj;
      DocList doclist = resultContext.docs;
      DocIterator dit = doclist.iterator();
      while (dit.hasNext()) {
        int docid = dit.nextDoc();
        Document doc = searcher.doc(docid, new HashSet<String>());
        Map<String,String> row = new HashMap<String,String>();
        for (String fl : fls) {
          row.put(fl, doc.get(fl));
        }
        rst.add(row);
      }
    } else if (rspObj instanceof SolrDocumentList) {
      SolrDocumentList docList = (SolrDocumentList) rspObj;
      Iterator<SolrDocument> docIt = docList.iterator();
      while (docIt.hasNext()) {
        SolrDocument doc = docIt.next();
        docIt.remove();
        Map<String,String> row = new HashMap<String,String>();
        for (String fl : fls) {
          Object tmp = doc.getFieldValue(fl);
          if (tmp != null) {
            row.put(fl, tmp.toString());
          }
        }
        rst.add(row);
      }
    }
    return rst;
  }    
  } 
}

SolrConfig.xml

<transformer name="anchors" class="AnchorTransformerFactory" >
    <int name="anchorRows">5</int>
  </transformer>
  <requestHandler name="/select" class="solr.SearchHandler"
  default="true">  
      <lst name="defaults">
          <str name="fl">otherfields,[anchors],anchorTag,anchorText</str>
       </lst>
   </requestHandler>

Resource
Using UpdateRequestProcessor to Store Anchor Tag and Content into Solr
Using Nutch to Extract Anchor Tag and Content
Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression

Using UpdateRequestProcessor to Store Anchor Tag and Content into Solr

This series talks about how to use Nutch and Solr to implement Google Search's "Jump to" and Anchor links features. This article introduces how to use Nutch, HTML Parser Jsoup and Regular Expression to Extract Anchor Tag and Content
The Problem 
In the search result, to help users easily jump to the section uses may be interested, we want to add anchor link below page description. Just like Google Search's "Jump to" and Anchor links features.
Main Steps
1. Extract anchor tag, text and content in Nutch.
Also refer to
Using Nutch to Extract Anchor Tag and Conten
Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression
2. Using UpdateRequestProcessor to Store Anchor Tag and Content into Solr
This is described in this article. 
3. Using DocTransformer to Add Anchor tag and content into response. 

Task: Using UpdateRequestProcessor to Store Anchor Tag and Content into Solr
In previous article, we have used Nutch to extract anchor tag, text and content from web page, and add into Solr documents: anchorTags, anchorTexts, anchorContents. These three fields are a list of string.

In Solr side, it will use a UpdateRequestProcessor to remove these three fields, and add a new Document for each anchor, set docType as 1: 0 means, this doc is a web page. 1 means an anchor.
The web page doc and anchor docs is a parent-child relationship.
Code

public class AnchorContentProcessorFactory extends
    UpdateRequestProcessorFactory {
  
  private String fromFlAnchorTags, fromFlAnchorTexts, fromFlAnchorContents;
  private String toFlAnchorTag, toFlAnchorText, toFlAnchorContent,
      toFlAnchorOrder, flForeignKey;
  
  public void init(NamedList args) {
    super.init(args);
    if (args != null) {
      SolrParams params = SolrParams.toSolrParams(args);
      fromFlAnchorTags = checkNotNull(params.get("fromFlAnchorTags"),
          "fromFlAnchorTags can't be null");
      fromFlAnchorTexts = checkNotNull(params.get("fromFlAnchorTexts"),
          "fromFlAnchorTexts can't be null");
      fromFlAnchorContents = checkNotNull(params.get("fromFlAnchorContents"),
          "fromFlAnchorContents can't be null");
      
      toFlAnchorTag = checkNotNull(params.get("toFlAnchorTag"),
          "toFlAnchorTag can't be null");
      toFlAnchorText = checkNotNull(params.get("toFlAnchorText"),
          "toFlAnchorText can't be null");
      toFlAnchorContent = checkNotNull(params.get("toFlAnchorContent"),
          "toFlAnchorContent can't be null");
      toFlAnchorOrder = checkNotNull(params.get("toFlAnchorOrder"),
          "toFlAnchorOrder can't be null");
      flForeignKey = checkNotNull(params.get("flForeignKey"),
          "flForeignKey can't be null");
    }
  }
  
  @Override
  public UpdateRequestProcessor getInstance(SolrQueryRequest req,
      SolrQueryResponse rsp, UpdateRequestProcessor next) {
    return new AnchorContentProcessor(next);
  }
  
  class AnchorContentProcessor extends UpdateRequestProcessor {
    
    public AnchorContentProcessor(UpdateRequestProcessor next) {
      super(next);
    }
    
    @Override
    public void processAdd(AddUpdateCommand cmd) throws IOException {
      
      SolrInputDocument oldDoc = cmd.solrDoc;
      // docType 0 means this item is full web page.
      // docType 1 means this item is anchor.
      oldDoc.setField("docType", 0);
      Collection<Object> fromAnchorTags = oldDoc
          .getFieldValues(fromFlAnchorTags);
      Collection<Object> fromAnchorTexts = oldDoc
          .getFieldValues(fromFlAnchorTexts);
      Collection<Object> fromAnchorContents = oldDoc
          .getFieldValues(fromFlAnchorContents);
      
      if (fromAnchorTags != null && fromAnchorTexts != null
          && fromAnchorContents != null) {
        if (fromAnchorTags.size() != fromAnchorTexts.size()
            || fromAnchorTags.size() != fromAnchorContents.size()) throw new RuntimeException(
            "size doesn't match: size of fromAnchorTags: "
                + fromAnchorTags.size() + ", size of fromAnchorTexts: "
                + fromAnchorTexts.size() + ", size of fromAnchorContents: "
                + fromAnchorContents.size());
        
        // add a new document
        AddUpdateCommand newCmd = new AddUpdateCommand(cmd.getReq());
        SolrInputDocument newDoc = new SolrInputDocument();
        
        Iterator<Object> it1 = fromAnchorTags.iterator(), it2 = fromAnchorTexts
            .iterator(), it3 = fromAnchorContents.iterator();
        int order = 0;
        while (it1.hasNext()) {
          // avoid construct new SolrInputDocument
          newDoc.clear();
          newDoc.addField(toFlAnchorTag, it1.next().toString());
          newDoc.addField(toFlAnchorText, it2.next().toString());
          newDoc.addField(toFlAnchorContent, it3.next().toString());
          newDoc.addField(toFlAnchorOrder, order++);
          
          String uniqueFl = newCmd.getReq().getSchema().getUniqueKeyField()
              .getName();
          newDoc.addField(uniqueFl,
              UUID.randomUUID().toString().toLowerCase(Locale.ROOT).toString());
          newDoc.addField(flForeignKey, oldDoc.getFieldValue(uniqueFl)
              .toString());
          // set docType 1 for the anchor item
          newDoc.addField("docType", 1);
          newCmd.solrDoc = newDoc;
          super.processAdd(newCmd);
        }
      }
      
      oldDoc.removeField(fromFlAnchorTags);
      oldDoc.removeField(fromFlAnchorTexts);
      oldDoc.removeField(fromFlAnchorContents);
      super.processAdd(cmd);
    }
  } 
}

SolrConfig.xml

<processor
   class="com.commvault.solr.update.processor.CVAnchorContentProcessorFactory">
      <str name="fromFlAnchorTags">anchorTags</str>
      <str name="fromFlAnchorTexts">anchorTexts</str>
      <str name="fromFlAnchorContents">anchorContents</str>

      <str name="toFlAnchorTag">anchorTag</str>
      <str name="toFlAnchorText">anchorText</str>
      <str name="toFlAnchorContent">anchorContent</str>
      <str name="toFlAnchorOrder">anchorOrder</str>
      <str name="flForeignKey">url</str>
    </processor>  

Schema.xml

<field name="docType" type="tint" indexed="true" stored="true" multiValued="false" /> 
    <field name="anchorTag" type="string" indexed="false" stored="true"  multiValued="false" /> 
    <field name="anchorText" type="string" indexed="false" stored="true" multiValued="false" /> 
    <field name="anchorContent" type="text_rev" indexed="true" stored="false" multiValued="false" /> 
    <field name="anchorOrder" type="tint" indexed="true" stored="true" multiValued="false" /> 

Resource
Using Nutch to Extract Anchor Tag and Content
Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression

Using Nutch to Extract Anchor Tag and Content

This series talks about how to use Nutch and Solr to implement Google Search's "Jump to" and Anchor links features.
The Problem 
In the search result, to help users easily jump to the section uses may be interested, we want to add anchor link below page description. Just like Google Search's "Jump to" and Anchor links features.
Main Steps
1. Extract anchor tag, text and content in Nutch.
This is described in this article and Using HTML Parser Jsoup and Regular Expression to Get Text between Tow Tags and Debugging and Optimizing Regular Expression
2. Save anchor information to Solr.
3. Return Anchor tag and text that matches the query. 

Task: Extract anchor tag, text and content in Nutch
We will write a Nutch plugin named index-anchor-content: it implements IndexingFilter extension point. 

Its getFields returns a collection that contains WebPage.Field.CONTENT field. This will tell Nutch to read Content field from the underlying data store. Without this step, the WebPage instance in filter(NutchDocument, String, WebPage) method would not have value for content field.

In filter method, we use jsoup to extract all anchor links in div[id=toc] ul>li section. 

Then use regular expression <span[^>]*\bid\s*=\s*(?:"|')?{0}(?:'|")?[^>]*>([^<]*)</span>(.*?)<span[^>]*\bid\s*=\s*(?:"|')?{1}(?:'|")?[^>]*>[^<]*</span> to extract tag, text and content for each anchor. {0} and {1} the anchor tag of anchor1 and anchor2. 

We then add them into NutchDocument fields: anchorTags, anchorTexts, anchorContents.

Please read more from Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression

The detailed step to build nutch plugin are omitted. Please refer to Writing Nutch Plugin Example.
Code

public class AnchorContentIndexingFilter implements IndexingFilter {

  public static final Logger LOG = LoggerFactory
      .getLogger(AnchorContentIndexingFilter.class);
  private Configuration conf;
  private static final Collection<WebPage.Field> FIELDS = new HashSet<WebPage.Field>();
  static {
    FIELDS.add(WebPage.Field.CONTENT);
  }
  private static final String DEFAULT_REGEX_TOC_ANCHOR = "div[id=toc] ul>li a[href^=#]:not([href=#])";
  private static final String DEFAULT_REGEX_PLAIN_ANCHOR_TAG = "a[href^=#]:not([href=#])";

  private static final int DEFAULT_MAX_ANCHOR_LINKS = 20;
  private static final String DEFAULT_FL_ANCHOR_TAGS = "anchorTags",
      DEFAULT_FL_ANCHOR_TEXTS = "anchorTexts",
      DEFAULT_FL_ANCHOR_CONTENTS = "anchorContents",
      DEFAULT_REGEX_BODY_ROOT = "article[id=sectionContent]",
      DEFAULT_REGEX_EXTRACT_CONTENT = "<span[^>]*?\bid\\s*=\\s*(?:\"|')?{0}(?:'|\")?[^>]*>([^<]*)</span>(.*?)<span[^>]*?\bid\\s*=\\s*(?:\"|')?{1}(?:'|\")?[^<]*>([^<]*)</span>";

  private String flAnchorTags, flAnchorTexts, flAnchorContents, regexTocAnchor,
      // if can't find tocAnchor in web page, revert to plainAnchorTag
      regexPlainAnchorTag,
      // if exists, only search content in this section
      regexBodyRoot;

  private boolean extractOtherAnchors = false;
  /**
   * the regex to extract content between two tags: <br>
   * 1. The string must have 2 place holders {0}, {1}, it will be replaced by the
   * anchor name at runtime.<br>
   * 2. There must be 3 regex group, the first group is to extract the text
   * of the first anchor, the second group is to extract content between the two
   * anchors, the third is to extract the text of the second anchor.<br>
   * 3. If ther is single quote ' in the regex string, have to replaced by
   * doubled single quotes '' due to the usage of MessageFormat.check:
   * http://docs.oracle.com/javase/7/docs/api/java/text/MessageFormat.html <br>
   * Check DEFAULT_REGEX_EXTRACT_CONTENT
   */
  private String regexExtractContent = DEFAULT_REGEX_EXTRACT_CONTENT;

  private int maxAnachorLinks = DEFAULT_MAX_ANCHOR_LINKS;
  private MessageFormat MSG_FORMAT;

  @Override
  public NutchDocument filter(NutchDocument doc, String url, WebPage page)
      throws IndexingException {

    ByteBuffer dataBuffer = page.getContent();
    String content = new String(dataBuffer.array());

    Document rootDoc = Jsoup.parse(content);
    try {
      List<Anchor> anchors = parseAnchors(rootDoc);
      for (Anchor anchor : anchors) {
        if (StringUtils.isNotBlank(anchor.getTag())
            && StringUtils.isNotBlank(anchor.getText())
            && StringUtils.isNotBlank(anchor.getContent())) {
          doc.add(flAnchorTags, anchor.getTag());
          doc.add(flAnchorTexts, anchor.getText());
          doc.add(flAnchorContents, anchor.getContent());
        }
      }
    } catch (IOException e) {
      throw new IndexingException(e);
    }
    return doc;
  }

  public List<Anchor> parseAnchors(Document rootDoc) throws IOException {
    List<Anchor> anchorContents = new LinkedList<Anchor>();
    Element rootElement = rootDoc;
    if (regexBodyRoot != null) {
      rootElement = rootDoc.select(regexBodyRoot).first();
    }
    if (rootElement == null)
      return anchorContents;
    Set<String> anchors = getAnchors(rootElement);
    if (anchors.isEmpty())
      return anchorContents;
    StringBuilder remainingTxt = new StringBuilder(rootElement.toString());

    Iterator<String> it = anchors.iterator();
    String curAnchorTag = it.next();
    String lastAnchorTag = null;
    while (it.hasNext() && remainingTxt.length() > 0) {
      String nextAnchorTag = it.next();
      Anchor anchor = getContentBetweenAnchor(remainingTxt, curAnchorTag, nextAnchorTag);
      anchorContents.add(anchor);
      if (!it.hasNext()) {
        // only for last anchor
        lastAnchorTag = anchor.getNextTagText();
      }
      curAnchorTag = nextAnchorTag;
    }
    // Don't forget last tag
    String lastTxt = Jsoup.parse(remainingTxt.toString()).text();
    if (StringUtils.isNotBlank(lastTxt)) {
      anchorContents.add(new Anchor(curAnchorTag, lastAnchorTag, lastTxt));
    }
    return anchorContents;
  }

  public Set<String> getAnchors(Element rootElement) {
    Set<String> anchors = new LinkedHashSet<String>() {
      private static final long serialVersionUID = 1L;

      @Override
      public boolean add(String e) {
        if (size() >= maxAnachorLinks)
          return false;
        return super.add(e);
      }
    };
    getAnchorsImpl(rootElement, regexTocAnchor, anchors);
    if (anchors.isEmpty() && extractOtherAnchors) {
      getAnchorsImpl(rootElement, regexPlainAnchorTag, anchors);
    }
    return anchors;
  }

  public void getAnchorsImpl(Element rootElement, String anchorPattern,
      Set<String> anchors) {
    Elements elements = rootElement.select(anchorPattern);
    if (!elements.isEmpty()) {
      for (Element element : elements) {
        String href = element.attr("href");
        anchors.add(href.substring(1));
      }
    }
  }
  public Anchor getContentBetweenAnchor(StringBuilder remainingTxt,
      String curAnchorTag, String nextAnchorTag) throws IOException {
    Anchor anchor = null;
    String regex = MSG_FORMAT.format(new String[] { curAnchorTag, nextAnchorTag });
    Matcher matcher = Pattern
        .compile(regex, Pattern.DOTALL | Pattern.MULTILINE).matcher(remainingTxt);
    if (matcher.find()) {
      String anchorText = Jsoup.parse(matcher.group(1)).text();
      String anchorContent = anchorText + " "
          + Jsoup.parse(matcher.group(2)).text();
      String nextTagText = matcher.group(3);
      anchor = new Anchor(curAnchorTag, anchorText, anchorContent, nextTagText);

      int g2End = matcher.end(2);
      remainingTxt.delete(0, g2End);
    }
    return anchor;
  }

  @Override
  public Collection<WebPage.Field> getFields() {
    return FIELDS;
  }
  
  private static class Anchor {
    private String tag, text, content,
    // used to get last tag text
    nextTagText;
  }
  public void setConf(Configuration conf) {
    this.conf = conf;
  
    flAnchorTags = getValue(conf, "indexer.anchorContent.field.anchorTags",
        DEFAULT_FL_ANCHOR_TAGS, false);
    flAnchorTexts = getValue(conf, "indexer.anchorContent.field.anchorTags",
        DEFAULT_FL_ANCHOR_TEXTS, false);
    flAnchorContents = getValue(conf,
        "indexer.anchorContent.field.anchorContents",
        DEFAULT_FL_ANCHOR_CONTENTS, false);
    regexTocAnchor = getValue(conf, "indexer.anchorContent.regex.tocAnchor",
        DEFAULT_REGEX_TOC_ANCHOR, false);
    String str = getValue(conf, "indexer.anchorContent.extractOtherAnchors",
        "false", true);
    if (StringUtils.isNotBlank(str)) {
      extractOtherAnchors = Boolean.parseBoolean(str);
    }
    if (extractOtherAnchors) {
      regexPlainAnchorTag = getValue(conf,
          "indexer.anchorContent.regex.plainAnchorTag",
          DEFAULT_REGEX_PLAIN_ANCHOR_TAG, false);
    }
    regexBodyRoot = getValue(conf, "indexer.anchorContent.regex.bodyRoot",
        DEFAULT_REGEX_BODY_ROOT, true);
  
    regexExtractContent = getValue(conf,
        "indexer.anchorContent.regex.extractContent",
        DEFAULT_REGEX_EXTRACT_CONTENT, false);
    MSG_FORMAT = new MessageFormat(regexExtractContent);
  
    str = conf.get("indexer.anchorContent.maxAnchorLinks");
    if (str != null) {
      maxAnachorLinks = Integer.parseInt(str);
    }
  }

  public String getValue(Configuration conf, String param, String oldValue,
      boolean blankable) {
    String newValue = oldValue;
    if (conf.get(param) != null) {
      newValue = conf.get(param);
    }
    if (!blankable && StringUtils.isBlank(newValue)) {
      throw new IllegalArgumentException(newValue + " is set to empty or null.");
    }
    return newValue;
  }
}

Configuration
We update plugin.includes in nutch-site.xml to include this plugin. In solrindex-mapping.xml, we map field in NutchDocument to field in Solr Document.

<field dest="anchorTags" source="anchorTags" />
<field dest="anchorTexts" source="anchorTexts" />
<field dest="anchorContents" source="anchorContents" />

Resource
Using HTML Parser Jsoup and Regex to Extract Text between Tow Tags
Debugging and Optimizing Regular Expression
Writing Nutch Plugin Example

Using HTML Parser Jsoup and Regular Expression to Get Text between Tow Tags

The Task
In this article, we are going to use jsoup to parse html pages to get all TOC(table of content) anchor links, and use regular expression to get all text content of each anchor link.

The Solution
Jsoup is a java HTML parser, its jquery-like and regex selector syntax makes it very easy to use to extract content form html page. 

Normally a site has some convention about where it puts the TOC anchor link: from this we can compose a css selector to select all anchor link. We will take this Java_Development_Kit wikipedia page as an example.

Use Jsoup to Get All Anchor Links
To try CSS selector, we can open Chrome Developer tools, in the console tab: use document.querySelectorAll("CSS_SELECTOR_HERE"); to test our css selector.

Our final css selector would be:
div[id=toc]>ul>li a[href^='#']:not([href='#'])
in the id=toc div section, get it's direct child ui element, then get direct child li elements, fina all link with href attribute: value of href should be started with #(means this points to an anchor link), but no '#".

The Code
One caveat: Jsoup doesn't like the ' or " around attribute value, the old css selector will cause no match. 
The final css selector for Jsoup is: div[id=toc] ul>li a[href^=#]:not([href=#])

Document doc = Jsoup.connect(url).get();
Element rootElement = doc.select(PATTERN_BODY_ROOT).first();
Set<String> anchors = new LinkedHashSet<String>();
Elements elements = rootElement.select(TOC_ANCHOR);
if (!elements.isEmpty()) {
  for (Element element : elements) {
    String href = element.attr("href");
    anchors.add(href.substring(1));
  }
}

Using Regular Expression and Jsoup to Get Text of each Anchor
First definition of the content of an anchor in our case: it's the all content between the current anchor and the next anchor.

The regular expression to get all html content between the the anchor JDK_contents and the anchor Ambiguity_between_a_JDK_and_an_SDK is like below:
<span[^>]*\s*(?:"|')?JDK_contents(?:'|")?[^>]*>([^<]*)</span>(.*)(<span[^>]*\s*(?:"|')?Ambiguity_between_a_JDK_and_an_SDK(?:'|")?[^>]*>[^<]*</span>.*)

In another post, we will introduce how to use tool RegexBuudy to test and compose this regular expression and improve the regulare expression to boost the performance.

After get the HMTL content, we call Jsoup.parse(html).text(); to get all combined text.

The Code

public String getContentBetweenAnchor(StringBuilder remaining,
    String anchor1, String anchor2, String anchorElement,
    String anchorAttribute) throws IOException {
  StringBuilder sb = new StringBuilder();
  // the first group is the anchor text
  sb.append(matchAnchorRegexStr(anchor1, anchorElement, true))
      // the second group is the text between these 2 anchors
      .append("(.*)")
      // the third group is the remaining text
      .append("(").append(matchAnchorRegexStr(anchor2, anchorElement, false))
      .append(".*)");

  System.out.println(sb);
  Matcher matcher = Pattern.compile(sb.toString(),
      Pattern.DOTALL | Pattern.MULTILINE).matcher(remaining);
  String matchedText = "";
  if (matcher.find()) {
    String anchorText = Jsoup.parse(matcher.group(1)).text();
    matchedText = anchorText + " " + Jsoup.parse(matcher.group(2)).text();
    String newRemaining = matcher.group(3);
    remaining.setLength(0);
    remaining.append(newRemaining);
  }
  return matchedText;
}

The Complete Code

package org.codeexample.lifelongprogrammer.anchorlinks;

import org.apache.commons.lang.StringUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.google.common.base.Stopwatch;

public class JsoupExample {
  private static final String TOC_ANCHOR = "div[id=toc] ul>li a[href^=#]:not([href=#])";
  private static final String PLAIN_ANCHOR_A_TAG = "a[href^=#]:not([href=#])";

  private static final int MAX_ANCHOR_LINKS = 5;
  // only <div id="bodyContent"> section
  private static final String PATTERN_BODY_ROOT = "div[id=bodyContent]";

  public Map<String, String> parseHTML(String url) throws IOException {
    Map<String, String> anchorContents = new LinkedHashMap<String, String>();

    Document doc = Jsoup.connect(url).get();
    Element rootElement = doc.select(PATTERN_BODY_ROOT).first();
    if (rootElement == null)
      return anchorContents;
    Set<String> anchors = getAnchors(rootElement);
    if (anchors.isEmpty())
      return anchorContents;
    StringBuilder remaining = new StringBuilder(rootElement.toString());

    Iterator<String> it = anchors.iterator();
    String current = it.next();
    while (it.hasNext() && remaining.length() > >0) {
      String next = it.next();
      anchorContents
          .put(
              current,
              getContentBetweenAnchorInWiki(remaining, current, next, "span",
                  "id"));
      current = next;
    }
    // last one
    String lastTxt = Jsoup.parse(remaining.toString()).text();
    if (StringUtils.isNotBlank(lastTxt)) {
      anchorContents.put(current, lastTxt);
    }
    return anchorContents;
  }

  public Set<String> getAnchors(Element rootElement) {
    Set<String> anchors = new LinkedHashSet<String>() {
      private static final long serialVersionUID = 1L;

      @Override
      public boolean add(String e) {
        if (size() >= MAX_ANCHOR_LINKS)
          return false;
        return super.add(e);
      }
    };
    getAnchorsImpl(rootElement, TOC_ANCHOR, anchors);
    if (anchors.isEmpty()) {
      // no toc anchor found, then use
      getAnchorsImpl(rootElement, PLAIN_ANCHOR_A_TAG, anchors);
    }
    return anchors;
  }

  public void getAnchorsImpl(Element rootElement, String anchorPattern,
      Set<String> anchors) {
    Elements elements = rootElement.select(anchorPattern);
    if (!elements.isEmpty()) {
      for (Element element : elements) {
        String href = element.attr("href");
        anchors.add(href.substring(1));
      }
    }
  }

  public String getContentBetweenAnchor(StringBuilder remaining,
      String anchor1, String anchor2, String anchorElement,
      String anchorAttribute) throws IOException {
    StringBuilder sb = new StringBuilder();
    // the first group is the anchor text
    sb.append(matchAnchorRegexStr(anchor1, anchorElement, true))
        // the second group is the text between these 2 anchors
        .append("(.*)")
        // the third group is the remaing text
        .append("(").append(matchAnchorRegexStr(anchor2, anchorElement, false))
        .append(".*)");

    System.out.println(sb);
    Matcher matcher = Pattern.compile(sb.toString(),
        Pattern.DOTALL | Pattern.MULTILINE).matcher(remaining);
    String matchedText = "";
    if (matcher.find()) {
      String anchorText = Jsoup.parse(matcher.group(1)).text();
      matchedText = anchorText + " " + Jsoup.parse(matcher.group(2)).text();
      String newRemaining = matcher.group(3);
      remaining.setLength(0);
      remaining.append(newRemaining);
    }
    return matchedText;
  }

  public String matchAnchorRegexStr(String anchor1, String anchorElement,
      boolean cpatureAnchorText) {
    StringBuilder sb = new StringBuilder().append("<").append(anchorElement)
        .append("[^>]*").append("\\s*").append("(?:\"|')?").append(anchor1)
        .append("(?:'|\")?[^>]*>");
    if (cpatureAnchorText) {
      sb.append("([^<]*)");
    } else {
      sb.append("[^<]*");
    }
    return sb.append("</").append(anchorElement).append(">").toString();
  }

  @Test
  public void testWiki() throws IOException {
    Stopwatch stopwatch = Stopwatch.createStarted();
    String url = "http://en.wikipedia.org/wiki/Java_Development_Kit";
    Map<String, String> anchorContents = parseHTML(url);
    System.out.println(anchorContents);
    System.out.println("Took " + stopwatch.elapsed(TimeUnit.MILLISECONDS));
    stopwatch.stop();
  }  
}

Resources
Comparison of HTML parsers
jsoup
CSS Selector Reference

Using Mergeindexes and PowerShell to Automate Deployment of Solr Index to Remote Production Machines

The Problem
We make change to our documentation site periodically, and use Nutch to crawl it and save index to solr server in local build machine and test it. 

On release date, we will deploy the new index into production machines. We want to minimize the downtime, so we can't restart Solr server in production machines.
The Solution
Luckily, Solr provides mergeindexes tool: it doesn't support merge remote indexes, but we can easily use Powershell to copy the new index to production machines, then run mergeindexes locally.

The reason we choose Window PowerShell is because PowerShell supports UNC path like(\\serverA\labelB\pathc), which Window batch doesn't support.
Steps and Script
1. Crawl vendorA doc to core core_vendorA in build machine.
PowerShell Script
This step is optional, as the site should be already crawled and tested before deploy. We include the script here for completeness.
We create a ServerResource in Nutch side to expose http API to start/stop/edit/delete a task to crawl a site and monitor crawl status. Please refer Nutch2: Extend Nutch2 to Crawl via Http API

$data = '{\"solrURL\":\"http://solrServerInbuildMachine/solr/vendorA/\",\"crawlID\":\"crawl_vendorA_ID1\",
\"taskName\":\"taskl_vendorA_ID1\",\"crawlDepth\":2,\"urls\":[\"http://docsite:port/rootpath/\"],
\"includePaths\":[\"+^(?i)http://docsite:port/rootpath/\"],\"subCollections\":[{\"name\":\"vendorA\",
\"id\":\"vendorA\",\"whiteList\":[\"http\",\"cifs\",\"file\",\"ftp\"]},{\"name\":\"vendorA\",
\"id\":\"vendorA\",\"whiteList\":[\"http\",\"cifs\",\"file\",\"ftp\"]}],\"solrindexParams\":\"update.chain=webCrawlerChain\",
\"delOldDataQuery\":\"subcollection:vendorA\",\"sync\":true,\"deleteIfExist\":true,\"updateDirectly\":false,
\"tmpCoreName\":\"core-tmp1\",\"cleanData\":true,\"startTask\":true,\"reuseIfExist\":false,
\"fileToFileMappings\":{\"conf/nutch-site.xml\":\"conf/predefinedTasks/nutch-site-templateA.xml\"}} '

&curl -X PUT -H "Content-Type: application/json" -d $data http://nutchServer:port/nutch/cvcrawler

2. After crawl is finished, copy and zip the index folder solr\data\core_vendorA\index to production machines, folder: %PREFIX%\new-index\core_vendorA\index, and unzip it.
PowerShell Script

if (-not (test-path "$env:ProgramFiles\7-Zip\7z.exe")) {throw "$env:ProgramFiles\7-Zip\7z.exe needed"} 
set-alias sz "$env:ProgramFiles\7-Zip\7z.exe" 
cd %LOCALBUILPATH%solr\data\core_vendorA\index
&sz a -tzip index.zip  -mx3

copy-item index.zip \\prodMachineA\new-index\vendorA\index
cd \\prodMachineA\new-index\vendorA\index
&sz x index.zip

Do the following steps for each production machine
3. Delete old index for vendor core_vendorA by sending a solr request:
curl "http://localhost:8080/solr/core_vendorA/update?commit=true&stream.body=<delete><query>*:*</query></delete>"

4. Merge index from new-index\core_vendorA\index to core core_vendorA by sending a solr request:
curl "http://localhost:8080/solr/admin/cores?action=mergeindexes&core=core_vendorA&indexDir=%PREFIX%\new-index\core_vendorA\index"

5. Commit the merged index by sending a commit request to solr: 
curl "http://localhost:8080/solr/core_vendorA/update?commit=true"

Step3,4,5 is pretty fast, usually take less than 1 minute.

Resources
Nutch2: Extend Nutch2 to Crawl via Http API