Text Mining: Integrate UIMA Regular Expression Annotator with Solr

UIMA RegexAnnotator
UIMA RegexAnnotator is an Apache UIMA analysis engine that uses regular expression to detect entities such as email addresses, URLs, phone numbers, zip codes or any other entity.

This article will introduce how to deploy RegexAnnotator as SOAP web service, add extra regex to extract other types of entities and integrate it with Solr.
Deploy RegexAnnotator as SOAP Web Service
For detailed steps, please refer to this post.
First copy all jars in %uima-addons-home%\addons\annotator\RegularExpressionAnnotator\lib\ to axis.war\WEB-INF\lib, copy RegularExpressionAnnotator\desc\concepts.xml to axis.war\WEB-INF\classes.

Then we need create web services deployment descriptor. Example WSDD files are provided in the examples/deploy/soap directory of the UIMA SDK. All we need do is to copy one wsdd(for example:Deploy_NamesAndPersonTitles.wsdd): change service name to urn:RegExAnnotator, change resourceSpecifierPath to point to %PEARS_HOME_REPLACE_THIS%\addons/annotator/RegularExpressionAnnotator/desc/RegExAnnotator.xml.

<deployment name="RegExAnnotator">
 <service name="urn:RegExAnnotator" provider="java:RPC">
  <parameter name="resourceSpecifierPath" value="%PEARS_HOME_REPLACE_THIS%\addons/annotator/RegularExpressionAnnotator/desc/RegExAnnotator.xml"/>
 </service>
</deployment>

If we are using tomcat, set CATALINA_HOME to the location where Tomcat is installed. if we are using other application server, we may update UIMA_CLASSPATH in runUimaClass.bat to include axis\WEB-INF\lib, axis\WEB-INF\classes.

Then run deploytool %FOLDER%\RegExAnnotator.wsdd to deploy the pear as SOAP service.
Test RegexAnnotator SOAP Service in CVD
Check How to Call a UIMA Service for detail.

We need define one SOAP Service Client Descriptor: RegExAnnotatorSoapServiceClient.xml

<uriSpecifier xmlns="http://uima.apache.org/resourceSpecifier">
 <resourceType>AnalysisEngine</resourceType>
 <uri>http://localhost:8080/axis/services/urn:RegExAnnotator</uri>
 <protocol>SOAP</protocol>
</uriSpecifier>

Then in CVD, click "Run" -> "Load AE" to load RegExAnnotatorSoapServiceClient.xml, then test it.
Adding RegEx to extract other types of entities
Regular expression can be used to extract many types of enties. 
We can use regex from this post: \(?([0-9]{3})\)?[-. ]?([0-9]{3})[-. ]?([0-9]{4}) to extract North American Phone Numbers.

To add this feature to UIMA, we create one ExtraRegExAnnotator.xml - similar as RegExAnnotator.xml except using different concept xml file and type definition:

<analysisEngineDescription xmlns="http://uima.apache.org/resourceSpecifier">
  <frameworkImplementation>org.apache.uima.java</frameworkImplementation>
  <primitive>true</primitive>
  <annotatorImplementationName>org.apache.uima.annotator.regex.impl.RegExAnnotator</annotatorImplementationName>
  <analysisEngineMetaData>
    <name>ExtraRegExAnnotator</name>
    <!--configurationParameters omitted here -->
    <configurationParameterSettings>
      <nameValuePair>
        <name>ConceptFiles</name>
        <value><array><string>extra-concepts.xml</string></array></value>
      </nameValuePair>      
    </configurationParameterSettings>
    <typeSystemDescription>
      <types>
        <typeDescription>
          <name>org.lifelongprogrammer.USAPhoneNumber</name>
          <description/>
          <supertypeName>uima.tcas.Annotation</supertypeName>
          <features>
            <featureDescription>
              <name>confidence</name>
              <description/>
              <rangeTypeName>uima.cas.Float</rangeTypeName>
            </featureDescription>            
          </features>
        </typeDescription>
      </types>
    </typeSystemDescription>
    <capabilities>
      <capability>
        <inputs/>
        <outputs>
          <type>org.lifelongprogrammer.USAPhoneNumber</type>
        </outputs>
        <languagesSupported/>
      </capability>
    </capabilities>
    <!-- operationalProperties omited here -->
  </analysisEngineMetaData>
</analysisEngineDescription>

extra-concepts.xml: - and copy it to axis.war\WEB-INF\classes

<conceptSet xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns="http://incubator.apache.org/uima/regex"
 xsi:schemaLocation="concept.xsd">
  <concept name="usaPhoneNumberDetection">
    <rules>
      <rule regEx="\(?([0-9]{3})\)?[-. ]?([0-9]{3})[-. ]?([0-9]{4})"
    matchStrategy="matchAll" matchType="uima.tcas.DocumentAnnotation"
    confidence="1.0" />
    </rules>
    <createAnnotations>
      <annotation id="usaPhoneNumber"
    type="org.lifelongprogrammer.USAPhoneNumber">
        <begin group="0" />
        <end group="0" />
        <setFeature name="confidence" type="Confidence" />
      </annotation>
    </createAnnotations>
  </concept>  
</conceptSet>

The web services deployment descriptor: Deploy_ExtraRegExAnnotator.wsdd: similar as Deploy_RegExAnnotator.wsdd.

The SOAP Service Client Descriptor: ExtraRegExAnnotatorSoapServiceClient.xml(similar as RegExAnnotatorSoapServiceClient.xml). Load it to CVD, and test it.
Integrate UIMA RegexAnnotator with Solr
Solr uses UIMAUpdateRequestProcessorFactory to send the text to SOAP web service and parse the soap response when add a document to Solr, UIMAUpdateRequestProcessorFactory will save the UIMA extracted information into Solr.


In order to call SOAP web service, we first need put the SOAP Service Client Descriptor: RegExAnnotatorSoapServiceClient.xml and ExtraRegExAnnotatorSoapServiceClient.xml in solr/collection1/conf folder.

Then wrap the two SOAP services urn:RegExAnnotator and urn:ExtraRegExAnnotator in an aggregate analysis engine.

<analysisEngineDescription xmlns="http://uima.apache.org/resourceSpecifier">
  <frameworkImplementation>org.apache.uima.java</frameworkImplementation>
  <primitive>false</primitive>
  <delegateAnalysisEngineSpecifiers>
    <delegateAnalysisEngine key="RegExAnnotatorService">
      <import location="RegExAnnotatorSoapServiceClient.xml"/>
    </delegateAnalysisEngine>   
    <delegateAnalysisEngine key="ExtraRegExAnnotatorService">
      <import location="ExtraRegExAnnotatorSoapServiceClient.xml"/>
    </delegateAnalysisEngine>  
  </delegateAnalysisEngineSpecifiers>
  <analysisEngineMetaData>
    <name>AllRegExAnnotatorService</name>
    <description/>
    <version>1.0</version>
    <vendor/>
    <configurationParameters searchStrategy="language_fallback">
    </configurationParameters>
    <flowConstraints>
      <fixedFlow>
        <node>RegExAnnotatorService</node>
        <node>ExtraRegExAnnotatorService</node>
      </fixedFlow>
    </flowConstraints>
    <fsIndexCollection/>
    <!-- capabilities omitted -->
    <!-- operationalProperties omitted -->
  </analysisEngineMetaData>
  <resourceManagerConfiguration/>
</analysisEngineDescription>

Then define update chain: uima-regex in solrconfig.xml:

<updateRequestProcessorChain name="uima-regex" default="true">
    <processor class="org.apache.solr.uima.processor.UIMAUpdateRequestProcessorFactory">
      <lst name="uimaConfig">
        <lst name="runtimeParameters">
        </lst>javax.xml.rpc.ServiceException
        <str name="analysisEngine">file:///%REPLACE_THIS%\solr\collection1\conf\RegExAnnotatorAE.xml</str>
        <bool name="ignoreErrors">false</bool>
        <lst name="analyzeFields">
          <bool name="merge">false</bool>
          <arr name="fields">
            <str>content</str>
          </arr>
        </lst>
        <lst name="fieldMappings">
        <lst name="type">
            <str name="name">org.lifelongprogrammer.USAPhoneNumber</str>
            <lst name="mapping">
              <str name="feature">coveredText</str>
              <str name="field">usaphone_mxf</str>
            </lst>
          </lst>
          <lst name="type">
            <str name="name">org.apache.uima.EmailAddress</str>
            <lst name="mapping">
              <str name="feature">normalizedEmail</str>
              <str name="field">email_mxf</str>
            </lst>
          </lst>          
          <lst name="type">
            <str name="name">org.apache.uima.ISBNNumber</str>
            <lst name="mapping">
              <str name="feature">coveredText</str>
              <str name="field">isbn_mxf</str>
            </lst>
          </lst>

          <lst name="type">
            <str name="name">org.apache.uima.MoneyAmount</str>
            <lst name="mapping">
              <str name="feature">coveredText</str>
              <str name="field">money_mxf</str>
            </lst>
          </lst>
          <lst name="type">
            <str name="name">org.apache.uima.CreditCardNumber</str>
            <lst name="mapping">
              <str name="feature">coveredText</str>
              <str name="field">creditcard_mxf</str>
            </lst>
            <lst name="mapping">
              <str name="feature">cardType</str>
              <str name="field">creditcardType_mxf</str>
            </lst>
          </lst>
        </lst>
      </lst>
    </processor>
    <processor class="solr.LogUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
  </updateRequestProcessorChain>

After that we can call 
http://localhost:8080/solr/update?update.chain=uima-regex&commit=true&stream.body=<add><doc><field name="id">1</field><field name="content">some text here</field></doc></add>
Then run http://localhost:8080/solr/select?q=id:1, we can see it extracts entities like phone number, email, credit card, isbn etc.
Resources
Text Mining: Integrate OpenNLP, UIMA and Solr via SOAP Web Service