Solr - Create custom data transformer to remove fields

Overview
Create custom data transformer to remove fields and remove field from json data in Solr.

The Problem
We store campaign message in Solr. One type of campaign is voucher. We return this user's voucher and other data based on user's accountId.

To support this, we add one searchable field: accountIds which includes all accountIds for this campaign. Add another field: details field which is a json string (mapping to java class) and non-searchable. It includes vouchers properties - a mapping from accountId to voucherCode.
-- We choose this approach to be consistent with existing data and make server code simpler.

accountIds and details.vouchers fields are big, and when return to client, actually we only need c this user's voucherCode.

The Solution
Excluding one field
We can build only fl to only include all fields except accountIds.  - This is kind of cumbersome, and every time we add a new field, we have to change the fl in SolrQuery.

[SOLR-3191] field exclusion from fl is promising, but it's not merged into Solr release.

So we create a data transformer which supports the following params:
removeFields - what fields to remove
Example: removeFields=accountIds,field1,field2

removeOthersVoucher - enable the feature if it's true
If removeOthersVoucher is true:
If accountId is empty, then remove all voucherCodes from details field.
If accountId is not empty, then remove all voucherCodes except accountId's voucher.

How to uses it
fl=*,[removeFeilds]&removeFields=accountIds&removeOthersVoucher=true&accountId=account1

Writing Custom Data Transformer
We use Jackson ObjectMapper to deserialize details field from String to  Map<String, Object>.

public class  MyTransformerFactory extends TransformerFactory {
    protected static Logger logger = LoggerFactory.getLogger(MyTransformerFactory.class);
    private boolean enabled = false;

    @Override
    public void init(@SuppressWarnings("rawtypes") final NamedList args) {
        try {
            super.init(args);
            if (args != null) {
                final SolrParams params = SolrParams.toSolrParams(args);
                enabled = SolrParams.toSolrParams(args).getBool("enabled", true);
            }
        } catch (final Exception e) {
            logger.error("MyTransformerFactory init failed", e);
        }
    }

    @Override
    public DocTransformer create(final String field, final SolrParams params, final SolrQueryRequest req) {
        final SolrParams reqParams = req.getParams();
        final String removeFields = reqParams.get("removeFields");
        final boolean removeOthersVoucher = reqParams.getBool("removeOthersVoucher", false);
        final String accountId = reqParams.get("accountId");
        if (!enabled || (removeFields == null && !removeOthersVoucher)) {
            return null;
        }
        return new  MyTransformer(removeFields, removeOthersVoucher, accountId);
    }

    private static class  MyTransformer extends DocTransformer {
        private static final String FIELD_DETAILS = "details";
        private static final String DETAIL_VOUCHER_CODES = "voucherCodes";

        private static ObjectMapper objectMapper = new ObjectMapper();
        private static Splitter splitter = Splitter.on(",").trimResults();

        private final String removeFields;
        private final boolean removeOthersVoucher;
        private final String accountId;

        public  MyTransformer(final String removeFields, final boolean removeOthersVoucher, final String accountId) {
            this.removeFields = removeFields;
            this.removeOthersVoucher = removeOthersVoucher;
            this.accountId = accountId;
        }
        @Override
        public String getName() {
            return  MyTransformer.class.getSimpleName();
        }

        @Override
        public void transform(final SolrDocument doc, final int docid) throws IOException {
            if (removeFields != null) {
                final Iterable<String> it = splitter.split(removeFields);
                for (final String removeField : it) {
                    doc.removeFields(removeField);
                }
            }
            try {
                if (removeOthersVoucher) {
                    removeOthersVoucher(doc);
                }
            } catch (final Exception e) {
                // ignore it if there is exception
                logger.error("MyTransformer transform failed", e);
            }
        }

        protected void removeOthersVoucher(final SolrDocument doc)
                throws IOException, JsonParseException, JsonMappingException, JsonProcessingException {
            final String detailsObj = getFieldValue(doc, FIELD_DETAILS);
            if (detailsObj == null) {
                return;
            }

            final Map<String, Object> detailsMap = objectMapper.readValue(detailsObj.toString(),
                    TypeFactory.defaultInstance().constructMapType(Map.class, String.class, Object.class));
            if (detailsMap == null) {
                return;
            }
            final Object voucherCodesObj = detailsMap.get(DETAIL_VOUCHER_CODES);
            if (!(voucherCodesObj instanceof HashMap)) {
                return;
            }
            final Map<String, String> voucherCodesMap = (Map<String, String>) voucherCodesObj;
            final String voucherCode = voucherCodesMap.get(accountId);

            final Map<String, String> myVoucherMap = new HashMap<String, String>();
            if (voucherCode != null) {
                myVoucherMap.put(accountId, voucherCode);
            }
            detailsMap.put(DETAIL_VOUCHER_CODES, myVoucherMap);

            doc.setField(FIELD_DETAILS, objectMapper.writeValueAsString(detailsMap));
        }
    }

    public static String getFieldValue(final SolrDocument doc, final String field) {
        final List<String> rst = new ArrayList<String>();
        final Object obj = doc.get(field);
        getFieldvalues(doc, rst, obj);

        if (rst.isEmpty()) {
            return null;
        }
        return rst.get(0);
    }

    public static void getFieldvalues(final SolrDocument doc, final List<String> rst, final Object obj) {
        if (obj == null) {
            return;
        }
        if (obj instanceof org.apache.lucene.document.Field) {
            final org.apache.lucene.document.Field field = (Field) obj;
            final String oldValue = field.stringValue();
            if (oldValue != null) {
                rst.add(oldValue);
            }
        } else if (obj instanceof IndexableField) {
            final IndexableField field = (IndexableField) obj;
            final String oldValue = field.stringValue();
            if (oldValue != null) {
                rst.add(oldValue);
            }
        } else if (obj instanceof Collection) {
            final Collection colls = (Collection) obj;
            for (final Object newObj : colls) {
                getFieldvalues(doc, rst, newObj);
            }
        } else {
            logger.error(MessageFormat.format("type: {0}", obj.getClass()));
            rst.add(obj.toString());
        }
    }
}

Add Transformer into solrConfig.xml

<lib dir="../../../lib" regex="lifelongprogrammer-solr-extension-jar-with-dependencies.jar" />

<transformer name="removeFeilds" class="com.lifelongprogrammer.solr.MyTransformerFactory">
  <bool name="enabled">true</bool>
</transformer>

pom.xml - Build solr-extension jar
We declare scope of solr-core as provided and use maven-assembly-plugin to build jar-with-dependencies.

<build>
  <finalName>lifelongprogrammer-solr-extension</finalName>
  <plugins>
    <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>2.6</version>
      <configuration>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
      <executions>
        <execution>
          <id>make-assembly</id>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

<dependencies>
  <dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-core</artifactId>
    <version>5.2.0</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.7.4</version>
  </dependency>
</dependencies>