Solr Data Schema Migration Practice

The Problem
When we design data schema, we should choose field name and type carefully. As it takes much more effort to change the schema, migrate old data without compared with changing code.
If we store data in Solr, we can explicitly store searchable fields as separate fields, store all other fields as a json string.

But sometimes we still have to change the schema: add fields or change field type.  

API to Getting the current Solr Schema Version
Solr stores current schema version like below: the version is a double number - not a string: means you can't store something like 1.6.0. 

The following code uses Spring Solr Data SolrSchemaRequest to get the version - you can also use solrj to do the same thing.

public double getVersion() {
    double version = -1;
    try {
        final NamedList<Object> nl = solrServer.request(SolrSchemaRequest.version(), getCollection());
        final Object object = nl.get("json");
        if (object != null) {
            version = MoreObjects.firstNonNull(createFailSafeObjectmapper()
                    .readValue(object.toString(), SchemaDefinition.class).getVersion(), -1d);
        }
    } catch (final Exception e) {
        LOGGER.error(MessageFormat.format("unable to get version for collection: {0}", getCollection()), e);
    }
    return version;
}

Store the version in the data
When we save data to Solr, call: entity.setObjectVersion(getVersion());

API/Script to upgrade data, check all versions
For example, recently we move some Solr fields that are not searched to a JSON body wich maps to a Java class - XXDetail.
- So when we need to add not-searched fields, we don;t have to update Solr schema - just put it into fields in XXDetail.

So we can write either java code or scripts to migrate the old version data to the new version schema.

Change field Type
Practice - Change Long to Date - Not Searchable
Previously we store the date(field:updateDate) as tlong in Solr, and we want to change it to date - As it will make the rest API more readable, easier to read the data and query Solr.

For Solr itself: after change type tlong to date, Solr can still read and query the old data.

If this field is not searched, and you already have Spring's LongToDateConverter, DateToLongConverter. Then you can just change its type from tlong to date.

    @ReadingConverter
    @WritingConverter
    public enum LongToDateConverter implements Converter {
        INSTANCE;

        @Override
        public Date convert(final Long source) {
            if (source == null) {
                return null;
            }
            return new Date(source);
        }
    }
    @ReadingConverter
    @WritingConverter
    public enum DateToLongConverter implements Converter {
        INSTANCE;
        @Override
        public Long convert(final Date source) {
            if (source == null) {
                return null;
            }
            return source.getTime();
        }
    }

Old code(using long) - new schema
When to save data, LongToDateConverter will convert long to date object automatically; when read data, DateToLongConverter will convert date and return long.
New code(using date) - old Schema
When to save data, DateToLongConverter will convert date object to long; when read data, LongToDateConverter will convert long and return Date.

Practice - Change Long to Date - Searchable
But if we search the field in code, then we can't just simply change the type.
Old code(using long) - new schema
When search: the query will be like:
field:[along to blong], but the field is already changed to date type, so the query will fail:
Invalid Date String:'2'
at org.apache.solr.schema.TrieDateField.parseMath(TrieDateField.java:150)
at org.apache.solr.schema.TrieField.getRangeQuery(TrieField.java:369)
at org.apache.solr.parser.SolrQueryParserBase.getRangeQuery(SolrQueryParserBase.java:761)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:382)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
New code(using date) - old Schema
When search in new code, the query will be like:
field:["yyyy-MM-dd:Thh:mm:ssZ" TO "yyyy-MM-dd:Thh:mm:ssZ"], but the field is still long in schema: so it will fail:
java.lang.NumberFormatException: For input string: "2016-02-23T20:08:15.208Z"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.solr.schema.TrieField.getRangeQuery(TrieField.java:343)
at org.apache.solr.parser.SolrQueryParserBase.getRangeQuery(SolrQueryParserBase.java:761)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:382)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)

The Solution
Now we will have to add a new field: updateTime - type date.
1. Update Schema - add fields, but not update or delete fields
-- Make sure old code works with new schema
The schema would contain both fields updateDate(tlong) and updateTime(date), the new code will store and query field updateTime(date).

2. Update to new code
-- Make sure new code work with old data

3. Migrate old version data to new version After we migrate all old version code to the new version, we will run the migrate script which will copy the old updateDate(tlong) to 
new field updateTime(date). Later we can get rid of the old field.

4. Remove code that handles the old version and old fields in next release

References
Database Migrations Done Right