Building Troubleshooting Friendly Application - Using Feature Toggle


In last post, I talked about how to change log level and add diagnosis information per request at runtime. It helps us to debug and trouble shooting problems in production environment.

But at that time, we use X-DEBUG-KEY header,  check whether it matches the key at server side, if so enable this feature. 

This is unsafe(as we may return internal implementation information to client). - even we encrypt the X-DEBUG-KEY in the source code and change it before push to production.

Recently, we developed a common feature toggle service. So now we can use the feature toggle service to control who can use the debug request feature and turn it on/off dynamically at server side.
FeatureService
Admin can create/update/delete feature. The feature can be stored in database(sql or nosql).
Client will call isFeatureOn with feature name and a map. isFeatureOn will get the feature from db and check whether it's on. It will return Decision.NEUTRAL if not exist.
public interface FeatureService {
    Decision isFeatureOn(String name, Map<String, Object> variables);
    public Optional<Feature> getFeature(final String name);
    public void addFeature(Feature feature);
    public void updateFeature(Feature feature);
    void deleteFeature(Iterator<String> iterator);
    Collection<Feature> getAllFeatures();
}
Feature and Expression
Feature contains a list of expression, and will call the expression's apply method to check whether the feature should be on or off. 


Currently we only support AnyMatchExpression, PercentageExpression and ScriptExpression, but it can be easily extended.

Notice we can use the optional ttlSeconds to control how long the feature will be on, after that it will be expired. This adds another safety layer to the feature toggle service.

-- We annotate updateTime with View.Viewable.class, so outside API can only view it but can't update it: check Using Jackson JSON View to Protect Mass Assignment Vulnerabilities

public class Feature {
    private String name;
    private String appId;
    /**
     * This is used as extension, in case later we support rejectFirst.
     */
    private boolean acceptFirst = true;
    public Feature() {}
    @JsonView(View.Editable.class)
    private List<Expression> expressionList = new ArrayList<>();
    @JsonView(View.Editable.class)
    protected int ttlSeconds = -1;

    @JsonView(View.Viewable.class) // all other properties are marked as Editable, except updateTime
    private Date updateTime;

    @JsonIgnore
    public boolean isEffective() {
        if (ttlSeconds != -1) {
            if (updateTime == null) {
                logger.error("Invalid feature {}, updateTime is null while timeToLive is {}", name, ttlSeconds);
                throw new XXException(ErrorCode.INTERNAL_ERROR,
                        String.format("Invalid feature %s: ", getName()));
            }
            final MutableDateTime expiredTime = new MutableDateTime(updateTime);
            expiredTime.addSeconds(ttlSeconds);

            return expiredTime.isAfterNow();
        }
        return true;
    }
    public Decision isFeatureOn(final Map<String, String> variables) {
        if (hasExpired()) {
            return Decision.NEUTRAL;
        }
        final List<Expression> expressionList = getExpressionList();
        for (final Expression expression : expressionList) {
            if (expression.apply(variables).equals(Decision.ACCEPT)) {
                return Decision.ACCEPT;
            }
        }
        return Decision.NEUTRAL;
    }
    // ignore equals, hashcode, toString, getter, setter..    
}    

@JsonIgnoreProperties(ignoreUnknown = true)
@org.codehaus.jackson.annotate.JsonIgnoreProperties(ignoreUnknown = true)
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY, property = AbstractExpression.FIELD_TYPE,
        visible = false)
@JsonSubTypes({@Type(value = PercentageExpression.class, name = PercentageExpression.TYPE),
        @Type(value = AnyMatchExpression.class, name = AnyMatchExpression.TYPE),
        @Type(value = ScriptExpression.class, name = ScriptExpression.TYPE)})
public interface Expression {
    public Decision apply(Map<String, Object> variables);
}

public class AnyMatchExpression extends AbstractExpression {
    public static final String TYPE = "anyMatch";
    /**
     * which variable name to look up
     */
    private String variableName;

    private boolean caseInsensitive;
    @JsonProperty("set")
    private Set<String> set = new HashSet<>();

    public AnyMatchExpression() {
        type = TYPE;
    }
    @Override
    public Decision apply(final Map<String, Object> variables) {
        final Object object = variables.get(variableName);
        if (object != null) {
            final Iterable<String> idIt = SomeUtil.COMMA_SPLITTER.split(object.toString());
            for (String id : idIt) {
                if (caseInsensitive) {
                    id = id.toLowerCase();
                }
                if (set.contains(id)) {
                    return Decision.ACCEPT;
                }
            }
        }
        return Decision.NEUTRAL;
    }
    /**
     * Precondition: There is no null value in the set. Otherwise it will throw NPE.
     */
    public void setSet(final HashSet<String> set) {
        if (caseInsensitive) {
            final HashSet<String> lowercaseSet = new HashSet<>();
            for (final String word : set) {
                lowercaseSet.add(word.toLowerCase());
            }

            this.set = lowercaseSet;
        } else {
            this.set = set;
        }
    }
    // ignore toString, getter, setter..
}
public class PercentageExpression extends AbstractExpression {
    public static final String TYPE = "percentage";
    /**
     * a value between 0 and 100
     */
    private int percentage;
    public PercentageExpression() {
        type = TYPE;
    }
    @Override
    public Decision apply(final Map<String, Object> variables) {
        if (percentage == 100) {
            return Decision.ACCEPT;
        }
        final Random random = new Random();
        final int rValue = random.nextInt(100);
        if (rValue < percentage) {
            return Decision.ACCEPT;
        }
        return Decision.REJECT;
    }
}
Check Whether Debug Request Feature is On
Now we can check whether the debug request feature is turned on for this user. - we only do this when X_DEBUG_REQUEST header exists. - Of course we cache the Feature object, so we don't do db call every time.
protected void checkDebugRequestFeature(final ContainerRequest request) {
    if (debugRequestEnabled) {
        final String debugRequest = request.getHeaderValue(X_DEBUG_REQUEST);
        if (StringUtils.isNotBlank(debugRequest)) {
            final User user = getUserObjectHere();

            if (SomeUtil.isFeatureOn(SomeUtil.FEATURE_DEBUG_REQUEST, user, featureService)) {
                MDC.put(X_DEBUG_REQUEST, debugRequest);
                RequestContextUtil.getRequestContext().setEnableDebug(Boolean.valueOf(debugRequest));
            } else {
                RequestContextUtil.getRequestContext().setEnableDebug(false);
            }
        }
    }
}
Other User Cases Using Feature Toggle
Feature Toggle is a quite simple but powerful service. With it, we can do A/B testing, we can test and trouble shooting on productions: such as we can allow some specific users to debug request, to mock user etc.

Improve Apache Commons ArrayUtils removeElements to O(n)


Recently, I read apache commons ArrayUtils removeElements source code and found that its implementation is not efficient when try to get all indexes that needed to be removed.

It's O(m*n): n is the length of origin array, m is the unique count of toBeRemoved.
    final BitSet toRemove = new BitSet();
    for (final Map.Entry<Byte, MutableInt> e : occurrences.entrySet()) {
        final Byte v = e.getKey();
        int found = 0;
        for (int i = 0, ct = e.getValue().intValue(); i < ct; i++) {
            found = indexOf(array, v.byteValue(), found);
            if (found < 0) {
                break;
            }
            toRemove.set(found++);
        }
    }

We can just scan the origin array once to get all indexes that needed to be removed.
    final BitSet toRemove = new BitSet();

    // the only change here, time complexity changed to O(n)
    for (int i = 0; i < array.length; i++) {
        final int key = array[i];

        final MutableInt count = occurrences.get(key);
        if (count != null) {
            count.decrement();
            if (count.intValue() == 0) {
                occurrences.remove(key);
            }
            toRemove.set(i);
        }
    }

I created one JIRA: Improve ArrayUtils removeElements time complexity to O(n)
The complete code
The implementation is quite elegant and robust.
 
    public static int[] removeElements(final int[] array, final int... values) {
        if (isEmpty(array) || isEmpty(values)) {
            return clone(array); // can't just return origin array
        }
        final HashMap<Integer, MutableInt> occurrences = new HashMap<Integer, MutableInt>(values.length);
        for (final int v : values) {
            final Integer boxed = Integer.valueOf(v);
            final MutableInt count = occurrences.get(boxed);
            if (count == null) {
                occurrences.put(boxed, new MutableInt(1));
            } else {
                count.increment();
            }
        }
        final BitSet toRemove = new BitSet();

        // My change here, time complexity changed to O(n)
        for (int i = 0; i < array.length; i++) {
            final int key = array[i];

            final MutableInt count = occurrences.get(key);
            if (count != null) {
                count.decrement();
                if (count.intValue() == 0) {
                    occurrences.remove(key);
                }
                toRemove.set(i);
            }
        }

        return (int[]) removeAll(array, toRemove);
    }

    static Object removeAll(final Object array, final BitSet indices) {
        final int srcLength = ArrayUtils.getLength(array);
        // No need to check maxIndex here, because method only currently called from removeElements()
        // which guarantee to generate on;y valid bit entries.
//        final int maxIndex = indices.length();
//        if (maxIndex > srcLength) { 
//            throw new IndexOutOfBoundsException("Index: " + (maxIndex-1) + ", Length: " + srcLength);
//        }
        final int removals = indices.cardinality(); // true bits are items to remove
        final Object result = Array.newInstance(array.getClass().getComponentType(), srcLength - removals);
        int srcIndex=0;
        int destIndex=0;
        int count;
        int set;
        while((set = indices.nextSetBit(srcIndex)) != -1){
            count = set - srcIndex;
            if (count > 0) {
                System.arraycopy(array, srcIndex, result, destIndex, count);
                destIndex += count;
            }
            srcIndex = indices.nextClearBit(set);
        }
        count = srcLength - srcIndex;
        if (count > 0) {
            System.arraycopy(array, srcIndex, result, destIndex, count);            
        }
        return result;
    } 

Find the element that Appears Once - Using BitSet


We can use different approaches:
1. Use two hashsets(as we don't need the real count): uniqueSet, duplicateSet
The first time we see the element, we put it into both sets, if we see it again, we remove it from uniqueSet.
2. We can use BitSet to reduce space - uniqueBitSet and duplicateBitSet
The first time we see the element, we put set its bit in uniqueBitSet and duplicateBitSet, if we see it again, we clear its bit in duplicateBitSet.
    // Using Bitset
    public static int findUnqiueNumber1(final int[] array) {
        final BitSet unique = new BitSet(), duplicate = new BitSet();
        for (final int number : array) {
            // first time see this number
            if (!duplicate.get(number)) {
                unique.set(number);
                duplicate.set(number);
            } else {
                // if see it again
                unique.clear(number);
            }
        }

        final int unqinueValue = unique.nextSetBit(0);

        if (unqinueValue == -1) {
            throw new NoUniqueException("No unique number");
        }
        final int nextUnqiueValue = unique.nextSetBit(unqinueValue + 1);
        if (nextUnqiueValue != -1) {
            throw new MoreThanOneUniqueException(String.format(
                    "More than 1 unique number found. We found at least %s, %s", unqinueValue, nextUnqiueValue));
        }

        return unqinueValue;
    }

    // Using HashSet
    public static int findUnqiueNumber2(final int[] array) {
        final Set<Integer> uniqueSet = new HashSet<>(), duplicateSet = new HashSet<>();
        for (final int number : array) {
            // first time see this number
            if (!duplicateSet.contains(number)) {
                uniqueSet.add(number);
                duplicateSet.add(number);
            } else {
                // if see it again
                uniqueSet.remove(number);
            }
        }

        if (uniqueSet.isEmpty()) {
            throw new NoUniqueException("No unique number");
        }
        if (uniqueSet.size() > 1) {
            throw new MoreThanOneUniqueException(
                    String.format("More than 1 unique number found. We found %s count", uniqueSet.size()));
        }
        final int unqinueValue = uniqueSet.iterator().next();
        return unqinueValue;
    }


    private static class MoreThanOneUniqueException extends RuntimeException {
        private static final long serialVersionUID = 1L;

        public MoreThanOneUniqueException(final String messge) {
            super(messge);
        }
    }
    private static class NoUniqueException extends RuntimeException {
        private static final long serialVersionUID = 1L;

        public NoUniqueException(final String messge) {
            super(messge);
        }
    }

    @Rule
    public ExpectedException thrown = ExpectedException.none();

    @Test
    public void test() {
        Assert.assertEquals(findUnqiueNumber2(new int[] {1}), 1);
        Assert.assertEquals(findUnqiueNumber2(new int[] {1, 2, 2}), 1);
        Assert.assertEquals(findUnqiueNumber2(new int[] {1, 2, 2}), 1);
    }

    @Test
    public void testNoUnqiue() {
        thrown.expect(NoUniqueException.class);
        findUnqiueNumber2(new int[] {});
    }

    @Test
    public void testNoUnqiue2() {
        thrown.expect(NoUniqueException.class);
        findUnqiueNumber2(new int[] {1, 1, 2, 2});
    }

    @Test
    public void testMoreThanONeUnqiue() {
        thrown.expect(MoreThanOneUniqueException.class);
        findUnqiueNumber2(new int[] {1, 2});
    } 
Related:
[CareerCup] 10.4 Find All Duplicates Elements
You have an array with all the numbers from 1 to N, where N is at most 32,000. The array may have duplicate entries and you do not know what N is. With only 4 kilobytes of memory available, how would you print all duplicate elements in the array?

It can also be solved by using BitSet. add i into bitset, if i happens again, print it out.
Bonus:
org.apache.commons.lang3.ArrayUtils.removeElements(int[], int...)
    public static int[] removeElements(final int[] array, final int... values) {
        if (isEmpty(array) || isEmpty(values)) {
            return clone(array);
        }
        final HashMap<Integer, MutableInt> occurrences = new HashMap<Integer, MutableInt>(values.length); // use MutableInt to avoid Integer creation again and again.
        for (final int v : values) {
            final Integer boxed = Integer.valueOf(v);
            final MutableInt count = occurrences.get(boxed);
            if (count == null) {
                occurrences.put(boxed, new MutableInt(1));
            } else {
                count.increment();
            }
        }
        final BitSet toRemove = new BitSet();
        for (final Map.Entry<Integer, MutableInt> e : occurrences.entrySet()) {
            final Integer v = e.getKey();
            int found = 0;
            for (int i = 0, ct = e.getValue().intValue(); i < ct; i++) {
                found = indexOf(array, v.intValue(), found);
                if (found < 0) {
                    break;
                }
                toRemove.set(found++);
            }
        }
        return (int[]) removeAll(array, toRemove);
    }
    static Object removeAll(final Object array, final BitSet indices) {
        final int srcLength = ArrayUtils.getLength(array);
        // No need to check maxIndex here, because method only currently called from removeElements()
        // which guarantee to generate on;y valid bit entries.
//        final int maxIndex = indices.length();
//        if (maxIndex > srcLength) { 
//            throw new IndexOutOfBoundsException("Index: " + (maxIndex-1) + ", Length: " + srcLength);
//        }
        final int removals = indices.cardinality(); // true bits are items to remove
        final Object result = Array.newInstance(array.getClass().getComponentType(), srcLength - removals);
        int srcIndex=0;
        int destIndex=0;
        int count;
        int set;
        while((set = indices.nextSetBit(srcIndex)) != -1){
            count = set - srcIndex;
            if (count > 0) {
               System.arraycopy(array, srcIndex, result, destIndex, count); // System.arraycopy is native and faster.
               destIndex += count;
            }
            srcIndex = indices.nextClearBit(set);
        }
        count = srcLength - srcIndex;
        if (count > 0) {
            System.arraycopy(array, srcIndex, result, destIndex, count);            
        }
        return result;
    }

java.util.BitSet.cardinality()
public int cardinality() {
    int sum = 0;
    for (int i = 0; i < wordsInUse; i++)
        sum += Long.bitCount(words[i]);
    return sum;
}    

ConcurrentModificationException: Remove during Loop - Java Common Bug Pattern


It's a common bug that we try to use collection.remove during looping the collection.

For example: in ehcache 2.7.0 getAll method
    public Map getAll(Collection keys) throws IllegalStateException, CacheException {
        getAllObserver.begin();
        checkStatus();

        if (disabled) {
            return null;
        }

        if (keys.isEmpty()) {
            return Collections.EMPTY_MAP; // optimize here: no need to create new map
        }

        Map elements = compoundStore.getAll(keys);

        long misses = 0;
        for (Entry entry : elements.entrySet()) {
            Object key = entry.getKey();
            Element element = entry.getValue();
            if (element == null) {
                misses++;
            } else {
                if (isExpired(element)) {
                    tryRemoveImmediately(key, true);
                    elements.remove(key); // bug here.
                    misses++;
                } else {
                    element.updateAccessStatistics();
                }
            }
        }
        int requests = elements.size();
        if (misses == 0) {
            getAllObserver.end(GetAllOutcome.ALL_HIT, requests, 0);
        } else if (misses == requests) {
            getAllObserver.end(GetAllOutcome.ALL_MISS, 0, misses);
        } else {
            getAllObserver.end(GetAllOutcome.PARTIAL, requests - misses, misses);
        }
        return elements;
    }

This was fixed in latest version: ehcache 2.10.0 getAll method
Usually we use Iterator> it=map.entrySet().iterator(); then use it.remove() to delete the element.
Here the iterator of the map returned by compoundStore.getAll() may not support remove method. 
That's why ehcache first adds all expired key to a HashSet, then call:
          try {
              elements.keySet().removeAll(expired);
          } catch (UnsupportedOperationException e) {
              elements = new HashMap(elements);
              elements.keySet().removeAll(expired);
          }

Here is the HashSet removeAll method:
     * Note that this implementation will throw an
     * UnsupportedOperationExceptionif the iterator returned by the

     * iteratormethod does not implement the removemethod.
    public boolean removeAll(Collection c) {
        Objects.requireNonNull(c);
        boolean modified = false;

        if (size() > c.size()) { // optimization: different branch based on size diff
            for (Iterator i = c.iterator(); i.hasNext(); )
                modified |= remove(i.next());
        } else {
            for (Iterator i = iterator(); i.hasNext(); ) {
                if (c.contains(i.next())) {
                    i.remove();
                    modified = true;
                }
            }
        }
        return modified;

    }
ehcache 2.10.0 getAll method
public Map getAll(Collection keys) throws IllegalStateException, CacheException {
        getAllObserver.begin();
        checkStatus();

        if (disabled) {
            return null;
        }

        if (keys.isEmpty()) {
            getAllObserver.end(GetAllOutcome.ALL_HIT, 0, 0);
            return Collections.EMPTY_MAP;
        }

        Map elements = compoundStore.getAll(keys);
        Set expired = new HashSet();
        for (Entry entry : elements.entrySet()) {
            Object key = entry.getKey();
            Element element = entry.getValue();
            if (element != null) {
                if (isExpired(element)) {
                    tryRemoveImmediately(key, true);
                    expired.add(key);
                } else {
                    element.updateAccessStatistics();
                }
            }
        }
        if (!expired.isEmpty()) {
          try {
              elements.keySet().removeAll(expired);
          } catch (UnsupportedOperationException e) {
              elements = new HashMap(elements);
              elements.keySet().removeAll(expired);
          }
        }

        int requests = keys.size();
        int hits = elements.size();
        if (hits == 0) {
            getAllObserver.end(GetAllOutcome.ALL_MISS, 0, requests);
        } else if (requests == hits) {
            getAllObserver.end(GetAllOutcome.ALL_HIT, requests, 0);
        } else {
            getAllObserver.end(GetAllOutcome.PARTIAL, hits, requests - hits);
        }
        return elements;
    }

Building Application to Be Troubleshooting Friendly


The Problem
There is always bug in code that will be found later in production, and in some cases we can't easily reproduce it in test environment.

Ideally we want to rerun the request, check the log in server or maybe change the log level to info or debug.

But usually we can't or at least not that easy: to check the realtime log in production machine or login to it, we have to talk with operation team or open ticket and get it approved; let alone restart the machine to change log level.

Solution
  • Change log level per request.
  • Return diagnosis information in API response.
Implementation
When client sends request with headers: X-Debug-Request=true, and X-DEBUG-KEY equals the encrypted debug.key defined in properties file, the application will change some loggers to debug level(so it will print more logging to help us trouble shooting the problem) and add diagnosis info in response. 

We can put the diagnosis info in header or request body. In this example, we choose to put it in request body.

-- We use debug.key to protect this feature and change it frequently (in each deployment).

The response would be like:
-- We wrap the real response in data tag, so we can easily extend it to add other related data later. 
{
 "data": { // here it can be an object(for getById) or an array(for query)
 }
 "diagnosis": [] // list of string that may help diagnose the problem
 // here can be some pagination meta data 
}

Code
The following code uses Logback and Jersey 1.

PerRequestTurboFilter
public class PerRequestTurboFilter extends DynamicThresholdFilter {
    @Override
    public FilterReply decide(final Marker marker, final Logger logger, final Level level, final String s,
            final Object[] objects, final Throwable throwable) {

        final String loggerName = logger.getName();

        if (loggerName.startsWith("com.myapp") || loggerName.startsWith("org.apache.http")
                || loggerName.startsWith("httpclient.wire")) {
            return super.decide(marker, logger, level, s, objects, throwable);
        }
        addMesToRequestConext(reply, loggerName, msg);
        return FilterReply.NEUTRAL;
    }
    protected void addMesToRequestConext(final FilterReply reply, final String loggerName, final String msg,) {
        final String mdcValue = MDC.get(getKey());
        if (mdcValue != null && reply.equals(FilterReply.ACCEPT) && StringUtils.isNotBlank(msg)
                && loggerName.startsWith("com.myapp")) {
            RequestContextUtil.getRequestContext().addDiagnosis(msg);
        }
    }
} 
PerRequestTurboFilter extends Logback DynamicThresholdFilter, it will only change logger level for com.app and apache http and httpclient. If MDC contains X-Debug-Request, and its value is true, the filter will accept the log request.

Also it will add the log message into the ThreadLocal RequestContext if the logger starts with com.myapp.
Logback.xml
<turboFilter class="com.myapp.PerRequestTurboFilter">
 <Key>X-Debug-Request</Key>
 <OnHigherOrEqual>ACCEPT</OnHigherOrEqual>
 <OnLower>NEUTRAL</OnLower>
 <MDCValueLevelPair>
  <value>true</value>
  <level>DEBUG</level>
 </MDCValueLevelPair>
</turboFilter>

Jersey Filter
@Provider
@Component
public class HttpHeaderFilter implements ContainerRequestFilter, ContainerResponseFilter {

    public static final String X_DEBUG_REQUEST = "X-Debug-Request";
    private static final String X_DEBUG_KEY = "X-DEBUG-KEY";
    // Used to protect debug header, we can change it from property file.
    @Value("${debug.key:}")
    private String debugKey;

    @Context
    HttpServletRequest servletRequest;

    @Override
    public ContainerRequest filter(final ContainerRequest request) {
        setLogDiagnosticContextParams(request);
        return request;
    }

    private void setLogDiagnosticContextParams(final ContainerRequest request) {
        final String debugRequest = request.getHeaderValue(X_DEBUG_REQUEST);;
        if (debugRequest != null) {

            final String clientKey = request.getHeaderValue(X_DEBUG_KEY);
            if (debugKey.equals(clientKey)) {
                MDC.put(X_DEBUG_REQUEST, debugRequest);
                RequestContextUtil.getRequestContext().setEnableDebug(Boolean.valueOf(debugRequest));
            } else {
                RequestContextUtil.getRequestContext().setEnableDebug(false);
            }
        }
    }

    @Override
    public ContainerResponse filter(final ContainerRequest request, final ContainerResponse response) {
        final MultivaluedMap<String, Object> headers = response.getHttpHeaders();

        final Object entity = response.getEntity();
 if (RequestContextUtil.getRequestContext().isEnableDebug()) {
     if (entity instanceof ClientData) {
         final ClientData<?> clientData = (ClientData<?>) entity;
         clientData.add(ClientData.DIAGNOSIS, RequestContextUtil.getRequestContext().getDiagnosises());
     } else {
         headers.add(ClientData.DIAGNOSIS, RequestContextUtil.getRequestContext().getDiagnosises());
     }
 }
        clear();
        return response;
    }

    private void clear() {
        MDC.clear();
        RequestContextUtil.removeRequestContext();
    }
}

We can also add request-id, logged userinto MDC, and change log pattern to include them in logback.xml.

In code, we can add diagnosis info like: RequestContextUtil.getRequestContext().getDiagnosises().add(String.format("query: %s", query));

RequestContextUtil and RequestContext
public class RequestContextUtil {
    private static ThreadLocal<RequestContext> requestContextLocal = new ThreadLocal<RequestContext>() {
        protected RequestContext initialValue() {
            return new RequestContext();
        };
    };

    public static ThreadLocal<RequestContext> getRequestContextLocal() {
        return requestContextLocal;
    }

    public static void setRequestContext(final RequestContext userContext) {
        requestContextLocal.set(userContext);
    }

    public static RequestContext getRequestContext() {
        return requestContextLocal.get();
    }

    public static void removeRequestContext() {
        requestContextLocal.remove();
    }

    public static class RequestContext implements Serializable {
        private static final long serialVersionUID = 1L;

        private boolean enableDebug;
        final List<String> diagnosises = new ArrayList<>();

        public List<String> getDiagnosises() {
            return diagnosises;
        }

        public void addDiagnosis(final String diagnosis) {
            diagnosises.add(diagnosis);
        }

        public boolean isEnableDebug() {
            return enableDebug;
        }

        public void setEnableDebug(final boolean enableDebug) {
            this.enableDebug = enableDebug;
        }
    }
}

Other Approaches
  • Expose API to change log level dynamically and recover it later.
Misc
Debug Feature
Return debug info that can check from response without having to check the log
- Of course secured, protected

Mock User Feature
Preview Feature
In memory test users in test env

How to Build Better Application Libaray


When we build library that will be used by many other projects(either internally or externally), we should think about how client will use it, build sample application, talk with client team: what they want us to improve, make the library easier for client team to use.

Examples
Build Common Authentication Library
For example, if we are building a common authentication library, we should package all configuration file(or Java Configuration classes), properties files in the library, also allow the client to overwrite the existing properties files.

The sensitive information(such as db password, aws key etc - at least for production) should be encrypted, and only the manager or operation team should know the encryption key.
Check How to Encrypt Properties in Spring Apllication

It should provide readme file about how to import it(when import it, whether need exclude some dependencies etc), what kind of properties client can overwrite etc.

The client only need import it, change some properties if needed, the application should just work.

Evolve the Library
The library should evolve, it may be developed with old versions such as Spring 3, Jersey 1, codehause Jackson, but over time, it should evolve to newer version Spring 4, Jersey 2, fasterxml Jackson. 

It may build different branch: old branch with old libraries, new branch with new libraries, client can choose when to upgrade to newer version.

How to Write Quality Code


Familiar with existing code in current project.
Familiar with common libraries or related projects in your company.
Look around before you make change
When you add field or function into a class which is extended or inherit from other classes, check whether it already existed in the class hierarchy, if so, reuse it and refactor the code.

Continuously refactor the code.
At least run every path in the changed code.
-- During dev, we may have to use debugger to change value or force throw exception.

More test.
Automate stuff.

Design data schema first
We can always refactor code but it's hard to change existing data and maintain data compatibility.
- Use ID(not name) as reference(in no-sql or solr).
- If there is some data is in-compatible(schema changed etc) in the query response, log it(and fix it later), but still return other data to client.

Readability
From Clean Code
Meaningful Names
Small function, classes.
Avoid long parameter lists.

Don’t add random check - only add a check if it may happen

Don't use boolean as parameter - never use three-state booleans
- use enum or split to different methods for constructor
- use static factory method

Don't return null
- Return Optional
- Or provide methods like getOrDefault()

Don't change input parameters


Put related code together
Don't swallow exceptions

Exceptions should be exceptional.

Always provide timeout

Use NPE-safe utils
- such as Objects.equals, apache commons

Use advanced data structures
- Use Multimap<K, V> instead of Map<K, List<V>>

Robust
Ask: What else?
Notice missing else or default in switch.

Ask: What may go wrong?

Be conservative or be liberal ?
Throw exception and reject the request or hide it and still service the request?
-- Depend on application and business, but be sure to think about it.

Security
Always validate input, set max length of input.
OWASP Top Ten Project
What should every programmer know about security?
A Guide to Building Secure Web Applications

Logging
Log what may help you trouble shoot the issue.
Logging for audit.

Practice
For example, to implement the function to upload image to CDN.
At first, build the basic version that works.
Then check the code to improve it.

What may go wrong?
Check uploaded file size.
Use Jersey @HeaderParam("Content-Length")  and reject big file.
Use limitinputstream and throw exception when read more than max bytes. 

Normalize file name and extension by replacing special character.
Limit length of file name.

Whitelist file types allowed
-- Use Tika to check file type.

Design Principles
S.O.L.I.D
Robustness Principle
Be conservative in what you send, be liberal in what you accept

Be conservative in what you do, be liberal in what you accept from others

Spring - Encrypt Properties by Customizing PropertySourcesPlaceholderConfigurer


Senario
Usually there are some sensitive properties(such as database password, aws key etc) in an application that we can't put it as plain text and push to git. We have to encrypt it, but decrypt when use it in the application.

Solution
We use some private password key to encrypt them, and put encrypted password in property file like below:
databse.password=ENC:encrypted_password

The ENC: prefix is used to tell the Spring application, this property is encrypted.

We pass the private password key to application server when start it by -DappPassword=password_key

Other approaches:
1. jasypt-spring-boot
You may consider to use jasypt-spring-boot in your sping-boot project. But I found one issue: By default it still decrypts the encrypted property every time when appContext.getEnvironment().getProperty is called for the same property.
com.ulisesbocchio.jasyptspringboot.resolver.DefaultPropertyResolver.resolvePropertyValue(String)

You may write your own MyEncryptablePropertyResolver to cache the already decrypted value in resolvePropertyValue.

2. PropertyPlaceholderConfigurer
Another option is to extend PropertyPlaceholderConfigurer
then implement methods convertPropertyValue, resolveSystemProperty, resolvePlaceholder to decrypt values.
The good part is it decrypt all values only once when spring creates PropertyPlaceholderConfigurer in PropertyResourceConfigurer.postProcessBeanFactory(ConfigurableListableBeanFactory). 

The bad part is that PropertyPlaceholderConfigurer is not EnvironmentAware which means we can not call appContext.getEnvironment().getProperty to get property value in static or non-spring-managed context.

Check the javadoc of PropertyPlaceholderConfigurer or PropertyResourceConfigurer:
As of Spring 3.1, PropertySourcesPlaceholderConfigurer should be used preferentially over this implementation; it is more flexible through taking advantage of the Environment and PropertySource mechanisms also made available in Spring 3.1.

How to Tell Spring to decrypt properties?
In Spring, we usually uses @PropertySource to specify property files. Then Spring uses PropertySourcesPlaceholderConfigurer to read them.

All we have to do is extend PropertySourcesPlaceholderConfigurer, so it(StringValueResolver) will decrypt property value when the values matches some pattern.

Problem of PropertySourcesPlaceholderConfigurer
One issue about PropertySourcesPlaceholderConfigurer: it handles @Value and appContext.getEnvironment().getProperty differently.

To decrypt value for placeholder in @Value, we can define our our StringValueResolver like below.

For @value, when spring tries to create the bean, it will call ValueResolver.resolveStringValue to parse it. We can define our our EncryptedValueResolver to decrypt value for placeholder.
EncryptedValueResolver.resolveStringValue(String) line: 32
DefaultListableBeanFactory(AbstractBeanFactory).resolveEmbeddedValue(String) line: 823
DefaultListableBeanFactory.doResolveDependency(DependencyDescriptor, String, Set, TypeConverter) line: 1084

DefaultListableBeanFactory.resolveDependency(DependencyDescriptor, String, Set, TypeConverter) line: 1064

But when you call appContext.getEnvironment().getProperty, the value is still not decrypted. One approach is to create one util SpringContextBridge, whose getProperty will decrypt the property value. 

Also we define our decrypt method to cache already decrypted value in a map.

The Implementation
First we register our custom EncryptedPropertySourcesPlaceholderConfigurer in configuration.
Notice it has to be static method, this bean has to be created first.
    @Bean
    public static PropertySourcesPlaceholderConfigurer properties() {
        final String password = System.getProperty(APP_ENCRYPTION_PASSWORD);
        if (StringUtils.isBlank(password)) {
            return new PropertySourcesPlaceholderConfigurer();
        } else {
            return new EncryptedPropertySourcesPlaceholderConfigurer(password);
        }
    }
Here we are using jasypt's BasicTextEncryptor, you are free to use any encryptor.
public class EncryptedPropertySourcesPlaceholderConfigurer extends PropertySourcesPlaceholderConfigurer {
    private final String password;

    public EncryptedPropertySourcesPlaceholderConfigurer(final String password) {
        super();
        this.password = password;
    }

    @Override
    protected void doProcessProperties(final ConfigurableListableBeanFactory beanFactoryToProcess,
            final StringValueResolver valueResolver) {
        super.doProcessProperties(beanFactoryToProcess, new EncryptedValueResolver(valueResolver, password));
    }
}
public class EncryptedValueResolver implements StringValueResolver {

    public static final String ENCRYPTED_PREFIX = "ENC:";

    private StringValueResolver valueResolver;

    private static PBEStringEncryptor encryptor;

    // Here we can use different encryptor
    // don't use StrongTextEncryptor, unless u have installed the Java Cryptography
    // Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this jvm.
    EncryptedValueResolver(final StringValueResolver stringValueResolver, final String password) {
        this.valueResolver = stringValueResolver;
        encryptor = getEncryptor(password);
    }

    @Override
    public String resolveStringValue(final String strVal) {

        // Values obtained from the property file to the naming
        // as seen with the encryption target
        String value = valueResolver.resolveStringValue(strVal);
        value = decrypt(value);
        return value;
    }

    private static Map<String, String> decryptValues = new HashMap<>();

    public static String decrypt(String originalValue) {
        if (originalValue != null && originalValue.startsWith(ENCRYPTED_PREFIX)) {
            return decryptValues.computeIfAbsent(originalValue,
                    oldValue -> encryptor.decrypt(oldValue.substring(ENCRYPTED_PREFIX.length())));
        }
        return originalValue;
    }

    private static final String SALT = "YOUR_SALT_HERE";

    public static StandardPBEStringEncryptor getEncryptor(final String password) {
        final StandardPBEStringEncryptor encryptor = new StandardPBEStringEncryptor();
        encryptor.setPassword(password); // we HAVE TO set a password
        // use default algorithm
        // don't use PBEWithMD5AndTripleDES
        encryptor.setAlgorithm("PBEWithMD5AndDES");

        final StringFixedSaltGenerator saltGenerator = new StringFixedSaltGenerator(SALT);
        encryptor.setSaltGenerator(saltGenerator);
        return encryptor;
    }

Problem in previous
There is one problem in previous code 


EncryptorUtil
Last, EncryptorUtil will use our private password key to encrypt text.
public class EncryptorUtil {
  protected static void decrypt(final String password, final String encryptedMessage) {
      final StandardPBEStringEncryptor encryptor = EncryptedValueResolver.getEncryptor(password);

      // don't use BasicTextEncryptor, as it's salt changes.
      // final BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
      // textEncryptor.setPassword(password);
      final String plainText = encryptor.decrypt(encryptedMessage);

      System.out.println("plainText: " + plainText);

  }

  protected static void encrypt(final String password, final String plainText) {
      final StandardPBEStringEncryptor encryptor = EncryptedValueResolver.getEncryptor(password);

      // don't use BasicTextEncryptor, as it's salt changes.
      // final BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
      // textEncryptor.setPassword(password);
      final String myEncryptedText = encryptor.encrypt(plainText);

      System.out.println("Encrypted value: " + EncryptedValueResolver.ENCRYPTED_PREFIX + myEncryptedText);
      // make sure we can decrypt from the encrypted text,
      System.out.println(
              "Decrypted value matches the actual value: " + plainText.equals(encryptor.decrypt(myEncryptedText)));
  }
  public static void main(final String[] args) {
       if (args.length < 3) {
       System.out.println("Please input the password(even length) and the text to be encrypted.");
       return;
       }
       final String action = args[0], password = args[1], text = args[2];
       if ("enc".equalsIgnoreCase(action)) {
       encrypt(password, text);
       } else {
       decrypt(password, text);
       }
  }
}
Resources
5.10. Properties Management
Extending Spring PropertyPlaceholderConfigurer to consider the OS Platform

How to Solve Problems


Don't overcomplicate it.
In most case, the solution is actually quite easy.
Try simplest solution first.

Understand the problem/environment first
Understand the problem.
-- Look carefully at the logging, the symptom.
Think about it first before google search otherwise it may just lead you to totally wrong directions.

Make sure you understand the root cause and the final solution.

Troubleshooting problem is thinking what may go wrong
- how it's likely caused by this?
- how easy we can verify the assumption?
- if not likely and not easily to verify it, try/list other possible root cause/approaches first

Try different approaches
Try to ask different questions (probable solution) and google it.

Track what change you have made
Check when it stops working, and what change u have made.


When help others
Ask what related change they have done

Ask help from others.
But be sure to understand the problem and have tried to fix by yourself.
Provide log or any information that may help others understand the problem.

When work with others
If you are trouble shooting problems together with others, communicate with them, know each other progress, doubt, what u will try next etc. They may try different approaches that may help to get final solution.

If no progress and too late or tired, go to rest and try it later or talk with others.

Solve problem quickly and Learn from it
Solve the problem first (so the project or others can move on) then try to learn related stuff later.
Write down what to check or learn later.

Take the chance to expand your knowledge.

Think More
Think over the code/problem, try to find better solution even it's already "fixed"

Check whether the problem also happens in other places(code, different server etc), if so, fix them.

Misc
Check configuration
-- If it works in dev-line, but fails in product line.

-- Happy hacking

Using Jackson JSON View to Protect Mass Assignment Vulnerabilities


Senario
We use JAX-RS to develop Restful Web Service and only consume and produce json data with Jackson.

In our model class(ModelA), there may be cases that:

  • Some fields are only viewable but not editable  - Client can view them but can't edit it, they are maintained by backend logic
  • Some fields are totally internal, shouldn't even return to client, and client is not allowed to edit.

We need use code to implement this logic, express what fields are viewable only, what fields are editable, and what fields are internal in whitelist mode; otherwise we may expose some security issue. - Check Mass-Assignment Vulnerabilities... Or How Github Got Hacked

Solution - Jackson @JsonView
We can create JSON view like below:
public class View {
    
    public static class Editable {}
    public static class Viewable extends Editable {}
    public static class Internal extends Viewable {}
}

Then annotate our mode class:
@JsonIgnoreProperties(ignoreUnknown = true)
public class Model implements Serializable {

 @JsonView(View.Editable.class)
 protected String editableField;

 @JsonView(View.Viewable.class)
 protected String viewableField; 

 @JsonView(View.Internal.class)
 protected String internalField;
}

At last, we annotate out jax-rs resource with @JsonView annotation.  
 @GET
 @Produces(MediaType.APPLICATION_JSON )
 @JsonView(View.Viewable.class)
 public Iterable<Model> search() {}

 @GET
 @Path("{id}")
 @Produces(MediaType.APPLICATION_JSON )
 @JsonView(View.Viewable.class)
 public Model getModel(@PathParam("id") final String id) {}

 @POST
 @Consumes({MediaType.APPLICATION_JSON})
 public Response add(@JsonView(View.Editable.class) final Model model) {}

In JAX-RS, if one model(either request or response) is annotated with @JsonView(View.Editable.class), in our case add method, Jackson will only serialize or deserialize fields that are annotated with @JsonView(View.Editable.class).
In our case, client can only pass editableField, if client pass any other fields, server will just silently ignore them.

If one model (either request or response) is annotated @JsonView(View.Viewable.class),  then Jackson will serialize or deserialize fields that are annotated with both @JsonView(View.Editable.class) and @JsonView(View.Viewable.class). child(Viewable) inherits view membership from parents(Editable).

In both cases, Jackson will not serialize or deserialize fields that are annotated with  @JsonView(View.Internal.class). So they are protected.

In our service implementation: in add method, we need make sure we add these non-editable fields; in update method, we may have to read and merge these non-editable fields from old value from database to the new value.

-- One trick: Don't mix-use @JsonIgnore and @JsonView, seems this will confuse Jackson, the field will be serialized or deserialized in all cases.

Misc
Spring MVC provides data binder that we can specify what fields are not allowed.
@InitBinder public void initBinder(WebDataBinder binder) { binder.setDisallowedFields(DISALLOWED_FIELDS); }


Read More
Jackson Essentials - the JSON Libaray
Using Jackson JSON View to Protect Mass Assignment Vulnerabilities
Merge JSON Objects: Jackson + BeanUtils.copyProperties
Jackson Generic Type + Java Type Erasure

Jackson Date Serialize + Deserialize
http://wiki.fasterxml.com/JacksonJsonViews
Mass-Assignment Vulnerabilities... Or How Github Got Hacked
Mass Assignment, Rails, and You

Debugging - DynamoDBLocal Status Code: 404


The Problem - Service: AmazonDynamoDBv2; Status Code: 404
Help one colleague to fix on issue today, in his laptop, when it talks to  DynamoDBLocal which is running on default port 8000, it failed with exception:
com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 404; Error Code: null; Request ID: null)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[aws-java-sdk-core-1.10.10.jar:na]

How To Debug
At first, we thought it was caused by some recent change, so we checked in some other's laptop, the code works fine. Then it seemed to be a problem related with his environment.

We restart DynamoDBLocal (no error during restart), try again, it fail with same error.

Then we try yo stop DynamoDBLocal, still same error. This looks strange now. 
As if DynamoDBLocal is not running, it should failed with exception like:
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection refused

As we set log level of org.apache.http.wire to debug, from the log, we can see that AmazonHttpClient established the connection to 8000, and send request like below.
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "POST / HTTP/1.1[\r][\n]"
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "Host: localhost:8000[\r][\n]"
2015-09-10 17:39:08,921 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "X-Amz-Date: 20150911T003908Z[\r][\n]"

This make us think that maybe there is another process is running on port 8000, run command "lsof -i :8080" (lsof -i :8000 -sTCP:LISTEN), it shows there are 2 applications listening on 
COMMAND   PID  USER   FD   TYPE    SIZE/OFF NODE NAME
java    40626  xx   53u  IPv6       0t0  TCP *:irdmi (LISTEN)
node    40981  xx   23u  IPv4       0t0  TCP localhost:irdmi (LISTEN) 

Now, the problem is obvious, the nodejs http server is running on 8000 and accepts all requests to 8000. So we kill the nodejs application, now, our application runs fine.

But why DynamoDBLocal is not throwing exception during start if another application is running on same port?
This issue seems related with nodejs http server.

In another test, I run tomcat at port 8000, then start  DynamoDBLocal, it does throw exception, but it continues running, when client calls DynamoDB API, it will fail with error code: 404 or 405.

Lesson Learned:

  • Trouble shooting problem is more about thinking what may go wrong.
  • Understand the problem, think about it, try it before google search with the error code as Google search may just lead you  to totally wrong direction.
  • Check when it stops working, and what change u have made before.

Java Config: Integrating JAX-RS Jersey with Spring Security


The Scenario
When we were developing the admin backend, we split it into 2 projects: backend application using jax-rs jersey, front end with angularjs - following single app design principle.

Later we decide to merge these 2 into one project, and use Spring security to protect front side(web url, page section etc) and jersey jax-rs.

We prefer Spring Java config over xml because it's more flexible, and XML configuration is kind of black box to developers,  using Java config, we can know more about the implementation and help us debug it later.

Update:
Later we upgrade to jersey 2:

@Priority(value = 1)
public class MyWebApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {

    @Override
    protected Class<?>[] getRootConfigClasses() {
        return new Class[] {MyAppConfig.class};
    }

    @Override
    protected Class<?>[] getServletConfigClasses() {
        return null;
    }

    @Override
    protected String[] getServletMappings() {
        return new String[] {"/"};
    }

    @Override
    public void onStartup(final ServletContext servletContext) throws ServletException {
        super.onStartup(servletContext);
        // this is to fix issues in jersey-srping3
        // org.glassfish.jersey.server.spring.SpringWebApplicationInitializer
        servletContext.setInitParameter("contextConfigLocation", "");
        servletContext.addListener(RequestContextListener.class);
        MyUtil.addDefaultUncaughtExceptionHandler();
    }

    @Override
    protected WebApplicationContext createRootApplicationContext() {
        final WebApplicationContext context = super.createRootApplicationContext();
        final ConfigurableEnvironment env = (ConfigurableEnvironment) context.getEnvironment();
        final String profile = (String) env.getSystemProperties().get("env");
        env.setActiveProfiles(profile);
        return context;
    }
}

public class MySecurityInitializer extends AbstractSecurityWebApplicationInitializer {}

@WebServlet(loadOnStartup = 1)
@ApplicationPath("/v1/*")
public class MyJerseyApplication extends ResourceConfig {
    public JerseyRestProvisionApplication() {
        packages("the_package_xx");
        property(ServerProperties.WADL_FEATURE_DISABLE, true);
        register(JacksonFeature.class);
        register(GZipEncoder.class);
        register(MultiPartFeature.class);

        register(new LoggingFilter(Logger.getLogger(MyJerseyApplication.class.getName()), true));
        register(RequestContextFilter.class);
        register(MyHttpHeaderFilter.class);
    }
}
How To
Basically we configure two servlets:

  • web.servlet.DispatcherServlet: map it to / (notice not /*). This handles Spring Security and Spring MVC.
  • Jersey SpringServlet: map to v1(all rest API starts with v1/).

Spring security is used to do authentication and authorization.

Jersery 1

public class AppInitializer implements WebApplicationInitializer {

    @Override
    public void onStartup(final ServletContext servletContext) {
        final AnnotationConfigWebApplicationContext rootContext = new AnnotationConfigWebApplicationContext();
        rootContext.register(AppConfig.class);

        final ConfigurableEnvironment env = rootContext.getEnvironment();

        final String profile = (String) env.getSystemProperties().get("env");
        env.setActiveProfiles(profile);

        servletContext.addListener(new ContextLoaderListener(rootContext));

        addSpringServlet(servletContext, profile);
        addSpringJersyServlet(servletContext, env);
    }

    protected void addSpringServlet(final ServletContext servletContext, final String profile) {

        final DispatcherServlet dispatcherServlet = new DispatcherServlet(new GenericWebApplicationContext());

        final Dynamic dispatcherDynamic = servletContext.addServlet("dispatcherServlet", dispatcherServlet);;
        dispatcherDynamic.setLoadOnStartup(1);

        dispatcherDynamic.addMapping("/");

        servletContext
                .addFilter(AbstractSecurityWebApplicationInitializer.DEFAULT_FILTER_NAME, DelegatingFilterProxy.class)
                .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), false, "/*");
    }

    protected void addSpringJersyServlet(final ServletContext servletContext, final ConfigurableEnvironment env) {
        final ServletRegistration.Dynamic appServlet = servletContext.addServlet("jersey-servlet", new SpringServlet());

        appServlet.setInitParameter(JSONConfiguration.FEATURE_POJO_MAPPING, "true");
        appServlet.setInitParameter(ResourceConfig.FEATURE_TRACE, env.getProperty("jersey.enable.trace", "false"));
        appServlet.setInitParameter(ResourceConfig.FEATURE_DISABLE_WADL,
                env.getProperty("jersey.disable.wadl", "true"));

        appServlet.setInitParameter(ResourceConfig.PROPERTY_CONTAINER_REQUEST_FILTERS,..);
        appServlet.setInitParameter(ResourceConfig.PROPERTY_CONTAINER_RESPONSE_FILTERS, ..);

        appServlet.setLoadOnStartup(2);
        appServlet.addMapping("/v1/*");
    }
}
Resources
Difference between / and /* in servlet mapping url pattern
http://javapapers.com/servlet/what-is-servlet-mapping/

SSH: Practical Guide for Developers


It's common that your server(destination_server) is running in cloud(AWS) which you have to access via another linux server(proxy_server), and you want to enable remote debug in the destination_server.

To do this, run this in the proxy_server:
ssh -f -N -L9999:localhost:9999 username@destination_server

Run this in your local server:
ssh -f -N -L9999:localhost:9999 username@proxy_server

In your destination_server, add this to the command that is used to start your server.
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999

Then start remote application in eclipse that connects to localhost:9999.

-- If Eclipse is very slow in remote debug mode, try to remove (all) breakpoints and expressions in eclipse.

-- Use ssh tunnel, you can do a lot of other stuff.
-- We can use ssh tunnel to test code locally without really deploy to aws.

ssh -f -N -L18983:remote-host:remote-port $USER@host-in-middle

SOCKS and dynamic port forwarding
1. .ssh/config
host server
  DynamicForward 1080


2. ssh -fND localhost:1080 bastion_server
ssh -fND 1080 bastion_server

curl --socks5 localhost:1080 remote_url

Java code
-DsocksProxyHost=127.0.0.1 -DsocksProxyPort=3000
System.setProperty("socksProxyHost", "127.0.0.1");

System.setProperty("socksProxyPort", "3000");


Install Proxy SwitchySharp in Chrome

-f  send this ssh operation into the background after password prompts
-N do not execute remote command or wait for the user to provide any commands
-L static port forward
-D the local port to listen on
  - for dynamic port forwarding

Related
Create tunnel to remote Cassandra via man-in-the-middle
ssh -f host-in-middle -L 9042:destination-cassandra-server:9042 -N -v

Resources
SSH Tunneling Explained
SSH Tunnel - Local and Remote Port Forwarding Explained With Examples

Implementing Inheritance in Spring Solr Data


In our solr data model, there is inheritance relationship between classes.

Please check about why we are mixing Spring Data Solr and SolrJ to make it work with Solr Cloud 5.

In the DAO:
public BaseModel findOne(final String id) {
    final SolrDocument doc = solrServer.getById(getCollection(), id);;
    if (doc == null) {
        throw new WrappedException(ErrorCode.INVALID_PARAMETER, "BaseModel not found: " + id);
    }
    return converter.read(BaseModel.class, doc);
}


But this will cause Spring Data Solr returns BaseModel object not its concrete sub-classes which we expected.

To fix this, we need change SolrConverter's read method to map to its concrete sub-classes in Spring configuration class.
@Bean
public static SolrConverter mappingSolrConverter() {
    final MappingContext<? extends SolrPersistentEntity<?>, SolrPersistentProperty> mappingContext =
            new SimpleSolrMappingContext();
    final MappingSolrConverter converter = new MappingSolrConverter(mappingContext) {
        @SuppressWarnings("unchecked")
        @Override
        protected <S> S read(final TypeInformation<S> targetTypeInformation, final Map<String, ?> source) {
            final Object obj = source.get(BaseModel.FIELD_TYPE);
            if (obj != null) {
                final String type = String.valueOf(obj);
                switch (type) {
                    case BaseModel.TYPE_A:
                        return (S) super.read(ClassTypeInformation.from(ModelA.class), source);
                    case BaseModel.TYPE_B:
                        return (S) super.read(ClassTypeInformation.from(ModelB.class), source);
                    case BaseModel.TYPE_C:
                        return (S) super.read(ClassTypeInformation.from(ModelC.class), source);
                    default:
                        throw new IllegalArgumentException(
                                "invalid " + BaseModel.FIELD_TYPE + ", value: " + type);
                }
            }
            return super.read(targetTypeInformation, source);
        }
    };
    return converter;
}

Mix Spring Data Solr and SolrJ in Solr Cloud 5


We are migrating from Solr 4(part of DataStax) to Solr Cloud 5(without Datastax). 
In previous code, we are using Spring Data Solr to talk with Solr, we are using SolrCrudRepository and also map closed set value to java enum class(not use solr.EnumField), map multiple value field to Set.

As Spring Data Solr doesn't work with Solr Cloud 5, we have to replace the DAO layer(SolrCrudRepository) to Use SolrJ directly.
But SolrJ doesn't support map field to Java Enum or map multiple value field to Set.

We don't want to change the Enum to String, change set to List, as if so, we need change a lot of code. 

We want keep the Model class untouched. So I decided to mix them: Use Spring Data Solr to convert between SolrInputDocument and Java object, use SolrJ to update or query Solr Server.
public abstract class XModelRepository<T extends XModel> implements IXModelRepository<T> {
    @Value("${solr.server.core.collection_name}")
    private String collection;

    @Autowired
    protected SolrClient solrServer;

    @Autowired
    protected SolrConverter converter;

    public final void save(final XModel model) {
        try {
            final Iterable<SolrInputDocument> docs = converter.write(Lists.newArrayList(model));
            final UpdateResponse response = solrServer.add(getCollection(), docs.iterator());
            softCommit();
        } catch (IOException | SolrServerException e) {
            throw new WrappedException(ErrorCode.INTERNAL_ERROR, e);
        }
    }
    public XModel findOne(final String id) {
        try {
            final SolrDocument doc = solrServer.getById(collection, id);
            return converter.read(XModel.class, doc);
        } catch (SolrServerException | IOException e) {
            throw new WrappedException(ErrorCode.INTERNAL_ERROR, e);
        }
    }
} 

This works in most case. But there is one issue: when it stores enum to solr, it saves something like xx.enums.UserType:TypeA, when it reads from solr server, tries to convert the string to enum, it fails with  the following error.
Java.lang.IllegalArgumentException: No enum constant xx.enums.UserType:TypeA
at java.lang.Enum.valueOf(Enum.java:236)


To fix this, I have to copy Spring org.springframework.core.convert.support.EnumToStringConverter and register it in SolrConverter in the configuration class like below:
    @Bean
    public static SolrConverter mappingSolrConverter() {
        final MappingContext<? extends SolrPersistentEntity<?>, SolrPersistentProperty> mappingContext =
                new SimpleSolrMappingContext();
        final MappingSolrConverter converter = new MappingSolrConverter(mappingContext);

        final List<Object> converters = new ArrayList<Object>();
        final DefaultConversionService conversionService = new DefaultConversionService();

        converters.add(new EnumToStringConverter(conversionService));

        final CustomConversions customConversions = new CustomConversions(converters) {
            @Override
            public boolean isSimpleType(final Class<?> clazz) {
                if (Enum.class.equals(clazz) || Enum.class.isAssignableFrom(clazz)) {
                    return false;
                }
                return super.isSimpleType(clazz);
            }

        };
        converter.setCustomConversions(customConversions);
        return converter;
    }
 // Copied from Spring org.springframework.core.convert.support.EnumToStringConverter 
 final class EnumToStringConverter implements Converter<Enum<?>, String>, ConditionalConverter {
     private final ConversionService conversionService;

     public EnumToStringConverter(final ConversionService conversionService) {
         this.conversionService = conversionService;
     }

     public boolean matches(final TypeDescriptor sourceType, final TypeDescriptor targetType) {
         for (@SuppressWarnings("rawtypes")
         final Class interfaceType : ClassUtils.getAllInterfacesForClass(sourceType.getType())) {
             if (this.conversionService.canConvert(TypeDescriptor.valueOf(interfaceType), targetType)) {
                 return false;
             }
         }
         return true;
     }

     public String convert(final Enum<?> source) {
         return source.name();
     }
 }

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)