Building Application to Be Troubleshooting Friendly

The Problem
There is always bug in code that will be found later in production, and in some cases we can't easily reproduce it in test environment.

Ideally we want to rerun the request, check the log in server or maybe change the log level to info or debug.

But usually we can't or at least not that easy: to check the realtime log in production machine or login to it, we have to talk with operation team or open ticket and get it approved; let alone restart the machine to change log level.

  • Change log level per request.
  • Return diagnosis information in API response.
When client sends request with headers: X-Debug-Request=true, and X-DEBUG-KEY equals the encrypted debug.key defined in properties file, the application will change some loggers to debug level(so it will print more logging to help us trouble shooting the problem) and add diagnosis info in response. 

We can put the diagnosis info in header or request body. In this example, we choose to put it in request body.

-- We use debug.key to protect this feature and change it frequently (in each deployment).

The response would be like:
-- We wrap the real response in data tag, so we can easily extend it to add other related data later. 
 "data": { // here it can be an object(for getById) or an array(for query)
 "diagnosis": [] // list of string that may help diagnose the problem
 // here can be some pagination meta data 

The following code uses Logback and Jersey 1.

public class PerRequestTurboFilter extends DynamicThresholdFilter {
    public FilterReply decide(final Marker marker, final Logger logger, final Level level, final String s,
            final Object[] objects, final Throwable throwable) {

        final String loggerName = logger.getName();

        if (loggerName.startsWith("com.myapp") || loggerName.startsWith("org.apache.http")
                || loggerName.startsWith("httpclient.wire")) {
            return super.decide(marker, logger, level, s, objects, throwable);
        addMesToRequestConext(reply, loggerName, msg);
        return FilterReply.NEUTRAL;
    protected void addMesToRequestConext(final FilterReply reply, final String loggerName, final String msg,) {
        final String mdcValue = MDC.get(getKey());
        if (mdcValue != null && reply.equals(FilterReply.ACCEPT) && StringUtils.isNotBlank(msg)
                && loggerName.startsWith("com.myapp")) {
PerRequestTurboFilter extends Logback DynamicThresholdFilter, it will only change logger level for and apache http and httpclient. If MDC contains X-Debug-Request, and its value is true, the filter will accept the log request.

Also it will add the log message into the ThreadLocal RequestContext if the logger starts with com.myapp.
<turboFilter class="com.myapp.PerRequestTurboFilter">

Jersey Filter
public class HttpHeaderFilter implements ContainerRequestFilter, ContainerResponseFilter {

    public static final String X_DEBUG_REQUEST = "X-Debug-Request";
    private static final String X_DEBUG_KEY = "X-DEBUG-KEY";
    // Used to protect debug header, we can change it from property file.
    private String debugKey;

    HttpServletRequest servletRequest;

    public ContainerRequest filter(final ContainerRequest request) {
        return request;

    private void setLogDiagnosticContextParams(final ContainerRequest request) {
        final String debugRequest = request.getHeaderValue(X_DEBUG_REQUEST);;
        if (debugRequest != null) {

            final String clientKey = request.getHeaderValue(X_DEBUG_KEY);
            if (debugKey.equals(clientKey)) {
                MDC.put(X_DEBUG_REQUEST, debugRequest);
            } else {

    public ContainerResponse filter(final ContainerRequest request, final ContainerResponse response) {
        final MultivaluedMap<String, Object> headers = response.getHttpHeaders();

        final Object entity = response.getEntity();
 if (RequestContextUtil.getRequestContext().isEnableDebug()) {
     if (entity instanceof ClientData) {
         final ClientData<?> clientData = (ClientData<?>) entity;
         clientData.add(ClientData.DIAGNOSIS, RequestContextUtil.getRequestContext().getDiagnosises());
     } else {
         headers.add(ClientData.DIAGNOSIS, RequestContextUtil.getRequestContext().getDiagnosises());
        return response;

    private void clear() {

We can also add request-id, logged userinto MDC, and change log pattern to include them in logback.xml.

In code, we can add diagnosis info like: RequestContextUtil.getRequestContext().getDiagnosises().add(String.format("query: %s", query));

RequestContextUtil and RequestContext
public class RequestContextUtil {
    private static ThreadLocal<RequestContext> requestContextLocal = new ThreadLocal<RequestContext>() {
        protected RequestContext initialValue() {
            return new RequestContext();

    public static ThreadLocal<RequestContext> getRequestContextLocal() {
        return requestContextLocal;

    public static void setRequestContext(final RequestContext userContext) {

    public static RequestContext getRequestContext() {
        return requestContextLocal.get();

    public static void removeRequestContext() {

    public static class RequestContext implements Serializable {
        private static final long serialVersionUID = 1L;

        private boolean enableDebug;
        final List<String> diagnosises = new ArrayList<>();

        public List<String> getDiagnosises() {
            return diagnosises;

        public void addDiagnosis(final String diagnosis) {

        public boolean isEnableDebug() {
            return enableDebug;

        public void setEnableDebug(final boolean enableDebug) {
            this.enableDebug = enableDebug;

Other Approaches

  • Expose API to change log level dynamically and recover it later.

How to Build Better Application Libaray

When we build library that will be used by many other projects(either internally or externally), we should think about how client will use it, build sample application, talk with client team: what they want us to improve, make the library easier for client team to use.

Build Common Authentication Library
For example, if we are building a common authentication library, we should package all configuration file(or Java Configuration classes), properties files in the library, also allow the client to overwrite the existing properties files.

The sensitive information(such as db password, aws key etc - at least for production) should be encrypted, and only the manager or operation team should know the encryption key.
Check How to Encrypt Properties in Spring Apllication

It should provide readme file about how to import it(when import it, whether need exclude some dependencies etc), what kind of properties client can overwrite etc.

The client only need import it, change some properties if needed, the application should just work.

Evolve the Library
The library should evolve, it may be developed with old versions such as Spring 3, Jersey 1, codehause Jackson, but over time, it should evolve to newer version Spring 4, Jersey 2, fasterxml Jackson. 

It may build different branch: old branch with old libraries, new branch with new libraries, client can choose when to upgrade to newer version.

How to Write Quality Code

Familiar with existing code in current project.
Familiar with common libraries or related projects in your company.
Look around before you make change
When you add field or function into a class which is extended or inherit from other classes, check whether it already existed in the class hierarchy, if so, reuse it and refactor the code.

Continuously refactor the code.
At least run every path in the changed code.
-- During dev, we may have to use debugger to change value or force throw exception.

More test.
Automate stuff.

Design data schema first
We can always refactor code but it's hard to change existing data and maintain data compatibility.
- Use ID(not name) as reference(in no-sql or solr).
- If there is some data is in-compatible(schema changed etc) in the query response, log it(and fix it later), but still return other data to client.

From Clean Code
Meaningful Names
Small function, classes.
Avoid long parameter lists.

Don’t add random check - only add a check if it may happen

Don't use boolean as parameter - never use three-state booleans
- use enum or split to different methods for constructor
- use static factory method

Don't return null
- Return Optional
- Or provide methods like getOrDefault()

Don't change input parameters

Put related code together
Don't swallow exceptions

Exceptions should be exceptional.

Always provide timeout

Use NPE-safe utils
- such as Objects.equals, apache commons

Use advanced data structures
- Use Multimap<K, V> instead of Map<K, List<V>>

Ask: What else?
Notice missing else or default in switch.

Ask: What may go wrong?

Be conservative or be liberal ?
Throw exception and reject the request or hide it and still service the request?
-- Depend on application and business, but be sure to think about it.

Always validate input, set max length of input.
OWASP Top Ten Project
What should every programmer know about security?
A Guide to Building Secure Web Applications

Log what may help you trouble shoot the issue.
Logging for audit.

For example, to implement the function to upload image to CDN.
At first, build the basic version that works.
Then check the code to improve it.

What may go wrong?
Check uploaded file size.
Use Jersey @HeaderParam("Content-Length")  and reject big file.
Use limitinputstream and throw exception when read more than max bytes. 

Normalize file name and extension by replacing special character.
Limit length of file name.

Whitelist file types allowed
-- Use Tika to check file type.

Design Principles
Robustness Principle
Be conservative in what you send, be liberal in what you accept

Be conservative in what you do, be liberal in what you accept from others

Spring - Encrypt Properties by Customizing PropertySourcesPlaceholderConfigurer

Usually there are some sensitive properties(such as database password, aws key etc) in an application that we can't put it as plain text and push to git. We have to encrypt it, but decrypt when use it in the application.

We use some private password key to encrypt them, and put encrypted password in property file like below:

The ENC: prefix is used to tell the Spring application, this property is encrypted.

We pass the private password key to application server when start it by -DappPassword=password_key

Other approaches:
1. jasypt-spring-boot
You may consider to use jasypt-spring-boot in your sping-boot project. But I found one issue: By default it still decrypts the encrypted property every time when appContext.getEnvironment().getProperty is called for the same property.

You may write your own MyEncryptablePropertyResolver to cache the already decrypted value in resolvePropertyValue.

2. PropertyPlaceholderConfigurer
Another option is to extend PropertyPlaceholderConfigurer
then implement methods convertPropertyValue, resolveSystemProperty, resolvePlaceholder to decrypt values.
The good part is it decrypt all values only once when spring creates PropertyPlaceholderConfigurer in PropertyResourceConfigurer.postProcessBeanFactory(ConfigurableListableBeanFactory). 

The bad part is that PropertyPlaceholderConfigurer is not EnvironmentAware which means we can not call appContext.getEnvironment().getProperty to get property value in static or non-spring-managed context.

Check the javadoc of PropertyPlaceholderConfigurer or PropertyResourceConfigurer:
As of Spring 3.1, PropertySourcesPlaceholderConfigurer should be used preferentially over this implementation; it is more flexible through taking advantage of the Environment and PropertySource mechanisms also made available in Spring 3.1.

How to Tell Spring to decrypt properties?
In Spring, we usually uses @PropertySource to specify property files. Then Spring uses PropertySourcesPlaceholderConfigurer to read them.

All we have to do is extend PropertySourcesPlaceholderConfigurer, so it(StringValueResolver) will decrypt property value when the values matches some pattern.

Problem of PropertySourcesPlaceholderConfigurer
One issue about PropertySourcesPlaceholderConfigurer: it handles @Value and appContext.getEnvironment().getProperty differently.

To decrypt value for placeholder in @Value, we can define our our StringValueResolver like below.

For @value, when spring tries to create the bean, it will call ValueResolver.resolveStringValue to parse it. We can define our our EncryptedValueResolver to decrypt value for placeholder.
EncryptedValueResolver.resolveStringValue(String) line: 32
DefaultListableBeanFactory(AbstractBeanFactory).resolveEmbeddedValue(String) line: 823
DefaultListableBeanFactory.doResolveDependency(DependencyDescriptor, String, Set, TypeConverter) line: 1084

DefaultListableBeanFactory.resolveDependency(DependencyDescriptor, String, Set, TypeConverter) line: 1064

But when you call appContext.getEnvironment().getProperty, the value is still not decrypted. One approach is to create one util SpringContextBridge, whose getProperty will decrypt the property value. 

Also we define our decrypt method to cache already decrypted value in a map.

The Implementation
First we register our custom EncryptedPropertySourcesPlaceholderConfigurer in configuration.
Notice it has to be static method, this bean has to be created first.
    public static PropertySourcesPlaceholderConfigurer properties() {
        final String password = System.getProperty(APP_ENCRYPTION_PASSWORD);
        if (StringUtils.isBlank(password)) {
            return new PropertySourcesPlaceholderConfigurer();
        } else {
            return new EncryptedPropertySourcesPlaceholderConfigurer(password);
Here we are using jasypt's BasicTextEncryptor, you are free to use any encryptor.
public class EncryptedPropertySourcesPlaceholderConfigurer extends PropertySourcesPlaceholderConfigurer {
    private final String password;

    public EncryptedPropertySourcesPlaceholderConfigurer(final String password) {
        this.password = password;

    protected void doProcessProperties(final ConfigurableListableBeanFactory beanFactoryToProcess,
            final StringValueResolver valueResolver) {
        super.doProcessProperties(beanFactoryToProcess, new EncryptedValueResolver(valueResolver, password));
public class EncryptedValueResolver implements StringValueResolver {

    public static final String ENCRYPTED_PREFIX = "ENC:";

    private StringValueResolver valueResolver;

    private static PBEStringEncryptor encryptor;

    // Here we can use different encryptor
    // don't use StrongTextEncryptor, unless u have installed the Java Cryptography
    // Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this jvm.
    EncryptedValueResolver(final StringValueResolver stringValueResolver, final String password) {
        this.valueResolver = stringValueResolver;
        encryptor = getEncryptor(password);

    public String resolveStringValue(final String strVal) {

        // Values obtained from the property file to the naming
        // as seen with the encryption target
        String value = valueResolver.resolveStringValue(strVal);
        value = decrypt(value);
        return value;

    private static Map<String, String> decryptValues = new HashMap<>();

    public static String decrypt(String originalValue) {
        if (originalValue != null && originalValue.startsWith(ENCRYPTED_PREFIX)) {
            return decryptValues.computeIfAbsent(originalValue,
                    oldValue -> encryptor.decrypt(oldValue.substring(ENCRYPTED_PREFIX.length())));
        return originalValue;

    private static final String SALT = "YOUR_SALT_HERE";

    public static StandardPBEStringEncryptor getEncryptor(final String password) {
        final StandardPBEStringEncryptor encryptor = new StandardPBEStringEncryptor();
        encryptor.setPassword(password); // we HAVE TO set a password
        // use default algorithm
        // don't use PBEWithMD5AndTripleDES

        final StringFixedSaltGenerator saltGenerator = new StringFixedSaltGenerator(SALT);
        return encryptor;

Problem in previous
There is one problem in previous code 

Last, EncryptorUtil will use our private password key to encrypt text.
public class EncryptorUtil {
  protected static void decrypt(final String password, final String encryptedMessage) {
      final StandardPBEStringEncryptor encryptor = EncryptedValueResolver.getEncryptor(password);

      // don't use BasicTextEncryptor, as it's salt changes.
      // final BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
      // textEncryptor.setPassword(password);
      final String plainText = encryptor.decrypt(encryptedMessage);

      System.out.println("plainText: " + plainText);


  protected static void encrypt(final String password, final String plainText) {
      final StandardPBEStringEncryptor encryptor = EncryptedValueResolver.getEncryptor(password);

      // don't use BasicTextEncryptor, as it's salt changes.
      // final BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
      // textEncryptor.setPassword(password);
      final String myEncryptedText = encryptor.encrypt(plainText);

      System.out.println("Encrypted value: " + EncryptedValueResolver.ENCRYPTED_PREFIX + myEncryptedText);
      // make sure we can decrypt from the encrypted text,
              "Decrypted value matches the actual value: " + plainText.equals(encryptor.decrypt(myEncryptedText)));
  public static void main(final String[] args) {
       if (args.length < 3) {
       System.out.println("Please input the password(even length) and the text to be encrypted.");
       final String action = args[0], password = args[1], text = args[2];
       if ("enc".equalsIgnoreCase(action)) {
       encrypt(password, text);
       } else {
       decrypt(password, text);
5.10. Properties Management
Extending Spring PropertyPlaceholderConfigurer to consider the OS Platform

How to Solve Problems

Don't overcomplicate it.
In most case, the solution is actually quite easy.
Try simplest solution first.

Understand the problem/environment first
Understand the problem.
-- Look carefully at the logging, the symptom.
Think about it first before google search otherwise it may just lead you  to totally wrong direction.

Make sure you understand the root cause and the final solution.

Trouble shooting problem is thinking what may go wrong
- how it's likely caused by this?
- how easy we can verify it?

- if not likely and not easily to verify it, try/list other possible root cause/approaches first

Try different approaches
Try to ask different questions (probable solution) and google it.

Track what change you have made
Check when it stops working, and what change u have made.

When help others
Ask what related change they have done

Ask help from others.
But be sure to understand the problem and have tried to fix by yourself.
Provide log or any information that may help others understand the problem.

When work with others
If you are trouble shooting problems together with others, communicate with them, know each other progress, doubt, what u will try next etc. They may try different approaches that may help to get final solution.

If no progress and too late or tired, go to rest and try it later or talk with others.

Solve problem quickly and Learn from it
Solve the problem first (so the project or others can move on) then try to learn related stuff later.
Write down what to check or learn later.

Take the chance to expand your knowledge.

Think More Think over the code/problem, try to find better solution even it's already "fixed"

Check whether the problem also happens in other places(code, different server etc), if so, fix them.

Check configuration
-- If it works in dev-line, but fails in product line.

-- Happy hacking

Using Jackson JSON View to Protect Mass Assignment Vulnerabilities

We use JAX-RS to develop Restful Web Service and only consume and produce json data with Jackson.

In our model class(ModelA), there may be cases that:

  • Some fields are only viewable but not editable  - Client can view them but can't edit it, they are maintained by backend logic
  • Some fields are totally internal, shouldn't even return to client, and client is not allowed to edit.

We need use code to implement this logic, express what fields are viewable only, what fields are editable, and what fields are internal in whitelist mode; otherwise we may expose some security issue. - Check Mass-Assignment Vulnerabilities... Or How Github Got Hacked

Solution - Jackson @JsonView
We can create JSON view like below:
public class View {
    public static class Editable {}
    public static class Viewable extends Editable {}
    public static class Internal extends Viewable {}

Then annotate our mode class:
@JsonIgnoreProperties(ignoreUnknown = true)
public class Model implements Serializable {

 protected String editableField;

 protected String viewableField; 

 protected String internalField;

At last, we annotate out jax-rs resource with @JsonView annotation.  
 @Produces(MediaType.APPLICATION_JSON )
 public Iterable<Model> search() {}

 @Produces(MediaType.APPLICATION_JSON )
 public Model getModel(@PathParam("id") final String id) {}

 public Response add(@JsonView(View.Editable.class) final Model model) {}

In JAX-RS, if one model(either request or response) is annotated with @JsonView(View.Editable.class), in our case add method, Jackson will only serialize or deserialize fields that are annotated with @JsonView(View.Editable.class).
In our case, client can only pass editableField, if client pass any other fields, server will just silently ignore them.

If one model (either request or response) is annotated @JsonView(View.Viewable.class),  then Jackson will serialize or deserialize fields that are annotated with both @JsonView(View.Editable.class) and @JsonView(View.Viewable.class). child(Viewable) inherits view membership from parents(Editable).

In both cases, Jackson will not serialize or deserialize fields that are annotated with  @JsonView(View.Internal.class). So they are protected.

In our service implementation: in add method, we need make sure we add these non-editable fields; in update method, we may have to read and merge these non-editable fields from old value from database to the new value.

-- One trick: Don't mix-use @JsonIgnore and @JsonView, seems this will confuse Jackson, the field will be serialized or deserialized in all cases.

Spring MVC provides data binder that we can specify what fields are not allowed.
@InitBinder public void initBinder(WebDataBinder binder) { binder.setDisallowedFields(DISALLOWED_FIELDS); }

Read More
Jackson Essentials - the JSON Libaray
Using Jackson JSON View to Protect Mass Assignment Vulnerabilities
Merge JSON Objects: Jackson + BeanUtils.copyProperties
Jackson Generic Type + Java Type Erasure

Jackson Date Serialize + Deserialize
Mass-Assignment Vulnerabilities... Or How Github Got Hacked
Mass Assignment, Rails, and You

Debugging - DynamoDBLocal Status Code: 404

The Problem - Service: AmazonDynamoDBv2; Status Code: 404
Help one colleague to fix on issue today, in his laptop, when it talks to  DynamoDBLocal which is running on default port 8000, it failed with exception:
com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 404; Error Code: null; Request ID: null)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse( ~[aws-java-sdk-core-1.10.10.jar:na]

How To Debug
At first, we thought it was caused by some recent change, so we checked in some other's laptop, the code works fine. Then it seemed to be a problem related with his environment.

We restart DynamoDBLocal (no error during restart), try again, it fail with same error.

Then we try yo stop DynamoDBLocal, still same error. This looks strange now. 
As if DynamoDBLocal is not running, it should failed with exception like:
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection refused

As we set log level of org.apache.http.wire to debug, from the log, we can see that AmazonHttpClient established the connection to 8000, and send request like below.
2015-09-10 17:39:08,920 [http-nio-] DEBUG o.a.h.i.c.Wire   >> "POST / HTTP/1.1[\r][\n]"
2015-09-10 17:39:08,920 [http-nio-] DEBUG o.a.h.i.c.Wire   >> "Host: localhost:8000[\r][\n]"
2015-09-10 17:39:08,921 [http-nio-] DEBUG o.a.h.i.c.Wire   >> "X-Amz-Date: 20150911T003908Z[\r][\n]"

This make us think that maybe there is another process is running on port 8000, run command "lsof -i :8080" (lsof -i :8000 -sTCP:LISTEN), it shows there are 2 applications listening on 
java    40626  xx   53u  IPv6       0t0  TCP *:irdmi (LISTEN)
node    40981  xx   23u  IPv4       0t0  TCP localhost:irdmi (LISTEN) 

Now, the problem is obvious, the nodejs http server is running on 8000 and accepts all requests to 8000. So we kill the nodejs application, now, our application runs fine.

But why DynamoDBLocal is not throwing exception during start if another application is running on same port?
This issue seems related with nodejs http server.

In another test, I run tomcat at port 8000, then start  DynamoDBLocal, it does throw exception, but it continues running, when client calls DynamoDB API, it will fail with error code: 404 or 405.

Lesson Learned:

  • Trouble shooting problem is more about thinking what may go wrong.
  • Understand the problem, think about it, try it before google search with the error code as Google search may just lead you  to totally wrong direction.
  • Check when it stops working, and what change u have made before.

Java Config: Integrating JAX-RS Jersey with Spring Security

The Scenario
When we were developing the admin backend, we split it into 2 projects: backend application using jax-rs jersey, front end with angularjs - following single app design principle.

Later we decide to merge these 2 into one project, and use Spring security to protect front side(web url, page section etc) and jersey jax-rs.

We prefer Spring Java config over xml because it's more flexible, and XML configuration is kind of black box to developers,  using Java config, we can know more about the implementation and help us debug it later.

Later we upgrade to jersey 2:

@Priority(value = 1)
public class MyWebApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {

    protected Class<?>[] getRootConfigClasses() {
        return new Class[] {MyAppConfig.class};

    protected Class<?>[] getServletConfigClasses() {
        return null;

    protected String[] getServletMappings() {
        return new String[] {"/"};

    public void onStartup(final ServletContext servletContext) throws ServletException {
        // this is to fix issues in jersey-srping3
        // org.glassfish.jersey.server.spring.SpringWebApplicationInitializer
        servletContext.setInitParameter("contextConfigLocation", "");

    protected WebApplicationContext createRootApplicationContext() {
        final WebApplicationContext context = super.createRootApplicationContext();
        final ConfigurableEnvironment env = (ConfigurableEnvironment) context.getEnvironment();
        final String profile = (String) env.getSystemProperties().get("env");
        return context;

public class MySecurityInitializer extends AbstractSecurityWebApplicationInitializer {}

@WebServlet(loadOnStartup = 1)
public class MyJerseyApplication extends ResourceConfig {
    public JerseyRestProvisionApplication() {
        property(ServerProperties.WADL_FEATURE_DISABLE, true);

        register(new LoggingFilter(Logger.getLogger(MyJerseyApplication.class.getName()), true));
How To
Basically we configure two servlets:

  • web.servlet.DispatcherServlet: map it to / (notice not /*). This handles Spring Security and Spring MVC.
  • Jersey SpringServlet: map to v1(all rest API starts with v1/).

Spring security is used to do authentication and authorization.

Jersery 1

public class AppInitializer implements WebApplicationInitializer {

    public void onStartup(final ServletContext servletContext) {
        final AnnotationConfigWebApplicationContext rootContext = new AnnotationConfigWebApplicationContext();

        final ConfigurableEnvironment env = rootContext.getEnvironment();

        final String profile = (String) env.getSystemProperties().get("env");

        servletContext.addListener(new ContextLoaderListener(rootContext));

        addSpringServlet(servletContext, profile);
        addSpringJersyServlet(servletContext, env);

    protected void addSpringServlet(final ServletContext servletContext, final String profile) {

        final DispatcherServlet dispatcherServlet = new DispatcherServlet(new GenericWebApplicationContext());

        final Dynamic dispatcherDynamic = servletContext.addServlet("dispatcherServlet", dispatcherServlet);;


                .addFilter(AbstractSecurityWebApplicationInitializer.DEFAULT_FILTER_NAME, DelegatingFilterProxy.class)
                .addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), false, "/*");

    protected void addSpringJersyServlet(final ServletContext servletContext, final ConfigurableEnvironment env) {
        final ServletRegistration.Dynamic appServlet = servletContext.addServlet("jersey-servlet", new SpringServlet());

        appServlet.setInitParameter(JSONConfiguration.FEATURE_POJO_MAPPING, "true");
        appServlet.setInitParameter(ResourceConfig.FEATURE_TRACE, env.getProperty("jersey.enable.trace", "false"));
                env.getProperty("jersey.disable.wadl", "true"));

        appServlet.setInitParameter(ResourceConfig.PROPERTY_CONTAINER_RESPONSE_FILTERS, ..);

Difference between / and /* in servlet mapping url pattern

SSH Tunnel and Eclipse Remote Debug

It's common that your server(destination_server) is running in cloud(AWS) which you have to access via another linux server(proxy_server), and you want to enable remote debug in the destination_server.

To do this, run this in the proxy_server:
ssh -f -N -L9999:localhost:9999 username@destination_server

Run this in your local server:
ssh -f -N -L9999:localhost:9999 username@proxy_server

In your destination_server, add this to the command that is used to start your server.

Then start remote application in eclipse that connects to localhost:9999.

-- If Eclipse is very slow in remote debug mode, try to remove (all) breakpoints and expressions in eclipse.

-- Use ssh tunnel, you can do a lot of other stuff.
-- We can use ssh tunnel to test code locally without really deploy to aws.

ssh -f -N -L18983:remote-host:remote-port $USER@host-in-middle

Create tunnel to remote Cassandra via man-in-the-middle
ssh -f host-in-middle -L 9042:destination-cassandra-server:9042 -N -v

SSH Tunneling Explained
SSH Tunnel - Local and Remote Port Forwarding Explained With Examples

Implementing Inheritance in Spring Solr Data

In our solr data model, there is inheritance relationship between classes.

Please check about why we are mixing Spring Data Solr and SolrJ to make it work with Solr Cloud 5.

In the DAO:
public BaseModel findOne(final String id) {
    final SolrDocument doc = solrServer.getById(getCollection(), id);;
    if (doc == null) {
        throw new WrappedException(ErrorCode.INVALID_PARAMETER, "BaseModel not found: " + id);
    return, doc);

But this will cause Spring Data Solr returns BaseModel object not its concrete sub-classes which we expected.

To fix this, we need change SolrConverter's read method to map to its concrete sub-classes in Spring configuration class.
public static SolrConverter mappingSolrConverter() {
    final MappingContext<? extends SolrPersistentEntity<?>, SolrPersistentProperty> mappingContext =
            new SimpleSolrMappingContext();
    final MappingSolrConverter converter = new MappingSolrConverter(mappingContext) {
        protected <S> S read(final TypeInformation<S> targetTypeInformation, final Map<String, ?> source) {
            final Object obj = source.get(BaseModel.FIELD_TYPE);
            if (obj != null) {
                final String type = String.valueOf(obj);
                switch (type) {
                    case BaseModel.TYPE_A:
                        return (S), source);
                    case BaseModel.TYPE_B:
                        return (S), source);
                    case BaseModel.TYPE_C:
                        return (S), source);
                        throw new IllegalArgumentException(
                                "invalid " + BaseModel.FIELD_TYPE + ", value: " + type);
            return, source);
    return converter;

Mix Spring Data Solr and SolrJ in Solr Cloud 5

We are migrating from Solr 4(part of DataStax) to Solr Cloud 5(without Datastax). 
In previous code, we are using Spring Data Solr to talk with Solr, we are using SolrCrudRepository and also map closed set value to java enum class(not use solr.EnumField), map multiple value field to Set.

As Spring Data Solr doesn't work with Solr Cloud 5, we have to replace the DAO layer(SolrCrudRepository) to Use SolrJ directly.
But SolrJ doesn't support map field to Java Enum or map multiple value field to Set.

We don't want to change the Enum to String, change set to List, as if so, we need change a lot of code. 

We want keep the Model class untouched. So I decided to mix them: Use Spring Data Solr to convert between SolrInputDocument and Java object, use SolrJ to update or query Solr Server.
public abstract class XModelRepository<T extends XModel> implements IXModelRepository<T> {
    private String collection;

    protected SolrClient solrServer;

    protected SolrConverter converter;

    public final void save(final XModel model) {
        try {
            final Iterable<SolrInputDocument> docs = converter.write(Lists.newArrayList(model));
            final UpdateResponse response = solrServer.add(getCollection(), docs.iterator());
        } catch (IOException | SolrServerException e) {
            throw new WrappedException(ErrorCode.INTERNAL_ERROR, e);
    public XModel findOne(final String id) {
        try {
            final SolrDocument doc = solrServer.getById(collection, id);
            return, doc);
        } catch (SolrServerException | IOException e) {
            throw new WrappedException(ErrorCode.INTERNAL_ERROR, e);

This works in most case. But there is one issue: when it stores enum to solr, it saves something like xx.enums.UserType:TypeA, when it reads from solr server, tries to convert the string to enum, it fails with  the following error.
Java.lang.IllegalArgumentException: No enum constant xx.enums.UserType:TypeA
at java.lang.Enum.valueOf(

To fix this, I have to copy Spring and register it in SolrConverter in the configuration class like below:
    public static SolrConverter mappingSolrConverter() {
        final MappingContext<? extends SolrPersistentEntity<?>, SolrPersistentProperty> mappingContext =
                new SimpleSolrMappingContext();
        final MappingSolrConverter converter = new MappingSolrConverter(mappingContext);

        final List<Object> converters = new ArrayList<Object>();
        final DefaultConversionService conversionService = new DefaultConversionService();

        converters.add(new EnumToStringConverter(conversionService));

        final CustomConversions customConversions = new CustomConversions(converters) {
            public boolean isSimpleType(final Class<?> clazz) {
                if (Enum.class.equals(clazz) || Enum.class.isAssignableFrom(clazz)) {
                    return false;
                return super.isSimpleType(clazz);

        return converter;
 // Copied from Spring 
 final class EnumToStringConverter implements Converter<Enum<?>, String>, ConditionalConverter {
     private final ConversionService conversionService;

     public EnumToStringConverter(final ConversionService conversionService) {
         this.conversionService = conversionService;

     public boolean matches(final TypeDescriptor sourceType, final TypeDescriptor targetType) {
         for (@SuppressWarnings("rawtypes")
         final Class interfaceType : ClassUtils.getAllInterfacesForClass(sourceType.getType())) {
             if (this.conversionService.canConvert(TypeDescriptor.valueOf(interfaceType), targetType)) {
                 return false;
         return true;

     public String convert(final Enum<?> source) {

How to Build Better Common Library - Lesson Learned

Make it easy to use
Convention over configuration
Package common shared configuration in library
When we are using spring to build (common enterprise) library that will be used by other teams, it's better that we include common shared configuration- using @Configuration class or XML and all other needed stuff in the library.
The client just need import the configuration class and add needed properties file. 

It would be great if we also include basic property files such as, and client can provide custom property file such as to overwrite it.

So all client need to do is just import the configuration class and add properties file if needed.

Build client library - so client can only import necessary libs
When build a service that is going to be called by other teams, provide a client library that only includes classes, libs that are going to be used by client.

Usually client only need some model classes to build request and parse response, 

Some times, we found that we have to use xxService-common library, which includes too many classes, 3-rd libs that client is not used at all.

Check what libs imported
Remove unneeded dependencies

Build Sample Client Application to demo how to use it
Build sample application to demonstrate how client should call the service, check the built application whether it includes unneeded dependencies.

Split functions into multiple small libraries instead of Big Library

Don't declare same dependencies multiple times
For example: if the commons module import XX lib, the service module imports common, then don't import XX-lib again in service module.

Evolve the Library
-- To use newer framework: from codehaus.jackson to fastxml.jackson, jersey 1 to jersey 2, old spring to newer spring.

Don't constrain client teams choices because they use your library.


Java (159) Lucene-Solr (110) All (60) Interview (59) J2SE (53) Algorithm (37) Eclipse (35) Soft Skills (35) Code Example (31) Linux (26) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (9) Continuous Integration (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Design (8) Dynamic Languages (8) Http Client (8) Maven (8) Security (8) Trouble Shooting (8) bat (8) blogger (8) Big Data (7) Google (7) Guava (7) JSON (7) Problem Solving (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) IDE (5) Lesson Learned (5) Miscs (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) OpenNLP (4) Project Managment (4) Python (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Firefox (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Google Drive (2) Gson (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Bit Operation (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Troubleshooting (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts