How to Build Better Common Library - Lesson Learned

Make it easy to use
Convention over configuration
Package common shared configuration in library
When we are using spring to build (common enterprise) library that will be used by other teams, it's better that we include common shared configuration- using @Configuration class or XML and all other needed stuff in the library.
The client just need import the configuration class and add needed properties file. 

It would be great if we also include basic property files such as thelib_common.properties, and client can provide custom property file such as thelib_overwrite.properties to overwrite it.

So all client need to do is just import the configuration class and add properties file if needed.
@Import({TheLibAppConfig.class})

Build client library - so client can only import necessary libs
When build a service that is going to be called by other teams, provide a client library that only includes classes, libs that are going to be used by client.

Usually client only need some model classes to build request and parse response, 

Some times, we found that we have to use xxService-common library, which includes too many classes, 3-rd libs that client is not used at all.

Check what libs imported

Remove unneeded dependencies

Build Sample Client Application to demo how to use it
Build sample application to demonstrate how client should call the service, check the built application whether it includes unneeded dependencies.

Split functions into multiple small libraries instead of Big Library

Don't declare same dependencies multiple times

For example: if the commons module import XX lib, the service module imports common, then don't import XX-lib again in service module.

Evolve the Library

-- To use newer framework: from codehaus.jackson to fastxml.jackson, jersey 1 to jersey 2, old spring to newer spring.

Don't constrain client teams choices because they use your library.

Debugging Hystrix CircuitBreaker

The Problem
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.

The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.

After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.

So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.

The circuit will only open after requestVolumeThreshold requests.

So in order to make test easier, I changed the 
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage

@HystrixCommand(fallbackMethod = "fallbackmethod",
commandProperties = {
    // Use this when we want to add debug, so the request will not timeout
    // @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",
    // value = "300000"),
    @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
    // because the third API has to be called in came thread: uses (thread local) @context UriInfo
    @HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE"),
    @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
    @HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "60000"),
    @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "1"),
    @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "50000"),},
ignoreExceptions = {IgnoreThisException.class})

@Configuration
// enable circuit breakers
@EnableCircuitBreaker

Code
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()

com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()

Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).

How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.

Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/

NoSuchMethodError - Troubleshooting Practice

The Problem
I am using DynamoDB in our project, and we add prefix before table names in different Environment such as: dev_, or product.

So I modified the code to add prefix which reads from property file based on environment.

@Configuration
@PropertySource("classpath:default/application.properties", "classpath:${env}/application.properties")
public class AppConfig {
    @Value("aws.product.line:")
    private String tableNamePrefix;

    @Bean
    public DynamoDBMapper dynamoDBMapper() {
        final TableNameOverride tableNameOverride = TableNameOverride.withTableNamePrefix(tableNamePrefix);
        final DynamoDBMapperConfig mapperConfig = new DynamoDBMapperConfig(tableNameOverride);
        final DynamoDBMapper mapper = new DynamoDBMapper(amazonDynamoDB(), mapperConfig);
        return mapper;
    }
    @Bean
    public AmazonDynamoDB amazonDynamoDB()...
}

It compiles but it failed with the following exception when I ran it:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper]: Factory method 'dynamoDBMapper' threw exception; nested exception is java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride;
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 60 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride; 

Troubleshooting
Next, I checked the built war, found there are 2 aws-java-sdk:
aws-java-sdk-1.4.2.1.jar
aws-java-sdk-core-1.10.10.jar

In 1.4.2, there is no TableNameOverride.withTableNamePrefix method.

Next, I ran mvn dependency:tree to figure out which library imports and depends on aws-java-sdk, then exclude from it.

<dependency>
    <groupId>xx</groupId>
    <artifactId>xxx</artifactId>
    <exclusions>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-dynamodb</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
        </exclusion>                    
    </exclusions>
</dependency>

ran mvn dependency:tree again to make sure aws-java-sdk-1.4.2 is gone, ran test, all works now.

Misc:
Check http://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#type_signatures:
to know about the type signature meaning:
L fully-qualified-class ;   fully-qualified-class
long f (int n, String s, int[] arr);
has the following type signature:
(ILjava/lang/String;[I)J