Debugging Hystrix CircuitBreaker

The Problem
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.

The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.

After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.

So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.

The circuit will only open after requestVolumeThreshold requests.

So in order to make test easier, I changed the 
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage

@HystrixCommand(fallbackMethod = "fallbackmethod",
commandProperties = {
    // Use this when we want to add debug, so the request will not timeout
    // @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",
    // value = "300000"),
    @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
    // because the third API has to be called in came thread: uses (thread local) @context UriInfo
    @HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE"),
    @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
    @HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "60000"),
    @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "1"),
    @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "50000"),},
ignoreExceptions = {IgnoreThisException.class})

@Configuration
// enable circuit breakers
@EnableCircuitBreaker
Code
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()

com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()


Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).

How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.

Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/

NoSuchMethodError - Troubleshooting Practice

The Problem
I am using DynamoDB in our project, and we add prefix before table names in different Environment such as: dev_, or product.

So I modified the code to add prefix which reads from property file based on environment.
@Configuration
@PropertySource("classpath:default/application.properties", "classpath:${env}/application.properties")
public class AppConfig {
    @Value("aws.product.line:")
    private String tableNamePrefix;

    @Bean
    public DynamoDBMapper dynamoDBMapper() {
        final TableNameOverride tableNameOverride = TableNameOverride.withTableNamePrefix(tableNamePrefix);
        final DynamoDBMapperConfig mapperConfig = new DynamoDBMapperConfig(tableNameOverride);
        final DynamoDBMapper mapper = new DynamoDBMapper(amazonDynamoDB(), mapperConfig);
        return mapper;
    }
    @Bean
    public AmazonDynamoDB amazonDynamoDB()...
}

It compiles but it failed with the following exception when I ran it:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper]: Factory method 'dynamoDBMapper' threw exception; nested exception is java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride;
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 60 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride; 

Troubleshooting
Next, I checked the built war, found there are 2 aws-java-sdk:
aws-java-sdk-1.4.2.1.jar
aws-java-sdk-core-1.10.10.jar

In 1.4.2, there is no TableNameOverride.withTableNamePrefix method.

Next, I ran mvn dependency:tree to figure out which library imports and depends on aws-java-sdk, then exclude from it.
<dependency>
    <groupId>xx</groupId>
    <artifactId>xxx</artifactId>
    <exclusions>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-dynamodb</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
        </exclusion>                    
    </exclusions>
</dependency>
ran mvn dependency:tree again to make sure aws-java-sdk-1.4.2 is gone, ran test, all works now.

Misc:
Check http://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#type_signatures:
to know about the type signature meaning:
L fully-qualified-class ;   fully-qualified-class
long f (int n, String s, int[] arr);
has the following type signature:
(ILjava/lang/String;[I)J

Labels

Java (159) Lucene-Solr (110) Interview (59) All (58) J2SE (53) Algorithm (45) Soft Skills (36) Eclipse (34) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) Troubleshooting (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) Problem Solving (7) ANT (6) Coding Skills (6) Database (6) Dynamic Languages (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) IDE (5) Lesson Learned (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts