Debugging Hystrix CircuitBreaker


The Problem
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.

The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.

After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.

So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.

The circuit will only open after requestVolumeThreshold requests.

So in order to make test easier, I changed the 
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage

@HystrixCommand(fallbackMethod = "fallbackmethod",
commandProperties = {
    // Use this when we want to add debug, so the request will not timeout
    // @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",
    // value = "300000"),
    @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
    // because the third API has to be called in came thread: uses (thread local) @context UriInfo
    @HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE"),
    @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
    @HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "60000"),
    @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "1"),
    @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "50000"),},
ignoreExceptions = {IgnoreThisException.class})

@Configuration
// enable circuit breakers
@EnableCircuitBreaker
Code
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()

com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()


Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).

How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.

Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)