The Problem
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.
The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.
After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.
So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.
The circuit will only open after requestVolumeThreshold requests.
So in order to make test easier, I changed the
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()
com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()
Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).
How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.
Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.
The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.
After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.
So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.
The circuit will only open after requestVolumeThreshold requests.
So in order to make test easier, I changed the
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage
@HystrixCommand(fallbackMethod = "fallbackmethod", commandProperties = { // Use this when we want to add debug, so the request will not timeout // @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", // value = "300000"), @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"), // because the third API has to be called in came thread: uses (thread local) @context UriInfo @HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE"), @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"), @HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "60000"), @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "1"), @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "50000"),}, ignoreExceptions = {IgnoreThisException.class}) @Configuration // enable circuit breakers @EnableCircuitBreakerCode
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()
com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()
Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).
How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.
Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/