The Problem
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.
The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.
After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.
So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.
The circuit will only open after requestVolumeThreshold requests.
So in order to make test easier, I changed the
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage
com.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()
com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()
Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).
How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.
Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/
We are using netflix CircuitBreaker in our project when call third party API via network, I am trying to debug and make sure the code works.
The caller calls @HystrixCommand annotated method via proxy(not directly), when it fails with exception(not within ignoreExceptions), it will call fallback method.
After it fails several times, i expected it bypasses @HystrixCommand annotated method and calls fallback method directly, but seems it doesn't work as I expect.
So I took some time to debug hystrix code, and fount out that:
In every metrics.rollingStats.timeInMilliseconds (default 10 seconds) window, after the first (circuitBreaker.requestVolumeThreshold[default 20]-1) requests, if error rates exceeds circuitBreaker.errorThresholdPercentage (default 50%), it will open the circuit, and reject request, also after circuitBreaker.sleepWindowInMilliseconds (default 5 seconds), it will try some requests to check whether it will succeed.
The circuit will only open after requestVolumeThreshold requests.
So in order to make test easier, I changed the
The caller calls @HystrixCommand like below:
increase the window: metrics.rollingStats.timeInMilliseconds
decrease circuitBreaker.requestVolumeThreshold to 1
decrease error rate circuitBreaker.errorThresholdPercentage
@HystrixCommand(fallbackMethod = "fallbackmethod",
commandProperties = {
// Use this when we want to add debug, so the request will not timeout
// @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds",
// value = "300000"),
@HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
// because the third API has to be called in came thread: uses (thread local) @context UriInfo
@HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE"),
@HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
@HystrixProperty(name = "metrics.rollingStats.timeInMilliseconds", value = "60000"),
@HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "1"),
@HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "50000"),},
ignoreExceptions = {IgnoreThisException.class})
@Configuration
// enable circuit breakers
@EnableCircuitBreaker
Codecom.netflix.hystrix.HystrixCircuitBreaker.HystrixCircuitBreakerImpl
allowRequest()
isOpen()
allowSingleTest()
com.netflix.hystrix.HystrixCommandMetrics.getHealthCounts()
Miscs
How Spring annotation(like @HystrixCommand, @Async) works?
@HystrixCommand only works because Spring makes a proxy to call that method on. If we call the method in the same class, it doesn't go through the interceptor. We need to call the @HystrixCommand from another @Component (or use AspectJ).
How do we know whether the configuration works?
If the caller calls @HystrixCommand annotated method directly, this means the configuration doesn't work.
Resources
Hystrix Configuration
http://callistaenterprise.se/blogg/teknik/2015/04/10/building-microservices-with-spring-cloud-and-netflix-oss-part-1/
http://callistaenterprise.se/blogg/teknik/2015/04/15/building-microservices-with-spring-cloud-and-netflix-oss-part-2/