Implement Circuit Breaker Pattern with Netflix Hystrix

When we design services, it's important to make them resilient and prevent cascading failures.

Circuit Breaker Pattern
From -Martin Fowler
The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you'll also want some kind of monitor alert if the circuit breaker trips.

There are several ways to apply circuit breaker pattern in java.

Using Netflix Hystrix

public class GetProductsCommand extends HystrixCommand<Set<Product>> {
  private final GetProductsConfiguration config;
  public GetEntitlementsCommand(final GetProductsConfiguration config, final String token) {
      super(HystrixCommandGroupKey.Factory.asKey("products"),config.getTimeoutInMilliseconds());
      this.config = config;
      this.token = token;
  }

  @Override
  protected Set<Product> run() throws Exception {
   // if it's client error, throws HystrixBadRequestException
   // it will not trigger fallback, not count against failure metrics and thus not trigger the circuit breaker.
  }
  @Override
  protected Set<Product> getFallback() throws Exception {}

  @Component
  public static class GetProductsConfiguration {
      @Autowired
      // auto wire services that's going to be used by GetProductsCommand
      @Value("${cobra.oauth.timeout.milliseconds:1000}")
      private int timeoutInMilliseconds;    
  }
}

Call HystrixCommand asynchronously, Get result later

@Autowired
private GetProductsConfiguration getProductsConfiguration;

// call it asycnchoursly
final Future<Set<Products>> productsFuture = new GetProductsCommand(getProductsConfiguration, token).queue();

// later
final Set<Products> products = productsFuture.get();

Propagating ThreadLocal to HystrixCommand

Sometimes, the service we are calling expects it's called in same http thread - it expects thread local from current http thread

requestAttributes = (ServletRequestAttributes) RequestContextHolder.currentRequestAttributes();

We can get current requestAttributes and pass to HystrixCommand:

public class MyHystrixCommand extends HystrixCommand<Result> {
  private final ServletRequestAttributes requestAttributes;
  public GetEntitlementsCommand() {
      super(HystrixCommandGroupKey.Factory.asKey("default"));
      this.requestAttributes = requestAttributes;
      this.requestAttributes = RequestContextHolder.getRequestAttributes();
      this.thread = Thread.currentThread();      
  }
  @Override
  protected Result run() throws Exception {
    try {
      RequestContextHolder.setRequestAttributes(requestAttributes);
      //do something
    } finally {
      clearThreadLocal();
    }
  }
  private void clearThreadLocal()
  {
    if (Thread.currentThread() != thread) {
      RequestContextHolder.resetRequestAttributes();
    }
    thread = null;
  }
}

Using Spring Cloud Hystrix

Spring cloud wraps Netflix Hystrix to make it easier to use.

First add @EnableCircuitBreaker in spring configuration class.
Then add @HystrixCommand annotation to service methods.

@HystrixCommand(fallbackMethod = "fallBack",
commandProperties = {
        @HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1000"),
        @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "2000")},
ignoreExceptions = {InvalidTokenException.class})
public Set<Product> getProducts() {}

public Set<Product> fallBack() {}

@HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE")

https://github.com/Netflix/Hystrix/wiki/Configuration

• THREAD — it executes on a separate thread and concurrent requests are limited by the number of threads in the thread-pool
• SEMAPHORE — it executes on the calling thread and concurrent requests are limited by the semaphore count

Resources:
Netflix Hystrix How to Use