Implement Circuit Breaker Pattern with Netflix Hystrix

When we design services, it's important to make them resilient and prevent cascading failures.

Circuit Breaker Pattern
From -Martin Fowler
The basic idea behind the circuit breaker is very simple. You wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all. Usually you'll also want some kind of monitor alert if the circuit breaker trips.

There are several ways to apply circuit breaker pattern in java.

Using Netflix Hystrix

public class GetProductsCommand extends HystrixCommand<Set<Product>> {
  private final GetProductsConfiguration config;
  public GetEntitlementsCommand(final GetProductsConfiguration config, final String token) {
      super(HystrixCommandGroupKey.Factory.asKey("products"),config.getTimeoutInMilliseconds());
      this.config = config;
      this.token = token;
  }

  @Override
  protected Set<Product> run() throws Exception {
   // if it's client error, throws HystrixBadRequestException
   // it will not trigger fallback, not count against failure metrics and thus not trigger the circuit breaker.
  }
  @Override
  protected Set<Product> getFallback() throws Exception {}

  @Component
  public static class GetProductsConfiguration {
      @Autowired
      // auto wire services that's going to be used by GetProductsCommand
      @Value("${cobra.oauth.timeout.milliseconds:1000}")
      private int timeoutInMilliseconds;    
  }
}

Call HystrixCommand asynchronously, Get result later

@Autowired
private GetProductsConfiguration getProductsConfiguration;

// call it asycnchoursly
final Future<Set<Products>> productsFuture = new GetProductsCommand(getProductsConfiguration, token).queue();

// later
final Set<Products> products = productsFuture.get();

Propagating ThreadLocal to HystrixCommand

Sometimes, the service we are calling expects it's called in same http thread - it expects thread local from current http thread

requestAttributes = (ServletRequestAttributes) RequestContextHolder.currentRequestAttributes();

We can get current requestAttributes and pass to HystrixCommand:

public class MyHystrixCommand extends HystrixCommand<Result> {
  private final ServletRequestAttributes requestAttributes;
  public GetEntitlementsCommand() {
      super(HystrixCommandGroupKey.Factory.asKey("default"));
      this.requestAttributes = requestAttributes;
      this.requestAttributes = RequestContextHolder.getRequestAttributes();
      this.thread = Thread.currentThread();      
  }
  @Override
  protected Result run() throws Exception {
    try {
      RequestContextHolder.setRequestAttributes(requestAttributes);
      //do something
    } finally {
      clearThreadLocal();
    }
  }
  private void clearThreadLocal()
  {
    if (Thread.currentThread() != thread) {
      RequestContextHolder.resetRequestAttributes();
    }
    thread = null;
  }
}

Using Spring Cloud Hystrix

Spring cloud wraps Netflix Hystrix to make it easier to use.

First add @EnableCircuitBreaker in spring configuration class.
Then add @HystrixCommand annotation to service methods.

@HystrixCommand(fallbackMethod = "fallBack",
commandProperties = {
        @HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1000"),
        @HystrixProperty(name = "execution.isolation.semaphore.maxConcurrentRequests", value = "1000"),
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "2000")},
ignoreExceptions = {InvalidTokenException.class})
public Set<Product> getProducts() {}

public Set<Product> fallBack() {}

@HystrixProperty(name = "execution.isolation.strategy", value = "SEMAPHORE")

https://github.com/Netflix/Hystrix/wiki/Configuration

• THREAD — it executes on a separate thread and concurrent requests are limited by the number of threads in the thread-pool
• SEMAPHORE — it executes on the calling thread and concurrent requests are limited by the semaphore count

Resources:
Netflix Hystrix How to Use

How to Build Better Common Library - Lesson Learned

Make it easy to use
Convention over configuration
Package common shared configuration in library
When we are using spring to build (common enterprise) library that will be used by other teams, it's better that we include common shared configuration- using @Configuration class or XML and all other needed stuff in the library.
The client just need import the configuration class and add needed properties file. 

It would be great if we also include basic property files such as thelib_common.properties, and client can provide custom property file such as thelib_overwrite.properties to overwrite it.

So all client need to do is just import the configuration class and add properties file if needed.
@Import({TheLibAppConfig.class})

Build client library - so client can only import necessary libs
When build a service that is going to be called by other teams, provide a client library that only includes classes, libs that are going to be used by client.

Usually client only need some model classes to build request and parse response, 

Some times, we found that we have to use xxService-common library, which includes too many classes, 3-rd libs that client is not used at all.

Check what libs imported

Remove unneeded dependencies

Build Sample Client Application to demo how to use it
Build sample application to demonstrate how client should call the service, check the built application whether it includes unneeded dependencies.

Split functions into multiple small libraries instead of Big Library

Don't declare same dependencies multiple times

For example: if the commons module import XX lib, the service module imports common, then don't import XX-lib again in service module.

Evolve the Library

-- To use newer framework: from codehaus.jackson to fastxml.jackson, jersey 1 to jersey 2, old spring to newer spring.

Don't constrain client teams choices because they use your library.