Writing Rest API to Export Data as CSV File - Jersey

The Problem

Sometimes we need support export data as CSV file, so people can download it and open it in excel and do some analysis.

The Data

The data we want to export it as below:

public class SurveyAnswers{
    public String surveyId;
    // some other meta data about survey, user
    public List<QuestionAnswer> answers;
}
// user's choice for this question
public class QuestionAnswer {
    public int questionId;
    public int optionId;
    // other meta data about the question and the option
}

The date in CSV file would be like this:

SurveyId, dataTime, /*fields about user meta data */, QuestionId1, AnswerId1, QuestionId2, AnswerId2...

The difficult part is that field headers are not fixed due to the list of QuestionAnswer.

Choose CSV Library

I choose Super-CSV, as we want to use its CsvMapWriter. We will convert the data to a hashmap, which contains fields like: QuestionId1, AnswerId1, QuestionId2, AnswerId2 etc.

Check more about superCSV at Writing CSV files

//Update:

It’s better to use CsvDozerBeanReader for our use case.

CsvBeanWritable Interface

In most cases, we are going to implement this interface.

/**
 * Check: http://stackoverflow.com/questions/21942042/using-supercsv-to-change-header-values
 *
 */
public interface CsvBeanWritable {
    /**
     * Header in csv file - it can be anything such as

     * new String[] { "First Name", "Last Name", "Birthday"};
     */
    @JsonIgnore
    public String[] getCsvHeader();
    /**
     * The mapping of bean field to cvv field - order matters

     * new String[] { "firstName", "lastName", "birthDate"};
     */
    @JsonIgnore
    public String[] getCsvMapping();
}

CsvMapWritable Interface

• When logic is complex like in our case, we can’t use CsvBeanWriter, we will implement CsvMapWritable interface.
• CsvMessageBodyWriter will use superCSV CsvMapWriter to write the object as csv data.

public interface CsvMapWritable {
    @JsonIgnore
    public Map<String, Object> getCsvBody();
    @JsonIgnore
    public String[] getCsvHeader();
}
CsvMessageBodyWriter - the Provider
CsvMessageBodyWriter  will marshall object as csv data if the object implements CsvBeanWritable or CsvMapWritable, of the object is a collection of CsvBeanWritable or CsvMapWritable.

We need register it in ResourceConfig or tell jersey to scan the package to find it.

@Component
@Provider
@Produces({CsvMessageBodyWriter.TEXT_CSV, CsvMessageBodyWriter.APPLICATION_EXCEL})
public class CsvMessageBodyWriter<T> implements MessageBodyWriter<T> {
    public static final String TEXT_CSV = "text/csv";
    public static final String APPLICATION_EXCEL = "application/vnd.ms-excel";

    @Override
    public boolean isWriteable(final Class<?> type, final Type genericType, final Annotation[] annotations,
            final MediaType mediaType) {
        return true;
    }
    @Override
    public long getSize(final T data, final Class<?> type, final Type genericType, final Annotation annotations[],
            final MediaType mediaType) {
        return -1;
    }
    @Override
    public void writeTo(final T data, final Class<?> type, final Type genericType, final Annotation[] annotations,
            final MediaType mediaType, final MultivaluedMap<String, Object> httpHeaders,
            final OutputStream entityStream) throws java.io.IOException, javax.ws.rs.WebApplicationException {
        try (AbstractCsvWriter csvWriter = getCsvWriter(data, entityStream)) {
            writeDate(data, csvWriter);
        }
    }
    private AbstractCsvWriter getCsvWriter(final T data, final OutputStream entityStream) {
        AbstractCsvWriter csvWriter = null;
        if (data instanceof CsvBeanWritable) {
            csvWriter = new CsvBeanWriter(new OutputStreamWriter(entityStream), CsvPreference.STANDARD_PREFERENCE);
        } else if (data instanceof CsvMapWritable) {
            csvWriter = new CsvMapWriter(new OutputStreamWriter(entityStream), CsvPreference.STANDARD_PREFERENCE);
        } else if (data instanceof Collection) {
            final Collection<?> collection = (Collection<?>) data;
            csvWriter = getCsvWritterFromCollection(collection, entityStream);
        }
        return csvWriter;
    }


    protected void writeDate(final T data, final AbstractCsvWriter csvWriter) throws IOException {
        if (data instanceof CsvBeanWritable) {
            final CsvBeanWritable writable = (CsvBeanWritable) data;
            csvWriter.writeHeader(writable.getCsvHeader());
            ((CsvBeanWriter) csvWriter).write(writable, writable.getCsvMapping());
        } else if (data instanceof CsvMapWritable) {
            final CsvMapWritable writable = (CsvMapWritable) data;
            csvWriter.writeHeader(writable.getCsvHeader());
            ((CsvMapWriter) csvWriter).write(writable.getCsvBody(), writable.getCsvHeader());
        } else if (data instanceof Collection) {
            writeCollection(data, csvWriter);
        } else {
            throw new XXException("doesn't support download as csv");
        }
    }

    protected void writeCollection(final T data, final AbstractCsvWriter csvWriter) throws IOException {
        final Collection<?> collection = (Collection<?>) data;
        boolean first = true;
        final Iterator<?> it = collection.iterator();
        while (it.hasNext()) {
            final Object obj = it.next();
            if (CsvBeanWritable.class.isAssignableFrom(obj.getClass())) {
                final CsvBeanWritable writable = (CsvBeanWritable) obj;
                if (first) {
                    csvWriter.writeHeader(writable.getCsvHeader());
                    first = false;
                }
                ((CsvBeanWriter) csvWriter).write(writable, writable.getCsvMapping());
            } else if (CsvMapWritable.class.isAssignableFrom(obj.getClass())) {
                final CsvMapWritable writable = (CsvMapWritable) obj;
                if (first) {
                    csvWriter.writeHeader(writable.getCsvHeader());
                    first = false;
                }
                ((CsvMapWriter) csvWriter).write(writable.getCsvBody(), writable.getCsvHeader());
            } else {
                throw new XXException("doesn't support download as csv");
            }
        }
    }
    protected static AbstractCsvWriter getCsvWritterFromCollection(final Collection<?> collection,
            final OutputStream entityStream) {
        AbstractCsvWriter csvWriter = null;
        final Iterator<?> it = collection.iterator();
        while (it.hasNext()) {
            final Object obj = it.next();
            if (CsvMapWritable.class.isAssignableFrom(obj.getClass())) {
                csvWriter = new CsvMapWriter(new OutputStreamWriter(entityStream), CsvPreference.STANDARD_PREFERENCE);
            } else if (CsvBeanWritable.class.isAssignableFrom(obj.getClass())) {
                csvWriter = new CsvBeanWriter(new OutputStreamWriter(entityStream), CsvPreference.STANDARD_PREFERENCE);
            }
        }
        return csvWriter;
    }
}

Add CSV ability to API

 * @param downloadAsFile: downloadAsFile=false, it seems not work in chrome, only work in safari,
 *        not try safari.
@GET
@Produces({MediaType.APPLICATION_JSON, CsvMessageBodyWriter.TEXT_CSV})
public Set getFlatSurveyChoiceResponse(@QueryParam("surveyId") final String surveyId,
        @QueryParam("downloadAsFile") @DefaultValue("true") final boolean downloadAsFile) {
    if (downloadAsFile) {
        servletResponse.addHeader("Content-Disposition", MessageFormat.format("attachment; filename={0}.csv",
                DateUtil.getThreadLocalDateFormat().format(new Date())));
    }
    return service.getSurveyAnswers(surveyId);
}

Using CsvMapWritable in our SurveyAnswers example

Create csv headers for this survey and add it into SurveyHeadersHolder.

for (final Integer questionId : questions) {
    extraHeaders.add(getQuestionIDHeader(questionId));
    extraHeaders.add(getQuestionTitleHeader(questionId));
    extraHeaders.add(getOptionIDHeader(questionId));
    extraHeaders.add(getOptionTextHeader(questionId));
}

SurveyHeadersHolder.INSTANCE.addSurveyHeaders(surveyId, extraHeaders);

Implementing CsvMapWritable in SurveyAnswers

@Override
public Map<String, Object> getCsvBody() {
    final Map<String, Object> map = new LinkedHashMap<>();
    map.put(HEADER_SURVEY_ID, surveyId);
    map.put(HEADER_DATE, DateUtil.getThreadLocalDateFormat().format(date));
    for (final SurveyAnswers answer : answers) {
        map.put(getQuestionIDHeader(answer.getQuestionId()), answer.getQuestionId());
        map.put(getQuestionTitleHeader(answer.getQuestionId()), answer.getQuestionText());
        map.put(getOptionIDHeader(answer.getQuestionId()), answer.getOptionId());
        map.put(getOptionTextHeader(answer.getQuestionId()), answer.getOptionText());
    }
    return map;
}

@Override
public String[] getCsvHeader() {
    return SurveyHeadersHolder.INSTANCE.getSurveyHeaders(surveyId);
}

Merge JSON Objects: Jackson + BeanUtils.copyProperties

The Problem
We store some configuration object as JSON string and want client able to update separate fields directly - without reading it first.

The Solution
In our java configuration class, we use wrapper class for simple types, then Spring BeanUtils.copyProperties(Object source, Object target, String... ignoreProperties) to copy from new config to old config and ignore the null values in new config.

If the configuration has f1=1, f2=1, f3=1, the client only wants to change f3 to 3, then the client can change just send {f3: 3} without having to read the data first.

This is why we use wrapper instead of primitive fields, as the value would be null in new config if the client doesn't specify, we can add all null fields to the ignoreProperties.

public class SimpleConfig implements Serializable {
    private static final long serialVersionUID = 1L;
    private  Boolean featureAEnabled;
    private  Integer minVersion;
    // other fields ignored
}

public void mergeSimpleConfig(final SimpleConfig newConfig) {
    final String configName = COLUMN_SIMPLE_CONFIG;
    final SimpleConfig oldConfig = getSimpleConfig();
    SimpleConfig mergedConfig = (SimpleConfig) SerializationUtils.clone(oldConfig);
    mergeIgnoreNullProperties(newConfig, mergedConfig);

    final ConfigElement newConfigElement = new ConfigElement();
    newConfigElement.setConfigName(configName);
    newConfigElement.setValue(objectMapper.writeValueAsString(mergedConfig));
    newConfigElement.setLastUpdateTime(new Date());
    configDao.saveConfig(newConfigElement);
}

public static <T> void mergeIgnoreNullProperties(final T source, final T target) {
    BeanUtils.copyProperties(source, target, CobraUtil.getNullPropertyNames(source));
}
public static String[] getNullPropertyNames(final Object source) {
    final BeanWrapper src = new BeanWrapperImpl(source);
    final java.beans.PropertyDescriptor[] pds = src.getPropertyDescriptors();

    final Set<String> emptyNames = new HashSet<String>();
    for (final java.beans.PropertyDescriptor pd : pds) {
        final Object srcValue = src.getPropertyValue(pd.getName());
        if (srcValue == null) {
            emptyNames.add(pd.getName());
        }
    }
    final String[] result = new String[emptyNames.size()];
    return emptyNames.toArray(result);
}

@Component
@Provider
public class JerseyObjectMapperProvider implements ContextResolver<ObjectMapper> {
    private static ObjectMapper objectMapper = createFailSafeObjectmapper();

    @Override
    public ObjectMapper getContext(final Class<?> type) {
        return objectMapper;
    }
    public static ObjectMapper createFailSafeObjectmapper() {
        return new ObjectMapper()
                .setDateFormat(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss"))
                .configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false)
                .configure(DeserializationFeature.FAIL_ON_INVALID_SUBTYPE, false)
                .configure(DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES, false)
                .configure(DeserializationFeature.FAIL_ON_NUMBERS_FOR_ENUMS, false)
                .configure(DeserializationFeature.FAIL_ON_READING_DUP_TREE_KEY, false)
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                .configure(DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS, false)
                .setSerializationInclusion(Include.NON_NULL);
    }    
}

Read More
Jackson Essentials - the JSON Libaray
Using Jackson JSON View to Protect Mass Assignment Vulnerabilities
Merge JSON Objects: Jackson + BeanUtils.copyProperties
Jackson Generic Type + Java Type Erasure
Jackson Date Serialize + Deserialize

JAX-RS: capture all error, don't expose internal stack trace

The Goal: Don't expose internal stack trace to client

Capture all Error
When we are developing restful API that are exposed to the internet,  we better capture all exception/error, and don't expose internal stack trace to outside.

In Jersey, we can use ExceptionMapper to capture any kind of exception or error.
For known exception such as ProductNotFound, we can capture it, and return meaningful error message to client.

But bad things always happen, our application may throw NullPointerExceptionn or exception that we don't ever expect or may throw OutOfMemoryError.

When this happens, we need capture it, return some message like: "internal error", and then asynchronously notify engineer owners(send email, store to db or or other approaches).

public class GenericExceptionMapper implements ExceptionMapper {
    private static final Logger LOGGER = LoggerFactory.getLogger(GenericExceptionMapper.class);
    private static final String MSG_INTERNAL_ERROR = "internal error.";
    public static final String HEADER_ERROR_CODE = "X-ErrorCode";

    @Override
    public Response toResponse(Throwable exception) {
        // including exception's type and message in the log message to facilitate log navigation

        // here we capture all error, so no stack trace is exposed to client.
        // TODO alert developers
        LOGGER.error("Unhanded exception detected {}:{}",
                new Object[] {exception.getClass(), exception.getMessage(), exception});

        if (exception instanceof OutOfMemoryError) {
            // notify engineer owner
        } else {
            // may caused by program bug
            // store the error to separate log or db, so we can check them easily
        }

        ResponseBuilder builder = Response.status(Response.Status.INTERNAL_SERVER_ERROR);
        builder.header(HEADER_ERROR_CODE, MSG_INTERNAL_ERROR);
        builder.entity(new ErrorMessage(MSG_INTERNAL_ERROR)).type(MediaType.APPLICATION_JSON);
        return builder.build();
    }
}

Code is the King
How Jersey find exception mapper for specific exception?
com.sun.jersey.spi.container.ContainerResponse.mapException(Throwable)
com.sun.jersey.server.impl.application.ExceptionMapperFactory.find(Class)

It checks all exception mappers, get the exception mapper whose exception type is isAssignableFrom and the distance from current exception is smallest.

If can't find. it will ResponseListener onError, which by default returns html page with error stack trace.

    public ExceptionMapper find(Class c) {
        int distance = Integer.MAX_VALUE;
        ExceptionMapper selectedEm = null;
        for (ExceptionMapperType emt : emts) {
            int d = distance(c, emt.c);
            if (d < distance) { 
                distance = d;
                selectedEm = emt.em;
                if (distance == 0) break;
            }
        }
        
        return selectedEm;
    }
    
    private int distance(Class c, Class emtc) {
        int distance = 0;
        if (!emtc.isAssignableFrom(c))
            return Integer.MAX_VALUE;
        
        while (c != emtc) {
            c = c.getSuperclass();
            distance++;
        }
        
        return distance;
    }