Eclipse: Add another Project as Dependency may Cause Unexpected Exception

The Problem
In local development, we run spring-boot application in eclipse tomcat - as we also deploy the project as a war.

But for some reason, one developer stills run it as a java application, and it fails with error - while it works well when run in (eclipse) tomcat.
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [com.amazonaws.services.s3.AmazonS3]: Factory method 's3Client' threw exception;
nested exception is java.lang.NoSuchMethodError: com.amazonaws.handlers.HandlerChainFactory.getGlobalHandlers()Ljava/util/List;

The Root Cause
Maven uses nearest wins strategy to determine which version to use, So we explicitly specify what version of aws-java-sdk-s3 to use in admin's pom.xml, but one library also implicitly depends on aws-java-sdk-s3 in common-module project. 

run mvn dependency:tree -Dverbose -Dincludes=com.amazonaws:aws-java-sdk-core, which shows that maven chooses the right version.
[INFO] |  \- com.amazonaws:aws-java-sdk-cloudfront:jar:1.11.32:compile
[INFO] |     \- (com.amazonaws:aws-java-sdk-core:jar:1.11.32:compile - omitted for conflict with 1.11.98)

- We can also get which version maven uses and why in eclipse: 
open pom,xml then go to dependency hierarchy tab , select the library in right panel.

I tried to run it as a java application, it works fine. But why it failed in his environment?

I compared the difference between his eclipse setup and mine, and found out that he manually added common-module in admin's Java Build Path -> Projects tab.

Now it's kind of clear why it failed: when we add a project as dependency, Eclipse also includes all libraries it depends on to the project. So now the project includes both versions, and Eclipse chooses the wrong version to use.

I created one bug Bug 514094 - Adding another Project as Dependency Causes Unexpected Exception to track it.

Maven Tips and Tricks - 2016

Maven auto-completion
brew tap homebrew/completions

brew install maven-completion
Then add this to ~/.bash_profile:
if [ -f ~/.maven-completion.bash ]; then
  . ~/.maven-completion.bash
fi

Options
-U forces updates, force maven to download/re-download the dependencies
-o Offline
-q,--quiet Quiet output - only show errors

Skip Test
mvn install -DskipTests
This will skip compiling the test:
mvn install -Dmaven.test.skip=true

-DskipTests will compile test classes but not run them
-Dmaven.test.skip=true will skip compiling the tests

How maven resolve versions
Maven uses nearest wins strategy to determine dependency version.
mvn dependency:tree -Dverbose -Dincludes=com.amazonaws:aws-java-sdk-core
[INFO] |  \- com.amazonaws:aws-java-sdk-cloudfront:jar:1.11.32:compile

[INFO] |     \- (com.amazonaws:aws-java-sdk-core:jar:1.11.32:compile - omitted for conflict with 1.11.98)

mvn dependency:build-classpath

Purge dependencies that may be corrupted
mvn dependency:purge-local-repository

Download source jars from maven repository
-DdownloadSources=true -DdownloadJavadocs=true

Download all dependencies into target/dependencies
mvn dependency:copy-dependencies Check how libraries are imported
mvn dependency:tree
Check several specific libraries
mvn dependency:tree -Dverbose -Dincludes=log4j:log4j
Find out conflicted(multiple) version
mvn dependency:tree -Dverbose | grep 'omitted for conflict'

Use dependency:analyze to find out unused dependencies
Use maven-enforcer-plugin(DependencyConvergence) to prevent multiple versions of same library

Use scala-maven-plugin for mixed java/scala projects

Properties
<properties>
  <m2eclipse.wtp.contextRoot>your-app-name</m2eclipse.wtp.contextRoot>
</properties>

Testing
Run specific method
mvn test -Dtest=className#methodName

mvn exec:java -Dexec.mainClass=full.qualified.class -Dexec.classpathScope=test -Dexec.args=local

NoSuchMethodError - Troubleshooting Practice

The Problem
I am using DynamoDB in our project, and we add prefix before table names in different Environment such as: dev_, or product.

So I modified the code to add prefix which reads from property file based on environment.

@Configuration
@PropertySource("classpath:default/application.properties", "classpath:${env}/application.properties")
public class AppConfig {
    @Value("aws.product.line:")
    private String tableNamePrefix;

    @Bean
    public DynamoDBMapper dynamoDBMapper() {
        final TableNameOverride tableNameOverride = TableNameOverride.withTableNamePrefix(tableNamePrefix);
        final DynamoDBMapperConfig mapperConfig = new DynamoDBMapperConfig(tableNameOverride);
        final DynamoDBMapper mapper = new DynamoDBMapper(amazonDynamoDB(), mapperConfig);
        return mapper;
    }
    @Bean
    public AmazonDynamoDB amazonDynamoDB()...
}

It compiles but it failed with the following exception when I ran it:
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper]: Factory method 'dynamoDBMapper' threw exception; nested exception is java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride;
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 60 more
Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapperConfig$TableNameOverride.withTableNamePrefix(Ljava/lang/String;)Lcom/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig$TableNameOverride; 

Troubleshooting
Next, I checked the built war, found there are 2 aws-java-sdk:
aws-java-sdk-1.4.2.1.jar
aws-java-sdk-core-1.10.10.jar

In 1.4.2, there is no TableNameOverride.withTableNamePrefix method.

Next, I ran mvn dependency:tree to figure out which library imports and depends on aws-java-sdk, then exclude from it.

<dependency>
    <groupId>xx</groupId>
    <artifactId>xxx</artifactId>
    <exclusions>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-dynamodb</artifactId>
        </exclusion>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk</artifactId>
        </exclusion>                    
    </exclusions>
</dependency>

ran mvn dependency:tree again to make sure aws-java-sdk-1.4.2 is gone, ran test, all works now.

Misc:
Check http://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#type_signatures:
to know about the type signature meaning:
L fully-qualified-class ;   fully-qualified-class
long f (int n, String s, int[] arr);
has the following type signature:
(ILjava/lang/String;[I)J

Lazy Developer: Using git pre-commit to auto tun test and findbugs

The Goal:

We developers wish our code is perfect, no bug, and we never want to check in code that will break test and found out later by peers.

We should always use tools like findbugs to check our code and run test locally before we push the change to git repository.

But would be better if git could just run test, check bugs for us before commit or push?

Using git pre-commit to run mvn test automatically

Luckily we can use git pre-commit hooks to do this for us.

We just need create pre-commit file in .git/hooks folder, make it executable(chmod 755), In this script, we can do anything we want.

In the following script, it will check whether we changed java code or pom file in this commit, if no, just commit it(we don't have test for UI side yet).

If yes, it will first stash(save and hide) unstaged changes before running tests, so we are really testing code that will be committed. 

#!/bin/bash
# At root folder project: run ln -s ../../utils/git/pre-commit .git/hooks/pre-commit
echo "running pre-commit hooks"

STAGED_FILES_CMD=`git diff --cached --name-only | grep -E "(\.java$)|(pom.xml)"`
# Determine if a file list is passed
ECHO "INPUT: $#"
if [ "$#" -eq 1 ]
then
 oIFS=$IFS
 IFS='
 '
 SFILES="$1"
 IFS=$oIFS
fi
SFILES=${SFILES:-$STAGED_FILES_CMD}

echo "STAGED_FILES_CMD: $STAGED_FILES_CMD, SFILES: $SFILES"
# Determine whether need run maven java test
if [ "$SFILES" == "" ]
then
 echo "no need to run maven test";
 exit 0;
fi

echo "Running maven clean test for errors"

# Stash changes
git stash -q --keep-index

# retrieving current working directory, if don't want to call this hook, run git commit --no-verify
CWD=`pwd`
MAIN_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# go to main project dir
cd $MAIN_DIR/../../
mkdir logs/

set -o pipefail
/usr/local/bin/mvn clean test package | tee logs/mvn.log

RESULT=$?
# Restore changes
git stash pop -q

if [ $RESULT -ne 0 ]; then
  echo "Error while testing the code"
  # in case when we run it in eclipse.
  open logs/mvn.log
  # go back to current working dir
  cd $CWD
  exit 1
fi
# go back to current working dir
cd $CWD

Add pre-commit hooks into git

Each developer should run ln -s ../../tools/git/pre-commit .git/hooks/pre-commit

So if we make any change, others can just pick the change.

Run findbugs during compile, fail build if find bugs

Add findbugs to pom:

<plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>findbugs-maven-plugin</artifactId>
 <version>${org.codehaus.mojo.version}</version>
 <configuration>
  <effort>Max</effort>
  <threshold>Medium</threshold>
  <xmlOutput>true</xmlOutput>
  <failOnError>true</failOnError>
 </configuration>
 <executions>
  <!-- Ensures that FindBugs inspects source code when project is compiled. -->
  <execution>
   <id>analyze-compile</id>
   <phase>compile</phase>
   <goals>
    <goal>check</goal>
   </goals>
  </execution>
 </executions>
</plugin>

We also added Google Error Prone as a step of mvn compile
Resources:

Tips for using a git pre-commit hook

Enforce code standards with composer, git hooks, and phpcs

Symbolic link to a hook in git

Maven: Non Existing Library jdk.tools.jar

The Problem
Today,after add some dependencies to maven,  I found that maven refuses to compile. In Problems view, it shows error:
The container 'Maven Dependencies' references non existing library 'C:\Users\administrator\.m2\repository\jdk\tools\jdk.tools\1.6\jdk.tools-1.6.jar'

Check my pom.xml, there is no direct dependency of jdk.tools-1.6.jar, then use maven dependency:tree tool to figure out which lib indirectly depends on it.
mvn dependency:tree -Dverbose -Dincludes=jdk.tools

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ learningLucene ---

[INFO] org.lifelongprogrammer:learningLucene:jar:1.0

[INFO] \- org.apache.solr:solr-core:jar:4.9.0:compile

[INFO]    \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile

[INFO]       \- jdk.tools:jdk.tools:jar:1.7:system

Turn out, solr-code depends on hadoop-annotation which then need jdk.tools.jar.

The Solution

Google search, find this page: 

http://maven.apache.org/general.html#tools-jar-dependency

<dependency>
  <groupId>jdk.tools</groupId>
  <artifactId>jdk.tools</artifactId>
  <scope>system</scope>
  <systemPath>C:/Program Files/Java/jdk1.8.0/lib/tools.jar</systemPath>
  <!-- have to include the version, otherwise eclipse throws exception: 
    Errors running builder 'Maven Project Builder' on project 'learningLucene'. 
    java.lang.NullPointerException -->
  <version>8.0</version>
</dependency>
</dependencies>
<build>
<plugins>
  <plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
      <source>1.8</source>
      <target>1.8</target>
    </configuration>
  </plugin>
</plugins>
</build>

Using Maven with Google App Engine

Maven is very good at managing the project's dependencies, so I also choose maven when develop GAE project.

Google Eclipse plugin doesn't support GAE maven development very well: we can't use Google Eclipse plugin to directly run, debug the app or deploy it to app engine.

To run the app in local GAE server:
cd ${mypp}\${mypp-ear}
mvn -f ..\pom.xml clean install && mvn appengine:devserver

To debug the app: add the following in pom.xml:

<plugins>
  <plugin>
    <groupId>com.google.appengine</groupId>
    <artifactId>appengine-maven-plugin</artifactId>
    <configuration>
      <jvmFlags>
        <jvmFlag>-Xdebug</jvmFlag>
        <jvmFlag>-agentlib:jdwp=transport=dt_socket,address=9999,server=y,suspend=n</jvmFlag>
      </jvmFlags>
      <disableUpdateCheck>true</disableUpdateCheck>
    </configuration>
  </plugin>
</plugins>

Start the local GAE server, then create a remote application to connect to localhost:9999. Now we can debug the GAE maven application in eclipse.

Change Application Id

For some reason, we may want to deploy the same application with multiple application id. - We may use GAE as backbone application, our client application maybe mobile app or even google blogger(as google doesn't allow to put ads in GAE app, we may use google bloger as the front side which talks with GAE server to do real task.).

When our application is getting popular, and exceeds the free quota. We may want to duplicate our applition to deploy under another application id.

If we are using maven to build and deploy, we need change the application id: ${mypp}\${mypp-ear}\src\main\application\META-INF\appengine-application.xml.

Then deploy it to the new application id:

cd ${mypp}\${mypp-ear}

mvn -f ..\pom.xml clean install && mvn appengine:update

Resources

GAE Using Apache Maven

Maven: Invalid maximum heap size

The Problem
I follow the stanbol tutorial to build stanbol. 
export MAVEN_OPTS="-Xmx1024M -XX:MaxPermSize=256M"

As I am building stanbol in Windows, so I changed it to:
set MAVEN_OPTS="-Xmx1024M -XX:MaxPermSize=256M"

Then whenI run "mvn clean install -Dmaven.test.skip=true", it failed with error:
Invalid maximum heap size: -Xmx1024M -XX:MaxPermSize=256M
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The Solution
Check mvn.bat: the MAVEN_OPTS is passed as JVM parameter directly:
SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe"
%MAVEN_JAVA_EXE% %MAVEN_OPTS% -classpath %CLASSWORLDS_JAR% "-Dclassworlds.conf=%M2_HOME%\bin\m2.conf" "-Dmaven.home=%M2_HOME%" %CLASSWORLDS_LAUNCHER% %MAVEN_CMD_LINE_ARGS%

If we run set MAVEN_OPTS="-Xmx1024M -XX:MaxPermSize=256M",the previous command would be replaced with %MAVEN_JAVA_EXE% "-Xmx1024M -XX:MaxPermSize=256M" ...

If we run java "-Xmx1024M -XX:MaxermSize=256M" -version: we will see same error message.

The solution is simple: 
We can run:
set "MAVEN_OPTS=-Xmx1024M -XX:MaxPermSize=256M"
 or remove the double quotes completely:
set MAVEN_OPTS=-Xmx1024M -XX:MaxPermSize=256M

The value of MAVEN_OPTS would be with no surrounding double quotes.

Now, we know why windows version batch file (mvn.bat) failed. But why the Linux version succeed? Check Linux shell file: mvn: it doesn't call java command directly, but call exec command.
exec "$JAVACMD" \
  $MAVEN_OPTS \
  -classpath "${M2_HOME}"/boot/plexus-classworlds-*.jar \
  "-Dclassworlds.conf=${M2_HOME}/bin/m2.conf" \
  "-Dmaven.home=${M2_HOME}"  \
  ${CLASSWORLDS_LAUNCHER} "$@"

Compile Using Memory Allocation Enhancements in pom.xml

Customizing Eclipse Project Names With maven-eclipse-plugin

The Problem
When learning Apache UIMA, I need to import it into eclipse. Also I need swtich between different UIMA version 2.4.2 and the latest(trunk) version frequently.

I can use maven-eclipse-plugin to import uima projects to eclipse. But I have to add a prefix to these uima projects.

The Solution
Check maven-eclipse-plugin, it provides several ways to change eclipse project name.
addGroupIdToProjectName
If set to true, the groupId of the artifact is appended to the name of the generated Eclipse project.
Expression: ${eclipse.addGroupIdToProjectName}

addVersionToProjectName
If set to true, the version number of the artifact is appended to the name of the generated Eclipse project. See projectNameTemplate for other options.
Expression: ${eclipse.addVersionToProjectName}

projectNameTemplate
Allows configuring the name of the eclipse projects. This property ifn set wins over addVersionToProjectName and addGroupIdToProjectName You can use [groupId], [artifactId] and [version] variables. eg. [groupId].[artifactId]-[version]
Expression: ${eclipse.projectNameTemplate}

So now the solution is obvious, we can add -Declipse.addVersionToProjectName=true
mvn -Declipse.addVersionToProjectName=true eclipse:eclipse
The project name would be updated as uimaj-core-2.4.2 or uimaj-core-2.5.1.


Or we can add -Declipse.projectNameTemplate=Prefix-[artifactId]-[version] 

Miscs
Use -DdownloadSources=true to tell maven to download source code.

Resource
Maven Eclipse Plugin
Get source jar files attached to Eclipse for Maven-managed dependencies