The Problem - Service: AmazonDynamoDBv2; Status Code: 404
Help one colleague to fix on issue today, in his laptop, when it talks to DynamoDBLocal which is running on default port 8000, it failed with exception:
Help one colleague to fix on issue today, in his laptop, when it talks to DynamoDBLocal which is running on default port 8000, it failed with exception:
com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 404; Error Code: null; Request ID: null)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[aws-java-sdk-core-1.10.10.jar:na]
How To Debug
At first, we thought it was caused by some recent change, so we checked in some other's laptop, the code works fine. Then it seemed to be a problem related with his environment.
We restart DynamoDBLocal (no error during restart), try again, it fail with same error.
Then we try yo stop DynamoDBLocal, still same error. This looks strange now.
As if DynamoDBLocal is not running, it should failed with exception like:
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection refused
As we set log level of org.apache.http.wire to debug, from the log, we can see that AmazonHttpClient established the connection to 8000, and send request like below.
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire >> "POST / HTTP/1.1[\r][\n]"
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire >> "Host: localhost:8000[\r][\n]"
2015-09-10 17:39:08,921 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire >> "X-Amz-Date: 20150911T003908Z[\r][\n]"
This make us think that maybe there is another process is running on port 8000, run command "lsof -i :8080" (lsof -i :8000 -sTCP:LISTEN), it shows there are 2 applications listening on
COMMAND PID USER FD TYPE SIZE/OFF NODE NAME
java 40626 xx 53u IPv6 0t0 TCP *:irdmi (LISTEN)
node 40981 xx 23u IPv4 0t0 TCP localhost:irdmi (LISTEN)
Now, the problem is obvious, the nodejs http server is running on 8000 and accepts all requests to 8000. So we kill the nodejs application, now, our application runs fine.
But why DynamoDBLocal is not throwing exception during start if another application is running on same port?
This issue seems related with nodejs http server.
In another test, I run tomcat at port 8000, then start DynamoDBLocal, it does throw exception, but it continues running, when client calls DynamoDB API, it will fail with error code: 404 or 405.
Lesson Learned:
- Trouble shooting problem is more about thinking what may go wrong.
- Understand the problem, think about it, try it before google search with the error code as Google search may just lead you to totally wrong direction.
- Check when it stops working, and what change u have made before.
