Debugging - DynamoDBLocal Status Code: 404

The Problem - Service: AmazonDynamoDBv2; Status Code: 404
Help one colleague to fix on issue today, in his laptop, when it talks to  DynamoDBLocal which is running on default port 8000, it failed with exception:
com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 404; Error Code: null; Request ID: null)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[aws-java-sdk-core-1.10.10.jar:na]

How To Debug
At first, we thought it was caused by some recent change, so we checked in some other's laptop, the code works fine. Then it seemed to be a problem related with his environment.

We restart DynamoDBLocal (no error during restart), try again, it fail with same error.

Then we try yo stop DynamoDBLocal, still same error. This looks strange now. 
As if DynamoDBLocal is not running, it should failed with exception like:
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection refused

As we set log level of org.apache.http.wire to debug, from the log, we can see that AmazonHttpClient established the connection to 8000, and send request like below.
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "POST / HTTP/1.1[\r][\n]"
2015-09-10 17:39:08,920 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "Host: localhost:8000[\r][\n]"
2015-09-10 17:39:08,921 [http-nio-0.0.0.0-8080-exec-3] DEBUG o.a.h.i.c.Wire   >> "X-Amz-Date: 20150911T003908Z[\r][\n]"

This make us think that maybe there is another process is running on port 8000, run command "lsof -i :8080" (lsof -i :8000 -sTCP:LISTEN), it shows there are 2 applications listening on 
COMMAND   PID  USER   FD   TYPE    SIZE/OFF NODE NAME
java    40626  xx   53u  IPv6       0t0  TCP *:irdmi (LISTEN)
node    40981  xx   23u  IPv4       0t0  TCP localhost:irdmi (LISTEN) 

Now, the problem is obvious, the nodejs http server is running on 8000 and accepts all requests to 8000. So we kill the nodejs application, now, our application runs fine.

But why DynamoDBLocal is not throwing exception during start if another application is running on same port?
This issue seems related with nodejs http server.

In another test, I run tomcat at port 8000, then start  DynamoDBLocal, it does throw exception, but it continues running, when client calls DynamoDB API, it will fail with error code: 404 or 405.

Lesson Learned:

  • Trouble shooting problem is more about thinking what may go wrong.
  • Understand the problem, think about it, try it before google search with the error code as Google search may just lead you  to totally wrong direction.
  • Check when it stops working, and what change u have made before.
Post a Comment

Labels

Java (159) Lucene-Solr (110) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (37) Eclipse (33) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) Problem Solving (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) Lesson Learned (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts