Bug Analysis Part 1

Bug Analysis Part 1

- StreamCorruptedException: unexpected block data

We met a field defect recently, client sent a command to server, but didn't get response. At Server there was no log indicating that it received the command.

At first, we thought it maybe a network related problem, as from client logs, we can see user tried same command several times, and run this command immediately after connected to server.

From server logs, we can see each time, after connection was created successfully and the application tried to read command, the application would throw 'StreamCorruptedException: unexpected block data ', and connection would be closed.

But we can't understand that if it’s a network connection problem, why it happened so frequently.
And we shouldn't blame the network easily, without real evidence, right?

From this book Debug It!, we learn that - it’s too easy to point the finger of blame to other components or hardware, we should suspect your own code first.

Meanwhile we didn't find some obvious problems about network.

What we should do next?
One of the most important skills for problem analysis is to think divergently, come up all possibilities, and check and test then in the order of probability.

Look at the Javadoc, StreamCorruptedException:
Thrown when control information that was read from an object stream violates internal consistency checks.

Maybe the connection was good, but only this type of command failed. – We should look at the log: does this client ever successfully executed one of this type of command, does it successfully executed other types of command?

Are the client and server version same and compatible? Version mismatch is a common source of bug in client/server application.

So check the version of client and server. Yep, it's different, client is new, and server is old.

What to do next?
In Debug It!, it teaches us that Always Reproduce the Problem First.
Sometimes we think we are just smart and we can find the root cause of the problem just by thinking, but if we can't reproduce it, how we can verify what we think is really the root cause, and how to verify that our fix does fix the problem in future.
Or sometimes, we are just lazy, as trying to reproduce the problem is usually not easy.
But this always pay off, as if we can find the way to produce the problem, then finding the root cause and fixing it would be easy.
Remember to use exact same version application as one used in client environment is important.

So next we used exact same client/server application and executed the command. Immediately we recreated the problem!

Next part is about the technical stuff.
Simply put, the reason is because in new version client, the command implementation changes, it introduces a new command class, and sends the new command object to server.

The old server doesn't have this command class, and can't deserialize the command object.

But normally, if one class is missing, java deserialization should throw ClassNotFoundException, why StreamCorruptedException? Is our analysis correct?

To figure it the reason, we wrote a simple java application, it writes the command object to the file, and reads it back.

During deserialization, we deleted the new command class, it does throw the StreamCorruptedException.

Check the source code, we figure it out this may be because the command overrides readObject and writeObject.

Sample code:
public class Deserialization
    public static void main(String[] args) throws Exception

    private static void writeZippedObject() throws FileNotFoundException, IOException, SecurityException,
       FileOutputStream fos = new FileOutputStream("command");
       GZIPOutputStream gos = new GZIPOutputStream(fos);
       ObjectOutputStream oos = new ObjectOutputStream(gos);

       Method method = Math.class.getMethod("sqrt", new Class[] { double.class });
       Command command = new ConfigCommand(method, new Object[] { Integer.valueOf(4) });
       System.out.println("write command: " + command);

    private static void readZippedObject() throws FileNotFoundException, IOException, ClassNotFoundException
       FileInputStream fis = new FileInputStream("command");
       GZIPInputStream gis = new GZIPInputStream(fis);
       ObjectInputStream ois = new ObjectInputStream(gis);

       Command command = (Command) ois.readObject();
       System.out.println("read command: " + command);

class Command implements Serializable
    private static final long serialVersionUID = 1L;
    private String serviceString = null;
    private String methodName = null;
    protected Object[] params = null;

    public Command(Method method, Object[] params)
       methodName = method.getName();
       Class service = method.getDeclaringClass();
       serviceString = service.getName();
       this.params = params;

    private Map fieldsTable = new HashMap();

    public String toString()
       return "[serviceString: " + serviceString + ", methodName: " + methodName + "]";

    private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException
       fieldsTable = (HashMap) in.readObject();
       serviceString = (String) fieldsTable.get("serviceString");
       methodName = (String) fieldsTable.get("methodName");
       params = (Object[]) fieldsTable.get("params");

    private void writeObject(ObjectOutputStream out) throws IOException
       fieldsTable.put("serviceString", serviceString);
       fieldsTable.put("methodName", methodName);
       fieldsTable.put("params", params);

class ConfigCommand extends Command
    public ConfigCommand(Method method, Object[] params)
       super(method, params);
    private static final long serialVersionUID = 1L;

Without readObject/writeObject in Command class and ConfigCommand class, it would throw ClassNotFoundException.

With these 2 serialization-related methods, deserialization without the ConfigCommand would throw ‘StreamCorruptedException: unexpected block data’:
Exception in thread "main" java.io.StreamCorruptedException: unexpected end of block data
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1951)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1875)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1757)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:352)
    at objectio.Deserialization.readZippedObject(Deserialization.java:47)
    at objectio.Deserialization.main(Deserialization.java:22)

Lesson Learned
Know your code well, and know its change history well, write them down.
If we know the implementation of this command changed significantly, we might find out the root cause faster.

Read defect description carefully, but keep your mind open, the report might not give too much information, or even give misleading or wrong information.

Think whether the same problem may occur elsewhere, and ensure it not occur again.

Why this bug go into field, and why it is not caught up in unit test and regression test.
Whether our unit test and regression test should be improved?

Client/server mismatch is a very common problem, in development we should think whether our change would cause client/server mismatch, if yes, do unit test carefully for all possible combinations, old client/new server, new client/old server, new client/new server.
And always run regression test for all possible combinations.
Post a Comment


Java (159) Lucene-Solr (110) All (60) Interview (59) J2SE (53) Algorithm (37) Eclipse (35) Soft Skills (35) Code Example (31) Linux (26) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (9) Continuous Integration (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Design (8) Dynamic Languages (8) Http Client (8) Maven (8) Security (8) Trouble Shooting (8) bat (8) blogger (8) Big Data (7) Google (7) Guava (7) JSON (7) Problem Solving (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) IDE (5) Lesson Learned (5) Miscs (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) OpenNLP (4) Project Managment (4) Python (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Firefox (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Google Drive (2) Gson (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Bit Operation (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Troubleshooting (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts