Bug Analysis Part 1
- StreamCorruptedException: unexpected block data
We met a field defect recently, client sent a command to server, but didn't get response. At Server there was no log indicating that it received the command.
At first, we thought it maybe a network related problem, as from client logs, we can see user tried same command several times, and run this command immediately after connected to server.
From server logs, we can see each time, after connection was created successfully and the application tried to read command, the application would throw 'StreamCorruptedException: unexpected block data ', and connection would be closed.
But we can't understand that if it’s a network connection problem, why it happened so frequently.
And we shouldn't blame the network easily, without real evidence, right?
From this book Debug It!, we learn that - it’s too easy to point the finger of blame to other components or hardware, we should suspect your own code first.
Meanwhile we didn't find some obvious problems about network.
What we should do next?
One of the most important skills for problem analysis is to think divergently, come up all possibilities, and check and test then in the order of probability.
Look at the Javadoc, StreamCorruptedException:
Thrown when control information that was read from an object stream violates internal consistency checks.
Maybe the connection was good, but only this type of command failed. – We should look at the log: does this client ever successfully executed one of this type of command, does it successfully executed other types of command?
Are the client and server version same and compatible? Version mismatch is a common source of bug in client/server application.
So check the version of client and server. Yep, it's different, client is new, and server is old.
What to do next?
In Debug It!, it teaches us that Always Reproduce the Problem First.
Sometimes we think we are just smart and we can find the root cause of the problem just by thinking, but if we can't reproduce it, how we can verify what we think is really the root cause, and how to verify that our fix does fix the problem in future.
Or sometimes, we are just lazy, as trying to reproduce the problem is usually not easy.
But this always pay off, as if we can find the way to produce the problem, then finding the root cause and fixing it would be easy.
Remember to use exact same version application as one used in client environment is important.
So next we used exact same client/server application and executed the command. Immediately we recreated the problem!
Next part is about the technical stuff.
Simply put, the reason is because in new version client, the command implementation changes, it introduces a new command class, and sends the new command object to server.
The old server doesn't have this command class, and can't deserialize the command object.
But normally, if one class is missing, java deserialization should throw ClassNotFoundException, why StreamCorruptedException? Is our analysis correct?
To figure it the reason, we wrote a simple java application, it writes the command object to the file, and reads it back.
During deserialization, we deleted the new command class, it does throw the StreamCorruptedException.
Check the source code, we figure it out this may be because the command overrides readObject and writeObject.
Sample code:
public class Deserialization
{
public static void main(String[] args) throws Exception
{
writeZippedObject();
readZippedObject();
}
private static void writeZippedObject() throws FileNotFoundException, IOException, SecurityException,
NoSuchMethodException
{
FileOutputStream fos = new FileOutputStream("command");
GZIPOutputStream gos = new GZIPOutputStream(fos);
ObjectOutputStream oos = new ObjectOutputStream(gos);
Method method = Math.class.getMethod("sqrt", new Class[] { double.class });
Command command = new ConfigCommand(method, new Object[] { Integer.valueOf(4) });
oos.writeObject(command);
oos.close();
fos.close();
System.out.println("write command: " + command);
}
private static void readZippedObject() throws FileNotFoundException, IOException, ClassNotFoundException
{
FileInputStream fis = new FileInputStream("command");
GZIPInputStream gis = new GZIPInputStream(fis);
ObjectInputStream ois = new ObjectInputStream(gis);
Command command = (Command) ois.readObject();
System.out.println("read command: " + command);
fis.close();
}
}
class Command implements Serializable
{
private static final long serialVersionUID = 1L;
private String serviceString = null;
private String methodName = null;
protected Object[] params = null;
public Command(Method method, Object[] params)
{
methodName = method.getName();
Class service = method.getDeclaringClass();
serviceString = service.getName();
this.params = params;
}
private Map fieldsTable = new HashMap();
public String toString()
{
return "[serviceString: " + serviceString + ", methodName: " + methodName + "]";
}
private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException
{
fieldsTable = (HashMap) in.readObject();
serviceString = (String) fieldsTable.get("serviceString");
methodName = (String) fieldsTable.get("methodName");
params = (Object[]) fieldsTable.get("params");
}
private void writeObject(ObjectOutputStream out) throws IOException
{
fieldsTable.put("serviceString", serviceString);
fieldsTable.put("methodName", methodName);
fieldsTable.put("params", params);
out.writeUnshared(fieldsTable);
}
}
class ConfigCommand extends Command
{
public ConfigCommand(Method method, Object[] params)
{
super(method, params);
}
private static final long serialVersionUID = 1L;
}
Without readObject/writeObject in Command class and ConfigCommand class, it would throw ClassNotFoundException.
With these 2 serialization-related methods, deserialization without the ConfigCommand would throw ‘StreamCorruptedException: unexpected block data’:
Exception in thread "main" java.io.StreamCorruptedException: unexpected end of block data
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1355)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1951)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1875)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1757)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:352)
at objectio.Deserialization.readZippedObject(Deserialization.java:47)
at objectio.Deserialization.main(Deserialization.java:22)
Lesson Learned
Know your code well, and know its change history well, write them down.
If we know the implementation of this command changed significantly, we might find out the root cause faster.
Read defect description carefully, but keep your mind open, the report might not give too much information, or even give misleading or wrong information.
Think whether the same problem may occur elsewhere, and ensure it not occur again.
Why this bug go into field, and why it is not caught up in unit test and regression test.
Whether our unit test and regression test should be improved?
Client/server mismatch is a very common problem, in development we should think whether our change would cause client/server mismatch, if yes, do unit test carefully for all possible combinations, old client/new server, new client/old server, new client/new server.
And always run regression test for all possible combinations.
