Daily Technical Logs

Daily Technical Logs

Create a project from an existing ANT buildfile:
Click on File->New->Project
Select "Java Project from existing Ant Buildfile" and click "next"
Export your project settings to an ANT build file:
In your navigator window, look for a file called ".project"
Right click on the file and click on "export"
On the window that pops up, select "Ant Buildfiles" and click on "next"

Linux date command
-s date, --set date
+format - Display current date in a nonstandard format.
date +"Hello%t Date is %D %n%t Time is %T"

Set Date
date -s "+2 days", date -s "-2 days",
The time unit can also be days, months, years, mins, hours, seconds etc.
date +%Y%m%d -s "20081128"

Set Time
date +%T -s "10:13:13"
FTP - mput and mget with no confirmation
ftp -i ftp.test.com
ftp> prompt
-i
Turns off interactive prompting during multiple file transfers.
prompt
   Toggle interactive prompting.
Setting bash as default shell
which bash
chsh -s /usr/local/bin/bash user
Port Forwarding Example
iptables -I FORWARD -p tcp -i eth2 -d 172.16.1.101 --dport 23 -j ACCEPT
iptables -I FORWARD -p tcp -i 172.16.0.3 -s 172.16.1.101 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -i eth2 -d ${PubIP} --dport 12301 -j DNAT --to 172.16.1.101:23
iptables -t nat -A POSTROUTING -o 172.16.0.3 -d 172.16.1.101 -j SNAT --to-source ${PrivateIP}

Unable to run shutdown command
# shutdown -r now
-bash: /sbin/shutdown: Permission denied
# ls -lart /sbin/shutdown
-rw-r--r-- 1 root root 18880 2007-09-29 10:01 /sbin/shutdown
# chmod a+x /sbin/shutdown

Constant Value in Java Class Binary

Constant Value in Java Class Binary

To do unit test or regression, we create a delta sandbox from base level code, and then upload the java source modified to the build machine, and run ant to build it.
It would only compile the modified java sources and build a jar with other old class files.

Today I find an interesting and weird thing.

I change the constant value in one source class. During my unit test, I find that in other classes, which I don't change, and are not recompiled, it still uses old value of the constant. This is weird.

I am wondering this may be because java class byte-code hard-codes the constant value in the class file for efficiency or other reasons.

So I write a small program to verify it.

A class that defines the constant:
public class Constants
{
    public static final int CONSTANT = 12345678;
}
A class - ConstantsReader that uses the constant:
public class ConstantsReader
{
    public static int getConstant()
    {
       return Constants.CONSTANT;
    }
}
A class - Main that prints out the value of the constant seen in ConstantsReader class.
public class Main
{
    public static void main(String[] args)
    {
       System.err.println("Constants.CONSTAN: " + ConstantsReader.getConstant());
    }
}
The copy these classes to another folder, and run Main class, it prints out:
Constants.CONSTAN: 12345678

Later, I change the constant value to 87654321, and copy the modified class ConstantsReader to the folder. Run Main class, we can see that it still prints old value:
Constants.CONSTAN: 12345678
Then I use Beyond Compare to compare the old class file of ConstantsReader and with the new one in Eclipse in Hex Viewer.
We can see that the constant value Constants.CONSTANT is replaced with literal constant value, ‘12345678’ in old version, ‘87654321’ in new version.

Use ‘javap -verbose -c -private ConstantsReader’ to compre the old and new version of the binary class.
Old version:
const #22 = String      #23;    //  Constants.CONSTAN: 12345678
const #23 = Asciz       Constants.CONSTAN: 12345678;
const #30 = int 12345678;
public static int getConstant();
 Code:
  Stack=2, Locals=0, Args_size=0
  0:   getstatic       #16; //Field java/lang/System.err:Ljava/io/PrintStream;
  3:   ldc     #22; //String Constants.CONSTAN: 12345678
  5:   invokevirtual   #24; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
  8:   ldc     #30; //int 12345678
  10:  ireturn
 LineNumberTable:
  line 5: 0
  line 6: 8

New version:
const #22 = String      #23;    //  Constants.CONSTAN: 87654321
const #23 = Asciz       Constants.CONSTAN: 87654321;
public static int getConstant();
 Code:
  Stack=2, Locals=0, Args_size=0
  0:   getstatic       #16; //Field java/lang/System.err:Ljava/io/PrintStream;
  3:   ldc     #22; //String Constants.CONSTAN: 87654321
  5:   invokevirtual   #24; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
  8:   ldc     #30; //int 87654321
  10:  ireturn
 LineNumberTable:
  line 5: 0
  line 6: 8

So my guess is verified, and it’s fun to know this truth.

Tricks for Debugging in Eclipse

Tips and Tricks for Debugging in Eclipse

Breakpoint Properties
After setting a breakpoint you can select the properties of the breakpoint to for example use a condition to restrict when the breakpoint should get toggeled
There are 3 settings that you can tweak.

1.          Hit count – Break only when code flow passes over this breakpoint N times.

2.          Condition – Break when a specific programmatic condition is reached.

3.          Suspend VM Vs Suspend Thread – This is useful when debugging multithreaded application. If you want to suspend only the current thread and not all threads that are running through this code flow, you can use this drop down.

Conditional breakpoints
We can declare conditional breakpoints triggered whenever the value of an expression changes.
Watchpoint
A watchpoint is a breakpoint at which is stop whenever a field read or changed. You can set a watchpoint through a double-click on the left side before the field declaration. Via the properties of the watchpoint you can define if the breakpoint should be hit during read access (Field Access) or during write access (Field Modification).
Exception Breakpoint
The application is stopped if the specified exception is thrown.
Method Breakpoint
A method breakpoint is defined via double-click in the left border of the editor and the method head. You can define if you want to stop the program during method entry of after leaving the method.
Class breakpoint
A Class Load Breakpoint will stop when the class is loaded. Right-click on a class in the Outline View and choose "Toggle Class Load Breakpoint"
Toggling class breakpoints on the class without the source code
Just go to the main menu and choose Run->Add Class Load Breakpoint…

Debug details formatter
Eclipse allows you to write your own formatter for displaying the content you are interested in. You can do it in the Eclipse preferences under Java->Debug->Detail Formatters.
Or in the Variables view, right click on the variable, then select "New Detail Format'.
This is also very useful for the standard classes such as map, list.
Logical Structure in Variables view
The logical structure is very handy for examining things like maps and other java collections classes, instead of showing the full detail of all the internal variables the view will only show the "logical" pieces.
Expression Evaluation
1. Inspect an expression
2. Use the Display View,
Write your expressions to evaluate and execute them by right clicking and selecting "Display" from the context menu.
Step Filtering
It filters out the classes that we don’t want to step into.
To specify which classes to filter out when "Step with Filters" is used, the preferences must be set.
Window > Preferences -> Java > Debug > Step Filtering and choose the classes/packages to be filtered out. When you step through the code, ensure that the "Use Step Filters" toggle button in Debug view is on.
Displaying all references of an object
If you want to display references of the specific object, right click on the object in the Variables view and choose All References. The popup window will appear with all references. If you want to see all references of all available objects go to the menu in the Variables view and choose Java->Show References.
Change variable values dynamically
You can use the Variables view to change the value of a variable dynamically.
Drop to frame
Eclipse allows you to select any level (frame) in the call stack during debugging and set the JVM to restart to that point.
This allows you to rerun a part of your program.
It also allows you to retrace your path after making a minor code change. The code flow goes back to the line of code you have dropped to and executes once again.

Hotswap Bug Fixing: On-the-fly code fixing
If you are running Java Virtual Machine (JVM) V1.4 or higher, Eclipse supports a feature called Hotswap Bug Fixing. It allows the changing of source code during a debugger session,
This is because JPDA implements the ability to substitute modified code in a running application.

Debug Java applications remotely with Eclipse
Run > Debug Configurations, then select the connector from the dropdown list. Two connectors are provided: Socket Attach, and Socket Listen
For the socket-listening connector, the Eclipse VM will be the host to be connected by the remote Java application.
For the socket-attaching connector, the target VM will be the host. There is no difference for application debugging between the two connectors — the user may choose.
A good rule of thumb is to use the faster, more powerful computer as the VM debug host because of the computational resources required.
Remote VM acts as debug server
Start remote java application as a debug server first:
java -agentlib:jdwp=transport=dt_socket,server=y,address="remote_server_public_ip:8000" DebugMe
java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address="remote_server_public_ip:8000" DebugMe(Prior to Java1.4)
Configure local eclipse:
Connection Type: Socket Attach

Remote Target VM acts as debug client
In this case, the remote Java application acts as a debug client, and the eclipse debugger front end acts as a debug server.
Eclipse uses the socket listen-mode connection type for listening. The debug front end must be started in advance to listen on a specific port.
Configure local eclipse:
Connection Type: Socket Listen
Then start remote java application as a debug client:
java -agentlib:jdwp=transport=dt_socket,suspend=y,address="eclipse_machine_ip:8000" DebugMe
java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=y,address="eclipse_machine_ip:8000" DebugMe(Prior to Java1.4)

You need to make sure the debug options are all enabled in in Window > Preferences > Java > Compiler.
 
Mae sure the jars in your remote server are build with these debug options enabled.

Remote Debugging Tomcat & JBoss Apps with Eclipse

Resources:
Java Debugging with Eclipse - Tutorial
Tips and Tricks for Debugging in Eclipse
5 Tips for Debugging Java Code in Eclipse
Eclipse tips and tricks – Part 2
Debug Java applications remotely with Eclipse
Remote debug with Java and Eclipse

OutOfMemory due to incorrect compareTo

Lesson learned from Defect

OutOfMemory due to incorrect implementation of compareTo

Recently, our application hit out of memory error. Through digging the heapdump generated, we quickly found the suspected class that may cause memory leak.

The class would put one item to TreeMap, and later remove it from the TreeMap when this item is finished.
We found that in the heapdump this TreeMap is huge, and this should not happen.

Add more logs to the class, and then we recreated this problem.
We found one strange thing, when after we put one item to TreeMap, later we remove it from the TreeMap, the treemap.remove(item.key) returns null, strange.

We wrote an example, which would put these items into the treemap, and then call contains(item.key), it returns false, en, very strange.
Then we debug the code, and found out that root cause.

See the code as below, have you seen the problem?
public class ClientID implements Serializable
public int compareTo(Object o)
{
   if (this == o)
    {
       return 0;
   }        
   if (o instanceof ClientID)
    {
       return (id - ((ClientID) o).id);
    }
   return -1;
}

In this code, it compares the two clients by subtracting their id(int).
But the id is a randomly-generated Hexadecimal int. If big negative number(a) subtracts another big negative number(b), a < b, it may overflow, and return a positive number.
So the compareTo method would return 1.

This would cause this item exists in the TreeMap, but TreeMap.contains/get can't find it.
The code should be changed like this:
public int compareTo(Object o)
{
   if (this == o)
    {
       return 0;
   }       
   if (o instanceof ClientID)
    {
           int thatClientId = ((ClientID) o).id;
           return (id == thatClientId) ? 0 : (id > thatClientId) ? 1 : -1;
    }
   return -1;
}

Except the fix above, we also change one TreeMap variable in the class to HashMap - as it doesn't need to be sorted by key, another TreeMap variable to TreeSet, as its key and value are same.

Lesson learned 1:
Ensure correct implementation of compareTo, equals, and hashcode.
In compareTo method, compare their value, not subtract them, this may cause overflow.
Be care of addition/subtraction and other operations of numbers, it may overflow.
Use correct data structure.

Lesson learned 2:
Recreate your problem.
Add more logs to help you recreate your problems.
Write example, prototype to verify your thoughts.
Think and list all possible reasons, order them by possibility, and the ease it can be tested and verified.
Then analyze one by one.

Developing Your Coding Skills

Developing Your Coding Skills

Measure code quality
Conceptual Integrity
Loose coupling, high cohesion
Simplicity
Correctness
Readability, extensibility, flexibility, maintainability, reusability

Coding Principles
Consistency
Completeness
DRY(Don't repeat yourself)
KISS(Keep it simple, Stupid) principle
"You ain't gonna need it"
Single-Responsibility Principle (SRP)
A class should have only one reason to change.
Open/Closed Principle (OCP)
Software entities (classes, modules, functions, etc.) should be open for extension but closed for modification.
If OCP is applied well, further changes of that kind are achieved by adding new code, not by changing old code that already works.
Figure out what may change, but resist premature abstraction.
Liskov Substitution Principle (LSP)
Subtypes must be substitutable for their base types.
Subclass should not violate invariants defined in base class, subclass's preconditions should not be stronger than those of base class, and derived classes must conform to all the postconditions of the base.
A derivative that does less than its base is usually not substitutable for that base and therefore violates LSP.
Design by contract
Dependency-Inversion Principle (DIP)
A. High-level modules should not depend on low-level modules. Both should depend on abstractions.
B. Abstractions should not depend upon details. Details should depend upon abstractions.
Interface Segregation Principle (ISP)
Interface pollution
Clients should not be forced to depend on methods they do not use.
Separate Client groups mean separate interfaces

Practices
Actively Evaluate Trade-Offs - Make sure the investment will pay off
Code in Increments - Write code in short edit/build/test cycles.
KISS (Keep it simple, Stupid)
Develop the simplest solution that works. Incorporate patterns, principles, and technology only if you have a compelling reason to use them.
Substitute by Contract
Use inheritance for is-a; use delegation for has-a or uses-a
Composition over inheritance
Don't swallow exceptions, report them.
Logging
Logging is vital to help to analyze problems, design your log mechanism, message need to log, rotation mechanism, record useful/consistent log.
Provide Useful Error Messages

Sensitive to Code Smell
A code smell is a hint that something has gone wrong somewhere in your code.
Common code smells
Duplicated code, Long method/class, excessively long identifiers,
Feature envy: a class that uses methods of another class excessively.
Inappropriate intimacy: a class that has dependencies on implementation details of another class.
Refused bequest: a class that overrides a method of a base class in such a way that the contract of the base class is not honored by the derived class.
Lazy class / Freeloader: a class that does too little.
Contrived complexity: forced usage of overly complicated design patterns where simpler design would suffice.

Beware of Anti-patterns
Methodological anti-patterns
Copy and paste, Golden hammer/Silver bullet, Reinventing the wheel, Premature optimization,
Programming by permutation/accident: Trying to approach a solution by successively modifying the code to see if it works
Improbability factor: Assuming that it is improbable that a known error will occur
Object-oriented design anti-patterns
BaseBean, Call super, Circle-ellipse(Square-Circle) problem, Circular dependency, Constant interface, God object,Object cesspool,Object orgy, Sequential coupling.
Yo-yo problem: A structure (e.g., of inheritance) that is hard to understand due to excessive fragmentation
Programming anti-patterns
Accidental complexity,Action at a distance, Blind faith, Boat anchor,Busy spin/wait,Caching failure,Cargo cult programming,Coding by exception,Error hiding,Hard code, Loop-switch sequence, Magic numbers, Magic strings,Paramesh,Soft code,Spaghetti code
Lava flow: Retaining undesirable (redundant or low-quality) code because removing it is too expensive or has unpredictable consequences

Design Smell
Rigidity - difficult to change.
Fragility - easy to break.
Immobility - difficult to reuse.
Viscosity - difficult to do the right thing.
Needless complexity - Overdesign.
Needless repetition - Mouse abuse.
Opacity - Disorganized expression.

Resources
A Taxonomy for "Bad Code Smells"
http://en.wikipedia.org/wiki/Code_smell
Practices of an Agile Developer: Working in the Real World
Agile Software Development, Principles, Patterns, and Practices
How To Improve Programming Skills

How to Fix Defect Effectively

How to Fix Defect Effectively

Understand the problem
Recreate the problem
Usually trying to recreate the problem is a good start.
Only we can recreate the problem, we can really make sure we know why this problem happen, why do find the root cause, It can help you to find the root cause, to isolate the problem, and it can make sure your change does really fix the problem.

May you have test team to help recreate the problem, still try to recreate the problem in your environment, this would be more time effective.

Ask customer or tester team how to recreate it, like what tests or operations they were doing.
Know more about the problem as possible as you can. Communicate well with customer and test team.
Get the bottom of the root cause.

1.         Never try to change the code before you figure out the real root cause.

2.         Isolate the problem and focus on the related code.

3.         List all possible causes, and examine them one by one.

4.         Add more log to help identify the root cause.

5.         If you are able to recreate the problem, you can add more log to help you to identify the root cause.

6.         Brain Storm.

Ask your team mate to help to brain storm on it.

7.         Write down your to-do-list for finding the root cause.

Fix the code

1.         Understand well the code you are trying to change.

Is there any special reason - we don't know yet, it is implemented like this? Why it doesn't work now.
Ask the originator to help to explain it.

2.         Choose the best solutions

Think whether there are other/alternative/better/simpler ways to fix this.
There are usually multiple ways to fix this problem, think them out, and compare them, make trade-off, and choose the best one that suits current situation.

3.         Know your fix well.

Know what it is trying to change, how to test your change.
Does your fix work well in all possible cases?
Know its side effects.
Beware of versioning problems
Will it break other things?
Know (and write down) possible breakage your code may introduce, pay attention to it, and use test to verify it.

4.         Use tools such as Findbugs, PMD etc to help avoid common programming errors.

Verify your fix.
Verify it fix this problem correctly.
Verify it doesn't introduce breakage.
Think about possible breakage it may introduce, and verify them.

Test your fix thoroughly to cover all possible situations.
Design the test cases to cover all good and error paths.
Write down the result of all test cases and analyze them.

Review your fix
Explain your fix clearly to your team mate, make sure they can understand the problem, agree the root cause, and understand your fix.
Explain the drawback of your fix, and ask for suggestions.

Debug It
The Core Debugging Process - Reproduce, diagnose, fix, reflect
Debugging Is More Than "Making the Bug Go Away"
Effective debugging requires that we take these steps:
1. Work out why the software is behaving unexpectedly.
2. Fix the problem.
3. Avoid breaking anything else.
4. Maintain or improve the overall quality (readability, architecture, test coverage, performance, and so on) of the code.
5. Ensure that the same problem does not occur elsewhere and can not occur again.

Understanding the problem and identifying the root cause is the cornerstone upon which everything else

Construct experiments, and observe the results.
Before start trying to reproduce the problem, hypothesizing about its cause, before make experiments, know What You’re Looking For, What is happening, and what should?
• Work on only one problem at a time.
Dig into the bug report.
Check Simple Things First
Ask whether there are similar problems seen and fixed before.
Reproduce
Reproducing the problem can help prove or disprove that your thoughts of the root cause is really root cause, can help to demonstrate your fix does work.
Ask your tester/users help (how) to recreate the problem.
Refining Your Reproduction
As Simple as Possible
Minimize the Time Required

Make Nondeterministic Bugs Deterministic
• Multithreading
Use sleep( ) method to force a thread to wait long enough to force a race condition to occur.

Automate the reproduction.
What If You Really Can’t Reproduce It?
Does It Really Exist?
Users don’t (normally) report bugs maliciously.

Work on a Different Problem in the Same Area
Get Others Involved
The best person to talk to is whoever reported the bug in the first place.
It can help to bring in someone who can attack the problem from an alternative direction.
Diagnose
Idealized process:
1. Examine what you know about the software’s behavior, and construct a hypothesis about what might cause it.
2. Design an experiment that will allow you to test its truth (or otherwise).
3. If the experiment disproves your hypothesis, come up with a new one, and start again.
4. If it supports your hypothesis, keep coming up with experiments until you have either disproved it or reached a high enough level of certainty to consider it proven.

• Construct hypotheses, and test them with experiments.
Experiments Must Prove or Disprove Something
Make sure you understand what your experiments are going to tell you. Your experiment must have a clear goal.
One Change at a Time
Keep a Record of What You’ve Tried
Periodically review what you’ve already tried and learned.

Ignore Nothing
If something unexpected happens, it means that some assumption you’re making is broken.
Anything that you don’t understand is potentially a bug.
Stratagems
Instrumentation the code
Divide and Conquer - binary chop
Leveraging Source Control when chase a regression
Focus on the Differences
Learn from Others, Google Searc
Explore the simple explanation/hypothesis first.
Use Debuggers

Validate Your Assumptions
Know what assumptions you’re making, and examine them critically.

Often the simple act of explaining the problem is all it takes for inspiration to strike.
Explaining your problem to someone else forces you to get your thoughts in order, enumerate your assumptions, and construct an argument from basic principles.
Let Problems Lie Fallow
Sleep on it, and Help your subconscious help you.
Change Something. Anything!

Persevere, there’s always a way through.
Validate Your Diagnosis
• Explain your diagnosis to someone else. Imagine that you are wrong—what mistake did you make?
Fix
There’s more to a good fix than just making the software behave correctly.
Make sure you know how you’re going to test it before designing your fix
Fix the Cause, Not the Symptoms
One logical change, one check-in.
Diff before check-in.
Get Your Code Reviewed
Reflect - Lesson Learned
Ask these questions: How the bug crept into the code? How Did It Ever Work? What Went Wrong?
Ensure it’ll Never Happen Again
Whether the code is leading people astray?
If you notice several examples of a particular problem, maybe the structure or the interface is making it too easy to make the same mistake repeatedly?

When a bug does slip through the cracks, we should learn as many lessons from it as possible and take whatever action we can to ensure that it doesn’t happen again.

Add identifying compatibility issues to your bug-fixing checklist.

Concurrency
Use sleep() or control exactly what order things happen in into your synchronization code to help reproduce concurreny-related bugs.
Avoid using sleep( ) when fixing concurrency bugs.
In concurrent software—making them less likely to happen is not an acceptable fix.

• Fixing performance bugs always starts with an accurate profile.
Suspect your own, ahead of third-party, code.
Software should be robust in production and fragile when debugging.

Resources:
Debug It!: Find, Repair, and Prevent Bugs in Your Code