Solr Data Schema Migration Practice


The Problem
When we design data schema, we should choose field name and type carefully. As it takes much more effort to change the schema, migrate old data without compared with changing code.
If we store data in Solr, we can explicitly store searchable fields as separate fields, store all other fields as a json string.

But sometimes we still have to change the schema: add fields or change field type.  

API to Getting the current Solr Schema Version
Solr stores current schema version like below: the version is a double number - not a string: means you can't store something like 1.6.0. 

The following code uses Spring Solr Data SolrSchemaRequest to get the version - you can also use solrj to do the same thing.
public double getVersion() {
    double version = -1;
    try {
        final NamedList<Object> nl = solrServer.request(SolrSchemaRequest.version(), getCollection());
        final Object object = nl.get("json");
        if (object != null) {
            version = MoreObjects.firstNonNull(createFailSafeObjectmapper()
                    .readValue(object.toString(), SchemaDefinition.class).getVersion(), -1d);
        }
    } catch (final Exception e) {
        LOGGER.error(MessageFormat.format("unable to get version for collection: {0}", getCollection()), e);
    }
    return version;
}

Store the version in the data
When we save data to Solr, call: entity.setObjectVersion(getVersion());

API/Script to upgrade data, check all versions
For example, recently we move some Solr fields that are not searched to a JSON body wich maps to a Java class - XXDetail.
- So when we need to add not-searched fields, we don;t have to update Solr schema - just put it into fields in XXDetail.

So we can write either java code or scripts to migrate the old version data to the new version schema.

Change field Type
Practice - Change Long to Date - Not Searchable
Previously we store the date(field:updateDate) as tlong in Solr, and we want to change it to date - As it will make the rest API more readable, easier to read the data and query Solr.

For Solr itself: after change type tlong to date, Solr can still read and query the old data.

If this field is not searched, and you already have Spring's LongToDateConverter, DateToLongConverter. Then you can just change its type from tlong to date.
    @ReadingConverter
    @WritingConverter
    public enum LongToDateConverter implements Converter {
        INSTANCE;

        @Override
        public Date convert(final Long source) {
            if (source == null) {
                return null;
            }
            return new Date(source);
        }
    }
    @ReadingConverter
    @WritingConverter
    public enum DateToLongConverter implements Converter {
        INSTANCE;
        @Override
        public Long convert(final Date source) {
            if (source == null) {
                return null;
            }
            return source.getTime();
        }
    }

Old code(using long) - new schema
When to save data, LongToDateConverter will convert long to date object automatically; when read data, DateToLongConverter will convert date and return long.
New code(using date) - old Schema
When to save data, DateToLongConverter will convert date object to long; when read data, LongToDateConverter will convert long and return Date.

Practice - Change Long to Date - Searchable
But if we search the field in code, then we can't just simply change the type.
Old code(using long) - new schema
When search: the query will be like:
field:[along to blong], but the field is already changed to date type, so the query will fail:
Invalid Date String:'2'
at org.apache.solr.schema.TrieDateField.parseMath(TrieDateField.java:150)
at org.apache.solr.schema.TrieField.getRangeQuery(TrieField.java:369)
at org.apache.solr.parser.SolrQueryParserBase.getRangeQuery(SolrQueryParserBase.java:761)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:382)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
New code(using date) - old Schema
When search in new code, the query will be like:
field:["yyyy-MM-dd:Thh:mm:ssZ" TO "yyyy-MM-dd:Thh:mm:ssZ"], but the field is still long in schema: so it will fail:
java.lang.NumberFormatException: For input string: "2016-02-23T20:08:15.208Z"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.solr.schema.TrieField.getRangeQuery(TrieField.java:343)
at org.apache.solr.parser.SolrQueryParserBase.getRangeQuery(SolrQueryParserBase.java:761)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:382)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:107)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)

The Solution
Now we will have to add a new field: updateTime - type date.
1. Update Schema - add fields, but not update or delete fields
-- Make sure old code works with new schema
The schema would contain both fields updateDate(tlong) and updateTime(date), the new code will store and query field updateTime(date).

2. Update to new code
-- Make sure new code work with old data

3. Migrate old version data to new version After we migrate all old version code to the new version, we will run the migrate script which will copy the old updateDate(tlong) to 
new field updateTime(date). Later we can get rid of the old field.

4. Remove code that handles the old version and old fields in next release

References
Database Migrations Done Right

Tools for Programmers - 2016


Atom
Alternatives - Sublime Text

iTerm2
brew cask install iterm2
Preferences > Profiles > General > Working Directory > Reuse previous seesion's directory

highlight errors and warnings
Preferences > Profiles > Advanced > Triggers > Edit
(?i:.*(conflict|fatal|error|warn).*) // Yellow on Black

Visual Studio Code: VSC
-- Auto complete for javascript
npm install tsd -g
tsd query -r -o -a install angular jquery

Auto Completion
Git Auto Completion
-- auto complete git commands, branche names, remote names etc
Maven Auto Completion
-- -DskipTests

Chrome
chrome://settings/cookies
chrome://about/
document.designMode='on'
- Enable copy & paste functions
Chrome Extensions
Quick Tabs
Sight
CheckChrome
Click to Remove Element
Postman
Grammarly Spell Checker & Grammar Checker
Octotree - Code tree for GitHub and GitLab

chrome flash is blocked on this website
Try setting chrome://flags/#prefer-html-over-flash as disable and restart Chrome.

JD-GUI - Java Decompiler
HomeBrew
brew update
brew upgrade $FORMULA

jq - command line json proccessor
jq ' .response.docs.[0].field | length ' solr-rsp.json
jq '.response.docs' solr-rsp.json
curl 'solr-query' | jq '.' > result.json collections_list = $(curl "http://host:port/solr/admin/collections?action=LIST&wt=json" | jq '.' curl "" | jq '.collections|join(" ")'')

Docker
docker run -d -p 80:80 --name webserver nginx
docker ps
docker exec -it container_id /bin/sh

docker kill/stop $(docker ps -q)
docker build --no-cache .
docker-compose build --no-cache mysql

Delete all containers
docker rm $(docker ps -a -q)
Delete all images
docker rmi $(docker images -q)

Redis
redis-server to start server
redis-cli
redis-cli shutdown
KEYS *
KEYS pattern

TTL key
TYPE key
FLUSHDB - Removes data from your connection's CURRENT database.

FLUSHALL - Removes data from ALL databases.
ZRANGE key start stop [WITHSCORES]

Kafka
Delete a topic in dev:
./bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name $topic-name --add-config retention.ms=1000
-- wait for several minutes, and verify there is no data:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic $topic-name  --from-beginning


bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name $topic-name --delete-config retention.ms

Commands
mysql.server stop
mysql.server start
SHOW CREATE TABLE tbl_name
DESCRIBE tbl_name

SourceTree
Postman

Markdown
List: +, *, -
Link:
[link text](link url "hover popup text")
```shell(java etc)
``` Java 
CFR 
- A decompiler that supports Java 8 language features, including lambda expressions and me
thod references.
JD-GUI

cygwi
jcmd: One JDK Command-Line Tool to Rule Them All
sudo -u tomcat jcmd pid Thread.print > thread.print
sudo -u tomcat jcmd pid GC.heap_dump heapdump.hprof

Thread dump Analyzer
http://fastthread.io/
https://spotify.github.io/threaddump-analyzer/

Heap dump Analyzer
Eclipse MAT

GC Log Analyzer
http://gceasy.io/
https://github.com/chewiebug/GCViewer

Misc
Blogspot
Search for multiple labels
search/label/LABEL1+LABEL2
/search/?q=label:LABEL1|label:LABEL2

What to Look for in Code Review


Good PR
Small Change/Diff/Pr
Stacked Change
Review often, merge often

What we want to achieve via Code review
Capture design and code issues
Sharing of knowledge

What to focus in Code review
Business logic
High level design
Coding sytle

Code review should be productive
– both for reviewers and the developer
Usually code review should be finished timely – within 3 to 5 days, not in weeks or months
- Unless we don’t need the function or have disagreement about the design and the code

- should be a pleasure for reviewers and the developer
- help the author to capture mistakes, bugs
- Reviewer can learn somethings such as:
   - domain knowledges, the new feature or others

Who do the code review?
How to pick code reviewers?
Who merge the code?


Bug always happens, how to prevent the developer and code reviewers make same mistake?

Before Send Code to Review
Only one change in one pr
Build succeed
Include new unit tests, integration tests
All tests passed
Existing functions work

New functions work

Use tools to analyze code first - eclipse compile warnings, findbugs, pmd, mvn site report.

Carefully reviewed your PR first
- Fix common issues such as commented-out code, indentation
- Code looks clean and readable to yourself
- Use tools such as FindBugs or Eclipse itself to check and clean your code
- It saves time for you and reviewers
- The reviewers don't pay attention to the trivial stuff and can focus on logic or things that can't be easily found out by tools

PR
Your PR should be pleasant for others to review.
Every PR should have clear description what has changed, and why (if needed)
Write descriptions/comments
- for reviewer
- for clients who is going to call your APIs
- share knowledge
Add comments to your code - such as why the code is removed

Highlight potential issues, or things you want reviewers to pay attention

- Don't hide them, instead highlight them
- it's for your won good as reviewers help improve your code, find potential issues, also this makes yourself a better engineer

Your Code
Your code should be pleasant for you and reviewers to read
- We not only write code, we write better code every day, write code that others want to use/call or read and maintain.

Write code for maintainer, reviewers, you in the future
Add comments to your code
- why you made the change this way: especially when there re other obvious approach, but maybe not work
- how you find the bug (internet link)
- when you think others(maybe yourself in the future) maybe confused
- write comments for your client/caller, how they should use it, what's the trick part

When others/reviewers ask questions about the code or when they feel difficult to understand it
It may be because others lack of the specific knowledge of the domain(logic) or it may (more likely) be a sign that we should improve the code, make it more readable

Share your knowledge to the team
- Such as in order to implement the PR, you learned some things new
- You don't have to share if it's something basic or we can find it via simple google search

- It's good to share if it takes you a lot of time, or you just want to share :)

During Code Review
Prefer to use PR to communicate
- as others would also know the conversation
- unless it's difficult to describe in words
- transparent and share knowledge
- What bugs we made, and how someone detect in code review

- how the reviewer reviews the code and finds out the issue

Don't forget the big picture
- What's the goal/new function of the PR
- Whether it solves the problem, implements the new function

What questions to ask?
- Not trivial things such as What is XX?
- Google it first, unless you still didn't find your answers

- Or you are the manager, tech skills/knowledge is not your primary goals any more

Use IDE to help understand the code if necessary
- Check the code change in Eclipse can help you to understand the code if you are not familiar with the function,

What to Look For
What may go wrong, what code may look suspicious, dangerous, have potential bugs, need pay extra attention, common source of bugs.

Whether the change improves code quality or make it worse?
Is there enough tests?
- Unit/Integration tests
- Happy, bad paths or corner cases

Admin what you don't know and ask others to review that part.
Ask for clarification/explanation if can't understand it.

Does it need documentation or notify other teams?

Using new features in the language?
- such as JDK8 time api, stream, new methods: getOrDefault, compute, putIfAbsent, computeIfAbsent etc

Don't invent here
- Whether we should use some well-known libraries, instead write our own code.

- Things that looks strange, that I don't understand or can't understand easily

Whether the new code is consistent (with existing code)
-- such as method name, order of parameters.

Focus on Main Function/Logic
-- Focus on the code that looks complicated, difficult to understand
-- Don't skip it if you can't understand it.

Performance
Whether the api is performance sensitive?
- Like database/networks calls
- If so, how long it usually takes in normal/extreme case?
- Is there duplication? do we need cache? or save the result first into a variable or a map?
--Did the code do enough optimization? such as use thread pool, call third part services asynchronously, in parallel etc. 

Do we need provide timeout for block call?
If so, whether the timeout value makes sense or is configurable?

Thread safe
Concurrent programming is hard, so pay more attention.
Example: Don't Use Mutable Variables in Multiple Threads
Variables may be Changed Unexpectedly in Multiple Threads
- Use mutable instance fields in services.
-- SimpleDateFormat

Does it reuse threadpool, is it configured correctly?
- Don't recreate threadpool and shutdown in each method, reuse it.
Is there really benefit to use multiple threads?
Is there race condition?
Is lock used correctly?

Code Quality
SOLID
Whether the method/class does only one thing?
Big interfaces: split them
use polymorphism to remove if.
Preconditions cannot be strengthened in a subtype.
Postconditions cannot be weakened in a subtype.

Review the data model
- As it's much harder to change data model after deployed to production.
-- is there any not needed fields?
How it is going to be used?
-- use redundant field/data to boost performance (avoid join query?)
-- will we search on it, define it as separate field, or store it as part of json object?

Is it using Right Data structures?
What's the frequent operations on the data?
If need search often: uses Set/Map instead of  List
If there is a Key in the data, use Map instead of List.
Don't use mutable object inSet, or as Key in Map
Consider using Guava Multimap instead of Map<K, List<V>>
-- Multimap put: the case when the key doesn't exist, get returns empty collection instead of null.

Is class in right place - module/application or package?
Whether it should be an instance field?

Security
Do we need protect the API? 
- If so, which role?
Are we storing sensitive info?
Should we audit the operation?

Code Quality
Bad Smell
Magic Number/String 
- change to use constant/enum and give a meaningful name.
Limit the number of parameters per unit to at most 4.
- extract parameters into objects.
Parameters
Too many parameters
Parameters are easily to be mixed

Use boolean as parameter -- consider to use enum or use different names.
Big Class/Methods

Method
Usually method should be short and easy to read, no too many nests.
Complex methods
-- extract small methods

Separate Concerns to different classes/functions

API Design
Think from client perspective.

Immutability
Unmodifiable of parameter, return value 
- defensive copy if needed

Whether API is easier to use, test/rollback in production

Code Readability
Whether the code is easy to read and understand?
 -- If not, ask the developer to clarify or add comments to the code, or refactor the code.
Bad signs - use boolean parameters, return Pair, long parameter lists

Naming
Naming(variable/function/class name) is important to make the code readable.
Whether names of variables, method, class are meaningful?
-- 20 Tips for Better Naming
Good names should answer all the big questions, be precise and thorough, reveal intent, and follow conventions.
-- Avoid use vague variable names.
waitTime -> waitTimeInMs

Arguments
Number of Method Arguments should be as small as possible.
- smaller is better, remove unused parameters
Order of arguments
- especially when the methods/constructors are overloaded. 

Order of statements
-- Put related statement together
-- whether some commands/functions should be always last one? If so, is it documented? 

API design
Unnecessary code
Is there any unnecessary code(field/methods/function)?

Configuration Driven
Should it be define as constant, in property file or stored in db?
Prefer to store in db, so we can change it dynamically.

Do we need wire on/off feature?
- Consider adding turn-on flag for for new features

Code Duplication - Don't repeat yourself
Does it cause duplication?

Is there anything that can be extracted and reused in future?
Whether this is common/util function?
Whether the function is put in right place?

Tools like PMD/CPD can help find copy-paste code.
Similar Methods:
Extract common logic?
Use Holder object to contain all data?
Before create a new constant/util methods, search the code base first to make sure it's not already there.

Common Bugs
NPE
- Change to use Optional, or use annotations: such as @Notnull, @CheckforNull
-- if(anint == anInteger), anint == null, 
anInteger = boolean?anint:anInteger
-- Use NullSafe API:
Objects.equals, MoreObjects.firstNonNull, StringUtils...

< or <+, off-by-one error
isEmpty or isNotEmpty?
Dangling If/Missing else
Missing default in switch

Allocate Unlimited Resources
-- Executors.newCachedThreadPool(); 
-- Create threads, make db/network calls in loop
Call expensive operations in for loop or for every record.

Look for signs of questionable code
Deep nesting of loops and conditionals.
AutoDetectParser instead of JpegParser

Not Needed Code
Remove commented code
YAGNI
Developer may write code that is not needed - such unused field/functions/features
Don't guess at future functionality.

Be cautious of clever implementation.
Is it over-engineered?

Is there enough Logging?
Whether there is enough logging to help debug if the function doesn't work as expected or to prove the code work.
Logging more info in lower log level.

Monitoring
Should we add monitor info such as API's latency/frequency? 

Don't
Don't return array

Mics
- Non-static inner class, anonymous class, lambda uses method or field from the enclosing class
  - Prefer to use static inner class
  - Whether it may cause memory leak

- Use Optional instead return null 
- Use Collections.emptyList/List.of()/ etc instead of new empty ArrayList
- Use new features unnecessary or inappropriately
- In some cases, the code is more efficient to just use regular for loop instead of stream
- we can collect/update all variables in one loop rather than use multiple streams(traverse) to update them separately
- Miss @Override

Testing
Json and Java serialization and deserialization of Http API response data


Resources
How to write the perfect pull request
thoughtbot - Code Review

How to Edit Blogger Mobile Template


To change blogger's default mobile template and add mobile widgets:

Choose custom mobile template
In Blogger dashboard, go to template panel, click the mobile gear button:
Make sure you select: Yes. Show mobile template on mobile devices.
Choose "Custom" as the mobile template.

Add a widget in layout, in the title, give it an easy-to-find name.
Edit template HTML, find the widget, set its mobile='only' or mobile='yes'(shown in both desktop and mobile).

Problem: An error occurred, please refresh the page and try again
After you add mobile widget, later when you try to move widgets in layout, it may fail with message: An error occurred, please refresh the page and try again

Preview also fails: The template is updated in another place. Please reload the original page and try again.

If this happens continuously, go to template the page to delete the mobile widget, make your change, then later add back the mobile widget.

Merge JSON Objects: Jackson + BeanUtils.copyProperties


The Problem
We store some configuration object as JSON string and want client able to update separate fields directly - without reading it first.

The Solution
In our java configuration class, we use wrapper class for simple types, then Spring BeanUtils.copyProperties(Object source, Object target, String... ignoreProperties) to copy from new config to old config and ignore the null values in new config.

If the configuration has f1=1, f2=1, f3=1, the client only wants to change f3 to 3, then the client can change just send {f3: 3} without having to read the data first.

This is why we use wrapper instead of primitive fields, as the value would be null in new config if the client doesn't specify, we can add all null fields to the ignoreProperties.

public class SimpleConfig implements Serializable {
    private static final long serialVersionUID = 1L;
    private  Boolean featureAEnabled;
    private  Integer minVersion;
    // other fields ignored
}

public void mergeSimpleConfig(final SimpleConfig newConfig) {
    final String configName = COLUMN_SIMPLE_CONFIG;
    final SimpleConfig oldConfig = getSimpleConfig();
    SimpleConfig mergedConfig = (SimpleConfig) SerializationUtils.clone(oldConfig);
    mergeIgnoreNullProperties(newConfig, mergedConfig);

    final ConfigElement newConfigElement = new ConfigElement();
    newConfigElement.setConfigName(configName);
    newConfigElement.setValue(objectMapper.writeValueAsString(mergedConfig));
    newConfigElement.setLastUpdateTime(new Date());
    configDao.saveConfig(newConfigElement);
}

public static <T> void mergeIgnoreNullProperties(final T source, final T target) {
    BeanUtils.copyProperties(source, target, CobraUtil.getNullPropertyNames(source));
}
public static String[] getNullPropertyNames(final Object source) {
    final BeanWrapper src = new BeanWrapperImpl(source);
    final java.beans.PropertyDescriptor[] pds = src.getPropertyDescriptors();

    final Set<String> emptyNames = new HashSet<String>();
    for (final java.beans.PropertyDescriptor pd : pds) {
        final Object srcValue = src.getPropertyValue(pd.getName());
        if (srcValue == null) {
            emptyNames.add(pd.getName());
        }
    }
    final String[] result = new String[emptyNames.size()];
    return emptyNames.toArray(result);
}

@Component
@Provider
public class JerseyObjectMapperProvider implements ContextResolver<ObjectMapper> {
    private static ObjectMapper objectMapper = createFailSafeObjectmapper();

    @Override
    public ObjectMapper getContext(final Class<?> type) {
        return objectMapper;
    }
    public static ObjectMapper createFailSafeObjectmapper() {
        return new ObjectMapper()
                .setDateFormat(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss"))
                .configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false)
                .configure(DeserializationFeature.FAIL_ON_INVALID_SUBTYPE, false)
                .configure(DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES, false)
                .configure(DeserializationFeature.FAIL_ON_NUMBERS_FOR_ENUMS, false)
                .configure(DeserializationFeature.FAIL_ON_READING_DUP_TREE_KEY, false)
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                .configure(DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS, false)
                .setSerializationInclusion(Include.NON_NULL);
    }    
}
Read More
Jackson Essentials - the JSON Libaray
Using Jackson JSON View to Protect Mass Assignment Vulnerabilities
Merge JSON Objects: Jackson + BeanUtils.copyProperties
Jackson Generic Type + Java Type Erasure
Jackson Date Serialize + Deserialize

Mastering Eclipse IDE - Tips, Tricks


Quick Access(Command+3)
-- For example, to change a string to upper case or lower case, and you don't know the short cuts, then just select the text, type "upper" or "lower" in quick access text box.
-- Open specific view, preference, call specific command.

Shortcuts
Command+./Ctrl+.      Go to next error/warning
- We can configure which annotations to traverse by clicking on the dropdown: like adding Tasks.
Command+1/Ctrl+1    Quick fix
Alt + Shift + T            Opens the context-sensitive refactoring menu
Alt + Shift + Z     Surround block with try and catch - config first
Command +M            Maximizes the current view or editor
Command +Shift+/     Insert block comment
Alt+Shift + N             Shortcut for the menu to create new objects

Command+D              Delete current line
Command +E
Search dialog to select an editor from the currently open editors
Command+,               Open preference 

Quick fix:
Convert "" + "" to MessageFormat or StringBuilder.
Convert anonymous class creation to lambda expression and vice versa
Show spelling suggestions

Refactoring
Eclipse Refactor Actions
https://bmwieczorek.wordpress.com/2009/11/28/eclipse-refactoring/
Move static methods/fields to another class
-- Select them in outline view -> Refactor -> Move 
Change method signature
-- Add/remove/move parameters and give default value
Introduce Parameter Type
-- When there are too many parameters in method
Rename
- Press cmd+option+r twice or click options to open a dialog to also rename the getter and setter.

Configuration
Auto Save
General > Editors > Auto-save
Configure Eclipse Compiler Warnings
Enable null analysis, unbox conversion, missing default in switch etc
Leveraging JSR-305 null annotations to prevent NullPointerExceptions
Use @CheckForNull, @Nonnulls

Enable Save Action
Configure and Run Clean up
- We can configure eclipse to auto run clean up when save 
- We can also run Clean up manually via quick access or assign it a shortcut such as ctrl+alt+command+c

Code Recommender
toString() Generator: Code Styles - Use StringBuilder or Skip Null
Configure hashCode and equals to use blocks in "if" statement.

Static import
To help write test cases easier, when we type "assertT" and hit Ctrl+Space, we want Eclipse to add static import automatically for us: import static org.hamcrest.MatcherAssert.assertThat;
Window > Preferences > Java > Editor > Content Assist > Favorites, then add:
org.hamcrest.Matchers.*
org.hamcrest.CoreMatchers.*
org.mockito.Mock.*
org.mockito.Mockito.*

org.mockito.Matchers.*
org.junit.*
org.junit.Assert.*
org.junit.Assume.*
org.junit.matchers.JUnitMatchers.*

JDK8:
java.util.stream.Collectors.*

com.jayway.restassured.RestAssured.* -> if use restAssured.

Transform static invocation to a static import - cmd(ctrl)-shift-M

Code Template
Use Template view to learn shortcuts
static_final, test, instanceof, lazy, lock, systrace, 
.var .null, .nnull

Debugging
Use Display view to execute (any) code in current context
CTRL+U (execute) or CTRL+SHIFT+I (inspect)

Launch Group
Group related debug/remote applications together.
Example - local solr remote debug

Force Return in Display view
Use conditional breakpoint to execute any code without stopping

Force Eclipse to throw exception in display view
Execute: {throw new AnyKindOfException();}

Navigation
Drop to Frame
Run to line(Ctrl+R)
Smart Step into selection -> Ctrl + Alt + Click
-- To step into a single method within a series of chained or nested method calls.
Detail Formatter in Variables View
Change preferences of detail formatters:
- So eclipse will show toStirng value directly in value field for variables like StringBuilder, Map etc.
Show variable details (toString() value) to one of the following:
As the labels for all variables
As the labels for all variables with detail formatters

SolrQuery : Url Decode
java.net.URLDecoder.decode(this.toString())
Date : Show utc time
java.text.SimpleDateFormat sdf = new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", java.util.Locale.ROOT);
sdf.setTimeZone(java.util.TimeZone.getTimeZone("UTC"));
sdf.format(this);
new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss").format(this)

Common Preference Settings
Remove java exception breakpoints
-- Go to Java>Debug, Uncheck "suspend on uncaught exception"

Improve Eclipse Performance
http://www.slideshare.net/kthoms/boosting-the-performance-of-your-eclipse-ide-eclipsecon-france-2016
Disable autobuild
- Uncheck 'Build Automatically'
- Eclipse auto build before launching the app(or tomcat): Pref > Run/Debug > Launching

Uninstall not-needed plugins such as mylyn at Help -> Installation Details
Close Unwanted Projects
eclipse.ini
-server
-Xms1g
-Xmx8g
-Xverify:none -> disable class verification

Show heap status at General
General -> Appearance
Disable theming
Disable Animations

Disable all plugins at General -> Startup and Shutdown
Disable unnecessary Validators -> Suspend all
Disable unnecessary Label Decorations

Click on Team -> Disconnect on project
- If you use git command

Maven:
Disable download repository index on startup
User interface: open xml page in the pom editor by default

Exclude workspace folder from Spotlight Index
System Preferences -> Spotlight -> Privacy tab, add folder to exclude from index
Exclude workspace from anti-virus

Find huge files
du -h -d 1 .metadata/.plugins | gsort -h
.m2/repository/.cache/m2e
metadata/.plugins/org.eclipse.m2e.core

Cleanup Metadata
.metadata/.plugins/org.eclipse.wst.server.core
.metadata/.plugins/org.eclipse.jdt.core

Search local history
grep -R -i yourQuery .metadata/.plugins/org.eclipse.core.resources/.history

Uncheck Re-open Remote Systems view in "Remote system".

JUnit - UnCheck Show Failures Only

Run Tests Across Multiple Projects
https://github.com/takari/takari-cpsuite
Create a maven project depending on all the projects you want to test.
pom.xml:
<dependencies>
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
  </dependency>
  <dependency>
    <groupId>io.takari.junit</groupId>
    <artifactId>takari-cpsuite</artifactId>
    <version>1.2.7</version>
  </dependency>
</dependencies>
Create Test code:
import org.junit.extensions.cpsuite.ClasspathSuite;
import org.junit.runner.RunWith;
@RunWith(ClasspathSuite.class)
public class TestRunner {
}

Effective Search
Filtering Call Hierarchy
- Filters: 
   exclude test classes: *.Test*, *.*Test, *.*TestCase, *.*TestBase
- Search in option to exclude application/jre libraries
- Field Access: Read or write access

Using Regular Expression
When we highlight words we want to search for, then Ctrl+H, the search dialog will contains the word we select.

If previously we selected "Regular Expression", Eclipse will automatically create the regex for us.

Press Ctrl+Space for content assist when write regex in search box.

Find empty lines: ^\s*\n
We can show file in terminal or package explorer in "Open Resource" dialog.

Show file in terminal or system explorer in package explorer view.

Eclipse Console View
We can open multiple consoles by "Open Console",  and pin it by "Pin Console", move them to different locations.

Click "Display selected console" button to switch console.
If don't want to open or switch console when there is update:
- Deselect "Show console when standard out changes" and "Show console when standard error changes"

Autosave for dirty editors
Preferences > General > Editors > Auto-save

Hot deploy
Eclipse supports limited hot deploy function: we can change the method implementation, but not add new classes/methods.

We can use dcevm and HotswapAgent for true hot deploy - check the link for how to setup them.
- Match the java build version and dcevm version: light-jdk8u112-8 works with Java 8u112.

Add -XXaltjvm=dcevm -javaagent:~/hotswap-agent-1.0.jar to tomcat VM arguments.

Plugins
MoreUnit
Ctrl+U: create test method

Ctrl+J:  jump to test method
Eclipse Grep Console
Using Eclipse Console plugin to show different colors for warning, error in console
ERROR:      (^(ERROR|SEVERE):)|Exception|(^\s+at)
Warning:      ^WARNING:

Spell Check
Eclipse default spell checker can detect spelling error in java doc, and plain text file.
Go to Window > Preferences > General > Editors > Text Editors > Spelling, to enable/disable or change the settings.

To detect spelling error in names of interfaces, classes, methods, constants, use jdt-spelling

We can use quick fix(cmd+1) to fix spelling error or add word to dictionary.

Misc
Enable "Link With Editor" in Package Explorer/Project Explorer/Navigator/Breakpoints
Use Working Sets to group different projects.

Linking an external folder to Eclipse

Turn off code formatter for certain sections of java code
Go to Preferences>Java>Code Style>Formatter.
Click on Edit, go to Off/On Tags, select Enable Off/On tags.
// @formatter:off
...
// @formatter:on

Determining who last modified a line with the Annotate command
Team > Show Annotation

Name template during project import
When need import maven projects multiple times into the same workspace,  in the advanced section, specify the name template like [artifactId]-something-here, so each project will have a different suffix.

Prevent line wrapping in XML and HTML files
Use: <![CDATA[ ]]>

Block Selection Mode
Use Ctrl+3 and type "block"
Mac:         Command+Option+A
Window:  Alt+Shift+A

Open multiple eclipse instances/workspaces
In Mac: open -n Eclipse.app
Or use OS X Eclipse Launcher

Other Plugins
Easy-shell
Open folder in explorer or command line.

Troubleshooting
Kill hang stopped tomcat process in Eclipse
Sometimes, when we stop tomcat, it may hang 
- it becomes gray square Cancel Requested.
(sometimes lsof -i :8080 shows nothing, but the tomcat process is still there) We can use command line like jps to find the tomcat process, and kill it. 

Find logs at yourWS/.metadata/.log

How to remove eclipse error "A cycle was detected in the build path of the project"
You can go to Preferences -> Java -> Compiler -> Building and change the Error to Warning.

Eclipse.ini


Misc
Docking a detached view
Click on the tab (not on the title of detached tab window) and drag


Java Date + TimzeZone + Locale + DateFormat


Time in Java Date is the number of milliseconds since January 1, 1970, 00:00:00 GMT, it's timezone-agnostic. (the getYear/Month/Date are dprecated because they are timezone-related.)

When we format the date to a string, we need specify what timezone and what locale to use.
If not specified, it will use time zone and locale in current system.

So if we use the number, time zone doesn't matter. But if we store the deate string or use it to query, we should always use UTC and locale incenstive SimpleDateFormat  to format date object.
DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT);
sdf.setTimeZone(TimeZone.getTimeZone("UTC"));
public static void main(final String[] args) {
    TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"));
    DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", new Locale("en", "US"));
    Date date = new Date();
    // 1456470936828 PST(-8:00):2016-02-25T23:15:36.828Z
    System.out.println(date.getTime() + " PST(-8:00):" + sdf.format(date));
    System.out.println(date.getDate());// 25

    TimeZone.setDefault(TimeZone.getTimeZone("America/New_York"));
    sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", new Locale("en", "US"));
    date = new Date();
    // 1456470936829 EST(-5:00):2016-02-26T02:15:36.829Z
    System.out.println(date.getTime() + " EST(-5:00):" + sdf.format(date));
    System.out.println(date.getDate()); // 26

    // use Locale.ROOT, UTC
    sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT);
    // 1456470936829 UTC:2016-02-26T07:15:36.829Z
    sdf.setTimeZone(TimeZone.getTimeZone("UTC"));
    System.out.println(date.getTime() + " UTC:" + sdf.format(date));
    System.out.println(date.getDate()); // 26

    // use org.apache.solr.common.util.DateUtil
    // 2016-02-26T07:15:36.829Z
    System.out.println(DateUtil.getThreadLocalDateFormat().format(date));
}
org.apache.solr.common.util.DateUtil
public static DateFormat getThreadLocalDateFormat() {
  return fmtThreadLocal.get();
}

public static TimeZone UTC = TimeZone.getTimeZone("UTC");
private static ThreadLocalDateFormat fmtThreadLocal = new ThreadLocalDateFormat();

private static class ThreadLocalDateFormat extends ThreadLocal<DateFormat> {
  DateFormat proto;

  public ThreadLocalDateFormat() {
    super();
    //2007-04-26T08:05:04Z
    SimpleDateFormat tmp = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", Locale.ROOT);
    tmp.setTimeZone(UTC);
    proto = tmp;
  }

  @Override
  protected DateFormat initialValue() {
    return (DateFormat) proto.clone();
  }
}

Solr removes DateUtil in later release. We can use Apache HttpClient org.apache.http.client.utils.DateUtils.parseDate(String, String[]) to parse the date using multiple date formats.
https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Troubsle shooting - Eclipse Tomcat Deploy Stops Working


The Problem
Sometimes, Eclipse Tomcat plugin fails to work: your web app is not deployed.

Check List
Try Server -> Clean
If not work, try Project -> Clean
Check Eclipse Problems View, make sure there is no build path issue that stops Eclipse from compiling.

Check Tomcat Depoyment folder:
- Check whether tomcat plugin copied your web apps to this folder:
INFO: Command line argument: -Dcatalina.base=/Users/USERNAME/ws/YOUR_WS/.metadata/.plugins/org.eclipse.wst.server.core/tmp1
Feb 12, 2016 11:50:29 AM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.home=/Users/USERNAME/src/apache/tomcat/apache-tomcat-8.0.21
Feb 12, 2016 11:50:29 AM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dwtp.deploy=/Users/USERNAME/ws/snei-qos/.metadata/.plugins/org.eclipse.wst.server.core/tmp1/wtpwebapps

Make sure classes are in WEB-INF/classes folder. 
--
Feb 12, 2016 11:50:43 AM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath

Root cause - Eclipse "Build Automatically" was unchecked somehow at some time, so tomcat does copy the web apps folder in target to tomcat deployment folder, but no classes in  WEB-INF/classes there.

Misc:
Use Grep Console to Colorize logs in eclipse console

Jackson Generic Type + Java Type Erasure


The problem
We are save configuration elements to Dynamodb: the key is the configuration name, the value is the JSON string.
Recently I am trying to refactor the code to make the service code much shorter and easier to read. But I made one mistake - related with Java Type erasure.

The following code tries to read configuration: its value is a Map. I want to create one generic method that can read different Map.
  @Nonnull
  private <T> Map<String, T> getMapConfig(final String configName) {
      final ConfigElement c = configDao.getConfigByName(configName);

      Map<String, T> resultMap = null;
      if (c != null) {
          String json = c.getValue();
          resultMap = objectMapper.readValue(json, new TypeReference<Map<String, T>>() {});
      }
      if (resultMap == null) {
          resultMap = new HashMap<>();
      }
      return resultMap;
  }

  private <T> T convertMapValue(final String json) {
      return objectMapper.readValue(json, new TypeReference<T>(){});
  }
But this doesn't work: because of type erasure, T is just a type variable, at runtime, Java can't get the real value of the type.

The Solution
We have to specify class type like below:
  @Nonnull
  private <T> Map<String, T> getMapConfig(final String configName, final Class<T> classType) {
      final ConfigElement c = configDao.getConfigByName(configName);

      Map<String, T> resultMap = null;
      if (c != null) {
          String json = c.getValue();
          final MapType mapType = TypeFactory.defaultInstance().constructMapType(Map.class, String.class, classType);            
          resultMap = objectMapper.readValue(json, mapType);
      }
      if (resultMap == null) {
          resultMap = new HashMap<>();
      }
      return resultMap;
  }
  private <T> T convertMapValue(final String json,final Class<T> valueType) {
      return objectMapper.readValue(json, valueType);
  }
One related stuff I found is that in eclipse you can change Eclipse's Warning settings so that if Eclipse will show warnings if some parameter are not used: Change -> Window/Preferences /Java/Compiler/ErrorsWarnings -> Unnecessary Code > Value of parameter is not used to warning.

Read More
Jackson Essentials - the JSON Libaray
Using Jackson JSON View to Protect Mass Assignment Vulnerabilities
Merge JSON Objects: Jackson + BeanUtils.copyProperties
Jackson Generic Type + Java Type Erasure
Jackson Date Serialize + Deserialize
Apache Commons TypeLiteral
Google Guice TypeLiteral
Effects of Type Erasure and Bridge Methods

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)