Tips and Tricks for Docker

docker run -d -p 80:80 --name webserver nginx
-v host-dir:container-dir
-p host-port:container-port

docker ps
Attach to existing container's shell
docker exec -it container_id /bin/sh

docker kill/stop $(docker ps -q)
docker build --no-cache .
docker-compose build --no-cache mysql

Delete all containers
docker rm $(docker ps -a -q)
Delete all images
docker rmi -f $(docker images -q)

Clean up disk space used by Docker
docker system df
docker system prune

Increase docker memory
In advance tab of preference, change the memory or cpu.
configure --memory 6g in docker run command to set a different value.

Disable container auto-run
docker update --restart=no my-container

List Docker Container Names and IPs
docker ps -q | xargs -n 1 docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} {{ .Name }}' | sed 's/ \// /'

docker stats
check memory and cpu usage

docker-compose up

Dockerfile
RUN creates an intermediate container, runs the script and freeze the new state of that container in a new intermediate image.

WORKDIR /path

ENTRYPOINT vs CMD 
Prefer exec form over shell form
docker run --entrypoint [my_entrypoint] containter_name [command 1] [arg1] [arg2]

Use COPY if don't need ADD's specical features

Use dockerfile to build each image, docker compose file to assemble them.

EXPOSE 8983-8986
USER builder

docker-compose
up, down, logs, 
scale service=number
docker-compose build

Volumes
1. absolute path or relative path started with ./ means local path
2. named volume that references from the volumes list
docker volume prune
docker volume ls
docker inspect $volume_name

Misc
Change hostname
docker run -it -h myhost ...
docker run --rm -it --cap-add SYS_ADMIN ...

Troubleshooting
docker logs $container_id
docker events &

Tips and Tricks for Atom Editor

Settings

Core Settings

  • Show/Hide files/folders ignored by .gitignore
    • (un)check “Hide VCS Ignored Files” in Tree View package
  • Make files/folders ignored by .gitignore searchable or not
    • (un)check “Exclude VCS Ignored Paths” in Settings

Atom -> Preference -> Editor -> Enable soft wrap

Tune Package Configuration

  • After install a new package, read its settings page to configure it, learn its Keybindings etc.
  • Find the package in “Preference -> Packages” #### Tree View #### Command Palette
  • Select Preserve Last Search #### Fuzzy Finder
  • Select Preserve Last Search #### AutoSave
    • select “enabled”.
  • Tune settings
    • Enable Auto Reveal: link tree view with editor
    • Enable Hide Ignored Names, Hide VSS Ignored Files
  • cmd-\ to open/close the tree view, ctrl-0 to focus it
  • a, shift-a, m, or delete to add, move or delete files and folders
  • up/down arrow keys to preview files
    • need reload the window

Whitespace package

Functions

Plugins

Plugins for Writing

Plugins for Developers

Other Plugins

Shortcuts

  • Search shortcut at Settings -> Keybindings
  • Shortcuts from vim-mode-plus
  • New File under current folder when select the folder or a file in project panel: A
Command+P Switch to File
Command+Shift+P Command Palette
Command+\ Toggle Tree View
Command+Enter Replace all
Cmd+Ctrl+G Select all matching characters
Shift+Ctrl+M Markdown Preview
Cmd+R File symbol navigation
Ctrl+Shift+C Copy full path
Ctrl + g Go to Line
Ctrl+m Go to Matching Bracket
Cmd+Shift+t Bring up the list of corrections (with spell-check extension)
Ctrl+Shift+left(or right) move tab to left(or right)
Cmd+k+u Convert to Upper Case
Cmd+k+l Convert to Lower Case
Ctrl+Shift+k Delete Line
Ctrl+K Cut to End of Line
Cmd+k+n(p) Focus next(previous) panel

Keymap

  • How to disable key binding
    • Click the clipboard to copy the shortcut definition in Keybindings settings
    • replace the function to unset! to remove it or change it to another function #### Solve the conflict between jumpy:toggle and markdown-preview-enhanced:run-code-chunk
    '.editor[data-grammar*="gfm"], .editor[data-grammar*="md"], .editor[data-grammar*="markdown"]':
    'shift-enter': 'jumpy:toggle'

    Tips

  • Use Rename to move files
  • Find previous: shift + enter in the search dialog
  • Use Cmd+/ to comment: it know the right syntax for different language
  • Click the file name at the bottom bar to copy the full path
  • Pending Pane Items
    • single-clicking a file will open it in pending state
    • use this to open files that we don’t want to keep and avoid open too many files
  • Use Settings -> Updates to update all extensions

File/Directory Pattern Syntax

  • Use minimatch and glob patterns
  • “globstar” pattern (**) can be used to match arbitrarily many subdirectories: docs/**/*.md
  • Exclude file types or paths: !
  • Search only java source (exclude test) code: **/src/main/**/*.java or *.java,!*Test.java(this also works in Eclipse)

apm

Resources

How Solr Create Collection - Learn Solr Code

Test Code to create collections
MiniSolrCloudCluster cluster = new MiniSolrCloudCluster(4 /*numServers*/, testBaseDir, solrXml, JettyConfig.builder().setContext("/solr").build());
cluster.createCollection(collectionName, 2/*numShards*/, 2/*replicationFactor*/, "cie-default", null);

CollectionsHandler
CollectionsHandler.handleRequestBody(SolrQueryRequest, SolrQueryResponse)

CollectionAction action = CollectionAction.get(a); // CollectionAction .CREATE(true)
CollectionOperation operation = CollectionOperation.get(action); //CollectionOperation .CREATE_OP(CREATE)

Map result = operation.call(req, rsp, this);
Return a mpa like this:
{name=collectionName, fromApi=true, replicationFactor=2, collection.configName=configName, numShards=2, stateFormat=2}

ZkNodeProps props = new ZkNodeProps(result);
if (operation.sendToOCPQueue) handleResponse(operation.action.toLower(), props, rsp, operation.timeOut);

CollectionsHandler. handleResponse
QueueEvent event = coreContainer.getZkController() .getOverseerCollectionQueue() .offer(Utils.toJSON(m), timeout);

This uses DistributedQueue.offer(byte[] data, long timeout) to add a task to /overseer/collection-queue-work/qnr-numbers.

It uses LatchWatcher to wait until this task is processed.

Overseer and OverseerCollectionProcessor

OverseerCollectionProcessor.processMessage(ZkNodeProps, String operation /*create*/)
OverseerCollectionProcessor.processMessage(ZkNodeProps, String)
{
  "name":"collectionName",
  "fromApi":"true",
  "replicationFactor":"2",
  "collection.configName":"configName",
  "numShards":"2",
  "stateFormat":"2",
  "operation":"create"}
  
OverseerCollectionProcessor.createCollection(ClusterState, ZkNodeProps, NamedList)

  ClusterStateMutator.getShardNames(numSlices, shardNames);
   positionVsNodes = identifyNodes(clusterState, nodeList, message, shardNames, repFactor); // round-robin if rule not set

  createConfNode(configName, collectionName, isLegacyCloud);
    Overseer.getInQueue(zkStateReader.getZkClient()).offer(Utils.toJSON(message));
// This message will be processed by ClusterStateUpdater
// wait for a while until we do see the collection in the clusterState

  for (Map.Entry e : positionVsNodes.entrySet()) {
  if (isLegacyCloud) {
    shardHandler.submit(sreq, sreq.shards[0], sreq.params);
  } else {
    coresToCreate.put(coreName, sreq);
  }
}

This will send http call and be handled by CoreAdminHandler.handleRequestBody.

ClusterStateUpdater
  // if there were any errors while processing
  // the state queue, items would have been left in the
  // work queue so let's process those first
  byte[] data = workQueue.peek();
  boolean hadWorkItems = data != null;
  while (data != null)  {
    final ZkNodeProps message = ZkNodeProps.load(data);
    clusterState = processQueueItem(message, clusterState, zkStateWriter, false, null);
    workQueue.poll(); // poll-ing removes the element we got by peek-ing
    data = workQueue.peek();
  }

ClusterStateUpdater.processQueueItem

  zkWriteCommand = processMessage(clusterState, message, operation);
  stats.success(operation);

  clusterState = zkStateWriter.enqueueUpdate(clusterState, zkWriteCommand, callback);
processMessage

  case CREATE:
    return new ClusterStateMutator(getZkStateReader()).createCollection(clusterState, message);

overseer.ClusterStateMutator.createCollection(ClusterState, ZkNodeProps)

Gradle Tips and Tricks - 2017

Run tasks on sub-projects only
./gradlew sub-project:build

Tasks
./gradlew classes
tasks

Gradle task options Skip Tasks
-x test -x testClasses -x javadoc -x javadocJar -x integrationTestClasses 
-x findbugsMain -x findbugsTest -x findbugsIntegrationTest -x pmdMain -x pmdTest -x pmdIntegrationTest 

--daemon
--dry-run 
--profile
--parallel
Troubleshooting
--debug

-s, --stacktrace

Run specific tests
gradle test --tests org.gradle.SomeTest.someMethod
gradle test --tests org.gradle.SomeTest
gradle test --tests org.gradle.internal*
//select all ui test methods from integration tests by naming convention
gradle test --tests *IntegTest*ui*
//selecting tests from different test tasks
gradle test --tests *UiTest integTest --tests *WebTest*ui

Install an artifact locally
apply plugin: 'maven-publish'
gradle publishToMavenLocal

Using artifacts from local maven
apply plugin: "maven"
allprojects {
  repositories {
      mavenLocal()
      mavenCentral()
  }
}

Run tasks in remote debug mode
--debug-jvm
export GRADLE_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"
- this is same as above, but more flexible.

gw - run gradle in sub folders
brew install gdub

Config files
Add extra steps into init script
~/.gradle/init.d

~/.gradle/init.gradle

allprojects {
  apply plugin: 'maven-publish'
  buildscript {
    repositories {
      mavenLocal()
      .... 
    }
  }

  repositories {
    mavenLocal()
    ....
  }

}

settings.gradle
- Figure out which projects are to take part in a build.
include ':repository', ':services', ':web-app'
gradle.properties vs settings.gradle

Change subproject name
include "foo"
rootProject.name = 'projectName'
project(":foo").name = "foofoo"

build.gradle
allprojects {}
subprojects {}

settings.gradle in root folder
build.gradle for each module

gradle.properties

Cache
~/.gradle/caches, it can also be configured to use local or remote maven.

Task
Use doLast instead of <<
task copyJarToBin(type: Copy) {
    from createJar // shortcut for createJar.outputs.files
    into "d:/tmp"

task stopTomcat(type:Exec)


Related
Maven Tips and Tricks - 2016

Lessons Learned about Programming and Soft Skills - 2017

How to compare different approaches
- Always think about different approaches (even if you already finished/committed code)

- Don't just choose one that looks good
- List them and compare them
- Always ask you why choose this approache
- Try hard to find problems in your current approach, and how to fix them
For small coding
- Implement them if possible
- Then compare which makes code cleaner, less change etc
Example: Exclude source and javadoc from -jar
APP_BIN=$APP_BIN_DIR/$(ls $APP_BIN_DIR | grep -E 'jarName-version.*jar' | grep -v sources | grep -v javadoc | grep -v pom)

How to quickly scan/learn new classes
Sometimes we need quickly scan/check a bunch of related classes to check how to implement a function, use a method etc
- Check the class's Javadoc
- Check the class signature
- Check main methods:
  - static methods
  - using ctrol+o or outline view
- Check call hierarchy in source code
- Check test code/examples
- Google search code example


When refactor/change the code, also check/change/improve its related code.

Find related doc, check/read the doc carefully.
- Mark/Note the important part of the doc.

For some task, we can use the trial and error approach, just do it, then fix it.
But for some task(production or physical hardware related), it's better to figure out the right way to do it first.

Evaluate the outcome of the action. 
- best/worst outcome

How to implement/work on a feature
- what's goal, what to achieve
- how to test/verify/deploy/enable it in test or production environment easily 
  - useful tricks: dry-run, 
  - able to enable/disable default configuration automatically, but override it manually
- how to measure whether the change makes improvement

Think different/3+/more approaches
Compare them
Don't stop until find a solution that looks good to you
Use tools(notebook, whiteboard)
- Usually we are not happy with the first approach that comes to our minds, and find out a different approach: maybe it's a little better or maybe just be different. In some cases, we stop there: maybe we started to talk it with others or present it (to get others' ideas)
-Example: add Explanation to transform actions

Before ask a question
- Try to Solve it by yourself
- Make sure you read all related code/doc: from top to bottom(quickly, scan but don't ignore any code that may be important)

- Example: NightlyTestRunner

Verify the assumption
- Be aware of the the assumption we or others made in the design or the code.
- Verify whether it's true or not
- Example: one line one record in csv, one-to-one between tms id and bam_id

Alternatives
- Check and realize alternatives
Sometimes, we want A, but maybe B also works and is even better.
- Example: asset letter or account statement

Use tools
Write down in whiteboard or notebook or app
Take a picture now 
- always bring the phone

Check carefully and verify your claim before blame others or think others are wrong
We incline to think others are wrong or made a mistake even if someone told you he/she did that - we made a very brief search and didn't check carefully, then started to think they are wrong.

Prefer to use code to enforce the rule than documentation
Example: all tests must extends XbaseUnitTest.

Make API/feature easier
- to use
- to test/rollback in production (feature flag)

Realize your assumption/decision and verify it first
- Otherwise you may go farther but on the wrong path
Step by step and verify each step

Identify useful info and active quickly (or you may forget about it)
Don't let your past experience affect you
- Try it, it may be different this time
Example: big item delivery

When some thing totally doesn't make sense:
- maybe they are totally different things, you are comparing Apple with Banaba
Example: 12/15, 1/15

Related
Lessons Learned about Programming and Soft Skills - 2016

Bash Scripting Essentials

Brace Expansion
chown root /usr/{ucb/{ex,edit},lib/{ex?.?*,how_ex}}

Special Variables
$? Exit value of last executed command.
pid=$!
wait $pid

$! Process number of last background command.
$0 First word; that is, the command name. This will have the full pathname if it was found via a PATH search.
$n Individual arguments on command line (positional parameters).
$# Number of command-line arguments.

“$*” All arguments on command line as one string (“$1 $2…”). The values are separated by the first character in $IFS.
“$@” All arguments on command line, individually quoted (“$1” “$2” …).

Tests
-n string is not null.
-z string is null, that is, has zero length

Testing for File Characteristics
-d File is a directory
-e File exists
-f File is a regular file
-s File has a size greater than zero
-r, -w, -x, -s - socket

[ -d "$dir" ] && echo "$dir exists." || echo "$dir doesn't exists."

Testing with Pattern Matches
== pattern
=~ ere
if [[ "${MYFILENAME}" == *.jpg ]]

-a, &&
-o, ||

if [ ! -d $param ]
if [ $? -ne 0 ]

Parsing
HEAP_DUMP_DIR=$(sed 's/-XX:HeapDumpPath=\([^ ]*\)/\1/' <<< $param)

Function
Use variable $1, $2..$n to access argument passed to the function.
Hello () {
   echo "Hello $1 $2"
   return 10
}
Return value in bash
echo in the function, capture the result in caller $()

Hello a b

for i in $( command ); do command $i; done

for i in $( command ); do
  command $i
done
if [ "$var" == "value" ];  then

else
fi

Google Shell Style Guide
quote your variables; prefer "${var}" over "$var",
Use $(command) instead of backticks.
[[ ... ]] is preferred over [test
- [[ ... ]] reduces errors as no pathname expansion or word splitting takes place between [[ and ]] and [[ ... ]] allows for regular expression matching where [ ... ] does not

Use readonly or declare -r to ensure they're read only.
Make variable readonly: readonly var=value
Make function readonly: readonly -f function
readonly -p/-f

if [[ -f ~/.bashrc ]]; then
   source ~/.bashrc
fi
[ ! -f $FILE ] && { echo "$FILE not found"; exit -1; }

$(( $a+$b )) to execute arithmetic expressions
Put ; do and ; then on the same line as the while, for or if.
Prefer brace-quoting all other variables.
Use "$@" unless you have a specific reason to use $*.
- "$@" will retain arguments as-is, so no args provided will result in no args being passed on;
- "$*" expands to one argument, with all args joined by (usually) spaces, so no args provided will result in one empty string being passed on.

while IFS=, read var1 var2 var3; do
  ...

done < file.txt

Use Local Variables
local var="something"
local var
var="$(func)" || return

if [[ "${my_var}" = "some_string" ]]; then
  do_something
fi
-z (string length is zero) and -n (string length is not zero)

if ! mv "${file_list}" "${dest_dir}/" ; then

fi

Use set -o errexit (a.k.a. set -e) to make your script exit when a command fails.
Then add || true to commands that you allow to fail.
set -e - enable exit immediately
set +e - disable exit immediately
set -x  - print a trace Use set -o nounset (a.k.a. set -u) to exit when your script tries to use undeclared variables.
Use set -o xtrace (a.k.a set -x) to trace what gets executed. Useful for debugging.
set -u   Fail for undefined variable (set -o nounset)

Use $(( ... )), not expr for executing arithmetic expressions. which is more forgiving about space
Use (( or let, not $(( when you don't need the result

Identify common problems with shellcheck.

Misc
while true; do some_commands_here; done
while true
do
  some_commands_here
done

Run command until success
until $the_command; do echo "Try again"; done
while [ -n $(the command) ]; do echo "Try again";done;

Related
Essential Linux Commands for Developers

Labels

Java (162) Lucene-Solr (112) Interview (63) J2SE (53) Algorithm (45) Soft Skills (39) Eclipse (32) Code Example (31) Troubleshooting (27) JavaScript (23) Linux (22) Spring (22) Tools (22) Windows (22) Web Development (20) Nutch2 (18) Bugs (17) Dev Tips (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (10) Problem Solving (10) Google (9) How to (9) Learning code (9) Performance (9) Security (9) UIMA (9) html (9) Design (8) Http Client (8) Maven (8) bat (8) blogger (8) Big Data (7) Database (7) Guava (7) JSON (7) Shell (7) System Design (7) ANT (6) Coding Skills (6) Lesson Learned (6) Programmer Skills (6) Scala (6) css (6) Algorithm Series (5) Cache (5) Continuous Integration (5) IDE (5) Testing (5) adsense (5) xml (5) AIX (4) Become a Better You (4) Code Quality (4) Concurrency (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Life (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Review (4) Spark (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Distributed (3) Dynamic Languages (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Mac (3) Python (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Fiddler (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Logging (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Firefox (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) Invest (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Tips (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts