Tips and Tricks for Docker

docker run -d -p 80:80 --name webserver nginx
-v host-dir:container-dir
-p host-port:container-port

docker ps
Attach to existing container's shell
docker exec -it container_id /bin/sh

docker kill/stop $(docker ps -q)
docker build --no-cache .
docker-compose build --no-cache mysql

Delete all containers
docker rm $(docker ps -a -q)
Delete all images
docker rmi -f $(docker images -q)

Clean up disk space used by Docker
docker system df
docker system prune

Increase docker memory
In advance tab of preference, change the memory or cpu.
configure --memory 6g in docker run command to set a different value.

Disable container auto-run
docker update --restart=no my-container

List Docker Container Names and IPs
docker ps -q | xargs -n 1 docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} {{ .Name }}' | sed 's/ \// /'

docker stats
check memory and cpu usage

docker-compose up

Dockerfile
RUN creates an intermediate container, runs the script and freeze the new state of that container in a new intermediate image.

WORKDIR /path

ENTRYPOINT vs CMD 
Prefer exec form over shell form
docker run --entrypoint [my_entrypoint] containter_name [command 1] [arg1] [arg2]

Use COPY if don't need ADD's specical features

Use dockerfile to build each image, docker compose file to assemble them.

EXPOSE 8983-8986
USER builder

docker-compose
up, down, logs, 
scale service=number
docker-compose build

Volumes
1. absolute path or relative path started with ./ means local path
2. named volume that references from the volumes list
docker volume prune
docker volume ls
docker inspect $volume_name

Misc
Change hostname
docker run -it -h myhost ...
docker run --rm -it --cap-add SYS_ADMIN ...

Troubleshooting
docker logs $container_id
docker events &

How Solr Create Collection - Learn Solr Code

Test Code to create collections
MiniSolrCloudCluster cluster = new MiniSolrCloudCluster(4 /*numServers*/, testBaseDir, solrXml, JettyConfig.builder().setContext("/solr").build());
cluster.createCollection(collectionName, 2/*numShards*/, 2/*replicationFactor*/, "cie-default", null);

CollectionsHandler

CollectionsHandler.handleRequestBody(SolrQueryRequest, SolrQueryResponse)

CollectionAction action = CollectionAction.get(a); // CollectionAction .CREATE(true)

CollectionOperation operation = CollectionOperation.get(action); //CollectionOperation .CREATE_OP(CREATE)

Map result = operation.call(req, rsp, this);

Return a mpa like this:

{name=collectionName, fromApi=true, replicationFactor=2, collection.configName=configName, numShards=2, stateFormat=2}

ZkNodeProps props = new ZkNodeProps(result);

if (operation.sendToOCPQueue) handleResponse(operation.action.toLower(), props, rsp, operation.timeOut);

CollectionsHandler. handleResponse

QueueEvent event = coreContainer.getZkController() .getOverseerCollectionQueue() .offer(Utils.toJSON(m), timeout);

This uses DistributedQueue.offer(byte[] data, long timeout) to add a task to /overseer/collection-queue-work/qnr-numbers.

It uses LatchWatcher to wait until this task is processed.

Overseer and OverseerCollectionProcessor

OverseerCollectionProcessor.processMessage(ZkNodeProps, String operation /*create*/)

OverseerCollectionProcessor.processMessage(ZkNodeProps, String)

{

  "name":"collectionName",

  "fromApi":"true",

  "replicationFactor":"2",

  "collection.configName":"configName",

  "numShards":"2",

  "stateFormat":"2",

  "operation":"create"}

  

OverseerCollectionProcessor.createCollection(ClusterState, ZkNodeProps, NamedList)

  ClusterStateMutator.getShardNames(numSlices, shardNames);

   positionVsNodes = identifyNodes(clusterState, nodeList, message, shardNames, repFactor); // round-robin if rule not set

  createConfNode(configName, collectionName, isLegacyCloud);

    Overseer.getInQueue(zkStateReader.getZkClient()).offer(Utils.toJSON(message));

// This message will be processed by ClusterStateUpdater

// wait for a while until we do see the collection in the clusterState

  for (Map.Entry e : positionVsNodes.entrySet()) {

  if (isLegacyCloud) {

    shardHandler.submit(sreq, sreq.shards[0], sreq.params);

  } else {

    coresToCreate.put(coreName, sreq);

  }

}

This will send http call and be handled by CoreAdminHandler.handleRequestBody.

ClusterStateUpdater

  // if there were any errors while processing

  // the state queue, items would have been left in the

  // work queue so let's process those first

  byte[] data = workQueue.peek();

  boolean hadWorkItems = data != null;

  while (data != null)  {

    final ZkNodeProps message = ZkNodeProps.load(data);

    clusterState = processQueueItem(message, clusterState, zkStateWriter, false, null);

    workQueue.poll(); // poll-ing removes the element we got by peek-ing

    data = workQueue.peek();

  }

ClusterStateUpdater.processQueueItem

  zkWriteCommand = processMessage(clusterState, message, operation);

  stats.success(operation);

  clusterState = zkStateWriter.enqueueUpdate(clusterState, zkWriteCommand, callback);

processMessage

  case CREATE:

    return new ClusterStateMutator(getZkStateReader()).createCollection(clusterState, message);

overseer.ClusterStateMutator.createCollection(ClusterState, ZkNodeProps)

Gradle Tips and Tricks - 2017

Run tasks on sub-projects only
./gradlew sub-project:build

Tasks
./gradlew classes
tasks

Gradle task options Skip Tasks
-x test -x testClasses -x javadoc -x javadocJar -x integrationTestClasses 
-x findbugsMain -x findbugsTest -x findbugsIntegrationTest -x pmdMain -x pmdTest -x pmdIntegrationTest 

--daemon
--dry-run 
--profile
--parallel
Troubleshooting
--debug

-s, --stacktrace

Run specific tests
gradle test --tests org.gradle.SomeTest.someMethod
gradle test --tests org.gradle.SomeTest
gradle test --tests org.gradle.internal*
//select all ui test methods from integration tests by naming convention
gradle test --tests *IntegTest*ui*
//selecting tests from different test tasks
gradle test --tests *UiTest integTest --tests *WebTest*ui

Install an artifact locally
apply plugin: 'maven-publish'
gradle publishToMavenLocal

Using artifacts from local maven
apply plugin: "maven"
allprojects {
  repositories {
      mavenLocal()
      mavenCentral()
  }
}

Run tasks in remote debug mode
--debug-jvm
export GRADLE_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"
- this is same as above, but more flexible.

gw - run gradle in sub folders
brew install gdub

Config files
Add extra steps into init script
~/.gradle/init.d

~/.gradle/init.gradle

allprojects {
  apply plugin: 'maven-publish'
  buildscript {
    repositories {
      mavenLocal()
      .... 
    }
  }

  repositories {
    mavenLocal()
    ....
  }

}

settings.gradle
- Figure out which projects are to take part in a build.
include ':repository', ':services', ':web-app'
gradle.properties vs settings.gradle

Change subproject name
include "foo"
rootProject.name = 'projectName'
project(":foo").name = "foofoo"

build.gradle
allprojects {}
subprojects {}

settings.gradle in root folder
build.gradle for each module

gradle.properties

Cache

~/.gradle/caches, it can also be configured to use local or remote maven.

Task
Use doLast instead of <<
task copyJarToBin(type: Copy) {
    from createJar // shortcut for createJar.outputs.files
    into "d:/tmp"
} 
task stopTomcat(type:Exec)


Related
Maven Tips and Tricks - 2016

Lessons Learned about Programming and Soft Skills - 2017

How to compare different approaches
- Always think about different approaches (even if you already finished/committed code)

- Don't just choose one that looks good
- List them and compare them
- Always ask you why choose this approache
- Try hard to find problems in your current approach, and how to fix them
For small coding
- Implement them if possible
- Then compare which makes code cleaner, less change etc
Example: Exclude source and javadoc from -jar
APP_BIN=$APP_BIN_DIR/$(ls $APP_BIN_DIR | grep -E 'jarName-version.*jar' | grep -v sources | grep -v javadoc | grep -v pom)

How to quickly scan/learn new classes
Sometimes we need quickly scan/check a bunch of related classes to check how to implement a function, use a method etc
- Check the class's Javadoc
- Check the class signature
- Check main methods:
  - static methods
  - using ctrol+o or outline view
- Check call hierarchy in source code
- Check test code/examples
- Google search code example


When refactor/change the code, also check/change/improve its related code.

Find related doc, check/read the doc carefully.
- Mark/Note the important part of the doc.

For some task, we can use the trial and error approach, just do it, then fix it.
But for some task(production or physical hardware related), it's better to figure out the right way to do it first.

Evaluate the outcome of the action. 
- best/worst outcome

How to implement/work on a feature
- what's goal, what to achieve
- how to test/verify/deploy/enable it in test or production environment easily 
  - useful tricks: dry-run, 
  - able to enable/disable default configuration automatically, but override it manually
- how to measure whether the change makes improvement

Think different/3+/more approaches
Compare them
Don't stop until find a solution that looks good to you
Use tools(notebook, whiteboard)
- Usually we are not happy with the first approach that comes to our minds, and find out a different approach: maybe it's a little better or maybe just be different. In some cases, we stop there: maybe we started to talk it with others or present it (to get others' ideas)
-Example: add Explanation to transform actions

Before ask a question
- Try to Solve it by yourself
- Make sure you read all related code/doc: from top to bottom(quickly, scan but don't ignore any code that may be important)

- Example: NightlyTestRunner

Verify the assumption
- Be aware of the the assumption we or others made in the design or the code.
- Verify whether it's true or not
- Example: one line one record in csv, one-to-one between tms id and bam_id

Alternatives
- Check and realize alternatives
Sometimes, we want A, but maybe B also works and is even better.
- Example: asset letter or account statement

Use tools
Write down in whiteboard or notebook or app
Take a picture now 
- always bring the phone

Check carefully and verify your claim before blame others or think others are wrong
We incline to think others are wrong or made a mistake even if someone told you he/she did that - we made a very brief search and didn't check carefully, then started to think they are wrong.

Prefer to use code to enforce the rule than documentation
Example: all tests must extends XbaseUnitTest.

Make API/feature easier
- to use
- to test/rollback in production (feature flag)

Realize your assumption/decision and verify it first
- Otherwise you may go farther but on the wrong path
Step by step and verify each step

Identify useful info and active quickly (or you may forget about it)
Don't let your past experience affect you
- Try it, it may be different this time
Example: big item delivery

When some thing totally doesn't make sense:
- maybe they are totally different things, you are comparing Apple with Banaba
Example: 12/15, 1/15

Related
Lessons Learned about Programming and Soft Skills - 2016

Bash Scripting Essentials

Brace Expansion
chown root /usr/{ucb/{ex,edit},lib/{ex?.?*,how_ex}}

Special Variables
$? Exit value of last executed command.
pid=$!
wait $pid

$! Process number of last background command.
$0 First word; that is, the command name. This will have the full pathname if it was found via a PATH search.
$n Individual arguments on command line (positional parameters).
$# Number of command-line arguments.

“$*” All arguments on command line as one string (“$1 $2…”). The values are separated by the first character in $IFS.
“$@” All arguments on command line, individually quoted (“$1” “$2” …).

Tests
-n string is not null.
-z string is null, that is, has zero length

Testing for File Characteristics
-d File is a directory
-e File exists
-f File is a regular file
-s File has a size greater than zero
-r, -w, -x, -s - socket

[ -d "$dir" ] && echo "$dir exists." || echo "$dir doesn't exists."

Testing with Pattern Matches
== pattern
=~ ere
if [[ "${MYFILENAME}" == *.jpg ]]

-a, &&
-o, ||

if [ ! -d $param ]
if [ $? -ne 0 ]

Parsing
HEAP_DUMP_DIR=$(sed 's/-XX:HeapDumpPath=\([^ ]*\)/\1/' <<< $param)

Function
Use variable $1, $2..$n to access argument passed to the function.
Hello () {
   echo "Hello $1 $2"
   return 10
}
Return value in bash
echo in the function, capture the result in caller $()

Hello a b

for i in $( command ); do command $i; done

for i in $( command ); do
  command $i
done
if [ "$var" == "value" ];  then

else
fi

Google Shell Style Guide
quote your variables; prefer "${var}" over "$var",
Use $(command) instead of backticks.

[[ ... ]] is preferred over [, test

- [[ ... ]] reduces errors as no pathname expansion or word splitting takes place between [[ and ]] and [[ ... ]] allows for regular expression matching where [ ... ] does not




Use readonly or declare -r to ensure they're read only.

Make variable readonly: readonly var=value

Make function readonly: readonly -f function

readonly -p/-f

if [[ -f ~/.bashrc ]]; then

   source ~/.bashrc

fi

[ ! -f $FILE ] && { echo "$FILE not found"; exit -1; }

$(( $a+$b )) to execute arithmetic expressions

Put ; do and ; then on the same line as the while, for or if.

Prefer brace-quoting all other variables.

Use "$@" unless you have a specific reason to use $*.

- "$@" will retain arguments as-is, so no args provided will result in no args being passed on;

- "$*" expands to one argument, with all args joined by (usually) spaces, so no args provided will result in one empty string being passed on.

while IFS=, read var1 var2 var3; do

  ...

done < file.txt

Use Local Variables

local var="something"
local var
var="$(func)" || return

if [[ "${my_var}" = "some_string" ]]; then
  do_something
fi
-z (string length is zero) and -n (string length is not zero)

if ! mv "${file_list}" "${dest_dir}/" ; then

fi

Best Practices
set -euxo pipefail

Use set -o errexit (a.k.a. set -e) to make your script exit when a command fails.

Then add || true to commands that you allow to fail.

set -e - enable exit immediately
set +e - disable exit immediately
set -x  - print a trace Use set -o nounset (a.k.a. set -u) to exit when your script tries to use undeclared variables.
Use set -o xtrace (a.k.a set -x) to trace what gets executed. Useful for debugging.
set -u   Fail for undefined variable (set -o nounset)

Use $(( ... )), not expr for executing arithmetic expressions. which is more forgiving about space

Use (( or let, not $(( when you don't need the result

Identify common problems with shellcheck.

Misc
while true; do some_commands_here; done
while true
do
  some_commands_here
done

Run command until success
until $the_command; do echo "Try again"; done
while [ -n $(the command) ]; do echo "Try again";done;

Related
Essential Linux Commands for Developers