How to Improve Design Skills


How to Improve Design Skills




How to Improve Design Skills

Jeffery Yuan
Updated at June 16, 2019

Agenda

  • How to Design
  • System Design Principles
  • Learning from Open Source
  • Learning from Existing Products
  • System Design Practices

How to Design

How to Design

  • Take time to think about your design
    • Minimize upfront design or YAGNI
    • It doesn’t mean you don’t take time to design the component
  • Components related
  • Impact to other components
  • What are alternatives?
  • Welcome different approaches and discussion

How to Design

  • Estimation
    • back-of-the-envelope calculation
  • Estimated data size, QPS
  • Take time to design data schema
    • As it’s difficult to change them after deploy to prod
  • Better user experience
    • Thinking from client/user perspective
    • How they use it, what they would like to know

Reflection – Lesson Learned

Reflection – Lesson Learned

  • What mistakes we made
    • Where to store data: dynamodb or not?
    • The key for Solr schema
  • Why they happened:
  • Not consider near-future requirements
  • Make decisions carelessly

Reflection – Lesson Learned

  • Better client library
    • Only contains library and code that client need
  • Package shared configuration in the library

System Design Principles

System Design Principles

  • Idempotent
  • Policy to expire/archive data - Less data
  • Optimize data for read
    • Denormalization
  • Read Heavy vs Write Heavy
  • Design to Be Disabled - feature toggle
  • Isolate Faults - Circular breaker
  • Throttling - Rate limit

System Design Principles Cont.

  • Stateless
  • Asynchronous
    • Back pressure with exponential backoff
  • Message queues
  • Cache
  • Visibility – monitoring
  • Separation of concerns
    • Separate read and write

System Design Principles Cont.

  • CAP
  • Graceful Degradation
  • Be Robust - Hide error as much as possible
  • Be conservative in what you send, be liberal in what you accept
  • Make your apps do something reasonable even if not all is right

Learning from Open Source

Learning from Open Source

  • What makes them popular
  • When to use them, when not

Cassandra

  • LSM(Log Structured Merge Trees)
    • append-only
  • SSTable
  • MemTable - SSTable in memory
  • How C* handles delete: Tombstone(grace period)
  • Merkle trees
  • Bloom Filter
  • Index
  • CommitLog

Cassandra Cont.

  • Serialize cache data (row-cache, key cache) to avoid cold restart
  • Session Coordinator
  • Gossip protocol
  • Seed nodes
  • Consistent Hashing
  • Eventual Consistency
    • W+R > N
  • Local Index (vs Global Index)

Kafka

  • Why it is fast
  • Sequentially read/write vs random read/write
  • Memory Mapped File
  • Zero copy
  • Batch data(compressed)
  • Partition: ordered, immutable, replicated
  • Consumer group

Database

  • Sharding
  • Replication
  • Master/Slave, Multi-master

Learning from Existing Products

  • Twitter/FB timeline
  • Pull/Push/Mixed Model
  • FB Haystack/Photo storage

System Design Practices

System Design Practices

  • URL shortener
    • read heavy
    • able to disable write functions
  • Design key-value store
  • Crawler
    • Re-crawling
    • cur+2t or cur+t/2 based on changed or not
  • Design search engine
    • In-memory version: Data structure
    • Distributed: Solr Cloud internal design

System Design Practices

  • Design score/rank system for social game
  • Search nearby places: GeoHash
  • Design Chat app
  • Design logging collection and analysis system
  • Design shopping cart
    • guest cart
  • Design Hit Counter
  • Design rate limiter
  • Design Miao Sha

Resource

Resource

  • Designing Data-Intensive Applications
  • Scalability Rules: Principles for Scaling Web Sites
  • The Art of Scalability

Resource Cont.

How to Improve Your Skills as a Programmer




How to Improve Your Skills as a Programmer

How to Improve Your Skills as a Programmer

Jeffery Yuan

Updated at June 16, 2019

Skills that are important for engineers

  • Problem Solving
  • Coding
  • Design
  • Eagerness to learn
  • Focus on soft skills in this presentation

Learn from others

  • Be humble and open-minded
    • If others are doing things different, use different tools, anything that you don’t know
    • Ask, learn and try them
  • Learn the knowledge itself
  • But also learn their thinking process
  • Ask help for same/similar things once, then you can do it

Learn how to do daily work better

Learn how to do daily work better

  • How to write clean and bug-free code
  • Source of common bugs
  • Learn from Bugs/Mistakes we made

Learn how to do daily work better Cont.

Learn the things you use

Learn the tools you use everyday

Learn the tools you use everyday

Mentoring/teaching

  • Improve your skills:
    • quick thinking, trouble shooting on spot
  • Save your time
    • don’t only tell the solution, but how you analyze the problem
  • Grow as a team/together
  • Good for the team

Share your knowledge

  • Take time to reflect and write down
  • you will have a better understanding
  • Create the environment that people share knowledge
    • you can also learn from others
  • Share formally or informally
    • slack, email, or group discussion
    • team-tech-only slack

Communication Skills

  • Listen before Ask/Talk
  • How to ask question/help
  • Be helpful
  • Keep calm and focused

What to learn

  • Trending(Atom)
  • Common tech used
    • Kafka/C*/Solr etc
  • How Big Tech Companies Build Things

Java Optional Best Practices


Linux File Manager Nemo Tips and Tricks


Spell and Grammar Check for Markdown


Best Practices to Handle Exceptions in Java


Best Practices to Handle Exceptions in Java

Throw Exception

Include descriptive contextual messages in the exception
Throw specific checked exceptions, don’t not RuntimeException, Exception, or Throwable
  • The checked exception is part of the contract of the method, so client knows what exceptions it can catch and handle.
  • If one method throws both base exception and one of its sub-class exception, we should declare both (the base and the specific sub-class) exception not just the base exception.
Throw exception in the right abstraction
  • Wrap low level exception in custom high level exception
Using Custom Exceptions
Don’t lose original exception
  • It’s common that we re-throw but when we do this, make sure to include the original exception as the cause otherwise we will lose the original error message and stack trace.
catch (SomeException e) {
   throw new MyServiceException("Some information: " , e);
}

Throw early catch late

You want to throw an exception as soon as possible because that makes it easier to find the cause

exceptions are handled at higher levels is because the lower levels don’t know what’s the appropriate course of action to handle the error

Valid parameter, user input early

Logging

Don’t Log the exception and re-throw
  • It will write same exception multiple times.
catch (DaoException e) {
   LOGGER.error("xxx", e);
   throw e;
}

Catch Exception

  • Catch specific exception, not the Exception class
  • Never swallow the exception
    • If you do not want to handle it, then re-throw it. The top layer must handle the exception and translate it into what the user can understand.
  • Never catch Error or Throwable
Only catch those exceptions that we can handle
  • We may catch it to add more information.
  • But if we can’t handle it, it’s better to let the caller decide what to do.

Use try-catch only if it is difficult to deal with pre-check, such as NumberFormatException.

Finally

Never use return within a finally block.
  • A return statement in a finally block will cause exceptions or result in a discarded return value in the try-catch block.
Don’t throw exception in the finally block
  • It will override the original exception.
Close resources: stream
  • IOUtils.closeQuietly(stream) from Apache commons-io
Using try-with-resources

Be careful of NullPointerException

Misc

  • Never use exceptions for control flow
Document all exceptions in public API
  • When the exception may throw.
  • How client can handle it.

Autoboxing and Unboxing Tips and Traps in Java


Autoboxing and Unboxing Tips and Traps in Java

What’s Autoboxing

  • Autoboxing is just syntactic sugar to convert a primitive value into an object of the corresponding wrapper class.
  • Example: Integer i = 2; is actually equivalent to Integer i = Integer.valueOf(2);

What’s Unboxing

  • Unboxing is also just syntactic sugar to convert an object of a wrapper type to its corresponding primitive value
  • Example: int i = new Integer(1); is actually equivalent to int i = new Integer(1).intValue();

When Autoboxing/unboxing Happens

  • Passed as a primitive type into a method that accepts a wrapper type or vice versa.
  • Assigned a primitive type to a variable of wrapper type or vice versa.
  • To use them in generics
    • Java generic types only accepts objects, so when we add primitive types into the collection, autoboxing happens, when int i=list.get(0);, unboxing happens.
  • ++ or -- on a wrapper instance.
  • Mix primitive and wrapper relational or equality operator
Map<String, Boolean> map = new HashMap<>();
// unboxing: map.get(key).booleanValue();
if(map.get(key)) {}

Integer i;
// unbox
if(i < 0) {}

Always use Objects.equals instead of == to compare wrapper class

  • From javadoc > This method will always cache values in the range -128 to 127, inclusive, and may cache other values outside of this range
Integer a = 127, b = 127;
// true
System.out.println(a==b);


a = 128; b = 128;
// false
System.out.println(a==b);

Unboxing and NullPointerException

When mix wrapper class and primitive type, Java will auto unbox the wrapper class to primitive type(boolean, int, long, etc), if the wrapper class is Null, then it would throw NullPointerException.

Map<String, Boolean> map = new HashMap<>();
if (map.get(key)) {}

In the above code, when map doesn’t contain the key, or its value is null, it would throw NullPointerException. One fix is to use Objects.equals

Widening or Boxing, but not both

  • Long l = 0; will not compile: Type mismatch: cannot convert from int to Long
  • s 0 is an int, java doesn’t first widen it to long, then autobox it to Long.

Some Puzzles

Long l = 0L;
System.out.println(l.equals(0L)); // true
// false
// Eclipse warning: Unlikely argument type for equals(): int seems to be
// unrelated to Long
System.out.println(l.equals(0)); // false
System.out.println(l == 0); // true

l = 128L;
System.out.println(128 == 128); // true
  • in l.equals(0), 0 is autoboxed to Integer then compared with Long l. > The result is true if and only if the argument is not null and is a Long object that contains the same long value as this object
public boolean java.lang.Long.equals(Object obj) {
    if (obj instanceof Long) {
        return value == ((Long)obj).longValue();
    }
    return false;
}
  • in l == 0, Long l is first unboxed to long, then we are compare two primitive types, so it returns true.

No Widen for wrapper type

  • The following code will not compile: Type mismatch: cannot convert from Integer to Long
Integer i = 0;
// Compare error: Type mismatch: cannot convert from Integer to Long
Long l = i;
  • Integer and Long is not direct related (no parent, child relation), so there is no way to convert from Integer to Long/
  • Java first unbox the Integer to an int, then widen it to a long.
public static void f(long l) {
  System.out.println("flong called");
}
public static void f(Long l) {
  System.out.println("fLong called");
}
Integer i = 0;
// print: flong called
f(i);

Widening takes priority than boxing

  • Auto widening transforms a variable to another with a wider type: from int to long etc.
  • It’s just a syntactic sugaring: (long)the_int_value;
  • Auto widening is easier and costs less than boxing.

Boxing takes priority than Varargs

  • Varargs is just a syntactic sugaring on top of arrays: javac has to create an array to wrap the parameters.
  • Boxing is easier and costs less than Varargs.

Varargs gotchas: passing an array of primitives

public static String ezFormat(Object... args) {
  String format = new String(new char[args.length]).replace("\0", "[ %s ]");
  return String.format(format, args);
}

int[] myNumbers = { 1, 2, 3 };
// [ [I@7852e922 ]
System.out.println(ezFormat(myNumbers));

// [ [I@4e25154f ]
System.out.println(ezFormat(new int[] { 1, 2, 3 }));

Performance Hit

  • Frequent autoboxing and unboxing will create unnecessary objects.

Add Google Custom Search to WebSite


Git Tips Tricks for Developers


Untitled

Install Bash git completion

  • brew install git bash-completion

Caching GitHub password in Git

  • git config --global credential.helper 'cache --timeout 72000'
  • Remove cached password (to reenter new password when it’s changed)
    • git credential-osxkeychain erase

Temporarily ignoring files

  • git update-index --assume-unchanged file-path
  • git update-index --no-assume-unchanged file-path

Remove file from the repository but keep it locally

  • git rm --cached -r somedir/somefile then add them to .gitignore

Reset current HEAD to the specified state

undo last commit
  • git reset --soft HEAD~1
  • Undo commits permanently: git reset --hard HEAD~3
    • The last three commits (HEAD, HEAD^, and HEAD~2) were bad and you do not want to ever see them again.
  • Undo a merge or pull inside a dirty working tree: git reset –merge ORIG_HEAD
  • return to any previous revision
    • git reset --soft commit_hash
  • Reset committed changes to local changes
    • git reset HEAD~

undo git pull

  • git reset --hard

Reset local branch to be same as remote branch

  • git fetch --all && git reset --hard upstream/master
  • If need delete everything(just like re-clone): git clean -x -d -f

~ vs ^

  • Both ~ and ^ on their own refer to the parent of the commit.
  • ~2 means up two levels in the hierarchy, via the first parent if a commit has more than one parent
  • ^2 means the second parent where a commit has more than one parent (i.e. because it’s a merge)
  • HEAD~2^3 means HEAD’s grandparent commit’s third parent commit.

rebase

Edit your X most recent commits interactively (squash, fixup, reword, drop)
  • git rebase -i HEAD~X
Modify a specific commit
Split a commit in two with Git
Delete a commit
  • If it is the last commit this is very straight forward. git reset HEAD

Branch

  • Delete local branch: git branch -d <local_branchname>
  • Delete remote branch: git push origin --delete <remote_branchname> or git push origin :<remote_branchname>
  • Rename local branch name: git branch -m <new-branch-name>
Checkout remote brach

git log

  • git log –oneline [–pretty=oneline]
Show commit id only
  • git log -1 –pretty=format:“%H”

View change in a specific commit

  • git show

Undo anything

Wipes your unstaged changes
  • git checkout
undo rm -rf * - git stash - if no local update

undo git commit –amend

  • git reset –soft HEAD@{1}
Checkout remote file after merge conflict
  • git checkout HEAD the_file

shallow clone

  • git clone --depth 1 https://path/to/repo/foo.git -b bar
  • –no–single–branch instead -b bar, to download first commit form ALL branches.
  • –no-checkout(-n): No checkout of HEAD is performed after the clone is complete.

ls-remote

git ls-remote --tags remote_git_url

Tags

  • git tag

git clean

  • -n: –dry-run
  • git clean -x -d -f

git commit

  • Add change to your last commit, instead of making a new commit
    • git commit --amend
Commit messages
  • <type>(<scope>): <subject>
  • type: feat(new feature), fix, docs, style(format code, no code change), refactor, test(add test cases), chore

git remote

  • List all remotes: git remote
  • Add another remote: git remote add upstream the_url
  • git remote show
  • git remote show upstream

git stash

  • -u|–include-untracked
  • List all stashes: git stash list
  • git stash
  • Also stack untracked files: git stash -u
  • Apply the stash: git stash apply <stash@{n}>
  • Apply last stash and remove the stash: git stash pop
  • git stash clear
  • git checkout <stash@{n}> -- <filePath>

git diff

  • Ignore the white space: git diff -w
  • Show local staged change(added but not committed): git diff --cached
  • Show committed but not pushed change: git diff origin/master..master

emoji

Misc

How to Copy Text From Copy-Protected Web Pages


How to Copy Text From Copy-Protected Web Pages

Awesome Tips about Chrome Series


The Problem

Sometimes we want to copy some text/code from a website, but find out that it disables copy text or even completely disables right click, right context menu. It’s annoying, but we can fix it easily.

Using Proxy Sites

Specific Chrome Extensions

We can use the following Specific Chrome extensions to make us able to copy text.

Reader View Chrome Extensions

If we don’t want to install the above specific Chrome extensions, then we can use the more general reader view chrome extension which can also make us able to copy text.

Read here for more must-have google chrome extensions

Reader View

  • This opens the page in chrome-extension:// and we have to wait until (at least) the main content of the page is loaded.
  • The greatest thing is it reduces the memory usage a LOT for pages that load a lot of flashes, ads.
    • e.g. reduces this page from around 450+ mb to 87mb.
  • This also works well together with The Great Suspender: the page in reader view will not be suspended as it runs in extension mode.
  • Assign shortcut Ctrl+Shift+R to Toggle the Reader View
  • Print and save the reader mode version, show or hide image.
  • Able to change the speed of text-to-speech
  • The Cron is it doesn’t work with other extensions, e.g., Chrome Vimium
    • as It opens the page in chrome-extension.
Tips

Just Read

  • Able to change theme(dark or light), width, font size and save the config.
  • Assign Ctrl+Shift+L to Open the Just Read format for the current page
  • Assign Ctrl+Shift+K to Enable user text selection mode using Just Read, then choose which part you want to select then enter (it will automatically try best to choose), it will show the section in read mode.
  • When it doesn’t work(e.g. in quip), use View this selection in Just Read.
  • Works with Chrome Vimium

Clearly

  • Assign shortcut Ctrl+Shift+C to Toggle Clearly
  • Make not easy to copy text easier to copy
  • Support outline, different themes, fullscreen, text-to-voice
  • Cron: Can’t change the width
  • Distill Page from Chrome
  • the result is not that good, and hard to run the command.

Mercury Reader

  • can’t change width

Summary

  • Use Reader View when the page(website) is slow: load lots of ads, flash, uses a lot of memory and you will take time(2+ mins) to read it.
  • Use Just Read in other case.
  • Use Clearly to check the outline.

Prefer Collections over Arrays


How to Make Money with the Amazon Associates Program


How to Make Money with the Amazon Associates Program

Amazon Affiliate Program provides different kinds of ads.

Native Shopping Ads

  • amzn_assoc_placement should be unique.
  • amzn_assoc_search_bar: false/true to whetehr show the search bar.

Custom Ads

Custom Ads allows us to curate specific products from Amazon that you would like to recommend and place the ad unit into your webpage. All we need is find a list ASINs of products and add them into amzn_assoc_asins.

<script type="text/javascript">
amzn_assoc_placement = "adunit0";
amzn_assoc_search_bar = "false";
amzn_assoc_tracking_id = "the_tracking_id";
amzn_assoc_ad_mode = "manual";
amzn_assoc_ad_type = "smart";
amzn_assoc_marketplace = "amazon";
amzn_assoc_region = "US";
amzn_assoc_title = "Best books for algorithms interview";
amzn_assoc_linkid = "the_amzn_assoc_linkid";
amzn_assoc_asins = "0984782850,1517671272,1617292230,032157351X,1537713949,0262033844,111941847X,819324527X";
</script>
<script src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US"></script>

<div id="amzn-assoc-ad-19c89cc6-6f8f-485d-a31a-7bc49e98e831"></div><script async src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US&adInstanceId=19c89cc6-6f8f-485d-a31a-7bc49e98e831"></script>

Recommendation Ads

  • Displays products related to your page content and visitors
  • We can choose “Emphasize Categories”.

Search Ads

  • Display search results from Amazon.com by specifying a search phrase and selecting a relevant category to your page content in a responsive ad unit that works across screen sizes.

  • We can specify a list of search phrases and sort them by priority, random pick one search phrase and stick to it in the X(10) days.

<script type="text/javascript">
  function createCookie(name, value, days) {
    if (days) {
      var date = new Date();
      date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
      var expires = "; expires=" + date.toGMTString();
    } else var expires = "";
    document.cookie = name + "=" + value + expires + "; path=/";
  }

  function readCookie(name) {
    var nameEQ = name + "=";
    var ca = document.cookie.split(";");
    for (var i = 0; i < ca.length; i++) {
      var c = ca[i];
      while (c.charAt(0) == " ") c = c.substring(1, c.length);
      if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length, c.length);
    }
    return null;
  }

  function deleteCookie(name, path, domain) {
    if (readCookie(name)) createCookie(name, "", -1, path, domain);
  }

  function pickOne(weightedPosts) {
    var totalWeight = 0;
    for (let post of weightedPosts) {
      totalWeight += post.weight;
    }
    var randomWeight = Math.random() * totalWeight;
    var currWeight = 0;
    for (let post of weightedPosts) {
      currWeight += post.weight;
      if (currWeight > randomWeight) {
        return post;
      }
    }
  }
</script>

<script type="text/javascript">
var pos = readCookie("amazonItemPos")
var products = Array("iphone",  "iPad Pro", "iPad Mini", "apple watch series 4", "MacBook Pro", "MacBook Air", "google pixel 3", "chromebook", "Amazon Echo");

var items = new Set();
var totalCount = products.length;
for (var i = 0; i < products.length; i++) {
  items.add(
    {
      product: products[i],
      weight: totalCount - i,
      pos: i
    }
  );
}
// var item = items[Math.floor(Math.random() * items.length)];
if (pos == null || pos <0 || pos > products.length) {
  // pos = Math.floor(Math.random() * items.length);
  pos = pickOne(items).pos
  createCookie("amazonItemPos", pos, 10);  
}

var item = products[pos];
amzn_assoc_placement = "adunit3";
amzn_assoc_search_bar = "true";
amzn_assoc_tracking_id = "the_tracking_id";
amzn_assoc_search_bar_position = "bottom";
amzn_assoc_ad_mode = "search";
amzn_assoc_ad_type = "smart";
amzn_assoc_marketplace = "amazon";
amzn_assoc_region = "US";
amzn_assoc_title = "";
amzn_assoc_default_search_phrase = item;
amzn_assoc_default_category = "All";
amzn_assoc_linkid = "the_amzn_assoc_linkid";
</script>
<script src="//z-na.amazon-adsystem.com/widgets/onejs?MarketPlace=US"></script>

Tips

Mockito Best Practices


User Scripts to Change Html5 Video Player Speed


The Goal

I Would like to change the speed of HTML5 video player at pan.baidu.com, so I can spend less time on the video and finish it sooner.

The Implementation

I found some user script at greasyfork for Chrome with tampermonkey extension

Based on these 2 script, I made the following simple user script that can change video speed to 1.5, 1.75, 2.0.

Other Ideas

  • Able to resume playback of a video at the point in time it was left.

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (4) ANT (6) bat (8) Become a Better You (4) Big Data (7) Blogger (14) Bugs (4) Cache (5) Chrome (17) Code Example (29) Code Quality (6) Coding Skills (5) Concurrency (4) Database (7) Debug (16) Design (5) Dev Tips (62) Eclipse (32) GAE (4) Git (5) Good Programming Practices (4) Google (27) Guava (7) How to (9) Http Client (8) IDE (6) Interview (88) J2EE (13) J2SE (49) Jackson (4) Java (177) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (22) Lucene-Solr (112) Mac (10) Maven (8) Memory Usage (4) Network (9) Nutch2 (18) OpenNLP (4) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Review (4) Scala (6) Security (9) Soft Skills (38) Spark (4) Spring (22) System Design (11) Testing (6) Text Mining (14) Tips (12) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)

Trending