A Complete DNS Setup Guide on Redhat(CentOS)

Background
When install cloudera cluster recently, I have to setup a private DNS server.
Environment
Private DNS server: 172.19.97.224(bigdatam.localdomain.com)
DNS client:  172.19.101.66(bigdata1.localdomain.com) and 172.19.102.56(bigdata2.localdomain.com.

Install bind and caching-nameserver
yum install bind  bind-utils bind-libs bind-chroot caching-nameserver -y

Run service named restart  to start named server first.

Configure DNS Server
Enable Caching nameserver and Create Zones
Edit /var/named/chroot/etc/named.conf:
1. Use forwarders block to forward DNS requests it can't resolve to upstream DNS server.
2. Add forward and reverse zones block for localdomain.com and 19.172.in-addr.arpa.
3. Add forward and reverse zones block for locahost and 0.0.127.in-addr.arpa.

vi /var/named/chroot/etc/named.conf 

acl localdomain-com { 172.19/16; };
options {
  directory  "/var/named";
  allow-query { localdomain-com; };
 # The block will cause the caching name server to forward DNS requests it can't resolve to upstream DNS server.
  forwarders { upstream-dns-server1; upstream-dns-server2};
  #forward only;
};
zone "localdomain.com" IN {
 type master;
 file "localdomain.com.zone";
};
zone "19.172.in-addr.arpa" IN {
 type master;
 file "172.19.zone";
};

zone "locahost" IN {
        type master;
        file "localhost.zone";
        allow-update{none;};
};

zone "0.0.127.in-addr.arpa" {
        type master;
        file "named.local";
};
zone "." {
 type hint;
 file "named.ca";
};

Add Zone files
Go to /var/named/chroot/var/named directory, create files: localdomain.com.zone and 19.172.in-addr.arpa.
cd /var/named/chroot/var/named
touch localdomain.com.zone && chown named:named localdomain.com.zone && chmod 644 localdomain.com.zone
touch 19.172.in-addr.arpa && chown named:named 19.172.in-addr.arpa && chmod 644 19.172.in-addr.arpa

vi localdomain.com.zone

$TTL 1D
$ORIGIN localdomain.com.
@             IN      SOA     bigdatam.localdomain.com. foo.bar.tld. (
                                200612060                 ; serial
                                2H                        ; refresh slaves
                                5M                        ; retry
                                1W                        ; expire
                                1M                        ; Negative TTL
                                )

@                       IN      NS      bigdatam

bigdatam       IN      A       172.19.97.224
bigdata1       IN      A       172.19.101.66
bigdata2       IN      A       172.19.102.56

vi 172.19.zone

$TTL 1D
$ORIGIN 19.172.IN-ADDR.ARPA.

@       IN      SOA     bigdatam.localdomain.com. foo.bar.tld. (
200612060       ; serial
2H              ; refresh slaves
5M              ; retry
1W              ; expire
1M              ; Negative TTL
)

        IN      NS      bigdatam.localdomain.com.
224.97      IN      PTR     bigdatam.localdomain.com.
66.101       IN      PTR     bigdata1.localdomain.com.
56.102       IN      PTR     bigdata2.localdomain.com.

localhost.zone, named.local and named.ca are already in /var/named/chroot/var/named, they are created automatically, we just need refer them in /var/named/chroot/etc/named.conf.
Restart named server
service named restart 
chkconfig named on
Reload configuration and zones
rndc reload 
Toggle query logging
rndc querylog

Sometimes, we need disable SELinux and firewall
Disable SELinux
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
SELINUXTYPE=targeted
Disable firewall
/etc/init.d/iptables stop
chkconfig iptables off

Configure DNS Client
Do following steps in all 3 servers.
Prevent /etc/resolv.conf get overwritten
/etc/sysconfig/network-scripts/ifcfg-eth0 (replace eth0 with your network interface if different) and change PEERDNS=yes to PEERDNS=no
Setup DNS Name resolution 
vi /etc/resolv.conf
search localdomain.com
nameserver 172.19.97.224 # the private dns server ip address.
Restart network
/etc/init.d/network restart

Test DNS Setup
Run nslookup to start a session, and run the folloiwng command in all hosts.
# nslookup

> 127.0.0.1
Server:         172.19.97.224
Address:        172.19.97.224#53

1.0.0.127.in-addr.arpa  name = localhost.
> localhost
Server:         172.19.97.224
Address:        172.19.97.224#53

Non-authoritative answer:
Name:   localhost
Address: 127.0.0.1
> bigdatam
Server:         172.19.97.224
Address:        172.19.97.224#53

Name:   bigdatam.localdomain.com
Address: 172.19.97.224
> bigdata1
Server:         172.19.97.224
Address:        172.19.97.224#53

Name:   bigdata1.localdomain.com
Address: 172.19.101.66

dig bigdaam
host -v -t A `hostname

vi /etc/hosts
127.0.0.1       localhost.localdomain localhost
Synchronize System Clock Between Servers
ntpdate pool.ntp.org

Resources
How to set up a home DNS server
How to set up a home DNS server, part II