[大数据]hadoop+hbase+zookeeper

Zookeeper+hadoop（ha）+hbase集群部署
部署zookeeper：
环境：

172.17.10.10 zk01 ZOOKEEPER
172.17.10.11 zk02 ZOOKEEPER
172.17.10.12 zk03 ZOOKEEPER
172.17.10.13 hdp01 HADOOP(NAMENODE)+HBASE(HMASTER)
172.17.10.14 hdp02 HADOOP(NAMENODE)+HBASE(HMASTER)
172.17.10.15 hdp03 HADOOP(NAMENODE)+HBASE
172.17.10.16 hdp04 HADOOP(NAMENODE)+HBASE
172.17.10.17 hdp05 HADOOP(NAMENODE)+HBASE

基础环境：
配置hosts文件：（将主节点的hosts分别拷贝到其他几个子节点）
vim /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.17.10.10 zk01
172.17.10.11 zk02
172.17.10.12 zk03
172.17.10.13 hdp01
172.17.10.14 hdp02
172.17.10.15 hdp03
172.17.10.16 hdp04
172.17.10.17 hdp05

配置免密登录：(七台依次执行）

ssh-keygen -t rsa

在主节点上执行：

cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

将其他子节点的公钥拷贝到主节点并添加进authorized_keys:

scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_hadoop02.pub
scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_hadoop03.pub
scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_hadoop04.pub
scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_hadoop05.pub
scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_master01.pub
scp ~/.ssh/id_rsa.pub hadoop01:~/.ssh/id_rsa_master02.pub

然后在主节点上，将拷贝过来的两个公钥合并到authorized_keys文件中去主节点上执行：

cat ~/.ssh/id_rsa_hadoop02.pub >> authorized_keys
cat ~/.ssh/id_rsa_hadoop03.pub >> authorized_keys
cat ~/.ssh/id_rsa_hadoop04.pub >> authorized_keys
cat ~/.ssh/id_rsa_hadoop05.pub >> authorized_keys
cat ~/.ssh/id_rsa_master01.pub >> authorized_keys
cat ~/.ssh/id_rsa_master02.pub >> authorized_keys

将主节点的authorized_keys文件分别替换子节点的authorized_keys文件,主节点上用scp命令将authorized_keys文件拷贝到子节点的相应位置:

scp authorized_keys hadoop02:~/.ssh/
scp authorized_keys hadoop03:~/.ssh/
scp authorized_keys hadoop04:~/.ssh/
scp authorized_keys hadoop05:~/.ssh/
scp authorized_keys master01:~/.ssh/
scp authorized_keys master02:~/.ssh/

最后测试是否配置成功,在hadoop01上分别执行:

ssh hadoop02
ssh hadoop03
ssh hadoop04
ssh hadoop05
ssh master01
ssh master02

能正确跳转到两台子节点的操作界面即可，同样在每个子节点通过相同的方式登录主节点和其他子节点也能无密码正常登录就表示配置成功。

保持7台时间同步

安装jdk（七台机器都要安装）：

rpm -ivh jdk-8u161-linux-x64.rpm

配置环境变量：

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_161/
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

安装zookeeper：

tar -zxvf zookeeper-3.4.14.tar.gz -C /home/userv/
cd /home/userv/zookeeper-3.4.14/conf
cp zoo-sample.cfg zoo.cfg
vim zoo.cfg
##########################################
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk01:2888:3888
server.2=zk02:2888:3888
server.3=zk03:2888:3888

进入设置的数据缓存目录，修改myid，如果当前服务器是server.1 myid内容就写1，server.2就写2 ，依次类推：

vim /home/appmanager/data/zookeeper/myid

1

手动创建缓存目录：

mkdir /home/appmanager/data/zookeeper

将配置好的zookeeper目录拷贝到其他两个节点：

scp -r /home/appmanager/rcs/zookeeper-3.4.14 zk02: /home/appmanager/rcs
scp -r /home/appmanager/rcs/zookeeper-3.4.14 zk03: /home/appmanager/rcs

修改zk02 zk03的myid：

vim /home/appmanager/data/zookeeper/myid
2

启动zookeeper集群：

cd  /home/appamanger/rcs/zookeeper-3.4.14/bin
./zkSERVER.sh  start
zookeeper 设置ACL：(有需要的再配置，没有需要跳过此步）
./zkCli.sh -server 
addauth digest user1:password1

安装Hadoop：

tar -zxvf hadoop-2.7.7.tar.gz -C /home/userv

配置Hadoop环境变量:

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_161/ #(注:这是JDK安装的位置)
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/home/appmanager/rcs/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

生效环境变量配置:
#：source /etc/profile

配置hadoop配置文件:
需要配置的文件的位置为/home/userv/hadoop-2.7.7/etc/hadoop，需要修改的有以下几个文件：
hadoop-env.sh
yarn-env.sh
core-site.xml
hdfs-site.xml
mapred-site.xml
yarn-site.xml
works
其中hadoop-env.sh和yarn-env.sh里面都要添加jdk的环境变量
hadoop-env.sh中添加如下代码：export JAVA_HOME=/usr/java/jdk1.8.0_161/ 到文件末尾位置：

cd /home/userv/hadoop-2.7.7/etc/hadoop

export JAVA_HOME=/usr/java/jdk1.8.0_161/
export HADOOP_HEAPSIZE=6144
export HADOOP_NAMENODE_INIT_HEAPSIZE=2048
export HADOOP_PID_DIR=/home/appmanager/data/hadoop/pids
export HADOOP_SECURE_DN_PID_DIR=/home/appmanager/data/hadoop/pids

yarn-env.sh中添加如下代码：export JAVA_HOME=/usr/java/jdk1.8.0_161/ 到文件末尾位置：

export JAVA_HOME=/usr/java/jdk1.8.0_161/
export YARN_PID_DIR=/home/appmanager/data/hadoop/pids

core-site.xml中添加如下代码：

<configuration>
 <property>
     <name>fs.defaultFS</name>
     <value>hdfs://hadoopha</value>
 </property>
 <property>
     <name>hadoop.tmp.dir</name>
     <value>file:/home/appmanager/data/hadoop/tempdata</value>
 </property>
 <property>
    <name>ha.zookeeper.quorum</name>
    <value>zk01:2181,zk02:2181,zk03:2181</value>
 </property>
 <property>
 
   <name>ha.zookeeper.auth</name>
   <value>@/home/appmanager/data/core/zk-auth.txt</value>
</property>
<property>
   <name>ha.zookeeper.acl</name>
   <value>@/home/appmanager/data/core/zk-acl.txt</value>
</property>
<property>
     <name>ha.zookeeper.parent-znode</name>
     <value>/hadoop-ha</value>
</property>
 <property>
    <name>ha.zookeeper.session-timeout.ms</name>
    <value>5000</value>
 </property>
       <property>
                <name>hadoop.http.filter.initializers</name>
                <value>org.apache.hadoop.security.AuthenticationFilterInitializer</value>
        </property>
        <property>
                <name>hadoop.http.authentication.type</name>
                <value>simple</value>
        </property>
        <property>
                <name>hadoop.http.authentication.signature.secret.file</name>
                <value>/home/appmanager/data/hadoop/hdfs/hadoop-http-auth-signature-secret</value>
        </property>
<property>  
  <name>hadoop.security.authorization</name>  
  <value>true</value>  
</property>
</configuration>

hdfs-site.xml中添加如下代码：

<configuration>
 <property>
     <name>dfs.namenode.name.dir</name>
     <value>file:/home/appmanager/data/hadoop/name</value>
 </property>
 <property>
     <name>dfs.datanode.data.dir</name>
     <value>file:/home/appmanager/data/hadoop/data</value>
 </property>
 <property>
     <name>dfs.replication</name>
     <value>2</value>
 </property>
 <!--HA配置 -->
 <property>
     <name>dfs.nameservices</name>
     <value>hadoopha</value>
 </property>
 <property>
     <name>dfs.ha.namenodes.hadoopha</name>
     <value>nn1,nn2</value>
 </property>
 <!--namenode1 RPC端口 -->
 <property>
     <name>dfs.namenode.rpc-address.hadoopha.nn1</name>
     <value>hdp01:9000</value>
 </property>
 <!--namenode1 HTTP端口 -->
 <property>
     <name>dfs.namenode.http-address.hadoopha.nn1</name>
     <value>hdp01:50070</value>
 </property>
 <!--namenode2 RPC端口 -->
 <property>
     <name>dfs.namenode.rpc-address.hadoopha.nn2</name>
     <value>hdp02:9000</value>
 </property>
  <!--namenode2 HTTP端口 -->
 <property>
     <name>dfs.namenode.http-address.hadoopha.nn2</name>
     <value>hdp02:50070</value>
 </property>
  <!--HA故障切换 -->
 <property>
     <name>dfs.ha.automatic-failover.enabled</name>
     <value>true</value>
 </property>
 <!-- journalnode 配置 -->
 <property>
     <name>dfs.namenode.shared.edits.dir</name>
     <value>qjournal://hdp01:8485;hdp02:8485;hdp03:8485;hdp04:8485;hdp05:8485/hadoopha</value>
 </property>
 <property>
     <name>dfs.client.failover.proxy.provider.hadoopha</name>
     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
 </property>
 <!--发生failover时，Standby的节点要执行一系列方法把原来那个Active节点中不健康的NameNode服务给杀掉，
             这个叫做fence过程。sshfence会通过ssh远程调用fuser命令去找到Active节点的NameNode服务并杀死它-->
 <property>
     <name>dfs.ha.fencing.methods</name>
     <value>shell(/bin/true)</value>
  </property>
   <!--SSH私钥 -->
  <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/home/appmanager/.ssh/id_rsa</value>
  </property>
 <!--SSH超时时间 -->
  <property>
      <name>dfs.ha.fencing.ssh.connect-timeout</name>
      <value>30000</value>
  </property>
  <!--Journal Node文件存储地址 -->
  <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/home/appmanager/data/hadoop/journal</value>
  </property>
</configuration>

mapred-site.xml中添加如下代码:

（注意要将mapred-site.xml.template重命名为 .xml的文件 Mv mapred-site.xml.template mapred-site.xml）
<configuration>
<property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>hdp01:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>hdp01:19888</value>
        </property>
</configuration>

yarn-site.xml中添加如下代码：

<configuration>
 
<!-- Site specific YARN configuration properties -->
<property>
         <name>yarn.resourcemanager.ha.enabled</name>
         <value>true</value>
    </property>
    <!-- 指定RM的cluster id -->
    <property>
         <name>yarn.resourcemanager.cluster-id</name>
         <value>yrc</value>
    </property>
    <!-- 指定RM的名字 -->
    <property>
         <name>yarn.resourcemanager.ha.rm-ids</name>
         <value>rm1,rm2</value>
    </property>
    <!-- 分别指定RM的地址 -->
    <property>
         <name>yarn.resourcemanager.hostname.rm1</name>
         <value>hdp01</value>
    </property>
    <property>
         <name>yarn.resourcemanager.hostname.rm2</name>
         <value>hdp02</value>
    </property>
    <!-- 指定zk集群地址 -->
    <property>
         <name>yarn.resourcemanager.zk-address</name>
         <value>zk01:2181,zk02:2181,zk03:2181</value>
    </property>
    <property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
    </property>
</configuration>

workers中修改成其他子节点hosts：

hdp03
hdp04
hdp05

拷贝hadoop安装文件到子节点,主节点上执行：

scp -r /home/appmanager/rcs/hadoop-3.1.3 hdp02:/home/appmanager/rcs/
scp -r /home/appmanager/rcs/hadoop-3.1.3 hdp03:/home/appmanager/rcs/
scp -r /home/appmanager/rcs/hadoop-3.1.3 hdp04:/home/appmanager/rcs/
scp -r /home/appmanager/rcs/hadoop-3.1.3 hdp05:/home/appmanager/rcs/

数据目录需要手动创建：

mkdir -p  /home/appmanager/data/hadoop/pids /home/appmanager/data/core

配置hadoopACL文件：

vim /home/yanfaapp/rcs/core/zk-auth.txt

digest:hbase:sda&fas 

在zookeeper服务器执行：

./zkCli.sh

addauth digest hbase:sda&fas
java -cp /home/yanfaapp/app/zookeeper-3.4.14/lib/*:/home/yanfaapp/app/zookeeper-3.4.14/zookeeper-3.4.14.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hbase:"sda&fas"

在hadoop节点执行：

vim /home/yanfaapp/rcs/core/zk-acl.txt

digest:hbase:UN0ykx/giGNLqVjeKUVFNY+hklo=:rwcda

拷贝到其他hadoop节点

启动journalnode：手动启动所有journalNode节点的journalNode功能

hdfs --daemon start journalnode

在其中一台NameNode格式化zkfc：

  hdfs zkfc -formatZK

格式化主节点namenode格式化主节点namenode，并启动：

hdfs namenode -format //格式化
hdfs --daemon start namenode //打开NameNode节点

副节点同步主节点格式化：

hdfs namenode -bootstrapStandby

启动集群：

stop-all.sh
start-all.sh

搭建Hbase
解压：

tar  -zxvf   hbase-2.1.0-bin.tar.gz  -C rcs/

配置hbase-site.xml文件：

<configuration>
 <property>
                <name>hbase.rootdir</name>
                <value>hdfs://hadoopha/hbase</value>
            </property>
             <!-- 指定HMaster主机 -->
  <property>
    <name>hbase.master</name>
    <value>hdfs://hdp01:60000</value>
  </property>
 
  <!-- 启用分布式模式 -->
            <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
            </property>
 <!-- 指定独立Zookeeper安装路径 -->
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/appmanager/rcs/zookeeper-3.4.14</value>
  </property>
 
  <!-- 指定ZooKeeper集群端口 -->
             <property>
                <name>hbase.zookeeper.property.clientPort</name>
                <value>2181</value>
           </property>
            <property>
                <name>hbase.zookeeper.quorum</name>
                <value>zk01,zk02,zk03</value>
            </property>
   <property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
  </property>
  
<property>
     <name>hbase.tmp.dir</name>
     <value>/home/appmanager/data/hbase/hbase-tmp</value>
</property>
<property>
     <name>zookeeper.session.timeout</name>
     <value>1200000</value>
</property>
<property>
     <name>hbase.client.write.buffer</name>
     <value>8388608</value>
</property>
<property>
     <name>hbase.regionserver.maxlogs</name>
     <value>128</value>
</property>
<property>
     <name>hbase.regionserver.hlog.blocksize</name>
     <value>536870912</value>
</property>
<property>
     <name>hbase.hstore.compaction.min</name>
     <value>10</value>
</property>
<property>
     <name>hbase.hstore.compaction.max</name>
     <value>30</value>
</property>
<property>
     <name>hbase.hstore.blockingStoreFiles</name>
     <value>100</value>
</property>
<property>
     <name>hbase.hregion.majorcompaction</name>
     <value>0</value>
</property>
<property>
     <name>hbase.regionserver.thread.compaction.large</name>
     <value>5</value>
</property>
<property>
     <name>hbase.regionserver.thread.compaction.small</name>
     <value>5</value>
</property>
<property>
     <name>hbase.regionserver.thread.compaction.throttle</name>
     <value>10737418240</value>
</property>
<property>
     <name>hbase.hstore.compaction.max.size</name>
     <value>21474836480</value>
</property>
<property>
     <name>hbase.rpc.timeout</name>
     <value>300000</value>
</property>
<property>
     <name>hbase.regionserver.regionSplitLimit</name>
     <value>150</value>
</property>
</configuration>

配置hbase-env.sh文件：

export JAVA_HOME=/usr/java/jdk1.8.0_161/
export HBASE_LOG_DIR=/home/appmanager/data/hbase/logs
 
export HBASE_OPTS="-verbose:gc -XX:+PrintGCDetails -Xloggc:${HBASE_LOG_DIR}/hbase-gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime \
-server -Xmx32768m -Xms32768m -Xmn3072m -Xss256k  -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -XX:MaxTenuringThreshold=15 \
-XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:CMSFullGCsBeforeCompaction=5 -XX:+UseCMSCompactAtFullCollection \
-XX:+CMSClassUnloadingEnabled  -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSMaxAbortablePrecleanTime=5000     \
"
export HBASE_MANAGES_ZK=false

配置regionservers文件：

hdp03
hdp04
hdp05

手动创建backup-masters文件：

hdp01
hdp02

将Hadoop文件中的core-site.xml和hdfs-site.xml软连接到hbase的conf下：

ln -s /home/appmanager/rcs/hadoop-3.1.3/etc/hadoop/hdfs-site.xml /home/appmanager/rcs/hbase-2.1.0/conf/
ln -s /home/appmanager/rcs/hadoop-3.1.3/etc/hadoop/core-site.xml /home/appmanager/rcs/hbase-2.1.0/conf/

Hbase启动时 lg4j会和 Hadoop中的lg4j冲突，只更改Hmaster节点。改jar文件格式就行

cd  /home/appmanager/rcs/hbase-2.1.0/lib/client-facing-thirdparty 
mv slf4j-log4j12-1.7.25.jar slf4j-log4j12-1.7.25.jar.bak

配置HBASE环境变量/etc/profile：

export HBASE_HOME=/home/appmanager/rcs/hbase-2.1.0
export PATH=${HBASE_HOME}/bin:$PATH
cp /home/yanfaapp/app/hbase-2.1.0/lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar  /home/yanfaapp/app/hbase-2.1.0/lib/

启动hbase集群：

./start-hbase.sh

最后jps查看一下集群每台运行状况

jps

3920 JournalNode
3574 NameNode
4488 ResourceManager
66249 Jps
4121 DFSZKFailoverController
56745 Main
53626 QuorumPeerMain
13707 HMaster