[大数据]Hadoop集群搭建

一、安装部署3台联网的服务器

1. 下载VMware Workstation Pro? ?https://www.vmware.com/cn/products/workstation-pro/workstation-pro-evaluation.html

安装VMware Workstation Pro

2. 下载CentOS-7-x86_64-DVD-2009.iso? ?https://mirrors.aliyun.com/centos/7/isos/x86_64/

安装CentOS7? ?https://www.runoob.com/w3cnote/vmware-install-centos7.html，https://blog.csdn.net/tsundere_x/article/details/104263100

3. 网络配置NAT模式? ?https://blog.csdn.net/qq_17011423/article/details/82947417

4. 配置完成的三台服务器的IP地址分别为：192.168.126.100、192.168.126.101、192.168.126.102

(base) [root@master ~]# cd /etc/sysconfig/network-scripts/
(base) [root@master network-scripts]# ls
ifcfg-ens33  ...
(base) [root@master network-scripts]# vim ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO=static
NM_CONTROLLED=no
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="261d2611-1417-4370-8f5e-7657cfd0ba22"
DEVICE="ens33"
ONBOOT="yes"
HWADDR=00:0c:29:21:c1:32
IPADDR=192.168.126.100
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=192.168.126.2
DNS1=114.114.114.114
DNS2=8.8.8.8

(base) [root@master ~]# vi /etc/resolv.conf

# Generated by NetworkManager
nameserver 114.114.114.114
nameserver 8.8.8.8

二、关闭防火墙

(base) [root@master ~]# systemctl stop firewalld.service
(base) [root@master ~]# firewall-cmd --state
not running

三、修改主机名

修改主机名为master、slave1和slave2。

(base) [root@localhost ~]# cat /etc/hostname
localhost.localdomain

(base) [root@localhost ~]# hostnamectl set-hostname master
(base) [root@localhost ~]# cat /etc/hostname
master

增加3台服务器IP和主机名的映射关系。

(base) [root@localhost ~]# vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.126.100 master
192.168.126.101 slave1
192.168.126.102 slave2

重启服务器生效。

(base) [root@localhost ~]# reboot

四、安装JDK

下载JDK? ?https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html

创建/usr/local/java目录，将jdk-8uxxx-linux-x64.tar.gz拷贝至该目录

[root@localhost ~]# cd /usr/local/java
[root@localhost java]# tar -zxvf jdk-8u271-linux-x64.tar.gz

[root@localhost java]# vim /etc/profile
export JAVA_HOME=/usr/local/java/jdk1.8.0_271
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin

[root@localhost java]# source /etc/profile

[root@localhost java]# java -version
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

五、安装配置Hadoop

下载hadoop-2.10.1.tar.gz? ?https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/

创建/usr/local/hadoop目录，将jhadoop-x.x.x.tar.gz拷贝至该目录

(base) [root@master ~]# cd /usr/local/hadoop
(base) [root@master hadoop]# tar -zxvf hadoop-2.10.1.tar.gz

1、配置环境变量

(base) [root@master hadoop]# vim /etc/profile
export HADOOP=/usr/local/hadoop/hadoop-2.10.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP/bin:$HADOOP/sbin

(base) [root@master hadoop]# source /etc/profile
(base) [root@master hadoop]# hadoop version
Hadoop 2.10.1

2、修改hadoop-2.10.1/etc/hadoop/目录下的配置文件

(base) [root@master hadoop]# cd /usr/local/hadoop/hadoop-2.10.1/etc/hadoop/
(base) [root@master hadoop]# ls

1) 修改slaves

配置从节点。

(base) [root@master hadoop]# vim slaves

slave1
slave2

2) 修改core-site.xml

使用fs.default.name还是fs.defaultFS，要判断是否开启了namenode的highavaliable，如果开启了NN HA，用fs.defaultFS，在单一namenode的情况下，用fs.default.name。如果在单一namenode节点的情况使用?fs.defaultFS，系统将报错：

?ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.

(base) [root@master hadoop]# vim core-site.xml

<configuration>
        <!-- 指定临时文件目录 -->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/hadoop/hadoop-2.10.1/tmp/</value>
        </property>
        <!-- 配置HDFS的NameNode所在节点服务器 -->
        <property>
                <name>fs.default.name</name>
                <value>hdfs://192.168.126.100:9000</value>
        </property>
</configuration>

(base) [root@master hadoop]# cd /usr/local/hadoop/hadoop-2.10.1/
(base) [root@master hadoop-2.10.1]# mkdir tmp

3) 修改hdfs-site.xml

(base) [root@master hadoop]# vim hdfs-site.xml 

<configuration>
        <!-- 指定HDFS副本的数量 -->
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
        <!-- 指定Hadoop辅助名称节点主机配置 -->
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>192.168.126.100:50090</value>
        </property>
        <!-- namenode节点数据保存的位置 -->
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <!-- datanode节点数据保存的位置 -->
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
</configuration>                         

4) 修改mapred-site.xml

(base) [root@master hadoop]# cp mapred-site.xml.template mapred-site.xml
(base) [root@master hadoop]# vim mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>192.168.126.100:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>192.168.126.100:19888</value>
        </property>
</configuration>

5) 修改yarn-site.xml

(base) [root@master hadoop]# vim yarn-site.xml 

<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>master</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

6) 修改hadoop-env.sh

将JAVA_HOME修改为本机Java路径。

(base) [root@master hadoop]# vim hadoop-env.sh

# The java implementation to use.
export JAVA_HOME=/usr/local/java/jdk1.8.0_271/

3、复制Hadoop至slave1和slave2

(base) [root@master hadoop]# scp -rp /usr/local/hadoop slave1:/usr/local/hadoop
(base) [root@master hadoop]# scp -rp /usr/local/hadoop slave2:/usr/local/hadoop

六、SSH免密登录

生成公钥：公钥文件id_rsa.pub、私钥文件id_rsa

[root@localhost java]# ssh-keygen
回车
回车
回车
[root@localhost java]# cd ~/.ssh/
[root@localhost .ssh]# ls
id_rsa  id_rsa.pub  known_hosts
[root@localhost .ssh]# cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjytOhnCY/Oahczd/OaalebQA4...5 root@master

三台服务器免密互相ssh访问

(base) [root@master .ssh]# cp id_rsa.pub ~/.ssh/authorized_keys
(base) [root@master .ssh]# vim authorized_keys 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQ.../WverOImLhJ root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCft...qOj2ABkh root@slave1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEE...RVsppaFn root@slave2

(base) [root@master .ssh]# scp -rp authorized_keys slave1:~/.ssh/
(base) [root@master .ssh]# scp -rp authorized_keys slave2:~/.ssh/

验证。

(base) [root@master .ssh]# ssh slave1
Last login: Sun Aug  8 00:22:10 2021 from 192.168.126.1
[root@slave1 ~]# ssh master
Last login: Sun Aug  8 00:22:07 2021 from 192.168.126.1
(base) [root@master ~]# ssh slave2
Last login: Sun Aug  8 00:22:12 2021 from 192.168.126.1
[root@slave2 ~]# exit
logout
Connection to slave2 closed.
(base) [root@master ~]# 

七、初始化、启动Hadoop

启动的命令在$HADOOP/bin:$HADOOP/sbin下面，已配置在环境变量中。

(base) [root@master ~]# hdfs namenode -format

(base) [root@master ~]# start-dfs.sh
(base) [root@master ~]# start-yarn.sh
(base) [root@master ~]# jps
14610 SecondaryNameNode
14852 Jps
14776 ResourceManager
14395 NameNode

[root@slave1 ~]# jps
49265 Jps
48970 DataNode
49101 NodeManager

(base) [root@master ~]# stop-yarn.sh
(base) [root@master ~]# stop-dfs.sh