[大数据]Hadoop 集群搭建，云服务器

先安装JDK与Hadoop

cd $HADOOP_HOME/etc/hadoop

=============core-site.xml

<configuration>
	<property> 
		<name>fs.default.name</name> 
		<value>hdfs://master:9000</value> 
	</property> 
	<property> 
		<name>hadoop.tmp.dir</name> 
		<value>file:/opt/module/hadoop-3.3.1/temp</value> 
	</property> 
	<property> 
		<name>io.file.buffer.size</name> 
		<value>131072</value> 
	</property>
</configuration>

===========hdfs-site.xml

<configuration>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/opt/module/hadoop-3.3.1/data/dfs/data</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/opt/module/hadoop-3.3.1/data/dfs/data</value>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>2</value>
	</property>
	<property>
		<name>dfs.permissions</name>
		<value>false</value> <description>need not permissions</description>
	</property>
	<property>
		<name>dfs.namenode.http-address</name>
		<value>master:50070</value>
	</property>
	<property>
        		<name>dfs.namenode.secondary.http-address</name>
        		<value>slave:9868</value>
    	</property>
</configuration>

==============yarn-site.xml

<configuration>
	<!-- Site specific YARN configuration properties --> 
	<property> 
		<name>yarn.resourcemanager.hostname</name> 
		<value>slave</value> 
	</property> 
	<property> 
		<description>The address of the applications manager interface in the RM.</description> 
		<name>yarn.resourcemanager.address</name> 
		<value>slave:8032</value> 
	</property> 
	<property> 
		<description>The address of the scheduler interface.</description> 
		<name>yarn.resourcemanager.scheduler.address</name> 
		<value>slave:8030</value> 
	</property> 
	<property> 
		<description>The http address of the RM web application.</description> 
		<name>yarn.resourcemanager.webapp.address</name> 
		<value>slave:18088</value> 
	</property> 
	<property> 
		<description>The https adddress of the RM web application.</description> 
		<name>yarn.resourcemanager.webapp.https.address</name> 
		<value>slave:18090</value> 
	</property> 
	<property> 
		<name>yarn.resourcemanager.resource-tracker.address</name> 
		<value>slave:8031</value> 
	</property> 
	<property> 
		<description>The address of the RM admin interface.</description> 
		<name>yarn.resourcemanager.admin.address</name> 
		<value>slave:8033</value> 
	</property> 
	<property> 
		<name>yarn.nodemanager.aux-services</name> 
		<value>mapreduce_shuffle</value> 
	</property>
</configuration>

=========mapred-site.xml

<configuration> 
	<property> 
		<name>mapreduce.framework.name</name> 
		<value>yarn</value> 
	</property> 
	<property> 
		<name>mapred.job.tracker</name> 
		<value>master:9001</value> 
	</property>
</configuration>

==========只需要添加worker节点的信息即可
vim /opt/module/hadoop-3.1.3/etc/hadoop/workers

情况：master : master_ip
worker : slave_ip

slave_ip（这种情况只需要添加这个IP到两个节点的worker配置文件中）

=====执行生效
source /etc/profile

======== 配置文件地址
/opt/module/hadoop-3.3.1/etc/hadoop

====添加主机列表，每个节点只需要添加其他节点的信息就好，不需要添加本身的ip。
vi /etc/hosts
master_ip master
slave_ip slave

========######第一次启动NameNode需要先进行格式化
cd /opt/module/hadoop-3.3.1/
/opt/module/hadoop-3.3.1/bin/hdfs namenode -format

=====配置在/etc/profile环境变量中
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOO{P}_{H}OME/lib/nativeexportHADOO{P}_{O}PTS="?Djava.library.path=$HADOOP_HOME/lib/native"

=====将节点间的公钥进行分享，即可使用主机名相互访问。
SSH免密登录
SSH免密登录是为了各个服务器之间访问不在需要密码。两台台服务器分别执行一遍如下命令。执行完该命令会在root/.ssh下生成密钥。
ssh-keygen -t rsa

#在slave id_rsa.pub发送到主机上，并重新命令
scp id_rsa.pub root@master:~/.ssh/id_rsa.pub.slave

在主机root/.ssh下把id_rsa.pub、id_rsa.pub.slave追加到authorized_keys中。
cat id_rsa.pub >> authorized_keys
cat id_rsa.pub.slave >> authorized_keys

然后把authorized_keys传回到slave1
scp authorized_keys root@slave:~/.ssh

最后修改文件权限。

chmod 755 ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

====添加信息到$HADOOP_HOME/etc/hadoop/hadoop-env.sh
export HADOOP_SHELL_EXECNAME=root
export HDFS_NAMENODE_USER=root

====启动
/opt/module/hadoop-3.3.1/sbin/start-dfs.sh
/opt/module/hadoop-3.3.1/sbin/start-yarn.sh
/opt/module/hadoop-3.3.1/sbin/start-all.sh

====关闭
/opt/module/hadoop-3.3.1/sbin/stop-dfs.sh
/opt/module/hadoop-3.3.1/sbin/stop-yarn.sh
/opt/module/hadoop-3.3.1/sbin/stop-all.sh

=====NameNode节点的访问地址

http://master:50070/
http://slave:8088/