[大数据]Hadoop【问题记录 03】【ipc.Client: Retrying connect to server:xxx/:8032+InvalidResourceRequestException】解决

起因

我使用的是 hadoop-3.1.3 版本，在进行基准测试时，执行以下代码；

[root@tcloud mapreduce]# hadoop jar ./hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -size 10kb -resFile home/hadoop/tmp/TestDFSIO.log

问题一【Retrying connect to server】

无法连接 ResourceManager：

2021-08-16 14:25:30,253 INFO client.RMProxy: Connecting to ResourceManager at tcloud/172.17.0.8:8032
2021-08-16 14:25:31,339 INFO ipc.Client: Retrying connect to server: tcloud/172.17.0.8:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

解决：

原因是未启动yarn，启动即可：

[root@tcloud mapreduce]# start-yarn.sh

问题二【InvalidResourceRequestException】

无效的资源请求，请求的资源 type=[memory-mb] < 0或大于允许的最大分配【这是使用基础配置云服务器的尴尬，由于服务器是1核心2G内存，所以配置memory-mb时配的低，造成请求资源大于允许的最大内存】：

2021-08-16 15:22:10,535 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
/tmp/hadoop-yarn/staging/root/.staging/job_1629098481992_0001

java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: 
Invalid resource request, requested resource type=[memory-mb] < 0 or greater than maximum 
allowed allocation. Requested resource=<memory:1536, vCores:1>, maximum allowed 
allocation=<memory:256, vCores:4>, please note that maximum allowed allocation is 
calculated by scheduler based on maximum resource of registered NodeManagers, which might 
be less than configured maximum allocation=<memory:256, vCores:4>

解决：

修改 ${HADOOP_HOME}/etc/hadoop/yarn-site.xml 文件：

# 停掉yarn
[root@tcloud hadoop]# stop-yarn.sh
# 修改配置文件
[root@tcloud hadoop]# vim ./yarn-site.xml
# 启动yarn
[root@tcloud hadoop]# start-yarn.sh

<!-- 调整内存大小即可 -->
 <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>256</value>
  </property>
  <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>1540</value>
  </property>
 <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1540</value>
  </property>

未解决的问题 但不影响执行

2021-08-16 15:10:55,499 INFO conf.Configuration: resource-types.xml not found
2021-08-16 15:10:55,500 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.