[大数据]ARM64环境利用 docker 编译 hadoop-2.7.7

浅言碎语

• x86_64 架构，直接从官网下载二进制文件就可以了
  • hadoop下载地址
  • maven下载地址
  • jdk下载地址
    • oracle 的 jdk 下载需要注册 oracle 的账号
• arm64 架构官方没有二进制文件，只能自己编译
  • 当然，也可以在 linux 服务上编译，利用 docker 其实是为了不’玷污’本地环境
    • 因为发行版是 麒麟V10 ，编译异常困难，还是要借助 docker 运行一个 centos
      • 为什么是 centos ？
        • 因为 hadoop-2.7.7 编译的时候要求 protoc 的版本是 2.5.0
        • 麒麟 V10 默认下载的版本已经是 3.x 了，而且没有 2.x 的历史版本
          • 尝试过去编译 protoc， 反正没成功
          • docker hub 上拉取的 centos 7 的镜像，yum 安装的就是 2.5.0 的 protoc
          • docker 镜像制作完成后，一定要先去容器里面执行 protoc --version，确认版本是不是2.5.0的，否则编译过程中会报错
            • 报错内容protoc version is 'libprotoc 3.9.0', expected version is '2.5.0'
      • 为什么用 oracle 的 jdk？
        • 因为 openjdk 在编译的时候，会出现 cannot find symbol 的报错，编译就进行不下去了
          • 用了 oracle 的 jdk1.8.0_321 就没有问题，百度云上传了一份
            • 链接：https://pan.baidu.com/s/1dY20MskH40KOscq0dXM61A
            • 提取码：vuuh
  • 编译过程还是比较依赖网络的，需要从 apache 的 maven 仓库里面获取 java 依赖
    • 当然，也可以提前下载好 maven 仓库，但是挺累的

准备环境

查看发行版

cat /etc/os-release

NAME="Kylin Linux Advanced Server"
VERSION="V10 (Tercel)"
ID="kylin"
VERSION_ID="V10"
PRETTY_NAME="Kylin Linux Advanced Server V10 (Tercel)"
ANSI_COLOR="0;31"

查看架构

uname -i

aarch64

目录结构

.
├── apache-maven-3.6.2.tar.gz
├── Dockerfile
├── hadoop-2.7.7-src.tar.gz
└── jdk1.8.0_321.tar.gz

编写 dockerfile

# 定义一个 centos 7 的初始镜像，方便自定义编译环境
FROM centos:7
# 定义变量
## 定义工作目录
ARG work_dir=/usr/local

# 定义容器的环境变量
ENV JAVA_HOME=${work_dir}/jdk1.8.0_321
## 我这里用的是 maven 3.6.2 版本，如果下载的版本和我的不一样，这里要修改
ENV MAVEN_HOME=${work_dir}/apache-maven-3.6.2
ENV PATH=${PATH}:${JAVA_HOME}/bin:${MAVEN_HOME}/bin

# 定义进入容器的默认目录
WORKDIR ${work_dir}
# 配置 yum 源为 阿里源
## 安装编译 hadoop 所需的工具，清理安装包和缓存
RUN curl -O /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/Centos-7.repo && \
    yum install -y gcc gcc-c++ make cmake protobuf-* automake libtool zlib-devel openssl-devel && \
    yum clean all
# 复制 tar 包到镜像内
ADD ./jdk1.8.0_321 ./jdk1.8.0_321
ADD ./apache-maven-3.6.2 ./apache-maven-3.6.2
ADD ./hadoop-2.7.7-src.tar.gz ./hadoop-2.7.7-src
# 整个脚本，让他睡十年
## docker 容器想要在后台常驻，需要有一个前台常驻进程
RUN echo '/usr/bin/sleep 315360000' > start.sh && \
    chmod +x start.sh
CMD ["/usr/bin/bash","start.sh"]

创建一个镜像

docker build -t hadoop:2.7.7 .

开始编译

先让容器在后台跑着

docker run -d --network host hadoop:2.7.7

进入容器

docker exec -it <容器id> bash

开始编译

cd hadoop-2.7.7-src
mvn package -e -X -Pdist,native -DskipTests -Dtar

整个编译时间长达 1小时22分钟（反正肯定一小时起步，至于时长，还是和机器性能以及网络有关）

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.7.7:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [14:57 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [11:03 min]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [03:02 min]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 54.759 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.407 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [02:16 min]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [03:00 min]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:35 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:06 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [01:01 min]
[INFO] Apache Hadoop Common ............................... SUCCESS [08:37 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.666 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:01 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.204 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:13 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.157 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [02:04 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.859 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.205 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.205 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 23.612 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [03:06 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.207 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  6.124 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 10.998 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.448 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  4.626 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 12.354 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  3.492 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  3.898 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.501 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.190 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.112 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.968 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.202 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.535 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.945 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.360 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 12.012 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [  9.888 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.641 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  6.284 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.903 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 17.598 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.778 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.951 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.957 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 14.252 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [01:42 min]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.917 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.753 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.136 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.143 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.645 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.321 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  6.265 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.007 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [03:45 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 24.236 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.014 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.844 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.275 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  7.215 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.199 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.403 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:22 h
[INFO] Finished at: 2022-04-17T04:51:48Z
[INFO] ------------------------------------------------------------------------

编译完成后的 tar 包文件在hadoop-dist/target/目录下

创作挑战赛

新人创作奖励来咯，坚持创作打卡瓜分现金大奖