一 hive1.0本地模式安装步骤
1)安装hadoop2.0伪分布式,启动hdfs和yarn
2)安装mysql,用户名root,密码SoftCup@2021
创建hive数据库create database hive
3)修改profile文件
# jdk export JAVA_HOME=/usr/local/java/jdk1.8 export PATH=$PATH:$JAVA_HOME/bin
# hadoop export HADOOP_HOME=/usr/local/hadoop-2.6.0-cdh5.7.0/ export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
# hive export HIVE_HOME=/usr/local/apache-hive-1.2.1-bin export PATH=$PATH:$HIVE_HOME/bin
4)解压缩hive安装包
5)修改配置文件hive-site.xml
<configuration> <property> ? ? <name>javax.jdo.option.ConnectionURL</name> ? ? <value>jdbc:mysql://hive:3306/hive?characterEncoding=UTF-8</value> </property> <property> ? ? <name>javax.jdo.option.ConnectionDriverName</name> ? ? <value>com.mysql.jdbc.Driver</value> </property>? <property> ? ? <name>javax.jdo.option.ConnectionUserName</name> ? ? <value>root</value> </property> <property> ? ? <name>javax.jdo.option.ConnectionPassword</name> ? ? <value>SoftCup@2021</value> </property> </configuration>
6)拷贝mysql驱动包mysql-connector-java-5.1.28.jar到hive的lib子目录
7)执行hive命令启动hive终端
进入hive终端后执行show databases;
二 hive3.0本地模式安装步骤
1)安装hadoop3.0伪分布式,启动hdfs和yarn
注意:hadoop3.0将hadoop2.0的slaves文件改名为workers
2)安装mysql,用户名root,密码SoftCup@2021
创建hive数据库create database hive
3)修改profile文件
# jdk export JAVA_HOME=/usr/local/java/jdk1.8 export PATH=$PATH:$JAVA_HOME/bin
# hadoop export HADOOP_HOME=/usr/local/hadoop-3.1.4/ export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HDFS_NAMENODE_USER=root export HDFS_DATANODE_USER=root export HDFS_SECONDARYNAMENODE_USER=root export YARN_RESOURCEMANAGER_USER=root export YARN_NODEMANAGER_USER=root
# hive export HIVE_HOME=/usr/local/apache-hive-3.1.2-bin/ export PATH=$PATH:$HIVE_HOME/bin
4)解压缩hive安装包
5)创建配置文件hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> ? ? <name>javax.jdo.option.ConnectionURL</name> ? ? <value>jdbc:mysql://hive:3306/hive?characterEncoding=UTF-8</value> </property> <property> ? ? <name>javax.jdo.option.ConnectionDriverName</name> ? ? <value>com.mysql.jdbc.Driver</value> </property>? <property> ? ? <name>javax.jdo.option.ConnectionUserName</name> ? ? <value>root</value> </property> <property> ? ? <name>javax.jdo.option.ConnectionPassword</name> ? ? <value>SoftCup@2021</value> </property> </configuration>
6)创建hive-env.sh
cp hive-env.sh.template hive-env.sh vi?hive-env.sh,末尾增加 export HADOOP_HOME=/usr/local/hadoop-3.1.4 export HIVE_CONF_DIR=/usr/local/apache-hive-3.1.2-bin/conf
7)修改yarn-site.xml,增加两个新的配置参数
<property> ? ? ? ? <name>yarn.application.classpath</name> ? ? ? ? <value>/usr/local/hadoop-3.1.4/etc/hadoop:/usr/local/hadoop-3.1.4//share/hadoop/common/lib/*:/usr/local/hadoop-3.1.4//share/hadoop/common/*:/usr/local/hadoop-3.1.4//share/hadoop/hdfs:/usr/local/hadoop-3.1.4//share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.4//share/hadoop/hdfs/*:/usr/local/hadoop-3.1.4//share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.4//share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.4//share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.4//share/hadoop/yarn/*</value> </property>
<property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>3.0</value> </property>
说明:yarn.application.classpath参数值来自hadoop classpaths命令的输出结果
不增加这两个新的配置参数,会出现select count(*)的mapreduce任务执行失败
8)拷贝mysql驱动包mysql-connector-java-5.1.28.jar到hive的lib子目录
9)初始化metadata
执行命令schematool -dbType mysql -initSchema
10)执行hive命令启动hive终端
进入hive终端后执行show databases;
三 hive终端命令测试
hive> show databases;
hive> create table student(id bigint, name string, score double, age int) row format delimited fields terminated by ','; hive> show tables;
hive> desc student; OK id ? ? ? ? ? ? ? ? ??? ?bigint ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? name ? ? ? ? ? ? ? ??? ?string ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? score ? ? ? ? ? ? ? ?? ?double ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? age ? ? ? ? ? ? ? ? ?? ?int ??
创建一个文本文件student.txt 内容如下:
43242341,zhangsan,99.5,19 76277567,lisi,98.4,20 87686553,wangwu,97.4,21 78568876,zhaoliu,99.9,22
hive> load data local inpath '/root/student.txt' into table student;
hive> select * from student;? ?? OK 43242341?? ?zhangsan?? ?99.5?? ?19 76277567?? ?lisi?? ?98.4?? ?20 87686553?? ?wangwu?? ?97.4?? ?21 78568876?? ?zhaoliu?? ?99.9?? ?22 Time taken: 0.577 seconds, Fetched: 4 row(s)
说明:select * 不执行mapreduce任务
hive> select count(*) from student;
2021-08-28 16:52:20,012 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.72 sec 2021-08-28 16:52:33,455 Stage-1 map = 100%, ?reduce = 100%, Cumulative CPU 5.96 sec MapReduce Total cumulative CPU time: 5 seconds 960 msec Ended Job = job_1630140616209_0001 MapReduce Jobs Launched:? Stage-Stage-1: Map: 1 ?Reduce: 1 ? Cumulative CPU: 5.96 sec ? HDFS Read: 12611 HDFS Write: 101 SUCCESS Total MapReduce CPU Time Spent: 5 seconds 960 msec OK 4
说明:select count(*) 执行mapreduce任务
|