[大数据]spark版本bug总结

spark sql

1、spark sql 删除分区报错 mismatched input ‘<=’ expecting

• 记录时间
  2021年11月23日20:11:07
• spark版本 2.1.0
• 报错详细信息

  Traceback (most recent call last):
  File "/tmp/voldemort/0000003358711/resource.test.py", line 28, in <module>
  spark.sql(sql_drop_part)
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera4-1.cdh5.13.3.p0.818552/lib/spark2/python/lib/pyspark.zip/pyspark/sql/session.py", 	line 545, in sql
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera4-1.cdh5.13.3.p0.818552/lib/spark2/python/lib/py4j-0.10.7-		src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera4-1.cdh5.13.3.p0.818552/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 73, in deco
  pyspark.sql.utils.ParseException: u"\nmismatched input '<=' expecting {')', ','}(line 1, pos 85)\n\n== SQL ==\nalter table ods.test_table_name drop if exists partition(dt <= '20211102')\n-------------------------------------------------------------------------------------^^^\n"
  

  spark版本问题，删除分区，无法匹配 '>' 或者 ‘<’，详情见 url
• 目前处理方式

  # 使用 = 
  sql = """
  alter table ods.test_table_name drop if exists partition(dt = '20211102')
  """