hive 为什么气流会(直线)受阻?

uurv41yg  于 2022-11-23  发布在  Hive
关注(0)|答案(1)|浏览(101)

不会出现如下所示的错误,但操作会在此状态下停止。
所以,如果我直接访问beeline并输入相同的sql语句并执行它,它就能工作。
有什么问题吗?

Reading local file: /root/airflow/logs/Batch_Migration_Data/create_hive_table/2022-04-18T03:12:24.861251+00:00/1.log
[2022-04-18, 21:13:16 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: Batch_Migration_Data.create_hive_table manual__2022-04-18T03:12:24.861251+00:00 [queued]>
[2022-04-18, 21:13:16 KST] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: Batch_Migration_Data.create_hive_table manual__2022-04-18T03:12:24.861251+00:00 [queued]>
[2022-04-18, 21:13:16 KST] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2022-04-18, 21:13:16 KST] {taskinstance.py:1244} INFO - Starting attempt 1 of 1
[2022-04-18, 21:13:16 KST] {taskinstance.py:1245} INFO - 
--------------------------------------------------------------------------------
[2022-04-18, 21:13:16 KST] {taskinstance.py:1264} INFO - Executing <Task(HiveOperator): create_hive_table> on 2022-04-18 03:12:24.861251+00:00
[2022-04-18, 21:13:16 KST] {standard_task_runner.py:52} INFO - Started process 6630 to run task
[2022-04-18, 21:13:16 KST] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'Batch_Migration_Data', 'create_hive_table', 'manual__2022-04-18T03:12:24.861251+00:00', '--job-id', '21', '--raw', '--subdir', 'DAGS_FOLDER/batch_migration.py', '--cfg-path', '/tmp/tmpdxjoky8n', '--error-file', '/tmp/tmp4qqlfuol']
[2022-04-18, 21:13:16 KST] {standard_task_runner.py:77} INFO - Job 21: Subtask create_hive_table
[2022-04-18, 21:13:17 KST] {logging_mixin.py:109} INFO - Running <TaskInstance: Batch_Migration_Data.create_hive_table manual__2022-04-18T03:12:24.861251+00:00 [running]> on host airflow
[2022-04-18, 21:13:17 KST] {taskinstance.py:1431} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=Batch_Migration_Data
AIRFLOW_CTX_TASK_ID=create_hive_table
AIRFLOW_CTX_EXECUTION_DATE=2022-04-18T03:12:24.861251+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-04-18T03:12:24.861251+00:00
[2022-04-18, 21:13:17 KST] {hive.py:132} INFO - Executing: create external table ctf_20220418 (tid int, tdate varchar(255), ttime varchar(255), test1 FLOAT, test2 int, test3 int, test4 float, test5 FLOAT, test6 FLOAT, test7 FLOAT, test8 FLOAT, test9 FLOAT, D0010 FLOAT, test11 FLOAT, test12 FLOAT, tset13 FLOAT, test14 FLOAT, test15 FLOAT, test16 FLOAT, test17 FLOAT, test18 FLOAT, test19 FLOAT, defect_rate int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
[2022-04-18, 21:13:18 KST] {base.py:79} INFO - Using connection to: id: beeline_hive_container. Host: hive, Port: 10000, Schema: default, Login: root, Password: ***, extra: {'use_beeline': True, 'auth': 'NONE'}
[2022-04-18, 21:13:18 KST] {hive.py:150} INFO - Passing HiveConf: {'airflow.ctx.dag_owner': 'airflow', 'airflow.ctx.dag_id': 'Batch_Migration_Data', 'airflow.ctx.task_id': 'create_hive_table', 'airflow.ctx.execution_date': '2022-04-18T03:12:24.861251+00:00', 'airflow.ctx.dag_run_id': 'manual__2022-04-18T03:12:24.861251+00:00'}
[2022-04-18, 21:13:18 KST] {hive.py:242} INFO - beeline -u "jdbc:hive2://hive:10000/default;auth=NONE" -n root -p *** -hiveconf airflow.ctx.dag_id=Batch_Migration_Data -hiveconf airflow.ctx.task_id=create_hive_table -hiveconf airflow.ctx.execution_date=2022-04-18T03:12:24.861251+00:00 -hiveconf airflow.ctx.dag_run_id=manual__2022-04-18T03:12:24.861251+00:00 -hiveconf airflow.ctx.dag_owner=airflow -hiveconf airflow.ctx.dag_email= -hiveconf mapred.job.name=Airflow HiveOperator task for airflow.Batch_Migration_Data.create_hive_table.2022-04-18T03:12:24.861251+00:00 -f /tmp/airflow_hiveop_jhcjl4f1/tmp25idpdt6
yvgpqqbh

yvgpqqbh1#

这是airflow的直线中的一个bug(位于miniconda 2/envs/airflow/bin/beeline),它阻止直线在后台模式下运行。
有关详细信息,请参阅:
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions
顺便说一句:这个解决方案对我不起作用。我终于黑进了spark类脚本。
导出HADOOP_CLIENT_OPTS="-Djline.终端=jline.不支持的终端”

相关问题