用于关闭impala查询的python脚本我正在使用urllib和json库

yqhsw0fo 于 2021-06-26 发布在 Impala

关注(0)|答案(1)|浏览(215)

我正在使用python脚本关闭impala查询。这个脚本可以很好地处理字符串变量，但是我不能为字符串+整数变量编写逻辑。我使用的变量是如果status=“finished”或“exception”且waiting=true，则使用urllib关闭查询。但是，我需要帮助包括整数变量。整数变量将类似于15秒10秒或15秒10毫秒。
下面是代码：

import urllib, json
from datetime import datetime

nodes = ["node1.com:1202",   #example destination servers
      "node2.com:1202",
      "node3.com:1202",       
    ]

for i, datanode in enumerate(nodes):
 print("Checking {}: {}".format(i, datanode))
 try:
     response = urllib.urlopen(datanode + "queries?json")
     data = json.loads(response.read())

     if data["num_waiting_queries"] > 0:
         print(data["num_waiting_queries"])
         for in_flight_query in data["in_flight_queries"]:
             if in_flight_query["waiting"] is True and in_flight_query['state'] == "FINISHED" or  "EXCEPTION" and in_flight_query['duration'][:2] > 15:  #I have included the duration here but it not working as expected. I guess the logic is incorrect. Code works after removing 'duration'.
                 cancel_url = datanode + "cancel_query?query_id={}".format(in_flight_query['query_id'])
                 print(cancel_url)
                 response = urllib.urlopen(cancel_url)

 except IOError:
     print("Skipping {}: {}".format(i, datanode))

 except Exception as e:
     print(e)

json对象：

This the json script,I want use variable "duration" from it. 
"in_flight_queries": [
    {
        "effective_user": "user4",
        "default_db": "testdb",
        "stmt": "select * from table",
        "stmt_type": "QUERY",
        "start_time": "2018-06-18 01:04:12.558731000",
        "end_time": "00:00:00",
        "duration": "30m34s",
        "progress": "16 / 30 (53.3333%)",
        "state": "FINISHED",
        "rows_fetched": 10,
        "query_id": "7243:954ed9414b96abaf",
        "last_event": "First row fetched",
        "waiting": true,
        "executing": false,
        "waiting_time": "30m30s"
    }

impala python python-3.x cloudera-cdh

来源：https://stackoverflow.com/questions/50906387/python-script-for-closing-impala-queries-i-am-using-urllib-and-json-libraries

1条答案

按热度按时间

ngynwnxp1#

基于上述讨论，我认为您可以使用下面的函数提取时间，并在稍后进行比较 if 声明：

from datetime import datetime

def extract_seconds(text: str):
    pat = '%Mm%Ss' # convenient reference: http://strftime.org/    
    try:
        x = datetime.strptime(text, pat)
    except ValueError:    
        raise ValueError('Cannot extract time from {} with {}'.format(x, pat))
    return x.minute * 60 + x.second    

assert extract_seconds('30m34s') == 1834

如果你愿意和 datetime 对象，这甚至更短：

def extract(text: str):
    pat = '%Mm%Ss' # convenient reference: http://strftime.org/    
    try:
        return datetime.strptime(text, pat).time()
    except ValueError:    
        raise ValueError('Cannot extract time from {} with {}'.format(x, pat))

赞(0）回复(0）举报 2021-06-26

我来回答

用于关闭impala查询的python脚本我正在使用urllib和json库

1条答案

相关问题

热门标签

最新问答