pyspark DataFactory ondemand HDInsight群集将日志记录到Azure Blob:“Bloblog RotatingFileLog”

jdgnovmf  于 5个月前  发布在  Spark
关注(0)|答案(1)|浏览(57)

我正在使用按需HDInsight群集在Azure Datafactory管道上运行spark脚本。我想将管道日志保存到Azure容器。正在尝试实现此:
Logging the errors to azure blob
但是,当我使用上面链接中提到的日志逻辑运行脚本时,我一直得到No模块名为azure_storage_logging.handlers。
我如何在我的spark脚本中为这个模块做一个“pip install”,我正在datafactory管道上运行?
或者任何其他替代方法来实现我的spark脚本中的日志逻辑?

5ssjco0h

5ssjco0h1#

在代码的开头添加以下Python代码,以便安装必要的包。

import os

try:
    from azure_storage_logging.handlers import BlobStorageRotatingFileHandler
    print("Module is installed.")
except ImportError:
    print("Module is not installed. Installing...")
    os.system('pip install azure-storage==0.36.0 azure-storage-logging')
    from azure_storage_logging.handlers import BlobStorageRotatingFileHandler

字符串
代码:

import logging
import os

try:
    from azure_storage_logging.handlers import BlobStorageRotatingFileHandler
    print("Module is installed.")
except ImportError:
    print("Module is not installed. Installing...")
    os.system('pip install azure-storage==0.36.0 azure-storage-logging')
    from azure_storage_logging.handlers import BlobStorageRotatingFileHandler

mystorageaccountname='acc_name'
mystorageaccountkey='key'

logger = logging.getLogger('service_logger')
log_formater = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(process)d - %(message)s')
azure_blob_handler = BlobStorageRotatingFileHandler(filename = 'service.log', 
                                                        account_name=mystorageaccountname,
                                                        account_key=mystorageaccountkey,
                                                        maxBytes=5,
                                                        container='service-log')
azure_blob_handler.setLevel(logging.INFO)
azure_blob_handler.setFormatter(log_formater)
logger.addHandler(azure_blob_handler)

logger.warning('warning message')

输出单位:service.log

2023-12-27  12:35:13,292 - service_logger - WARNING - 16456 - warning message

相关问题