websphere mq与flume的集成

c2e8gylq  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(388)

我正在尝试将websphere(ibm)mq与flume集成。我有一些xml文件将来自mq&我想在flume中获取这些消息并在spark中进行进一步处理。
我在AWSEC2示例上做这个集成,我的hadoop也安装在这个示例上。下面是我遵循的集成步骤。
创建队列管理器:https://www.ibm.com/support/knowledgecenter/ssfksj_7.5.0/com.ibm.mq.con.doc/q015210_.htm
./crtmqm-q-d my.default.bts.queue-u system.dead.letter.queue csbts.queue.manager
启动队列管理器:
./strmqm csbts.queue.manager ./runmqsc CSBTS.QUEUE.MANAGER 定义服务器通道
define channel(chan2)chltype(svrconn)trptype(tcp)descr('到客户端的服务器连接2')
定义客户端通道
define channel(chan2)chltype(clntconn)trptype(tcp)conname(10.0.0.238)qmname(csbts.queue.manager)descr('客户端到服务器的连接2')
alter qmgr chlauth(已禁用)
定义本地队列
定义qlocal(csbts.deal)
添加消息
./amqsput csbts.deal csbts.queue.manager
创建连接工厂
def cf(cf4)qmgr(csbts.queue.manager)传输(客户端)主机名(10.0.0.238)通道(chan2)
启动侦听器
定义侦听器(csbts\u list)trptype(tcp)端口(1414)控制(qmgr)
显示侦听器(csbts\U列表)
显示lsstatus(csbts\U列表)
(重新启动qm)
流Flume
/usr/bin/flume ng agent-c/etc/flume ng/conf-f/etc/flume ng/conf/flume.conf-n第1层
flume通过linux用户“hduser”执行。
下面是我们得到的身份验证问题。

20 Jul 2016 09:17:33,236 ERROR [PollableSourceRunner-JMSSource-source1] (org.apache.flume.source.PollableSourceRunner$PollingRunner.run:156)  - Unhandled exception, logging and sleeping for 5000ms
org.apache.flume.FlumeException: Source had error configuring or starting
        at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:48)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: Could not create connection to broker
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:76)
        at org.apache.flume.source.jms.JMSMessageConsumerFactory.create(JMSMessageConsumerFactory.java:32)
        at org.apache.flume.source.jms.JMSSource.createConsumer(JMSSource.java:317)
        at org.apache.flume.source.jms.JMSSource.doStart(JMSSource.java:299)
        at org.apache.flume.source.BasicSourceSemantics.start(BasicSourceSemantics.java:83)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:74)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'CSBTS.QUEUE.MANAGER' with connection mode 'Client' and host name '10.0.0.238(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.
        at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:531)
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
        at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:413)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
        at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7814)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:299)
        at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
        at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6024)
        at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6073)
        at org.apache.flume.source.jms.JMSMessageConsumer.<init>(JMSMessageConsumer.java:69)
        ... 13 more
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').
        at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
        ... 21 more

我被困在这从过去的3个星期,如果有人曾经面对过这个问题,那么请帮助我们。
来自amqerr01.log的错误日志

----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5540: Application 'flume.node.Application' did not supply a user ID and
password

EXPLANATION:
The queue manager is configured to require a user ID and password, but none was
supplied.
ACTION:
Ensure that the application provides a valid user ID and password, or change
the queue manager configuration to OPTIONAL to allow applications to connect
which have not supplied a user ID and password.
----- amqzfuca.c : 4287 -------------------------------------------------------
07/21/2016 10:08:17 AM - Process(16169.50) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ5541: The failed authentication check was caused by the queue manager
CONNAUTH CHCKCLNT(REQDADM) configuration.

EXPLANATION:
The user ID 'mqm' and its password were checked because the user ID is
privileged and the queue manager connection authority (CONNAUTH) configuration
refers to an authentication information (AUTHINFO) object named
'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with CHCKCLNT(REQDADM). 

This message accompanies a previous error to clarify the reason for the user ID
and password check.
ACTION:
Refer to the previous error for more information. 

Ensure that a password is specified by the client application and that the
password is correct for the user ID. The authentication configuration of the
queue manager connection determines the user ID repository. For example, the
local operating system user database or an LDAP server. 

To avoid the authentication check, you can either use an unprivileged user ID
or amend the authentication configuration of the queue manager. You can amend
the CHCKCLNT attribute in the CHLAUTH record, but you should generally not
allow unauthenticated remote access.
-------------------------------------------------------------------------------
07/21/2016 10:08:18 AM - Process(18280.13) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'mqm'.

EXPLANATION:
The call to initialize the User ID 'mqm' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------
07/21/2016 10:10:58 AM - Process(16169.51) User(mqm) Program(amqzlaa0)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ8077: Entity 'hduser' has insufficient authority to access object
'CSBTS.QUEUE.MANAGER'.

EXPLANATION:
The specified entity is not authorized to access the required object. The
following requested permissions are unauthorized: connect
ACTION:
Ensure that the correct level of authority has been set for this entity against
the required object, or ensure that the entity is a member of a privileged
group.
----- amqzfubx.c : 687 --------------------------------------------------------
07/21/2016 10:10:58 AM - Process(18280.14) User(mqm) Program(amqrmppa)
                    Host(ip-10-0-0-238) Installation(Installation1)
                    VRMF(8.0.0.4) QMgr(CSBTS.QUEUE.MANAGER)

AMQ9557: Queue Manager User ID initialization failed for 'hduser'.

EXPLANATION:
The call to initialize the User ID 'hduser' failed with CompCode 2 and Reason
2035.
ACTION:
Correct the error and try again.
----- cmqxrsrv.c : 2321 -------------------------------------------------------
sshcrbum

sshcrbum1#

感谢@moraghughson,这是基于您的建议,请查找amqerr01.log以了解更多问题。它帮助我克服了mq与flume交互的问题。
我所做的是将队列管理器配置更改为可选,以允许未提供用户id和密码的应用程序进行连接。在我的例子中,我没有在flume.conf文件中传递密码。
我跟踪了这个链接-http://www-01.ibm.com/support/docview.wss?uid=swg21680930 使用选项b。
下面是我用来使身份验证可选的命令-

ALTER AUTHINFO(SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL)

REFRESH SECURITY TYPE(CONNAUTH)

我希望这对任何试图将mq与flume/kakfa集成的人都有帮助。

相关问题