使用python向flume avro源代码发送消息

pvcm50d1  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(571)

我想编写一个python程序,将json文档流转换为avro,并将它们流到flume(这样我就可以将它们发送到solr和parquet)。
我正在看一个使用pythonavro库的示例,它声称实现了avrorpc协议。https://github.com/phunt/avro-rpc-quickstart/blob/master/src/main/python/send_message.py
但是当我尝试将示例发送到我的flume avro服务器时,它似乎只是关闭了连接。例如

$ ./atest.py jnkjn kjnkjn e3e3
Have requester
About to request... REQUEST>Ú­òs±3ô8RÍsÊT¿ÌQÚ­òs±3ô8RÍsÊT¿Ìsend
jnkjn
     kjnkje3e3<
RESPONSE><
Traceback (most recent call last):
  File "atest.py", line 35, in <module>
    print("Result: " + requestor.request('send', params))
  ...
  File "/usr/lib64/python2.6/httplib.py", line 991, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 392, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 356, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

我不得不在httplib中放入一个 print语句,以查看响应是否为空,或者连接是否已关闭。
通过将avropython库指向flume avro源代码,我是否已经走上了写的轨道?他们谈的是同一个协议吗?
我正在运行最新的cdh5.1堆栈。
在查看flume日志时,我注意到每当我尝试连接时都会抛出一个非常特定的错误:

2014-09-16 16:35:15,745 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 => /172.31.1.204:19999] OPEN
2014-09-16 16:35:15,745 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 => /172.31.1.204:19999] BOUND: /172.31.1.204:19999
2014-09-16 16:35:15,746 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 => /172.31.1.204:19999] CONNECTED: /192.168.150.84:38516
2014-09-16 16:35:15,747 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 :> /172.31.1.204:19999] DISCONNECTED
2014-09-16 16:35:15,747 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 :> /172.31.1.204:19999] UNBOUND
2014-09-16 16:35:15,747 INFO org.apache.avro.ipc.NettyServer: [id: 0x0633c6d1, /192.168.150.84:38516 :> /172.31.1.204:19999] CLOSED
2014-09-16 16:35:15,747 INFO org.apache.avro.ipc.NettyServer: Connection to /192.168.150.84:38516 disconnected.
2014-09-16 16:35:15,747 WARN org.apache.avro.ipc.NettyServer: Unexpected exception from downstream.
org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected: 539959368 items! Connection closed.
    at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167)
    at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:422)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
erhoui1w

erhoui1w1#

当我遇到这个问题时,是因为Flume通道的缓冲区不够大。调整了通道的缓冲区大小以解决它。
对于内存通道,属性为:bytecapacitybufferpercentage

相关问题