aws-elasticmapreduce似乎没有正确地将流转换为jar

8wtpewkr  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(246)

当我在管道版本中运行它们时,我有一个Map器和还原器可以正常工作:

cat data.csv | ./mapper.py | sort -k1,1 | ./reducer.py

我使用了elastic mapreducer向导,加载了输入、输出、引导等。引导成功,但在执行过程中仍然出现错误。
这是我在步骤1的标准中遇到的错误。。。

+ /etc/init.d/hadoop-state-pusher-control stop
+ PID_FILE=/mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ LOG_FILE=/mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ SVC_FILE=/mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ case $1 in
+ stop
+ echo 0
/etc/init.d/hadoop-state-pusher-control: line 35: /mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher: No such file or directory
+ /etc/init.d/hadoop-state-pusher-control start
+ PID_FILE=/mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ LOG_FILE=/mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ SVC_FILE=/mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ case $1 in
+ start
++ dirname /mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ sudo -u hadoop mkdir -p /mnt/var/lib/hadoop-state-pusher
+ echo 1
++ dirname /mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ sudo -u hadoop mkdir -p /mnt/var/run/hadoop-state-pusher
++ dirname /mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ sudo -u hadoop mkdir -p /mnt/var/log/hadoop-state-pusher
+ disown %1
+ sleep 5
+ sudo -u hadoop /usr/bin/hadoop-state-pusher -server --pidfile /mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ exit 0
Command exiting with ret '0'

这很神秘。这到底是什么意思?
装东西好像有问题?其他哪些日志文件可能会说一些信息,我应该在哪里寻找?
我尝试了一个我在这里找到的解决方案,只是把示例变大了,但是没有成功,同样的错误消息。

8oomwypt

8oomwypt1#

我查错了日志文件。有一个不同的(有6个?)实际上给了我一些有用的python调试信息。原来我用的是 string interpolation.format("of this kind {}, not this kind with a digit {1}".vars(a,b)) 这在python<2.7中是不受支持的,这是默认安装在elasticmapreduce中使用的ec2映像上的。

相关问题