异常长的pig作业开始时间

aurhwmvo 于 2021-06-25 发布在 Pig

关注(0)|答案(1)|浏览(296)

一个pig脚本（并不比我构建的任何其他脚本都复杂）在作业开始之前，它似乎循环了很长一段时间：

2013-10-08 10:46:07,655 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 10
2013-10-08 10:46:07,659 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 10
2013-10-08 10:46:09,168 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 10
2013-10-08 10:46:09,168 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 10
2013-10-08 10:46:11,381 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 10
2013-10-08 10:46:11,381 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 10
2013-10-08 10:46:13,875 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 10
2013-10-08 10:46:13,875 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 10
2013-10-08 10:46:16,303 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 10

当这一步通常在几秒钟内完成时，它会重复上述步骤大约4分钟。我还不能确定原因-除了删除脚本的一部分，但问题似乎不是由脚本的任何特定部分引起的。我有其他像这个一样复杂的脚本，我没有这个问题。是什么导致了这个问题？

apache-pig

来源：https://stackoverflow.com/questions/19251662/unusually-long-pig-job-start-time