hadoop计数器文档?

u3r8eeie  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(390)

这个问题在这里已经有了答案

hadoop mapreduce控制台输出说明(1个答案)
两年前关门了。
在我的 MapReduce 工作结束后,我得到了很多 Counter 信息:

File System Counters
                FILE: Number of bytes read=4386096368
                FILE: Number of bytes written=8805370803
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=54583718086
                HDFS: Number of bytes written=4382090874
                HDFS: Number of read operations=1479
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=369
                Launched reduce tasks=1
                Data-local map tasks=369
                Total time spent by all maps in occupied slots (ms)=34288552
                Total time spent by all reduces in occupied slots (ms)=232084
                Total time spent by all map tasks (ms)=8572138
                Total time spent by all reduce tasks (ms)=58021
                Total vcore-seconds taken by all map tasks=8572138
                Total vcore-seconds taken by all reduce tasks=58021
                Total megabyte-seconds taken by all map tasks=35111477248
                Total megabyte-seconds taken by all reduce tasks=237654016
        Map-Reduce Framework
                Map input records=14753874
                Map output records=666776
                Map output bytes=4383426830
                Map output materialized bytes=4386098552
                Input split bytes=47970
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=4386098552
                Reduce input records=666776
                Reduce output records=666776
                Spilled Records=1333552
                Shuffled Maps =369
                Failed Shuffles=0
                Merged Map outputs=369
                GC time elapsed (ms)=1121584
                CPU time spent (ms)=23707900
                Physical memory (bytes) snapshot=152915259392
                Virtual memory (bytes) snapshot=2370755190784
                Total committed heap usage (bytes)=126644912128
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=49449743227
        File Output Format Counters
                Bytes Written=4382090874

我在哪里可以找到这些字段含义的解释?其中有些是相当明显的( Number of bytes read ),但其他人则更加模棱两可( Total time spent by all maps in occupied slotsTotal time spent by all map tasks ).
我找到了所有默认计数器的列表,但似乎找不到它们的解释或描述。
我相当惊讶,我似乎不能很容易地找到有关这个输出的文档。有人能提供一个链接或解释吗?

50pmv0ei

50pmv0ei1#

hadoop的第8章:权威指南(华盛顿州立大学链接中的完整pdf)提供了计数器的详细信息 MapReduce . 从第225页开始,见表8-1。safari books online上提供了此资源的最新版本(第4版)(您需要先登录)。

相关问题