如何在Hadoop2.6中访问作业计数器和文件系统计数器?

hec6srdp  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(303)

Reducer 在我的mapreduce程序中,我想读一本 JobCounter 和一个 FileSystemCounter . 运行命令时 mapred job -status <job id> ,我需要的计数器按其显示名称列出:

...
File System Counters
    FILE: Number of bytes read=148874
    FILE: Number of bytes written=22010065
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=135823
    HDFS: Number of bytes written=44423504133
    HDFS: Number of read operations=2185
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=1316
Job Counters 
    Launched map tasks=1
    Launched reduce tasks=200
    Rack-local map tasks=1
    Total time spent by all maps in occupied slots (ms)=5293
    Total time spent by all reduces in occupied slots (ms)=972893
    Total time spent by all map tasks (ms)=5293
    Total time spent by all reduce tasks (ms)=972893
    Total vcore-seconds taken by all map tasks=5293
    Total vcore-seconds taken by all reduce tasks=972893
    Total megabyte-seconds taken by all map tasks=5420032
    Total megabyte-seconds taken by all reduce tasks=996242432
...

如何在运行时从系统内部访问这些计数器 Reducer 的代码?
使用google,我找不到任何关于如何访问这些计数器的有用信息。使用context.getcounter(string groupname,string countername)的直接尝试无法检索 Counter 示例,因此抛出 NullPointerException 一经召唤 getValue() :

long bytes = context.getCounter(
    FileSystemCounter.class.getName(),
    FileSystemCounter.BYTES_WRITTEN.name()
).getValue();
long milliseconds = context.getCounter(
    JobCounter.class.getName(),
    JobCounter.MILLIS_REDUCES.name()
).getValue();
snvhrwxg

snvhrwxg1#

Counters counters = job.getCounters();

for (CounterGroup group : counters) {
      System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")");
      System.out.println("  number of counters in this group: " + group.size());
      for (Counter counter : group) {
        System.out.println("  - " + counter.getDisplayName() + ": " + counter.getName() + ": "+counter.getValue());
      }
    }

我认为这将有助于打印所有计数器及其值。

相关问题