org.apache.hadoop.mapred.Task.getJobID()方法的使用及代码示例

x33g5p2x  于2022-01-30 转载在 其他  
字(12.8k)|赞(0)|评价(0)|浏览(102)

本文整理了Java中org.apache.hadoop.mapred.Task.getJobID()方法的一些代码示例，展示了Task.getJobID()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Task.getJobID()方法的具体详情如下：
包路径：org.apache.hadoop.mapred.Task
类名称：Task
方法名：getJobID

Task.getJobID介绍

[英]Get the job name for this task.
[中]获取此任务的作业名称。

代码示例

代码示例来源：origin: com.facebook.hadoop/hadoop-core

private List<String> buildTaskCleanupArgs(
  TaskControllerPathDeletionContext context) {
 List<String> commandArgs = new ArrayList<String>(3);
 commandArgs.add(context.mapredLocalDir.toUri().getPath());
 commandArgs.add(context.task.getJobID().toString());
 String workDir = "";
 if (context.isWorkDir) {
  workDir = "/work";
 }
 if (context.task.isTaskCleanupTask()) {
  commandArgs.add(context.task.getTaskID() + TaskTracker.TASK_CLEANUP_SUFFIX
          + workDir);
 } else {
  commandArgs.add(context.task.getTaskID() + workDir);
 }
 return commandArgs;
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
 * Builds the path of taskAttemptDir OR taskWorkDir based on
 * mapredLocalDir, jobId, taskId, etc
 */
String buildPathForDeletion() {
 String subDir = TaskTracker.getLocalTaskDir(task.getJobID().toString(),
   task.getTaskID().toString(), task.isTaskCleanupTask());
 if (isWorkDir) {
  subDir = subDir + Path.SEPARATOR + "work";
 }
 return mapredLocalDir.toUri().getPath() + Path.SEPARATOR + subDir;
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
 * Run the given task asynchronously.
 */
void runTask(Task task) {
 JobID jobId = task.getJobID();
 boolean isMap = task.isMapTask();
 JVMId jvmId = new JVMId(jobId, isMap, taskCounter++);
 synchronized(this) {
  taskJvms.put(jvmId.getId(), jvmId);
  runningTasks.put(jvmId.getId(), task);
 }
 TaskRunnable taskRunnable = new TaskRunnable(task, jvmId.getId());
 executor.execute(taskRunnable);
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-mapreduce-client-common

static void setupChildMapredLocalDirs(Task t, JobConf conf) {
 String[] localDirs = conf.getTrimmedStrings(MRConfig.LOCAL_DIR);
 String jobId = t.getJobID().toString();
 String taskId = t.getTaskID().toString();
 boolean isCleanup = t.isTaskCleanupTask();
 String user = t.getUser();
 StringBuffer childMapredLocalDir =
   new StringBuffer(localDirs[0] + Path.SEPARATOR
     + getLocalTaskDir(user, jobId, taskId, isCleanup));
 for (int i = 1; i < localDirs.length; i++) {
  childMapredLocalDir.append("," + localDirs[i] + Path.SEPARATOR
    + getLocalTaskDir(user, jobId, taskId, isCleanup));
 }
 LOG.debug(MRConfig.LOCAL_DIR + " for child : " + childMapredLocalDir);
 conf.set(MRConfig.LOCAL_DIR, childMapredLocalDir.toString());
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapreduce-client-common

static void setupChildMapredLocalDirs(Task t, JobConf conf) {
 String[] localDirs = conf.getTrimmedStrings(MRConfig.LOCAL_DIR);
 String jobId = t.getJobID().toString();
 String taskId = t.getTaskID().toString();
 boolean isCleanup = t.isTaskCleanupTask();
 String user = t.getUser();
 StringBuffer childMapredLocalDir =
   new StringBuffer(localDirs[0] + Path.SEPARATOR
     + getLocalTaskDir(user, jobId, taskId, isCleanup));
 for (int i = 1; i < localDirs.length; i++) {
  childMapredLocalDir.append("," + localDirs[i] + Path.SEPARATOR
    + getLocalTaskDir(user, jobId, taskId, isCleanup));
 }
 LOG.debug(MRConfig.LOCAL_DIR + " for child : " + childMapredLocalDir);
 conf.set(MRConfig.LOCAL_DIR, childMapredLocalDir.toString());
}

代码示例来源：origin: io.hops/hadoop-mapreduce-client-common

static void setupChildMapredLocalDirs(Task t, JobConf conf) {
 String[] localDirs = conf.getTrimmedStrings(MRConfig.LOCAL_DIR);
 String jobId = t.getJobID().toString();
 String taskId = t.getTaskID().toString();
 boolean isCleanup = t.isTaskCleanupTask();
 String user = t.getUser();
 StringBuffer childMapredLocalDir =
   new StringBuffer(localDirs[0] + Path.SEPARATOR
     + getLocalTaskDir(user, jobId, taskId, isCleanup));
 for (int i = 1; i < localDirs.length; i++) {
  childMapredLocalDir.append("," + localDirs[i] + Path.SEPARATOR
    + getLocalTaskDir(user, jobId, taskId, isCleanup));
 }
 LOG.debug(MRConfig.LOCAL_DIR + " for child : " + childMapredLocalDir);
 conf.set(MRConfig.LOCAL_DIR, childMapredLocalDir.toString());
}

代码示例来源：origin: ch.cern.hadoop/hadoop-mapreduce-client-common

static void setupChildMapredLocalDirs(Task t, JobConf conf) {
 String[] localDirs = conf.getTrimmedStrings(MRConfig.LOCAL_DIR);
 String jobId = t.getJobID().toString();
 String taskId = t.getTaskID().toString();
 boolean isCleanup = t.isTaskCleanupTask();
 String user = t.getUser();
 StringBuffer childMapredLocalDir =
   new StringBuffer(localDirs[0] + Path.SEPARATOR
     + getLocalTaskDir(user, jobId, taskId, isCleanup));
 for (int i = 1; i < localDirs.length; i++) {
  childMapredLocalDir.append("," + localDirs[i] + Path.SEPARATOR
    + getLocalTaskDir(user, jobId, taskId, isCleanup));
 }
 LOG.debug(MRConfig.LOCAL_DIR + " for child : " + childMapredLocalDir);
 conf.set(MRConfig.LOCAL_DIR, childMapredLocalDir.toString());
}

代码示例来源：origin: mesos/hadoop

mesosTracker.jobs.add(task.getJobID());

代码示例来源：origin: org.jvnet.hudson.hadoop/hadoop-core

public TaskRunner(TaskTracker.TaskInProgress tip, TaskTracker tracker, 
  JobConf conf) {
 this.tip = tip;
 this.t = tip.getTask();
 this.tracker = tracker;
 this.conf = conf;
 this.mapOutputFile = new MapOutputFile(t.getJobID());
 this.mapOutputFile.setConf(conf);
 this.jvmManager = tracker.getJvmManagerInstance();
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred

/**
 * Prepare the Configs.LOCAL_DIR for the child. The child is sand-boxed now.
 * Whenever it uses LocalDirAllocator from now on inside the child, it will
 * only see files inside the attempt-directory. This is done in the Child's
 * process space.
 */
static void setupChildMapredLocalDirs(Task t, JobConf conf) {
 String[] localDirs = conf.getTrimmedStrings(MRConfig.LOCAL_DIR);
 String jobId = t.getJobID().toString();
 String taskId = t.getTaskID().toString();
 boolean isCleanup = t.isTaskCleanupTask();
 String user = t.getUser();
 StringBuffer childMapredLocalDir =
   new StringBuffer(localDirs[0] + Path.SEPARATOR
     + TaskTracker.getLocalTaskDir(user, jobId, taskId, isCleanup));
 for (int i = 1; i < localDirs.length; i++) {
  childMapredLocalDir.append("," + localDirs[i] + Path.SEPARATOR
    + TaskTracker.getLocalTaskDir(user, jobId, taskId, isCleanup));
 }
 LOG.debug(MRConfig.LOCAL_DIR + " for child : " + childMapredLocalDir);
 conf.set(MRConfig.LOCAL_DIR, childMapredLocalDir.toString());
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
  * Called in case the task needs to be killed. Canceling will kill any map
  * wait threads and also remove it from the queue of tasks that should be
  * marked as finished.
  * @param tip the killed TaskInProgress
  */
 public void cancel(TaskInProgress tip) {
  LOG.info("Canceling task "  + tip.getTask().getTaskID() + " of job " +
    tip.getTask().getJobID());
  // Cancel & remove the map completion finish thread for reduce tasks.
  if (!tip.getTask().isMapTask() && !tip.getTask().isTaskCleanupTask()) {
   if (!mapperWaitThreadMap.containsKey(tip)) {
    throw new RuntimeException("Mapper wait thread doesn't exist " +
      "for " + tip.getTask().getTaskID());
   }
   LOG.debug("Interrupting mapper wait thread for " +
     tip.getTask().getTaskID() + " job " +
     tip.getTask().getJobID());
   mapperWaitThreadMap.get(tip).interrupt();
   LOG.debug("Removing mapper wait thread for " +
     tip.getTask().getTaskID() + " job " + tip.getTask().getJobID());
   mapperWaitThreadMap.remove(tip);
  } else {
   LOG.debug(tip.getTask().getTaskID() + " is not a reduce task, so " +
     "not canceling mapper wait thread");
  }
  removeFromFinishingQueue(tip);
 }
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

public TaskRunner(
  TaskTracker.TaskInProgress tip,
  Task task,
  TaskTracker tracker,
  JobConf conf) {
 this.tip = tip;
 this.t = task;
 this.tracker = tracker;
 this.conf = conf;
 this.mapOutputFile =
  new MapOutputFile(task.getJobID(), tracker.getAsyncDiskService());
 this.mapOutputFile.setConf(conf);
 this.jvmManager = tracker.getJvmManagerInstance();
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
 * @param tip a reduce task in progress that we should wait for the mappers
 * to finish
 * @param taskRunner the task runner thread that the TIP should be sent
 * to after all the mappers are done.
 * @param umbilicalProtocol The umbilical
 * events
 */
public MapperWaitThread(TaskInProgress tip,
  SimulatedTaskRunner taskRunner, TaskUmbilicalProtocol umbilicalProtocol) {
 this.taskRunner = taskRunner;
 this.umbilicalProtocol = umbilicalProtocol;
 this.tip = tip;
 this.setName("Map-waiting thread for job: " + tip.getTask().getJobID() +
   " reduce task: " + tip.getTask().getTaskID());
 // Don't want to prevent the TT from shutting down just because of this
 // thread
 this.setDaemon(true);
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred

/**
 * Write the child's configuration to the disk and set it in configuration so
 * that the child can pick it up from there.
 * 
 * @param lDirAlloc
 * @throws IOException
 */
void setupChildTaskConfiguration(LocalDirAllocator lDirAlloc)
  throws IOException {
 Path localTaskFile =
   lDirAlloc.getLocalPathForWrite(TaskTracker.getTaskConfFile(
     t.getUser(), t.getJobID().toString(), t.getTaskID().toString(), t
       .isTaskCleanupTask()), conf);
 // write the child's task configuration file to the local disk
 writeLocalTaskFile(localTaskFile.toString(), conf);
 // Set the final job file in the task. The child needs to know the correct
 // path to job.xml. So set this path accordingly.
 t.setJobFile(localTaskFile.toString());
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred

/**
 * Returns the taskWorkDir or taskLocalDir based on whether 
 * {@link TaskControllerTaskPathDeletionContext} is configured to delete
 * the workDir.
 */
@Override
protected String getPath() {
 String subDir = (isWorkDir) ? TaskTracker.getTaskWorkDir(task.getUser(),
   task.getJobID().toString(), task.getTaskID().toString(),
   task.isTaskCleanupTask())
  : TaskTracker.getLocalTaskDir(task.getUser(),
   task.getJobID().toString(), task.getTaskID().toString(),
   task.isTaskCleanupTask());
 return subDir;
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred

RunningJob localizeJob(TaskInProgress tip
             ) throws IOException, InterruptedException {
 Task t = tip.getTask();
 JobID jobId = t.getJobID();
 RunningJob rjob = addTaskToJob(jobId, tip);
 // Initialize the user directories if needed.
 getLocalizer().initializeUserDirs(t.getUser());
 synchronized (rjob) {
  if (!rjob.localized) {
      JobConf localJobConf = localizeJobFiles(t, rjob);
   // initialize job log directory
   initializeJobLogDir(jobId, localJobConf);
   // Now initialize the job via task-controller so as to set
   // ownership/permissions of jars, job-work-dir. Note that initializeJob
   // should be the last call after every other directory/file to be
   // directly under the job directory is created.
   JobInitializationContext context = new JobInitializationContext();
   context.jobid = jobId;
   context.user = t.getUser();
   context.workDir = new File(localJobConf.get(JOB_LOCAL_DIR));
   taskController.initializeJob(context);
   rjob.jobConf = localJobConf;
   rjob.keepJobFiles = ((localJobConf.getKeepTaskFilesPattern() != null) ||
              localJobConf.getKeepFailedTaskFiles());
   rjob.localized = true;
  }
 }
 return rjob;
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred

/**
 * Remove the tip and update all relevant state.
 * 
 * @param tip {@link TaskInProgress} to be removed.
 * @param wasFailure did the task fail or was it killed?
 */
private void purgeTask(TaskInProgress tip, boolean wasFailure) 
throws IOException {
 if (tip != null) {
  LOG.info("About to purge task: " + tip.getTask().getTaskID());
   
  // Remove the task from running jobs, 
  // removing the job if it's the last task
  removeTaskFromJob(tip.getTask().getJobID(), tip);
  tip.jobHasFinished(wasFailure);
  if (tip.getTask().isMapTask()) {
   indexCache.removeMap(tip.getTask().getTaskID().toString());
  }
 }
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
 * Remove the tip and update all relevant state.
 *
 * @param tip {@link TaskInProgress} to be removed.
 * @param wasFailure did the task fail or was it killed?
 */
private void purgeTask(TaskInProgress tip, boolean wasFailure)
throws IOException {
 if (tip != null) {
  LOG.info("About to purge task: " + tip.getTask().getTaskID());
  // Remove the task from running jobs,
  // removing the job if it's the last task
  removeTaskFromJob(tip.getTask().getJobID(), tip);
  tip.jobHasFinished(wasFailure);
  if (tip.getTask().isMapTask()) {
   indexCache.removeMap(tip.getTask().getTaskID().toString());
  }
 }
}

代码示例来源：origin: org.jvnet.hudson.hadoop/hadoop-core

/**
 * Remove the tip and update all relevant state.
 * 
 * @param tip {@link TaskInProgress} to be removed.
 * @param wasFailure did the task fail or was it killed?
 */
private void purgeTask(TaskInProgress tip, boolean wasFailure) 
throws IOException {
 if (tip != null) {
  LOG.info("About to purge task: " + tip.getTask().getTaskID());
   
  // Remove the task from running jobs, 
  // removing the job if it's the last task
  removeTaskFromJob(tip.getTask().getJobID(), tip);
  tip.jobHasFinished(wasFailure);
  if (tip.getTask().isMapTask()) {
   indexCache.removeMap(tip.getTask().getTaskID().toString());
  }
 }
}

代码示例来源：origin: com.facebook.hadoop/hadoop-core

/**
 * The primary public method that should be called to 'run' a task. Handles
 * both map and reduce tasks and marks them as completed after the configured
 * time interval
 * @param tip
 */
public void launchTask(TaskInProgress tip) throws IOException {
 LOG.info("Launching simulated task " + tip.getTask().getTaskID() +
   " for job " + tip.getTask().getJobID());
 TaskUmbilicalProtocol umbilicalProtocol = taskTracker.getUmbilical(tip);
 // For map tasks, we can just finish the task after some time. Same thing
 // with cleanup tasks, as we don't need to be waiting for mappers to finish
 if (tip.getTask().isMapTask() || tip.getTask().isTaskCleanupTask() ||
  tip.getTask().isJobCleanupTask() || tip.getTask().isJobSetupTask() ) {
  addTipToFinish(tip, umbilicalProtocol);
 } else {
  MapperWaitThread mwt =
    new MapperWaitThread(tip, this, umbilicalProtocol);
  // Save a reference to the mapper wait thread so that we can stop them if
  // the task gets killed
  mapperWaitThreadMap.put(tip, mwt);
  mwt.start();
 }
}

内容来源于网络，如有侵权，请联系作者删除！

task

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    Task类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com