org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler类的使用及代码示例

x33g5p2x  于2022-02-05 转载在 其他  
字(12.0k)|赞(0)|评价(0)|浏览(180)

本文整理了Java中org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler类的一些代码示例，展示了YarnScheduler类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。YarnScheduler类的具体详情如下：
包路径：org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler
类名称：YarnScheduler

YarnScheduler介绍

[英]This interface is used by the components to talk to the scheduler for allocating of resources, cleaning up resources.
[中]组件使用该接口与调度器进行对话，以分配资源、清理资源。

代码示例

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

private static YarnScheduler mockYarnScheduler() {
 YarnScheduler yarnScheduler = mock(YarnScheduler.class);
 when(yarnScheduler.getMinimumResourceCapability()).thenReturn(
   Resources.createResource(
     YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB));
 when(yarnScheduler.getMaximumResourceCapability()).thenReturn(
   Resources.createResource(
     YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB));
 when(yarnScheduler.getAppsInQueue(QUEUE_1)).thenReturn(
   Arrays.asList(getApplicationAttemptId(101), getApplicationAttemptId(102)));
 when(yarnScheduler.getAppsInQueue(QUEUE_2)).thenReturn(
   Arrays.asList(getApplicationAttemptId(103)));
 ApplicationAttemptId attemptId = getApplicationAttemptId(1);
 when(yarnScheduler.getAppResourceUsageReport(attemptId)).thenReturn(null);
 ResourceCalculator rc = new DefaultResourceCalculator();
 when(yarnScheduler.getResourceCalculator()).thenReturn(rc);
 return yarnScheduler;
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

@Override
 public void handle(SchedulerEvent event) {
  scheduler.handle(event);
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

/**
 * Utility method to validate a list resource requests, by insuring that the
 * requested memory/vcore is non-negative and not greater than max
 */
public static void normalizeAndValidateRequests(List<ResourceRequest> ask,
  Resource maximumResource, String queueName, YarnScheduler scheduler,
  RMContext rmContext)
  throws InvalidResourceRequestException {
 QueueInfo queueInfo = null;
 try {
  queueInfo = scheduler.getQueueInfo(queueName, false, false);
 } catch (IOException e) {
 }
 for (ResourceRequest resReq : ask) {
  SchedulerUtils.normalizeAndvalidateRequest(resReq, maximumResource,
    queueName, scheduler, rmContext, queueInfo);
 }
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

/**
 * Move all apps in the set of queues to the parent plan queue's default
 * reservation queue in a synchronous fashion
 */
private void moveAppsInQueueSync(String expiredReservation,
  String defReservationQueue) {
 List<ApplicationAttemptId> activeApps =
   scheduler.getAppsInQueue(expiredReservation);
 if (activeApps.isEmpty()) {
  return;
 }
 for (ApplicationAttemptId app : activeApps) {
  // fallback to parent's default queue
  try {
   scheduler.moveApplication(app.getApplicationId(), defReservationQueue);
  } catch (YarnException e) {
   LOG.warn(
     "Encountered unexpected error during migration of application: {}"
       + " from reservation: {}",
     app, expiredReservation, e);
  }
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

/**
 * First sets entitlement of queues to zero to prevent new app submission.
 * Then move all apps in the set of queues to the parent plan queue's default
 * reservation queue if move is enabled. Finally cleanups the queue by killing
 * any apps (if move is disabled or move failed) and removing the queue
 */
protected void cleanupExpiredQueues(String planQueueName,
  boolean shouldMove, Set<String> toRemove, String defReservationQueue) {
 for (String expiredReservationId : toRemove) {
  try {
   // reduce entitlement to 0
   String expiredReservation = getReservationQueueName(planQueueName,
     expiredReservationId);
   setQueueEntitlement(planQueueName, expiredReservation, 0.0f, 0.0f);
   if (shouldMove) {
    moveAppsInQueueSync(expiredReservation, defReservationQueue);
   }
   if (scheduler.getAppsInQueue(expiredReservation).size() > 0) {
    scheduler.killAllAppsInQueue(expiredReservation);
    LOG.info("Killing applications in queue: {}", expiredReservation);
   } else {
    scheduler.removeQueue(expiredReservation);
    LOG.info("Queue: " + expiredReservation + " removed");
   }
  } catch (YarnException e) {
   LOG.warn("Exception while trying to expire reservation: {}",
     expiredReservationId, e);
  }
 }
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

scheduler.getMaximumResourceCapability(),
   submissionContext.getQueue(), scheduler, isRecovery, rmContext);
} catch (InvalidResourceRequestException e) {
SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
  scheduler.getClusterResource(),
  scheduler.getMinimumResourceCapability(),
  scheduler.getMaximumResourceCapability(),
  scheduler.getMinimumResourceCapability());
return amReq;

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

appAttempt.scheduler.allocate(appAttempt.applicationAttemptId,
   EMPTY_CONTAINER_REQUEST_LIST, EMPTY_CONTAINER_RELEASE_LIST, null,
   null);
  .get(0));
RMContainerImpl rmMasterContainer = (RMContainerImpl)appAttempt.scheduler
  .getRMContainer(appAttempt.getMasterContainer().getId());
rmMasterContainer.setAMContainer(true);

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

rScheduler.getMaximumResourceCapability(), app.getQueue(),
   rScheduler, rmContext);
} catch (InvalidResourceRequestException e) {
  this.rScheduler.allocate(appAttemptId, ask, release, 
    blacklistAdditions, blacklistRemovals);
 for(RMNode rmNode: updatedNodes) {
  SchedulerNodeReport schedulerNodeReport =  
    rScheduler.getNodeReport(rmNode.getNodeID());
  Resource used = BuilderUtils.newResource(0, 0);
  int numContainers = 0;
allocateResponse.setAvailableResources(allocation.getResourceLimit());
allocateResponse.setNumClusterNodes(this.rScheduler.getNumClusterNodes());

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

Assert.assertEquals(0, scheduler.getNumClusterNodes());
int maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[0], maxVCores);
scheduler.handle(new NodeAddedSchedulerEvent(node1));
Assert.assertEquals(1, scheduler.getNumClusterNodes());
maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[1], maxVCores);
scheduler.handle(new NodeRemovedSchedulerEvent(node1));
Assert.assertEquals(0, scheduler.getNumClusterNodes());
maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[2], maxVCores);
scheduler.handle(new NodeAddedSchedulerEvent(node2));
Assert.assertEquals(1, scheduler.getNumClusterNodes());
maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[3], maxVCores);
scheduler.handle(new NodeAddedSchedulerEvent(node3));
Assert.assertEquals(2, scheduler.getNumClusterNodes());
maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[4], maxVCores);
scheduler.handle(new NodeRemovedSchedulerEvent(node3));
Assert.assertEquals(1, scheduler.getNumClusterNodes());
maxVCores = scheduler.getMaximumResourceCapability().getVirtualCores();
Assert.assertEquals(expectedMaxVCores[5], maxVCores);

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

try {
 QueueInfo queueInfo = 
  scheduler.getQueueInfo(request.getQueueName(),  
    request.getIncludeChildQueues(), 
    request.getRecursive());
 if (request.getIncludeApplications()) {
  List<ApplicationAttemptId> apps =
    scheduler.getAppsInQueue(request.getQueueName());
  appReports = new ArrayList<ApplicationReport>(apps.size());
  for (ApplicationAttemptId app : apps) {

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

getScheduler().getMaximumResourceCapability(app.getQueue());
} else {
 try {
  allocation = getScheduler().allocate(appAttemptId, ask,
    request.getSchedulingRequests(), release,
    blacklistAdditions, blacklistRemovals, containerUpdateRequests);
    .getApplicationAttempt(appAttemptId).pullUpdateContainerErrors());
response.setNumClusterNodes(getScheduler().getNumClusterNodes());

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

/**
 * {@link RMAppAttemptState#ALLOCATED}
 */
@SuppressWarnings("unchecked")
private void testAppAttemptAllocatedState(Container amContainer) {
 assertEquals(RMAppAttemptState.ALLOCATED, 
   applicationAttempt.getAppAttemptState());
 assertEquals(amContainer, applicationAttempt.getMasterContainer());
 // Check events
 verify(applicationMasterLauncher).handle(any(AMLauncherEvent.class));
 verify(scheduler, times(2)).allocate(any(ApplicationAttemptId.class),
   any(List.class), any(List.class), any(List.class), any(List.class), any(List.class),
   any(ContainerUpdates.class));
 verify(nmTokenManager).clearNodeSetForAttempt(
  applicationAttempt.getAppAttemptId());
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

DEFAULT_RM_CONTAINER_ALLOC_EXPIRY_INTERVAL_MS);
opCtx.updateAllocationParams(
  getScheduler().getMinimumResourceCapability(),
  getScheduler().getMaximumResourceCapability(),
  getScheduler().getMinimumResourceCapability(),
  tokenExpiryInterval);
appAttempt.setOpportunisticContainerContext(opCtx);

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

.newRecordInstance(RegisterApplicationMasterResponse.class);
response.setMaximumResourceCapability(rScheduler
  .getMaximumResourceCapability(app.getQueue()));
response.setApplicationACLs(app.getRMAppAttempt(applicationAttemptId)
  .getSubmissionContext().getAMContainerSpec().getApplicationACLs());
 .getSchedulingResourceTypes());

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

List<ApplicationAttemptId> appsInA1 = scheduler.getAppsInQueue("a1");
assertEquals(1, appsInA1.size());
List<ApplicationAttemptId> appsInA = scheduler.getAppsInQueue("a");
assertTrue(appsInA.contains(appAttemptId));
assertEquals(1, appsInA.size());
List<ApplicationAttemptId> appsInRoot = scheduler.getAppsInQueue("root");
assertTrue(appsInRoot.contains(appAttemptId));
assertEquals(1, appsInRoot.size());
 scheduler.killAllAppsInQueue("DOES_NOT_EXIST");
 Assert.fail();
} catch (YarnException e) {
appsInA1 = scheduler.getAppsInQueue("a1");
assertEquals(1, appsInA1.size());
appsInA = scheduler.getAppsInQueue("a");
assertTrue(appsInA.contains(appAttemptId));
assertEquals(1, appsInA.size());
appsInRoot = scheduler.getAppsInQueue("root");
assertTrue(appsInRoot.contains(appAttemptId));
assertEquals(1, appsInRoot.size());

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

List<NMContainerStatus> containerReports = new ArrayList<>();
containerReports.add(containerReport);
scheduler.handle(new NodeAddedSchedulerEvent(node1, containerReports));
RMContainer rmContainer = scheduler.getRMContainer(containerId);

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

queueInfo = appAttempt.scheduler.getQueueInfo(queue, false,
    false);
 } catch (IOException e) {
appAttempt.scheduler.allocate(
  appAttempt.applicationAttemptId,
  appAttempt.amReqs, null, EMPTY_CONTAINER_RELEASE_LIST,

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

applicationAttemptId);
response.setMaximumResourceCapability(getScheduler()
  .getMaximumResourceCapability(app.getQueue()));
response.setApplicationACLs(app.getRMAppAttempt(applicationAttemptId)
  .getSubmissionContext().getAMContainerSpec().getApplicationACLs());
  .getKeepContainersAcrossApplicationAttempts()) {
 List<Container> transferredContainers = getScheduler()
   .getTransferredContainers(applicationAttemptId);
 if (!transferredContainers.isEmpty()) {
  response.setContainersFromPreviousAttempts(transferredContainers);
  .getSchedulingResourceTypes());
response.setResourceTypes(ResourceUtils.getResourcesTypeInfo());
if (getRmContext().getYarnConfiguration().getBoolean(

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

@Override
public GetNewApplicationResponse getNewApplication(
  GetNewApplicationRequest request) throws YarnException {
 GetNewApplicationResponse response = recordFactory
   .newRecordInstance(GetNewApplicationResponse.class);
 response.setApplicationId(getNewApplicationId());
 // Pick up min/max resource from scheduler...
 response.setMaximumResourceCapability(scheduler
   .getMaximumResourceCapability());       
 
 return response;
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

RMContainer container = scheduler.getRMContainer(containerId);
if (container != null) {
 this.rmContext.getDispatcher().getEventHandler().handle(

内容来源于网络，如有侵权，请联系作者删除！

YarnScheduler

相关文章

    查看更多
    微信公众号

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com