在hadoop/cluster上提交作业后超时

xtupzzrd  于 2021-06-21  发布在  Flink
关注(0)|答案(2)|浏览(421)

我正在尝试升级我们的工作,从flink 1.4.2到1.7.1,但我不断遇到超时后提交的工作。flink作业运行在我们的hadoop集群(2.7版)上。
我见过以下行为:
使用与我们在1.4.2中使用的相同的flink-conf.yaml:1.5.6/1.6.3/1.7.1所有版本在1.4.2工作时超时。
将1.5.6与“mode:legacy”(关闭flip-6)一起使用是可行的
将1.7.1与“mode:legacy”一起使用会导致超时(我假设此选项已被删除,但文档已过时?https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#legacy)
当超时发生时,我得到以下堆栈跟踪:

INFO    class java.time.Instant does not contain a getter for field seconds
INFO    class com.bol.fin_hdp.cm1.domain.Cm1Transportable does not contain a getter for field globalId
INFO    Submitting job 5af931bcef395a78b5af2b97e92dcffe (detached: false).
INFO    ------------------------------------------------------------
INFO    The program finished with the following exception:
INFO    org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
INFO    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
INFO    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
INFO    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
INFO    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:798)
INFO    at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:289)
INFO    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:215)
INFO    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1035)
INFO    at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1111)
INFO    at java.security.AccessController.doPrivileged(Native Method)
INFO    at javax.security.auth.Subject.doAs(Subject.java:422)
INFO    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
INFO    at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
INFO    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1111)
INFO    Caused by: java.lang.RuntimeException: org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result.
INFO    at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:43)
INFO    at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJobWithConfig(IntervalJobStarter.java:32)
INFO    at com.bol.fin_hdp.Main.main(Main.java:8)
INFO    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO    at java.lang.reflect.Method.invoke(Method.java:498)
INFO    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
INFO    ... 12 more
INFO    Caused by: org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result.
INFO    at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:258)
INFO    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
INFO    at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
INFO    at com.bol.fin_hdp.cm1.job.Job.execute(Job.java:54)
INFO    at com.bol.fin_hdp.job.starter.IntervalJobStarter.startJob(IntervalJobStarter.java:41)
INFO    ... 19 more
INFO    Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
INFO    at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:371)
INFO    at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
INFO    at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
INFO    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
INFO    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
INFO    at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:216)
INFO    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
INFO    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
INFO    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
INFO    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
INFO    at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:301)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
INFO    at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:214)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
INFO    at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
INFO    at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
INFO    at java.lang.Thread.run(Thread.java:748)
INFO    Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
INFO    at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:213)
INFO    ... 17 more
INFO    Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: shd-hdp-b-slave-01...
INFO    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
INFO    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
INFO    at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943)
INFO    at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
INFO    ... 15 more
INFO    Caused by: org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: connection timed out: shd-hdp-b-slave-017.example.com/some.ip.address:46500
INFO    at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:212)
INFO    ... 7 more

flip-6中发生了哪些可能导致这种行为的更改?如何修复?

xghobddn

xghobddn1#

我们在flink 1.6上的工作,必须把 web.timeout 设置通过 -yD web.timeout=100000 .

vd8tlhqk

vd8tlhqk2#

在我们的例子中,提交作业的机器和hadoop集群之间有一个防火墙。
在较新的flink版本(1.7及更高版本)中,flink使用rest提交作业。此rest服务的端口号在Yarn设置中是随机的,无法设置。
flink 1.8.0引入了一个配置选项,使用以下命令将其设置为端口或端口范围:

rest.bind-port: 55520-55530

相关问题