无法在aws emr中运行oozie工作流

js5cn81o  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(409)

我们正在尝试实现一个poc,我们正在尝试在aws emr中运行oozie。由于安全原因,我不能发布工作流,但这是一个简单的例子,我们只有一个重命名操作,重命名文件名。其余的操作都是标准的,比如start、end、fatal error、error handler等等。同样的工作流在ec2示例上运行良好。但是当我们尝试在emr上运行oozie工作流时,我们得到了以下错误

2019-09-12 19:34:41,300  WARN ActionStartXCommand:523 - SERVER[<hostname>] USER[hadoop] GROUP[-] TOKEN[] APP[<WorkflowName>] JOB[0000006-190911195656052-oozie-oozi-W] ACTION[0000006-190911195656052-oozie-oozi-W@ErrorHandler] Error starting action [ErrorHandler]. ErrorType [ERROR], ErrorCode [EM007], Message [EM007: Encountered an error while sending the email message over SMTP.]
org.apache.oozie.action.ActionExecutorException: EM007: Encountered an error while sending the email message over SMTP.
        at org.apache.oozie.action.email.EmailActionExecutor.email(EmailActionExecutor.java:304)
        at org.apache.oozie.action.email.EmailActionExecutor.validateAndMail(EmailActionExecutor.java:173)
        at org.apache.oozie.action.email.EmailActionExecutor.start(EmailActionExecutor.java:112)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
        at org.apache.oozie.command.XCommand.call(XCommand.java:291)
        at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:459)
        at org.apache.oozie.command.wf.SignalXCommand.execute(SignalXCommand.java:82)
        at org.apache.oozie.command.XCommand.call(XCommand.java:291)
        at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:283)
        at org.apache.oozie.command.wf.ActionEndXCommand.execute(ActionEndXCommand.java:62)
        at org.apache.oozie.command.XCommand.call(XCommand.java:291)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:244)
        at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
        at org.apache.oozie.command.XCommand.call(XCommand.java:291)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.mail.MessagingException: Could not connect to SMTP host: <hostname>, port: 25;
  nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
        at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1961)
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:654)

当我们检查应用程序日志时,得到以下错误

Launcher AM execution failed
java.lang.UnsupportedOperationException: Not implemented by the S3FileSystem FileSystem implementation
        at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.oozie.action.hadoop.FSLauncherURIHandler.create(FSLauncherURIHandler.java:36)
        at org.apache.oozie.action.hadoop.PrepareActionsHandler.execute(PrepareActionsHandler.java:86)
        at org.apache.oozie.action.hadoop.PrepareActionsHandler.prepareAction(PrepareActionsHandler.java:73)
        at org.apache.oozie.action.hadoop.LauncherAM.executePrepare(LauncherAM.java:371)
        at org.apache.oozie.action.hadoop.LauncherAM.access$000(LauncherAM.java:55)
        at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:220)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
        at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Exception in thread "main" java.lang.UnsupportedOperationException: Not implemented by the S3FileSystem FileSystem implementation
        at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1060)
        at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile.java:1371)

hadoop磁盘tribution:amazon 2.8.5 奥齐version:oozie 5.1.0 emr版本:emr-5.26.0
感谢您的指导。

eqoofvh9

eqoofvh91#

在我们使用oozie的旧版本(即4.3)之后,问题得到了解决。没有其他更改。很好用。在aws的一个链接中看到,有些人无法用5.x版本执行oozie。一旦我们得到aws的具体回复,我们将更新答案。

相关问题