Spark塞插入数据中断

eagi6jfj  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(523)

我正在努力学习Spark和Hive。
上下文:我的配置单元数据存储中存在一个表。我想把它的内容复制到另一个表中。我正在使用sparksql从同一个查询中触发一个简单的insert。
执行上述操作失败,thriftserver日志中跟踪到以下异常-

[ERROR] 2020-08-03 12:14:34,201 org.apache.spark.sql.execution.datasources.FileFormatWriter - Aborting job a9619c2d-5786-4e66-a21d-59a749876bae.
        java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:206)
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
        at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:243)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:729)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:167)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:109)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:107)
        at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:120)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195)
        at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
        at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:195)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:80)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
        ...

[WARN] 2020-08-03 12:14:34,201 org.apache.thrift.transport.TIOStreamTransport - Error closing output stream.
        java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        ...

[ERROR] 2020-08-03 12:14:34,215 org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation - Error executing query, currentState CLOSED, 
        org.apache.spark.SparkException: Job aborted.
            at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
            at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
          ...
        Caused by: java.lang.InterruptedException
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)

有人能解释一下这里会发生什么吗?
如果你需要更多的信息,请告诉我。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题