我正在Pyspark中运行一个小脚本，从hbase表中提取一些数据并创建一个Pyspark Dataframe 。我试图将 Dataframe 保存回本地hdfs，遇到了一个 *exit 50错误 *。
我能够对相对较小的 Dataframe 成功地执行相同的操作，但不能对大文件执行。我很乐意分享任何代码片段，并将感谢任何帮助。此外，SparkUI的整个环境可以作为屏幕截图共享。
这是我的Spark（2.0.0）属性的配置（这里显示为字典）。部署在yarn-client上。

configuration={'spark.executor.memory':'4g',
        'spark.executor.instances':'32',
        'spark.driver.memory':'12g',
        'spark.yarn.queue':'default' 
       }

在我获得 Dataframe 后，我试图将其保存为：

df.write.save('user//hdfs//test_df',format = 'com.databricks.spark.csv',mode = 'append')

以下错误块不断重复，直到作业失败。我相信这可能是一个OOM错误，但我已经尝试过给多达128个执行器，每个执行器有16 GB内存，但无济于事。任何变通方案将不胜感激。

Container exited with a non-zero exit code 50

17/09/25 15:19:35 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 64, fslhdppdata2611.imfs.micron.com): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container marked as failed: container_e37_1502313369058_6420779_01_000043 on host: fslhdppdata2611.imfs.micron.com. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_e37_1502313369058_6420779_01_000043
Exit code: 50
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
    at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Shell output: main : command provided 1
main : run as user is hdfsprod
main : requested yarn user is hdfsprod
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /opt/hadoop/data/03/hadoop/yarn/local/nmPrivate/application_1502313369058_6420779/container_e37_1502313369058_6420779_01_000043/container_e37_1502313369058_6420779_01_000043.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...