我试图运行来自精通ApacheSpark2.x的书中的示例。
scala> val df = sc.parallelize(Array(1,2,3)).toDF
df: org.apache.spark.sql.DataFrame = [value: int]我是spark world的新手，但我想Dataframe应该保存到hdfs中
scala> df.write.json("hdfs://localhost:9000/tmp/account.json")
java.net.ConnectException: Call From miki/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused我和dfsadmin联系过了
hadoop dfsadmin -safemode enter
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
safemode: FileSystem file:/// is not an HDFS file systemjps输出
miki@miki:~$ jps
13798 Jps
10906 SparkSubmit如何解决这个问题？
1条答案
按热度按时间92dk7w1h1#
根据您的jps输出，您没有运行读写hdfs（namenode、datanode、resourcemanager）所需的hadoop守护进程。确保在机器上运行start yarn和start dfs，以使hdfs正常运行。