我是hadoop/giraph和java的新手。作为任务的一部分,我下载了cloudera快速启动vm和giraph。我用这本书命名为“实用图形分析与Apache吉拉夫;作者:shaposhnik,roman,martella,claudio,logothetis,dionysios“我试着在第111页(twitter followership图)上运行第一个例子。
在群集2.6.0-mr1-cdh5.12.0上运行hadoop版本的pom.xml文件时,请查找以下错误
`[cloudera@quickstart first]$ mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building book-examples 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ book-examples ---
[INFO] Deleting /home/cloudera/workspace/first/target
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ book-examples ---
[debug] execute contextualize
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ book-examples ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/cloudera/workspace/first/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[5,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[6,27] error: package org.apache.hadoop.io does not exist
[ERROR] /home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[7,29] error: cannot find symbol
[ERROR] package org.apache.hadoop.util
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,17] error: cannot find symbol
[ERROR] class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[14,30] error: cannot find symbol
[ERROR] class IntWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,0] error: cannot find symbol
[ERROR] class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[15,14] error: cannot find symbol
[ERROR] class NullWritable
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[17,28] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,3] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[18,16] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[19,12] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,12] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[24,25] error: cannot find symbol
[ERROR] class GiraphHelloWorld
/home/cloudera/workspace/first/src/main/java/GiraphHelloWorld.java:[34,14] error: cannot find symbol
[INFO] 14 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.495s
[INFO] Finished at: Fri Dec 08 14:57:01 PST 2017
[INFO] Final Memory: 18M/57M
`
我根据堆栈溢出响应添加了cloudera存储库。请查找上述错误适用的更新pom xml:
`<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>giraph</groupId>
<artifactId>book-examples</artifactId>
<version>1.0.0</version>
<dependencies>
<dependency>
<groupId>org.apache.giraph</groupId>
<artifactId>giraph-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.6.0-mr1-cdh5.12.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>create-jar-bundle</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</project>`
书中的hadoop版本是1.2.1。两个版本之间的图书依赖性存在问题。
如果有人能帮助我理解如何处理这个错误就太好了。
提前谢谢。
1条答案
按热度按时间uqxowvwt1#
这个
pom.xml
在你的书里,这本书已经过时了。用这个代替。来源:github上的图书示例库。编辑:
您要使用的是最新版本的
hadoop-core
,但maven central repository(默认respository)提供的最新版本是1.2.1。您需要使用cloudera存储库来获取库的最新版本。为此,只需将存储库添加到pom.xml
:现在您应该看到,maven试图首先在cloudera找到jar,然后返回到中心:
编辑2:
好吧,我终于明白了。从版本2开始,hadoop改变了它的打包方式,而不是声明对
hadoop-core
,您应该使用hadoop-client
它是一个元包,为您聚合了所有必需的依赖项。拆下hadoop-core
依赖于你的pom.xml
并添加: