spark cluster和cassandra的janusgraph的设置和配置

7qhs6swi  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(472)

我在一台机器上运行janusgraph(0.1.0)和spark(1.6.1)。我按照这里的描述进行了配置。当使用sparkgraphcomputer访问gremlin控制台上的图形时,它总是空的。我在日志文件中找不到任何错误,它只是一个空图。
是否有人将janusgraph与spark一起使用,并可以共享其配置和属性?
使用janusgraph,我得到了预期的输出:

gremlin> graph=JanusGraphFactory.open('conf/test.properties')
==>standardjanusgraph[cassandrathrift:[127.0.0.1]]
gremlin> g=graph.traversal()
==>graphtraversalsource[standardjanusgraph[cassandrathrift:[127.0.0.1]], standard]
gremlin> g.V().count()
14:26:10 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
==>1000001
gremlin>

使用带有spark的hadoopgraph作为graphcomputer,图形为空:

gremlin> graph=GraphFactory.open('conf/test.properties')
==>hadoopgraph[cassandrainputformat->gryooutputformat]
gremlin> g=graph.traversal().withComputer(SparkGraphComputer)
==>graphtraversalsource[hadoopgraph[cassandrainputformat->gryooutputformat], sparkgraphcomputer]
gremlin> g.V().count()
            ==>0==============================================>   (14 + 1) / 15]

我的conf/test.properties:


# 

# Hadoop Graph Configuration

# 

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphInputFormat=org.janusgraph.hadoop.formats.cassandra.CassandraInputFormat
gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat
gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
gremlin.hadoop.memoryOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat

gremlin.hadoop.deriveMemory=false
gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output

# 

# Titan Cassandra InputFormat configuration

# 

janusgraphmr.ioformat.conf.storage.backend=cassandrathrift
janusgraphmr.ioformat.conf.storage.hostname=127.0.0.1
janusgraphmr.ioformat.conf.storage.keyspace=janusgraph
storage.backend=cassandrathrift
storage.hostname=127.0.0.1
storage.keyspace=janusgraph

# 

# Apache Cassandra InputFormat configuration

# 

cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner
cassandra.input.keyspace=janusgraph
cassandra.input.predicate=0c00020b0001000000000b000200000000020003000800047fffffff0000
cassandra.input.columnfamily=edgestore
cassandra.range.batch.size=2147483647

# 

# SparkGraphComputer Configuration

# 

spark.master=spark://127.0.0.1:7077
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.executor.memory=100g

gremlin.spark.persistContext=true
gremlin.hadoop.defaultGraphComputer=org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer

hdfs的配置似乎正确,如下所述:

gremlin> hdfs
==>storage[DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_178390072_1, ugi=cassandra (auth:SIMPLE)]]]
ecr0jaav

ecr0jaav1#

尝试修复这些属性:

janusgraphmr.ioformat.conf.storage.keyspace=janusgraph
storage.keyspace=janusgraph

替换为:

janusgraphmr.ioformat.conf.storage.cassandra.keyspace=janusgraph
storage.cassandra.keyspace=janusgraph

默认的键空间名称是 janusgraph ,因此,尽管属性名称上有错误,但我不认为您会注意到这个问题,除非您使用不同的键空间名称加载数据。
配置参考中描述了后一个属性。另外,请关注这个开放性问题,以改进hadoop图形使用的文档。

相关问题