使用hue的cdh5中的pig作业问题

9ceoxa92  于 2021-06-21  发布在  Pig
关注(0)|答案(1)|浏览(339)

我在cdh5.1中使用hue3.6。我面临着Pig读Hive表的问题。我认为必须有一些配置需要做的hcatalog。我使用cm安装cloudera。有人能指出文件配置色调与Pig工作。以下是错误堆栈:

2014-09-09 10:55:58,010 [main] INFO  org.apache.pig.Main  - Apache Pig version 0.12.0-cdh5.1.0 (rexported) compiled Jul 12 2014, 08:41:26
2014-09-09 10:55:58,012 [main] INFO  org.apache.pig.Main  - Logging error messages to: /yarn/nm/usercache/cloudera/appcache/application_1410276487009_0001/container_1410276487009_0001_01_...
2014-09-09 10:55:58,067 [main] INFO  org.apache.pig.impl.util.Utils  - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
2014-09-09 10:55:58,189 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-09-09 10:55:58,189 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-09-09 10:55:58,189 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  - Connecting to hadoop file system at: hdfs://txwlcloud2:8020
2014-09-09 10:55:58,197 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  - Connecting to map-reduce job tracker at: txwlcloud2:8032
2014-09-09 10:55:59,013 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
2014-09-09 10:55:59,013 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
2014-09-09 10:55:59,013 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
2014-09-09 10:55:59,014 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
2014-09-09 10:55:59,014 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
2014-09-09 10:55:59,014 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-09-09 10:55:59,014 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2014-09-09 10:55:59,276 [main] INFO  org.apache.hadoop.hive.metastore.HiveMetaStore  - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2014-09-09 10:55:59,327 [main] INFO  org.apache.hadoop.hive.metastore.ObjectStore  - ObjectStore, initialize called
2014-09-09 10:55:59,692 [main] INFO  DataNucleus.Persistence  - Property datanucleus.cache.level2 unknown - will be ignored
2014-09-09 10:56:00,004 [main] ERROR org.apache.pig.PigServer  - exception during parsing: Error during parsing. Cannot get schema from loadFunc org.apache.hcatalog.pig.HCatLoader
Failed to parse: Can not retrieve schema from loader org.apache.hcatalog.pig.HCatLoader@d47f419
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:198)
    at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1676)
    at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1409)
    at org.apache.pig.PigServer.parseAndBuild(PigServer.java:342)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:367)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:769)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:478)
    at org.apache.pig.PigRunner.run(PigRunner.java:49)
    at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:287)
    at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:227)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
    at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: Can not retrieve schema from loader org.apache.hcatalog.pig.HCatLoader@d47f419
    at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:91)
    at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:853)
    at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
    at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
    at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
    at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
    at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
    ... 30 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245: Cannot get schema from loadFunc org.apache.hcatalog.pig.HCatLoader
    at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:179)
    at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
    ... 37 more
Caused by: java.io.IOException: java.lang.Exception: Could not instantiate a HiveMetaStoreClient connecting to server uri:[null]
    at org.apache.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:191)
    at org.apache.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:194)
    at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
    ... 38 more
Caused by: java.lang.Exception: Could not instantiate a HiveMetaStoreClient connecting to server uri:[null]
    at org.apache.hcatalog.pig.PigHCatUtil.getHiveMetaClient(PigHCatUtil.java:152)
    at org.apache.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:186)
    ... 40 more
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
    at org.apache.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:167)
    at org.apache.hcatalog.common.HiveClientCache.get(HiveClientCache.java:143)
    at org.apache.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:548)
    at org.apache.hcatalog.pig.PigHCatUtil.getHiveMetaClient(PigHCatUtil.java:150)
    ... 41 more
rfbsl7qr

rfbsl7qr1#

你需要加上 hive-site.xml 作为“属性”->“资源”下的文件,告诉pig要使用哪个元存储。
来源
http://blog.cloudera.com/blog/2013/08/demo-using-hue-to-access-hive-data-through-pig/ 以及
http://gethue.com/hadoop-tutorial-how-to-access-hive-in-pig-with/

相关问题