我想从导入数据 Mysql
至 HDFS
与 Sqoop2
我的table看起来像
CREATE TABLE `campaign` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(128) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
我已经创建了2个链接
sqoop:000> show link
+----+-------+--------------+------------------------+---------+
| Id | Name | Connector Id | Connector Name | Enabled |
+----+-------+--------------+------------------------+---------+
| 2 | hdfs | 3 | hdfs-connector | true |
| 7 | mysql | 1 | generic-jdbc-connector | true |
+----+-------+--------------+------------------------+---------+
我创造了一份工作
Table SQL statement: SELECT id, name FROM campaign WHERE ${CONDITIONS}
我也试过这个sql语句-它也不起作用
Table SQL statement: SELECT cast(id as UNSIGNED INTEGER) id, name FROM campaign WHERE ${CONDITIONS}
我的工作看起来像:
sqoop:000> show job
+----+-----------------+----------------+--------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+-----------------+----------------+--------------+---------+
| 4 | campaign_import | 1 | 3 | true |
+----+-----------------+----------------+--------------+---------+
我已成功运行此作业
sqoop:000> start job -j 4
然后我在Map上发现了错误
原因:java.lang.classcastexception:java.math.biginteger无法转换为java.lang.long at
错误的全文看起来像
Error: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:99) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.lang.Long at org.apache.sqoop.connector.common.SqoopIDFUtils.toCSVFixedPoint(SqoopIDFUtils.java:133) at org.apache.sqoop.connector.common.SqoopIDFUtils.toCSV(SqoopIDFUtils.java:588) at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.toCSV(CSVIntermediateDataFormat.java:116) at org.apache.sqoop.connector.idf.CSVIntermediateDataFormat.setObjectData(CSVIntermediateDataFormat.java:87) at org.apache.sqoop.job.mr.SqoopMapper$SqoopMapDataWriter.writeArrayRecord(SqoopMapper.java:125) at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:96) at org.apache.sqoop.connector.jdbc.GenericJdbcExtractor.extract(GenericJdbcExtractor.java:38) at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:95) ... 7 more
这看起来像是 id
将(bigint)键入 campaign
表,但我不知道如何在不更改此字段类型的情况下修复它:)
也许有人知道?
1条答案
按热度按时间bq8i3lrv1#
当我将表sql语句改为
我看得更仔细了,看到了
1) 带表sql语句:
SELECT id, name FROM campaign WHERE ${CONDITIONS}
我发现错误:原因:java.lang.classcastexception:java.math.biginteger无法转换为java.lang.long
2) 带表sql语句:
SELECT cast(id as UNSIGNED INTEGER) id, name FROM campaign WHERE ${CONDITIONS}
我发现错误:原因:java.lang.classcastexception:java.lang.long不能转换为java.lang.integer
所以问题出在铸造上,我需要铸造
BIGINT
至SIGNED INTEGER