SQOOP从Oracle导入大表报错java.lang.OutOfMemoryError

ilovemk 2015-06-04 03:36:48
版本信息:
sqoop-1.4.5
hadoop-2.6.0

sqoop import --connect jdbc:oracle:thin:@10.222.55.28:1521:xxxx --username xxx --password xxx --table xxx -m 1 --warehouse-dir /usr/hive/warehouse --delete-target-dir --fields-terminated-by , --null-string '\\N' --null-non-string '\\N' --hive-import --create-hive-table --hive-overwrite

15/06/04 15:26:06 INFO manager.SqlManager: Using default fetchSize of 1000
15/06/04 15:26:06 INFO tool.CodeGenTool: Beginning code generation
15/06/04 15:26:07 INFO manager.OracleManager: Time zone has been set to GMT
15/06/04 15:26:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM XXX t WHERE 1=0
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:75)
at oracle.jdbc.driver.BufferCache.get(BufferCache.java:226)
at oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7672)
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:1005)
at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:277)
at oracle.jdbc.driver.T4CTTIdcb.receive(T4CTTIdcb.java:146)
at oracle.jdbc.driver.T4C8Oall.readDCB(T4C8Oall.java:844)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:358)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:750)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

这个错误是经典的JVM Heap Size不足导致的,但这里比较奇怪的是执行SQOOP一开始就报错了,并没有到执行Mapper或Reducer或Hive时报错,可能的原因是SQOOP的JVM Heap设置太小导致的,但是小弟不知道如何设置SQOOP的JVM Heap Size,还请各位帮忙看看,谢谢。
...全文
539 1 打赏 收藏 转发到动态 举报
写回复
用AI写文章
1 条回复
切换为时间正序
请发表友善的回复…
发表回复
Emory1949 2015-06-25
  • 打赏
  • 举报
回复
资源不足,你可以oracle表分批次入hive

20,807

社区成员

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区,致力于有Hadoop,hive,Spark,Hbase,Flink,ClickHouse,Kafka,数据仓库,大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧