SQOOP从Oracle导入大表报错java.lang.OutOfMemoryError

ilovemk 2015-06-04 03:36:48

版本信息：
sqoop-1.4.5
hadoop-2.6.0

sqoop import --connect jdbc:oracle:thin:@10.222.55.28:1521:xxxx --username xxx --password xxx --table xxx -m 1 --warehouse-dir /usr/hive/warehouse --delete-target-dir --fields-terminated-by , --null-string '\\N' --null-non-string '\\N' --hive-import --create-hive-table --hive-overwrite

15/06/04 15:26:06 INFO manager.SqlManager: Using default fetchSize of 1000
15/06/04 15:26:06 INFO tool.CodeGenTool: Beginning code generation
15/06/04 15:26:07 INFO manager.OracleManager: Time zone has been set to GMT
15/06/04 15:26:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM XXX t WHERE 1=0
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:75)
at oracle.jdbc.driver.BufferCache.get(BufferCache.java:226)
at oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7672)
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:1005)
at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:277)
at oracle.jdbc.driver.T4CTTIdcb.receive(T4CTTIdcb.java:146)
at oracle.jdbc.driver.T4C8Oall.readDCB(T4C8Oall.java:844)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:358)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:750)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)

这个错误是经典的JVM Heap Size不足导致的，但这里比较奇怪的是执行SQOOP一开始就报错了，并没有到执行Mapper或Reducer或Hive时报错，可能的原因是SQOOP的JVM Heap设置太小导致的，但是小弟不知道如何设置SQOOP的JVM Heap Size，还请各位帮忙看看，谢谢。

...全文