Spark Sql 报错:GeneratedIterator grows beyond 64 KB

Useful丶M 2018-07-23 10:58:22
程序运行现在这几天一直报这个错,一直搞不明白原因,还请不吝赐教...


23-07-2018 02:01:38 CST NsspAnaly ERROR - 18/07/23 02:01:38 ERROR CodeGenerator: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "agg_doAggregateWithKeys$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIterator;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator" grows beyond 64 KB

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 001 */ public Object generate(Object[] references) {

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 002 */   return new GeneratedIterator(references);

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 003 */ }

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 004 */

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 006 */   private Object[] references;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 007 */   private scala.collection.Iterator[] inputs;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 008 */   private boolean agg_initAgg;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 009 */   private boolean agg_bufIsNull;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 010 */   private long agg_bufValue;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 011 */   private boolean agg_bufIsNull1;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 012 */   private long agg_bufValue1;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 013 */   private boolean agg_bufIsNull2;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 014 */   private long agg_bufValue2;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 015 */   private boolean agg_bufIsNull3;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 016 */   private long agg_bufValue3;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 017 */   private boolean agg_bufIsNull4;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 018 */   private long agg_bufValue4;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 019 */   private boolean agg_bufIsNull5;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 020 */   private long agg_bufValue5;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 021 */   private boolean agg_bufIsNull6;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 022 */   private long agg_bufValue6;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 023 */   private boolean agg_bufIsNull7;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 024 */   private long agg_bufValue7;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 025 */   private boolean agg_bufIsNull8;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 026 */   private long agg_bufValue8;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 027 */   private boolean agg_bufIsNull9;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 028 */   private long agg_bufValue9;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 029 */   private boolean agg_bufIsNull10;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 030 */   private long agg_bufValue10;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 031 */   private boolean agg_bufIsNull11;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 032 */   private long agg_bufValue11;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 033 */   private boolean agg_bufIsNull12;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 034 */   private long agg_bufValue12;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 035 */   private boolean agg_bufIsNull13;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 036 */   private long agg_bufValue13;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 037 */   private boolean agg_bufIsNull14;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 038 */   private long agg_bufValue14;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 039 */   private boolean agg_bufIsNull15;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 040 */   private long agg_bufValue15;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 041 */   private boolean agg_bufIsNull16;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 042 */   private long agg_bufValue16;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 043 */   private boolean agg_bufIsNull17;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 044 */   private long agg_bufValue17;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 045 */   private boolean agg_bufIsNull18;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 046 */   private long agg_bufValue18;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 047 */   private boolean agg_bufIsNull19;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 048 */   private long agg_bufValue19;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 049 */   private boolean agg_bufIsNull20;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 050 */   private long agg_bufValue20;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 051 */   private boolean agg_bufIsNull21;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 052 */   private long agg_bufValue21;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 053 */   private boolean agg_bufIsNull22;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 054 */   private long agg_bufValue22;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 055 */   private boolean agg_bufIsNull23;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 056 */   private long agg_bufValue23;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 057 */   private boolean agg_bufIsNull24;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 058 */   private long agg_bufValue24;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 059 */   private boolean agg_bufIsNull25;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 060 */   private long agg_bufValue25;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 061 */   private boolean agg_bufIsNull26;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 062 */   private long agg_bufValue26;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 063 */   private boolean agg_bufIsNull27;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 064 */   private long agg_bufValue27;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 065 */   private boolean agg_bufIsNull28;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 066 */   private long agg_bufValue28;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 067 */   private boolean agg_bufIsNull29;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 068 */   private long agg_bufValue29;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 069 */   private boolean agg_bufIsNull30;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 070 */   private long agg_bufValue30;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 071 */   private boolean agg_bufIsNull31;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 072 */   private long agg_bufValue31;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 073 */   private boolean agg_bufIsNull32;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 074 */   private long agg_bufValue32;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 075 */   private boolean agg_bufIsNull33;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 076 */   private long agg_bufValue33;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 077 */   private boolean agg_bufIsNull34;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 078 */   private long agg_bufValue34;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 079 */   private boolean agg_bufIsNull35;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 080 */   private long agg_bufValue35;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 081 */   private boolean agg_bufIsNull36;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 082 */   private long agg_bufValue36;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 083 */   private boolean agg_bufIsNull37;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 084 */   private long agg_bufValue37;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 085 */   private boolean agg_bufIsNull38;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 086 */   private long agg_bufValue38;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 087 */   private boolean agg_bufIsNull39;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 088 */   private long agg_bufValue39;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 089 */   private boolean agg_bufIsNull40;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 090 */   private long agg_bufValue40;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 091 */   private boolean agg_bufIsNull41;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 092 */   private long agg_bufValue41;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 093 */   private boolean agg_bufIsNull42;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 094 */   private long agg_bufValue42;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 095 */   private boolean agg_bufIsNull43;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 096 */   private long agg_bufValue43;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 097 */   private boolean agg_bufIsNull44;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 098 */   private long agg_bufValue44;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 099 */   private boolean agg_bufIsNull45;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 100 */   private long agg_bufValue45;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 101 */   private boolean agg_bufIsNull46;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 102 */   private long agg_bufValue46;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 103 */   private boolean agg_bufIsNull47;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 104 */   private long agg_bufValue47;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 105 */   private boolean agg_bufIsNull48;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 106 */   private long agg_bufValue48;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 107 */   private boolean agg_bufIsNull49;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 108 */   private long agg_bufValue49;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 109 */   private boolean agg_bufIsNull50;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 110 */   private long agg_bufValue50;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 111 */   private boolean agg_bufIsNull51;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 112 */   private long agg_bufValue51;

23-07-2018 02:01:38 CST NsspAnaly ERROR - /* 113 */
...全文
1037 2 打赏 收藏 转发到动态 举报
写回复
用AI写文章
2 条回复
切换为时间正序
请发表友善的回复…
发表回复
qq_38599944 2020-07-17
  • 打赏
  • 举报
 回复
spark.conf.set("spark.sql.codegen.wholeStage",false)
javartisan 2019-09-19
  • 打赏
  • 举报
 回复
Spark优化生成的代码超过64KB 导致编译错误
Spark报错：grows beyond 64 KB
（作者：陈玓玏 data-master） spark脚本执行中，碰到错误： 2020-06-24 16:59:53,246 [Thread-44] ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of meth
sparksql执行报错grows beyond 64 KB
报错内容Code of method “apply_2(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClassSpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 K
sparksql Exception :GeneratedClass$SpecificUnsafeProjection 异常 2020-03-30 15:18:47,161 WARN TaskSetManager Lost task 13.0 in stage 3.0 (TID 35, hdp-dn12.cluster01.xxxxx, executor 2): java.util.concur...
GeneratedClass$GeneratedIteratorForCodegenStage1“ grows beyond 64 KB
如果列"industry"的值满足某个条件，则替换为指定的新值；在上述示例中，使用表达式CASE WHEN industry = 'Energy' THEN '能源' ELSE industry END将"industry"列中所有值为"Energy"的行替换为"能源"。出现这个错误的原因通常是在使用Spark SQL进行复杂计算时，生成的代码量太大，超过了Janino编译器的限制。这是由于数据量过大、复杂的逻辑或者有大量列引起的。这个错误是由于编译器生成的代码超出了Janino编译器的限制导致的。
spark报错处理
Spark报错处理 1、问题：org.apache.spark.SparkException: Exception thrown in awaitResult 分析：出现这个情况的原因是spark启动的时候设置的是hostname启动的，导致访问的时候DNS不能解析主机名导致。 问题解决： 第一种方法：确保URL是spark://服务器ip:7077，而不是spark://hostname...
Spark

1,261

社区成员

1,169

社区内容

发帖
与我相关
我的任务
社区描述
Spark由Scala写成，是UC Berkeley AMP lab所开源的类Hadoop MapReduce的通用的并行计算框架，Spark基于MapReduce算法实现的分布式计算。
社区管理员
  • Spark
  • shiter
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章