Structured Streaming + kafka碰上@Resource注入遇到问题

Hong4741 2019-01-23 04:48:37
最近在用Structured Streaming + kafka开发,初始化显示Resource注解的的beCalTask对象正常,当kafka中有数开始消费的时候beCalTask又提示为null, window单机部署的时候没问题,在服务器上集群部署的时候会出现,请教一下各位如何解决?

@Repository
public class RapidCalculateTask implements Serializable{
private static Logger log = LoggerFactory.getLogger(RapidCalculateTask.class);
/**
*
*/
private static final long serialVersionUID = -7664074371811203339L;

@Resource
BeCalculateTaskDao beCalTask;

public void startStruStram() {
try {
CalculateService calService = new CalculateService();

Properties prop = calService.getProperties();
String servers_kafka = prop.getProperty("spark.servers_kf");
String subscribe = prop.getProperty("spark.subscribe");

SparkConf conf = calService.getSparkConfig(prop);
conf.setJars(SparkContext.jarOfClass( this.getClass() ).toList());
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();

Dataset<Row> lines = spark.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", servers_kafka)
.option("subscribe", subscribe)
.load().selectExpr("CAST(value AS STRING)");


StreamingQuery query = lines.writeStream()
.foreach(new RapidForeachWriter<Row>(beCalTask))
.start();

query.awaitTermination();
} catch (StreamingQueryException e) {
e.printStackTrace();
}
}
}


public class RapidForeachWriter<Rows> extends ForeachWriter<Rows> implements Serializable{
/**
*
*/
private static final long serialVersionUID = 5549303376220035143L;
private static Logger log = LoggerFactory.getLogger(RapidForeachWriter.class);


private BeCalculateTaskDao beCalTask;
private JavaSparkContext javaCont;



public RapidForeachWriter (BeCalculateTaskDao beCalTask){
log.info("RapidForeachWriter 检查 BeCalculateTaskDao 是否为空"+beCalTask);
this.beCalTask = beCalTask;
}

@Override
public void close(Throwable errorOrNull) {
// TODO Auto-generated method stub
}

@Override
public boolean open(long partitionId, long version) {
log.info("检查 beCalTask 是否为空-[partitionId:"+partitionId+";version:"+version+"]"+beCalTask);
try {
if(beCalTask.equals(null)) {
return false;
}
return true;
} catch (Exception e) {
return false;
}
}

@Override
public void process(Rows value){

}
}
...全文
58 回复 打赏 收藏 转发到动态 举报
写回复
用AI写文章
回复
切换为时间正序
请发表友善的回复…
发表回复

1,258

社区成员

发帖
与我相关
我的任务
社区描述
Spark由Scala写成,是UC Berkeley AMP lab所开源的类Hadoop MapReduce的通用的并行计算框架,Spark基于MapReduce算法实现的分布式计算。
社区管理员
  • Spark
  • shiter
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧