Failed to send requests for topics mykafka1 with correlation ids in [0,20]

weixin_39630690 2018-03-27 03:31:09
最近写了个kafka_producer测试程序,可是每当程序运行到一定时间后就报错,重新创建新topic也不行:

Fetching topic metadata with correlation id 15 for topics [Set(mykafka1)] from broker [id:1,host:192.168.231.12,port:9092] fail
ERROR DefaultEventHandler: Failed to send requests for topics mykafka1 with correlation ids in [0,20]

producer:
object MyProducer {

def getProducerConfig(brokerAddr: String): Properties = {
val props = new Properties()
props.put("metadata.broker.list", brokerAddr)
props.put("serializer.class", classOf[ItelogEncoder[Person]].getName)
props.put("key.serializer.class", classOf[StringEncoder].getName)
// props.put("replica.fetch.max.bytes", "" + 1024 * 1024 * 6)
// props.put("message.max.bytes", "" + 1024 * 1024 * 4)
props
}

def sendMessages(topic: String, messages: List[Person], brokerAddr: String) = {
val producer = new Producer[String, Person](new ProducerConfig(getProducerConfig(brokerAddr)))
producer.send(messages.map {
new KeyedMessage[String, Person](topic, "Iteblog", _)
}: _*)

producer.close()
}

def main(args: Array[String]) {

val Array(brokerAddr, topic) = args

val sparkConf = new SparkConf().setAppName(this.getClass.getSimpleName).setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(2))

for (i <- 0.to(10000)) {
val data = List(Person("xiaoming" + i, 23), Person("xiaohua" + i, 24), Person("xiaoliu" + i, 26))
sendMessages(topic, data, brokerAddr)

}
}
case class Person(var name: String, var age: Int)


求大神们帮帮忙,小白在此谢过
...全文
1483 2 打赏 收藏 转发到动态 举报
写回复
用AI写文章
2 条回复
切换为时间正序
请发表友善的回复…
发表回复
weixin_39630690 2018-03-30
  • 打赏
  • 举报
回复
val producer = new Producer[String, Person](new ProducerConfig(props)) for (j <- 0.to(10000)) { producer.send(data.map { new KeyedMessage[String, Person](topic, "Iteblog", _) }: _*) producer.close() } 如果将新建对象写到for循环外,就会报错: Exception in thread "main" kafka.producer.ProducerClosedException: producer already closed
peak-c 2018-03-29
  • 打赏
  • 举报
回复
刚好也犯了同样的错误,而且时间也最近几天,希望对你有用; 问题部分在于这段代码:(每次发送消息都创建新实例) def sendMessages(topic: String, messages: List[Person], brokerAddr: String) = { val producer = new Producer[String, Person](new ProducerConfig(getProducerConfig(brokerAddr))) producer.send(messages.map { new KeyedMessage[String, Person](topic, "Iteblog", _) }

1,258

社区成员

发帖
与我相关
我的任务
社区描述
Spark由Scala写成,是UC Berkeley AMP lab所开源的类Hadoop MapReduce的通用的并行计算框架,Spark基于MapReduce算法实现的分布式计算。
社区管理员
  • Spark
  • shiter
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧