KafkaConsumer报错

2016-07-13 02:34:17 448 阅读 AI 写同款 · GEO 优化AI 选题数据监测一键分发 ›

Flink消费Kafka出现异常：KafkaConsumer.assign方法报错 Flink消费Kafka出现异常：KafkaConsumer.assign方法报错大数据领域中，Apache Flink是一个流处理框架，而Apache Kafka则是一个分布式消息队列。在使用Flink来消费Kafka的过程中，有时会遇到KafkaConsumer.assign方法报错的情况。本文将探讨该问题的原因，并给出相应的解决方案。阅读详情

执行之后错误如下：
Exception in thread "Thread-0" org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'brokers': Error reading field 'host': Error reading string of length 28271, only 593 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:973)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
at KafkaNew.Consumer$ConsumerThread.run(Consumer.java:40)

代码下面贴出

...全文

kafka spark Structured streaming整合后集群报错KafkaConsumer.subscribe(Ljava/util/Collection;)V 简介整个项目架构是在CDH中，，然后spark Structured streaming消费kafka。 spark 2.3版本 kafka0.10版本  <dependency> <groupId>org.apache.spark</groupId> ... 阅读详情

曼森的玛丽莲

Java EE 67,535 成员 225,843 内容

+ 关注 + 加入社区

切换为时间正序

曼森的玛丽莲 2016-07-13

打赏
举报

小弟对新的API不太熟，请各路大神帮忙看看咋回事，搞了好久了

曼森的玛丽莲 2016-07-13

打赏
举报

import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.common.errors.WakeupException; import java.util.Arrays; import java.util.Properties; import java.util.Scanner; public class Consumer { private static Scanner in; public static void main(String[] argv)throws Exception{ String topicName = xxx String groupId = xxx ConsumerThread consumerRunnable = new ConsumerThread(topicName,groupId); consumerRunnable.start(); } private static class ConsumerThread extends Thread{ private String topicName; private String groupId; private KafkaConsumer<String,String> kafkaConsumer; public ConsumerThread(String topicName, String groupId){ this.topicName = topicName; this.groupId = groupId; } public void run() { Properties configProperties = new Properties(); configProperties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); configProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArrayDeserializer"); configProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); configProperties.put(ConsumerConfig.GROUP_ID_CONFIG, groupId); configProperties.put(ConsumerConfig.CLIENT_ID_CONFIG, "simple"); kafkaConsumer = new KafkaConsumer<String, String>(configProperties); kafkaConsumer.subscribe(Arrays.asList(topicName)); while (true) { ConsumerRecords<String, String> records = kafkaConsumer.poll(100); //poll这一步报错 for (ConsumerRecord<String, String> record : records) System.out.println(record.value()); } public KafkaConsumer<String,String> getKafkaConsumer(){ return this.kafkaConsumer; } } }

KafkaConsumer报错

相关推荐