找到你要的答案

Q:Kafka have both partitioning and consumer group

Q:卡夫卡既有分区又有消费群

Can we use both partitioning and consumer group for same topic. So we want to create a topic which have partitioning and then create multiple consumers to it and out of them 2-3 will be generic listening to all messages(needs to be in a consumer group so that message in not processed multiple times) and then one consumer for specific partion.

Is partitioning and consumer are mutually exclusive?

我们可以同时使用分区和消费群为同一主题。所以我们要创建一个分区,然后创建多个消费者和他们2-3将通用听所有邮件主题(需要在一个消费群体,在多次处理消息)然后具体划分一个消费者。

分区和消费者是互斥的吗?

answer1: 回答1:

With the high level consumer API you can´t pin a consumer instance to a particular partition but there is nothing preventing you from having a set of consumers using the high level API and another set of consumers using the simple API for the same topic.

With this you could have a simple consumer consuming from a specific partition and a set of high level consumers in a consumer group consuming messages across all partitions.

与高层次消费的API可以´引脚一个消费者实例特定分区但没有什么能够阻止你有一套消费者使用高级别的API和另一组消费者使用简单的API相同的话题。

有了这一点,你可以有一个简单的消费从一个特定的分区和一组高层次的消费者在消费群体消费信息在所有分区。

answer2: 回答2:

High Level Consumer for kafka 0.8.x doesn't allow you to specify partition. It reads the data from all partitions and does complex failure detection and rebalancing. Probably it will be supported in future versions according to the API redesign - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design#ConsumerClientRe-Design-Allowmanualpartitionassignment.

If you need to read from specific topic and partition - use SimpleConsumer(it requires manual leader/offset/exclusivity/error handling).

Or you can use high level consumer and filter the data by partition(if you accept the overhead).

Also as an option you can redesign your topics to write data to separate topic instead of 'specific partition'.

卡夫卡0.8的高级用户不允许您指定分区。它读取所有分区的数据,并进行复杂的故障检测和再平衡。也许这将支持根据API设计- HTTPS在未来的版本:/ / cwiki。Apache。org /融合/显示/卡夫卡/消费+客户端+重新设计# consumerclientre设计allowmanualpartitionassignment。

如果你需要读取特定主题和分区使用simpleconsumer(需要手动的领导者/偏移/独家/错误处理)。

或者您可以使用高级用户和分区过滤数据(如果您接受开销)。

此外,作为一个选项,您可以重新设计您的主题写数据分离的主题,而不是“具体的分区”。

partitioning  apache-kafka  producer-consumer