Я использую Kafka Server 0.9 с потребительской версией kafka-client 0.9 и kafka-maker 0.8.2.
Все отлично работает, за исключением того, что я получаю много информации о том, что координатор мертв на потребителе
2016-02-25 19:30:45.046 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.048 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.049 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.050 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.051 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.052 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.053 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.054 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.055 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.056 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.057 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.058 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.059 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.060 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.061 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.062 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.063 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.064 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.065 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.066 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.067 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.068 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.069 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.070 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.071 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.072 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.073 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.074 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.075 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.076 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.077 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.078 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.079 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.080 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.081 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.082 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.083 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.084 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.085 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.086 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.087 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.088 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.089 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.090 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.091 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.093 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-02-25 19:30:45.094 INFO 10263 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
Я также заметил, что производитель отключает связь каждые 10 минут, как показано ниже
2016-03-12 15:55:36 INFO [pool-1-thread-1] - Fetching metadata from broker id:0,host:192.168.72.30,port:9092 with correlation id 41675 for 1 topic(s) Set(act)
2016-03-12 15:55:36 INFO [pool-1-thread-1] - Connected to 192.168.72.30:9092 for producing
2016-03-12 15:55:36 INFO [pool-1-thread-1] - Disconnecting from 192.168.72.30:9092
2016-03-12 15:55:36 INFO [pool-1-thread-1] - Disconnecting from kafkauk.XXXXXXXXXX.co:9092
2016-03-12 15:55:36 INFO [pool-1-thread-1] - Connected to kafkauk.XXXXXXXXXX.co:9092 for producing
это моя конфигурация производителя
metadata.broker.list=192.168.72.30:9092
serializer.class=kafka.serializer.StringEncoder
request.required.acks=1
linger.ms=2000
batch.size=500
и потребительская конфигурация
bootstrap.servers: kafkauk.xxxxxxxx.co:9092
group.id: cdrServer
client.id: cdrServer
enable.auto.commit: true
auto.commit.interval.ms: 1000
session.timeout.ms: 30000
key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
Я не мог понять, что это значит, и я должен пренебрегать ими, или я что-то упустил в конфигурации
После того, как я изменил kafka на уровень отладки у потребителя, я нашел ниже
2016-03-13 18:21:55.586 DEBUG 5469 --- [ cdrServer] org.apache.kafka.clients.NetworkClient : Node 2147483647 disconnected.
2016-03-13 18:21:55.586 INFO 5469 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Marking the coordinator 2147483647 dead.
2016-03-13 18:21:55.586 DEBUG 5469 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Issuing group metadata request to broker 0
2016-03-13 18:21:55.586 DEBUG 5469 --- [ cdrServer] org.apache.kafka.clients.NetworkClient : Sending metadata request ClientRequest(expectResponse=true
, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=183025,client_id=cdrServer}, body={topics=[act]}), isInitiatedByNetworkCli
ent, createdTimeMs=1457893315586, sendTimeMs=0) to node 0
2016-03-13 18:21:55.591 DEBUG 5469 --- [ cdrServer] org.apache.kafka.clients.Metadata : Updated cluster metadata version 296 to Cluster(nodes = [N
ode(0, kafkauk.xxxxxxxxx.co, 9092)], partitions = [Partition(topic = act, partition = 0, leader = 0, replicas = [0,], isr = [0,]])
2016-03-13 18:21:55.592 DEBUG 5469 --- [ cdrServer] o.a.k.c.c.internals.AbstractCoordinator : Group metadata response ClientResponse(receivedTimeMs=1457
893315592, disconnected=false, request=ClientRequest(expectResponse=true, callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFu
[email protected], request=RequestSend(header={api_key=10,api_version=0,correlation_id=183024,client_id=cdrServer}, body={group_id=cdrServer}), c
reatedTimeMs=1457893315586, sendTimeMs=1457893315586), responseBody={error_code=0,coordinator={node_id=0,host=kafkauk.xxxxxxxx.co,port=9092}})
Я не уверен, что это сетевая проблема, потому что это происходит каждые 9 минут точно
Обновить
Я обнаружил, что это напрямую связано с
connections.max.idle.ms: 300000
Что бы я ни сказал, тогда я отключусь от этого значения