<>Kafka common problem
<> abstract
This paper records kafka-0.8.2.1,Kafka-1.1.1,Kafka-2.x Common problems and solutions of version .
<>1 Kafka-0.8.2.1
<>1.1 Consumer report errors
* Problem description ： WARN [
console-consumer-23149_hadoop2-1476872289945-6edaec5e-leader-finder-thread],
Failed to add leaderfor partitions [filelog_v3_json_test_1019,13],....; will
retry(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)WARN [
console-consumer-23149_hadoop2-1476872289945-6edaec5e-leader-finder-thread],
Failed to add leaderfor partitions [filelog_v3_json_test_1019,13],....; will
retry(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
java.nio.channels.ClosedChannelException[2016-10-19 18:18:11,801] WARN Fetching
topic metadata with correlationid 4 for topics [Set(filelog_v3_json_test_1019)]
from broker[id:0,host:es-gangw,port:9092] failed (kafka.client.ClientUtils$)
* Solution ：
inspect server.properties file Of host.name field , Is the configuration effective ?（ Pay attention to the front # Number , It takes effect only if it is removed ）
<>1.2 First push topic Error in data , It's normal to push again
* Problem description ： bin/kafka-console-producer.sh --broker-list
10.1.17.51:9092,10.1.17.52:9092,10.1.17.53:9092 --topic chengc_1104 222[
2016-11-04 10:45:43,274] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,291] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,293] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadatafor topic: chengc_1104 (
kafka.producer.async.DefaultEventHandler) [2016-11-04 10:45:43,402] WARN Error
while fetching metadata [{TopicMetadata for topic chengc_1104 -> No partition
metadatafor topic chengc_1104 due to kafka.common.LeaderNotAvailableException}]
for topic [chengc_1104]: class kafka.common.LeaderNotAvailableException (
kafka.producer.BrokerPartitionInfo) [2016-11-04 10:45:43,407] WARN Error while
fetching metadata[{TopicMetadata for topic chengc_1104 -> No partition metadata
for topic chengc_1104 due to kafka.common.LeaderNotAvailableException}] for
topic[chengc_1104]: class kafka.common.LeaderNotAvailableException (
kafka.producer.BrokerPartitionInfo) [2016-11-04 10:45:43,407] ERROR Failed to
collate messages by topic, partition due to: Failed to fetch topic metadatafor
topic: chengc_1104(kafka.producer.async.DefaultEventHandler) [2016-11-04
10:45:43,521] WARN Error while fetching metadata [{TopicMetadata for topic
chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,527] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,527] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadatafor topic: chengc_1104 (
kafka.producer.async.DefaultEventHandler)
* Solution ：
You need to create it manually first topic Push data again ,0.8.2.1 This is a problem
<>1.3 because Kafka auto rebalance , Causes the disk IO Hit full , Network traffic speed limit exceeds disk IO
*
Problem description
Kafka In both cases, the number of copies will be true or partition unbalanced , This leads to auto rebalance：
1.Kafka Replica synchronization timeout
2. There is a node hang point or restart
*
Solution
In this case, the shutdown can be considered rebalance : auto.leader.rebalance.enable
, Then perform the command line manual balance, You can choose your own time , Otherwise, it may lead to a very busy time rebalance Cause the load to be too high .
*
About manual self motion rebalance explain ：
Whenever one broker Close or leader When the mechanism fails , On top of it topic Of partition Copies are transferred to other nodes . This means that , By default, when broker It will only be used as the partition Copy of follower existence , That is to say, client read and write requests will not be processed .
To avoid this ,Kafka There is a priority copy (preferred replicas
) The concept of . for instance , Some partition A copy of exists in 1,5,9 node , here 1 Node No leader Preferred copy of , Because he joined the replica list earlier .
You can make Kafka The cluster attempts to restore the leader of the restored replica by running the following command ：
bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot
<>1.4 topic Of leader All for none
* Solution
Number of copies checked factor Is it greater than the number of nodes , For example, only a single node is set factor by 2. This time it was changed to 1 Just fine
<>2 Kafka-1.1.1
<>2.1 kafka-connect
Launch official example :
./bin/connect-standalone.sh config/connect-standalone.properties
config/connect-file-source.properties config/connect-file-sink.properties&
The error report is as follows ：
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.
collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
This problem has been investigated for a long time , For a long time, there was no clear solution , Now let me talk about my solution .
First of all, the reason for this problem is because OS It's already configured classpath Lower version included (18.0) Of guava package , It's not what we expected $KAFKA_HOME/libs/guava-20.0.jar, So it leads to an error .
see connect-standalone.sh It turns out that he's calling kafka-run-class.sh, And it's used in this CLASSPATH
, So we add a line directly to the front ：CLASSPATH="/home/kafka/kafka_2.11-1.1.1/libs/*".
Restart connect-standalone.sh, It's all right .
Another idea is to use what we're going to use Kafka Of lib Path added to OS Of classpath in , Pay attention to the front , Priority loading .
<>2.2 Timeout of 60000ms expired before the position for partition xxx_topic
could be determined
This problem is generally due to Kafka One of the clusters Broker Node load is too high , Unable to respond in time producer The request to send data results in .
<>3 Kafka-2.x
<>2.1 Source code gradle initialization Could not find method scala
I started from github
clone Is the latest code , Version is 2.7, then gradle（ The version is up to date 6.5） Keep reporting mistakes , Say there's a what Configuration Class not found .
Then the gradle Demote to 5.6.4, The error was solved .
But there are new mistakes ：
Could not find method scala() for arguments [
build_2ofsm9wzcvhxxkvqblsz8ogy5$_run_closure5$_closure73$_closure105@729633ff]
on object of type org.gradle.api.plugins.scala.ScalaPlugin.
There's no answer on the Internet , Finally, I tried to demote kafka edition , Specifically, it is to branch from 2.7 And for the sake of 2.3, There was no mistake ...
Technology