关于 kafka 日志清理策略的问题

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

这是一个创建于 2147 天前的主题，其中的信息可能已经有所发展或是发生改变。

现象：搭建了一个 kafka 服务, 使用 kafka-python 包正常生产数据, 但是 kafka 过五分钟就把我的 topic 删除掉. 但是配置 log 的已经配置了, 我认为 kafka 不应该删除我的 topic 历史数据.

关于 log 清理的配置文件:

############################# Log Flush Policy #############################

log.flush.interval.messages=10000

log.flush.interval.ms=1000

############################# Log Retention Policy #############################

log.retention.hours=168

log.retention.bytes=-1

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

打印的 log:

[2019-11-26 10:19:11,478] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log) [2019-11-26 10:19:11,485] INFO [ProducerStateManager partition=falcon_source_created-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager) [2019-11-26 10:19:11,487] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 8 ms. (kafka.log.Log) [2019-11-26 10:19:11,488] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 5861] for deletion. (kafka.log.Log) [2019-11-26 10:19:11,491] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log) [2019-11-26 10:19:11,497] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log) [2019-11-26 10:19:11,499] INFO [ProducerStateManager partition=event_mapped-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 2 ms. (kafka.log.Log) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 6452] for deletion. (kafka.log.Log) [2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log) [2019-11-26 10:20:11,489] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log) [2019-11-26 10:20:11,493] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log) [2019-11-26 10:20:11,500] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.log.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.index.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment) [2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

查看一些资料发现应该跟他的偏移量的删除策略有关~ 但是我设置了 LOG_CLEANER_ENABLE 关闭删除策略, kafka 依然会删除我的 topic

1 条回复 2019-11-26 19:40:27 +08:00

amwyyyy

2019-11-26 19:40:27 +08:00

默认的配置，数据是可以保存 7 天。你这个日志有点像 topic 被删了，或者调整分区位置。