关于 Gunicorn + Flask 在多进程中，日志轮转的一个 BUG

多进程模式下肯定这样是不行的。 考虑用其它东西，比如 redis 订阅之类的，交给另一个服务去处理日志

我 Python 开发的时候一直用 Loguru ，推荐你试试

我自己又重新测试了下，假设 1001 对 app.log 进行轮转：

1. app.log 该名称变成 app.log.bak ，但实际上我发现 app.log.bak 的 inode 值没变

2. 1001 进程创建一个新的文件叫 app.log ，然后自己一个人把日志写入到这个新的文件里（新的 inode 值）

3. 1002 进程和 1003 进程被 1001 骗了，还在往 app.log.bak 写日志

大概，似乎，是这样子。。。

我踩过坑，解决方法，写一个多线程兼容的 handler 去处理

`
class MultiCompatibleTimedRotatingFileHandler(TimedRotatingFileHandler):
def doRollover(self):
if self.stream:
self.stream.close()
self.stream = None
# get the time that this sequence started at and make it a TimeTuple
currentTime = int(time.time())
dstNow = time.localtime(currentTime)[-1]
t = self.rolloverAt - self.interval
if self.utc:
timeTuple = time.gmtime(t)
else:
timeTuple = time.localtime(t)
dstThen = timeTuple[-1]
if dstNow != dstThen:
if dstNow:
addend = 3600
else:
addend = -3600
timeTuple = time.localtime(t + addend)
dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
# 兼容多进程并发 LOG_ROTATE
if not os.path.exists(dfn):
f = open(self.baseFilename, "a")
fcntl.lockf(f.fileno(), fcntl.LOCK_EX)
if not os.path.exists(dfn):
os.rename(self.baseFilename, dfn) # 释放锁 释放老 log 句柄
f.close()
if self.backupCount > 0:
for s in self.getFilesToDelete():
os.remove(s)
if not self.delay:
self.stream = self._open()
newRolloverAt = self.computeRollover(currentTime)
while newRolloverAt <= currentTime:
newRolloverAt = newRolloverAt + self.interval
# If DST changes and midnight or weekly rollover, adjust for this.
if (self.when == "MIDNIGHT" or self.when.startswith("W")) and not self.utc:
dstAtRollover = time.localtime(newRolloverAt)[-1]
if dstNow != dstAtRollover:
if (
not dstNow
): # DST kicks in before next rollover, so we need to deduct an hour
addend = -3600
else: # DST bows out before next rollover, so we need to add an hour
addend = 3600
newRolloverAt += addend
self.rolloverAt = newRolloverAt
`

用 stream 写就行了。日志存盘那是另外的事

如果不是日志量大到惊悚，建议打到 syslog 去，用 logrotate 管理切分。比应用程序或者框架自己管理日志切分省心。

@adoal syslog 倒不必……大量的应用程序日志会影响调查系统级别的问题

import logging
from logging import getLogger, DEBUG
from concurrent_log_handler import ConcurrentRotatingFileHandler

formatter_log = logging.Formatter('%(asctime)s%(filename)s[%(lineno)d]%(funcName)s %(levelname)s %(message)s')

logger = getLogger()
logfile = os.path.abspath("log/resource.log")
rotateHandler = ConcurrentRotatingFileHandler(logfile, "a", 3000 * 1024, 50)
rotateHandler.setFormatter(formatter_log)
logger.addHandler(rotateHandler)
logger.setLevel(DEBUG)
我的也是 gunicorn 多进程,使用 ConcurrentRotatingFileHandler 正常记录多进程日志.

使用 ConcurrentRotatingFileHandler 可以解决多进程日志记录的问题。ConcurrentRotatingFileHandler 是 ConcurrentLogHandler 库的一部分，专门设计用于在进程环境中安全地进行日志记录和日志轮转。

python 自带的日志库想解决这个问题很麻烦，最简单的就是不在 python 内做轮转，直接打到 std ，用类似 supervisor 之类的捕获 std 输出做轮转

现在大部分不都是用 docker 部署，直接打到 std ，用 docker 自带的日志管理实现轮转

import os
import time

from logging.handlers import TimedRotatingFileHandler


class SafeRotatingFileHandler(TimedRotatingFileHandler):
def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False):
TimedRotatingFileHandler.__init__(self, filename, when, interval, backupCount, encoding, delay, utc)

"""
多进程, 文件处理.
Override doRollover
lines commanded by "##" is changed by cc
https://www.jianshu.com/p/d615bf01e37b
"""

def doRollover(self):
"""
do a rollover; in this case, a date/time stamp is appended to the filename
when the rollover happens. However, you want the file to be named for the
start of the interval, not the current time. If there is a backup count,
then we have to get a list of matching filenames, sort them and remove
the one with the oldest suffix.

Override, 1. if dfn not exist then do rename
2. _open with "a" model
"""
if self.stream:
self.stream.close()
self.stream = None
# get the time that this sequence started at and make it a TimeTuple
currentTime = int(time.time())
dstNow = time.localtime(currentTime)[-1]
t = self.rolloverAt - self.interval
if self.utc:
timeTuple = time.gmtime(t)
else:
timeTuple = time.localtime(t)
dstThen = timeTuple[-1]
if dstNow != dstThen:
if dstNow:
addend = 3600
else:
addend = -3600
timeTuple = time.localtime(t + addend)

# 多进程, 文件处理.
dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
# if os.path.exists(dfn):
# os.remove(dfn)

# Issue 18940: A file may not have been created if delay is True.
# if os.path.exists(self.baseFilename):
if not os.path.exists(dfn) and os.path.exists(self.baseFilename):
os.rename(self.baseFilename, dfn)

if self.backupCount > 0:
for s in self.getFilesToDelete():
os.remove(s)
if not self.delay:
self.mode = "a"
self.stream = self._open()
newRolloverAt = self.computeRollover(currentTime)
while newRolloverAt <= currentTime:
newRolloverAt = newRolloverAt + self.interval
# If DST changes and midnight or weekly rollover, adjust for this.
if (self.when == 'MIDNIGHT' or self.when.startswith('W')) and not self.utc:
dstAtRollover = time.localtime(newRolloverAt)[-1]
if dstNow != dstAtRollover:
if not dstNow: # DST kicks in before next rollover, so we need to deduct an hour
addend = -3600
else: # DST bows out before next rollover, so we need to add an hour
addend = 3600
newRolloverAt += addend
self.rolloverAt = newRolloverAt

init:
class Logger(object):
def __init__(self):
handlers.SafeRotatingFileHandler = SafeRotatingFileHandler

yaml:
info:
class: logging.handlers.SafeRotatingFileHandler
level: INFO
formatter: console
filename: ./log/info.log
when: MIDNIGHT
interval : 1
backupCount: 7
encoding: utf8


表现就是在切分日志的时候, 会发生日志丢失

https://www.jianshu.com/p/d615bf01e37b
重写 TimedRotatingFileHandler, 之前遇到过相似问题, 希望可以帮到你.

这是 21 年左右的方式了, 现在也许会有更好的 :)

```py
import datetime
```

我们直接在日志文件后面加了进程号 app_name_{pid}.log

@zhangchunjiiw 你在简书写的这篇博客对我很有帮助，感谢

@yumenlong https://pypi.org/project/concurrent-log-handler/ 这个库对么，我试试

@Koril 是这个库，我最近在 django 中用这个库解决了类似的问题