NGINX 处于 D 状态,阻塞 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
yuyuyu
V2EX    NGINX

NGINX 处于 D 状态,阻塞

  •  
  •   yuyuyu 2020-08-02 01:17:19 +08:00 3816 次点击
    这是一个创建于 1905 天前的主题,其中的信息可能已经有所发展或是发生改变。

    centos7,4CPU 8G,nginx 时不时在 top 里出现 d 状态,查了下是 uninterruptible sleep,出现的时候 iftop 网络全部跌至低谷,io 飙升,nginx 不接受也不处理新请求,业务阻塞死,大概 1 秒以后恢复,期间 CPU 、内存都正常,看 IOPS 也很低才几百(通过阿里云的监控看的),nginx 使用了 cache,把 proxy_cache 相关的全部注释掉关闭 cache 后不再出现 d 状态,这到底是啥问题?服务器是阿里 ECS,CPU 和内存肯定够用,IO 也不高,下面是出现问题时的状态:

    avg-cpu: %user %nice %system %iowait %steal %idle 10.61 0.00 12.12 25.25 0.00 52.02

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 51.00 217.00 163.00 1123.00 880.00 6212.00 11.03 11.63 9.05 2.46 10.00 0.41 53.20

    avg-cpu: %user %nice %system %iowait %steal %idle 12.56 0.00 13.57 39.20 0.00 34.67

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 0.00 257.00 168.00 930.00 716.00 5100.00 10.59 38.18 34.77 5.36 40.08 0.51 56.20

    avg-cpu: %user %nice %system %iowait %steal %idle 16.58 0.00 16.58 63.32 0.00 3.52

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util vda 29.00 291.00 184.00 1635.00 1040.00 37052.00 41.88 56.54 31.08 7.70 33.71 0.47 86.00

    iotop:

    Total DISK READ : 625.15 K/s | Total DISK WRITE : 680.54 K/s Actual DISK READ: 625.15 K/s | Actual DISK WRITE: 31.36 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 3174 be/4 nginx 320.49 K/s 328.40 K/s 0.00 % 42.88 % nginx: worker process 4012 be/4 root 0.00 B/s 0.00 B/s 0.00 % 39.93 % [kworker/u4:3+flush-253:0] 3173 be/4 nginx 304.66 K/s 352.14 K/s 0.00 % 36.35 % nginx: worker process Total DISK READ : 1148.47 K/s | Total DISK WRITE : 1089.07 K/s Actual DISK READ: 1148.47 K/s | Actual DISK WRITE: 3.77 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND 4012 be/4 root 0.00 B/s 0.00 B/s 0.00 % 41.17 % [kworker/u4:3-events_unbound] 3173 be/4 nginx 570.28 K/s 574.24 K/s 0.00 % 37.87 % nginx: worker process 3174 be/4 nginx 578.20 K/s 514.83 K/s 0.00 % 37.18 % nginx: worker process

    下面是我的 NGINX 配置: worker_processes 2; worker_shutdown_timeout 180s;

    error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid;

    worker_rlimit_nofile 65535; events {

    worker_connections 65535; 

    } http {

    include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" $upstream_cache_status $request_time $upstream_response_time'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; gzip on; gzip_types application/json text/plain application/x-Javascript application/Javascript text/Javascript text/css application/xml text/xml; gzip_min_length 1k; proxy_buffering on; proxy_buffer_size 32k; proxy_buffers 8 512k; server_tokens off; underscores_in_headers on; proxy_temp_path /tmp/cache_tmp; proxy_cache_path /tmp/cache levels=1:2 keys_zOne=cache1:100m inactive=7d max_size=20g; upstream backend { server xxx:xx max_fails=3 weight=3 fail_timeout=60s; } server { listen 80; location / { proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_cache cache1; proxy_cache_convert_head off; proxy_cache_key $uri proxy_cache_revalidate on; proxy_cache_methods GET HEAD POST; } } 

    }

    9 条回复    2020-08-03 11:23:54 +08:00
    vk42
        1
    vk42  
       2020-08-02 01:24:24 +08:00
    查下阻塞的时候 wchan 是什么?
    mml
        2
    mml  
       2020-08-02 02:00:46 +08:00
    几百 iops 和下面那些 iowait 都不低啊。
    yuyuyu
        3
    yuyuyu  
    OP
       2020-08-02 10:47:46 +08:00
    @mml 但阿里 ecs 给的性能完全不止这点啊,他们的云盘有 10000 iops,我也给阿里发了工单,他们说 iops 不高,让我检查下 nginx 的 cache 设置,但我实在不知道这个 proxy_cache 还有什么可以优化的地方
    realpg
        4
    realpg  
    PRO
       2020-08-02 11:50:18 +08:00
    基本是 IO 的问题。
    运作良好时候会有 10000 IOPS
    映射的块有问题时候 就上不去了啊……
    yuyuyu
        5
    yuyuyu  
    OP
       2020-08-02 11:52:17 +08:00
    @realpg 那这个要怎么解决?多挂几块云盘分散一下?
    realpg
        6
    realpg  
    PRO
       2020-08-02 11:54:33 +08:00
    另外,虚拟环境,考虑 DIRECTIO,与 VM 的 IPSAN 、Local SSD IO cache 的问题,这个不好调试
    yuyuyu
        7
    yuyuyu  
    OP
       2020-08-02 19:25:17 +08:00
    @realpg 用 fio 测了下随机写 4K,只有 9M/s 左右,是不是这个原因? 32K 就有 72M/S,NGINX 怎么优化?
    linuxmap
        8
    linuxmap  
       2020-08-03 11:22:19 +08:00
    可以考虑一下这样的架构。nginx1+ats+nginx2:nginx1 专心的做业务处理,使用 lua 会非常的方便 ats 专门做缓存 nginx2 做回源处理。
    linuxmap
        9
    linuxmap  
       2020-08-03 11:23:54 +08:00
    proxy_cache 在磁盘满了之后的 磁盘调整的策略好像有点问题, 楼主可以分析一下这块的代码。
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     3447 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 24ms UTC 10:58 PVG 18:58 LAX 03:58 JFK 06:58
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86