hyper-v 搭建的 centos7.8.2003 空载情况下内存消耗 80%,找不到消耗的进程,真是奇怪 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
ic2y
V2EX    Linux

hyper-v 搭建的 centos7.8.2003 空载情况下内存消耗 80%,找不到消耗的进程,真是奇怪

  •  
  •   ic2y 2020-06-06 15:47:24 +08:00 5068 次点击
    这是一个创建于 2036 天前的主题,其中的信息可能已经有所发展或是发生改变。
    简介:
    在 hyper-v 上搭建了一个 centos 系统,几乎没有做什么事情,但是内存使用率在几分钟后,会突然使用到 80%,ps 、top 手段看不到 占用内存的进程。

    环境信息如下:

    宿主机:
    win10 专业版 1909 版本、E5 2696 v2 、64G 的 ECC 内存。
    hyper-v 虚拟机:
    1 核心处理器,6000MB 的内存分配,启动动态内存,最小 RAM 2000MB,最大 RAM 6000MB 、内存权重:中。
    centos 系统配置:
    centos 7.8.2003 X64 server 版,无 GUI.

    表现:
    centos 在空载的情况下,前 1 分钟内,使用的内存都是 200MB,但是在 1-3 分钟后,突然上升到 5000MB 的使用量。而且找不到消耗的进程。

    命令执行情况如下:
    free -m
    [root@server10 ~]# free -m
    total used free shared buff/cache available
    Mem: 5663 4151 1301 8 210 1280
    Swap: 8063 0 8063

    df -h
    [root@server10 ~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    devtmpfs 2.8G 0 2.8G 0% /dev
    tmpfs 2.8G 0 2.8G 0% /dev/shm
    tmpfs 2.8G 9.0M 2.8G 1% /run
    tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup
    /dev/mapper/centos-root 50G 1.7G 49G 4% /
    /dev/sda1 1014M 191M 824M 19% /boot
    /dev/mapper/centos-home 192G 918M 191G 1% /home
    tmpfs 567M 0 567M 0% /run/user/0


    ps 根据内存排序
    [root@server10 ~]# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem
    PID PPID CMD %MEM %CPU
    689 1 /usr/bin/python2 -Es /usr/s 0.5 0.0
    1044 1 /usr/bin/python2 -Es /usr/s 0.3 0.0
    661 1 /usr/lib/polkit-1/polkitd - 0.2 0.0
    690 1 /usr/sbin/NetworkManager -- 0.1 0.0
    1047 1 /usr/sbin/rsyslogd -n 0.1 0.0
    482 1 /usr/lib/systemd/systemd-jo 0.1 0.0
    1 0 /usr/lib/systemd/systemd -- 0.1 0.0
    1497 1045 sshd: root@pts/0 0.0 0.0
    502 1 /usr/sbin/lvmetad -f 0.0 0.0
    819 690 /sbin/dhclient -d -q -sf /u 0.0 0.0
    512 1 /usr/lib/systemd/systemd-ud 0.0 0.0
    1045 1 /usr/sbin/sshd -D 0.0 0.0
    663 1 /usr/bin/dbus-daemon --syst 0.0 0.0
    1501 1497 -bash 0.0 0.0
    670 1 /usr/sbin/chronyd 0.0 0.0
    658 1 /usr/lib/systemd/systemd-lo 0.0 0.0
    8255 1501 ps -eo pid,ppid,cmd,%mem,%c 0.0 0.0
    681 1 /usr/sbin/crond -n 0.0 0.0
    635 1 /sbin/auditd 0.0 0.0
    687 1 /sbin/agetty --noclear tty1 0.0 0.0
    2 0 [kthreadd] 0.0 0.0


    [root@server10 ~]# ps -eo rss,pmem,pcpu,vsize,args | sort -k 1 -r -n
    29524 0.5 0.0 358764 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid
    17432 0.3 0.0 574304 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
    12956 0.2 0.0 613020 /usr/lib/polkit-1/polkitd --no-debug
    10996 0.1 0.0 550336 /usr/sbin/NetworkManager --no-daemon
    7332 0.1 0.0 218552 /usr/sbin/rsyslogd -n
    6724 0.1 0.0 39080 /usr/lib/systemd/systemd-journald
    6640 0.1 0.0 128024 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    5680 0.0 0.0 156796 sshd: root@pts/0
    5560 0.0 0.0 201104 /usr/sbin/lvmetad -f
    5508 0.0 0.0 102904 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-acb65408-a188-4419-9d1c-5c58f456bc41-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
    5220 0.0 0.0 48284 /usr/lib/systemd/systemd-udevd
    4316 0.0 0.0 112924 /usr/sbin/sshd -D
    3996 0.0 0.0 227552 /usr/libexec/nm-dispatcher
    2588 0.0 0.0 66476 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
    2152 0.0 0.0 115680 -bash
    1844 0.0 0.0 117808 /usr/sbin/chronyd
    1756 0.0 0.0 26384 /usr/lib/systemd/systemd-logind
    1668 0.0 0.0 126384 /usr/sbin/crond -n
    1488 0.0 0.0 153348 ps -eo rss,pmem,pcpu,vsize,args
    912 0.0 0.0 116600 sort -k 1 -r -n
    856 0.0 0.0 55532 /sbin/auditd
    848 0.0 0.0 110204 /sbin/agetty --noclear tty1 linux

    dmesg 超出长度了,删减一点 0-2 时间的日志
    [root@server10 ~]# dmesg
    [ 1.941334] sda: sda1 sda2
    [ 1.945116] sd 0:0:0:0: [sda] Attached SCSI disk
    [ 2.422141] ata1.01: NODEV after polling detection
    [ 2.505749] ata1.00: host indicates ignore ATA devices, ignored
    [ 2.506095] ata2.01: NODEV after polling detection
    [ 2.592571] ata2.00: ATAPI: Virtual CD, , max MWDMA2
    [ 2.595149] ata2.00: configured for MWDMA2
    [ 2.597455] scsi 3:0:0:0: CD-ROM Msft Virtual CD/ROM 1.0 PQ: 0 ANSI: 5
    [ 2.633490] sr 3:0:0:0: [sr0] scsi3-mmc drive: 0x/0x tray
    [ 2.633494] cdrom: Uniform CD-ROM driver Revision: 3.20
    [ 2.633822] sr 3:0:0:0: Attached scsi CD-ROM sr0
    [ 3.277974] SGI XFS with ACLs, security attributes, no debug enabled
    [ 3.297495] XFS (dm-0): Mounting V5 Filesystem
    [ 3.618658] XFS (dm-0): Ending clean mount
    [ 3.954048] random: crng init done
    [ 4.019475] systemd-journald[92]: Received SIGTERM from PID 1 (systemd).
    [ 4.577895] type=1404 audit(1591358205.434:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
    [ 4.634156] SELinux: 2048 avtab hash slots, 112685 rules.
    [ 4.699424] SELinux: 2048 avtab hash slots, 112685 rules.
    [ 4.730215] SELinux: 8 users, 14 roles, 5046 types, 316 bools, 1 sens, 1024 cats
    [ 4.730219] SELinux: 130 classes, 112685 rules
    [ 4.734688] SELinux: Completing initialization.
    [ 4.734690] SELinux: Setting up existing superblocks.
    [ 4.738802] type=1403 audit(1591358205.595:3): policy loaded auid=4294967295 ses=4294967295
    [ 4.754962] systemd[1]: Successfully loaded SELinux policy in 173.104ms.
    [ 4.965923] ip_tables: (C) 2000-2006 Netfilter Core Team
    [ 4.965951] systemd[1]: Inserted module 'ip_tables'
    [ 4.994376] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 24.600ms.
    [ 5.189157] psmouse serio1: trackpoint: IBM TrackPoint firmware: 0x01, buttons: 3/3
    [ 5.190570] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input3
    [ 5.191489] input: AT Translated Set 2 keyboard as /devices/LNXSYSTM:00/device:00/PNP0A03:00/device:08/VMBUS:01/d34b2567-b9b6-42b9-8778-0a4ec0b955bf/serio2/input/input4
    [ 6.309916] systemd-journald[482]: Received request to flush runtime journal from PID 1
    [ 7.060905] piix4_smbus 0000:00:07.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
    [ 7.755829] hv_vmbus: registering driver hv_balloon
    [ 7.760236] hv_balloon: Using Dynamic Memory protocol version 2.0
    [ 7.782817] pps_core: LinuxPPS API ver. 1 registered
    [ 7.782819] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
    [ 7.806385] PTP clock support registered
    [ 7.828141] hv_utils: Registering HyperV Utility Driver
    [ 7.828145] hv_vmbus: registering driver hv_util
    [ 7.828808] hv_utils: Heartbeat IC version 3.0
    [ 7.830498] hv_utils: Shutdown IC version 3.0
    [ 7.831212] hv_utils: TimeSync IC version 4.0
    [ 7.832551] hv_utils: VSS IC version 5.0
    [ 7.852739] sd 0:0:0:0: Attached scsi generic sg0 tpe 0
    [ 7.853294] sr 3:0:0:0: Attached scsi generic sg1 type 5
    [ 7.862171] input: PC Speaker as /devices/platform/pcspkr/input/input5
    [ 7.939175] cryptd: max_cpu_qlen set to 1000
    [ 8.153763] AVX version of gcm_enc/dec engaged.
    [ 8.153766] AES CTR mode by8 optimization enabled
    [ 8.165736] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
    [ 8.165774] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
    [ 8.301663] EDAC sbridge: Seeking for: PCI ID 8086:0ea0
    [ 8.301668] EDAC sbridge: Ver: 1.1.2
    [ 8.675971] Adding 8257532k swap on /dev/mapper/centos-swap. Priority:-2 extents:1 across:8257532k FS
    [ 8.722279] XFS (sda1): Mounting V5 Filesystem
    [ 8.846217] XFS (sda1): Ending clean mount
    [ 10.421871] XFS (dm-2): Mounting V5 Filesystem
    [ 10.518566] XFS (dm-2): Ending clean mount
    [ 10.866185] type=1305 audit(1591358210.636:4): audit_pid=635 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
    [ 13.479185] ip6_tables: (C) 2000-2006 Netfilter Core Team
    [ 13.626047] Ebtables v2.0 registered
    [ 13.774889] Netfilter messages via NETLINK v0.30.
    [ 13.817275] ip_set: protocol 7
    [ 13.992945] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [ 13.994648] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [ 13.996028] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [ 14.017224] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 14.017908] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 14.072969] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 14.338697] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
    [ 14.543551] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
    [ 14.970829] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [ 14.998091] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
    [ 54.801674] hv_balloon: Max. dynamic memory size: 6000 MB
    10 条回复    2020-06-06 20:10:01 +08:00
    patrickyoung
        1
    patrickyoung  
       2020-06-06 15:57:48 +08:00
    请使用 markdown 对您提交的内容进行排版,大量日志烦请使用 pastebin 等服务贴出。这么排,完全没法看啊
    ic2y
        2
    ic2y  
    OP
       2020-06-06 15:58:57 +08:00
    @patrickyoung 更新过了。我第一把发上去,就发现乱了。后续 死活打不开 V2EX 了。好像网站断断续续被攻击。
    patrickyoung
        3
    patrickyoung  
       2020-06-06 16:01:44 +08:00
    @ic2y 信息不足,烦请使用 pastebin 等服务贴出完整日志。
    ic2y
        4
    ic2y  
    OP
       2020-06-06 16:06:04 +08:00
    @patrickyoung 完整的 dmesg 的地址 https://pastebin.com/3Zn2LtGN
    12101111
        5
    12101111  
       2020-06-06 16:44:21 +08:00
    Hyper-V Memory ballooning,就你粘的 dmesg 最后一行
    简言之,半虚拟化的内核可以将不用的内存还给宿主系统。
    tms
        6
    tms  
       2020-06-06 16:48:22 +08:00
    我猜是 Memory ballooning 的锅
    Osk
        7
    Osk  
       2020-06-06 16:53:32 +08:00 via Android
    更新内核
    关闭动态内存
    先试试看是什么问题

    另外微软建议 Linux guest 的内存是 128M 的整数倍
    ic2y
        8
    ic2y  
    OP
       2020-06-06 17:47:33 +08:00
    @12101111 @tms @Osk

    问题已经解决了。是 Hyper-v 的动态内存的问题。关闭了 Hyper-v 的动态内存,Centos 的内存消耗就变正常。

    刚开始,解决问题的思路偏了,一直以为有什么恶意脚本。但是始终找不到进程。

    猜测的最终原因:应该是 Hyper-v 与 Centos7 的适配有问题,Hyper-v 的动态内存我设定最小 2G,最大 6G,中间应该有 4G 没有真的分配,被宿主机挪走了。但是 Centos 就显示 4G+真正的使用的内存。就变成 80%的内存使用率了。
    msg7086
        9
    msg7086  
       2020-06-06 18:00:59 +08:00
    只看标题就知道是 ballooning 。
    这东西开着一般也不影响什么,反正虚拟机里跑程序的时候宿主会把内存还回去的。
    ps1aniuge
        10
    ps1aniuge  
       2020-06-06 20:10:01 +08:00
    centos 内核太老,对 hyperv 的内存动态调节兼容差?
    建议用 centos8,ubuntu1804,debian9,debian10.
    分享 debian9.12 ,bt 种子下载:
    https://cdimage.debian.org/cdimage/archive/9.12.0/amd64/bt-dvd/
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     2522 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by soltude
    VERSION: 3.9.8.5 25ms UTC 14:38 PVG 22:38 LAX 06:38 JFK 09:38
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86