拷贝大文件(并且数量多)到移动硬盘的过程中经常死机 - V2EX
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Distributions
Ubuntu
Fedora
CentOS
中文资源站
网易开源镜像站
atom
V2EX    Linux

拷贝大文件(并且数量多)到移动硬盘的过程中经常死机

  •  
  •   atom 2014-12-18 10:02:05 +08:00 7857 次点击
    这是一个创建于 3998 天前的主题,其中的信息可能已经有所发展或是发生改变。
    我的系统是Fedora20, 3.16.6-200.fc20.i686+PAE
    有两块希捷移动硬盘(FreeAgent Go 和 Backup Plus)。

    我有定期备份文档和照片的习惯,这些文件数量大(4000多个),并且累积起来容量也不小,(10GB-20GB左右),每次就是把文件夹拖到移动硬盘保存。

    最近发现拷贝到移动硬盘的过程中,大概3、4分钟后,进度条不动了,上面的数字也不变了, 鼠标和键盘没有任何反应,系统就这样死掉了。过了一会儿,移动硬盘的工作灯也不亮了,但文件实际上没有复制完。

    一开始我以为是移动硬盘的问题,在Windows和Mac上试了下,运行良好。

    这是不是赶上什么模块bug了?
    8 条回复    2014-12-19 08:44:14 +08:00
    carmark
        1
    carmark  
       2014-12-18 10:31:05 +08:00   1
    看样子是linux的file cache的问题,你可以按照下面的说明试试。
    http://askubuntu.com/questions/122857/can-i-copy-large-files-faster-without-using-the-file-cache

    如果这个不好用,可以尝试找找linux下的kernel-bypass的拷贝命令,具体我忘记了,我记得是有类似的工具。主要就是减少用户态内存和内核态内存的数据拷贝。
    GPU
        2
    GPU  
       2014-12-18 11:55:28 +08:00 via iPhone
    如果系统的硬盘与资料的硬盘分开呢
    likuku
        3
    likuku  
       2014-12-18 12:49:24 +08:00
    用 rsync -avP source/ target/

    别用cp,也别文件管理器里拖放。
    bellchu
        4
    bellchu  
       2014-12-18 14:32:27 +08:00   1
    一楼说的没错,缓存的问题,KERNEL的写入缓存太大,以至于你从高速存储写往低速设备时出现拥塞瓶颈

    限定一下dirty background bytes到比较小的数值测试下卡顿现象,至于多少数值就要你自己测试了,设4M试试看先

    下面是一些相关设置的默认值,
    vm.dirty_background_bytes = 0
    vm.dirty_background_ratio = 10
    vm.dirty_bytes = 0
    vm.dirty_expire_centisecs = 3000
    vm.dirty_ratio = 20
    vm.dirty_writeback_centisecs = 500

    以下是Linus Torvalds对相关问题的解释

    From: Linus Torvalds <torvalds@linux-foundation.org>
    Newsgroups: fa.linux.kernel
    Subject: Re: [ext3][kernels >= 2.6.20.7 at least] KDE going comatose when FS
    Date: Fri, 27 Apr 2007 15:19:40 UTC
    Message-ID: <fa.+jJfsPdWk8uCcJT/[email protected]>

    On Fri, 27 Apr 2007, Mike Galbraith wrote:
    >
    > As subject states, my GUI is going away for extended periods of time
    > when my very full and likely highly fragmented (how to find out)
    > filesystem is under heavy write load. While write is under way, if
    > amarok (mp3 player) is running, no song change will occur until write is
    > finished, and the GUI can go _entirely_ comatose for very long periods.
    > Usually, it will come back to life after write is finished, but
    > occasionally, a complete GUI restart is necessary.

    One thing to try out (and dammit, I should make it the default now in
    2.6.21) is to just make the dirty limits much lower. We've been talking
    about this for ages, I think this might be the right time to do it.

    Especially with lots of memory, allowing 40% of that memory to be dirty is
    just insane (even if we limit it to "just" 40% of the normal memory zone.
    That can be gigabytes. And no amount of IO scheduling will make it
    pleasant to try to handle the situation where that much memory is dirty.

    So I do believe that we could probably do something about the IO
    scheduling _too_:

    - break up large write requests (yeah, it will make for worse IO
    througput, but if make it configurable, and especially with
    controllers that don't have insane overheads per command, the
    difference between 128kB requests and 16MB requests is probably not
    really even noticeable - SCSI things with large per-command overheads
    are just stupid)

    Generating huge requests will automatically mean that they are
    "unbreakable" from an IO scheduler perspective, so it's bad for latency
    for other reqeusts once they've started.

    - maybe be more aggressive about prioritizing reads over writes.

    but in the meantime, what happens if you apply this patch?

    Actually, you don't need to apply the patch - just do

    echo 5 > /proc/sys/vm/dirty_background_ratio
    echo 10 > /proc/sys/vm/dirty_ratio

    and say if it seems to improve things. I think those are much saner
    defaults especially for a desktop system (and probably for most servers
    too, for that matter).

    Even 10% of memory dirty can be a whole lot of RAM, but it should
    hopefully be _better_ than the insane default we have now.

    Historical note: allowing about half of memory to contain dirty pages made
    more sense back in the days when people had 16-64MB of memory, and a
    single untar of even fairly small projects would otherwise hit the disk.
    But memory sizes have grown *much* more quickly than disk speeds (and
    latency requirements have gone down, not up), so a default that may
    actually have been perfectly fine at some point seems crazy these days..

    Linus

    ---
    diff --git a/mm/page-writeback.c b/mm/page-writeback.c
    index f469e3c..a794945 100644
    --- a/mm/page-writeback.c
    +++ b/mm/page-writeback.c
    @@ -67,12 +67,12 @@ static inline long sync_writeback_pages(void)
    /*
    * Start background writeback (via pdflush) at this percentage
    */
    -int dirty_background_ratio = 10;
    +int dirty_background_ratio = 5;

    /*
    * The generator of dirty data starts writeback at this percentage
    */
    -int vm_dirty_ratio = 40;
    +int vm_dirty_ratio = 10;

    /*
    * The interval between `kupdate'-style writebacks, in jiffies
    atom
        5
    atom  
    OP
       2014-12-18 15:41:24 +08:00
    @bellchu
    @carmark
    晚上试试,先谢过
    hiboshi
        6
    hiboshi  
       2014-12-18 18:36:12 +08:00
    试试rsync
    webjin
        7
    webjin  
       2014-12-19 00:03:33 +08:00 via Android
    不错
    atom
        8
    atom  
    OP
       2014-12-19 08:44:14 +08:00
    @bellchu
    改参数后复制成功,多谢指点
    关于     帮助文档     自助推广系统     博客     API     FAQ     Solana     1032 人在线   最高记录 6679       Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 21ms UTC 23:36 PVG 07:36 LAX 15:36 JFK 18:36
    Do have faith in what you're doing.
    ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86