Unix/Linux Go Back    

Filesystems, Disks and Memory Discuss NAS, SAN, RAID, Robotic Libraries, backup devices, RAM, DRAM, SCSI, IDE, EIDE topics here.

Slow copy (cp) performance when overwriting files

Filesystems, Disks and Memory

copy cp performance

Thread Tools Search this Thread Display Modes
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
ces55's Unix or Linux Image
ces55 ces55 is offline
Registered User
Join Date: Jul 2013
Last Activity: 16 July 2013, 10:02 AM EDT
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system - another unix system I'm using does not show the same results.

I don't know the system specifications, so I won't be able to post them (sorry). I'm just looking for ideas as to why I could be getting these results, and ideas to try to eliminate the problem. Thanks!
Sponsored Links
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
rbatte1's Unix or Linux Image
rbatte1 rbatte1 is offline Forum Staff  
Root armed
Join Date: Jun 2007
Last Activity: 21 June 2018, 9:53 AM EDT
Location: Lancashire, UK
Posts: 3,559
Thanks: 1,566
Thanked 699 Times in 627 Posts
I'm guessing that these filesystems are on the same local disk and therefore the IO overheads really mount up.

If you have a source directory src_dir and a target directory trg_dir you could try something like:-

cd src_dir
ls -1 | (cd trg_dir;xargs rm)
cp * trg_dir

Does that give you something to work from?

Sponsored Links
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
jim mcnamara's Unix or Linux Image
jim mcnamara jim mcnamara is offline Forum Staff  
Join Date: Feb 2004
Last Activity: 21 June 2018, 9:11 PM EDT
Location: NM
Posts: 11,416
Thanks: 629
Thanked 1,170 Times in 1,077 Posts
Also consider that
directory has large numbers of entries - >40K (varies with filesystem type)

Cure: make a directory tree, with each subdirectory having way smaller numbers of files.

Overly full disks can result in fragmented files which trashes update (cp overlay) performance of larger files.

So: You can experience lots of delay finding and overwriting files.

Another cause is poor inode caching. This is OS specific, I cannot comment specifically. But when you try to find a file in a directory, the kernel checks the inode cache first. If it finds the file in the cache it can open the file right away without having to scan thru a huge directory. inode caching also impacts the efficiency of PATH searching.
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
achenle's Unix or Linux Image
achenle achenle is offline Forum Advisor  
Registered User
Join Date: Jun 2009
Last Activity: 20 June 2018, 11:19 AM EDT
Posts: 1,014
Thanks: 3
Thanked 156 Times in 148 Posts
It's tough to help with so little data.

The problem could be a lot of things. The slow system could just be a slow system. The file system could be almost full, making it slower. It could be on disk(s) that are shared with other very busy file systems.

The more data you provide, the more help you'll get.

What type of Unix? What's the result from running "uname -a"?

What type of file system? How big is the file system? How full is it?
Sponsored Links

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Copy files from multiple directories into one directory without overwriting them shoaibjameel123 Shell Programming and Scripting 1 08-27-2011 04:14 AM
copy files using scp without overwriting sivarajb Shell Programming and Scripting 3 12-11-2010 02:04 PM
Slow copy/performance... between volumes kopper UNIX for Dummies Questions & Answers 0 07-14-2010 09:01 PM
Slow Copy(CP) performance b_sri Filesystems, Disks and Memory 3 10-08-2009 05:29 PM
Help! Slow Performance Neo Post Here to Contact Site Administrators and Moderators 6 08-25-2003 03:08 PM

All times are GMT -4. The time now is 05:21 AM.