Slow copy (cp) performance when overwriting files


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 07-12-2013
Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system - another unix system I'm using does not show the same results.

I don't know the system specifications, so I won't be able to post them (sorry). I'm just looking for ideas as to why I could be getting these results, and ideas to try to eliminate the problem. Thanks!
# 2  
Old 07-12-2013
I'm guessing that these filesystems are on the same local disk and therefore the IO overheads really mount up.

If you have a source directory src_dir and a target directory trg_dir you could try something like:-
Code:
cd src_dir
ls -1 | (cd trg_dir;xargs rm)
cp * trg_dir


Does that give you something to work from?


Robin
Liverpool/Blackburn
UK
# 3  
Old 07-12-2013
Also consider that
IF:
directory has large numbers of entries - >40K (varies with filesystem type)

Cure: make a directory tree, with each subdirectory having way smaller numbers of files.

Overly full disks can result in fragmented files which trashes update (cp overlay) performance of larger files.

So: You can experience lots of delay finding and overwriting files.

Another cause is poor inode caching. This is OS specific, I cannot comment specifically. But when you try to find a file in a directory, the kernel checks the inode cache first. If it finds the file in the cache it can open the file right away without having to scan thru a huge directory. inode caching also impacts the efficiency of PATH searching.
# 4  
Old 07-12-2013
It's tough to help with so little data.

The problem could be a lot of things. The slow system could just be a slow system. The file system could be almost full, making it slower. It could be on disk(s) that are shared with other very busy file systems.

The more data you provide, the more help you'll get.

What type of Unix? What's the result from running "uname -a"?

What type of file system? How big is the file system? How full is it?
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Solaris 11.1 Slow Network Performance christr Solaris 7 06-27-2013 08:01 PM
GFS file system performance is very slow susindram Red Hat 3 04-01-2012 10:14 AM
99% performance wa, slow server. draiphod Infrastructure Monitoring 2 10-07-2011 01:47 PM
Copy files from multiple directories into one directory without overwriting them shoaibjameel123 Shell Programming and Scripting 1 08-27-2011 04:14 AM
Slow performance filtering file Miila Shell Programming and Scripting 5 02-11-2011 03:23 AM
Hard disk write performance very slow jegaraman Solaris 9 12-13-2010 02:47 AM
copy files using scp without overwriting sivarajb Shell Programming and Scripting 3 12-11-2010 02:04 PM
Slow copy/performance... between volumes kopper UNIX for Dummies Questions & Answers 0 07-14-2010 09:01 PM
Unzip files without overwriting donisback Shell Programming and Scripting 1 03-04-2010 01:28 PM
Slow Copy(CP) performance b_sri Filesystems, Disks and Memory 3 10-08-2009 05:29 PM
Performance of log parsing shell script very slow sowmitr Shell Programming and Scripting 7 04-08-2009 01:37 PM
egrep is very slow : How to improve performance hidnana Shell Programming and Scripting 7 02-12-2008 06:13 AM
Help! Slow Performance Neo Post Here to Contact Site Administrators and Moderators 6 08-25-2003 03:08 PM