Home
Man
Search
Today's Posts
Register

Discuss NAS, SAN, RAID, Robotic Libraries, backup devices, RAM, DRAM, SCSI, IDE, EIDE topics here.

Slow copy (cp) performance when overwriting files

Tags
copy cp performance, filesystem

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 07-12-2013
Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system - another unix system I'm using does not show the same results.

I don't know the system specifications, so I won't be able to post them (sorry). I'm just looking for ideas as to why I could be getting these results, and ideas to try to eliminate the problem. Thanks!
# 2  
Old 07-12-2013
I'm guessing that these filesystems are on the same local disk and therefore the IO overheads really mount up.

If you have a source directory src_dir and a target directory trg_dir you could try something like:-
Code:
cd src_dir
ls -1 | (cd trg_dir;xargs rm)
cp * trg_dir


Does that give you something to work from?


Robin
Liverpool/Blackburn
UK
# 3  
Old 07-12-2013
Also consider that
IF:
directory has large numbers of entries - >40K (varies with filesystem type)

Cure: make a directory tree, with each subdirectory having way smaller numbers of files.

Overly full disks can result in fragmented files which trashes update (cp overlay) performance of larger files.

So: You can experience lots of delay finding and overwriting files.

Another cause is poor inode caching. This is OS specific, I cannot comment specifically. But when you try to find a file in a directory, the kernel checks the inode cache first. If it finds the file in the cache it can open the file right away without having to scan thru a huge directory. inode caching also impacts the efficiency of PATH searching.
# 4  
Old 07-12-2013
It's tough to help with so little data.

The problem could be a lot of things. The slow system could just be a slow system. The file system could be almost full, making it slower. It could be on disk(s) that are shared with other very busy file systems.

The more data you provide, the more help you'll get.

What type of Unix? What's the result from running "uname -a"?

What type of file system? How big is the file system? How full is it?
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
GFS file system performance is very slow susindram Red Hat 3 04-01-2012 11:14 AM
99% performance wa, slow server. draiphod Infrastructure Monitoring 2 10-07-2011 02:47 PM
Copy files from multiple directories into one directory without overwriting them shoaibjameel123 Shell Programming and Scripting 1 08-27-2011 05:14 AM
Slow performance filtering file Miila Shell Programming and Scripting 5 02-11-2011 04:23 AM
copy files using scp without overwriting sivarajb Shell Programming and Scripting 3 12-11-2010 03:04 PM
Slow copy/performance... between volumes kopper UNIX for Dummies Questions & Answers 0 07-14-2010 10:01 PM
Slow Copy(CP) performance b_sri Filesystems, Disks and Memory 3 10-08-2009 06:29 PM
egrep is very slow : How to improve performance hidnana Shell Programming and Scripting 7 02-12-2008 07:13 AM
Help! Slow Performance Neo Post Here to Contact Site Administrators and Moderators 6 08-25-2003 04:08 PM


All times are GMT -4. The time now is 09:24 AM.

Unix & Linux Forums Content Copyright 1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password