Unix/Linux Go Back    


Filesystems, Disks and Memory Discuss NAS, SAN, RAID, Robotic Libraries, backup devices, RAM, DRAM, SCSI, IDE, EIDE topics here.

Slow copy (cp) performance when overwriting files

Filesystems, Disks and Memory


Tags
copy cp performance

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
ces55's Unix or Linux Image
ces55 ces55 is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 16 July 2013, 10:02 AM EDT
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system - another unix system I'm using does not show the same results.

I don't know the system specifications, so I won't be able to post them (sorry). I'm just looking for ideas as to why I could be getting these results, and ideas to try to eliminate the problem. Thanks!
Sponsored Links
    #2  
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
rbatte1's Unix or Linux Image
rbatte1 rbatte1 is offline Forum Staff  
Root armed
 
Join Date: Jun 2007
Last Activity: 14 December 2017, 6:33 AM EST
Location: Lancashire, UK
Posts: 3,407
Thanks: 1,477
Thanked 669 Times in 602 Posts
I'm guessing that these filesystems are on the same local disk and therefore the IO overheads really mount up.

If you have a source directory src_dir and a target directory trg_dir you could try something like:-
Code:
cd src_dir
ls -1 | (cd trg_dir;xargs rm)
cp * trg_dir


Does that give you something to work from?


Robin
Liverpool/Blackburn
UK
Sponsored Links
    #3  
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
jim mcnamara's Unix or Linux Image
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 13 December 2017, 6:33 PM EST
Location: NM
Posts: 11,249
Thanks: 577
Thanked 1,121 Times in 1,033 Posts
Also consider that
IF:
directory has large numbers of entries - >40K (varies with filesystem type)

Cure: make a directory tree, with each subdirectory having way smaller numbers of files.

Overly full disks can result in fragmented files which trashes update (cp overlay) performance of larger files.

So: You can experience lots of delay finding and overwriting files.

Another cause is poor inode caching. This is OS specific, I cannot comment specifically. But when you try to find a file in a directory, the kernel checks the inode cache first. If it finds the file in the cache it can open the file right away without having to scan thru a huge directory. inode caching also impacts the efficiency of PATH searching.
    #4  
Old Unix and Linux 07-12-2013   -   Original Discussion by ces55
achenle's Unix or Linux Image
achenle achenle is offline Forum Advisor  
Registered User
 
Join Date: Jun 2009
Last Activity: 5 December 2017, 1:02 PM EST
Posts: 1,008
Thanks: 3
Thanked 154 Times in 146 Posts
It's tough to help with so little data.

The problem could be a lot of things. The slow system could just be a slow system. The file system could be almost full, making it slower. It could be on disk(s) that are shared with other very busy file systems.

The more data you provide, the more help you'll get.

What type of Unix? What's the result from running "uname -a"?

What type of file system? How big is the file system? How full is it?
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Copy files from multiple directories into one directory without overwriting them shoaibjameel123 Shell Programming and Scripting 1 08-27-2011 05:14 AM
copy files using scp without overwriting sivarajb Shell Programming and Scripting 3 12-11-2010 03:04 PM
Slow copy/performance... between volumes kopper UNIX for Dummies Questions & Answers 0 07-14-2010 10:01 PM
Slow Copy(CP) performance b_sri Filesystems, Disks and Memory 3 10-08-2009 06:29 PM
Help! Slow Performance Neo Post Here to Contact Site Administrators and Moderators 6 08-25-2003 04:08 PM



All times are GMT -4. The time now is 09:18 AM.