Slow copy (cp) performance when overwriting files


 
Thread Tools Search this Thread
Special Forums Hardware Filesystems, Disks and Memory Slow copy (cp) performance when overwriting files
# 1  
Old 07-12-2013
Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system - another unix system I'm using does not show the same results.

I don't know the system specifications, so I won't be able to post them (sorry). I'm just looking for ideas as to why I could be getting these results, and ideas to try to eliminate the problem. Thanks!
# 2  
Old 07-12-2013
I'm guessing that these filesystems are on the same local disk and therefore the IO overheads really mount up.

If you have a source directory src_dir and a target directory trg_dir you could try something like:-
Code:
cd src_dir
ls -1 | (cd trg_dir;xargs rm)
cp * trg_dir


Does that give you something to work from?


Robin
Liverpool/Blackburn
UK
# 3  
Old 07-12-2013
Also consider that
IF:
directory has large numbers of entries - >40K (varies with filesystem type)

Cure: make a directory tree, with each subdirectory having way smaller numbers of files.

Overly full disks can result in fragmented files which trashes update (cp overlay) performance of larger files.

So: You can experience lots of delay finding and overwriting files.

Another cause is poor inode caching. This is OS specific, I cannot comment specifically. But when you try to find a file in a directory, the kernel checks the inode cache first. If it finds the file in the cache it can open the file right away without having to scan thru a huge directory. inode caching also impacts the efficiency of PATH searching.
# 4  
Old 07-12-2013
It's tough to help with so little data.

The problem could be a lot of things. The slow system could just be a slow system. The file system could be almost full, making it slower. It could be on disk(s) that are shared with other very busy file systems.

The more data you provide, the more help you'll get.

What type of Unix? What's the result from running "uname -a"?

What type of file system? How big is the file system? How full is it?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are... (7 Replies)
Discussion started by: christr
7 Replies

2. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%.... (2 Replies)
Discussion started by: draiphod
2 Replies

3. Shell Programming and Scripting

Copy files from multiple directories into one directory without overwriting them

I have several directories and all those directories have .dat files in them. I want to copy all those .dat files to one directory say "collected_directory" The problem is I don't want to overwrite files. So, if two file names match, I don't want the old file to be overwritten with a new one. ... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

4. Shell Programming and Scripting

Slow performance filtering file

Please, I need help tuning my script. It works but it's too slow. The code reads an acivity log file with 50.000 - 100.000 lines and filters error messages from it. The data in the actlog file look similar to this: 02/08/2011 00:25:01,ANR2034E QUERY MOUNT: No match found using this criteria.... (5 Replies)
Discussion started by: Miila
5 Replies

5. Shell Programming and Scripting

copy files using scp without overwriting

Hi, i need to use "scp" to copy a file without overwriting the same in destinations. any suggestion? thanks Sivaji (3 Replies)
Discussion started by: sivarajb
3 Replies

6. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we... (0 Replies)
Discussion started by: kopper
0 Replies

7. Filesystems, Disks and Memory

Slow Copy(CP) performance

Hi all We have got issues with copying a 2.6 GB file from one folder to another folder. Well, this is not the first issue we are having on the box currently, i will try to explain everything we have done from the past 2 days. We got a message 2 days back saying that our Production is 98%... (3 Replies)
Discussion started by: b_sri
3 Replies

8. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other... (7 Replies)
Discussion started by: hidnana
7 Replies

9. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo (6 Replies)
Discussion started by: Neo
6 Replies
Login or Register to Ask a Question