Slow Copy(CP) performance Post: 302360369

Sponsored Content

Special Forums Hardware Filesystems, Disks and Memory Slow Copy(CP) performance Post 302360369 by jim mcnamara on Thursday 8th of October 2009 05:29:58 PM

10-08-2009

Registered User

Rule of thumb:
Seriously consider never letting a given filesystem get above 80-85% full.

Filesystems under heavy I/O loads suffer from various kinds of latency issues
when free space becomes tight, file allocation times increase as well.

The other caveat:
Assuming loads of available free inodes, huge directory files (the directory file itself, not what is in the directory) are the result of adding lots of files to a single directory. As the directory file itself grows, system performance against it - ls, find, stat, etc. - becomes very poor.

This is because any operation that does a readdir, which is sequential, is really slow if it has to read thru 2 million entries to find one filename.

When files are deleted from the bloated directory, it does not shrink. You have to park the remaining files somewhere, delete the directory, recreate the directory, then move the files back into it. And you have a new, smaller size directory file. Having broader directory trees solves this problem in the first place.

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

9 More Discussions You Might Find Interesting

1. Post Here to Contact Site Administrators and Moderators

Help! Slow Performance

Is the performance now very, very slow (pages take a very long time to load)? Or is it just me? Neo

2. Shell Programming and Scripting

egrep is very slow : How to improve performance

We have an egrep search in a while loop. egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new ${PICKUP_DIR}/new_update is 210 MB file In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other...

3. UNIX for Dummies Questions & Answers

Slow copy/performance... between volumes

hi guys We are seeing weird issues on my Linux Suse 10, it has lotus 8.5 and 1 filesystem for OS and another for Lotus Database. the issue is when the Lotus service starts wait on top is very high about 25% percent and in general CPU usage is very high we found that when this happens if we...

4. Solaris

Hard disk write performance very slow

Dear All, I have a hard disk in solaris on which the write performanc is too slow. The CPU , RAM memory are absolutely fine. What might be reason. Kindly explain. Rj

5. Shell Programming and Scripting

Slow performance filtering file

Please, I need help tuning my script. It works but it's too slow. The code reads an acivity log file with 50.000 - 100.000 lines and filters error messages from it. The data in the actlog file look similar to this: 02/08/2011 00:25:01,ANR2034E QUERY MOUNT: No match found using this criteria....

6. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%....

7. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system...

8. Solaris

Solaris 11.1 Slow Network Performance

I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are...

9. Filesystems, Disks and Memory

Slow copy (cp) performance when overwriting files

I have a lot of binary files I need to copy to a folder. The folder is already filled with files of the same name. Copying on top of the old files takes MUCH longer than if I were to delete the old files then copy the new files to the now-empty folder. This result is specific to one system -...

LEARN ABOUT HPUX

delete_directory

delete directory(1m)													      delete directory(1m)

NAME

       delete directory - Deletes a directory

SYNOPSIS

       cdscp delete directory directory-name

ARGUMENTS

       The full name of the directory.

DESCRIPTION

       The  delete  directory  command	deletes a directory.  The directory cannot contain any object entries, soft links, or child pointers.  The
       master replica must be the only remaining replica in the cell.  Use the delete replica command if you need to remove read-only replicas.

   Privilege Required
       You must have delete permission to the directory and write permission to the clearinghouse that stores the master replica of the directory.
       The  server  principal  needs  administer  permission  to the parent directory or delete permission to the child pointer that points to the
       directory you intend to delete.

NOTE

       This command is replaced at Revision 1.1 by the dcecp command and may not be provided in future releases of DCE.

EXAMPLE

       The following command deletes the directory /.:/eng from the namespace: cdscp> delete directory /.:/eng

RELATED INFORMATION

       Commands: create directory(1m), delete replica(1m), list directory(1m), set directory(1m), set directory to skulk(1m), show directory(1m)

															      delete directory(1m)