Sponsored Content
Operating Systems Linux Red Hat Unable to copy files due to many files in directory Post 302696405 by Don Cragun on Wednesday 5th of September 2012 02:36:17 AM
Old 09-05-2012
If you have a billion files in a directory, anything you try to do accessing a file in that directory is going to take a LONG time. In this case you have to read the directory twice for each file found that is less than 48 hours old. You said you're copying files, but there is nothing in the command line you listed that will copy anything. The command line you gave will search for groups of names of files that have been modified within the last 48 hours and then read the directory again for each group to print the names of those files.

The time needed to run this pipeline would probably be cut in half (while producing the same results except that the order in which the files are printed may be different) if you used the command:
Code:
find . -mtime -2

instead of the command:
Code:
find . -mtime -2 -print | xargs ls -d

Note that if a lot of files have been changed in the last two days, neither of these will give you a sorted list of the files that have changed.

Note also that if any filenames in the file hierarchy rooted in the current directory contain a space, tab, or newline character, the command you have using xargs will fail, and the output from the replacement command I suggested may be ambiguous.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copy files from one directory to another

I need to copy about 13 Tb of data from one directory and subdirectories to the other (another mount point). If I run this as a cron, say between 10 pm and 7 am, not all of the files will be copied over. Is there a way of 'resuming' the copy the following evenings until all files are copied over? (0 Replies)
Discussion started by: hd2006
0 Replies

2. Shell Programming and Scripting

Copy files from one directory to another

Hi when copy the files from one directory to another as like below,it is tried to copy *. as a file. cp /home/rha/*. My objective is to copy all the files (don't care about case sensitive), Thanks in advance for your valuable reply. (1 Reply)
Discussion started by: HAA
1 Replies

3. Linux

Unable to view files in a particular directory under /opt

Hi Everybody, I am Unable to view files in a particular directory under /opt. But, when I reboot the server, I am able to view the files.. Its happening daily. Do u 've n e answers/suggestions. Kindly help.. :confused: (1 Reply)
Discussion started by: its.simron
1 Replies

4. UNIX for Dummies Questions & Answers

Unable to view files in a particular directory under /opt

Hi Everybody, I am Unable to view files in a particular directory under /opt. But, when I reboot the server, I am able to view the files.. Its happening daily. Do u 've n e answers/suggestions. Kindly help.. :eek: (1 Reply)
Discussion started by: its.simron
1 Replies

5. UNIX for Dummies Questions & Answers

Copy directory tree with files

Iam in the process of copying a directory with thousands of directories and files into a new directory. I need to preserve permissions, owner, group, date and timestamps, everything. Iam using AIX and would need help of writing the command whether it is cp-RP or cpio. Apprecaite your... (3 Replies)
Discussion started by: baanprog
3 Replies

6. UNIX for Dummies Questions & Answers

How to copy all files into the same directory

Dear All, Again I have another simple question. :confused: I want to write a csh which can copy all files of a current directory with a new name in the same directory, I mean: If I have tree bird apple as files in a directory I want to give ,say number 007 as argument to my csh and it copies... (3 Replies)
Discussion started by: dreamer0085
3 Replies

7. UNIX for Dummies Questions & Answers

Copy files into another directory

I have a folder will a lot of documents (pdf, xls, doc etc.) which users have uploaded but only 20% of them are currently linking from my html files. So my goal is to copy only the files which are linked in my html files from my Document directory into another directory. Eg: My documents exist... (5 Replies)
Discussion started by: ankitha
5 Replies

8. Shell Programming and Scripting

Copy the files in directory and sub folders as it is to another directory.

How to copy files from one directory to another directory with the subfolders copied. If i have folder1/sub1/sub2/* it needs to copy files to folder2/sub1/sub2/*. I do not want to create sub folders in folder2. Can copy command create them automatically? I tried cp -a and cp -R but did... (4 Replies)
Discussion started by: santosh2626
4 Replies

9. Shell Programming and Scripting

How to copy a directory without specific files?

Hi I need to copy a huge directory with thousands of files onto another directory but without *.WMV files (and without *.wmv - perhaps we need to use *.). Pls advise how can I do that. Thanks (17 Replies)
Discussion started by: reddyr
17 Replies

10. Solaris

Unable to list files in a directory

Hi have a system running solaris with a mount point running 58% capacity used, but unfortunately when I type ls -lrtin a specific directory, it returns: ls: Not enough space. I suspect there are millions of files in this directory. So what I did is to create a script like this: touch -mt... (13 Replies)
Discussion started by: fretagi
13 Replies
cpio(1) 						      General Commands Manual							   cpio(1)

Name
       cpio - copy file archives in and out

Syntax
       cpio -i [ -C ] [keys] [patterns]

       cpio -o [keys]

       cpio -p [keys] directory

Description
       The  command  is  a filter designed to let you copy files to or from an archive.  The command differs from the command in that lets you ar-
       chive any kind of file, while is limited to program object files.

Options
       -i   Copies files that match the specified pattern.  If the pattern is not specified, copies in all files.  Extracts files from	the  stan-
	    dard  input,  which  is  assumed to be the product of a previous cpio -o, and places them into the user's current directory tree.  For
	    files with the same name, the newer file replaces the older file unless the -u option is used.

	    Only files with names that match patterns are selected.  The patterns are specified using the notation for names described in In  pat-
	    terns,  the  slash	for  directories (/) is included in searches using meta-characters.  For example, suppose the archive contains the
	    file and the pathname information in the archive indicates that the directory below contains the file This command copies  both  files
	    into the user's current directory:
	    cpio -i *p < /dev/rmt0l

	    Multiple  patterns	may be specified and if no patterns are specified, the default for patterns is * (that is, select all files).  The
	    extracted files are conditionally created and copied into the current directory tree based upon the options described below.  The com-
	    mand has three function keys, each with its own set of options.

       -C   Specifies  compatability mode for reading archives created with the old style header format.  Use this option to copy in archives gen-
	    erated by pre-V4.2 This option is valid only for copy-in (-i) operations.

       -o   Copies out the specified files.  Reads the standard input to obtain a list of path names and copies those files onto the standard out-
	    put together with path name and status information.

       -p   Copies  files  into  the specified destination directory, which must already exist.  Reads the standard input to obtain a list of path
	    names of files that are conditionally created.  This list of files is copied into  the  destination  directory  tree  based  upon  the
	    options used.  For files with the same name, the newer file replaces the older file unless the -u option is used.

Keys
       6    Processes a file with the UNIX System Sixth Edition format.

       a    Retains  original  access  times  of  input  files, and can be used with -o and -p.  Normally, the read(s) used in the copy update the
	    copied file's access time.

       B    Determines input/output is to be blocked 5,120 bytes to the record.  This option is meaningful only with data directed to or from or

       b    Swaps both bytes and halfwords.

       c    Creates header information in ASCII format and can be used with -i and -o.

       d    Creates subdirectories, as needed, below the specified destination directory.

       f    Copies all files except those that match the specified pattern.

       k    Enables symbolic link handling and is used with the -i, -o, and -p options.

       l    Creates links wherever possible.

       m    Retains modification time for each copied file.  This option does not work on directories or symbolic links that are being copied; the
	    directory is always reset to show the access time when the copy was made.

       r    Interactively renames files.  If you respond with a null line, the file is skipped (not copied).  Use only with the -i option.

       s    Swaps bytes while copying files in.

       S    Swaps half words while copying files in.

       t    Prints a table of contents of the input (no files are created).

       u    Copies files unconditionally.  (Otherwise, an older file will not replace a newer file with the same name).

       v    Displays  detailed	(verbose)  information as it copies and/or creates file.  When used with the t option, the table of contents looks
	    like the output of an ls -l command.  For further information, see

Examples
       This example shows how to copy the contents of the user's current directory into an archive.
       ls | cpio -o  > /dev/rmt0l
       This example shows how to duplicate a directory hierarchy.
       mkdir ~phares/newdir
       cd ~phares/olddir
       find . -print | cpio -pdl ~phares/newdir
       This example shows how to copy all files and directories with names containing the characters "chapter" in user smith's home directory  and
       underlying directories.
       find ~smith -name '*chapter*' -print | cpio -o > /dev/rmt0h
       This example shows the results of using the r option with the -i function key.
       ls | cpio -ir > ~smith/newdir
       Rename <file1>
       newnamefile1
       Rename <file2>
       <RETURN>
       Skipped
       Rename <file3>
       newnamefile3
       In  some  cases,  the  -cpio  option of the command can be used more effectively than pipes and redirects using For instance, the following
       example
       find . -print | cpio -oB > /dev/rmt0l
       can be handled more efficiently by:
       find . -cpio /dev/rmt0l
       To copy the contents of a directory (with symbolic link handling enabled) to the tape drive, type:
       ls | cpio -ok > /dev/rmt0h
       To restore the archived files back into a directory, type:
       cpio -ik < /dev/rmt0h
       The following example moves files, including symbolic links, from an old directory to a new directory:
       mkdir ~craig/newdir
       cd ~craig/olddir
       ls | cpio -pdk ~craig/newdir

Restrictions
       Pathnames are restricted to 128 characters.

       When there are too many unique linked files, the program runs out of memory and cannot trace them.  In this case,  linking  information	is
       lost.

       Only the superuser can copy special files.

See Also
       ar(1), find(1), cpio(5)

																	   cpio(1)
All times are GMT -4. The time now is 09:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy