Sponsored Content
Operating Systems Linux Red Hat Unable to copy files due to many files in directory Post 302696405 by Don Cragun on Wednesday 5th of September 2012 02:36:17 AM
Old 09-05-2012
If you have a billion files in a directory, anything you try to do accessing a file in that directory is going to take a LONG time. In this case you have to read the directory twice for each file found that is less than 48 hours old. You said you're copying files, but there is nothing in the command line you listed that will copy anything. The command line you gave will search for groups of names of files that have been modified within the last 48 hours and then read the directory again for each group to print the names of those files.

The time needed to run this pipeline would probably be cut in half (while producing the same results except that the order in which the files are printed may be different) if you used the command:
Code:
find . -mtime -2

instead of the command:
Code:
find . -mtime -2 -print | xargs ls -d

Note that if a lot of files have been changed in the last two days, neither of these will give you a sorted list of the files that have changed.

Note also that if any filenames in the file hierarchy rooted in the current directory contain a space, tab, or newline character, the command you have using xargs will fail, and the output from the replacement command I suggested may be ambiguous.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copy files from one directory to another

I need to copy about 13 Tb of data from one directory and subdirectories to the other (another mount point). If I run this as a cron, say between 10 pm and 7 am, not all of the files will be copied over. Is there a way of 'resuming' the copy the following evenings until all files are copied over? (0 Replies)
Discussion started by: hd2006
0 Replies

2. Shell Programming and Scripting

Copy files from one directory to another

Hi when copy the files from one directory to another as like below,it is tried to copy *. as a file. cp /home/rha/*. My objective is to copy all the files (don't care about case sensitive), Thanks in advance for your valuable reply. (1 Reply)
Discussion started by: HAA
1 Replies

3. Linux

Unable to view files in a particular directory under /opt

Hi Everybody, I am Unable to view files in a particular directory under /opt. But, when I reboot the server, I am able to view the files.. Its happening daily. Do u 've n e answers/suggestions. Kindly help.. :confused: (1 Reply)
Discussion started by: its.simron
1 Replies

4. UNIX for Dummies Questions & Answers

Unable to view files in a particular directory under /opt

Hi Everybody, I am Unable to view files in a particular directory under /opt. But, when I reboot the server, I am able to view the files.. Its happening daily. Do u 've n e answers/suggestions. Kindly help.. :eek: (1 Reply)
Discussion started by: its.simron
1 Replies

5. UNIX for Dummies Questions & Answers

Copy directory tree with files

Iam in the process of copying a directory with thousands of directories and files into a new directory. I need to preserve permissions, owner, group, date and timestamps, everything. Iam using AIX and would need help of writing the command whether it is cp-RP or cpio. Apprecaite your... (3 Replies)
Discussion started by: baanprog
3 Replies

6. UNIX for Dummies Questions & Answers

How to copy all files into the same directory

Dear All, Again I have another simple question. :confused: I want to write a csh which can copy all files of a current directory with a new name in the same directory, I mean: If I have tree bird apple as files in a directory I want to give ,say number 007 as argument to my csh and it copies... (3 Replies)
Discussion started by: dreamer0085
3 Replies

7. UNIX for Dummies Questions & Answers

Copy files into another directory

I have a folder will a lot of documents (pdf, xls, doc etc.) which users have uploaded but only 20% of them are currently linking from my html files. So my goal is to copy only the files which are linked in my html files from my Document directory into another directory. Eg: My documents exist... (5 Replies)
Discussion started by: ankitha
5 Replies

8. Shell Programming and Scripting

Copy the files in directory and sub folders as it is to another directory.

How to copy files from one directory to another directory with the subfolders copied. If i have folder1/sub1/sub2/* it needs to copy files to folder2/sub1/sub2/*. I do not want to create sub folders in folder2. Can copy command create them automatically? I tried cp -a and cp -R but did... (4 Replies)
Discussion started by: santosh2626
4 Replies

9. Shell Programming and Scripting

How to copy a directory without specific files?

Hi I need to copy a huge directory with thousands of files onto another directory but without *.WMV files (and without *.wmv - perhaps we need to use *.). Pls advise how can I do that. Thanks (17 Replies)
Discussion started by: reddyr
17 Replies

10. Solaris

Unable to list files in a directory

Hi have a system running solaris with a mount point running 58% capacity used, but unfortunately when I type ls -lrtin a specific directory, it returns: ls: Not enough space. I suspect there are millions of files in this directory. So what I did is to create a script like this: touch -mt... (13 Replies)
Discussion started by: fretagi
13 Replies
find(1) 						      General Commands Manual							   find(1)

Name
       find - find files

Syntax
       find pathname-list  expression

Description
       The  command  recursively  descends the directory hierarchy for each pathname in the pathname-list (that is, one or more pathnames) seeking
       files that match a boolean expression written in the primaries given below.  In the descriptions, the argument n is used as a decimal inte-
       ger where +n means more than n, -n means less than n , and n means exactly n.

Options
       -atime n       Tests true if the file has been accessed in n days.

       -cpio output   Writes current file on output in the format (5120-byte records) specified in the reference page.	The output can be either a
		      file or tape device.  If output is a tape device the B key must be used to read data from the tape.

       -ctime n       Tests true if the file has been changed in n days.

       -depth	      Always true; causes descent of the directory hierarchy to be done so that all entries in a directory are acted on before the
		      directory  itself (that is, postorder instead of preorder).  This can be useful when is used with to transfer files that are
		      contained in directories without write permission.

       -exec command  Tests true if specified command returns a 0 on exit.  The end of the command must be punctuated by an escaped semicolon.	 A
		      command argument `{}' is replaced by the current pathname.

       -group gname   Tests true if group ID matches specified group name.

       -inum n	      Tests true if the file has inode number n.

       -links n       Tests true if the file has n links.

       -mount	      Tests true if the current file is on the same file system as the current starting pathname.

       -mtime n       Tests true if the file has been modified in n days.

       -name filename Tests  true  if  the  filename  argument matches the current file name.  Normal Shell argument syntax may be used if escaped
		      (watch out for `[', `?' and `*').

       -newer file    Tests true if the current file has been modified more recently than the argument file.

       -ok command    Executes specified command on standard output, then standard input is read and command executed only upon response y.

       -perm onum     Tests true if file has specified octal number.  For further information, see If onum is prefixed by a minus sign, more  flag
		      bits (017777) become significant and the flags are compared: (flags&onum)==onum.	For further information, see

       -print	      Prints current pathname.

       -size n	      Tests true if the file is n blocks long (512 bytes per block).

       -type c	      Tests  true  if  file is c type ( c = b, block special file: c, character special file: d, directory: f, plain file: l, sym-
		      bolic link: p, type port: s, type socket).

       -user uname    Tests true if file owner is login name or numeric user ID.

       The primaries may be combined using the following operators (in order of decreasing precedence):

       1)  A parenthesized group of primaries and operators (parentheses are special to the Shell and must be escaped).

       2)  The negation of a primary (`!' is the unary not operator).

       3)  Concatenation of primaries (the and operation is implied by the juxtaposition of two primaries).

       4)  Alternation of primaries (`-o' is the or operator).

Examples
       To remove all files named `a.out' or `*.o' that have not been accessed for a week:
       find / ( -name a.out -o -name '*.o' ) 
       -atime +7 -exec rm {} ;

       To find all files on the root file system type:
       find / -mount -print

       To write all the files on the root file system to tape:
       find / -mount -print -cpio /dev/rmt?h
       cpio -iBvt < /dev/rmt?h

       To find all the mount points on the root file system type:
       find / ! -mount -print

Files
See Also
       cpio(1), sh(1), test(1), cpio(5), fs(5)

																	   find(1)
All times are GMT -4. The time now is 08:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy