Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Too many files to list / remove Post 25662 by Perderabo on Friday 2nd of August 2002 08:06:46 AM
Old 08-02-2002
Handling a directory this large is going to require very careful attention to performance considerations. I usually hold my tongue when I see someone suggest the -exec option on a "find" command. But in this case, it will be a very large problem. A command like:

find /path/directory/ -mtime +29 -exec ls {} \;

is going to launch one "ls" process for each file. In this case, that is way too many. We need to get as many files on the "ls" (or "rm") command line as possible. That way, a single process will be handling dozens or maybe hundreds of files at once. We can do this with:

cd /path/directory
find . -mtime +29 -print | xargs ls -d

(I always use -d in a case like this in case the "find" output a subdirectory.) By cd'ing to the directory first and then use "." in the "find" command, we shorten the pathname that find will output. This means that xargs can collect more of them for each "ls" process that it invokes.

Using xargs is always better than -exec, but with a small number of files, it's not a big deal.

Peter may have meant "ls", the OP did request help obtaining such a listing. But can anyone read a listing that is 600,000 lines long? There is really no point to such a listing.

Any shell script written to process these files will also need careful attention to performance.
This:

for each_file in /path/directory/*

is not going to work. The shell will try to expand that asterisk and it will fail. Something like this:

#! /usr/bin/ksh
cd /path/directory
find . -print | while read each_file ; do

will work, but whatever the loop does it must be carefully coded. It must use only shell built-in commands and maybe some pre-launched co-processes. Invoking even 4 or 5 processes per loop will mean millions of total processes. Such a script would take a very long time to run.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Can't list or remove files from a directory

Hi I have a problem. One of the directories in /var, on one of my servers, has filled up with loads of files. I have tried to list the directory to see what is in it, but the 'ls' command just hangs. Same thing happens when I try to use 'rm' to remove files from that directory. I can see what... (2 Replies)
Discussion started by: m_smith
2 Replies

2. Shell Programming and Scripting

read list of filenames from text file and remove these files in multiple directories

I have a large list of filenames from an Excel sheet, which I then translate into a simple text file. I'd like to use this list, which contains various file extensions , to archive these files and then remove them recursively through multiple directories and subdirectories. So far, it looks like... (5 Replies)
Discussion started by: fxvisions
5 Replies

3. Shell Programming and Scripting

Remove a value from a string list...

I have something a bit tricky here and not sure how to go about fixing it. I am doing a ls on a directory to obtain a list of databases within a directory and assigning it to a string as you can see from below. The problem is there is a folder in there called apache. I do not want this to be... (3 Replies)
Discussion started by: LRoberts
3 Replies

4. UNIX for Dummies Questions & Answers

remove characters from list of files

done some homework on this-- after i remove up to and including the ) i want to take newfile.txt and use that list to remove the files from a file in my the directory pwd i have a input.txt file cat input,txt 1)mary.jpg 12)john.jpg 100)frankkfkdf .jpg i want to remove the characters in the... (1 Reply)
Discussion started by: plener
1 Replies

5. Shell Programming and Scripting

Remove first row in a list of files

Hi I want to remove the first row in a list of files and am trying to do via the following. I've tried various quotes to redirect the modifed file to a newly named version of itself but no joy. Can you help? > for i in 'ls A*'; do sed '1d' $i > $i"_complete"; done bash: $i"_complete":... (4 Replies)
Discussion started by: ksexton
4 Replies

6. Shell Programming and Scripting

Remove files from subdirectories given a list of filenames

Dear all, I have a dir structure like main_dir At_nn Ag_js Nf_hc .... mcd32 mgd43... mcd32 mgd43... mcd32 mgd43... and each subdir (e.g. mcd32, mgd43) contains files. Now, i... (15 Replies)
Discussion started by: yogeshkumkar
15 Replies

7. Shell Programming and Scripting

List files with *.i extension in a directory and all its subdirectories + 30days old then remove

I need to write a script to : list files with *.i extension in a directory and all its subdirectories + 30days old, save it in a file and then remove (2 Replies)
Discussion started by: lena keung
2 Replies

8. Solaris

Command to remove existing files in the tar files in Solaris 10

Hi, I am using solaris 10 OS.Please help me out with the commands needed in below two scenarios. 1)How to delete the existing files in the tar file. suppose i have a main tarfile named application.tar and it contains a file called ingres.tar. what is the command to remove ingres.tar... (2 Replies)
Discussion started by: muraliinfy04
2 Replies

9. Linux

Remove matching files from a list

hi, i have a cache file with below file list more gtda_11.cache GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies

10. Shell Programming and Scripting

Remove last '1' in list of variables

Hi folks, I have a list of variables as follows: CDBTEST1 messdba1 sat11cru1 s12tgts1 sa12ss1 I need to remove the last '1' so I can use the remaining variables in a for loop: CDBTEST messdba sat11cru s12tgts sa12ss Something like this: (3 Replies)
Discussion started by: jonnyd
3 Replies
rmdir(1)						      General Commands Manual							  rmdir(1)

NAME
rmdir - Removes a directory SYNOPSIS
rmdir [-p] [-s] directory... STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: rmdir: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
Removes all directories in a path name. For each directory argument, the directory entry it names is removed. If the directory argument includes more than one path name component, effects equivalent to the following command occur: rmdir -p $(dirname directory) That is, rmdir recursively removes each directory in the path name. OPERANDS
The path name of an empty directory to be removed. DESCRIPTION
The rmdir command removes a directory from the system. The directory must be empty before you can remove it, and you must have write per- mission in its parent directory. Use the ls -al command to see if a directory is empty. If a directory and a subdirectory of that directory are specified in a single invocation of rmdir, the subdirectory must be specified before the parent directory so that the parent directory will be empty when rmdir tries to remove it. RESTRICTIONS
A directory must be empty before you can remove it, and you must have write permission in its parent directory. If the -p option is used, all directories in the path must be empty except for the directory being recursively removed. EXIT STATUS
The following exit values are returned: Each directory specified by directory operand was successfully removed. An error occurred. EXAMPLES
To empty and remove a directory, enter: rm mydir/* mydir/.* rmdir mydir This removes the contents of mydir, then removes the empty directory. The rm command displays an error message about trying to remove the directories . (dot) and .. (dot dot), and then rmdir removes them. Note that rm mydir/* mydir/.* first removes files with names that do not begin with a (dot), then those with names that do begin with a (dot). You may not realize that the directory contains file names that begin with a (dot) because the ls command does not normally list them unless you use the -a option to see the files whose names begin with a (dot). To remove all of the directories in the path name a/b/c, enter: rmdir -p a/b/c Use a command like this one if directory a in the current directory is empty except that it contains a directory b and a/b is empty except that it contains a directory c. ENVIRONMENT VARIABLES
The following environment variables affect the execution of rmdir: Provides a default value for the internationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari- ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments). Determines the locale for the for- mat and contents of diagnostic messages written to standard error. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: mkdir(1), ls(1), rm(1) Functions: rmdir(2), unlink(2), remove(3) Standards: standards(5) rmdir(1)
All times are GMT -4. The time now is 08:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy