08-02-2002
Handling a directory this large is going to require very careful attention to performance considerations. I usually hold my tongue when I see someone suggest the -exec option on a "find" command. But in this case, it will be a very large problem. A command like:
find /path/directory/ -mtime +29 -exec ls {} \;
is going to launch one "ls" process for each file. In this case, that is way too many. We need to get as many files on the "ls" (or "rm") command line as possible. That way, a single process will be handling dozens or maybe hundreds of files at once. We can do this with:
cd /path/directory
find . -mtime +29 -print | xargs ls -d
(I always use -d in a case like this in case the "find" output a subdirectory.) By cd'ing to the directory first and then use "." in the "find" command, we shorten the pathname that find will output. This means that xargs can collect more of them for each "ls" process that it invokes.
Using xargs is always better than -exec, but with a small number of files, it's not a big deal.
Peter may have meant "ls", the OP did request help obtaining such a listing. But can anyone read a listing that is 600,000 lines long? There is really no point to such a listing.
Any shell script written to process these files will also need careful attention to performance.
This:
for each_file in /path/directory/*
is not going to work. The shell will try to expand that asterisk and it will fail. Something like this:
#! /usr/bin/ksh
cd /path/directory
find . -print | while read each_file ; do
will work, but whatever the loop does it must be carefully coded. It must use only shell built-in commands and maybe some pre-launched co-processes. Invoking even 4 or 5 processes per loop will mean millions of total processes. Such a script would take a very long time to run.
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hi
I have a problem. One of the directories in /var, on one of my servers, has filled up with loads of files. I have tried to list the directory to see what is in it, but the 'ls' command just hangs. Same thing happens when I try to use 'rm' to remove files from that directory.
I can see what... (2 Replies)
Discussion started by: m_smith
2 Replies
2. Shell Programming and Scripting
I have a large list of filenames from an Excel sheet, which I then translate into a simple text file. I'd like to use this list, which contains various file extensions , to archive these files and then remove them recursively through multiple directories and subdirectories. So far, it looks like... (5 Replies)
Discussion started by: fxvisions
5 Replies
3. Shell Programming and Scripting
I have something a bit tricky here and not sure how to go about fixing it.
I am doing a ls on a directory to obtain a list of databases within a directory and assigning it to a string as you can see from below. The problem is there is a folder in there called apache. I do not want this to be... (3 Replies)
Discussion started by: LRoberts
3 Replies
4. UNIX for Dummies Questions & Answers
done some homework on this--
after i remove up to and including the ) i want to take newfile.txt and use that list to remove the files from a file in my the directory pwd
i have a input.txt file
cat input,txt
1)mary.jpg
12)john.jpg
100)frankkfkdf .jpg
i want to remove the characters in the... (1 Reply)
Discussion started by: plener
1 Replies
5. Shell Programming and Scripting
Hi
I want to remove the first row in a list of files and am trying to do via the following.
I've tried various quotes to redirect the modifed file to a newly named version of itself but no joy.
Can you help?
> for i in 'ls A*'; do sed '1d' $i > $i"_complete"; done
bash: $i"_complete":... (4 Replies)
Discussion started by: ksexton
4 Replies
6. Shell Programming and Scripting
Dear all,
I have a dir structure like
main_dir
At_nn Ag_js Nf_hc ....
mcd32 mgd43... mcd32 mgd43... mcd32 mgd43...
and each subdir (e.g. mcd32, mgd43) contains files.
Now, i... (15 Replies)
Discussion started by: yogeshkumkar
15 Replies
7. Shell Programming and Scripting
I need to write a script to :
list files with *.i extension in a directory and all its subdirectories + 30days old, save it in a file and then remove (2 Replies)
Discussion started by: lena keung
2 Replies
8. Solaris
Hi,
I am using solaris 10 OS.Please help me out with the commands needed in below two scenarios.
1)How to delete the existing files in the tar file.
suppose i have a main tarfile named application.tar and it contains a file called ingres.tar.
what is the command to remove ingres.tar... (2 Replies)
Discussion started by: muraliinfy04
2 Replies
9. Linux
hi,
i have a cache file with below file list
more gtda_11.cache
GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv
GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv
GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv
GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies
10. Shell Programming and Scripting
Hi folks,
I have a list of variables as follows:
CDBTEST1
messdba1
sat11cru1
s12tgts1
sa12ss1
I need to remove the last '1' so I can use the remaining variables in a for loop:
CDBTEST
messdba
sat11cru
s12tgts
sa12ss
Something like this: (3 Replies)
Discussion started by: jonnyd
3 Replies
LEARN ABOUT HPUX
remove_directory
remove directory(1m) remove directory(1m)
NAME
remove directory - Removes a value from a set-valued or single-valued attribute (including application-defined attributes) of a directory
SYNOPSIS
cdscp remove directory directory-name attribute-name [= attribute-value]
ARGUMENTS
The full name of the directory. The name of a particular attribute. Specify only one attribute at a time. See the cds_attributes file
for the list of attributes and corresponding data types that your application uses. The value of a particular attribute. The value of an
application-defined attribute is dependent on the type of attribute.
DESCRIPTION
The remove directory command removes a value from a set-valued or single-valued attribute (including application-defined attributes) of a
directory. If you do not specify a value, the command removes the entire attribute. This command can delete attributes created by the add
directory and set directory commands. Usually this task is performed through the client application. See the OSF DCE Administration Guide
for more information about attributes.
Privilege Required
You must have write permission to the directory.
NOTE
This command is replaced at Revision 1.1 by the dcecp command and may not be provided in future releases of DCE.
EXAMPLE
To remove the value 1 from the user-defined, set-valued attribute dirregion of a directory named /.:/sales, follow these steps: Read the
cds_attributes file to check that the attribute dirregion is listed, as shown in the following display:
OID LABEL SYNTAX 1.3.22.1.3.66 dirregion small Enter the following command to remove the value 1 from the
attribute dirregion. cdscp> remove directory /.:/sales dirregion = 1
RELATED INFORMATION
Commands: add directory(1m), list directory(1m), set directory(1m), set directory to skulk(1m), show directory(1m)
Books: OSF DCE Administration Guide
remove directory(1m)