Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Too many files to list / remove Post 25662 by Perderabo on Friday 2nd of August 2002 08:06:46 AM
Old 08-02-2002
Handling a directory this large is going to require very careful attention to performance considerations. I usually hold my tongue when I see someone suggest the -exec option on a "find" command. But in this case, it will be a very large problem. A command like:

find /path/directory/ -mtime +29 -exec ls {} \;

is going to launch one "ls" process for each file. In this case, that is way too many. We need to get as many files on the "ls" (or "rm") command line as possible. That way, a single process will be handling dozens or maybe hundreds of files at once. We can do this with:

cd /path/directory
find . -mtime +29 -print | xargs ls -d

(I always use -d in a case like this in case the "find" output a subdirectory.) By cd'ing to the directory first and then use "." in the "find" command, we shorten the pathname that find will output. This means that xargs can collect more of them for each "ls" process that it invokes.

Using xargs is always better than -exec, but with a small number of files, it's not a big deal.

Peter may have meant "ls", the OP did request help obtaining such a listing. But can anyone read a listing that is 600,000 lines long? There is really no point to such a listing.

Any shell script written to process these files will also need careful attention to performance.
This:

for each_file in /path/directory/*

is not going to work. The shell will try to expand that asterisk and it will fail. Something like this:

#! /usr/bin/ksh
cd /path/directory
find . -print | while read each_file ; do

will work, but whatever the loop does it must be carefully coded. It must use only shell built-in commands and maybe some pre-launched co-processes. Invoking even 4 or 5 processes per loop will mean millions of total processes. Such a script would take a very long time to run.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Can't list or remove files from a directory

Hi I have a problem. One of the directories in /var, on one of my servers, has filled up with loads of files. I have tried to list the directory to see what is in it, but the 'ls' command just hangs. Same thing happens when I try to use 'rm' to remove files from that directory. I can see what... (2 Replies)
Discussion started by: m_smith
2 Replies

2. Shell Programming and Scripting

read list of filenames from text file and remove these files in multiple directories

I have a large list of filenames from an Excel sheet, which I then translate into a simple text file. I'd like to use this list, which contains various file extensions , to archive these files and then remove them recursively through multiple directories and subdirectories. So far, it looks like... (5 Replies)
Discussion started by: fxvisions
5 Replies

3. Shell Programming and Scripting

Remove a value from a string list...

I have something a bit tricky here and not sure how to go about fixing it. I am doing a ls on a directory to obtain a list of databases within a directory and assigning it to a string as you can see from below. The problem is there is a folder in there called apache. I do not want this to be... (3 Replies)
Discussion started by: LRoberts
3 Replies

4. UNIX for Dummies Questions & Answers

remove characters from list of files

done some homework on this-- after i remove up to and including the ) i want to take newfile.txt and use that list to remove the files from a file in my the directory pwd i have a input.txt file cat input,txt 1)mary.jpg 12)john.jpg 100)frankkfkdf .jpg i want to remove the characters in the... (1 Reply)
Discussion started by: plener
1 Replies

5. Shell Programming and Scripting

Remove first row in a list of files

Hi I want to remove the first row in a list of files and am trying to do via the following. I've tried various quotes to redirect the modifed file to a newly named version of itself but no joy. Can you help? > for i in 'ls A*'; do sed '1d' $i > $i"_complete"; done bash: $i"_complete":... (4 Replies)
Discussion started by: ksexton
4 Replies

6. Shell Programming and Scripting

Remove files from subdirectories given a list of filenames

Dear all, I have a dir structure like main_dir At_nn Ag_js Nf_hc .... mcd32 mgd43... mcd32 mgd43... mcd32 mgd43... and each subdir (e.g. mcd32, mgd43) contains files. Now, i... (15 Replies)
Discussion started by: yogeshkumkar
15 Replies

7. Shell Programming and Scripting

List files with *.i extension in a directory and all its subdirectories + 30days old then remove

I need to write a script to : list files with *.i extension in a directory and all its subdirectories + 30days old, save it in a file and then remove (2 Replies)
Discussion started by: lena keung
2 Replies

8. Solaris

Command to remove existing files in the tar files in Solaris 10

Hi, I am using solaris 10 OS.Please help me out with the commands needed in below two scenarios. 1)How to delete the existing files in the tar file. suppose i have a main tarfile named application.tar and it contains a file called ingres.tar. what is the command to remove ingres.tar... (2 Replies)
Discussion started by: muraliinfy04
2 Replies

9. Linux

Remove matching files from a list

hi, i have a cache file with below file list more gtda_11.cache GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies

10. Shell Programming and Scripting

Remove last '1' in list of variables

Hi folks, I have a list of variables as follows: CDBTEST1 messdba1 sat11cru1 s12tgts1 sa12ss1 I need to remove the last '1' so I can use the remaining variables in a for loop: CDBTEST messdba sat11cru s12tgts sa12ss Something like this: (3 Replies)
Discussion started by: jonnyd
3 Replies
DH-EXEC(1)							      dh-exec								DH-EXEC(1)

NAME
dh-exec - Debhelper executable file helpers SYNOPSIS
#! /usr/bin/dh-exec src/libfoo-*.so.* debian/foo-plugins/usr/lib/foo/${DEB_HOST_MULTIARCH}/ etc/example.conf => debian/foo/etc/foo/foo.conf DESCRIPTION
dh-exec is a simple program, meant to be used as the interpreter for executable debhelper config files. It is a wrapper around the various other sub-commands (see below), and will pipe the input file through all of them in turn, using an ordering that makes most sense in the vast majority of cases. The order as of now is dh-exec-subst gets run first, followed by dh-exec-install, so that variable expansion happens before files need to be copied. ARCHITECTURE
dh-exec is built up from three layers: there is the dh-exec utility, its single entry point, the only thing one will need to call. Below that, there are the various sub-commands, such as dh-exec-subst, dh-exec-installs and dh-exec-illiterate, which are thin wrappers around the various dh-exec scripts, that make sure they only run those that need to be run. And the lowest layer are the various scripts that do the actual work. One can control which sub-commands to run, or if even more granularity is desired, one can limit which scripts shall be run, too. See below for the options! OPTIONS
--with=command[,command ...] Replace the list of sub-commands to run the input through with a custom list (where entries are separated by whitespace or commas). This option will always replace the existing list with whatever is specified. This can be used to explicitly set which sub-commands to use. The list must not include the dh-exec- prefix. Defaults to subst,install. --without=command[,command ...] Inversely to the option above, this lists all the sub-commands which should not be used. The list must not include the dh-exec- prefix. --with-scripts=script[,script ...] Replace the list of scripts to run the input through with a custom list (where entries are separated by whitespace or commas). This option will always replace the existing list with whatever is specified. This can be used to explicitly specify which scripts to use, limiting even beyond what the --with option is capable of. The list must not include the dh-exec- prefix. By default it is empty, meaning there is no filtering done, and whatever scripts the sub-commands find, will be run. --no-act Do not really do anything, but print the pipeline that would have been run instead. --list List the available sub-commands and scripts, grouped by sub-command. --help, --version Display a short help or the package version, respectively. SUB-COMMANDS dh-exec-subst Substitutes various variables (either from the environment, or from dpkg-architecture(1)). dh-exec-install An extension to dh_install(1), that supports renaming files during the copy process, using a special syntax. ENVIRONMENT
DH_EXEC_LIBDIR The directory in which the wrapped sub-commands reside. Defaults to /usr/lib/dh-exec/. DH_EXEC_SCRIPTDIR The directory in which the scripts that do the heavy work live. Defaults to /usr/share/dh-exec/. FILES
$DH_EXEC_LIBDIR/dh-exec-* The various sub-commands. $DH_EXEC_SCRIPTDIR/dh-exec-* The various scripts ran by the sub-commands. SEE ALSO
debhelper(1), dh-exec-subst(1), dh-exec-install(1) AUTHOR
dh-exec is copyright (C) 2011-2012 by Gergely Nagy <algernon@madhouse-project.org>. 2012-05-03 DH-EXEC(1)
All times are GMT -4. The time now is 11:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy