Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Too many files to list / remove Post 25662 by Perderabo on Friday 2nd of August 2002 08:06:46 AM
Old 08-02-2002
Handling a directory this large is going to require very careful attention to performance considerations. I usually hold my tongue when I see someone suggest the -exec option on a "find" command. But in this case, it will be a very large problem. A command like:

find /path/directory/ -mtime +29 -exec ls {} \;

is going to launch one "ls" process for each file. In this case, that is way too many. We need to get as many files on the "ls" (or "rm") command line as possible. That way, a single process will be handling dozens or maybe hundreds of files at once. We can do this with:

cd /path/directory
find . -mtime +29 -print | xargs ls -d

(I always use -d in a case like this in case the "find" output a subdirectory.) By cd'ing to the directory first and then use "." in the "find" command, we shorten the pathname that find will output. This means that xargs can collect more of them for each "ls" process that it invokes.

Using xargs is always better than -exec, but with a small number of files, it's not a big deal.

Peter may have meant "ls", the OP did request help obtaining such a listing. But can anyone read a listing that is 600,000 lines long? There is really no point to such a listing.

Any shell script written to process these files will also need careful attention to performance.
This:

for each_file in /path/directory/*

is not going to work. The shell will try to expand that asterisk and it will fail. Something like this:

#! /usr/bin/ksh
cd /path/directory
find . -print | while read each_file ; do

will work, but whatever the loop does it must be carefully coded. It must use only shell built-in commands and maybe some pre-launched co-processes. Invoking even 4 or 5 processes per loop will mean millions of total processes. Such a script would take a very long time to run.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Can't list or remove files from a directory

Hi I have a problem. One of the directories in /var, on one of my servers, has filled up with loads of files. I have tried to list the directory to see what is in it, but the 'ls' command just hangs. Same thing happens when I try to use 'rm' to remove files from that directory. I can see what... (2 Replies)
Discussion started by: m_smith
2 Replies

2. Shell Programming and Scripting

read list of filenames from text file and remove these files in multiple directories

I have a large list of filenames from an Excel sheet, which I then translate into a simple text file. I'd like to use this list, which contains various file extensions , to archive these files and then remove them recursively through multiple directories and subdirectories. So far, it looks like... (5 Replies)
Discussion started by: fxvisions
5 Replies

3. Shell Programming and Scripting

Remove a value from a string list...

I have something a bit tricky here and not sure how to go about fixing it. I am doing a ls on a directory to obtain a list of databases within a directory and assigning it to a string as you can see from below. The problem is there is a folder in there called apache. I do not want this to be... (3 Replies)
Discussion started by: LRoberts
3 Replies

4. UNIX for Dummies Questions & Answers

remove characters from list of files

done some homework on this-- after i remove up to and including the ) i want to take newfile.txt and use that list to remove the files from a file in my the directory pwd i have a input.txt file cat input,txt 1)mary.jpg 12)john.jpg 100)frankkfkdf .jpg i want to remove the characters in the... (1 Reply)
Discussion started by: plener
1 Replies

5. Shell Programming and Scripting

Remove first row in a list of files

Hi I want to remove the first row in a list of files and am trying to do via the following. I've tried various quotes to redirect the modifed file to a newly named version of itself but no joy. Can you help? > for i in 'ls A*'; do sed '1d' $i > $i"_complete"; done bash: $i"_complete":... (4 Replies)
Discussion started by: ksexton
4 Replies

6. Shell Programming and Scripting

Remove files from subdirectories given a list of filenames

Dear all, I have a dir structure like main_dir At_nn Ag_js Nf_hc .... mcd32 mgd43... mcd32 mgd43... mcd32 mgd43... and each subdir (e.g. mcd32, mgd43) contains files. Now, i... (15 Replies)
Discussion started by: yogeshkumkar
15 Replies

7. Shell Programming and Scripting

List files with *.i extension in a directory and all its subdirectories + 30days old then remove

I need to write a script to : list files with *.i extension in a directory and all its subdirectories + 30days old, save it in a file and then remove (2 Replies)
Discussion started by: lena keung
2 Replies

8. Solaris

Command to remove existing files in the tar files in Solaris 10

Hi, I am using solaris 10 OS.Please help me out with the commands needed in below two scenarios. 1)How to delete the existing files in the tar file. suppose i have a main tarfile named application.tar and it contains a file called ingres.tar. what is the command to remove ingres.tar... (2 Replies)
Discussion started by: muraliinfy04
2 Replies

9. Linux

Remove matching files from a list

hi, i have a cache file with below file list more gtda_11.cache GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies

10. Shell Programming and Scripting

Remove last '1' in list of variables

Hi folks, I have a list of variables as follows: CDBTEST1 messdba1 sat11cru1 s12tgts1 sa12ss1 I need to remove the last '1' so I can use the remaining variables in a for loop: CDBTEST messdba sat11cru s12tgts sa12ss Something like this: (3 Replies)
Discussion started by: jonnyd
3 Replies
INTRO(1)						     Linux Programmer's Manual							  INTRO(1)

NAME
intro - Introduction to user commands DESCRIPTION
Linux is a flavour of Unix, and as a first approximation all user commands under Unix work precisely the same under Linux (and FreeBSD and lots of other Unix-like systems). Under Linux there are GUIs (graphical user interfaces), where you can point and click and drag, and hopefully get work done without first reading lots of documentation. The traditional Unix environment is a CLI (command line interface), where you type commands to tell the com- puter what to do. That is faster and more powerful, but requires finding out what the commands are. Below a bare minimum, to get started. Login In order to start working, you probably first have to login, that is, give your username and password. See also login(1). The program login now starts a shell (command interpreter) for you. In case of a graphical login, you get a screen with menus or icons and a mouse click will start a shell in a window. See also xterm(1). The shell One types commands to the shell, the command interpreter. It is not built-in, but is just a program and you can change your shell. Every- body has her own favourite one. The standard one is called sh. See also ash(1), bash(1), csh(1), zsh(1), chsh(1). A session might go like knuth login: aeb Password: ******** % date Tue Aug 6 23:50:44 CEST 2002 % cal August 2002 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 % ls bin tel % ls -l total 2 drwxrwxr-x 2 aeb 1024 Aug 6 23:51 bin -rw-rw-r-- 1 aeb 37 Aug 6 23:52 tel % cat tel maja 0501-1136285 peter 0136-7399214 % cp tel tel2 % ls -l total 3 drwxr-xr-x 2 aeb 1024 Aug 6 23:51 bin -rw-r--r-- 1 aeb 37 Aug 6 23:52 tel -rw-r--r-- 1 aeb 37 Aug 6 23:53 tel2 % mv tel tel1 % ls -l total 3 drwxr-xr-x 2 aeb 1024 Aug 6 23:51 bin -rw-r--r-- 1 aeb 37 Aug 6 23:52 tel1 -rw-r--r-- 1 aeb 37 Aug 6 23:53 tel2 % diff tel1 tel2 % rm tel1 % grep maja tel2 maja 0501-1136285 % and here typing Control-D ended the session. The % here was the command prompt - it is the shell's way of indicating that it is ready for the next command. The prompt can be customized in lots of ways, and one might include stuff like user name, machine name, current direc- tory, time, etc. An assignment PS1="What next, master? " would change the prompt as indicated. We see that there are commands date (that gives date and time), and cal (that givs a calendar). The command ls lists the contents of the current directory - it tells you what files you have. With a -l option it gives a long listing, that includes the owner and size and date of the file, and the permissions people have for reading and/or changing the file. For example, the file "tel" here is 37 bytes long, owned by aeb and the owner can read and write it, others can only read it. Owner and permissions can be changed by the commands chown and chmod. The command cat will show the contents of a file. (The name is from "concatenate and print": all files given as parameters are concate- nated and sent to "standard output", here the terminal screen.) The command cp (from "copy") will copy a file. On the other hand, the command mv (from "move") only renames it. The command diff lists the differences between two files. Here there was no output because there were no differences. The command rm (from "remove") deletes the file, and be careful! it is gone. No wastepaper basket or anything. Deleted means lost. The command grep (from "g/re/p") finds occurrences of a string in one or more files. Here it finds Maja's telephone number. Path names and the current directory Files live in a large tree, the file hierarchy. Each has a path name describing the path from the root of the tree (which is called /) to the file. For example, such a full path name might be /home/aeb/tel. Always using full path names would be inconvenient, and the name of a file in the current directory may be abbreviated by only giving the last component. That is why "/home/aeb/tel" can be abbreviated to "tel" when the current directory is "/home/aeb". The command pwd prints the current directory. The command cd changes the current directory. Try "cd /" and "pwd" and "cd" and "pwd". Directories The command mkdir makes a new directory. The command rmdir removes a directory if it is empty, and complains otherwise. The command find (with a rather baroque syntax) will find files with given name or other properties. For example, "find . -name tel" would find the file "tel" starting in the present directory (which is called "."). And "find / -name tel" would do the same, but starting at the root of the tree. Large searches on a multi-GB disk will be time-consuming, and it may be better to use locate(1). Disks and Filesystems The command mount will attach the filesystem found on some disk (or floppy, or CDROM or so) to the big filesystem hierarchy. And umount detaches it again. The command df will tell you how much of your disk is still free. Processes On a Unix system many user and system processes run simultaneously. The one you are talking to runs in the foreground, the others in the background. The command ps will show you which processes are active and what numbers these processes have. The command kill allows you to get rid of them. Without option this is a friendly request: please go away. And "kill -9" followed by the number of the process is an imme- diate kill. Foreground processes can often be killed by typing Control-C. Getting information There are thousands of commands, each with many options. Traditionally commands are documented on man pages, (like this one), so that the command "man kill" will document the use of the command "kill" (and "man man" document the command "man"). The program man sends the text through some pager, usually less. Hit the space bar to get the next page, hit q to quit. In documentation it is custumary to refer to man pages by giving the name and section number, as in man(1). Man pages are terse, and allow you to find quickly some forgotten detail. For newcomers an introductory text with more examples and explanations is useful. A lot of GNU/FSF software is provided with info files. Type "info info" for an introduction on the use of the program "info". Special topics are often treated in HOWTOs. Look in /usr/share/doc/howto/en and use a browser if you find HTML files there. Linux 2002-08-06 INTRO(1)
All times are GMT -4. The time now is 12:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy