Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to delete a huge number of files at a time
Top Forums Shell Programming and Scripting How to delete a huge number of files at a time Post 302484256 by m.d.ludwig on Thursday 30th of December 2010 09:30:41 AM
Old 12-30-2010
@sandholm is quite correct:
Quote:
Originally Posted by sandholm
xargs is your friend.
And so is grep.

If you want to delete the files and monitor how many are left:
Code:
cd directory
ls -f1 | grep file.......\.dat$ | xargs -P 4 rm -f &
while sleep 10; do
ls -f1 | grep -c file.......\.dat
done

With a large number of files, the -f option keeps ls from sorting -- in what order the files are deleted, or counted, should not matter.

This is a quick-n-dirty script, ymmv.

---------- Post updated at 09:30 AM ---------- Previous update was at 09:26 AM ----------

If the directory only contains these files, it would be easier to:
Code:
rm -rf directory
mkdir directory

 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

Time taken for creation of a huge core file

Hi, I needed to know how I can find out the time needed for an Unix machine(HP) to create a corefile as huge as 500MB(core created either by a SEGV or a kill -6 command). An approximate figure of the time taken would be really helpful.:confused: (4 Replies)
Discussion started by: nayeem
4 Replies

2. Shell Programming and Scripting

Delete lines from huge file

I have to delete 1st 7000 lines of a file which is 12GB large. As it is so large, i can't open in vi and delete these lines. Also I found one post here which gave solution using perl, but I don't have perl installed. Also some solutions were redirecting the o/p to a different file and renaming it.... (3 Replies)
Discussion started by: rahulrathod
3 Replies

3. UNIX for Dummies Questions & Answers

Need to delete the files based on the time stamp of the file

Hi Everyone, I want to delete some files in a path based on the time stamp of the file that is i want to delete the file once in a month. Can any one help me on this? Thanks in advance (2 Replies)
Discussion started by: samudha
2 Replies

4. UNIX for Dummies Questions & Answers

Delete large number of files

Hi. I need to delete a large number of files listed in a txt file. There are over 90000 files in the list. Some of the directory names and some of the file names do have spaces in them. In the file, each line is a full path to a file: /path/to/the files/file1 /path/to/some other/files/file 2... (4 Replies)
Discussion started by: inakajin
4 Replies

5. Shell Programming and Scripting

search a number in very very huge amount of data

Hi, I have to search a number in a very long listing of files.the total size of the files in which I have to search is 10 Tera Bytes. How to search a number in such a huge amount of data effectively.I used fgrep but it is taking many hours to search. Is there any other feasible solution to... (3 Replies)
Discussion started by: vsachan
3 Replies

6. Shell Programming and Scripting

Shell Script to delete files within a particular time frame under multiple sub folders

Greetings! I'm looking for starting information for a shell script. Here's my scenario: I have multiple folders(100) for example: /www/test/applications/app1/logs /www/test/applications/app2/logs Within these folders there are log files files that need to be deleted after a month. ... (3 Replies)
Discussion started by: whysolucky
3 Replies

7. Shell Programming and Scripting

Search and replace ---A huge number of files

Hello Friends, I have the below scenario in my current project. Suggest me which tool ( perl,python etc) is best to this scenario. Or should I go for Programming language ( C/Java ).. (1) I will be having a very big file ( information about 200million subscribers will be stored in it ). This... (5 Replies)
Discussion started by: panyam
5 Replies

8. Shell Programming and Scripting

Split a folder with huge number of files in n folders

We have a folder XYZ with large number of files (>350,000). how can i split the folder and create say 10 of them XYZ1 to XYZ10 with 35,000 files each. (doesnt matter which files go where). (12 Replies)
Discussion started by: AlokKumbhare
12 Replies

9. UNIX for Beginners Questions & Answers

Count the number of files to delete doesnt match

Good evening, need your help please Need to delete certain files before octobre 1 2016, so need to know how many files im going to delete, for instance ls -lrt file_20160*.lis!wc -l but using grep -c to another file called bplist which contains the list of all files backed up doesn match... (7 Replies)
Discussion started by: alexcol
7 Replies

10. Shell Programming and Scripting

Help for Count number of files in certain time

Chaps, I need to count number of files in a remote directory from Linux (FreeBSD) as if 10 trace files (log files) been generated within 5min of time. So this is the script then I can setup a monitoring. I came across with ls -1 \ip\d:\Logs | wc -l but then what else requires to check time... (8 Replies)
Discussion started by: samwijekoon
8 Replies

LEARN ABOUT V7

zgrep

ZGREP(1)						      General Commands Manual							  ZGREP(1)

NAME

       zgrep - search possibly compressed files for a regular expression

SYNOPSIS

       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION

       Zgrep  invokes  grep  on  compressed  or  gzipped  files.   These  grep	options  will  cause  zgrep  to  terminate  with  an  error  code:
       (-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*).  All other options specified are passed directly to grep.  If no file is specified,  then  the
       standard input is decompressed if necessary and fed to grep.  Otherwise the given files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program to be invoked.

EXIT CODE

       2 - An option that is not supported was specified.

AUTHOR

       Charles Levert (charles@comm.polymtl.ca)

SEE ALSO

       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

																	  ZGREP(1)
All times are GMT -4. The time now is 02:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy