While that solution is just fine for most cases, it doesn't scale very well. For 30,000 files it will need to fork-exec 1 find + 1 xargs + 30,000 bash + 30,000 wc + 30,000 cut + somewhere between 0 to 30,000 rm processes.
I used the following script to generate some test files in an empty directory:
Except for the file named "200", all files are empty.
On my 4 yr old laptop, even when all but one file has zero bytes, and after using redirection to eliminate the need for cut (no filename in wc's output), it takes a while:
The following solution is not as succinct as bartus11's, but it is much faster. Aside from find and xargs, it only fork-execs one shell and (on my system) one rm per 5000 files to delete
The run-time difference is substantial:
Over 6 minutes versus 9 seconds, to process 30,001 files of which 30,000 are empty and the remaining file only has 200 lines totaling just 692 bytes.
Note that as the files grow the disparity between the approaches will only increase. wc in the original solution must read the files in their entirey whereas the while-read loop in my alternative aborts a file's processing if the threshold line count (200) is reached. If the files become large, so do the i/o savings.
Again, I realize that bartus11's solution is perfectly fine for 99.9% of cases; I'm not criticizing it. My intent is only to show that there's a lot to be gained should a 0.1% situation arise.
Regards,
Alister
These 2 Users Gave Thanks to alister For This Post:
I am having a two files and different days, and this is example:
file1: 06.09.2017.
abcd
123
file2: 07.09.2017.
abcd
1234
So what I want is that file2 with today's date contains only 1234, so where is a problem you would ask?
Problem is here that I put these commands into routers,. and... (3 Replies)
I have some text files in a folder named ff as follows. I need to delete the lines (in-place editing)in these files based on another file aa.txt.
32bm.txt:
249 253 A P - 0 0 8 0, 0.0 6,-1.4 0, 0.0 2,-0.4 -0.287 25.6-102.0 -74.4 161.1 37.1 13.3 10.9
250... (2 Replies)
Hi,
I have a process which creates almost 200K files. Each file ranging from 1kb to 5kb. This is almost 2GB of data in all files.
I have a requirement where the business needs only 100MB worth of files.
Is there a way to get files around 100MB (doesn't have to be exactly 100MB) from all the... (2 Replies)
Simply, I have a directory of text files and I need to delete every line in every file containing a specific string. I want to write the modified files to an empty sub directory.
I can't seem to get the sed command to delete the lines containing the string, and not just the string, in other... (3 Replies)
I want just delete the first 3 lines of files,nothing else to do,
but sed -i.bak '1,3d' *txt
seems very slow for huge files, and I tried sed -i.bak '1,3d;4q' *.txt, remaining only one line, is there any suggestion? (3 Replies)
Hello,
I have a group of text files with many lines in each file.
I need to delete all the lines in each and only leave 2 lines in each file. (3 Replies)
hi all,
i have enabled audit in our server it is working fine,but now i want to reduce the space with out removing audit.i.e..i want to delete some lines from audit file. here if i use vi editer, audit is not geting up.
i want to delete the data i.e..logs for every 6 days in audit file
plz... (3 Replies)
Hi
I want to delete first five and last five lines in text files without opening the file and also i want to keep the same file name for all the files.
Thanks in advance!!!
Ragav (10 Replies)