Shell Programming and Scripting

Hi All,

I have some 30,000 files in one directory. The files look like this:

Code:

computer
networks
router
wire

I want to remove those files which have less than 200 lines for example in the above file, the number of lines is 4.

I am trying something like this:

Code:

find /path/to/dir -type f -size -200c -exec rm {}  \;

I am not sure whether this is the correct way of deleting files with 200 lines. I am using Linux with BASH.

Try: (make a backup first )

Code:

find . -type f | xargs -i bash -c 'if [ $(wc -l {}|cut -d" " -f1) -lt 200 ]; then rm -f {}; fi'

Code:

find . -type f | while read
do
    (($(cat $REPLY | /usr/compat/linux/usr/bin/wc -l) < 200)) && rm -vf "$REPLY"
done

And another one:

Code:

find . -type f -exec sh -c '
  [ $(
        wc -l < "$1"
		) -lt 200 ] &&
          rm  -- "$1"
		  ' inline {} \;

While that solution is just fine for most cases, it doesn't scale very well. For 30,000 files it will need to fork-exec 1 find + 1 xargs + 30,000 bash + 30,000 wc + 30,000 cut + somewhere between 0 to 30,000 rm processes.

I used the following script to generate some test files in an empty directory:

Code:

$ ./create.sh 30000
$ cat create.sh
#!/bin/sh

jot 200 > 200
i=$1
while [ $i -gt 0 ]; do
    > empty.$i
    i=$((i-1))
done

Except for the file named "200", all files are empty.

On my 4 yr old laptop, even when all but one file has zero bytes, and after using redirection to eliminate the need for cut (no filename in wc's output), it takes a while:

Code:

$ cat bartus.sh
#!/bin/sh

find "$1" -type f | xargs -I {} bash -c 'if [ $(wc -l < {}) -lt 200 ]; then rm -f {}; fi'

Code:

$ time ./bartus.sh test

real    6m13.537s
user    1m15.476s
sys     4m7.636s

The following solution is not as succinct as bartus11's, but it is much faster. Aside from find and xargs, it only fork-execs one shell and (on my system) one rm per 5000 files to delete

Code:

$ cat alister.sh
#/bin/sh

find "$1" -type f |
while read f; do
	i=0
	while read line; do
		i=$((i+1))
		[ $i -eq 200 ] && continue 2
	done < "$f"
	printf %s\\n "$f"
done |
xargs rm -f

The run-time difference is substantial:

Code:

$ time ./alister.sh test

real    0m8.800s
user    0m2.287s
sys     0m5.345s

Over 6 minutes versus 9 seconds, to process 30,001 files of which 30,000 are empty and the remaining file only has 200 lines totaling just 692 bytes.

Note that as the files grow the disparity between the approaches will only increase. wc in the original solution must read the files in their entirey whereas the while-read loop in my alternative aborts a file's processing if the threshold line count (200) is reached. If the files become large, so do the i/o savings.

Again, I realize that bartus11's solution is perfectly fine for 99.9% of cases; I'm not criticizing it. My intent is only to show that there's a lot to be gained should a 0.1% situation arise.

Regards,
Alister

Very good point again alister!
Going to the next file right after the line limit is reached is definitely better.