Shell Programming and Scripting

Maybe something like :

Code:

eval egrep -e '$regex' *.csv >/tmp/regex_csv.out

with a little more parsing could get the info as well.
This would only generate one *.out file, but the info would still be in

If i/o (hard drive access, for example) is the bottleneck, and if all the files are on the same device, then your simultaneous instances of the script will be contending for the same limited i/o resource, which will likely result in a drop in performance.

Regardless of the bottleneck, using find with exec-+ will involve much less process creation -- which can be expensive if your files are many and a substantial portion of the work if they are also small -- thereby speeding things up (perhaps substantially).

Code:

regex=abx
find . -type f -name '*.csv' -exec awk -v re="$regex" '
    FNR == 1 { close(f); f=FILENAME".out"  }
    $6 ~ re { print $1, $3, $4 > f }
' {} +

Regards,
Alister