Shell Programming and Scripting

(sem is part of parallel) You may want to limit the number of files running at the same time to avoid eating the system when other users want resources:

Code:

for dir in `find /home -type d -name "Folder*"`; do
   sem -j 10 first $dir
done 
sem

for cpu intensive operations sem (part of parallel) can control the number of cpu cores being used. Or limit access to a resource like a semaphore, hence the name. This example limits sem to the number of available cores on the system

Code:

for dir in `find /home -type d -name "Folder*"`
do
  sem -j+0  first $fname
done
sem

sem on the last line wait for all the other sem invocations to complete.

parallel is perl, so it runs on systems with perl 5.8 or higher

http://ftp.gnu.org/gnu/parallel/

Your program already does use multiple processes -- the tr happens simultanelusly -- but it's a bit of a waste really, since doing it in that fashion isn't faster than doing it inside awk.

awk is hardly a one-trick pony, you can run it once here to replace everything you've been doing by running awk, tr, and echo 10,000 times apiece. Since there is a large cost to running small programs over and over, this will speed up performance a lot.

Perhaps something like this:

Code:

awk -v OFS="" 'FNR==1 { printf("\n") }; /<.*/ , /.*<\/.*>/' *.txt

I think with thousands of files: globbing *.txt is a problem with ARG_MAX on lots of UNIX platforms. Correct me if I messed something. I thought that was why the OP used find to start with.

Actually OP didn't use find. I did, to get list of directories containing files to be processed

Hi.

The purpose of my post was to:

1) clarify the difference between the terms thread and process,

2) demonstrate that running simultaneous processes can be easy (but may require some tinkering with options, checking times for completion of a set of tasks, etc.),

I didn't know specifically about sem -- thanks Jim. Apparently version 20111122 didn't have it (at least on the install), but I see that ... GNU sem is an alias for GNU parallel --semaphore ... Other options can set the number of jobs relative to the number of CPUs or cores -- potentially very useful.

The man page for parallel contains lots of examples as well as comparisons between parallel and other utilities of the same kind, e.g. xargs, paexec, etc.

Best wishes ... cheers, drl

UPDATE:

I finally found sem in the parallel install directory, along with niceload, etc., including man pages.

( Edit 2: add sem discovery )
( Edit 1: correct minor typo )

Thanks drl.

It helped me understand the concept.

---------- Post updated at 02:06 AM ---------- Previous update was at 01:11 AM ----------

Hi Corona,

Actually i wanted the XML to be in one single line from one file.
So i was using script that way.

Can you please let me know how to get the xml into one single line from one file using the code above?

Code:

awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n'
echo -ne '\n'

Thanks,
Chetan.C

I know what you're trying to do, and thought my script did that, but it had an error:

Code:

awk -v ORS="" 'FNR==1 { printf("\n") }; /<.*/ , /.*<\/.*>/' *.txt

Try it again, please.

It ought to do everything you were trying to do in 5 lines and 3 external programs and a parallel program, in one line with one program, faster...