Shell Programming and Scripting

Hi RudiC, Thank you for that hint, I was not aware of this command. I have now added the following script to split the files:

Code:

#!/usr/bin/bash
fspec=2015-11-09.querylogs
num_files=10

# Work out lines per file.
total_lines=$(wc -l <${fspec})
((lines_per_file = (total_lines + num_files - 1) / num_files))

# Split the actual file, maintaining lines.
split --lines=${lines_per_file} ${fspec} new.

Now there are 10 files, how to I start 10 scripts to run through these files?

Output files:

Code:

Total lines     = 27622480
Lines  per file = 2762248
   2762248 new.aa
   2762248 new.ab
   2762248 new.ac
   2762248 new.ad
   2762248 new.ae
   2762248 new.af
   2762248 new.ag
   2762248 new.ah
   2762248 new.ai
   2762248 new.aj
  27622480 total

Back in post #14 I showed you a way to get rid of lots of extra processes you are invoking for each line you read. In post #16 you said my suggestion improved performance of your script by a factor of 5. But, in your latest script in post #24 in this thread, you have thrown away that improvement and added several more duplicated invocations of geoiplookup piped through various combinations of awk, cut, awk again, and tr. Why? If you would have taken that earlier advice and applied it to the geoiplookup invocations as well, you would easily get an improvement by at least a factor of 10.

Your script in post #24 has two distinct parts. The 1st 49 lines process the stuff we have been talking about in this thread. The last line processes a file that is not mentioned in the rest of the script. Why not move that command to the start of your script and run it asynchronously (and add a wait at the end of your script if your script should not be allowed to terminate before that command completes)?

How many cores can you use on your system? If you want to run thirty copies of your script simultaneously, you will probably be fighting against yourself if you don't have at least thirty cores at your disposal.

In your latest script (n post #24) there is no reference to $fspec or to any files ending with .querylogs.

If you want to run that script 10 times in parallel, are the new.* files supposed to be used as replacements for the file named tmp or for the file named DNS1_DOM or for some other input file?

What is the name of your current script?

How do you invoke your current script?

In what directory are the new.* files located?

In what directory do you invoke your current script?

Given that there are two independent actions occurring in that script, why would you want to run one of those actions 10 times instead of just running it once?