Shell Programming and Scripting

Hi, first, try these modifications to that code:

Code:

awk 'NR==FNR{A[$1]; next} { for(i in A) if($0~i){f=i ".txt"; print>>f; close(f) }}' infile "$file"

That seems to work like a charm.

So if I wanted to create a separate file, that has the contents of EVERYTHING else (not included in the "infile"), how would that else statement look?

Try:

Code:

awk 'NR==FNR{A[$1]; next} { for(i in A) if($0~i){f=i ".txt"; print>>f; close(f); next }}1'  infile "$file" > file.other

With this one, that I already mentioned had worked. Is there a reason why it would take a pretty large amount of time to complete the script, while creating just one file for one search string is done in 2 seconds.

Is there a more effecient way of writing the code to make it faster? Is there another way to write this code, in order to find errors easier, where I can print "something" when there's an error?

---------- Post updated at 04:57 PM ---------- Previous update was at 01:49 PM ----------

Good Afternoon Scrutinizer,

I'm testing this code and there's something very funny about it. First off, the things that are being searched have a file created IMMEDIATELY, but not yet finished being searched until 30 minutes. To manually grep these strings out of the whole directory, it takes less than 2 seconds.

The results between manual search and script finishing are different. Also, the scripts returns different results when ran once, deleted and then ran again. None of the data changed, only the output files have discrepancies. I'm not sure why this would be...

-David

Hi David, I had not really looked at your problem, just provided a fix for the ambiguous redirection and then suggested some improvements on that ..
OK, I see the awk is supposed to be part of a for loop is that correct? How many files are in that directory typically?

There most likely should be for loop. While I ask these questions and try to come up with solutions for easier processing on these logs, I'm taking small tutorials on the different commands (awk, sed, unique, cut, sort). Obviously these things will give me statistics that would take extremely long to manually perform, considering the amount of data that's present. So, in short, I believe it's a loop I'm looking for. =}

The directory contains roughly 50 files. The amount of files will continue to grow, 1 per day. So the idea is to incorporate, into the script, something that will search the NEW files and append to the current, existing file.

Then after that is smoothly running on a cron job, I will look into making a script that will give me stats on particular fields (I'm assuming, using the awk, cut, unique, and sed commands).

Scrutinizer, feel free to point me in a general direction to get started, instead of spelling everything out. It does seem that the examples you've listed in this thread are at an expert level of scripting. Where many tutorials I've been through seem to make separate lines for each (command?) and have a debug included incase something with the files change or it simply just doesn't work someday.

-David