Shell Programming and Scripting

Hello Chubler_XL,

I cannot say anything more than Awesome! what do you eat? jaja

It can become a little bit complex extracting 5 more parameters that I've suppressed from the sample input, to have it all completely mapped I can show you if you want to increase the challenge if not is ok, is much much more that I thought to get in help.

Please if you have a little chance, please may you explain the parts of the script that you think must be more difficult to understand for newbies like me ?

For example,

Why you use the input file twice?
How is the logic of the script?
How you separate and store for each block each parameter? and things like that.

I hope you could explain a little bit, your solution is a complete class/treaty of awk for me and for many for sure!

Thanks again for help other and your great contributions for many people.

---------- Post updated at 05:33 PM ---------- Previous update was at 01:32 AM ----------

Hello again Chubler_XL,

In addition to what I've mentioned in my previous post, I've tested in real
files and works fine for files that have up to 1 million of lines (it took 3:58 min to process).

But when I used an input file that have 4,252,871 lines, the processing took 1:31 hours

I'm not sure why the time in processing grew up in such manner.

I don't know if there is something that could be done in order to speed up a little more the
processing of big files like that of 4 million lines.

Below the times for different files:

Code:

With 100,000 lines (3 seconds)
$ . Script.sh 
Wed, Sep 12, 2012  1:13:43 PM
Wed, Sep 12, 2012  1:13:46 PM

With 500,000 lines(52 seconds)
$ . Script.sh
Wed, Sep 12, 2012  1:15:30 PM
Wed, Sep 12, 2012  1:16:22 PM

With 1,000,000 lines(3:58 minutes)
$ . Script.sh
Wed, Sep 12, 2012  1:27:37 PM
Wed, Sep 12, 2012  1:31:35 PM

With 4,252,871 lines(1:31 hours)
$ . Script.sh
Wed, Sep 12, 2012  1:38:32 PM
Wed, Sep 12, 2012  3:10:24 PM

Thanks for your help again.

Regards

OK I'll start with performance, It's probably related GNU array sorting try removing the PROCINFO["sorted_in"] = "@ind_str_asc" line and process a mid-sized file.

If it's a lot quicker we can try turning off array sorting until the processing is done.

OK now for a quick description on what this script is doing.

I'll start with the last line first:

Code:

' FS="\n" RS="" infile infile

As you said the file is processed twice and the reason for this is the heading line. This appears as the first output line and it must know at this point all the heading strings that the document uses. We could read the whole document into memory then output the headings and the data in the END block but this would be a poor solution for big input files.



Pass 1 - Build tags array

To simplify code pass 1 does pretty much the same work as pass 2, most of this is unnecessary and could be skipped by adding FNR!=NR to all the conditions except for the "PERMANENT SUBSCRIBER DATA" (as this is where tags is built).



FS=”\n” RS=””

RS is record separator and FS is field separator. When a blank RS is used a blank line operates as the record separator, so for:

Code:

SUBSCRIBER IDENTITY
MSISDN           IMSI             STATE          AUTHD
99949091700      123450011753067  CONNECTED      AVAILABLE
 
NAM
1

Awk passes 1 record with field 1 as “SUBSCRIBER IDENTITY” and field 2 as “MSISDN IMSI STATE AUTHD”, and field 3 with the data.
Then a second record appears with field 1 as "NAM" and field 2 as "1".

As you can see awk has done a lot of the work for us now as each part of the document comes in as an individual record.

Hello Chubler_XL,

Many thanks for your explanation. I'll study it in depth seeing your code.

Regarding the performance tests removing the line PROCINFO["sorted_in"] = "@ind_str_asc". The time was better with an input file of 1 million lines, 1:20 minutes,
but with an input of 4 millions of lines, the script stucked again. After 30 minutes
any output was shown .

I'm not sure why with a file 4 times bigger (4 million line) than the other (1 million lines), the processing decrease in such manner.

Thanks in advance for your great help.