UNIX for Advanced & Expert Users

Thanks a lot,

Two questions here:

1) Radoulov - Your code is amazing, but what is the difference between the two option which you have given? I was not clear for "If the filenames contain no spaces". But the first option is working for my requirement.

2)bobbygsk - your code is also working for my requirement. But my concern here is the "performance". Since in real environment the datafiles will have "millions" of records speed up our script is very essential.

Hence please suggest me, which code is faster ?

Thanks again......

With Regards / Lokesha

Try the first one with a filename like this:

data file1.txt


I'd try both solutions and see.

Lokesha,

My code looks simple and easy to understand.
I'm in intermediate stage of unix scripting.
I do not know about my script performance.
It is better to go with AWK.

You need to try out the alternatives before picking one that works efficiently specially since you need to process millions of records which would not be an easy feat to accomplish.

Code:

awk '{
   s = sprintf("\"%s\"", $0)
   re = "*TM*"
   a[re] = 0
   while ("cat "s | getline l) {
     if (l == re)
        a[re]++
     if (a[re] == 1 && l != re)
        print l
   }
}' list_file

One more problem,

Radoulov- I'm using the your below code:

nawk 'f[FILENAME]==1&&!/^\*TM\*/;/^\*TM\*/{f[FILENAME]++}' $(<list_file) > complete.txt

Since datafile and list will not be in the same location. Also list file will only have the datafile names to be selected and not the location. We've to declare this explicitly.
ex: FTPIN/ is the location for datafile and FILES/ is the location for list file and FTPOUT/ is the location for complete.txt file.
I've to mention these location in your code as below:

nawk 'f[FILENAME]==1&&!/^\*TM\*/;/^\*TM\*/{f[FILENAME]++}' $(<FILES/list_file) >> FTPOUT/complete.txt

But where to mention the datafile location FTPIN/ to select the datafiles?

Also please correct me if Iam wrong for the above code.

You can use something like this:

Code:

nawk 'f[FILENAME] == 1 && !/^\*TM\*/
/^\*TM\*/ { f[FILENAME]++ }
' $(printf "FTPIN/%s\n" $(<FILES/list_file)) > FTPOUT/complete.txt

P.S. You don't need to append here:

Code:

... >> FTPOUT/complete.txt

This should be sufficient:

Code:

... > FTPOUT/complete.txt

Thanks Radoulov,

Iam using your below script:

nawk 'f[FILENAME] == 1 && !/^\*TM\*/
/^\*TM\*/ { f[FILENAME]++ }
' $(printf "FTPIN/%s\n" $(<FILES/list_file)) > FTPOUT/complete.txt

One more question here. Is it possible to check the "read" permission for the file whose names are present in the "FILES/list_file" ?

This is really challenging for me to check this, as this is mandatory...