UNIX for Dummies Questions & Answers

Thanks,
Tomorrow I will try and let you know if I need more help.

---------- Post updated at 10:47 PM ---------- Previous update was at 10:42 AM ----------

Hi Corona,

I tried your code and it working fine, Thank you so much but I have couple of doubt which I wanted to share with you

Code:

awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print >  PRF "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

After Running the above code I check the data inside the file using more and cat comment. Its coming as per expectation but when I opening these files in notepad in windows system all records are coming in one line
Notepad file

Code:

PP1234512345671234567BABCPP1234512345671234567BABC

Using more command

Code:

$ more Pplus_GIC_*
::::::::::::::
Pplus_GIC_B.txt
::::::::::::::
PP1234512345671234567BABC
PP1234512345671234567BABC
::::::::::::::
Pplus_GIC_C.txt
::::::::::::::
PP1234512345671234567CABC
PP1234512345671234567CABC
::::::::::::::
Pplus_GIC_D.txt
::::::::::::::
PP1234512345671234567DABC
PP1234512345671234567DABC
PP1234512345671234567DABC
::::::::::::::
Pplus_GIC_E.txt
::::::::::::::
PP1234512345671234567EABC
PP1234512345671234567EABC

Do we have to insert newline for each record ?

Also, Can you suggest abount the performance of this command on 2 million records?

Regards
Ibrar Ahmad

*nix and windows are two seriously different systems. One of the differences is the designation of line ends in text files. *nix uses <NL> (= <new line>, 0x0A, \n), windows uses the combination of <CR> (= <carriage return>, 0x0D, \r) and <NL>.
So you should stay on one system. If working on both is unavoidable, you'll need to take extra care to convert the files. You can use conversion tools like unix2dos, iconv, or recode. Or you can print the \r explicitly in the above awk solution.

I don't have a file with 2 million records at hand, so its difficult to predict, esp. on a windows system. Still I guess it should work in a few seconds unless many files are to be opened/closed again and again..

Using:

Code:

awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print >  PRF "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

could have a problem on some versions of awk because the precedence between concatenation of strings and the output redirection operator in print statements is not specified by the standards. Some implementations of awk will treat this code as:

Code:

awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{(print >  PRF) "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

(possibly giving a syntax error, and certainly not producing the output files you want) and others will treat it as:

Code:

awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print > (PRF "_" substr($0, POS, NXT-POS+1) ".txt")}'  $Filename

Since it is working on your system, we can assume that your version of awk is doing the latter.

This code doesn't close and reopen files, so, if it doesn't give an error (too many open files) it should run pretty quickly. It might run slightly faster if you move appending the "_" out of the loop and just do it once instead of two million times:

Code:

awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="${TargetfilePrefix}_" '{print >  (PRF substr($0, POS, NXT-POS+1) ".txt")}'  $Filename

As was noted before, if this fails on your two million record file with a too many open files error, you'll have to build in code to close and open files. If you open and close files for each output line, that will run considerably slower. But, doing anything smarter than that would require you to evaluate the input file to see if lines directed to the same output file are closely grouped in the input, if there are common occurrences of adjacent lines that will be directed to the same output file, etc. that can be used to make smarter decisions about when to close a file and when (if ever) a file needs to be reopened.

Since you couldn't even guess at how many output files would be produced from your input file, I assume you have not tried to evaluate any of the above questions that might help produce more efficient code if you do have to close and reopen files.

Hi.

In general for collecting, gating lines that are related, one could do an initial stable sort on the field of interest, then, while reading with a subsequent (say awk) code, close previous file and open new file when the field content changes.

Obviously this touches the data an extra time (at least), but is conceptually simple and would never run into the limit for open files. Assuming a pipe connection, then no extra files (other than from the sort) are created.

Best wishes ... cheers, drl