Shell Programming and Scripting

Hi guys,

Wish you all a very Happy New Year!!!.

Thanks in advance.

I want to read a file and merge the rows which have '\n' in it.
The rows could be > 50,000 bytes. The script should merge all the rows till the next row starts with word 'Type|'.
ex.
Type|Ticket|TradeID|42224A|SystemID|DDDD61266|SourceSystem|RMS|Version|5|LatestVersionYN|Y|Counterpa rty|OB|DBLegalEntity|London|Notes|cal events added|Salesperson|DHX|LastUpdated|DEC 2008 18:31:40|Action|Exp|DealCaptureSystem|Sales|CustomerSourceSystem|RMS|Customer|AUGS|ActionEffectiveDa te|DEC 2008 00:00:00|BookingLocation|London|EnteredBy|GONU|MarkupAmt|100|MarkupCcy|KEX|PremiumCurrency|EUR|Premi umAmount|000|PremiumDate|DEC 2008|PremiumHasPaidYN|N|BookingDetails|9000/mihl
Client sells KEZUSD
st ref 1.3724
Leveraged Sele is at ST
52 sele, starting

Type|FxDiscreteDNT|TradeID|SSA32553H|SystemID|GGT04481|SourceSystem|RMS|SourceSystemBook|OT|Book|OT| BookBranchParent|Go|Version|8|LatestVersionYN|Y|Counterparty|DOB|Status|KnockedOut|IsLiveYN|N|DBLega lEntity|London|LastUpdated|JAN 2009 00:26:35|Action|KnockedOut|CounterpartySourcesystem|RMS|TradeDate|JUL 2006|ProductID|KKI99106|StructureID|DDS32553|DealCaptureSystem|RMS|CustomerSourceSystem|RMS|Customer |ESL|PricingSite|Option|UnderCurrency1|JPY|AccCurrency1|US|UpBarrier1|1.5|LowBarrier1|1|UnderCurrenc y2|JPY|AccCurrency2|CH|UpBarrier2|1.64|LowBarrier2|1.49|UnderCurrency3|EUR|AccCurrency3|JPY|UpBarrie r3|175|LowBarrier3|105|LastFixed

The outout should be
The below record should be in one line.

Type|Ticket|TradeID|42224A|SystemID|DDDD61266|SourceSystem|RMS|Version|5|LatestVersionYN|Y|Counterpa rty|OB|DBLegalEntity|London|Notes|cal events added|Salesperson|DHX|LastUpdated|DEC 2008 18:31:40|Action|Exp|DealCaptureSystem|Sales|CustomerSourceSystem|RMS|Customer|AUGS|ActionEffectiveDa te|DEC 2008 00:00:00|BookingLocation|London|EnteredBy|GONU|MarkupAmt|100|MarkupCcy|KEX|PremiumCurrency|EUR|Premi umAmount|000|PremiumDate|DEC 2008|PremiumHasPaidYN|N|BookingDetails|9000/mihl Client sells KEZUSD st ref 1.3724 Leveraged Sele is at ST 52 sele, starting

Type|FxDiscreteDNT|TradeID|SSA32553H|SystemID|GGT04481|SourceSystem|RMS|Version|8|LatestVersionYN|Y| Counterparty|DOB|DBLegalEntity|London|LastUpdated|JAN 2009 00:26:35|Action|KnockedOut|DealCaptureSystem|RMS|CustomerSourceSystem|RMS|Customer|ESL|PricingSite|O ption|UnderCurrency1|JPY|AccCurrency1|US|UnderCurrency3|EUR|AccCurrency3|JPY|UpBarrier3|175|LowBarri er3|105|LastFixed

I tried using below mentioned script but AWK can't handle > 20,000 bytes. Also, using String as 'LINE_BREAK' for identification rows got merged.

FileName=LINE_TEST.TXT
FileName1=O_LINE_TEST.TXT
OUTPUT_FILE=OUTPUT_WITHOUT_LINE_BREAK.tmp
if [[ -e $FileName ]]; then
sed "s/%/percentage/g" $FileName > $FileName1
## To remove '%' as 'awk' can not handle it.

/usr/xpg4/bin/awk 'NR==1{printf $0;next}
!/^TradeType/ {printf "LINE_BREAK" $0;next}
{printf "\n" $0}
END{print ""}
' $FileName1 > $OUTPUT_FILE
fi

Try with nawk:

Code:

nawk '$1=$1' FS='\n' RS= ORS='\n\n' infile

I am getting the below mentioned error :

13460 Segmentation Fault(coredump). Is it related Memory ?

The Actual File size is : 95532469 bytes.

Try with Perl:

Code:

perl -00ple'tr.\n. .' infile

How to identfy which lines were merged while processing the file ?

Just append some string to identify them:

Code:

perl -00ple'tr.\n. . and $_.=" __FIXED__"' infile

Hi Radoulov,

It's working !!!

Thank you very much for quick reply. Appriciated !!!

One last request

Could you please confirm how this script works ?