Shell Programming and Scripting

Hi,

I have a weird requirement. I am having a file with 12fields in it and the end of the line for each record is "\n" (Just \n and no carriage returns) and the field delimiter is "|". Problem is I can have new line characters in any field in the data and these new line characters can even come in the last field (This is the problem as it is very tricky to identify where the field is completing).

Can any one help me how can I delete these new line characters which are present in the data.

Please help me how can I do
1. If I have a decimal field as first field
2. Having string field as first field and this field as well having new line characters in it -- Looks to me it is almost impossible to cleane this file but want a confirmation from the expertise.

Let me know if I am not clear with my question.

Thanks in advance.

In a correct record are there always exactly 11 pipe delimiters.
When there is a broken record, do the components always contain 11 pipe delimiters?

If so, the "awk" programmers should be able to help.

Please post you Operating System and version. There is much variation in awk/nawk/gawk .

Try this:

Code:

sed -i -e 's:\\n::g' filename

This will replace the \n with nothing.

Does that work?

@brianj
I can answer that ... no.

The \n referred to by the O/P is a newline character. Of course the normal line terminator in a unix text file is newline. Therefore newlines inside text file records are a complete no-no.
This situation often arises from bad field validation at the terminal when entering data into a database. When data is extracted from the database any embedded newlines get confused with real line terminators.
The real solution is to build a filter into your data extract program, or to do all the programming with a database programming language not unix Shell tools.

Thnxs for the replies. Yes methyl. For correct record and for the broken records number of delimiters will be always 11. Could u let me know the script.

---------- Post updated at 05:18 PM ---------- Previous update was at 05:14 PM ----------

True the real solution is to get this fixed by the source but unfortunately I can't get this done by source. So no other option except using a shell.

---------- Post updated 02-08-12 at 04:12 AM ---------- Previous update was 02-07-12 at 05:18 PM ----------

Sorry methyl...Didn't look your message fully. OS I am using is
Linux-x86-gcc3p32

Try this:

Code:

awk -F \| '{while(NF<12 && getline p)$0=$0p}1' infile

Thanks Scrutinizer. This handles the new line characters present in all the fields except in last field but my requirement to handle these as well.

My input records are like:

Code:

3025|DUM_
MY_1|Class DUMMY_1
|5000|GBP|3025|2|14|T|1|1|Dummy
Invoice
for dummy1
3026|DUMMY_2|class DUMMY_2|5000|GBP|3026|2|0|T|1|1|N/A

I want the output to be:

Code:

3025|DUM_MY_1|ClassDUMMY_1|5000|GBP|3025|2|14|T|1|1|DummyInvoicefor dummy1
3026|DUMMY_2|class DUMMY_2|5000|GBP|3026|2|0|T|1|1|N/A

with te command given you the output is coming as

Code:

3025|DUM_MY_1|ClassDUMMY_1|5000|GBP|3025|2|14|T|1|1|Dummy
Invoicefor dummy13026|DUMMY_2|class DUMMY_2|5000|GBP|3026|2|0|T|1|1|N/A

some of the data in the last field is going to next record.

Only facility I have here is the first field is always a decimal field.