Shell Programming and Scripting

Hi

I am using delimited sequence file. Delimter we are using is pipe .But for some of the records for one of the column the values are getting split into different lines as shown below

Code:

"113"|"0155"|"2016-04-27 07:59:04"|"1930"|"TEST@TEST"|"2016-04-27 11:04:04.357000000"|"BO"|"Hard BO"|"10"|"5.1.0 e This is a permanent error. Please verify the address(es) and try again.
<TEST>:
123.123.12 does not like recipient.
Remote host said: 123 Address rejected test@test
Giving up on 1200.
--- "|"I"|"191"|"212"|"DAM"|"PIl"

But my expectation is it should come in sing line before processing the file

Code:

"113"|"0155"|"2016-04-27 07:59:04"|"1930"|"TEST@TEST"|"2016-04-27 11:04:04.357000000"|"BO"|"Hard BO"|"10"|"5.1.0 e This is a permanent error. Please verify the address(es) and try again.<TEST>:123.123.12 does not like recipient.Remote host said: 123 Address rejected test@testGiving up on 1200.--- "|"I"|"191"|"212"|"DAM"|"PIl"

Please help me on this

How are those files produced?

Its a file which we are getting from different source.Not very sure how they are getting produced.Usually the error comes for the 10h column data

could some one please help on this

Not much to go on. You have spurious line feeds in the middle of the line. I am guessing.
Assumption: if the FS = "|" and the first column is a quoted number, this is the correct start of the the line. Anything else is bad.

Is this correct? And is the file from a Windows application or changed by anything like windows FTP? Why I am asking -> because the carriage control may be messed up as well.

If you provide a clear description of your problem (instead of just one sample of a problem with no description), you will be more likely to get a response.

What are the line delimiters in your input file? UNIX single newline character delimiters or DOS carriage-return/newline character pair delimiters? Are the delimiters on split lines the same as in a correctly formatted line? Or, is the delimiter on split lines different from the delimiter on correctly formatted lines? What delimiters do you need in your output file?

How are we supposed to know when a line is complete?

Is your verification program supposed to know that there should be a specific number of fields in each input line? Is that number of fields the same for every file your verification program will process?

Is an invalid input line ALWAYS split between a pair of double quotes? Can we assume that a line needs to be combined with the next line from the input file if and only if an input line does not end with a double quote character?

Are invalid records always split by converting a space to a newline? How do we know whether or not a character needs to be added when lines are joined? Is a space character ALWAYS supposed to be added when lines are joined?

What operating system and shell are you using?

What have you tried to solve this problem on your own?

The standard way of tackling this would be something like this:

Code:

awk -F\| 'NF<15{while(getline s>0) $0=$0 s}1'  file

Which should work with your sample..

However as others have pointed out, unless you provide more information it will be difficult to tell if this would be a solution to your problem..


---
Note that this will not work if the newlines appear in the very last field.

Also note that the csv format allows for newlines within quoted fields, so the sample you posted seems to within specification, so by joining the lines you are effectively changing the content by removing the newlines..

If you want to process the file you do not need remove the newlines in order to process the file. For example to print field 10 (without the enclosing double quotes), you could do something like this:

Code:

$> awk -F\| 'NF<15{while(getline s>0) $0=$0 RS s}{gsub(/^"|"$/,x,$10); print $10}' file
5.1.0 e This is a permanent error. Please verify the address(es) and try again.
<TEST>:
123.123.12 does not like recipient.
Remote host said: 123 Address rejected test@test
Giving up on 1200.
--- 
$>