Shell Programming and Scripting

Hi Gurus,

Apologies as I feel like this must be answered already on here somewhere but I just can't find it. I find many people looking to remove all \n and \r (CR and LF) or one or the other but the only times I've found someone trying to remove them only when both are together they've found workarounds instead eg: https://www.unix.com/unix-advanced-ex...ited-file.html

So my issue is: I have data like

Code:

a,b,c,d,e
a,b,c,d,e
a,b,c,^M
d,e
a,b,c,d,e

Removing the ^M alone I can do with

Code:

tr -d '\r'

But this still leaves the broken line

I tried

Code:

tr -d '\r\n'

but of course that moves ALL linefeeds, not just the ones after a ^M

I can find the lines with ease using grep and <ctrlv><ctrlm> and fix them manually but I need this in an automated script, as simple to understand as possible please as I'm obviously no unix wiz.

Your help is greatly appreciated, as always!

Lee

Hi, Lee:

Perhaps the following will do the trick for you:

Code:

sed '/^M$/{N; s/.\n//;}'

That solution will discard a final line ending with \r\n since there's nothing to merge it with. If that's undesirable, see the next offering.

The following, when it encounters a final line which ends in \r\n, will strip the \r but leave the \n. Nothing follows the line so it cannot be merged with another, but the final result is a text file with unix line endings as per your example:

Code:

sed '/^M$/{s///; $p; N; s/\n//;}'

Note that everything said above only applies to a posix-compliant sed implementation. GNU sed by default chooses to ignore the standard and writes the pattern space to stdout when N executes and there's no further text. The following (I think .. untested) are the equivalents for the GNU sed commonly found on linux systems:

Code:

sed '/^M$/{$d; N; s/.\n//;}'

Code:

sed '/^M$/{s///; N; s/\n//;}'

Regards,
Alister

P.S. In case the rationale for GNU sed's N behavior is of interest to anyone, it's discussed @ http://www.gnu.org/software/sed/manu...ting-Bugs.html (I found that myself when looking to report the "bug" )

I had this issue with files that were ftp-ed from windows to unix .. using the dos2unix command cleans it up pretty well. Try it. I used it in a script to clean up data files before loading into the database and it worked too.

Code:

dos2unix <inputfile> <outputfile>

mvijayv:

I think you misunderstood the problem. dos2unix, to my knowledge, only converts the line ending; it does not merge lines by deleting one type of line ending while leaving others intact.

Regards,
Alister

Code:

echo "a,b,c,d,e
a,b,c,d,e
a,b,c,^M
d,e
a,b,c,d,e" |sed 'N;s/^M\n//'
a,b,c,d,e
a,b,c,d,e
a,b,c,d,e
a,b,c,d,e

Thanks Alister. Missed that piece .. Yup dos2unix doesn't merge the lines.

That solution will not work for even numbered lines ending with \r\n. In those cases, there will be no \r\n in the pattern space. The pattern space will end with \r alone.

Regards,
Alister