Issue with Removing Carriage Return (^M) in delimited file - UNIX for Advanced & Expert Users

sirahc · #1 03-19-2008

Hi - I tried to remove ^M in a delimited file using "tr -d "\r" and "sed 's/^M//g'", but it does not work quite well. While the ^M is removed, the format of the record is still cut in half, like
a,b, c
c,d,e

The delimited file is generated using sh script by outputing a SQL query result to tab delimited file. The ^M embedded in the record is captured from user input, I believe.

Any insight is appreciated.

joeyg · #2 03-19-2008

Or at least the first few lines of data?

try something like the following which will look at the first five lines:
(use your file in place of infile)

>head -5 infile | od -An -t oC -w10

Note, in octal
<CR> carriage return = 015
<LF> line feed = 012

sirahc · #3 03-19-2008

Hi joeyg,

Thanks for your reply. Unfortunately, this is sensitive data so I can't send out a dump. I did use the od command to pull the data and checked for \r (od -c). I see that there is carriage return (\r)+ new line control (\n) embedded in the record.

When I used tr -d "\r\n" , however it remove all line breaks with just "\n". Do you know how I can remove only the string "\r\n" without touching those with only "\n"?

Thanks,
sirahc

joeyg · #4 03-20-2008

How about something like:

Code:

infile="original.dat"
outfile="revised.dat"
while read zf
  do
    printf "$zf\n" >>$outfile
done < $infile

kanu_kanu · #5 03-20-2008

how abt dos2ux / dtox command ?

sirahc · #6 03-21-2008

Thanks for all your suggestions.

The printf/dos2unix commands did work to some extend but cause format problem to other columns. I eventually used SQL to remove ^M before outputting it to the file.

ednix · #7 03-24-2008

in vi:

# vi datafile
:s/^V^M//g
:wq
#

the ^V^M sequence means control-V control-M

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Carriage Return at end of file	bd_joy	Shell Programming and Scripting	14	10-20-2006 10:20 AM
Removing Carriage Return and or line feed from a file	tbone231	Shell Programming and Scripting	1	02-18-2005 01:37 PM
sed removing carriage return and newline	mored	Shell Programming and Scripting	2	05-06-2004 09:28 AM
Removing Carriage return to create one record	r1500	Shell Programming and Scripting	3	02-06-2004 12:45 PM
Removing carriage return characters from file	b1saini	UNIX for Dummies Questions & Answers	3	09-10-2003 06:41 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode