Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Trying to remove '^M' characters from a file.

Shell Programming and Scripting


Tags
shell script, shell scripting, unix scripting, unix scripting basics

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 06-30-2006
Krispy Krispy is offline
Registered User
 
Join Date: Dec 2005
Last Activity: 10 November 2006, 12:18 PM EST
Posts: 9
Thanks: 0
Thanked 1 Time in 1 Post
Trying to remove '^M' characters from a file.

Hi guys,

Hope you are all well.

This is a line of data from a csv file. I have used vi and set the 'set list' option to display the trailing $ character.

"01","Grocery","01006","eat Fish & Spreads"$

I have tried the following commands, but neither of them appear to be working?

1) tr -d "^M" < originalfile.csv > newfile.csv
2) tr -d "\015" < originalfile.csv > newfile.csv

....but when I vi & set list the new file, I am still seeing the trailing $ character in the file.

Can anyone please tell me where I am going wrong?

Thanks in advance,
Kev
Sponsored Links
    #2  
Old Unix and Linux 06-30-2006
ShawnMilo ShawnMilo is offline
Registered User
 
Join Date: Jun 2006
Last Activity: 10 November 2009, 8:27 AM EST
Posts: 252
Thanks: 0
Thanked 1 Time in 1 Post
Hope this helps.

I had a big issue with this while trying to get a flat file into Oracle. I tried using vi and sed, and ultimately came upon a solution using strings and tr.

Here is my shell script:

#make a backup
cp $1 $1.orig

#Strip out nulls.
tr -d ‘\0′ < $1.orig > $1.temp

#Strip out other unprintable characters
strings $1.temp > $1

#Remove temp file.
rm $1.temp

#Fix permissions. (You may not need this for your application)
chmod 744 $1
Sponsored Links
    #3  
Old Unix and Linux 06-30-2006
thestevew thestevew is offline
Registered User
 
Join Date: Mar 2006
Last Activity: 11 July 2008, 4:53 AM EDT
Location: South Yorkshire, UK
Posts: 114
Thanks: 0
Thanked 1 Time in 1 Post
Have you misunderstood vi's "set list". What it does is print a "$" sign at the end of each line (and show tabs as ^I). The $ sign isn't in the file itself it is just displayed to make it easier for you to spot things like trailing spaces at the end of lines.

cheers
    #4  
Old Unix and Linux 06-30-2006
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 27 May 2017, 10:47 PM EDT
Location: NM
Posts: 11,069
Thanks: 526
Thanked 1,057 Times in 980 Posts
The "^M" things are from dos/windows files. dos carriage returns are "\n\r", unix "\n"

The extra character shows up as "^M".

Use dos2ux (or dos2unix depending on your flavor of unix) to convert windows text files.


Code:
dos2ux oldfile > newfile

Sponsored Links
    #5  
Old Unix and Linux 06-30-2006
Hitori's Unix or Linux Image
Hitori Hitori is offline Forum Advisor  
kharkovpromenade.com.ua
 
Join Date: Jun 2006
Last Activity: 10 December 2011, 9:33 AM EST
Posts: 360
Thanks: 0
Thanked 9 Times in 8 Posts
in vi you can replace all ^M characters with command
:%s/^M//g

(Note that to type ^M in vi type Ctrl-V Enter)

I realise that this note may be offensive for vi users Linux
The Following 2 Users Say Thank You to Hitori For This Useful Post:
alex_5161 (05-18-2010), valuequest (02-21-2011)
Sponsored Links
    #6  
Old Unix and Linux 07-03-2006
Krispy Krispy is offline
Registered User
 
Join Date: Dec 2005
Last Activity: 10 November 2006, 12:18 PM EST
Posts: 9
Thanks: 0
Thanked 1 Time in 1 Post
Thanks for the swift response guys - much appreciated.

Thestevew was right - I was misunderstanding the vi 'set list' option, so thanks for putting me straight.

I eventually came up with a couple of possibilites - I could use the unix2dos command within a batch file to properly format my csv file.

I also found another thread on this forum with the following perl command...


Code:
perl -i -pe 's/$/\r/' myfile.csv

I'll probably use the perl command to avoid implementing a batch file. Plus, I don't have to output the results to a new file.

Out of interest, does anyone know of a sed command that would do the same as the above perl command?

Thanks again,
Kev
The Following User Says Thank You to Krispy For This Useful Post:
alex_5161 (05-18-2010)
Sponsored Links
    #7  
Old Unix and Linux 07-03-2006
Hitori's Unix or Linux Image
Hitori Hitori is offline Forum Advisor  
kharkovpromenade.com.ua
 
Join Date: Jun 2006
Last Activity: 10 December 2011, 9:33 AM EST
Posts: 360
Thanks: 0
Thanked 9 Times in 8 Posts

Code:
sed -i 's/$/\r/' myfile.csv

I don't think that all implementations of sed has this feature. I've used GNU sed 4.1.4
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
remove last characters after %EOF (pdf binary file) diegugawa Shell Programming and Scripting 1 06-26-2010 09:49 AM
Remove the characters from the file Kattoor Shell Programming and Scripting 6 02-16-2010 12:48 AM
How to remove ^M characters from a zip file? bhaskar_m Shell Programming and Scripting 2 06-25-2009 07:01 AM
Remove characters from file name mrlayance Shell Programming and Scripting 5 04-28-2009 08:04 PM
sed to remove last 2 characters of txt file ajp7701 Shell Programming and Scripting 1 10-29-2008 07:23 PM



All times are GMT -4. The time now is 04:43 AM.