Pattern to replace ^M and ^Y in a 4.2 AIX text file | Unix Linux Forums | AIX

  Go Back    


AIX AIX is IBM's industry-leading UNIX operating system that meets the demands of applications that businesses rely upon in today's marketplace.

Pattern to replace ^M and ^Y in a 4.2 AIX text file

AIX


Tags
aix, unix

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 05-16-2009
Browser_ice Browser_ice is offline
Registered User
 
Join Date: Nov 2005
Last Activity: 6 April 2010, 6:04 PM EDT
Location: Canada
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
Question Pattern to replace ^M and ^Y in a 4.2 AIX text file

I have files on my AIX 4.2 client system where I need to do the following replacements below but have no clue how ? They are control characters (linefeed, chariage return, ...).


First, replace "^M^Y^M" with ^char_for_end_of_line
Then replace "^M" with " "
Trim all left spaces

In VI, my files contents look like this :

aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
...

I want it to be:
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
...

Nb of records is unknown.
'zzzzzz' can have any combinations of "(", ")", "'", """, ",", "[", "]", ".", ";" (in other words anything with printable characters)
Sponsored Links
    #2  
Old 05-16-2009
TonyFullerMalv's Avatar
TonyFullerMalv TonyFullerMalv is offline Forum Advisor  
Registered User
 
Join Date: Sep 2008
Last Activity: 1 July 2013, 6:45 PM EDT
Location: Malvern, Worcs. U.K.
Posts: 1,033
Thanks: 0
Thanked 8 Times in 8 Posts
What you could really do with is dos2unix(1) that comes with Solaris and Linux but not with AIX IIRC, so instead you can use sed:

dos2unix:

Code:
sed -i 's/\r//' file

unix2dos:

Code:
sed -i 's/\n/\n\r/' file

from: UNIX BASH scripting: Linux flip command - alternative of dos2unix,unix2dos

or

dos2unix:

Code:
$ sed 's/^M$//'  input.txt > output.txt

unix2dos:

Code:
$ sed 's/$'"/`echo \\\r`/"   input.txt > output.txt

from: Howto: UNIX or Linux convert DOS newlines CR-LF to Unix/Linux format

The dos2unix examples will get rid of the carriage returns for you I will leave a scripting guru to work out the removal of particular unwanted line feeds.
Sponsored Links
    #3  
Old 05-17-2009
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 30 July 2014, 6:40 AM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 4,130
Thanks: 44
Thanked 773 Times in 608 Posts
In your example it looks like you have groups of 3 lines of text followed by 2 lines. You want to combine the three lines of text into a single line and remove the two separating lines completely.

If this is the case:


Code:
sed -n 'N;N;s/[^M^Y]//g;s/\n//gp;N;N

This will first read two additional lines (to the first read line) from the file and combine these into the pattern space. The first replacement then throws out the control characters (^M and ^Y, enter them via <CTRL-V> in vi), the second replacement removes the newline characters combining the lines to one line and prints it. Then two additional lines (the separator lines) are read and discarded, since they are not printed at all, then repeat from start.

I hope this helps.

bakunin
    #4  
Old 05-19-2009
Browser_ice Browser_ice is offline
Registered User
 
Join Date: Nov 2005
Last Activity: 6 April 2010, 6:04 PM EDT
Location: Canada
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by bakunin View Post
In your example it looks like you have groups of 3 lines of text followed by 2 lines. You want to combine the three lines of text into a single line and remove the two separating lines completely.

If this is the case:


Code:
sed -n 'N;N;s/[^M^Y]//g;s/\n//gp;N;N

This will first read two additional lines (to the first read line) from the file and combine these into the pattern space. The first replacement then throws out the control characters (^M and ^Y, enter them via <CTRL-V> in vi), the second replacement removes the newline characters combining the lines to one line and prints it. Then two additional lines (the separator lines) are read and discarded, since they are not printed at all, then repeat from start.

I hope this helps.

bakunin
What if the number of lines of the original file is unknown ?

In my example I gave 3 lines but it can be anything between 1 and 20 lines. The file contains any multi-line amount of records. Each records is totally independent from the previous one. One record could have 2 lines, the next 20, the next 5, ... No regular patterns for the amount of lines. The file contains a list of system generated alarms coming from 20 different servers, numerous amount of workstations, ...

Sorry I forgot to mention it.
Sponsored Links
    #5  
Old 05-21-2009
Browser_ice Browser_ice is offline
Registered User
 
Join Date: Nov 2005
Last Activity: 6 April 2010, 6:04 PM EDT
Location: Canada
Posts: 118
Thanks: 0
Thanked 0 Times in 0 Posts
I tried the combinations below which do not change anything or are not recognized

\n
\^m
\^Y
Ctrl-V + Ctrl-M
Ctrl-V + Ctrl-Y => nothing is typed in the console, I have to do a Ctrl-C to get out
\x0D$
\xC1$
[^M^Y]
[^M]
[^Y]
\c[m => not recognized

sed 's/.$//' does remove the ^M at the end of each line but then it is still a multi-line format. Its like removing the last character of each line but keeping the end-of-line linefeed.

[added comments]
Is there a way to find out in VI what is the ascii value of the character under the cursor ?
It would help me identify the right decimal value to use in a replacement string.

[added comments]
I found out that ^M is actually \015. So I can remove it with tr -d '\015'
But I still haven't found out what ^Y is.

Last edited by Browser_ice; 05-21-2009 at 02:34 PM..
Sponsored Links
    #6  
Old 05-21-2009
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 30 July 2014, 6:40 AM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 4,130
Thanks: 44
Thanked 773 Times in 608 Posts
Quote:
Originally Posted by Browser_ice View Post
What if the number of lines of the original file is unknown ?

In my example I gave 3 lines but it can be anything between 1 and 20 lines.
In this case you will have to have some indication for a "record" being complete. Maybe you will need some record starting criteria too, for which one could match. Provide some data and i will provide some solution.

Quote:
Originally Posted by Browser_ice View Post
I tried the combinations below which do not change anything or are not recognized
This is just a way to enter non-printing (control-) characters into vi: enter input mode, press "CTRL-V", then press CTRL-M (for example for "^M"). You should be still in input mode and see "^M" under the cursor.

Quote:
sed 's/.$//' does remove the ^M at the end of each line but then it is still a multi-line format.
It removes the last character in a line, regardless which character this is - this is the problem. You have to specifically match "^M" (CTRL-M) and throw that out. You can throw out linefeeds by searching for "\n". Try the following with some test file:


Code:
sed 'N;s/\n/@/' /some/file

to see the effect: two lines combined to one and the linefeed is replaced by an at.

[quote]Is there a way to find out in VI what is the ascii value of the character under the cursor ?[/qoute]

No, but you can use "od -ax <file> | more".

I hope this helps.

bakunin
Sponsored Links
    #7  
Old 05-22-2009
ravager's Avatar
ravager ravager is offline
Registered User
 
Join Date: Sep 2005
Last Activity: 20 December 2011, 8:04 AM EST
Posts: 128
Thanks: 0
Thanked 1 Time in 1 Post
You may also use vi

vi file

:.%s/^M//

THIS STRING WILL NOT WORK JUST LIKE THIS AS YOU HAVE TO USE CTRL SET

SO THE COMMAND TO GET THIS SAME STRING IS THIS
:.%s/(ctrl+v)(ctrl+M)//


I hope this helps you
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to replace text in a file with text entered cleanden Shell Programming and Scripting 9 04-11-2009 08:03 PM
Error: Find the pattern in the file and replace shreekrishnagd Shell Programming and Scripting 9 03-04-2009 04:04 AM
serach and replace a specific pattern or value in a xml file kiranreddy1215 Shell Programming and Scripting 1 08-13-2008 12:21 PM
search and replace a specific text in text file? santosham UNIX for Dummies Questions & Answers 4 06-25-2008 05:53 PM
find and replace a pattern in a file krishnamaraju Shell Programming and Scripting 1 08-29-2006 10:02 AM



All times are GMT -4. The time now is 09:08 AM.