Pattern to replace ^M and ^Y in a 4.2 AIX text file


 
Thread Tools Search this Thread
Operating Systems AIX Pattern to replace ^M and ^Y in a 4.2 AIX text file
# 1  
Old 05-16-2009
Question Pattern to replace ^M and ^Y in a 4.2 AIX text file

I have files on my AIX 4.2 client system where I need to do the following replacements below but have no clue how ? They are control characters (linefeed, chariage return, ...).


First, replace "^M^Y^M" with ^char_for_end_of_line
Then replace "^M" with " "
Trim all left spaces

In VI, my files contents look like this :

aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
aaaa zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzzzz^M
zzzzzzzzzzzzzzzzzzzz^M
^M
^Y^M
...

I want it to be:
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
aaaa zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz zzzzzzzzzzzzzzzzzzzzzz
...

Nb of records is unknown.
'zzzzzz' can have any combinations of "(", ")", "'", """, ",", "[", "]", ".", ";" (in other words anything with printable characters)
# 2  
Old 05-16-2009
What you could really do with is dos2unix(1) that comes with Solaris and Linux but not with AIX IIRC, so instead you can use sed:

dos2unix:
Code:
sed -i 's/\r//' file

unix2dos:
Code:
sed -i 's/\n/\n\r/' file

from: UNIX BASH scripting: Linux flip command - alternative of dos2unix,unix2dos

or

dos2unix:
Code:
$ sed 's/^M$//'  input.txt > output.txt

unix2dos:
Code:
$ sed 's/$'"/`echo \\\r`/"   input.txt > output.txt

from: Howto: UNIX or Linux convert DOS newlines CR-LF to Unix/Linux format

The dos2unix examples will get rid of the carriage returns for you I will leave a scripting guru to work out the removal of particular unwanted line feeds.
# 3  
Old 05-17-2009
In your example it looks like you have groups of 3 lines of text followed by 2 lines. You want to combine the three lines of text into a single line and remove the two separating lines completely.

If this is the case:

Code:
sed -n 'N;N;s/[^M^Y]//g;s/\n//gp;N;N

This will first read two additional lines (to the first read line) from the file and combine these into the pattern space. The first replacement then throws out the control characters (^M and ^Y, enter them via <CTRL-V> in vi), the second replacement removes the newline characters combining the lines to one line and prints it. Then two additional lines (the separator lines) are read and discarded, since they are not printed at all, then repeat from start.

I hope this helps.

bakunin
# 4  
Old 05-19-2009
Quote:
Originally Posted by bakunin
In your example it looks like you have groups of 3 lines of text followed by 2 lines. You want to combine the three lines of text into a single line and remove the two separating lines completely.

If this is the case:

Code:
sed -n 'N;N;s/[^M^Y]//g;s/\n//gp;N;N

This will first read two additional lines (to the first read line) from the file and combine these into the pattern space. The first replacement then throws out the control characters (^M and ^Y, enter them via <CTRL-V> in vi), the second replacement removes the newline characters combining the lines to one line and prints it. Then two additional lines (the separator lines) are read and discarded, since they are not printed at all, then repeat from start.

I hope this helps.

bakunin
What if the number of lines of the original file is unknown ?

In my example I gave 3 lines but it can be anything between 1 and 20 lines. The file contains any multi-line amount of records. Each records is totally independent from the previous one. One record could have 2 lines, the next 20, the next 5, ... No regular patterns for the amount of lines. The file contains a list of system generated alarms coming from 20 different servers, numerous amount of workstations, ...

Sorry I forgot to mention it.
# 5  
Old 05-21-2009
I tried the combinations below which do not change anything or are not recognized

\n
\^m
\^Y
Ctrl-V + Ctrl-M
Ctrl-V + Ctrl-Y => nothing is typed in the console, I have to do a Ctrl-C to get out
\x0D$
\xC1$
[^M^Y]
[^M]
[^Y]
\c[m => not recognized

sed 's/.$//' does remove the ^M at the end of each line but then it is still a multi-line format. Its like removing the last character of each line but keeping the end-of-line linefeed.

[added comments]
Is there a way to find out in VI what is the ascii value of the character under the cursor ?
It would help me identify the right decimal value to use in a replacement string.

[added comments]
I found out that ^M is actually \015. So I can remove it with tr -d '\015'
But I still haven't found out what ^Y is.

Last edited by Browser_ice; 05-21-2009 at 03:34 PM..
# 6  
Old 05-21-2009
Quote:
Originally Posted by Browser_ice
What if the number of lines of the original file is unknown ?

In my example I gave 3 lines but it can be anything between 1 and 20 lines.
In this case you will have to have some indication for a "record" being complete. Maybe you will need some record starting criteria too, for which one could match. Provide some data and i will provide some solution.

Quote:
Originally Posted by Browser_ice
I tried the combinations below which do not change anything or are not recognized
This is just a way to enter non-printing (control-) characters into vi: enter input mode, press "CTRL-V", then press CTRL-M (for example for "^M"). You should be still in input mode and see "^M" under the cursor.

Quote:
sed 's/.$//' does remove the ^M at the end of each line but then it is still a multi-line format.
It removes the last character in a line, regardless which character this is - this is the problem. You have to specifically match "^M" (CTRL-M) and throw that out. You can throw out linefeeds by searching for "\n". Try the following with some test file:

Code:
sed 'N;s/\n/@/' /some/file

to see the effect: two lines combined to one and the linefeed is replaced by an at.

[quote]Is there a way to find out in VI what is the ascii value of the character under the cursor ?[/qoute]

No, but you can use "od -ax <file> | more".

I hope this helps.

bakunin
# 7  
Old 05-22-2009
You may also use vi

vi file

:.%s/^M//

THIS STRING WILL NOT WORK JUST LIKE THIS AS YOU HAVE TO USE CTRL SET

SO THE COMMAND TO GET THIS SAME STRING IS THIS
:.%s/(ctrl+v)(ctrl+M)//


I hope this helps you
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Replace pattern in text

hi unix expert is there any command in linux to repace a pattern in the text to another pattern? many thanks samad (2 Replies)
Discussion started by: abdossamad2003
2 Replies

2. Shell Programming and Scripting

Grab text after pattern and replace

i have a file which contains data seperated by comma. i want to replace text after 3rd occurrence of a comma. the input file looks like this abcdef,11/02/2015 11:55:47,1001,1234567812345678,12364,,abc abcdefg,11/02/2015 11:55:47,01,1234567812345678,123,,abc abcdefhih,11/02/2015... (4 Replies)
Discussion started by: gpk_newbie
4 Replies

3. Shell Programming and Scripting

Pattern replace from a text file using sed

I have a sample text format as given below <Text Text_ID="10155645315851111_10155645333076543" From="460350337461111" Created="2011-03-16T17:05:37+0000" use_count="123">This is the first text</Text> <Text Text_ID="10155645315851111_10155645317023456" From="1626711840902323"... (3 Replies)
Discussion started by: my_Perl
3 Replies

4. Shell Programming and Scripting

Search and Replace a text if the line contains a pattern

My text file looks like below . . . abcdefghi jklmnop $Bad_ptrq_GTS=rcrd_ip.txt $Bad_abcd_REJ=rcrd_op.txt ghijklm $Bad_abcd_TYHS=rcrd_op.txt abcgd abcdefghi jklmnop $Bad_ptrq_GTS=rcrd_ip.txt (2 Replies)
Discussion started by: machomaddy
2 Replies

5. Shell Programming and Scripting

sed help, Find a pattern, replace it with same text minus leading 0

HI Folks, I'm looking for a solution for this issue. I want to find the Pattern 0/ and replace it with /. I'm just removing the leading zero. I can find the Pattern but it always puts literal value as a replacement. What am I missing?? sed -e s/0\//\//g File1 > File2 edit by... (3 Replies)
Discussion started by: SirHenry1
3 Replies

6. Shell Programming and Scripting

Sed command to replace with pattern except for text and closing parentheses

Can someone help me with a sed command: There will be multiple occurences in a file that look like this: MyFunction(12c34r5) and I need to replace that with just the 12c34r5 for every occurrence. The text between the parentheses will be different on each occurrence, so I can't search for that.... (4 Replies)
Discussion started by: missb
4 Replies

7. Shell Programming and Scripting

find pattern and replace the text before it

i am editing a big log file with the following pattern: Date: xxxx Updated: name Some log file text here Date: eee Updated: ny Some log file text here Basically i want to remove all the text in a line before the "Updated" pattern. I sill want to print the other... (4 Replies)
Discussion started by: balan1983a
4 Replies

8. Shell Programming and Scripting

Create multiple text file from a single text file on AIX

Hi I need to create multiple text files from onc text file on AIX. The data of text files is as below: ********************************************** ********************************************** DBVERIFY: Release 10.2.0.4.0 - Production on Tue Nov 10 13:45:42 2009 Copyright (c) 1982,... (11 Replies)
Discussion started by: lodhi1978
11 Replies

9. Shell Programming and Scripting

Replace Text Based On Pattern

Hi I'm trying to replace text in a file based upon a pattern. The pattern I'm looking for is: <styleURL>#style0002</styleURL> <name>#######6105#######</name>The # are seven alphanumeric characters before and after 6105. I need it to replace that with this recursively: ... (4 Replies)
Discussion started by: Grizzly
4 Replies

10. Shell Programming and Scripting

pattern replace inside text file using sed

Hi, I have a situation where I want to replace some occurrences of ".jsp" into ".html" inside a text file. For Example: If a pattern found like <a href="http://www.mysite.com/mypage.jsp"> it should be retained. But if a pattern found like <a href="../mypage.jsp"> it should be changed to... (4 Replies)
Discussion started by: meharo
4 Replies
Login or Register to Ask a Question