removing spurious charactors from file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting removing spurious charactors from file
# 1  
Old 07-08-2004
Question removing spurious charactors from file

Morning,

Hopefully someone can help an newbie!

I have been getting a number of file corruptions with
Rogue "Control M" charactors [ carriage return ].

The effect is:

Should be

aaaa
bbbb
cccc
dddd

(4 lines)

But getting

aaaa
bb<Ctl M>
bb
ccccc
dddd

(5 lines)

I am having to correct this manually using vi...


Anyone know a way to do this in a script?

Effectively I want to delete any occurrances of the string
"^M\n"

I can't use tr as

tr -d "^M" leaves two lines.
tr -d "^M\n" also deletes the genuine "\n" at end of line.


Any suggestions? sed?

The problem with manuals is that you need to know what to look up!
# 2  
Old 07-08-2004
in vi:

esc. to cmd mode, then enter

Code:
:%s/CTRL+V CTRL+M//g

Note I left a space betw. the 2 CTRL sequences for readability, you shouldn't have any spaces.
# 3  
Old 07-08-2004
Thanks.

However, I am able to do this using vi.

I am asking how to do it in a script,
so it can be automated.
# 4  
Old 07-08-2004
Sorry I relied too much on your title and assumed it was a single file.

Try playing with this:
Code:
for fl in *.php; do
mv $fl $fl.old
sed 's/FINDSTRING/REPLACESTRING/g' $fl.old > $fl
#rm -f $fl.old
done

You could also do it something like:
Code:
for i in `find ./ -type f -exec grep -l "yourstringhere" {} \;`
        do 
        # use our sed rules to make changes. put them in a temp file
        sed -f ./sedlist $i > $i.tmp
        # this is optional. make a back up to be safe.
        #cp $i $i.sedbak
        # move the new temp file over the original
        mv $i.tmp $i
        done

without the "sedlist" file

Of course, change the values to suit your needs.
# 5  
Old 07-08-2004
Try...

awk '/\r/{sub("\r","");printf $0;next};1' file1 > file2
# 6  
Old 07-08-2004
the problem is that the
corruption inserts a <Ctl M> charactor and throws a line.

Hence

instead of bbbb

we get :

bb<CtlM>
bb

By simply substituting <Ctl M> with ''
I get

bb
bb

when I want

bbbb


In vi I have to use "J" to concatinate the two lines after subsituting <Ctl M> for ''

Hence I think I need to substitute the string
^M\n with ''

but I can't get this to work
# 7  
Old 07-08-2004
Eureka!


Thanks Ygor.


That's done the trick!

(albeit with nawk rather than awk!)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

File Management: Removing of files from Server2 IF the same file is removed from Server1.

Hi Folks, I have a requirement of file management on different servers. Source Server is SERVER-A. Two servers will fetch files from SERVER-A: SERVER1 and SERVER2. 4th SERVER is SERVER-B, It will fetch files from SERVER1. If SERVER1 goes DOWN, SERVER-B will fetch pending files from... (2 Replies)
Discussion started by: Raza Ali
2 Replies

2. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

3. Linux

File conversion and removing special characters from a file in Linux

I have a .CSV file when I check for the special characters in the file using the command cat -vet filename.csv, i get very lengthy lines with "^@", "^I^@" and "^@^M" characters in between each alphabet in all of the records. Using the code below file filename.csv I get the output as I have a... (2 Replies)
Discussion started by: dhruuv369
2 Replies

4. Programming

[Solved] Removing duplicates from the file and saving as new file

Dear All I have 200 data files and each files has many duplicates. I am looking for the automated awk script such that it checks and removes the duplicates from the each file and saving them as new files for all 200 files in the respective folder. For example my data looks like this.. ... (12 Replies)
Discussion started by: bala06
12 Replies

5. Shell Programming and Scripting

Removing part of a file name and appending into a single file

I have two files like ABC_DEF_yyyyymmdd_hhmiss_XXX.txt and ABC_DEF_yyyyymmdd_hhmiss_YYY.txt. The date part is going to be changing everytime. How do i remove this date part of the file and create a single file like ABC_DEF_XXX.txt. (8 Replies)
Discussion started by: varlax
8 Replies

6. UNIX for Dummies Questions & Answers

Help removing multiple lines from one file with a list from a second file!

I am trying to remove the lines listed in example File A from File B to achieve File C. Both files are much larger than the examples below. (File B has up to 6,000 lines). I have searched the forums and I have not been able to find an answer to this particular question. I tried grep -v -f... (2 Replies)
Discussion started by: pezziza
2 Replies

7. Shell Programming and Scripting

rename file by removing some part of the file name

I am special requirements to rename file. I have files with names like below: 1_firstname1_lastname1.html 2_firstname2_lastname2.html 3_fistname3_lastname2.html I would like these file to be renamed as below firstname1_lastname1.html firstname2_lastname2.html... (5 Replies)
Discussion started by: McLan
5 Replies

8. UNIX for Dummies Questions & Answers

Help removing strings from one file that match any of the values in a second file.

Hello, I have a file that lists a few hundred values. Example: abca abcb abcc abcd I have a 2nd file with a few thousand lines. I need to remove every line from the 2nd file that contains any of the values listed in first file. Example of strings to delete: line1 *abca* end of... (1 Reply)
Discussion started by: upstate_boy
1 Replies

9. Shell Programming and Scripting

Spurious line feeds

Hi all, I know this is **awfully** general but..... I have a script which does, basically... for file in `find command`; do some stuff more stuff echo '.\c' done I want to output the '.' char just to give an idea of progress. However, it works fine for a while and then I... (2 Replies)
Discussion started by: ajcannon
2 Replies

10. UNIX for Dummies Questions & Answers

removing a line from a file and then placing into another file

grep `whoami` $1 >> file this lets me take out the username from a file and then i move it to a file but i need it to do one step at a time because i want the occurences to be numbered like 1)HOME=/home/joe.bloggs 2)LOGNAME=joe.bloggs instead of just HOME=/home/joe.bloggs... (1 Reply)
Discussion started by: iago
1 Replies
Login or Register to Ask a Question