Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Awk to remove carriage return from 65th field

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk to remove carriage return from 65th field
# 1  
Old 03-21-2012
Awk to remove carriage return from 65th field
Hi,

I have a pipe delimited file. There are around 700 columns in the file.

The 65th column has carriage return which is causing read issue with our ETL process. I would like to replace the new line characters in 65th field with "nothing"

i have return the following code and need help to fix it.

Code:
 
nawk -F"|" 'NF=65 && NF like \n {gsub(NF,\n,"") } file

Please help me fix this.
# 2  
Old 03-21-2012
\n is a newline, not a carriage return. Which do you mean?

If you mean an extra newline is breaking the line early, awk doesn't know the difference between the 'right' newlines and the 'wrong' ones, they're all just bytes; and it won't be able to delete a newline it hasn't read yet.

But you can count the number of fields to see if it broke early and save the partial record for next time.

Code:
awk -F"|" 'NF==65 { T=$0; next}; T { print T $0; T=""; next} 1

# 3  
Old 03-21-2012
When i do octal dump on the line of concern, it looks like the following. It has \r and \n. This is happening at field 65. The only thing i like to do is if the new line character is found in 65th field i want to join this line with the next line.

Help is really appreciated.


Quote:
0001160 060 040 174 060 061 061 065 065 062 174 120 104 120 124 130 015
0 | 0 1 1 5 5 2 | P D P T X \r
0001200 042 012
" \n
---------- Post updated at 03:23 PM ---------- Previous update was at 03:21 PM ----------

In the 65th field we are expecting data like

"ABC DEF"

but what we are receiving is

"ABC
DEF"

we like to convert to "ABC DEF" using awk

Appreciate help.
# 4  
Old 03-21-2012
Did my code do anything?

I'd make a slight change knowing there's actual CR's in there:

Whenever it sees a line ending in \r, it'll change \r to space, save that line, fetch the next, and print both together. Otherwise it will print lines unmodified.
Code:
awk -F"|" '/\r/ { sub(/\r/, " "); T=$0; getline; print T $0; next } 1'

# 5  
Old 03-21-2012
Quote:
Originally Posted by Corona688
Did my code do anything?

I'd make a slight change knowing there's actual CR's in there:

Whenever it sees a line ending in \r, it'll change \r to space, save that line, fetch the next, and print both together. Otherwise it will print lines unmodified.
Code:
awk -F"|" '/\r/ { sub(/\r/, " "); T=$0; getline; print T $0; next } 1'


Thanks Corona this works but needs few changes.

These works but need 3 small changes.
Quote:
nawk -F"|" '/\r/ { sub(/\r/, " "); T=$0; getline; print T $0; next } 1'
  • The replacement should not happen for all records, should happen only when the carriage return is present in the 65th field.
  • Carriage return needs to be replaced with "" (empty) instead of " " space

  • And to replace double quotes " in 65th field with empty
Appreciate help
# 6  
Old 03-22-2012
Why would carriage returns be wanted in any of the records, unless the entire file is delimited with \r\n ? Whether it is would be good to know.

Code:
awk -F"|" 'NF==65 { sub(/\r/, ""); T=$0; getline; print T $0; next} 1'

# 7  
Old 03-22-2012
Quote:
Originally Posted by Corona688
Why would carriage returns be wanted in any of the records, unless the entire file is delimited with \r\n ? Whether it is would be good to know.

Code:
awk -F"|" 'NF==65 { sub(/\r/, ""); T=$0; getline; print T $0; next} 1'

Thanks Corona !!

There is a possibility that all the files at the end may have carriage return as the file is ftped to us, hence i want to look only in the 65th field.

This works
Quote:
Now my data looks in 65th field like this
|ABCTX"" |
I like it to look like this.
|ABCTX |
I modified and tried the below but does not work. I dont know where i am going wrong.

Code:
 
awk -F"|" 'NF==65 { sub(/\r/, ""); sub(/"/, " "); T=$0; getline; print T $0; next} 1' 

awk -F"|" 'NF==65 { sub(/\r/, ""); sub(/\"/, " "); T=$0; getline; print T $0; next} 1' 

awk -F"|" 'NF==65 { sub(/\r/, ""); sub(/\"\"/, " "); T=$0; getline; print T $0; next} 1'

Help is appreciated.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Add a carriage return after every sorted field

Hi All, I have a text file - nmn-smt-1039.test.com:SearchService-WW:x:8277 nmn-smt-1102.test.com:AdminConsole-ww:x:8536 nmn-smt-1041.test.com:SearchService-WW:x:8277 nmn-wsf-1007.test.com:Service-ww:x:8532 nmn-smt-1042.test.com:SearchService-WW:x:8277... (3 Replies)
Discussion started by: jacki
3 Replies

2. Shell Programming and Scripting

Remove carriage return and append the next line

Hi All, My requirement is to remove the carriage return in from the lines which i am reading if the length is lesser than 1330 and append the next line with it. Below is the realistic example of file structure. Input file: Blah blah blah blah Blah blah blah blah Blah blah blah blah Blah... (16 Replies)
Discussion started by: mad man
16 Replies

3. Shell Programming and Scripting

Remove Carriage Return (CRLF) within double quotes

How to remove Carriage Return (CRLF) within double quotes in a file. There are multiple CRLFs within double quotes. We are on Ubuntu 14.04.2 LTS. The file that we are importing is a csv file from unix to windows and the file was formatted to unix2dos. Therefore all lines in the file all have... (12 Replies)
Discussion started by: covina
12 Replies

4. UNIX for Dummies Questions & Answers

Remove carriage return

I need to remove the carriage return comes inbetween the record. Need to have CR only at the end. I used the below command. tr -d '\n' < filewithcarriagereturns > filewithoutcarriagereturns But its removing all the CR and giving one line output. Input File: 12345 abcdegh... (11 Replies)
Discussion started by: srvn_saru
11 Replies

5. Shell Programming and Scripting

Remove carriage return from the variable

Hi, I try to handle very large numbers with a bash script. I run ssh command in a remote server and store the output in a local variable. But this output contains a return carriage at the end. So I try to remove it by tr But I can't figure out the right notation with printf. So my problem... (6 Replies)
Discussion started by: Meacham12
6 Replies

6. Shell Programming and Scripting

Remove carriage return in a record

Hi all gurus, I need help in removing carriage return existed within a record delimited by pipe <|>. Sample: A_01|Test1|Testing1|Remarks1 A_02|Test2|Test ing2|Remarks2 A_03|Test3|Testing3| Remarks3 Desire output: A_01|Test1|Testing1|Remarks1 A_02|Test2|Testing2|Remarks2... (10 Replies)
Discussion started by: agathaeleanor
10 Replies

7. UNIX for Dummies Questions & Answers

Remove ^M (carriage return) with string manipulation

Hello, I want remove ^M at end of my files line if I use command : tr -d '\r' <inp>out it work fine but get I the same result by manipulating the string ? I want this because in my text file I manipulate some other part I have input "the cat^M" I want output "the cat" I have made... (3 Replies)
Discussion started by: aquila_1
3 Replies

8. Shell Programming and Scripting

sqlplus returning value - remove carriage return '\r' - Please help

Guys - Simple code, i am trying to get a number back from sqlplus call to a query. After that, i need to use that number in a loop. --------------------------------- #!/bin/ksh VALUE=`sqlplus -silent sh/password@sh <<END set pagesize 0 feedback off verify off heading off echo off select... (10 Replies)
Discussion started by: sunshine1974
10 Replies

9. UNIX for Dummies Questions & Answers

To remove carriage return between the line

Hi, I have a situation where I need to remove the carriage return between the lines. For.eg. The input file: 1,ad,"adc sdfd",edf 2,asd,"def fde",asd The output file should be 1,ad,adc sdfd,edf 2,asd,def fde,asd Thanks Shash (5 Replies)
Discussion started by: shash
5 Replies

10. UNIX for Dummies Questions & Answers

Remove a carriage return at end of variable

Is there a command in unix to remove a carriage return character(^M) at the end of a variable value? (5 Replies)
Discussion started by: flagship99
5 Replies
Login or Register to Ask a Question