Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove Carriage Return (CRLF) within double quotes

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove Carriage Return (CRLF) within double quotes
# 1  
Old 07-24-2015
Remove Carriage Return (CRLF) within double quotes
How to remove Carriage Return (CRLF) within double quotes in a file. There are multiple CRLFs within double quotes. We are on Ubuntu 14.04.2 LTS.

The file that we are importing is a csv file from unix to windows and the file was formatted to unix2dos. Therefore all lines in the file all have CRLF at the end. However, there is a comment field where we see multiple CRLFs within this single field and the contents of the field is enclosed with double quotes. So, when we open the file in notepad it looks like a single line is broken into several lines. If the file is opened in the excel it comes out OK.

My requirement is to replace all CRLFs within double quotes should be replaced by a single space.
# 2  
Old 07-24-2015
Hello Covina,

Welcome to forums, hope you will enjoy learning here. Following are some examples which may help you to remove carriage characters.
Code:
tr -d '\r' < Input_file > Output_file
awk '{gsub(/\r/, " ");print}' Input_file > Output_file

Apart from this you can have dos2unix utility installed in your box too. Hope this helps.
Enjoy learning Smilie


Thanks,
R. Singh
# 3  
Old 07-24-2015
I'd be surprised if a field in a .csv file had <CR><LF> chars in it, as usually EXCEL uses single <LF> chars for in-cell line breaks.
Please post (attach) a small but meaningful sample file on which we can work.
Last edited by RudiC; 07-24-2015 at 02:31 PM..
# 4  
Old 07-24-2015
The problem started when the file was transferred to Unix and all CRLF became LF, now no longer distinguishable from the embedded LF.
Converting all to CRLF does not help.
Maybe you can redo the original download/transfer to Windows?
In case of ftp transfer use binary mode!
# 5  
Old 07-24-2015
Please copy and paste in Note++ and you will see it (display the end of line symbol to see CRLF)
Here is the sample file:
Code:
6,,G0570405,037112,13907187-P06241,P06241,,B1249998,,,03/10/2014,12:42:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,1301,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --

Processed in purification",Small Box,,WB for DNA,Yes,3/11/2014 11:55 AM,130100010,,,,,,0,,
9,,,,12112008,BAF312A2304,EH07583,G0521371,2024001,D-D,02/21/2014,09:22:00,03/11/2014 09:00:00,,,,,Male,1240924436,,Frozen Acceptable ,2024,DNA PaxGene Tube,V1CYP,,"Partsample Barcode: 01240924436011
Material: BA
Sample Description: CYP2C9 Haplotype
Eurofins Study Code: 2181261098

-- Added 3/11/2014 2:36 PM by abc --

Processed in purification",Small Box,,WB for DNA,No,3/11/2014 12:18 PM,,,,,,,0,,
10,,,,CL15312003,12-102-13,,G0573245,0135-1306,,03/10/2014,,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --

Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
11,,,,CL15312003,12-102-13,,G0576563,0135-1308,,03/10/2014,15:45:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --

Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
12,,,,CL15312003,12-102-13,,G0576562,0135-1307,,03/10/2014,12:48:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --

Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
24,,,,19012034,BAN2401-G000-201,I6720150214,G0521407,10901054,,03/04/2014,,03/11/2014 10:00:00,,,,,Female,,,Frozen Acceptable ,1090,EDTA Tube,2 (Baseline),,"Sample Name: PG (APOE4)
Year of Birth: 1933",Small Box,,WB for DNA,No,3/11/2014 3:20 PM,,,,,,,0,28456103,

I wanted to be formatted like this:
Code:
5,,G0570409,037112,13907187-P06241,P06241,,B1249997,,,03/10/2014,12:42:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,1301,SST Tube,,,,Large Box,,Serum,Yes,3/11/2014 11:07 AM,130100010,,,,,,0,,
6,,G0570405,037112,13907187-P06241,P06241,,B1249998,,,03/10/2014,12:42:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,1301,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --Processed in purification",Small Box,,WB for DNA,Yes,3/11/2014 11:55 AM,130100010,,,,,,0,,
9,,,,12112008,BAF312A2304,EH07583,G0521371,2024001,D-D,02/21/2014,09:22:00,03/11/2014 09:00:00,,,,,Male,1240924436,,Frozen Acceptable ,2024,DNA PaxGene Tube,V1CYP,,"Partsample Barcode: 01240924436011 Material: BA Sample Description: CYP2C9 Haplotype Eurofins Study Code: 2181261098 -- Added 3/11/2014 2:36 PM by abc -- Processed in purification",Small Box,,WB for DNA,No,3/11/2014 12:18 PM,,,,,,,0,,
10,,,,CL15312003,12-102-13,,G0573245,0135-1306,,03/10/2014,,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
11,,,,CL15312003,12-102-13,,G0576563,0135-1308,,03/10/2014,15:45:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
12,,,,CL15312003,12-102-13,,G0576562,0135-1307,,03/10/2014,12:48:00,03/11/2014 10:00:00,,,,,,,,Cool/Acceptable,135,DNA PaxGene Tube,,,"-- Added 3/11/2014 2:36 PM by abc --Processed in purification",Small Box,3064,WB for DNA,Yes,3/11/2014 12:36 PM,,,,,,,0,,
24,,,,19012034,BAN2401-G000-201,I6720150214,G0521407,10901054,,03/04/2014,,03/11/2014 10:00:00,,,,,Female,,,Frozen Acceptable ,1090,EDTA Tube,2 (Baseline),,"Sample Name: PG (APOE4) Year of Birth: 1933",Small Box,,WB for DNA,No,3/11/2014 3:20 PM,,,,,,,0,28456103

Last edited by Scrutinizer; 07-24-2015 at 06:56 PM.. Reason: better file; mod: code tags
# 6  
Old 07-24-2015
Code:
 perl -07 -pe 's/\n\n//g' broken_cvs > fixed_cvs

# 7  
Old 07-24-2015
no, I am still not getting it. There should be only one CRLF at the end of the line. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove carriage return and append the next line

Hi All, My requirement is to remove the carriage return in from the lines which i am reading if the length is lesser than 1330 and append the next line with it. Below is the realistic example of file structure. Input file: Blah blah blah blah Blah blah blah blah Blah blah blah blah Blah... (16 Replies)
Discussion started by: mad man
16 Replies

2. UNIX for Dummies Questions & Answers

Remove carriage return

I need to remove the carriage return comes inbetween the record. Need to have CR only at the end. I used the below command. tr -d '\n' < filewithcarriagereturns > filewithoutcarriagereturns But its removing all the CR and giving one line output. Input File: 12345 abcdegh... (11 Replies)
Discussion started by: srvn_saru
11 Replies

3. Shell Programming and Scripting

Remove carriage return from the variable

Hi, I try to handle very large numbers with a bash script. I run ssh command in a remote server and store the output in a local variable. But this output contains a return carriage at the end. So I try to remove it by tr But I can't figure out the right notation with printf. So my problem... (6 Replies)
Discussion started by: Meacham12
6 Replies

4. Shell Programming and Scripting

Awk to remove carriage return from 65th field

Hi, I have a pipe delimited file. There are around 700 columns in the file. The 65th column has carriage return which is causing read issue with our ETL process. I would like to replace the new line characters in 65th field with "nothing" i have return the following code and need help to... (7 Replies)
Discussion started by: pinnacle
7 Replies

5. Shell Programming and Scripting

Excel CSV with Double Quotes and Carriage Returns

Hi List, I am sure this issue must have been encountered before. I have saved an excel CSV and sent it to my linux box where I require to parse it as a CSV line by line. The file displays fine in Excel with csv headers and consistent records and fields. However when I look at it in linux I see... (2 Replies)
Discussion started by: landossa
2 Replies

6. Shell Programming and Scripting

Remove carriage return in a record

Hi all gurus, I need help in removing carriage return existed within a record delimited by pipe <|>. Sample: A_01|Test1|Testing1|Remarks1 A_02|Test2|Test ing2|Remarks2 A_03|Test3|Testing3| Remarks3 Desire output: A_01|Test1|Testing1|Remarks1 A_02|Test2|Testing2|Remarks2... (10 Replies)
Discussion started by: agathaeleanor
10 Replies

7. UNIX for Dummies Questions & Answers

Remove ^M (carriage return) with string manipulation

Hello, I want remove ^M at end of my files line if I use command : tr -d '\r' <inp>out it work fine but get I the same result by manipulating the string ? I want this because in my text file I manipulate some other part I have input "the cat^M" I want output "the cat" I have made... (3 Replies)
Discussion started by: aquila_1
3 Replies

8. Shell Programming and Scripting

sqlplus returning value - remove carriage return '\r' - Please help

Guys - Simple code, i am trying to get a number back from sqlplus call to a query. After that, i need to use that number in a loop. --------------------------------- #!/bin/ksh VALUE=`sqlplus -silent sh/password@sh <<END set pagesize 0 feedback off verify off heading off echo off select... (10 Replies)
Discussion started by: sunshine1974
10 Replies

9. UNIX for Dummies Questions & Answers

To remove carriage return between the line

Hi, I have a situation where I need to remove the carriage return between the lines. For.eg. The input file: 1,ad,"adc sdfd",edf 2,asd,"def fde",asd The output file should be 1,ad,adc sdfd,edf 2,asd,def fde,asd Thanks Shash (5 Replies)
Discussion started by: shash
5 Replies

10. UNIX for Dummies Questions & Answers

Remove a carriage return at end of variable

Is there a command in unix to remove a carriage return character(^M) at the end of a variable value? (5 Replies)
Discussion started by: flagship99
5 Replies
Login or Register to Ask a Question