Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replace Double quotes within double quotes in a column with space while loading a CSV file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replace Double quotes within double quotes in a column with space while loading a CSV file
# 1  
Old 05-11-2015
Replace Double quotes within double quotes in a column with space while loading a CSV file
Hi All,

I'm unable to load the data using sql loader where there are double quotes within the double quotes As these are optionally enclosed by double quotes.

Sample Data :

Code:
"221100",138.00,"D","0019/1477","44012075","49938","49938/15043000","Television - 22" Refurbished - Airwave","Supply of Delivery & Collection 1st Unit (1), Delivery & Collection Additions (1), Whitbread Refurb LCD (2)","Airwave Europe Ltd","15/04/2015","2520",""

Desired output :

Code:
"221100",138.00,"D","0019/1477","44012075","49938","49938/15043000","Television  - 22 Refurbished - Airwave","Supply of Delivery & Collection 1st  Unit (1), Delivery & Collection Additions (1), Whitbread Refurb LCD  (2)","Airwave Europe Ltd","15/04/2015","2520",""

I have checked for many threads posted in this site. and tried
Code:
sed 's/\([^",]\)"\([^",]\)/\1\2/' < infile > outfile

Code:
perl -anle 'my @fields = ($_ =~ /(?:^|,)(".*?"|[^,]*?)(?=,|$)/g);foreach my $f(@fields){$f=~s/"//g;$f=sprintf("\"%s\"",$f);}my $line=join(",",@fields);print $line' file

But it didn't work. If the last column of the data is blank. then it is changing for that as well and getting the below output.

Code:
"221100",138.00,"D","0019/1477","44012075","49938","49938/15043000","Television  - 22 Refurbished - Airwave","Supply of Delivery & Collection 1st  Unit (1), Delivery & Collection Additions (1), Whitbread Refurb LCD  (2)","Airwave Europe Ltd","15/04/2015","2520","

Could anyone help me out to fix this issue.

Regards,
Lavanya.
Last edited by rbatte1; 05-11-2015 at 12:45 PM..
# 2  
Old 05-11-2015
You sed snippet works for me - it removes exactly the " after the 22. Why don't you like that solution?
# 3  
Old 05-11-2015
Change outer double quotes to an unprintable (here a literal ^B, so remember that not carat B, but ctrl-B). Then eliminate the remaining double quotes and replace all ^Bs with double quotes.

Code:
sed -e 's/^"/^B/' -e 's/"$/^B/' -e 's/",/^B,/g' -e 's/,"/,^B/g' -e 's/"//g' -e 's/^B/"/g'

This User Gave Thanks to cjcox For This Post:
rbatte1
# 4  
Old 05-11-2015
Hi RudiC,

It doesnt work for me as for the data should be some thing like below:

Input:

Code:
   560003_07.28,292.47,"D","1073/1220","44536370","16520","16520/14103000","Vacuum   - Upright (c) - "Vaclensa","Supply of BS36 Upright (3yr NO QUIBBLE   Guarantee) (1)","Vaclensa PLC","03/10/2014","2510","PINON15N001"

After using

sed 's/\([^",]\)"\([^",]\)/\1\2/' < Input file> output_file


Output:
Code:
   560003_07.28,292.47,"D","1073/1220","44536370","16520","16520/14103000","Vacuum   - Upright (c) - Vaclensa","Supply of BS36 Upright (3yr NO QUIBBLE   Guarantee) (1)","Vaclensa PLC","03/10/2014","2510","PINON15N001

Expected Ouptut:

Code:
   560003_07.28,292.47,"D","1073/1220","44536370","16520","16520/14103000","Vacuum   - Upright (c) - Vaclensa","Supply of BS36 Upright (3yr NO QUIBBLE   Guarantee) (1)","Vaclensa PLC","03/10/2014","2510","PINON15N001"


Can u please help me out with this.

Regards,
Lavanya.

---------- Post updated at 01:04 AM ---------- Previous update was at 01:01 AM ----------

Hi Cjcox,

Can u please confirm if i can write the whole code you provided as a single SED command.
As im new to this technology and trying to learn .

Regards,
Lavanya.
# 5  
Old 05-11-2015
Quote:
Originally Posted by mlavanya
... ... ...

Hi Cjcox,

Can u please confirm if i can write the whole code you provided as a single SED command.
As im new to this technology and trying to learn .

Regards,
Lavanya.
Yes Lavanya, you can use a single invocation of the sed utility to perform all six sed substitute commands as shown in the suggestion cjcox provided.

Obviously, you will have to provide input for that command to process, and, unless you just want the output to go to standard output, you'll need to redirect the output.
# 6  
Old 05-11-2015
Hi Don,

I tried using :

sed -e 's/^"/^B/' -e 's/"$/^B/' -e 's/",/^B,/g' -e 's/,"/,^B/g' -e 's/"//g' -e 's/^B/"/g'
but i could see that all the double quotes are replaced by
^B
and also eliminating the last double quote in the data file.

Input:

Code:
   221100,37.20,"C","0073/1454","44019120","16395","16395/14103000","Safety   Workwear - "Screwfix","","Screwfix Direct   Ltd","10/10/2014","2520",""

Output:

Code:
   ^B221100^B,37.20,^BC^B,^B0073/1454^B,^B44019120^B,^B16395^B,^B16395/14103000^B,^BSafety   Workwear - Screwfix^B,^B^B,^BScrewfix Direct Ltd^B,^B10/10/2014^B,^B2520^B,^B

Expected Output:

Code:
   221100,37.20,"C","0073/1454","44019120","16395","16395/14103000","Safety   Workwear - Screwfix","","Screwfix Direct   Ltd","10/10/2014","2520",""

If you can see in the output , the last column with null value has been replaced only with a single double quote.

Please help me to resolve this issue.

Regards,
Lavanya.

---------- Post updated at 08:29 AM ---------- Previous update was at 08:15 AM ----------

Actually i have problem only with 8th column of the data file. Can we do the change only for column 8 , to check if there is any double quotes between double quotes and replace it with space.
# 7  
Old 05-12-2015
Go back and look at message #3 in this thread again. You seem to have used the two characters circumflex (^) and capital letter b (B) instead of the single character that you get by pressing and holding the control key (control, ctl, or cntl on your keyboard depending on your keyboard manufacturer) while you press and release the B key. This key combination would show up on your editing screen as ^B if you were using common UNIX/Linux/POSIX editing tools like vi.

If, for some reason, you are unable to use the ctl-B key combination to create that character, you can replace all occurrences of that character in the sed command line with any other character that CANNOT appear as a legitimate character in your input file except that you cannot use a character that has a special meaning in a basic regular expression nor that has a special meaning in a sed s command replacement string.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need shell script to append double quotes for each column in a file

Hi Experts, I am beginner to the shell scripting, My requirement is to append double quotes for each column in a file if double quotes does not exist. Example: "abc"|123|"gh-ch"|23.067 Use code tags, thanks. (10 Replies)
Discussion started by: spidy
10 Replies

2. Shell Programming and Scripting

Shell script that should remove unnecessary commas between double quotes in CSV file

i have data as below 123,"paul phiri",paul@yahoo.com,"po.box 23, BT","Eco Bank,Blantyre,Malawi" i need an output to be 123,"paul phiri",paul@yahoo.com,"po.box 23 BT","Eco Bank Blantyre Malawi" (5 Replies)
Discussion started by: mathias23
5 Replies

3. Shell Programming and Scripting

Replace double quotes with a single quote within a double quoted string

Hi Froum. I have tried in vain to find a solution for this problem - I'm trying to replace any double quotes within a quoted string with a single quote, leaving everything else as is. I have the following data: Before: ... (32 Replies)
Discussion started by: pchang
32 Replies

4. Shell Programming and Scripting

How to delete the commas in a .CSV file that are enclosed in a string with double quotes?

Okay, I would like to delete all the commas in a .CSV file (TEST.CSV) or at least substitute them with empty space, that are enclosed in double quote. Please see the sample file as below: column 1,column 2,column 3,column 4,column 5,column 6,column 7,column 8,column 9,column 10... (8 Replies)
Discussion started by: dhruuv369
8 Replies

5. Shell Programming and Scripting

Issue with Single Quotes and Double Quotes for prompt PS1

Hi, Trying to change the prompt. I have the following code. export PS1=' <${USER}@`hostname -s`>$ ' The hostname is not displayed <abc@`hostname -s`>$ uname -a AIX xyz 1 6 00F736154C00 <adcwl4h@`hostname -s`>$ If I use double quotes, then the hostname is printed properly but... (3 Replies)
Discussion started by: bobbygsk
3 Replies

6. Shell Programming and Scripting

Replace newline character between a double quotes to a space

Hi Guys, I have a file with content as below aj.txt "Iam allfine" abcdef abcd "all is not well" What I'm trying to say is my data has some new line characters in between quoted text. I must get ride of the newline character that comes in between the quoted text. output must be:... (8 Replies)
Discussion started by: ajahuja
8 Replies

7. Shell Programming and Scripting

HELP with AWK or SED. Need to replace the commas between double quotes in CSV file

Hello experts, I need to validate a csv file which contains data like this: Sample.csv "ABCD","I",23,0,9,,"23/12/2012","OK","Street,State, 91135",0 "ABCD","I",23,0,9,,"23/12/2012","OK","Street,State, 91135",0 I just need to check if all the records contain exactly the number of... (5 Replies)
Discussion started by: shell_boy23
5 Replies

8. Shell Programming and Scripting

Replace double double quotes using AWK/SED

Hi, I have data as "01/22/97-"aaaaaaaaaaaaaaaaa""aaa""aabbbbbbbbcccccc""zbcd""dddddddddeeeeeeeeefffffff" I want to remove only the Consequitive double quotes and not the one which occurs single. My O/P must be ... (2 Replies)
Discussion started by: Bhuvaneswari
2 Replies

9. Shell Programming and Scripting

Replacing comma with in double quotes in a csv file

Hello, I need to read a csv file and I am trying to replace a comma with a text DSEE?DSEE. Example Input "Chapter","NewTrains, "oldTrains","Delayed",10,"London" "Chapter","Newbuses,oldbuses","On Time",20,"London" Output "Chapter","NewTrainsDSEE?DSEE... (5 Replies)
Discussion started by: venkatvani
5 Replies

10. Shell Programming and Scripting

replace value with double quotes of specific coulmn value in csv file

Hi, I am trying to replace a specific column values in a csv file with double quotes. Example: SNO,NAME,ZIPCODE,RANK 1,Robert,74538,12 2,Sam,07564,13 3,Kim, Ed,12345,14 Desired Output: SNO,NAME,ZIPCODE,RANK 1,Robert Ken,74538,12 2,Sam Mik,"07564",13 3,"Kim, Ed",12345,14 I... (3 Replies)
Discussion started by: techmoris
3 Replies
Login or Register to Ask a Question