Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replace double quotes inside the string data for all the columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replace double quotes inside the string data for all the columns
# 8  
Old 05-16-2018
It's following the same logic as does your sed approach but takes advantage of awk having a "finer grained" access to parts of the line, i.e. fields. Double any double quote in each field with gsub (/"/, "&&", $i), and then remove the first and the last in the field.
# 9  
Old 05-17-2018
Hi Rudic,

Thanks for the email.
Could you please explain what is the reason NF<7(number of field) conditions? due to this,some of the flat files are not working i guess.

Code:
awk  -F\| '
        {while (NF < 7) {getline X
                         $0 = $0 "\001" X
                        }
         for (i=1; i<=NF; i++)  {gsub (/"/, "&&", $i)
                                 sub (/^"/, "", $i)
                                 sub (/"$/, "", $i)
                                }
         gsub (/\001/, "\n")
        }
1
' OFS=\| test file


Thanks,

Regards,
Srinivasan S

Thanks,

Srini




Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!
Last edited by RudiC; 05-17-2018 at 05:58 AM.. Reason: Added CODE tags.
# 10  
Old 05-17-2018
Quote:
Originally Posted by SSrini
. . .
some of the flat files are not working i guess.

Then, the sample given was not representative, I guess.



Quote:
explain what is the reason NF<7(number of field) conditions?

. . .
The <new line> char in some fields starts a new line in *nix text files, and awk sees exactly those. So the code repeatedly appends lines until all fields are present. If the fields count is not constant across lines or even files, things become way more complicated. You need to define a criterion to tell a complete line from a broken one.
# 11  
Old 05-17-2018
H srini...
Code:
emp|^^|ename|^^|sal|^^|refname
"PM"|^^|"R""A""M"|^^|100|^^|"X"""YZ"
"RM"|^^|"PR
RED""
ORANG""
SRI"|^^|500|^^|"ABCD"
"AM"|^^|"""MAR"""|^^|200|^^|"ABC"""

Just an observation here, on line 2 there are 13 double quotes. Should there be 14 or should there be 12 as an odd number seems just exactly that - very ODD.
# 12  
Old 05-18-2018
Hi Rudic and wisecracker,

Thanks for the email.
We have requested the source team for changing the delimited format from |^^| to |~~|.

After changing this. the NF command count is working as expected.
somehow the carrot symbol delimiter number of field count is not working.


Code:
dlmt="[|]~~[|]"
vcol_cnt=`head -1 $Inpufile|awk -F $dlmt '{print NF}'`
echo "Number of Columns:$vcol_cnt"
 

awk  -v colcount=$vcol_cnt -F $dlmt  '
        {while (NF < colcount) {getline X
                         $0 = $0 "\001" X
                        }
         for (i=1; i<=NF; i++)  {gsub (/"/, "&&", $i)
                                 sub (/^"/, "", $i)
                                 sub (/"$/, "", $i)
                                }
         gsub (/\001/, "\n")
        }
1
' OFS=\, $Inpufile

Thanks,
Srini
# 13  
Old 05-18-2018
I don't think the delimiter influences the logics. And, you seem to be sure that the first line definitely determines the field count. If you can rely on that - why not do all of it in one awk script?
# 14  
Old 05-18-2018
Hi Rudic,

Thanks for the email.
Could you please possible to provide the sed command for the above scenario instead of awk command? the awk command is working fine but am seeing the performance wise slow.

Thanks,

Regard,
Srinivasan S
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace Double quotes within double quotes in a column with space while loading a CSV file

Hi All, I'm unable to load the data using sql loader where there are double quotes within the double quotes As these are optionally enclosed by double quotes. Sample Data : "221100",138.00,"D","0019/1477","44012075","49938","49938/15043000","Television - 22" Refurbished - Airwave","Supply... (6 Replies)
Discussion started by: mlavanya
6 Replies

2. Programming

Replace comma which is not inside brackets,quotes or paranthesis

Hi All, I want to replace the commas which are not inside parenthesis,quotes if input is abc,,lm,(no,pq,rs),{tu,vw,xy},zs,"as,as,fr",'ab,cd,ef' output should be abc lm (no,pq,rs) {tu,vw,xy} zs "as,as,fr" 'ab,cd,ef' I tried this str.replaceAll("\\(.*?\\)|(,)", " "); say my string... (3 Replies)
Discussion started by: preethy
3 Replies

3. Shell Programming and Scripting

sed command to replace string that contain blackslash,double quotes

Hi All, I have been trying to replace a string using the sed command string value contain blackslash and double quotes. I am not a expert writer of unix script but do try not to ask question. I have almost given up. Hope you all can give me some suggestion I want to replace a place string... (6 Replies)
Discussion started by: thanush9sep
6 Replies

4. Shell Programming and Scripting

Replace double quotes with a single quote within a double quoted string

Hi Froum. I have tried in vain to find a solution for this problem - I'm trying to replace any double quotes within a quoted string with a single quote, leaving everything else as is. I have the following data: Before: ... (32 Replies)
Discussion started by: pchang
32 Replies

5. Shell Programming and Scripting

Preserve commas inside double quotes (perl)

Hi, I have an input file like this $ cat infile hi,i,"am , sam", y hello ,good, morning abcd, " ef, gh " ,ij no, "good,morning", yes, "good , afternoon" from this file I have to split the fields on basis of comma"," however, I the data present inside double qoutes should be treated as... (3 Replies)
Discussion started by: sam05121988
3 Replies

6. Shell Programming and Scripting

Replace double double quotes using AWK/SED

Hi, I have data as "01/22/97-"aaaaaaaaaaaaaaaaa""aaa""aabbbbbbbbcccccc""zbcd""dddddddddeeeeeeeeefffffff" I want to remove only the Consequitive double quotes and not the one which occurs single. My O/P must be ... (2 Replies)
Discussion started by: Bhuvaneswari
2 Replies

7. Shell Programming and Scripting

To append two columns without double quotes

Hi i have a file with follw data "20090427","0","","16371311","-100200","","","","16371311","JUL","09" In the 10th column i need to convert the month name into month number in this case JUL will be 7 and append the 10th and 11th column which shows me the output as 709. Can you suggest a shell... (11 Replies)
Discussion started by: vee789
11 Replies

8. Shell Programming and Scripting

To Replace comma with Pipe inside double quotes

Hi, I have a requirement to replace the comma's inside the double quotes. The comma's inside the double quotes will get changed dynamically. Input Record: "Washington, DC,Prabhu,aju",New York Output Record: "Washington| DC|Prabhu|aju",New York I tried with the below command but it... (3 Replies)
Discussion started by: prabhutkl
3 Replies

9. Shell Programming and Scripting

sed removing comma inside double quotes

I have a csv file with lines like the followings 123456,"ABC CO., LTD","XXX" 789012,"DEF LIMITED", "XXX" before I bcp this file to database, the comma in "CO.," need to be removed first. My script is cat <filename> | sed 's/"CO.,"/"CO."/g' but it doesn't work. Can anyone here able to... (2 Replies)
Discussion started by: joanneho
2 Replies

10. Shell Programming and Scripting

escaping double-quotes inside the script?

I'm having a strange problem with escaping double-quotes. I have a script that looks like this: #!/bin/bash for HOST in `cat $INFILE | grep -v ^#` do for VFILER in `some_command` do echo " " echo -e '\E The problem with ssh command... (3 Replies)
Discussion started by: GKnight
3 Replies
Login or Register to Ask a Question