Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to delete a column/columns of a CSV file which has cell values with a string enclosed in " , "?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to delete a column/columns of a CSV file which has cell values with a string enclosed in " , "?
# 1  
Old 06-20-2013
How to delete a column/columns of a CSV file which has cell values with a string enclosed in " , "?
How can I delete a column from a CSV file which has comma separated value with a string enclosed in double quotes and a comma in between? I have a file 44.csv with 4 lines including the header like the below format:

Code:
column1, column2, column3, column 4, column5, column6
12,455,"string with quotes, and with a comma in between",4432,6787,890,88
4432,6787,"another, string with quotes, and with two comma in between",890,88,12,455
11,22,"simple string",77,777,333,22

I need to cut the 1,2,3 columns from the file, so I used the cut command as below

Code:
cut -d"," -f1,2,3 44.csv > 444.csv

I am getting the output as

Code:
column1, column2, column3
12,455,"string with quotes
4432,6787,"another string with quotes
11,22,"simple string"

But I need the output to be

Code:
column1, column2, column3
12,455,"string with quotes, and with a comma in between"
4432,6787,"another, string with quotes, and with two comma in between"
11,22,"simple string"

Any help is greatly appreciated.

Thanks
Moderator's Comments:
Mod Comment Please use CODE tags.
Last edited by Don Cragun; 06-20-2013 at 09:01 PM.. Reason: Added CODE tags
# 2  
Old 06-20-2013
Please use code tags for posting data samples.

Here is an awk approach that might work for your data:
Code:
awk -F, 'NR==1{NF-=3}NF>1{sub(/",.*/,"\"")}1' OFS=, file

# 3  
Old 06-24-2013
Thanks Yoda, I am new to UNIX and did not know about the CODE TAGS. I will make sure to use them on my next questions... However, the code worked, but can you please explain the code... How do I select other columns other than a given range... Lets say, I need to select columns which are not consecutive... I appreciate your help...
# 4  
Old 06-24-2013
Quote:
Originally Posted by dhruuv369
can you please explain the code...
Code:
awk -F, '                               # Set comma as input field separator
        NR == 1 {                       # If NR (total records read) == 1
                NF -= 3                 # Subtract 3 from NF (number of fields in current record)
        }
        NF > 1 {                        # If NR (total records read) > 1
                sub (/",.*/, "\"")      # Substitute " followed by , followed by .* (0 or more occurrence of any char) with "
        }
        1                               # 1 == true (if true, default awk action is print)
' OFS=, file                            # Set comma as output field separator

These 2 Users Gave Thanks to Yoda For This Post:
dhruuv369 RavinderSingh13
# 5  
Old 06-24-2013
Thanks Yoda for the details in explaining the code, can you please answer the second part of my question too...
Quote:
How do I select other columns other than a given range... Lets say, I need to select specific columns(1,5,10 from a file) which are not consecutive... I appreciate your input on this
# 6  
Old 06-24-2013
Yoda's proposal is working fine if the quoted column is the last one you want to keep. Try this for an arbitrary last column:
Code:
awk     '       {for (i=2; i<=NF; i+=2) {                               #  every second field is one inside double quotes
                         gsub (/,/, "\001", $i)                         # replace every comma in quoted field by something
                        }
                 split ($0, TMP, ",")                                   # get the comma separated fields into an temp array
                 $0=""
                 for (i=1; i<=COLS; i++) $0=$0 ($0?",":"") TMP[i]       # rebuild $0 from this array using only COLS columns
                 gsub ("\001", ",")                                     # replace something back to commas
                }
         1                                                              # print new $0 string
        ' FS="\"" OFS="\"" COLS=4 file                                  # parameter COLS has number of columns to keep

This is not yet what you requested in your last post, to keep any arbitrary column, but you can use it as a starting point...


EDIT: try this to keep arb. columns:
Code:
awk     'FNR==1 {CNT=split(COLS, CNo, ",")}                             # get desired columns into array
                {for (i=2; i<=NF; i+=2) {                               # every second field is one inside double quotes
                         gsub (/,/, "\001", $i)                         # replace every comma in quoted field by something
                        }
                 split ($0, TMP, ",")                                   # now get the comma separated fields into an temp array
                 $0=""
                 for (i=1; i<=CNT; i++) $0=$0 ($0?",":"") TMP[CNo[i]]   # rebuild $0 from this array using only COLS columns
                 gsub ("\001", ",")                                     # replace something back to commas
                }
         1                                                              # print new $0 string
        ' FS="\"" OFS="\"" COLS=1,3,6 file                              # parameter COLS has number of columns to keep
column1, column3, column6
455,12,890
4432,"another, string with quotes, and with two comma in between",12
11,"simple string",333

(in line 2, I moved the quoted string to field 2)
Last edited by RudiC; 06-24-2013 at 03:20 PM..
This User Gave Thanks to RudiC For This Post:
dhruuv369
# 7  
Old 06-25-2013
Thank you RudiC for explaining the code, it works perfectly for any given columns. Cheers
Last edited by dhruuv369; 06-25-2013 at 11:36 AM.. Reason: No Specific reason...
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete all log files older than 10 day and whose first string of the first line is "MSH" or "<?xml"

Dear Ladies & Gents, I have a requirement to delete all the log files in /var/log/test directory that are older than 10 days and their first line begin with "MSH" or "<?xml" or "FHS". I've put together the following BASH script, but it's erroring out: for filename in $(find /var/log/test... (2 Replies)
Discussion started by: Hiroshi
2 Replies

2. UNIX for Dummies Questions & Answers

Replacing "." with "GG" in a certain column of a file that has heading

Hi, all, I have a file that looks like: ## XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ## YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY #AA AB AC AD AE AF AG AH AI AJ AK AL 20 60039 60039 ... (5 Replies)
Discussion started by: kush
5 Replies

3. Shell Programming and Scripting

How to delete a columns of a CSV file which has cell values with a string enclosed in " , "?

Hi How can I delete a columns from a CSV file which has comma separated value with a string enclosed in double quotes or square bracket and a comma in between? I have a csv file with below format. Template,Target Server,Target Component,Rule Group,Rule,Rule Reference Number,Rule... (7 Replies)
Discussion started by: Litu19
7 Replies

4. Linux

To get all the columns in a CSV file based on unique values of particular column

cat sample.csv ID,Name,no 1,AAA,1 2,BBB,1 3,AAA,1 4,BBB,1 cut -d',' -f2 sample.csv | sort | uniq this gives only the 2nd column values Name AAA BBB How to I get all the columns of CSV along with this? (1 Reply)
Discussion started by: sanvel
1 Replies

5. Shell Programming and Scripting

How to delete the commas in a .CSV file that are enclosed in a string with double quotes?

Okay, I would like to delete all the commas in a .CSV file (TEST.CSV) or at least substitute them with empty space, that are enclosed in double quote. Please see the sample file as below: column 1,column 2,column 3,column 4,column 5,column 6,column 7,column 8,column 9,column 10... (8 Replies)
Discussion started by: dhruuv369
8 Replies

6. Shell Programming and Scripting

Working with CSV files values enclosed with ""

I have a CSV file as shown below "1","SANTHA","KUMAR","SAM,MILLER","DEVELOPER","81,INDIA" "2","KAPIL","DHAMI","ECO SPORT","DEVELOPER","82,INDIA" File is comma delimited.All the field values are enclosed by double quotes. But while using awk or cut, it interprets the comma which is present in... (6 Replies)
Discussion started by: santhansk
6 Replies

7. Shell Programming and Scripting

Choosing between repeated entries based on the "absolute values" of a column

Hello, I was looking for a way to select between the repeated entries (column1) based on the values of absolute values of column 3 (larger value). For example if the same gene id has FC value -2 and 1, I should get the output as -2. Kindly help. GeneID Description FC ... (2 Replies)
Discussion started by: Sanchari
2 Replies

8. UNIX for Dummies Questions & Answers

replace "," with "." only in specific columns of a file?

Hi all, I have this text file containing 9 columns separated by space. The 8th columns contains the numbers. C1 C2 C3 C4 C5 C6 C7 C8 C9 er rt yt gh iu nk il 0.07 xs yt lr ty bg iu zk nh 0,0005 lt ...etc. I want to replace the comma with full stop only in 8th coloumn. the output... (8 Replies)
Discussion started by: Unilearn
8 Replies

9. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies
Login or Register to Ask a Question