Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Converting comma separated to pipe delimited file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Converting comma separated to pipe delimited file
# 1  
Old 05-11-2010
Error Converting comma separated to pipe delimited file
Hi,

I came across a very good script to convert a comma seperated to pipe delimited file in this forum. the script serves most of the requirement but looks like it does not handle embedded double quotes and commas i.e if the input is like
Code:
 
1234, "value","first,second", "LDC5"monitor", "three""","four"

The desired output should be
Code:
1234|value|first,second|LDC5"monitor|three""|four

but the output is like
Code:
1234|value|first|second|LDC5monitor|three"|four"

please suggest if it possible to modify the code such that it handles comma and double quotes in the substring.
Code:
BEGIN { FS=SUBSEP; OFS="|" }

{
  result = setcsv($0, ",")
  print
}

function setcsv(str, sep, i) {
  gsub(/""/, "\035", str)
  gsub(sep, FS, str)

  while (match(str, /"[^"]*"/)) {
    middle = substr(str, RSTART+1, RLENGTH-2)
    gsub(FS, sep, middle)
    str = sprintf("%.*s%s%s", RSTART-1, str, middle,
      substr(str, RSTART+RLENGTH))
  }
  if (index(str, "\"")) {
    return ((getline) > 0) ? setcsv(str (RT != "" ? RT : RS) $0, sep) : !setcsv(str "\"", sep)
  } else {
    gsub(/\035/, "\"", str)
    $0 = str

    for (i = 1; i <= NF; i++)
      if (match($i, /^"+$/))
        $i = substr($i, 2)

    $1 = $1 ""
    return 1
  }
}

Last edited by zaxxon; 05-11-2010 at 07:10 AM.. Reason: use code tags also for data and logs, ty
# 2  
Old 05-11-2010
Something like this?
Code:
awk -F, '{gsub(",", "|"); gsub("\"", "")}1' OFS="|" file

# 3  
Old 05-11-2010
this will replace the comma in the substring with pipe

i.e 1234,"test,file" will be replaced as 1234|test|file
# 4  
Old 05-11-2010
There should be a better way in sed but this will do what you want...

Code:
sed -e 's/ //g' -e 's/\"\,\"/\|/g' -e 's/\,\"/\|/g' -e 's/\"$//' infile



---------- Post updated at 01:52 PM ---------- Previous update was at 01:29 PM ----------

Or awk...

Code:
awk '{gsub(" ","");gsub("\"$", "");gsub("\",\"", "|");gsub(",\"","|")}1' infile

This User Gave Thanks to malcomex999 For This Post:
pseudocoder
# 5  
Old 05-11-2010
thanks for the updates.. But this doesn seem to work.. Please try the solution with the below data.

Code:
571283,1,"R","01/15/2002","IBMS,SL","IBM/POSSL5M7"","000019826","000019826",,,571283,"D","D","N","N","N","N","N","N","N","N",1,1,1808946.09,1808946.09,,,1808946.09,1808946.09,,,2,2,2,"USD","USD","08/01/1987","08/01/1987",200,200,"1098","1098",,,"12006","12006",,,"BASIC","BASIC","1098","1098",,,,,,,200,"04/12/2002",17:18:18,"finl421",,,"P",,,,,




Moderator's Comments:
Mod Comment Added code tags.
Last edited by radoulov; 05-11-2010 at 09:19 AM..
# 6  
Old 05-11-2010
This differs from the sample file you provided.
so what is your desired output for this example?
# 7  
Old 05-11-2010
With Perl and the CPAN module Text:CSV:


Code:
perl -MText::CSV -nle'BEGIN {
  $csv = Text::CSV->new();
    }  
  $csv->parse($_) and 
    print join "|", $csv->fields();
  ' infile

Code:
% cat infile
571283,1,"R","01/15/2002","IBMS,SL","IBM/POSSL5M7"","000019826","000019826",,,571283,"D","D","N","N","N","N","N","N","N","N",1,1,1808946.09,1808946.09,,,1808946.09,1808946.09,,,2,2,2,"USD","USD","08/01/1987","08/01/1987",200,200,"1098","1098",,,"12006","12006",,,"BASIC","BASIC","1098","1098",,,,,,,200,"04/12/2002",17:18:18,"finl421",,,"P",,,,,

% perl -MText::CSV -nle'BEGIN {
  $csv = Text::CSV->new();
    }
  $csv->parse($_) and
    print join "|", $csv->fields();
  ' infile                         
571283|1|R|01/15/2002|IBMS,SL|IBM/POSSL5M7",00019826|000019826|||571283|D|D|N|N|N|N|N|N|N|N|1|1|1808946.09|1808946.09|||1808946.09|1808946.09|||2|2|2|USD|USD|08/01/1987|08/01/1987|200|200|1098|1098|||12006|12006|||BASIC|BASIC|1098|1098|||||||200|04/12/2002|17:18:18|finl421|||P|||||


Text::CSV is not included in the standard Perl distribution.

---------- Post updated at 03:01 PM ---------- Previous update was at 02:41 PM ----------

Actually I'm not sure if the output matches the OP expectations ...

---------- Post updated at 03:27 PM ---------- Previous update was at 03:01 PM ----------

I suppose this code produces the desired result:

Code:
perl -MText::CSV -nle'BEGIN {
  $csv = Text::CSV->new({
    escape_char          => "\\"
      });
    }  
  $csv->parse($_) and 
    print join "|", $csv->fields();
  ' infile

Code:
% perl -MText::CSV -nle'BEGIN {
  $csv = Text::CSV->new({
    escape_char          => "\\"
      });
    }
  $csv->parse($_) and
    print join "|", $csv->fields();
  ' infile
571283|1|R|01/15/2002|IBMS,SL|IBM/POSSL5M7"|000019826|000019826|||571283|D|D|N|N|N|N|N|N|N|N|1|1|1808946.09|1808946.09|||1808946.09|1808946.09|||2|2|2|USD|USD|08/01/1987|08/01/1987|200|200|1098|1098|||12006|12006|||BASIC|BASIC|1098|1098|||||||200|04/12/2002|17:18:18|finl421|||P|||||
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Linux convert Comma delimited file to pipe

I have file in linux with comma delimited and string fields in double quotations ", I need to convert them to pipe delimiter please share your inputs. Example: Input: "2017-09-30","ACBD,TVF","01234",NULL,18,NULL,"686091802","BANK OF ABCD, LIMITED, THE",790456 Output: ... (4 Replies)
Discussion started by: shieksir
4 Replies

2. Shell Programming and Scripting

Help/Advise please for converting space delimited string variable to comma delimited with quote

Hi, I am wanting to create a script that will construct a SQL statement based on a a space delimited string that it read from a config file. Example of the SQL will be For example, it will read a string like "AAA BBB CCC" and assign to a variable named IN_STRING. I then concatenate... (2 Replies)
Discussion started by: newbie_01
2 Replies

3. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies

4. Shell Programming and Scripting

How to cut a pipe delimited file and paste it with another file to form a comma separated outputfile

Hello ppl I have a requirement to split (cut in unix) a file (A.txt) which is a pipe delimited file into A1.txt and A2.txt Now I have to join (paste in unix) this A2.txt with external file A3.txt to form output file A4.txt which should be CSV (comma separated file) so that third party can... (25 Replies)
Discussion started by: etldev
25 Replies

5. Shell Programming and Scripting

Oracle table extract: all columns are not converting into pipe delimited in flat file

Hi All, I am writing a shell script to extract oracle table into a pipe dilemited flat file. Below is my code and I have attached two files that I have abled to generate so far. 1. Table.txt ==> database extract file 2. flat.txt ==> pipe delimited after some manipulation of the original db... (5 Replies)
Discussion started by: express14
5 Replies

6. UNIX for Dummies Questions & Answers

[solved] Comma separated values to space separated

Hi, I have a large number of files which are written as csv (comma-separated values). Does anyone know of simple sed/awk command do achieve this? Thanks! ---------- Post updated at 10:59 AM ---------- Previous update was at 10:54 AM ---------- Guess I asked this too soon. Found the... (0 Replies)
Discussion started by: lost.identity
0 Replies

7. Shell Programming and Scripting

Help with converting Pipe delimited file to Tab Delimited

I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use cat file | sed 's/|//t/g' The above command substituted "/t" not tab in the place of pipe. Sample file: abc|123|2012-01-30|2012-04-28|xyz have to convert to: abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies

8. Shell Programming and Scripting

Linux - Script to generate the output delimited by Comma/Pipe

Hi All, I have a requirement where I need to go to a directory, list all the files that start with person* (for eg) & read the most recent file from the list of files. While browsing through the forum, i found that the command ls -t will list the files. I am trying to generate the output... (1 Reply)
Discussion started by: dsfreddie
1 Replies

9. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

10. Shell Programming and Scripting

Converting Tab delimited file to Comma delimited file in Unix

Hi, Can anyone let me know on how to convert a Tab delimited file to Comma delimited file in Unix Thanks!! (22 Replies)
Discussion started by: charan81
22 Replies
Login or Register to Ask a Question