Sponsored Content
Top Forums Shell Programming and Scripting Pivoting csv field from pipe delimited file Post 302926000 by durden_tyler on Thursday 20th of November 2014 04:13:58 PM
Old 11-20-2014
Code:
awk -F"|" '{n=split($2,a,","); for(i=1;i<=n;++i){print $1"|"a[i]}}' test.dat

The awk code performs 3 steps for every line that it reads from "test.dat":

Step 1:
Split the line on the "|" character, since -F"|" has been specified. After splitting, the variables $1 and $2 are set to the two values. Each line will have only two values since there is exactly one "|" per line.

Step 2:
Use the "split" function on the value of $2 from Step 1. Use the comma "," as separator here. After splitting, set the values to the array "a". Set the value of "n" to the size of the array "a".

Step 3:
Run the "for" loop from value of i = 1 to "n" that was determined in Step 2. For each iteration, print the value of $1 from Step 1, the pipe character "|" and the value of a[i]. "a" was determined in Step 2 and i is the iterator value.

Once you understand these 3 steps, you can apply that knowledge to a couple of lines read.

---------------------------------------------------
Line 1 => Read "Field1|Field2"
---------------------------------------------------
Step 1:
After splitting on "|" character, value of $1 = Field1 and that of $2 = Field2

Step 2:
After splitting $2 = Field2 on the "," character, the array "a" has only one element. a[1] = Field2 and n = 1.

Step 3:
Loop from i=1 to n i.e. 1. Print $1 then "|" then a[1] i.e. print "Field1|Field2"

---------------------------------------------------
Line 2 => Read "123|345,567,789"
---------------------------------------------------

Step 1:
After splitting on "|" character, value of $1 = 123 and that of $2 = 345,567,789


Step 2:
After splitting $2 = 345,567,789 on the "," character, the array "a" has 3 elements.
a[1] = 345
a[2] = 567
a[3] = 789
n = 3


Step 3:
Loop from i=1 to n i.e. 3.
i = 1 => Print $1 then "|" then a[1] i.e. print "123|345"
i = 2 => Print $1 then "|" then a[2] i.e. print "123|567"
i = 3 => Print $1 then "|" then a[3] i.e. print "123|789"

And so on...

Since -F forces awk to split the line anyway, the "split" function could be avoided like so:

Code:
awk -F'[|,]' '{for(i=2;i<=NF;++i){print $1"|"$i}}' test.dat

A test run follows:

Code:
$
$ cat test.dat
Field1|Field2
123|345,567,789
234|563,560
345|975,098,985,397,984
456|736
$
$ awk -F'[|,]' '{for(i=2;i<=NF;++i){print $1"|"$i}}' test.dat
Field1|Field2
123|345
123|567
123|789
234|563
234|560
345|975
345|098
345|985
345|397
345|984
456|736
$
$

The OFS variable could also be used, as others have shown.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Replacing a field in pipe delimited TEXT File

Hi, I want to replace a field in a text delimited file with the actual number of records in the same file. HDR|ABCD|10-13-2008 to 10-19-2008.txt|10-19-2008|XYZ DTL|0|5464-1|0|02-02-2008|02-03-2008||||F||||||||| DTL|1|5464-1|1|02-02-2008|02-03-2008|1||JJJ... (3 Replies)
Discussion started by: ravi0435
3 Replies

2. Shell Programming and Scripting

Convert CSV file (with double quoted strings) to pipe delimited file

Hi, could some help me convert CSV file (with double quoted strings) to pipe delimited file: here you go with the same data: 1,Friends,"$3.99 per 1,000 listings",8158here " 1,000 listings " should be a single field. Thanks, Ram (8 Replies)
Discussion started by: Ram.Math
8 Replies

3. Shell Programming and Scripting

How to insert a sequence number column inside a pipe delimited csv file using shell scripting?

Hi All, I need a shell script which could insert a sequence number column inside a dat file(pipe delimited). I have the dat file similar to the one as shown below.. |A|B|C||D|E |F|G|H||I|J |K|L|M||N|O |P|Q|R||S|T As shown above, the column 4 is currently blank and i need to insert sequence... (5 Replies)
Discussion started by: nithins007
5 Replies

4. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

5. UNIX for Dummies Questions & Answers

add new 'date field' in a pipe delimited file

i need to add a new field in a pipe delimited line. the field will be the current date today. aa|a|s|w|1 as|oiy|oiy|oiy|2 given that all lines are uniformed in the number of fields i want it to look like this:\ aa|a|s|w|1|20120126 as|oiy|oiy|oiy|2|20120126 please help :) (3 Replies)
Discussion started by: kokoro
3 Replies

6. Shell Programming and Scripting

Help with converting Pipe delimited file to Tab Delimited

I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use cat file | sed 's/|//t/g' The above command substituted "/t" not tab in the place of pipe. Sample file: abc|123|2012-01-30|2012-04-28|xyz have to convert to: abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies

7. Shell Programming and Scripting

Convert csv to pipe delimited except the ones in double quotes

I have a csv data file : A,B,C,D,"A,B",E,"GG,H" E,F,G,H,I,J,"S,P" I need to replace all "," with "|" except the ones between double quotes i.e A|B|C|D|"A,B"|E|"GG,H" E|F|G|H|I|J|"S,P" CAn someone assist? (8 Replies)
Discussion started by: Shivdatta
8 Replies

8. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

9. Shell Programming and Scripting

Pipe delimited to csv

I have a small quandry. I had server reports that I pulled from a database that came out pipe "|" delimited. The developers have now changed the format to CSV. The issue is that some fields have quotes around the text and other fields are blank with strings of commas denoting each field. To further... (2 Replies)
Discussion started by: dagamier
2 Replies

10. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies
Regexp::Common::delimited(3)				User Contributed Perl Documentation			      Regexp::Common::delimited(3)

NAME
Regexp::Common::delimited -- provides a regex for delimited strings SYNOPSIS
use Regexp::Common qw /delimited/; while (<>) { /$RE{delimited}{-delim=>'"'}/ and print 'a " delimited string'; /$RE{delimited}{-delim=>'/'}/ and print 'a / delimited string'; } DESCRIPTION
Please consult the manual of Regexp::Common for a general description of the works of this interface. Do not use this module directly, but load it via Regexp::Common. $RE{delimited}{-delim}{-esc} Returns a pattern that matches a single-character-delimited substring, with optional internal escaping of the delimiter. When "-delim=S" is specified, each character in the sequence S is a possible delimiter. There is no default delimiter, so this flag must always be specified. If "-esc=S" is specified, each character in the sequence S is the delimiter for the corresponding character in the "-delim=S" list. The default escape is backslash. For example: $RE{delimited}{-delim=>'"'} # match "a " delimited string" $RE{delimited}{-delim=>'"'}{-esc=>'"'} # match "a "" delimited string" $RE{delimited}{-delim=>'/'} # match /a / delimited string/ $RE{delimited}{-delim=>q{'"}} # match "string" or 'string' Under "-keep" (See Regexp::Common): $1 captures the entire match $2 captures the opening delimiter (provided only one delimiter was specified) $3 captures delimited portion of the string (provided only one delimiter was specified) $4 captures the closing delimiter (provided only one delimiter was specified) $RE{quoted}{-esc} A synonym for $RE{delimited}{q{-delim='"`}{...}} SEE ALSO
Regexp::Common for a general description of how to use this interface. AUTHOR
Damian Conway (damian@conway.org) MAINTAINANCE
This package is maintained by Abigail (regexp-common@abigail.be). BUGS AND IRRITATIONS
Bound to be plenty. For a start, there are many common regexes missing. Send them in to regexp-common@abigail.be. LICENSE and COPYRIGHT This software is Copyright (c) 2001 - 2009, Damian Conway and Abigail. This module is free software, and maybe used under any of the following licenses: 1) The Perl Artistic License. See the file COPYRIGHT.AL. 2) The Perl Artistic License 2.0. See the file COPYRIGHT.AL2. 3) The BSD Licence. See the file COPYRIGHT.BSD. 4) The MIT Licence. See the file COPYRIGHT.MIT. perl v5.18.2 2013-03-08 Regexp::Common::delimited(3)
All times are GMT -4. The time now is 06:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy