Hello All,
Thanks for taking time to read through the thread and for providing any possible solution.
I am trying to pivot a comma separated field in a pipe delimited file. Data looks something like this:
Desired output:
Please note that the comma separated field can have any number of separator. That is, there is no fixed number that Field2 can have a maximum of 10 delimiters (commas).
Thing that I could think of is:
But this gives output like this
I need Field1 as well as that is my key field and without that the pivoting will not make any sense.
Any help would be greatly appreciated!
Thanks very much.
Moderator's Comments:
Please use code tags next time for your code and data. Thanks
Thanks very much junior-helper and durden_tyler.
Both the solutions worked. However, with junior_helper's solution, I am getting a warning message
Also, I would really appreciate if you guys can explain the code as well.
Thanks again!
awk -F'[\|,]'
Telling awk to use pipe and comma as field separator/delimiter for the input file.
I used backslash "\" to escape the pipe, but obviously it's not mandatory here, you can remove the backslash to avoid the warning.
'BEGIN {OFS="|"}
Defining "Output Field Separator" as pipe. This portion is executed only once.
Hint: Delete this part to see the difference.
This is one way of defining the OFS. Alternatively one can "hard-code" it in the print command, eg. print $1"|"$i
NR==1 {print;next}
NR is an internal awk variable, meaning "Number of Row" or line number, respectively.
The above line means if the line number is 1, print the line unmodified; read next line.
This portion is executend only once too.
{for (i=2;i<=NF;i++) print $1, $i}'
NF is an internal awk variable, meaning (total) "Number of Fields" in the particular line.
awk is looping here from field 2 to last field and printing $1, $i
($1 is the first field, $i is the second; in the next loop awk will print $1 and the third field and so on)
The awk code performs 3 steps for every line that it reads from "test.dat":
Step 1:
Split the line on the "|" character, since -F"|" has been specified. After splitting, the variables $1 and $2 are set to the two values. Each line will have only two values since there is exactly one "|" per line.
Step 2:
Use the "split" function on the value of $2 from Step 1. Use the comma "," as separator here. After splitting, set the values to the array "a". Set the value of "n" to the size of the array "a".
Step 3:
Run the "for" loop from value of i = 1 to "n" that was determined in Step 2. For each iteration, print the value of $1 from Step 1, the pipe character "|" and the value of a[i]. "a" was determined in Step 2 and i is the iterator value.
Once you understand these 3 steps, you can apply that knowledge to a couple of lines read.
---------------------------------------------------
Line 1 => Read "Field1|Field2"
---------------------------------------------------
Step 1:
After splitting on "|" character, value of $1 = Field1 and that of $2 = Field2
Step 2:
After splitting $2 = Field2 on the "," character, the array "a" has only one element. a[1] = Field2 and n = 1.
Step 3:
Loop from i=1 to n i.e. 1. Print $1 then "|" then a[1] i.e. print "Field1|Field2"
---------------------------------------------------
Line 2 => Read "123|345,567,789"
---------------------------------------------------
Step 1:
After splitting on "|" character, value of $1 = 123 and that of $2 = 345,567,789
Step 2:
After splitting $2 = 345,567,789 on the "," character, the array "a" has 3 elements.
a[1] = 345
a[2] = 567
a[3] = 789
n = 3
Step 3:
Loop from i=1 to n i.e. 3.
i = 1 => Print $1 then "|" then a[1] i.e. print "123|345"
i = 2 => Print $1 then "|" then a[2] i.e. print "123|567"
i = 3 => Print $1 then "|" then a[3] i.e. print "123|789"
And so on...
Since -F forces awk to split the line anyway, the "split" function could be avoided like so:
A test run follows:
The OFS variable could also be used, as others have shown.
Hi, I have a rquirement in unix as below .
I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column.
ex:
Input Text file:
1|A|apple
2|B|bottle
excel file to be generated as output as... (9 Replies)
I have a small quandry. I had server reports that I pulled from a database that came out pipe "|" delimited. The developers have now changed the format to CSV. The issue is that some fields have quotes around the text and other fields are blank with strings of commas denoting each field. To further... (2 Replies)
Hi guys,
I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file
for eg:
file 1:
xx|yy|"xyz|zzz"|zzz|12...
using below awk command
awk 'BEGIN {FS=OFS="|" } print $3
i would get xyz
But i want as :
xyz|zzz to consider as whole column... (13 Replies)
I have a csv data file :
A,B,C,D,"A,B",E,"GG,H"
E,F,G,H,I,J,"S,P"
I need to replace all "," with "|" except the ones between double quotes i.e
A|B|C|D|"A,B"|E|"GG,H"
E|F|G|H|I|J|"S,P"
CAn someone assist? (8 Replies)
I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use
cat file | sed 's/|//t/g'
The above command substituted "/t" not tab in the place of pipe.
Sample file:
abc|123|2012-01-30|2012-04-28|xyz
have to convert to:
abc 123... (6 Replies)
i need to add a new field in a pipe delimited line. the field will be the current date today.
aa|a|s|w|1
as|oiy|oiy|oiy|2
given that all lines are uniformed in the number of fields
i want it to look like this:\
aa|a|s|w|1|20120126
as|oiy|oiy|oiy|2|20120126
please help :) (3 Replies)
Hi All,
I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted...
AA ATIU2345098809 009697 005374
BB ATIU2345097809 005445 006518
CC ATIU9685098809 003215 003571
DD... (7 Replies)
Hi All,
I need a shell script which could insert a sequence number column inside a dat file(pipe delimited).
I have the dat file similar to the one as shown below..
|A|B|C||D|E
|F|G|H||I|J
|K|L|M||N|O
|P|Q|R||S|T
As shown above, the column 4 is currently blank and i need to insert sequence... (5 Replies)
Hi, could some help me convert CSV file (with double quoted strings) to pipe delimited file:
here you go with the same data:
1,Friends,"$3.99 per 1,000 listings",8158here " 1,000 listings " should be a single field.
Thanks,
Ram (8 Replies)
Hi,
I want to replace a field in a text delimited file with the actual number of records in the same file.
HDR|ABCD|10-13-2008 to 10-19-2008.txt|10-19-2008|XYZ
DTL|0|5464-1|0|02-02-2008|02-03-2008||||F|||||||||
DTL|1|5464-1|1|02-02-2008|02-03-2008|1||JJJ... (3 Replies)