Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

awk command to split pipe delimited file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers awk command to split pipe delimited file
# 1  
Old 10-06-2018
awk command to split pipe delimited file
Hello,

I need to split a pipe de-limited file based on the COLUMN 7 value . If the column value changes I need to split the file

Source File

Code:
Payment|ID|DATE|TIME|CONTROLNUMBER|NUMBER|NAME|INDICATOR
42156974|1137937|10/1/2018|104440|4232|2054391|CARE|1
42156978|1137937|10/1/2018|104440|4232|2054391|CARE|0
42156982|1137937|10/1/2018|104440|4230|2054391|UNIVERSAL|8
42157000|1137937|10/1/2018|104440|4230|2054391|UNIVERSAL|6
42157012|1137937|10/1/2018|104440|4235|2054391|ALLIED|10

Split File 1 : Output File Name should be : 1_CARE

Code:
42156974|1137937|10/1/2018|104440|4232|2054391|CARE|1
42156978|1137937|10/1/2018|104440|4232|2054391|CARE|0

Split File 2 : Output File Name should be : 2_UNIVERSAL

Code:
42156982|1137937|10/1/2018|104440|4230|2054391|UNIVERSAL|8
42157000|1137937|10/1/2018|104440|4230|2054391|UNIVERSAL|6

Split File 3: Output File Name should be :3_ALLIED

Code:
42157012|1137937|10/1/2018|104440|4235|2054391|ALLIED|10

Please advise
# 2  
Old 10-06-2018
Have you tried anything, or you just expect an awk one-liner ?

Regards
Peasant.
# 3  
Old 10-06-2018
Hi Peasant

I tried few options but I am no way closer. I captured column seven in variable, I need to now check this variable against the next line and then split when it changes.

Code:
 awk -F\|  '{ var1=$7; var2=$1; print var1, var2 }' Input.txt

Please advise
# 4  
Old 10-07-2018
Code:
awk -F"|" 'NR>1 { a[$0]=$7 } END { for ( i in a ) print i > "1_"a[i] } ' input

If you have gigabyte files, a different approach would be needed to minimize memory usage, since array a would become huge on those files.

But that's another problem, which would require a bit larger and more efficient program.
If that is the case, get back here, and we shall think of something.

Hope that helps
Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
rosebud123
# 5  
Old 10-07-2018
A bit simpler, no memory hogger:


Code:
awk -F\| 'NR>1 {if (!X[$7]) X[$7] = ++CNT; print > (X[$7] "_" $7)}' file


If you have a large number of different output files (exceeding system limits) you'll need to append to the files and close them after writing.
These 2 Users Gave Thanks to RudiC For This Post:
Peasant rosebud123
# 6  
Old 10-11-2018
Rudi C.

The three output file names have 1_ as prefix , can you please update the code to go in sequential manner i.e. 1_xx, 2_xxx e.t.c

Thanks
# 7  
Old 10-11-2018
Mine has _1 hard coded.

RudiC code enumerates properly, creating three files from current input.
Code:
1_CARE  2_UNIVERSAL  3_ALLIED

Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Get the output of w command in pipe delimited format

Since output of w command have variable number of columns I want to get the output in pipe delimited format. I tried export OFS="|"; w but that does not work. Any ideas? (4 Replies)
Discussion started by: Soham
4 Replies

2. Shell Programming and Scripting

Split a content in a file with specific interval base on the delimited values using UNIX command

Hi All, we have a requirement to split a content in a text file every 5 rows and write in a new file . conditions: if 5th line falls between center of the statement . it should look upto after ";" files are below format: 1 UPDATE TABLE TEST1 SET VALUE ='AFDASDFAS' 2 WHERE... (3 Replies)
Discussion started by: KK230689
3 Replies

3. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies

4. Shell Programming and Scripting

Insert a value in a pipe delimited line (unsig sed,awk)

Hi, I want to insert a value (x) in the 3rd position of each line in a file like below a|b|c|d|1 a|b|c|d a|b|c|d|e|1 a|b|cso that output file looks like a|b|x|c|d|1 a|b|x|c|d a|b|x|c|d|e|1 a|b|x|cI can do that using perl as below #!/usr/bin/perl -w use strict; #inserting x at... (5 Replies)
Discussion started by: sam05121988
5 Replies

5. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

6. Shell Programming and Scripting

Help with converting Pipe delimited file to Tab Delimited

I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use cat file | sed 's/|//t/g' The above command substituted "/t" not tab in the place of pipe. Sample file: abc|123|2012-01-30|2012-04-28|xyz have to convert to: abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies

7. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

8. UNIX for Dummies Questions & Answers

Delete last value from pipe delimited file

I have a large(ish) pipe delimited file. The last line of the file contains a total row count and a checksum: END|1537451|1328569446 After making other adjustments to the file, I need to strip out the checksum and apply a new value - I have a script to generate the checksum and 'cat' it... (3 Replies)
Discussion started by: relentl3ss
3 Replies

9. Shell Programming and Scripting

convert a pipe delimited file to a':" delimited file

i have a file whose data is like this:: osr_pe_assign|-120|wg000d@att.com|4| osr_evt|-21|wg000d@att.com|4| pe_avail|-21|wg000d@att.com|4| osr_svt|-11|wg000d@att.com|4| pe_mop|-13|wg000d@att.com|4| instar_ready|-35|wg000d@att.com|4| nsdnet_ready|-90|wg000d@att.com|4|... (6 Replies)
Discussion started by: priyanka3006
6 Replies

10. Shell Programming and Scripting

How to split pipe delimited file

I have a pipe delimited input file as below. First byte of the each line indicate the record type. Then i need to split the file based on record_type = null,0,1,2,6 and create 5 files. How do i do this in a ksh script? Pls help |sl||SL|SL|SL|1996/04/03|1988/09/15|C|A|sl||||*|... (4 Replies)
Discussion started by: njgirl
4 Replies
Login or Register to Ask a Question