Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Split into multiple files by using Unique columns in a UNIX file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Split into multiple files by using Unique columns in a UNIX file
# 1  
Old 07-15-2019
Split into multiple files by using Unique columns in a UNIX file
I have requirement to split below file (sample.csv) into multiple files by using the unique columns (first 3 are unique columns)

sample.csv

Code:
123|22|56789|ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|1|1
123|12|5679|BCDEFG|34CD56|2019-07-10|2019-07-10|896.7200|1|2
123|12|5679|CDEFGH|45DE67|2019-07-10|2019-07-10|680.9200|1|1
123|22|56789|DEFGHI|56EF78|2019-07-10|2019-07-10|405.8800|1|2

required output files should be as below

Code:
sample_123_22_56789.csv

123|22|56789|ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|1|1
123|22|56789|DEFGHI|56EF78|2019-07-10|2019-07-10|405.8800|1|2

Code:
sample_123_12_5679.csv

123|12|5679|BCDEFG|34CD56|2019-07-10|2019-07-10|896.7200|1|2
123|12|5679|CDEFGH|45DE67|2019-07-10|2019-07-10|680.9200|1|1

I am using below command and it is working but want to make it generic instead of hard coding the $4"|"$5"|"$6"|"$7"|"$8"|"$9"|"$10

Code:
awk -F\| '{print $4"|"$5"|"$6"|"$7"|"$8"|"$9"|"$10 > "sample_"$1"_"$2"_"$3".CSV"}' sample.csv

Thanks.
# 2  
Old 07-15-2019
Did you search this forum? There might be an answer already.
Here is a simple one:
Code:
awk -F"|" ' { print > ("sample_" $1 "_" $2 "_" $3 ".csv") }' sample.csv

# 3  
Old 07-15-2019
Thank you for the reply, Sorry for the confusion, my desired output would be as below (Excludeing first 3 columns)

Code:
sample_123_22_56789.csv
ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|1|1
DEFGHI|56EF78|2019-07-10|2019-07-10|405.8800|1|2

Code:
sample_123_12_5679.csv
BCDEFG|34CD56|2019-07-10|2019-07-10|896.7200|1|2
CDEFGH|45DE67|2019-07-10|2019-07-10|680.9200|1|1

Last edited by jim mcnamara; 07-15-2019 at 01:18 PM..
# 4  
Old 07-15-2019
Ah, now I understand
It's a challenge to split and print in awk.
Here is a RE
Code:
awk -F"|" '{ x=$0; sub(/[^|]*[|][^|]*[|][^|]*[|]/, "", x); print x > ("sample_" $1 "_" $2 "_" $3 ".csv") }' sample.csv

Not elegant.
The following in a loop composes the strings for the output and the file name:
Code:
awk -F"|" ' { fn="sample"; out=sep=""; for(i=1; i<=NF; i++) if (i<=3) { fn=(fn "_" $i) } else { out=(out sep $i); sep=FS } print out > (fn ".csv") }' sample.csv

Last edited by MadeInGermany; 07-15-2019 at 09:38 AM.. Reason: out="" missing
This User Gave Thanks to MadeInGermany For This Post:
jim mcnamara
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading multiple values from multiple lines and columns and setting them to unique variables.

Hello, I would like to ask for help with csh script. An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that... (7 Replies)
Discussion started by: FMMOLA
7 Replies

2. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

3. Shell Programming and Scripting

awk script to split file into multiple files based on many columns

So I have a space delimited file that I'd like to split into multiple files based on multiple column values. This is what my data looks like 1bc9A02 1 10 1000 FTDLNLVQALRQFLWSFRLPGEAQKIDRMMEAFAQRYCQCNNGVFQSTDTCYVLSFAIIMLNTSLHNPNVKDKPTVERFIAMNRGINDGGDLPEELLRNLYESIKNEPFKIPELEHHHHHH 1ku1A02 1 10... (9 Replies)
Discussion started by: viored
9 Replies

4. Shell Programming and Scripting

Split file in unix into multiple files

Hi Gurus I have to split the incoming source file into multiple file. File contains some unwanted XML tags also . Files looks like some XML tags FILEHEADERABC 12 -- --- ---- EOF some xml tags xxxFILEHEADERABC 13 -- --- ---- EOF I have to ignore XML tags and only split file... (6 Replies)
Discussion started by: manish2608
6 Replies

5. Shell Programming and Scripting

split file into multiple files

Hi, I have a file of the following syntax that has around 120K records that are tab separated. input.txt abc def klm 20 76 . + . klm_mango unix_00000001; abc def klm 83 84 . + . klm_mango unix_0000103; abc def klm 415 439 . + . klm_mango unix_00001043; I am looking for an awk oneliner... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

6. Shell Programming and Scripting

Split file into multiple files

Hi I have a file that has multiple sequences; the sequence name is the line starting with '>'. It looks like below: infile.txt: >HE_ER tttggtgccttgactcggattgggggacctcccttgggagatcaatcccctgtcctcctgctctttgctc cgtgaaaaggatccacctatgacctctagtcctcagacccaccagcccaaggaacatctcaccaatttca >M7B_Ho_sap... (2 Replies)
Discussion started by: jdhahbi
2 Replies

7. Shell Programming and Scripting

Split a file into multiple files

Hi, i have a file like this: 1|2|3|4|5| 1|2|8|4|6| Trailer1||||| 1|2|3| Trailer2||| 3|4|5|6| 3|4|5|7| 3|4|5|8| Trailer2||| I want to generate 3 files out of this based on the trailer record. Trailer record string can be different for each file or it may be same for one or two. No... (24 Replies)
Discussion started by: pparthji
24 Replies

8. UNIX for Dummies Questions & Answers

Join multiple Split files in Unix

Hi, I have a big file of 50GB size. I need copy it to a second ftp from a ftp. I am not able to do the full 50GB transfer as it timesout after some time. SO i am trying to split the file into 5gb each 10 files with the below command. split -b 5368709120 pack.tar.gz backup.gz After I... (2 Replies)
Discussion started by: venu_nbk
2 Replies

9. UNIX for Dummies Questions & Answers

split a file into multiple files

Hi All, I have a file ABC.txt and I need to split this file on every 250 rows. And the file name should be ABC1.txt , ABC2.txt and so on. I tried with split command split -l 250 <filename> '<filename>' but the file name returned was ABC.txtaa ABC.txtab. Please... (8 Replies)
Discussion started by: kumar66
8 Replies

10. Shell Programming and Scripting

Split a file into multiple files

I have a file ehich has multiple create statements as create abc 123 one two create xyz 456 four five create nnn 666 six four I want to separte each create statement in seperate files (3 Replies)
Discussion started by: glamo_2312
3 Replies
Login or Register to Ask a Question