Split a file into several files using a data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split a file into several files using a data
# 1  
Old 05-07-2012
Split a file into several files using a data

Hi All,

I have file(File1) with data like below:

Code:
102100|LName|Gender|Company|Branch|Bday|Salary|Age
102100|bbbb|male|cccc|dddd|19900814|15000|20|
102101|asdg|male|gggg|ksgu|19911216|||
102102|bdbm|male|kkkk|acke|19931018||23|
102102|kfjg|male|kkkc|gkgg|19921213|14000|24|
102102|bprb|male|cccc|pppp||15000|20|

Need a shell script to move the above specific data to files like below

Output1.txt
Code:
102100|LName|Gender|Company|Branch|Bday|Salary|Age
102100|bbbb|male|cccc|dddd|19900814|15000|20|

Output2.txt
Code:
102101|asdg|male|gggg|ksgu|19911216|||

Output3.txt
Code:
102102|bdbm|male|kkkk|acke|19931018||23|
102102|kfjg|male|kkkc|gkgg|19921213|14000|24|
102102|bprb|male|cccc|pppp||15000|20|

The logic is compare first coloumn data of a file and move the lines to separate file when the first coloumn in each line matches.

Thanks in advance.

Last edited by Scrutinizer; 05-07-2012 at 07:01 AM..
# 2  
Old 05-07-2012
Code:
 
nawk -F\| -v OFS=\| '{filename=$1".txt";print > filename}' input.txt

# 3  
Old 05-07-2012
Thanks!!!It worked like a charm.I have one more query,say i have a requirement like the above code should be
move to diffent files only if the count of the file is more than 10.Say in the above example,it consist of 14 lines.
This should be moved logically using the data in the fisrt coloumn.
File 1:

Code:
102100|LName|Gender|Company|Branch|Bday|Salary|Age
102100|bbbb|male|cccc|dddd|19900814|15000|20|
102101|asdg|male|gggg|ksgu|19911216|||
102102|bdbm|male|kkkk|acke|19931018||23|
102102|kfjg|male|kkkc|gkgg|19921213|14000|24|
102102|bprb|male|cccc|pppp||15000|20|

File 2
Code:
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|

Note : In file 1,i cannot put 4 records of 102103 bcos my data(102103) will be in 2(file 1 and file2) files.

---------- Post updated at 04:16 PM ---------- Previous update was at 04:14 PM ----------

Sorry ignore the previous reply.
Thanks!!!It worked like a charm.I have one more query,say i have a requirement like the below code should be
move to diffent files only if the count of the file is more than 10.Say in the above example,it consist of 14 lines.
This should be moved logically using the data in the fisrt coloumn.

Code:
102100|LName|Gender|Company|Branch|Bday|Salary|Age
102100|bbbb|male|cccc|dddd|19900814|15000|20|
102101|asdg|male|gggg|ksgu|19911216|||
102102|bdbm|male|kkkk|acke|19931018||23|
102102|kfjg|male|kkkc|gkgg|19921213|14000|24|
102102|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|

File 1:

Code:
102100|LName|Gender|Company|Branch|Bday|Salary|Age
102100|bbbb|male|cccc|dddd|19900814|15000|20|
102101|asdg|male|gggg|ksgu|19911216|||
102102|bdbm|male|kkkk|acke|19931018||23|
102102|kfjg|male|kkkc|gkgg|19921213|14000|24|
102102|bprb|male|cccc|pppp||15000|20|

File 2
Code:
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|
102103|kfjg|male|kkkc|gkgg|19921213|14000|24|
102103|bprb|male|cccc|pppp||15000|20|

Note : In file 1,i cannot put 4 records of 102103 bcos my data(102103) will be in 2(file 1 and file2) files.

Moderator's Comments:
Mod Comment Welcome to the UNIX and Linux Forums. Please use [code]...[/code] tags. Video tutorial on how to use them

Last edited by Scrutinizer; 05-07-2012 at 07:50 AM.. Reason: code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split JSON to different data files

Hi Gurus, I have below JSON file, now I want to rewrite this file into a new file. I will appreciate if anyone can help me to provide the solution...I can't use jq. { "_id": "3ad893cb4cf1560add7b4caffd4b6126", "_rev": "1-1f0ce165e1d210319cf6e9f9c6ff654f", "name":... (4 Replies)
Discussion started by: manas_ranjan
4 Replies

2. Shell Programming and Scripting

Split a file into 10 different files

OS : RHEL 6.7 Shell : bash I have a text file with 5.97 million lines. I want to split this big file into 12 different files (in sequential order) so that each file will contain roughly 500K lines. I tried the following awk command after googling. But, it just created 2 files... (5 Replies)
Discussion started by: omega3
5 Replies

3. Shell Programming and Scripting

Split File data using awk

HI Guys, I need to split the file in to number of files . file contains FILEHEADER and EOF . I have to split n number of times . I have to form the file with each splitted message between FILEHEADER and EOF using awk beign and end . how to implement please suggest. (2 Replies)
Discussion started by: manish8484
2 Replies

4. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

5. Shell Programming and Scripting

split input data file and put into same output file

Hi All, I have two input file and need to generate a CSV file. The existing report just "GREP" the records with the Header and Tailer records with the count of records. Now i need to split the data into 25 records each in the same CSV file. id_file (Input file ) 227050994 232510151... (4 Replies)
Discussion started by: rasmith
4 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

Split file by data group

Hi all, I'm having a little trouble solving a file split I need to get done. I have the following data: 1. Light 1A. Light Soft texture: it's soft color: the color value is that of something light vital statistics: srm: 23 og: 1.035 sp: 1.065 comment: this is nice if you like... (8 Replies)
Discussion started by: mkastin
8 Replies

8. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

9. Shell Programming and Scripting

Can I split a 10GB file into 1 GB sizes using my repeating data pattern

I'm not a unix guy so excuses my ignorance... I'm the database ETL guy. I'm trying to be proactive and devise a plan B for a ETL process where I expect a file 10X larger than what I process daily for a recast job. The ETL may handle it but I just don't know. This file may need to be split... (3 Replies)
Discussion started by: john091
3 Replies

10. HP-UX

Need to split a large data file using a Unix script

Greetings all: I am still new to Unix environment and I need help with the following requirement. I have a large sequential file sorted on a field (say store#) that is being split into several smaller files, one for each store. That means if there are 500 stores, there will be 500 files. This... (1 Reply)
Discussion started by: SAIK
1 Replies
Login or Register to Ask a Question