Help Needed : Split one big file to multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help Needed : Split one big file to multiple files
# 1  
Old 03-02-2008
Help Needed : Split one big file to multiple files

Hi friends,

I have data in flat file as following, first filed is the customer number. We have almost 50-100 customers in the system


100 ABC A123
100 BVC D234
100 BNC N324
200 CBC A122
200 AVC D294
200 HNC N324
300 GBC A173
300 FVC D234
300 DNC N344

I want to split the file and create individual file for each customer with customer number in the file name as follows,

File Name : 100.<TimeStamp>.txt

100 ABC A123
100 BVC D234
100 BNC N324

File Name : 200.<TimeStamp>.txt

200 CBC A122
200 AVC D294
200 HNC N324

File Name : 300.<TimeStamp>.txt

300 GBC A173
300 FVC D234
300 DNC N344

I really apprciate if anyone can help me writing this script.
# 2  
Old 03-02-2008
This is one line with awk...
Code:
$ ls
data
$
$
$
$
$
$ cat data
100 ABC A123
100 BVC D234
100 BNC N324
200 CBC A122
200 AVC D294
200 HNC N324
300 GBC A173
300 FVC D234
300 DNC N344
$ awk -v timestamp=whatever  '{print $0 >> ($1timestamp".txt")}'  data
$ ls
100whatever.txt 200whatever.txt 300whatever.txt data
$ cat 100whatever.txt
100 ABC A123
100 BVC D234
100 BNC N324
$

# 3  
Old 03-02-2008
Code:
$  awk '{print > $1".ts.txt"}' cus.out

# 4  
Old 03-02-2008
wow thanks Perderabo it worked like a charm!!! but it is creating one more extra file also i.e, whatever.txt. I dont want this file. I really appreciate ur input.

39 Mar 2 05:42 100whatever.txt
39 Mar 2 05:42 200whatever.txt
39 Mar 2 05:42 300whatever.txt
2 Mar 2 05:42 whatever.txt

THanks
Monica
# 5  
Old 03-02-2008
Quote:
Originally Posted by monicasgupta
wow thanks Perderabo it worked like a charm!!! but it is creating one more extra file also i.e, whatever.txt. I dont want this file. I really appreciate ur input.

39 Mar 2 05:42 100whatever.txt
39 Mar 2 05:42 200whatever.txt
39 Mar 2 05:42 300whatever.txt
2 Mar 2 05:42 whatever.txt

THanks
Monica
you must have some blank lines. You can ignore blank line by checking if the number of fields is non-zero....

awk -v timestamp=whatever 'NF {print $0 >> ($1timestamp".txt")}' data

And you're supposed to replace "whatever" with your timestamp.
# 6  
Old 03-03-2008
Thanks Perderabo, it worked really fine, i really appreciate if you can be more helpful to me

once the file r created i need to do following two things.

1) Following file are created with data.

100whatever.txt
100 ABC A123
100 BVC D234
100 BNC N324

100whatever.txt
200 CBC A122
200 AVC D294
200 HNC N324

100whatever.txt
300 GBC A173
300 FVC D234
300 DNC N344

Now I want to remove first record from all the file, should look like
100whatever.txt
ABC A123
BVC D234
BNC N324

100whatever.txt
CBC A122
AVC D294
HNC N324

100whatever.txt
GBC A173
FVC D234
DNC N344

2) Zip all the files into one file as WHATEVER.zip

THanks
MOnica
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

Split a big file system to several files

Gents Actually I have question and i need your support. I have this NAS file system mounted as /coresys has size of 7 TB I need to Split this file system into several file systems as mount points I mean how to can I Split it professionally to different NAS mount points how to can I decide... (2 Replies)
Discussion started by: AbuAliiiiiiiiii
2 Replies

2. Shell Programming and Scripting

Split a big file into multiple files using awk

this thread is a continuation from previous thread https://www.unix.com/shell-programming-and-scripting/223901-split-big-file-into-multiple-files-based-first-four-characters.html ..I am using awk to split file and I have a syntax error while executing the below code I am using AIX 7.2... (4 Replies)
Discussion started by: etldev
4 Replies

3. Shell Programming and Scripting

Split a big file into multiple files based on first four characters

I have a requirement to split a huge file to smaller text files based on first four characters which look like ABCD 1234 DFGH RREX : : : : : 0000 Each of these records are OF EQUAL bytes with a different internal layout based on the above first digit identifier.. Any help to start... (5 Replies)
Discussion started by: etldev
5 Replies

4. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

5. Emergency UNIX and Linux Support

Combine multiple Files into one big file

Hi Ppl, I have a requirement like i will be getting files of huge size daily and if the file size is so huge ,the files will be split into many parts and sent.The first file will have the header details followed by detail records and the consecutive files will have detail records and the last... (11 Replies)
Discussion started by: ganesh_248
11 Replies

6. Shell Programming and Scripting

Need to Split Big XML into multiple xmls

Hi friends.. We have urgent requirement.We need to split the big xml having multiple orders into multiple xmls having each order in each xml. For Example In input XMl will be in following format with multiple line orders.. <OrderDetail BillToKey="20100805337" Createuserid="CreateGuestOrder"... (8 Replies)
Discussion started by: dprakash
8 Replies

7. UNIX for Advanced & Expert Users

Split a big file into two others files

Hello, i have a very big file that has more then 80 MBytes (100MBytes). So with my CVS Application I cannot commit this file (too Big) because it must have < 80 MBytes. How can I split this file into two others files, i think the AIX Unix command : split -b can do that, buit how is the right... (2 Replies)
Discussion started by: steiner
2 Replies

8. Shell Programming and Scripting

Split file into multiple files

Hi I have a file that has multiple sequences; the sequence name is the line starting with '>'. It looks like below: infile.txt: >HE_ER tttggtgccttgactcggattgggggacctcccttgggagatcaatcccctgtcctcctgctctttgctc cgtgaaaaggatccacctatgacctctagtcctcagacccaccagcccaaggaacatctcaccaatttca >M7B_Ho_sap... (2 Replies)
Discussion started by: jdhahbi
2 Replies

9. UNIX for Dummies Questions & Answers

split a file into multiple files

Hi All, I have a file ABC.txt and I need to split this file on every 250 rows. And the file name should be ABC1.txt , ABC2.txt and so on. I tried with split command split -l 250 <filename> '<filename>' but the file name returned was ABC.txtaa ABC.txtab. Please... (8 Replies)
Discussion started by: kumar66
8 Replies

10. Shell Programming and Scripting

Split a file into multiple files

I have a file ehich has multiple create statements as create abc 123 one two create xyz 456 four five create nnn 666 six four I want to separte each create statement in seperate files (3 Replies)
Discussion started by: glamo_2312
3 Replies
Login or Register to Ask a Question