Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

AWK Shell Program to Split Large Files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AWK Shell Program to Split Large Files
# 8  
Old 06-29-2009
Code:
awk '{if(out) close(out);out=$(NF-2) ".dat"; print >> out}' mka.txt

# 9  
Old 06-29-2009
Is it possible to substring the field instead of using the 'NF' option because in the files it is possible to there to be anywhere from 6 - 50 populated fields between the field to split on and the 'X' at the EOL. Sorry that I'm being such a pain, but I really appreciate the help. Also I'm looking for the output names to be like this <infile name>_<string>.dat

Thanks again and again!
# 10  
Old 06-29-2009
could you provide a couple of varying samples so that we could see a 'pattern' and identify the fields/strings, please.
# 11  
Old 06-29-2009
Code:
0001  Ronald   McDonald  01 H81 0001256 0100111               X
0002  Elmo     St. Elmo  02 H82 0089621  001  10 11 01 1      X
0003  Cookie   Monster   01 H81 0887141    1  .  0   0  .  1  X
0004  Oscar    Grouche   03 H83 0364471                   110.X
0005  Dumb     Name      02 H82 0000233   111 00 1111 00000000X
0006  Butter   Face      04 H84 0014666 1111111111111111111111X
0007  Ford     F150      01 H81 0000001 00111 110 110  0011 ..X
0008  Last     One       03 H83 7741668 0 1 0 1 0 1 0 1 0 1 0 X

Have I mentioned yet that you're awesome, Smilie
# 12  
Old 06-29-2009
Is it safe to assume that the 'strings' are always in the form 'H<digit><digit><digit>..' ?
Are the fields TAB separated by any chance?
What 'safe' assumption can be made to get to the string?
# 13  
Old 06-29-2009
Okay, this is a fixed length file, so no tabs, the string is a 25 character varchar so it does not conform to my H### example. The 'safe' assumption is the the string to split is always are the same byte position in my example byte 29 for length of 3. There is no consistent number of fields before or after the split string as they can be populated or not depending on the data available for the individual represented.
# 14  
Old 06-29-2009
Code:
nawk '{if(out) close(out);out=FILENAME "_" substr($0,29,3) ".dat"; print >> out}' mka.txt
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ... (20 Replies)
Discussion started by: Raghuram717
20 Replies

2. Shell Programming and Scripting

Split large xml into mutiple files and with header and footer in file

Split large xml into mutiple files and with header and footer in file tried below it splits unevenly and also i need help in adding header and footer command : csplit -s -k -f my_XML_split.xml extrfile.xml "/<Document>/" {1} sample xml <?xml version="1.0" encoding="UTF-8"?><Recipient>... (36 Replies)
Discussion started by: karthik
36 Replies

3. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this... (12 Replies)
Discussion started by: Kamesh G
12 Replies

4. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

5. Shell Programming and Scripting

Process multiple large files with awk

Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk. I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside. For example: a sample_1 200 a.b sample_2 10 a sample_3 10 a sample_1 10 a... (4 Replies)
Discussion started by: camor
4 Replies

6. Shell Programming and Scripting

Split Large Files Based On Row Pattern..

Hi all. I've tried searching the web but could not find similar problem to mine. I have one large file to be splitted into several files based on the matching pattern found in each row. For example, let's say the file content: ... (13 Replies)
Discussion started by: aimy
13 Replies

7. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

8. Shell Programming and Scripting

Split large zone file dump into multiple files

I have a large zone file dump that consists of ; DNS record for the adomain.com domain data1 data2 data3 data4 data5 CRLF CRLF CRLF ; DNS record for the anotherdomain.com domain data1 data2 data3 data4 data5 data6 CRLF (7 Replies)
Discussion started by: Bluemerlin
7 Replies

9. Shell Programming and Scripting

Split line to multiple files Awk/Sed/Shell Script help

Hi, I need help to split lines from a file into multiple files. my input look like this: 13 23 45 45 6 7 33 44 55 66 7 13 34 5 6 7 87 45 7 8 8 9 13 44 55 66 77 8 44 66 88 99 6 I want to split every 3 lines from this file to be written to individual files. (3 Replies)
Discussion started by: saint2006
3 Replies

10. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (7 Replies)
Discussion started by: ashish4422
7 Replies
Login or Register to Ask a Question