awk for splitting file in constant chunks


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk for splitting file in constant chunks
# 1  
Old 06-20-2012
Question awk for splitting file in constant chunks

Hi gurus,
I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error.
Code:
awk '{ for (i = 1; i <= 20; i++)  { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f; close(f)}  } }' main_file

I am new to awk, but I wanted to split using awk. Following is the error ->
Code:
awk: syntax error at source line 1
 context is
    { for (i = 1; i <= 20; i++)  { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends >>>  { <<< 
awk: illegal statement at source line 1
awk: illegal statement at source line 1

Moderator's Comments:
Mod Comment code tags for code, please. (corona688)


Moderator's Comments:
Mod Comment edit by bakunin: corrected the typo in the thread title so it can be found

Last edited by bakunin; 06-21-2012 at 08:35 AM..
# 2  
Old 06-20-2012
Hi

Code:
 awk '!(NR%2500){i++;}{print > "f"i;}' i=1 file

Guru
This User Gave Thanks to guruprasadpr For This Post:
# 3  
Old 06-20-2012
Can't you just use the split command which has especially been designed for such kind of task ?

Code:
man split

# 4  
Old 06-20-2012
Remember that the entire outer code block runs once per line. 'print' doesn't cause it to read another line, even if you didn't have syntax errors your code would just be printing the same line over and over.

Also, $ doesn't mean "variable", $ means "column". If you just want the value of the variable you don't need $.

How the above program works is

Code:
# If NR is a multiple of 2500, increment i.
!(NR%2500){i++;}
# Print into fi.
{print > "f"i;}

This User Gave Thanks to Corona688 For This Post:
# 5  
Old 06-21-2012
Thanks guys. I understood the concept. But this is ending up in error -
awk: syntax error at source line 1

Code:
awk '!(NR%2500){i++;}{print > "f"i;}' i=1 file

Code:
 context is
    !(NR%2500){i++;}{print > >>>  "f"i <<< ;}
awk: illegal statement at source line 1

# 6  
Old 06-21-2012
Try:
Code:
awk '!(NR%2500){close(f); f="f" i++}{print>f}' i=1 file


--
(or use split as ctsgnb suggested)
# 7  
Old 06-21-2012
Split is taking a lot of time, so I wanted to check with awk. Also the output of split is suffixed with aa, ab, ac ... which is not desired, so I need to rename all after splitting.
Dear Scrutinizer
your suggested code is also giving following error -
Code:
awk '!(NR%2500){f="f" i++}{print>f}' i=1 main_file

Code:
awk: null file name in print or getline
 input record number 1, file main_file
 source line number 1

Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting duplicated chunks in a file using awk/sed

Hi all, I'd always appreciate all helps from this site. I would like to delete duplicated chunks of strings on the same row(?). One chunk is comprised of four lines such as: path name starting point ending point voltage number I would like to delete duplicated chunks on the same... (5 Replies)
Discussion started by: jypark22
5 Replies

2. Shell Programming and Scripting

Splitting a file into chunks of 1TB

Hi I have a file with different filesystems with there sizes. I need to split them in chucks of 1TB. The file looks like vf_MTLHQNASF07_Wkgp2 187428400 10601AW1 vf_MTLHQNASF07_Wkgp2 479504596 10604AW1 vf_MTLHQNASF07_Wkgp2 19940 10605AID vf_MTLHQNASF07_Wkgp2 1242622044... (4 Replies)
Discussion started by: bombcan
4 Replies

3. Shell Programming and Scripting

Splitting the file using awk

Hi, I have a requirement in which I am going to receive one file and should be splitted to 9 different files based on one distinguisher called TYPE. I heard that this can be done using awk or sed. Can any one advise regardint the logic and simpler way other than using awk or sed is also... (15 Replies)
Discussion started by: sagar.cumar
15 Replies

4. Shell Programming and Scripting

Splitting file using awk

I have file with below content FG1620000|20000 FG1623000|23000 FG1625000|25000 FG1643894|43894 FG1643895|43895 FG1643896|43896 FG1643897|43897 FG1643898|43898 My aim is to split the above file into two files based on the value in the second field. If the value in second field is... (2 Replies)
Discussion started by: anijan
2 Replies

5. UNIX for Dummies Questions & Answers

Awk: Print out overlapping chunks of file - rows 0-20,10-30,20-40 etc.

First time poster, but the forum has saved my bacon more times than... Lots. Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file 1 2 3 4 5 etc... I want the out put... (3 Replies)
Discussion started by: matfald
3 Replies

6. Shell Programming and Scripting

Splitting a complex file using awk

I have a file that contains the following format delete from table1; delete from table2; insert into table1 (col1, col2) values (value1, value2)@ insert into table1 (col1, col2) values(value3, value4)@ insert into table2(col1, col2,col3) values(value1, value2, value3)@ etc etc This is... (9 Replies)
Discussion started by: hukcjv
9 Replies

7. UNIX for Advanced & Expert Users

Help with splitting lines in a file using awk

I have a file which is one big long line of text about 10Kb long. Can someone provide a way using awk to introduce carriage returns every 40 chars in this file. Any other solutions would also be welcome. Thank you in advance. (5 Replies)
Discussion started by: martinbarretto
5 Replies
Login or Register to Ask a Question