Visit Our UNIX and Linux User Community


Split file using awk


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Split file using awk
# 1  
Old 08-25-2018
Split file using awk

I need to split the incoming source file in to multiple files using awk.

Split position is (6,13) : 8 positions
  1. All the records that are greater than 20170101 and less than or equal to 20181231 should go in a split file with file name as source filename_greaterthan_20170101_lessthan_20181231 + yyyymmddhhmmss
  • All records that are less than 20170101 should go in a file with file name as source filename_lessthan_20170101 + yyyymmddhhmmss
  • All records that are greater than 20190101 should go in a file with file name as source filename_20190101 + yyyymmddhhmmss

Additionally instead of hard coding the condition in the script/command, can we pass it as a variable to the script , so the script remains dynamic.

Source File:

Code:
001  20991231
002  20190101
003  20231231
004  20231231
005  20261231
006  20271231
007  20281231
008  20301231
009  20161231
010  20161230
011  20161010
012  19880101
013  20000101
014  20110121
015  20130121
016  20170121
017  19870121

# 2  
Old 08-26-2018
Try and adjust something like:
Code:
awk '{y=substr($2,1,4); f=b} y<lt{f=a} y>gt{f=c} {print>f} ' lt=2017 gt=2018 a=y1 b=y2 c=y3 infile

This should split the input file into the files y1, y2 and y3.
This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 08-26-2018
Thanks..

Can you please explain the code in few lines
# 4  
Old 08-26-2018
Sure:
Code:
awk '                                            
  {                                            
    y=substr($2,1,4)                              # Set the variable y to first 4 characters of 
                                                  # the second field of the input file                        
    f=b                                           # set the output to the name in variable b
  }                                            
  y<lt {                                          # if the year is less than the min treshold  
    f=a                                           # set the variable f to the name in variable a
  }                                             
  y>gt {                                          # if the year is more than the max treshold 
    f=c                                           # set the variable f to the name in variable c
  }                                            
  {                                            
    print>f                                       # print the line to the appropriate file          
  }                                            
' lt=2017 gt=2018 a=y1 b=y2 c=y3 infile           # set variables lt, gt, a, b, and c and specify file name.

# 5  
Old 08-27-2018
Thanks.

Is there a way to not hard code 2017 and 2018 , rather pass them as a parameters ?
# 6  
Old 08-27-2018
Quote:
Originally Posted by rosebud123
Thanks.

Is there a way to not hard code 2017 and 2018 , rather pass them as a parameters ?
they are being passed as parameters.
# 7  
Old 08-27-2018
I think s/he means shell variables

Code:
awk '...' lt="${PAR1}" gt="${PAR2}"


Previous Thread | Next Thread
Test Your Knowledge in Computers #762
Difficulty: Medium
The L2 cache, and higher-level caches, are never shared between the CPU cores.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk file split

Hi all, First of all I' like to mention that I'm pretty new to unix scripting. :( I'm trying to split an large xml with awk and rename it based on the values of two attributes. Example XML <RECORD> <element1>11</element1> <element2>22</element2> <element3>33</element3>... (18 Replies)
Discussion started by: f0usk4s
18 Replies

2. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

3. Shell Programming and Scripting

Split a file with awk

Hi! I have a file like this: a,b,c,12,d,e a,b,c,13,d,e a,b,c,14,d,e a,b,c,15,d,e a,b,c,16,d,e a,b,c,17,d,e I need to split that file in two: If field 4 is equal or higher than 14 that row goes to one file and if it is equal or higher than 15 to another. Can anyone point me in the... (2 Replies)
Discussion started by: Tr0cken
2 Replies

4. Shell Programming and Scripting

AWK File Split

Hi All, Input.txt XYZONEABC                  CZXTWOJJJ KKKSIXOOO asdfhajlsdhfajs asdfasfasdf Output Files: ONE.txt XYZONEABC                 TWO.txt CZXTWOJJJ SIX.txt KKKSIXOOO I had a script (2 Replies)
Discussion started by: kmsekhar
2 Replies

5. Shell Programming and Scripting

How to split a file using AWK?

Hello, I have a file like the following: david,a,b,c,20,r thomas,a,b,c,30,r willaiam,a,b,c,80,r barbara,a,b,c,100,r I would like to split the file into other files using a condition for the contents of column 5. The condition should be a if the contents of column 5 is in a range... (4 Replies)
Discussion started by: keenboy100
4 Replies

6. Shell Programming and Scripting

split a file using awk

Hi , I just need to split a file and outputfiles are redirected to gzip file need: Input file - A.gz content of A.gz is 100|sfdds|dffdds|200112|sdfdf 100|sfdds|dffdds|200112|sdfdf 100|sfdds|dffdds|200112|sdfdf 100|sfdds|dffdds|200212|sdfdf 100|sfdds|dffdds|200212|sdfdf... (3 Replies)
Discussion started by: mohan_xunil
3 Replies

7. Shell Programming and Scripting

awk - split file

How can I split a text file (in awk) in n others with number of record given in input? Thanks (6 Replies)
Discussion started by: pinguc
6 Replies

8. Shell Programming and Scripting

split file with awk

I did a lot of search on this forum on spiting file; found a lot, but my requirement is a bit different, please guide. Master file: x:start:5 line1:23 line2:12 2:90 x:end:5 x:start:2 45:56 22:90 x:end:2 x:start:3 line1:23 line2:12 x:end:3 x:start:2 line5:23 (1 Reply)
Discussion started by: uwork72
1 Replies

9. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

10. Shell Programming and Scripting

Split file using awk

I am trying to read a file and split the file into multiple files. I need to create new files with different set of lines from the original file. ie, the first output file may contain 10 lines and the second 100 lines and so on. The criteria is to get the lines between two lines starting with some... (8 Replies)
Discussion started by: pvar
8 Replies

Featured Tech Videos