Split a txt file on the basis of line number


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Split a txt file on the basis of line number
# 1  
Old 04-08-2019
Split a txt file on the basis of line number

I have to split a file containing 100 lines to 5 files say from lines ,1-20 ,21-30 ,31-40 ,51-60 ,61-100


Here is i can do it for 2 file but how to handle it for more than 2 files

Code:
awk 'NR < 21{ print >> "a"; next } {print >> "b" }' $input_file

Please advidse.

Thanks
# 2  
Old 04-08-2019
how about - 20 lines/records per file:
Code:
 awk -v div=20 '{print > ("file_" int(FNR/div))}' myFile

or split into 5 files:
Code:
awk -v div=5 '{print > ("file_" FNR%div)}' myFile


Last edited by vgersh99; 04-08-2019 at 12:28 PM..
# 3  
Old 04-08-2019
Try:
Code:
split -l20 inputfile outputfile.

# 4  
Old 04-08-2019
Hello abhaydas,

I believe your number of lines which you need in output split files are NOT even. So I am coming up here with an approach where one could mention the line numbers when they want to generate a new output file. Let's say your own example posted one, you need first 20 lines in file1, then next 10 lines to file2, next 10 lines to file3, next 10 lines to file4 and then next 40 lines to file5. If this is the case then could you please try following(I have NOT tested it, I am cooking right now, I am pretty sure it should WORK Smilie ).

Code:
awk '
BEGIN{
  split("20,10,10,10,40",array,",")
  count=1
  file="file"count
}
FNR==array[count]{
  print $0 > file
  close(file)
  file="file"++count
  next
}
{
  print $0 > file
}'  Input_file

NOTE: Point to be noted here is split("20,10,10,10,40",array,",") code's part is responsible for mentioning number of lines in output file so please be sure you are providing proper values here.

NOTE2: Also I am closing the output files with close(file) statement to avoid errors like
Quote:
Too many files opened etc..
Thanks,
R. Singh
# 5  
Old 04-09-2019
A meta-answer with a non-standard, local utility

Hi.

I think segment 41-50 is supposed to be skipped: 1-20 ,21-30 ,31-40 ,51-60 ,61-100

Our shop needed something like this long ago so we created a split utility, r3split, to handle these situations. When passing a file of lines f1 ... f100 through this command, we get:
Code:
 Results, lines in receiving files:
 Expecting first lines, 1(20), 21(10), 31(10), 51(10), 61-100
 Note - accepting partial completion on r05, exit.
Edges: 1:0:0 of 20 lines in file "r01"
f1
   ---

Edges: 1:0:0 of 10 lines in file "r02"
f21
   ---

Edges: 1:0:0 of 10 lines in file "r03"
f31
   ---

Edges: 1:0:0 of 10 lines in file "r04"
f51
   ---

Edges: 1:0:0 of 40 lines in file "r05"
f61
   ---

And this is the command:
Code:
r3split -c "20,10+2,-10,10," -p "r" $FILE

translating to: write to files beginning with "r", 20 lines, 10 lines repeated twice, skip 10 lines, everthing else to last receiving file.

Sadly we have not yet released any of our library of utilities.

The reason it is called r3split is for the 3 functions it performs. If anyone would like to use it as inspiration, here is the help page:
Code:
r3split - Split text file into Ratio, iRregular, or Random chunks.

usage: r3split [options] -- [files]

options:

Options which allow arguments require either a space or an "=" as
a separator, e.g. "-r 3" or "-r=3".

--help | -h
  print this message and quit.

--chunks | -c [-]c1,[-]c2 ... cn
  Transfer ci lines to next output file.  If preceded by "-" skip
  ci lines.  For convenience, the ci sequence may be the first
  parameter (or second after the --debug option).

  The chunks may be integers, ratios ( 1 < ci < 0 ), percentages
  (similar to ratios but 100 < ci < 0, followed by "%")  Ratios
  and percentages will require a preliminary (fast) pass over the
  file to find the length in lines (see "estimate:).  A count of
  zero will be treated as 1.

  If the symbol "+" occurs after the chunk, an integer repeat
  count may be specified.

  Usually the chunks are transferred in complete blocks, and
  remaining lines less than a chunk are ignored.  However if a
  final comma, ",", appears, then the incomplete chunk will be
  transferred to the final file. (Essentially a large number is
  created for the final chunk.)

  Ex: -c=1,1    --chunks 20%,.3,100     --chunks=-4,13,.75
  -c 3,3+       -c3,3+,

--random | -r n
  Transfer to n files, each file containing a random number of
  lines.  The total lines will equal the number of lines in the
  input file.  The NUMBER of lines is random here, the sequence
  is as it appears in the input.

--debug | -d
  Print debugging output.  Must be first option.
  
--suffix | -s format
  Suffix format, the default is "%02d".  To specify a ".txt" for
  each file use -s="%02d.txt".  The format is used to create the
  sequence of output filenames.

--prefix -p string
  Prefix, default is "xx", like many other split utilities.

--estimate | -e
  Allow the file length to be estimated, generally useful only
  for "long" files, perhaps 1 GB or more.  (This is accomplished
  by external routine "esmele" which might not be available on
  all systems.)

cheers, drl
This User Gave Thanks to drl For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ... (20 Replies)
Discussion started by: Raghuram717
20 Replies

2. UNIX for Dummies Questions & Answers

Split Every Line In Txt Into Separate Txt File, Named Same As The Line

Hi All Is there a way to export every line into new txt file where by the title of each txt output are same as the line ? I have this txt files containing names: Kandra Vanhooser Rhona Menefee Reynaldo Hutt Houston Rafferty Charmaine Lord Albertine Poucher Juana Maes Mitch Lobel... (2 Replies)
Discussion started by: Nexeu
2 Replies

3. Shell Programming and Scripting

Splitting XML file on basis of line number into multiple file

Hi All, I have more than half million lines of XML file , wanted to split in four files in a such a way that top 7 lines should be present in each file on top and bottom line of should be present in each file at bottom. from the 8th line actual record starts and each record contains 15 lines... (14 Replies)
Discussion started by: ajju
14 Replies

4. Shell Programming and Scripting

How to split a file based on pattern line number?

Hi i have requirement like below M <form_name> sdasadasdMklkM D ...... D ..... M form_name> sdasadasdMklkM D ...... D ..... D ...... D ..... M form_name> sdasadasdMklkM D ...... M form_name> sdasadasdMklkM i want split file based on line number by finding... (10 Replies)
Discussion started by: bhaskar v
10 Replies

5. UNIX for Dummies Questions & Answers

Splitting the file basis of Line Number

Can u pls advise the unix command as I have a file which contain the records in the below format 333434 435435 435443 434543 343536 Now the total line count is 89380 , now i want to create a separate I am trying to split my large big file into small bits using the line... (2 Replies)
Discussion started by: punpun66
2 Replies

6. Shell Programming and Scripting

Need to append the date | abcddate.txt to the first line of my txt file

I want to add/append the info in the following format to my.txt file. 20130702|abcd20130702.txt FN|SN|DOB I tried the below script but it throws me some exceptions. <#!/bin/sh dt = date '+%y%m%d'members; echo $dt+|+members+$dt; /usr/bin/awk -f BEGIN { FS="|"; OFS="|"; } { print... (6 Replies)
Discussion started by: harik1982
6 Replies

7. Shell Programming and Scripting

How to split this txt file into small files?

Dear shell experts, I would like to spilt a txt file into small ones. However, I did not know how to program use shell. If someone could help, it is greatly appreciated! Specifically, I supposed there is file named A.txt. The content of the file likes this: Subject run condtion ACC time... (3 Replies)
Discussion started by: psychmyluo
3 Replies

8. Shell Programming and Scripting

KSH script for split a txt file

I have a problem which I would like to solve by using UNIX power and inspired minds around world. Here is the problem I have a text file and it has data as follows 1X.....................1234567890123456789T1234598765XT1 (header) 1Z01............(sub HEADER) P100001............ Q1........... (4 Replies)
Discussion started by: ask.chowhan
4 Replies

9. Shell Programming and Scripting

Split one file to Multiple file with report basis in unix

Hi, Please help on this. i want split the below file(11020111.CLT) to more files with some condition. :b: 1) %s stating of the report 2) %e ending of the report example starting of the report: %sAEGONCA| |MUMBAI | :EXPC|N|D ending of the report %eAEGONCA| |MUMBAI | :EXPC 3)so the... (10 Replies)
Discussion started by: krbala1985
10 Replies

10. Shell Programming and Scripting

Split File Based on Line Number Pattern

Hello all. Sorry, I know this question is similar to many others, but I just can seem to put together exactly what I need. My file is tab delimitted and contains approximately 1 million rows. I would like to send lines 1,4,& 7 to a file. Lines 2, 5, & 8 to a second file. Lines 3, 6, & 9 to... (11 Replies)
Discussion started by: shankster
11 Replies
Login or Register to Ask a Question