Extract n number of lines from a file successively


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Extract n number of lines from a file successively
# 1  
Old 03-09-2011
Question Extract n number of lines from a file successively

Hello,

I have a file with over 100,000 lines. I would like to be able extract 5000 lines at a time and give it as an input to another program.

sed -n '1,5000p' <myfile> > myOut

Similarly for
5001-10000
10001-15000
....

How can I do this in a loop?

Thanks,
Guss
# 2  
Old 03-09-2011
Just thinking out loud...
would a combination of

Code:
Loop:
 head -n 5000 <infile >file1
 tail -n +5001 <infile >file2
 cp file2 infile

be more efficient?
# 3  
Old 03-09-2011
Hi.
Code:
NAME
       split - split a file into pieces

SYNOPSIS
       split [OPTION] [INPUT [PREFIX]]

DESCRIPTION
       Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
       size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when
       INPUT is -, read standard input.

See entire man page for details ... cheers, drl
# 4  
Old 03-09-2011
I did think about split; however I do not want multiple split files. Every break of 5000 gets output to a single file that is an input for another program. I do not want to clog up resources with multiple x00s.

Thanks,
Guss
# 5  
Old 03-09-2011
Hi.

Adapt as needed:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate successive slices of file.

FILE=${1-data1}

n=$( wc -l < $FILE )
echo " Info: processing $n lines of $FILE"
for (( i=1,e=5,j=1 ; i<=$n ; i+=5,e+=5 ))
do
  echo " Pass $j"
  sed -n "$i,${e}p" $FILE 
  # Pipe into process here or write file and process.
  (( j++ ))
done

producing:
Code:
% ./s1
 Info: processing 16 lines of data1
 Pass 1
1
2
3
4
5
 Pass 2
6
7
8
9
10
 Pass 3
11
12
13
14
15
 Pass 4
16

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
# 6  
Old 03-09-2011
Thank you drl!

~Guss
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract some lines from one file and add those lines to current file

hi, i have two files. file1.sh echo "unix" echo "linux" file2.sh echo "unix linux forums" now the output i need is $./file2.sh unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies

2. Shell Programming and Scripting

Perl extract number from file & write to file

I have 1 file that has elements as follows. Also the CVR(10) and the word "SAUCE" only appear once in the file so maybe a grep command would work? file1 CVR( 9) = 0.385E+05, ! VEHICLE CVR(10) = 0.246E+05, ! SAUCE CVR(11) = 0.162E+03, ! VEHICLE I need to extract the... (6 Replies)
Discussion started by: austinj
6 Replies

3. Shell Programming and Scripting

extract the lines by index number

Hi All, I want to extract the lines from file1 by using the index numbers from file2. In example, cat file1.txt 265 ABC 956 ... 698 DFA 456 ... 456 DDD 145 ... 125 DSG 154 ... 459 CGB 156 ... 490 ASF 456 ... 484 XFH 489 ... 679 hgt 481 ... 111 dfg 986 ... 356 vhn 444 ...... (7 Replies)
Discussion started by: senayasma
7 Replies

4. Shell Programming and Scripting

Extract a number from a line in a file and sed in another copied file

Dear all, I am trying to extract a number from a line in one file (task 1), duplicate another file (task 2) and replace all instances of the strings 300, in duplicated with the extracted number (task 3). Here is what I have tried so far: for ((k=1;k<4;k++)); do temp=`sed -n "${k}p"... (2 Replies)
Discussion started by: mnaqvi
2 Replies

5. Emergency UNIX and Linux Support

Urgent help pls.how to extract two lines having same starting number

Hi , I have a huge file like this =245 this is testing =035 abc123 =245 this is testing1 =035 abc124 =245 this is testing2 =035 abc125 =035 abc126 =245 this is testing3 here i have to pull out those lines having two =035 instead of alternative 035 and 245 i.e extract... (18 Replies)
Discussion started by: umapearl
18 Replies

6. Shell Programming and Scripting

Awk to extract lines with a defined number of characters

This is my problem, my file (file A) contains the following information: Now, I would like to create a file (file B) containing only the lines with 10 or more characters but less than 20 with their corresponding ID: Then, I need to compare the entries and determine their frequency. Thus, I... (7 Replies)
Discussion started by: Xterra
7 Replies

7. Shell Programming and Scripting

extract number range from a file

Hi Everyone, a.txt 1272904667;1272904737;1 1272904747;1272904819;1 1272904810;1272904857;1 1272904889;1272904926;1 1272905399;1272905406;1 1272905411;1272905422;1 if i want to get the record, when the a.txt 1st field is between 1272904749 and 1272905399, any simple way by using awk,... (1 Reply)
Discussion started by: jimmy_y
1 Replies

8. Shell Programming and Scripting

Number lines of file and assign variable to each number

I have a file with a list of config files numbered on the lefthand side 1-300. I need to have bash read each lines number and assign it to a variable so it can be chosen by the user called by the script later. Ex. 1 some data 2 something else 3 more stuff which number do you... (1 Reply)
Discussion started by: glev2005
1 Replies

9. Shell Programming and Scripting

extract the lines between specific line number from a text file

Hi I want to extract certain text between two line numbers like 23234234324 and 54446655567567 How do I do this with a simple sed or awk command? Thank you. ---------- Post updated at 06:16 PM ---------- Previous update was at 05:55 PM ---------- found it: sed -n '#1,#2p'... (1 Reply)
Discussion started by: return_user
1 Replies

10. Shell Programming and Scripting

extract a number within an xml file

Hi Everyone, I have an sh script that I am working on and I have run into a little snag that I am hoping someone here can assist me with. I am using wget to retrieve an xml file from thetvdb.com. This part works ok but what I need to be able to do is extract the series ID # from the xml and put... (10 Replies)
Discussion started by: tret
10 Replies
Login or Register to Ask a Question