Want to process file in 1000 record increments


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Want to process file in 1000 record increments
# 1  
Old 11-24-2008
Want to process file in 1000 record increments

I have a file that varies in the # of records. I want to read 1000 records, do something, read the next 1000 records, do something, going in 1000 increment chunks, and then finish up with whatever is left over at end of file.
# 2  
Old 11-24-2008
Hammer & Screwdriver

Take a look at the split command to break the file into pieces. The -l option sets the linecount, but the default is already 1000 lines.

Do a man split on your unix system to read more. Pay special attention to the output filenaming conventions.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract timestamp from first record in xml file and it checks if not it will replace first record

I have test.xml <emp><id>101</id><name>AAA</name><date>06/06/14 1811</date></emp> <Join><id>101</id><city>london</city><date>06/06/14 2011</date></join> <Join><id>101</id><city>new york</city><date>06/06/14 1811</date></join> <Join><id>101</id><city>sydney</city><date>06/06/14... (2 Replies)
Discussion started by: vsraju
2 Replies

2. UNIX for Dummies Questions & Answers

Process the record only for defined length

I have two files. One file have length defined and other file have data. I need to do some processing in data file only for the length mentioned in the length file. my lenth file has one record with five bytes, need to store the last two bytes as length. for example, if my file has a... (3 Replies)
Discussion started by: srvn_saru
3 Replies

3. Shell Programming and Scripting

Help with write a master script to automatic 1000 process

Hi, I just random create 1000 folder which range from 0-20,000 . Inside all the folder, I have copy over the same program named as run.sh to run in each folder. Can I know that how to write a master script to let it automatic run the "run.sh" in each different random folder create? Some... (5 Replies)
Discussion started by: perl_beginner
5 Replies

4. Shell Programming and Scripting

how to print the value > 1000 in a file

if the file test include 1233 0.123 0.128 0.165 0.14 14.5 134556 48968 can i show the output value > 1000 by command grep or awk or...? 1233 134556 48968 Thank you. (6 Replies)
Discussion started by: bleach8578
6 Replies

5. Shell Programming and Scripting

Increments in a loop

Hi All, I am trying to implement a for loop in a .sh file and execute it. My for loop is for i in 1, 200, 400, 600, 800, 1000, 1200,......100000 do ... .. .. done As you could see there is an increment of 200 in each for loop until it reaches 100,000 Instead of specifying all the... (8 Replies)
Discussion started by: Lucky Ali
8 Replies

6. Shell Programming and Scripting

retain last 1000 line in a file

I have large file with around 100k+ lines. I wanted to retain only the last 100 lines in that file. One way i thought was using tail -1000 filename > filename1 mv filename1 filename But there should be a better solution.. Is there a way I can use sed or any such command to change the... (9 Replies)
Discussion started by: nss280
9 Replies

7. UNIX for Dummies Questions & Answers

how to read record by record from a file in unix

Hi guys, i have a big file with the following format.This includes header(H),detail(D) and trailer(T) information in the file.My problem is i have to search for the character "6h" at 14 th and 15 th position in all the records .if it is there i have to write all those records into a... (1 Reply)
Discussion started by: raoscb
1 Replies

8. Shell Programming and Scripting

splitting a record and adding a record to a file

Hi, I am new to UNIX scripting and woiuld appreicate your help... Input file contains only one (but long) record: aaaaabbbbbcccccddddd..... Desired file: NEW RECORD #new record (hardcoded) added as first record - its length is irrelevant# aaaaa bbbbb ccccc ddddd ... ... ... (1 Reply)
Discussion started by: rsolap
1 Replies

9. Programming

ip address octet increments

Hi all, Situation is as below. I would get an IP address and port from eithe r a file or command line. It probably would be as char * or string. So was wondering how I could accept this and increment the last octets? Incrementing the port is fine. I could get that into an integer by atoi()... (8 Replies)
Discussion started by: Naanu
8 Replies

10. Shell Programming and Scripting

Working with field increments

I have a new challenge that I need some help with. Each morning I have two files that contain working units and failed units that I join together to form file3 which is FTP to server for comparison against billing system. My problem, I would like to take joined data in file3 and add a field... (2 Replies)
Discussion started by: greengrass
2 Replies
Login or Register to Ask a Question