File handling issue

Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers File handling issue
# 1  
Old 08-08-2013
File handling issue

Hi All,

I am running into an issue. I have a very big file. Wants to split it in smaller chunks. This file has multiple header/ trailers. Also, between each header/trailer there are records. Number of records in each header trailer combination can vary. Also, headers can start with 202XXXXXXXXXXXXXXX, same for trailer.
Records start like 1101XXXXXXXXXXXXXXX.

I want to split file say like
split -l 99999 filename.

Say last 8 records are like

My script should put till record 3 in first file and records from 4 onwards in the next file as maching trailer for header starting at record 4 is missing.

Can someone help. Please ask all queries.


Last edited by Scott; 08-08-2013 at 12:05 PM.. Reason: Added Code tags [7th time]
# 2  
Old 08-08-2013
please post what you have done so far so people can help you ...
# 3  
Old 08-08-2013
Hi Just Ice,

I think I got the way to code this solution.

But I need one help


grep -n ^202 filename > test

 cut -d ':' -f1,1 test >test1
 test1 will have

Say I want to have maximum 5 records in first file
So, what I will do is add to first header + 5 = 6

And find a number in test1 less than 6
which is 4. Check if it is header or trailer

Since this is header will cut 1thru 3 and 4 thru 9 and generate 2 split files.

All I need is, Is there a way to find a number less than 6 in my test1 file without reading each record line by line.

Reason I want to do that is my file is very very big. It has trillions of records.

Reading header trailer number line by line will make my processing too slow.

Last edited by Scott; 08-08-2013 at 02:26 PM.. Reason: Added code tags [8th time]
# 4  
Old 08-08-2013
the tool/command you use will eventually read the whole file based on your requirements ...

however, see this thread as the requirements are similar to yours ...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File handling

Hi All, I need to extract the data from the text file. The data of the text file is shown below #L 0.000017 4.329939 0.000017 4.716267 r7.9 P 1 1;Net=IN32 The extracted data should be IN32. Could anyone help to script in c shell.? (4 Replies)
Discussion started by: gopishrine
4 Replies

2. Shell Programming and Scripting

Issue handling single quoted argument in shell script.

Below is my script that works fine and prints the desired output: #!/bin/ksh echo "$1" | while IFS= read -r dirpath do echo "DIRR_PATH:$dirpath" install_dir=$install_dir" "$dirpath done echo "Desired Output:$install_dir" Output: ./ (10 Replies)
Discussion started by: mohtashims
10 Replies

3. Shell Programming and Scripting

ISSUE in handling multiple same name files :-(

Dear all, My work is completely stuck cos of the following issue. Please find it here and kindly help me. Task is following: I have set of files with such pattern 1t-rw-rw-r-- 1 emily emily 119 Jun 11 10:45 vgtree_5_1_pfs.root 3t-rw-rw-r-- 1 emily emily 145 Jun 11 10:46 vgtree_5_3_pfs.root... (4 Replies)
Discussion started by: emily
4 Replies

4. UNIX for Dummies Questions & Answers

File Handling

Hi Team, I am trying to cut a large file into multiple files. It has Header 50,050 records Trailer ------------------------------------------- I need to cut the files into multiple files of 1000 records and should have the same header and trailer as the original files. ... (4 Replies)
Discussion started by: Gurkamal83
4 Replies

5. UNIX for Dummies Questions & Answers

Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long" I need to actually remove the last | symbol from this file. sed -e 's/\|*$//' filename is working fine for small files. But not working on this big... (13 Replies)
Discussion started by: Gurkamal83
13 Replies

6. Shell Programming and Scripting

UNIX file handling issue

I have a huge file semicolon( ; ) separated records are Pipe(|) delimited. e.g abc;def;ghi|jkl;mno;pqr|123;456;789 I need to replace the 50th field(semicolon separated) of each record with 9006. The 50th field can have no value e.g. ;; Can someone help me with the appropriate command. (3 Replies)
Discussion started by: Gurkamal83
3 Replies

7. Shell Programming and Scripting

Issue with Error handling,not able to continue the script further

Hi, I am trying to write a script to cleanup files in a log directory .. cd log find Datk** -mtime +7 -exec rm -f {} \; 2> /dev/null Have used the above to clean up files in log directory more then 7 days older. The file can be something like ( auto-generate by some processes and... (2 Replies)
Discussion started by: nss280
2 Replies

8. Shell Programming and Scripting

UNIX File handling -Issue in reading a file

I have been doing automation of daily check activity for a server, i have been using sqls to retrive the data and while loop for reading the data from the file for several activities. BUT i got a show stopper the below one.. where the data is getting store in $temp_file, but not being read by while... (1 Reply)
Discussion started by: KuldeepSinghTCS
1 Replies

9. UNIX for Advanced & Expert Users

please help me in file handling

sir i have to get first line from a file for example >cat file1 abc zxc asd adsf from that file1 i need only first line expected result >abc please help me ! (1 Reply)
Discussion started by: ponmuthu
1 Replies

10. Programming

File Handling in C

Hi all, I have a problem in handling files through C. here is the problem im having: i will query the database (for instance consider employees table ) for empno,ename,job,salary fields.The query returns me some 100 of rows. now i need to place them in a file in row wise pattern as they... (3 Replies)
Discussion started by: trinath
3 Replies
Login or Register to Ask a Question