Split file based on records


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split file based on records
# 1  
Old 07-11-2013
Oracle Split file based on records

I have to split a file based on number of lines and the below command works fine:

Code:
split -l 2 Inputfile -d Outputfile

My input file contains header, detail and trailor info as below:
Code:
H
D
D
D
D
T

My split files for the above command contains:
First File:
Code:
H
D

Second File:
Code:
D
D

Third File
Code:
D
T

But i do not want the H and T records. I want only the detail record.

Can any one help me on achieving this.

Thanks,
Ajay

Last edited by radoulov; 07-11-2013 at 07:58 AM..
# 2  
Old 07-11-2013
you can use below awk to get rid of H and T

Code:
 
awk -v nl=$(wc -l < filename) 'NR>1 && NR<nl{print $0}' filename

However you can pass it to split when its using -d option as it might throw error so i suggest forming a temp file or use diff method other than split to create the files.
# 3  
Old 07-12-2013
Hi Vidyadhar,

Thanks for the reply, this works but when I do not have the header or trailor record in the input file, even then it strips the first and the last line. I wanted to split the files based on the TAGS. Only for lines which starts with a substring say "D", i want to split the file.

I thought of the idea of doing it with temp file but since I am dealing with huge files upto 30GB, taking it to a temp will take more time.

Is there any other solution for this.

Thanks,
Ajay
# 4  
Old 07-12-2013
Can you give your exact requirements?

Whats the exact string?
What and how will u decide its header or trailer?
How many records in per file? if you go for 2 records per file for 30GB file you end up consuming all inum Smilie
# 5  
Old 07-12-2013
Ok. I will have Header, Detail and Trailor records, of which Detail will looping (infinite) and Heasder/Trailor will not loop and they might of migth not occur in the inuput (In other workd they are not mandatory). I want to split the file the file based on a particualr number of lines only for the Detail records, omiting the Header and Trailor records.

I can as well delete the header and trailor records and then split the file. But i require them for other purpose.

If you require any clarification, please do ask me.


Thanks,
Ajay

---------- Post updated at 12:10 PM ---------- Previous update was at 12:09 PM ----------

The records will be identified based on the TAGS.
# 6  
Old 07-12-2013
How about this ? Assuming header is the first line and trailer is the last line

Code:
sed -e '1d' -e '$d' file  | split -l2

# 7  
Old 07-12-2013
Hi Raja,adhvan,

This work. But when there is no Header and trailor in the file. Then this will strip off the first record which acuallly would have been the detail.

Instead i tried someting like this:
Code:
awk '{if (substr($0,1,1) == "D") {print $0;}}' Filename

This gets you the list of detail records. But i am not sure how to pass the output of this command to the split command?
Any suggetions on that?

Thanks,
Ajay

Last edited by Franklin52; 07-14-2013 at 05:59 PM.. Reason: Please use code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Want to grep records in alphabetical order from a file and split into other files

Hi All, I have one file containing thousands of table names in single column. Now I want that file split into multiple files e.g one file containing table names starting from A, other containing all tables starting from B...and so on..till Z. I tried below but it did not work. for i in... (6 Replies)
Discussion started by: shekhar_4_u
6 Replies

2. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

3. Shell Programming and Scripting

Filter records based on 2nd file

Hello, I want to filter records of a file if they fall in range associated with a second file. First the chr number (2nd col of 1st file and 1st col of 2nd file) needs to be matched. Then if the 3rd col of the first file falls within any of the ranges specified by the 2nd and 3rd cols , then... (4 Replies)
Discussion started by: ritakadm
4 Replies

4. Shell Programming and Scripting

Split records into multiple records

Hi All, I am trying to split a record into multiple records based on a value. Input.txt "A",1,0,10 "B",2,0,10,15,20 "C",3,11,14,16,19,21,23 "D",1,0,5 My desired output is: "A",1,0,10 "B",2,0,10 "B",2,15,20 "C",3,11,14 "C",3,16,19 "C",3,21,23 (4 Replies)
Discussion started by: kmsekhar
4 Replies

5. UNIX for Advanced & Expert Users

Split records based on '-'

HI, I have a pipe delimiter file , I have to search for second field pattern, if the second field does not contain a '-' , I need to start capturing the record from this line till I find another second field with '-' value. Below is the sample data SOURCE DATA ABC|ABC_702148-PARAM... (3 Replies)
Discussion started by: mora
3 Replies

6. Shell Programming and Scripting

parallel while loop based on the file records

Hi, I need to execute parallel with while loop. Input File(source_file.csv) contains filenames the below source_file.csv file contains Customer1.txt Product1.txt Sales.txt Emp.txt Dept.txt Based on the number of rows that file I want to run the script ‘n' times. while... (2 Replies)
Discussion started by: onesuri
2 Replies

7. UNIX for Dummies Questions & Answers

How to split multiple records file in n files

Hello, Each record has a lenght of 7 characters I have 2 types of records 010 and 011 There is no character of end of line. For example my file is like that : 010hello 010bonjour011both 011sisters I would like to have 2 files 010.txt (2 records) hello bonjour and ... (1 Reply)
Discussion started by: jeuffeu
1 Replies

8. Shell Programming and Scripting

Based on num of records in file1 need to check records in file2 to set some condns

Hi All, I have two files say file1 and file2. I want to check the number of records in file1 and if its atleast 2 (i.e., 2 or greater than 2 ) then I have to check records in file2 .If records in file2 is atleast 1 (i.e. if its not empty ) i have to set some conditions . Could you pls... (3 Replies)
Discussion started by: mavesum
3 Replies

9. UNIX for Dummies Questions & Answers

Filtering records of a file based on a value of a column

Hi all, I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g. COL1 COL2 COL3 ............... COL47 1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies

10. Shell Programming and Scripting

Sort & Split records in a file

Hi, I am new to scripting. I need a script to sort and the records in a file and then split them into different files. For example, the file is: H1...................... H2...................... D2.................... D2.................... H1........................... (15 Replies)
Discussion started by: Sunitha_edi82
15 Replies
Login or Register to Ask a Question