Visit Our UNIX and Linux User Community

Removing Header & Trailer from a file

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing Header & Trailer from a file
# 1  
Old 10-02-2009
Removing Header & Trailer from a file

Hi All,

I am karthik. I am new to this forum. I have one requirement.

I have a file with header and footer.

Header may be like


(Assume it is unknown so far, but i am sure there is a header in the file)

likewise file has the trailer too.

I just want to remove the header & footer and BCP data into a table.

Note: The file size would be 1.5 GB to 2.0 GB.

So i am looking for some good way to remove the header and trailer.

Inputs are welcome!
# 2  
Old 10-02-2009
This should do the trick:

awk 'NR<3{s=$0;next}{print s;s=$0}' file > newfile


sed '1d;$d' file > newfile

# 3  
Old 10-02-2009
Thanks Franklin!

I tested the second one. It works fine.

But i don't want to create another file. I tested the script as below

filename > filename. After executing the script, the <filename> becomes empty.

so isn't possible to remove the header & footer and store it in the same file itself?

Also can you tell me what is the difference between the first one and second one? i mean which will give give good performance?

Last edited by karthi_gana; 10-02-2009 at 02:30 PM..
# 4  
Old 10-02-2009
awk 'NR<3{s=$0;next}{print s;s=$0}' file > newfile

Explanation of awk code:


If line number < 3 assign the current line to variable s and read the next line

{print s;s=$0}

Print the previous line and assign the current line to variable s

Explanation of sed code:

sed '1d;$d'

Delete the first (1d) and the last line ($d)
# 5  
Old 10-02-2009
Answer for one of your question:

How to make the edit in the same file itself,
use -i switch in sed as


# 6  
Old 10-07-2009
Originally Posted by thegeek
Answer for one of your question:

How to make the edit in the same file itself,
use -i switch in sed as


sed: illegal option -- i

i got the above error message.

i am using ksh.

is there any other option to do this?
# 7  
Old 10-08-2009
Originally Posted by karthi_gana
sed: illegal option -- i

i got the above error message.

i am using ksh.

is there any other option to do this?
No, but you can do something like:

sed '1d;$d' file > tempfile && mv tempfile file

Previous Thread | Next Thread
Test Your Knowledge in Computers #134
Difficulty: Easy
Linus Torvalds is known for disagreeing with other developers strongly and calls himself a 'really unpleasant person'.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Removing Header and Trailer record of a EBCDIC file

I have a EBCDIC multi layout file which has a header record which is 21 bytes, The Detail records are 2427 bytes long and the trailer record is 9 bytes long. Is there a command to remove the header as well as trailer record and read only the detail records while at the same time not altering... (1 Reply)
Discussion started by: abhilashnair
1 Replies

2. Shell Programming and Scripting

Split and add header and trailer from input file

I need to split the file based on pattern from position 34-37 while retaining the header and trailer records in each individual split file Also is it possible to output the TOM and PAT records in the same output file ? I need the output file names same as xyz_pattern_Datetimestamp.txt ... (23 Replies)
Discussion started by: techedipro
23 Replies

3. Shell Programming and Scripting

Verify the header and trailer in file

please see my requirement, I hope I am clear. (9 Replies)
Discussion started by: mirwasim
9 Replies

4. Shell Programming and Scripting

Script to validate file header and trailer

Hi, I need a script that validates a file header/detail/trailer. File layout is: Header - Rec_Type|File_name|File_Date Detail - Rec_Type|field1|field2|field3... Trailder - Rec_Type|File_name|File_Date|Record_count Sample Data: HDR|customer_data.dat|20120709... (7 Replies)
Discussion started by: ash_sh
7 Replies

5. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

6. UNIX for Dummies Questions & Answers

Adding header and trailer into a file

Hi, I want to add the below Header to all the files in sequence File1,File2,File3...etc "ABC,<number of chracter in the file>" e,g - If File1 is as below pqrstuvdt abcdefgh then I want to add the above header into it ,So that File1 becomes as below ABC,17 pqrstuvdt abcdefgh ... (9 Replies)
Discussion started by: spari2
9 Replies

7. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

8. Shell Programming and Scripting

Creating Header & Trailer for bulk volume data file

Hi all, I have a requirement to create a Header &Trailer for a file which is having 20 millions of records. If I use the following method, i think it will take more time. cat "Header"> file1.txt cat Data_File>>file1.txt cat "Trailer">>file1.txt since second CAT command has to read all... (4 Replies)
Discussion started by: Raamc
4 Replies

9. UNIX for Dummies Questions & Answers

Copy all the files with time stamp and remove header,trailer from file

All, I am new to unix and i have the following requirement. I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file.. Could... (35 Replies)
Discussion started by: ksrams
35 Replies

10. UNIX for Dummies Questions & Answers

Removing trailer from a flat file!!!

Hi, I get some flat files with trailer which gives the totol records count and i want to remove the trailer from the file. i used the following command it works fine with a single file. cat file_name | grep -v 'Total records:' > file1 mv file file_name But i dont know how to remove the... (12 Replies)
Discussion started by: kumarsaravana_s
12 Replies

Featured Tech Videos