Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Strip header and footer

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Strip header and footer
# 1  
Old 06-14-2013
Strip header and footer
Hi I have below requirements on the script below :
(1) I receive 2 pipe seperated file called OUT.psv and DIFF.psv with a column header.I concatenate the 2 files and create a final.psv file. I want to add another header as START_FILE to the final.psv file . How to achieve this ?

(2) I have added 3 footers using echo in the script successfully.

(3)Next day i run the script again which needs to first strip the first header which is START_FILE and last 3 headers and do the concatenate of a OUT and DIFF file and then sort . How to remove the first header and last 3 headers and the continue with my usual script to concatenate and sort ?

Code:
#!/bin/ksh
cat /app/reporting/daily/OUT.psv /app/reporting/daily/temp/DIFF.psv >> /app/reporting/daily/final.psv
sort -u -t'|' -k1,1r /app/reporting/daily/final.psv
COUNT=$( wc -l) < /app/reporting/daily/final.psv
echo "END_OF_FILE" >> /app/reporting/daily/final.psv
echo "Time_Finished"$NOW >> /app/reporting/daily/final.psv
echo "RECORD_COUNT"$COUNT >> /app/reporting/daily/final.psv
rm /app/reporting/daily/OUT.psv 
rm /app/reporting/daily/temp/DIFF.psv

Output:
-------

Code:
START_OF_FILE
time|tourit|nofdays|rbcid|blank|type|value|nill|valuedesc|name
2013-05-16T00:52:31.662-04:00|12|3|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO ABLatest
2013-05-15T00:52:31.672-04:00|1|40|39693766-NA-NA||Common Stick|ESHR||Common Stock|HS AG
2013-05-14T00:52:31.672-04:00|1|45|16111278-TSX-NA||Common Stk|ESQHR||Common Stock|STANDARD REGISTER CO
2013-05-14T00:52:31.662-04:00|1|4|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO AB
2013-05-13T00:52:31.672-04:00|1|44|15277105-NA-NA||Common Stock|DOMESHR||Common Stock|CYRO AB
2013-05-10T00:52:31.672-04:00|1|5|39693766-NYSE-EUR||Common HSAGStick|ESHR||Common Stock|HS AG
END_OF_FILE
Time_Finished=Fri Jun 14 16:41:09 EDT 2013
RECORD_COUNT= 7

Last edited by Scrutinizer; 06-14-2013 at 08:26 PM.. Reason: code tags also for data samples
# 2  
Old 06-14-2013
This is one way, You can create the output file with just the header you want to add and then concatenate the other files into it:
Code:
echo START_OF_FILE > /app/reporting/daily/final.psv
cat /app/reporting/daily/OUT.psv /app/reporting/daily/temp/DIFF.psv >> /app/reporting/daily/final.psv

You can use sed or grep to retreive only the 'data' lines from the file for processing:
Code:
$ cat t
START_OF_FILE
time|tourit|nofdays|rbcid|blank|type|value|nill|valuedesc|name
2013-05-16T00:52:31.662-04:00|12|3|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO ABLatest
2013-05-15T00:52:31.672-04:00|1|40|39693766-NA-NA||Common Stick|ESHR||Common Stock|HS AG
2013-05-14T00:52:31.672-04:00|1|45|16111278-TSX-NA||Common Stk|ESQHR||Common Stock|STANDARD REGISTER CO
2013-05-14T00:52:31.662-04:00|1|4|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO AB
2013-05-13T00:52:31.672-04:00|1|44|15277105-NA-NA||Common Stock|DOMESHR||Common Stock|CYRO AB
2013-05-10T00:52:31.672-04:00|1|5|39693766-NYSE-EUR||Common HSAGStick|ESHR||Common Stock|HS AG
END_OF_FILE
Time_Finished=Fri Jun 14 16:41:09 EDT 2013
RECORD_COUNT= 7


$ sed -n '/^[0-9]/p' t
2013-05-16T00:52:31.662-04:00|12|3|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO ABLatest
2013-05-15T00:52:31.672-04:00|1|40|39693766-NA-NA||Common Stick|ESHR||Common Stock|HS AG
2013-05-14T00:52:31.672-04:00|1|45|16111278-TSX-NA||Common Stk|ESQHR||Common Stock|STANDARD REGISTER CO
2013-05-14T00:52:31.662-04:00|1|4|15277105-NA-YEN||Common Stick|ESHR||Common Stock|CYRO AB
2013-05-13T00:52:31.672-04:00|1|44|15277105-NA-NA||Common Stock|DOMESHR||Common Stock|CYRO AB
2013-05-10T00:52:31.672-04:00|1|5|39693766-NYSE-EUR||Common HSAGStick|ESHR||Common Stock|HS AG

# 3  
Old 06-15-2013
HI thanks for input but just to let you know the number of columns may be added more in future . So in your statement below is it only existing 10 columns you counted? .
Code:
sed -n '/^[0-9]/p' t

# 4  
Old 06-15-2013
Quote:
Originally Posted by samrat dutta
HI thanks for input but just to let you know the number of columns may be added more in future . So in your statement below is it only existing 10 columns you counted? .
Code:
sed -n '/^[0-9]/p' t

This sed command does not care how many columns are in its input file; it prints lines from its input if and only if the first character on the line is a digit. Since all of the header and trailer lines in your input file start with an alphabetic character, this sed command just discards headers and trailers from the input file and prints all of the non-header and non-trailer lines.
This User Gave Thanks to Don Cragun For This Post:
samrat dutta
# 5  
Old 06-17-2013
Thanks for clarification. Just want to clarify one more thing. If i have to write the output of this sed to a temp file, using > or >> gives same result. Which one to use > or >> ?

Code:
sed -n '/^[0-9]/p' final.psv > temp.psv

OR
Code:
sed -n '/^[0-9]/p' final.psv >> temp.psv

# 6  
Old 06-17-2013
">" overwrites
">>" appends

---------- Post updated at 12:08 ---------- Previous update was at 12:06 ----------

Additional info: I/O Redirection
Last edited by spacebar; 06-17-2013 at 02:08 PM.. Reason: Additional info
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

[solved] how to remove header and footer

it still display header and footer header SQL*Plus: Release 10.2.0.1.0 - Production on Mon Sep 24 13:41:51 2012 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production With the Partitioning, Real... (0 Replies)
Discussion started by: ment0smintz
0 Replies

2. Shell Programming and Scripting

Add header and footer with record count in footer

This is my file(Target.txt) name|age|locaction abc|23|del xyz|24|mum jkl|25|kol The file should be like this 1|03252012 1|name|age|location 2|abc|23|del 2|xyz|24|mum 2|jkl|25|kol 2|kkk|26|hyd 3|4 Column 1 is row indicator for row 1 and 2, column indicator is 1,for data rows... (1 Reply)
Discussion started by: itsranjan
1 Replies

3. Shell Programming and Scripting

Removing header and footer

I have two files which are getting sent to a UNIX server in order to be bcp'd into a database. The bcp is failing because there's a header and footer row on the file which give the date of the file and the number of rows in it. That's because the file is also being used for another process, so we... (1 Reply)
Discussion started by: Tom Sawyer
1 Replies

4. Shell Programming and Scripting

Header and Footer...

Hi All, I need to write a script that In my file I have to check header and footer records are available or not. If it is available I have to run the script, otherwise I should not. But current script it is checking only the data inside the script. It is avoiding to check Header and Footer... (1 Reply)
Discussion started by: suresh_target
1 Replies

5. UNIX for Dummies Questions & Answers

remove the header and footer using sed

I want to delete the header and footer in the file by using sed for that i ran the below script and my text file looks like emp.txt # This file contain employee # information abc 12300 34 'FGH' # This is confidential as per the firm rules. my intention is to remove the header... (8 Replies)
Discussion started by: vmachava
8 Replies

6. UNIX for Dummies Questions & Answers

Help with the Header and Footer check

Hi, I need to check whether the incoming file has a header and footer using a UNIX script. The pattern of the header and footer is fixed as follows: Header: Name,Date Footer: Count, Total Name,Date ------------------------- ------------------------- ------------------------- Count,... (5 Replies)
Discussion started by: Sunny_teotia
5 Replies

7. Shell Programming and Scripting

How to add header and footer?

Hi, Guys, I want add header and footer in a file. I can add footer using following command: echo "Footer" >>file. I don't know how to add header. Thanks in advance (4 Replies)
Discussion started by: ken002
4 Replies

8. Shell Programming and Scripting

Inserting Header and footer

Hi All, I have several txt files i need to enter specific header and footer (both are separate) to all these files how can i do this? plz help.. Regards, Raghav (4 Replies)
Discussion started by: digitalrg
4 Replies

9. Shell Programming and Scripting

rowcnt except Header & Footer

Hi Gurus, My requirement is, I am passing a file1.dat into this(rowcnt.sh) script,but returning a wrong value of -2.(it should be 4).Becoz my file1.dat contains 6records incl: Header & Footer.(6-2=4) wrong output: ------- #sh rowcnt.sh file1.dat -2 actual_cnt except HDR & FTR should be:... (3 Replies)
Discussion started by: vsubbu1000
3 Replies

10. Shell Programming and Scripting

remove header and footer rows

I would like to remove some lines from begining of file (header) and some lines from end of file (footer). The header/footer lines generated by web-browser when the user upload a file to my webserver. Example: -----------------------------7d62af20c052c Content-Disposition: form-data;... (2 Replies)
Discussion started by: seaky
2 Replies
Login or Register to Ask a Question