Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Split and add header and trailer from input file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split and add header and trailer from input file
# 1  
Old 06-02-2014
Split and add header and trailer from input file
I need to split the file based on pattern from position 34-37 while retaining the header and trailer records in each individual split file

Also is it possible to output the TOM and PAT records in the same output file ?

I need the output file names same as xyz_pattern_Datetimestamp.txt

Code:
 
H00000012345678900000000 xxxxxxxxxxxxxx
D00000012300000000000000 xxxxxxxxTOMxxx
D00000045600000000000000 xxxxxxxxTOMxxx
D00000078900000000000000 xxxxxxxxPATxxx
D00000065000000000000000 xxxxxxxxPATxxx
D00000023100000000000000 xxxxxxxxPATxxx
D00000013200000000000000 xxxxxxxxSAMxxx
D00000036500000000000000 xxxxxxxxSAMxxx
T00000025800000000000000 xxxxxxxxxxxxxxx

So far I was able to split the file with the below command but struggling with header and trailer on individual split file and also with the output file name

Code:
 
awk -F "" 'BEGIN {x=""} NR>1 { x=substr($0,34,3); print $0 >> x".txt" }' Test.txt

# 2  
Old 06-02-2014
Welcome to the forum, and congrats for using code tags correctly right from the start!

Still, some questions remain:
Where's the header and the trailer?
Does the order of lines matter?
If you want TOM and PAT to go to the same file, which pattern should be used for the file name?
Do they have to be combined in the awk script, or can they be concatenated afterwards?
And, surely this is a typo, but you extract three chars in your script, but specify four (34 - 37).
# 3  
Old 06-02-2014
Thanks!

The first record in the input file is the Header record , the last record in the input file is the trailer record . Yes the order matters, the records on the output file should appear in the same sequence as they are on the input file

The first pattern TOM should be used for the output file name in the scenario where we are grouping the records i.e. TOM and PAT records going in to the same output file

Yes the pattern to look for is 3 characters from position 34
# 4  
Old 06-02-2014
This is a true split:
Code:
awk '
{
  if (/^H/) {header=$0}
  else if (/^T/) {trailer=$0}
  else {
    fname=substr($0,34,3)".txt"
    if (!(fname in A)) {A[fname]; print header > fname}
    print $0 >> fname
  }
}
END {
  for (fname in A) print trailer >> fname
}
' Test.txt

This User Gave Thanks to MadeInGermany For This Post:
techedipro
# 5  
Old 06-02-2014
Thank you , the above awk command splitted the files with header and trailer in each individual file

left with grouping of TOM and PAT records and the output file name
as xyz_pattern_Datetimestamp.txt
# 6  
Old 06-02-2014
Try this:

Code:
awk -v stamp=$(date +%Y%m%d%H%M%S) '
{
  if (/^H/) {header=$0}
  else if (/^T/) {trailer=$0}
  else {
    typ=substr($0,34,3)
    if(typ == "PAT") typ="TOM"
    fname="xyz_" typ "_" stamp ".txt"
    if (!(fname in A)) {A[fname]; print header > fname}
    print $0 >> fname
    close(fname)
  }
}
END {
  for (fname in A) {
    print trailer >> fname
    close(fname)
  }
}
' Test.txt

Note: You will need the close(fname) statements if you ever get more than 20 or so record types as awk has a limited number of open files.
# 7  
Old 06-03-2014
Thanks Chubler!

Can you please explain your code
Last edited by techedipro; 06-03-2014 at 12:14 AM.. Reason: typo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Removing Header and Trailer record of a EBCDIC file

I have a EBCDIC multi layout file which has a header record which is 21 bytes, The Detail records are 2427 bytes long and the trailer record is 9 bytes long. Is there a command to remove the header as well as trailer record and read only the detail records while at the same time not altering... (1 Reply)
Discussion started by: abhilashnair
1 Replies

2. Shell Programming and Scripting

Verify the header and trailer in file

please see my requirement, I hope I am clear. (9 Replies)
Discussion started by: mirwasim
9 Replies

3. Shell Programming and Scripting

Script to validate file header and trailer

Hi, I need a script that validates a file header/detail/trailer. File layout is: Header - Rec_Type|File_name|File_Date Detail - Rec_Type|field1|field2|field3... Trailder - Rec_Type|File_name|File_Date|Record_count Sample Data: HDR|customer_data.dat|20120709... (7 Replies)
Discussion started by: ash_sh
7 Replies

4. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

5. UNIX for Dummies Questions & Answers

Adding header and trailer into a file

Hi, I want to add the below Header to all the files in sequence File1,File2,File3...etc "ABC,<number of chracter in the file>" e,g - If File1 is as below pqrstuvdt abcdefgh then I want to add the above header into it ,So that File1 becomes as below ABC,17 pqrstuvdt abcdefgh ... (9 Replies)
Discussion started by: spari2
9 Replies

6. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

7. Shell Programming and Scripting

Creating Header & Trailer for bulk volume data file

Hi all, I have a requirement to create a Header &Trailer for a file which is having 20 millions of records. If I use the following method, i think it will take more time. cat "Header"> file1.txt cat Data_File>>file1.txt cat "Trailer">>file1.txt since second CAT command has to read all... (4 Replies)
Discussion started by: Raamc
4 Replies

8. Shell Programming and Scripting

Removing Header & Trailer from a file

Hi All, I am karthik. I am new to this forum. I have one requirement. I have a file with header and footer. Header may be like HDR0001 or FILE20090110 (Assume it is unknown so far, but i am sure there is a header in the file) likewise file has the trailer too. I just... (7 Replies)
Discussion started by: karthi_gana
7 Replies

9. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (7 Replies)
Discussion started by: ashish4422
7 Replies

10. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies
Login or Register to Ask a Question