Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Split and add header and trailer from input file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split and add header and trailer from input file
# 15  
Old 06-09-2014
Thank you.

New line characters are appearing in the error file if they happen to present in the input file , how can I avoid them ?

Also can you briefly explain the new portion of the code

Appreciate your help!
# 16  
Old 06-09-2014
Can you supply a demo of the input file with this new line characters, are you talking about blank/empty lines?

To explain the new code:

Code:
    -v f="TOM PAT SAM BOB KIM" \
    -v t="TOM TOM TOM BOB KIM" '
BEGIN {
   split(f,from)
   for(i=split(t,to);i;i--) CONV[from[i]]=to[i]
}

Above we pass into awk two space separated lists f and t (for from and to). These are loaded into a CONV lookup array. Such that:
Code:
CONV[TOM] => TOM 
CONV[PAT] => TOM
CONV[KIM] => KIM

Below we check the line and if it's setup in the CONV array we assign the fname and write the header line to the file when this is the first time this file is written. If line is not setup in CONV array fname is "xyz_error_" plus time-stamp instead:

Code:
   if (typ in CONV) {
       fname="xyz_" CONV[typ] "_" stamp ".txt"
       if (!(fname in A)) {A[fname]; print header > fname}
   } 
   else fname="xyz_error_" stamp ".txt"

# 17  
Old 06-10-2014
Yes, I am talking about line feed characters at the end of the file


Code:
 
H00000012345678900000000 xxxxxxxxxxxxxx
D00000012300000000000000 xxxxxxxxTOMxxx
D00000045600000000000000 xxxxxxxxTOMxxx
D00000078900000000000000 xxxxxxxxPATxxx
D00000065000000000000000 xxxxxxxxPATxxx
D00000023100000000000000 xxxxxxxxPATxxx
D00000013200000000000000 xxxxxxxxSAMxxx
D00000036500000000000000 xxxxxxxxSAMxxx
D00000036500000000000000 xxxxxxxxBOBxxx
D00000036500000000000000 xxxxxxxxBOBxxx
D00000036500000000000000 xxxxxxxx   xxx
D00000036500000000000000 xxxxxxxx   xxx
T00000025800000000000000 xxxxxxxxxxxxxx

# 18  
Old 06-11-2014
How about this change to ignore lines with 3 or less characters:

Code:
awk -v stamp=$(date +%Y%m%d%H%M%S) \
    -v f="TOM PAT SAM BOB KIM" \
    -v t="TOM TOM TOM BOB KIM" '
BEGIN {
   split(f,from)
   for(i=split(t,to);i;i--) CONV[from[i]]=to[i]
}
/^H/ {header=$0 ; next}
/^T/ {trailer=$0 ; next}
length>3{
   typ=substr($0,34,3)
   if (typ in CONV) {
       fname="xyz_" CONV[typ] "_" stamp ".txt"
       if (!(fname in A)) {A[fname]; print header > fname}
   } 
   else fname="xyz_error_" stamp ".txt"
   print $0 >> fname
   close(fname)
}
END {
  for (fname in A) print trailer >> fname
}' Test.txt

# 19  
Old 06-16-2014
Thanks Chubler ..greatly appreciate it !
# 20  
Old 07-06-2014
One more change ,I need to replace a bunch of special characters on the input file while I am splitting it

Below is the list that I need to find and then replace with the corresponding value

This list could grow..I need to be able to control this

Code:
Character Found    Character Replaced
ED                SPACE
E9                SPACE
D9                SPACE
C2                SPACE
80                SPACE
99                SPACE
E2                SPACE
0D0A              0A
C3                SPACE
C4                SPACE
C8                SPACE
C9                SPACE
A0                SPACE
FF                SPACE
B1                SPACE
83                SPACE
C1                SPACE
E1                SPACE
B7                SPACE
F6                SPACE
F1                SPACE
F3                SPACE
E3                SPACE
1A                SPACE

Appreciate your help!
# 21  
Old 07-06-2014
How about having your replace.txt file formatted like this:

Code:
ED:20:
E9:20:
D9:20:
C2:20:
80:20:
99:20:
E2:20:
0D24::
C3:20:
C4:20:
C8:20:
C9:20:
A0:20:
FF:20:
B1:20:
83:20:
C1:20:
E1:20:
B7:20:
F6:20:
F1:20:
F3:20:
E3:20:
1A:20:

Note how I replaced your OD0A value with OD24 this is because awk strips the CR characters from each line. To match end-of-line we use the dollar sign (0x24), this being the RE meta-character for end-of-string.

This also means if you want to replace any RE meta-characters (eg star or dot) you will need to backslash them so to replace star with hash we would use 5C2A:23


Below is the code which does rely on the gawk --non-decimal-data command line switch:

Code:
awk --non-decimal-data \
-v stamp=$(date +%Y%m%d%H%M%S) \
    -v f="TOM PAT SAM BOB KIM" \
    -v t="TOM TOM TOM BOB KIM" '
function hextoascii(str,i,ret) {
   for(i=1;i<length(str);i+=2)
      ret=ret sprintf("%c", 0 + ("0x" substr(str,i,2)))
   return ret
}
BEGIN {
   split(f,from)
   for(i=split(t,to);i;i--) CONV[from[i]]=to[i]
}
FNR==NR{
   v=split($0,vals,":")
   repl[hextoascii(vals[1])]=hextoascii(vals[2])
   next
}
{ for(i in repl) gsub(i, repl[i]) }
/^H/ {header=$0 ; next}
/^T/ {trailer=$0 ; next}
length>3{
   typ=substr($0,34,3)
   if (typ in CONV) {
       fname="xyz_" CONV[typ] "_" stamp ".txt"
       if (!(fname in A)) {A[fname]; print header > fname}
   }
   else fname="xyz_error_" stamp ".txt"
   print $0 >> fname
   close(fname)
}
END {
  for (fname in A) print trailer >> fname
}' replace.txt Test.txt
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Removing Header and Trailer record of a EBCDIC file

I have a EBCDIC multi layout file which has a header record which is 21 bytes, The Detail records are 2427 bytes long and the trailer record is 9 bytes long. Is there a command to remove the header as well as trailer record and read only the detail records while at the same time not altering... (1 Reply)
Discussion started by: abhilashnair
1 Replies

2. Shell Programming and Scripting

Verify the header and trailer in file

please see my requirement, I hope I am clear. (9 Replies)
Discussion started by: mirwasim
9 Replies

3. Shell Programming and Scripting

Script to validate file header and trailer

Hi, I need a script that validates a file header/detail/trailer. File layout is: Header - Rec_Type|File_name|File_Date Detail - Rec_Type|field1|field2|field3... Trailder - Rec_Type|File_name|File_Date|Record_count Sample Data: HDR|customer_data.dat|20120709... (7 Replies)
Discussion started by: ash_sh
7 Replies

4. Shell Programming and Scripting

Remove last few characters in a file but keeping Header and trailer intact

Hi All, I am trying write a simple command using AWK and SED to this but without any success. Here is what I am using: head -1 test1.txt>test2.txt|sed '1d;$d' test1.txt|awk '{print substr($0,0,(length($0)-2))}' >>test2.txt|tail -1 test1.txt>>test2.txt Input: Header 1234567 abcdefgh... (2 Replies)
Discussion started by: nvuradi
2 Replies

5. UNIX for Dummies Questions & Answers

Adding header and trailer into a file

Hi, I want to add the below Header to all the files in sequence File1,File2,File3...etc "ABC,<number of chracter in the file>" e,g - If File1 is as below pqrstuvdt abcdefgh then I want to add the above header into it ,So that File1 becomes as below ABC,17 pqrstuvdt abcdefgh ... (9 Replies)
Discussion started by: spari2
9 Replies

6. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

7. Shell Programming and Scripting

Creating Header & Trailer for bulk volume data file

Hi all, I have a requirement to create a Header &Trailer for a file which is having 20 millions of records. If I use the following method, i think it will take more time. cat "Header"> file1.txt cat Data_File>>file1.txt cat "Trailer">>file1.txt since second CAT command has to read all... (4 Replies)
Discussion started by: Raamc
4 Replies

8. Shell Programming and Scripting

Removing Header & Trailer from a file

Hi All, I am karthik. I am new to this forum. I have one requirement. I have a file with header and footer. Header may be like HDR0001 or FILE20090110 (Assume it is unknown so far, but i am sure there is a header in the file) likewise file has the trailer too. I just... (7 Replies)
Discussion started by: karthi_gana
7 Replies

9. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (7 Replies)
Discussion started by: ashish4422
7 Replies

10. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies
Login or Register to Ask a Question