11-17-2008
Merge text files while combining the multiple header/trailer records into one each.
Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to reduce its run-time. This works fine, however, we're faced with multiple interface files that need to be merged prior transferring to the vendor.
Some Details on the File:
The file generated has a header and a trailer record, and the trailer record has pertinent total values (i.e., employee count, records approved, etc). There are no field separators -- these are fixed length fields.
Predicament in Detail:
We'd like to concatenate the files -- that's the easy part. What makes this difficult is that we need to eliminate the multiple header records and retain only the first one. Also, we need to eliminate the multiple trailer records, but we need to add all the value totals from each trailer into the one trailer record we'll retain at the end.
As you might have surmised by now, I've written some UNIX scripts, but lack some key knowledge related to individual record and field manipulation within a text file. In particular, I'd like to know how I can define specific fields when I read each record -- these are the fields for the trailer records I need to keep a rolling total on. Also, I'd like to know how I can delete individual records.
Any assistance will be greatly appreciated.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a flat file and need to count no of records in the file less the header and the trailer record.
I would appreciate any and all asistance
Thanks
Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies
2. UNIX for Dummies Questions & Answers
All,
I am new to unix and i have the following requirement.
I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file..
Could... (35 Replies)
Discussion started by: ksrams
35 Replies
3. Shell Programming and Scripting
Hi All,
In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'.
I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance?
... (1 Reply)
Discussion started by: HemaV
1 Replies
4. Shell Programming and Scripting
How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited
I need append 3 files as above: File1:
P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01
File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies
5. UNIX for Dummies Questions & Answers
Friends,
I need help with the following in UNIX.
Merge all csv files in one folder considering only 1 header row and ignoring header of all other files.
FYI - All files are in same format and contains same headers.
Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies
6. UNIX for Dummies Questions & Answers
:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux.
I am working in Linux terminal.
I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies
7. Shell Programming and Scripting
Hi All,
I am trying to merge all the text files into one file using below snippet
cat /home/Temp/Test/Log/*.txt >> all.txt
But it seems it is not working.
I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt
I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies
8. Shell Programming and Scripting
Hello expert friends,
I'm writing a script to capture stats using sar and stuck up at report generation.
I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies
9. Shell Programming and Scripting
Hi all,
Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),,
How can I merge them to one singe file "X" in shell scripting
Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies
10. UNIX for Beginners Questions & Answers
Hi Experts
Need help...
I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp
e.g.
Header
row1
row2
row3
Trailer
The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies
shift(1) User Commands shift(1)
NAME
shift - shell built-in function to traverse either a shell's argument list or a list of field-separated words
SYNOPSIS
sh
shift [n]
csh
shift [variable]
ksh
* shift [n]
DESCRIPTION
sh
The positional parameters from $n+1 ... are renamed $1 ... . If n is not given, it is assumed to be 1.
csh
The components of argv, or variable, if supplied, are shifted to the left, discarding the first component. It is an error for the variable
not to be set or to have a null value.
ksh
The positional parameters from $n+1 $n+1 ... are renamed $1 ..., default n is 1. The parameter n can be any arithmetic expression that
evaluates to a non-negative number less than or equal to $#.
On this man page, ksh(1) commands that are preceded by one or two * (asterisks) are treated specially in the following ways:
1. Variable assignment lists preceding the command remain in effect when the command completes.
2. I/O redirections are processed after variable assignments.
3. Errors cause a script that contains them to abort.
4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari-
able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not
performed.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
SEE ALSO
csh(1), ksh(1), sh(1), attributes(5)
SunOS 5.10 15 Apr 1994 shift(1)