11-17-2008
Merge text files while combining the multiple header/trailer records into one each.
Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to reduce its run-time. This works fine, however, we're faced with multiple interface files that need to be merged prior transferring to the vendor.
Some Details on the File:
The file generated has a header and a trailer record, and the trailer record has pertinent total values (i.e., employee count, records approved, etc). There are no field separators -- these are fixed length fields.
Predicament in Detail:
We'd like to concatenate the files -- that's the easy part. What makes this difficult is that we need to eliminate the multiple header records and retain only the first one. Also, we need to eliminate the multiple trailer records, but we need to add all the value totals from each trailer into the one trailer record we'll retain at the end.
As you might have surmised by now, I've written some UNIX scripts, but lack some key knowledge related to individual record and field manipulation within a text file. In particular, I'd like to know how I can define specific fields when I read each record -- these are the fields for the trailer records I need to keep a rolling total on. Also, I'd like to know how I can delete individual records.
Any assistance will be greatly appreciated.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a flat file and need to count no of records in the file less the header and the trailer record.
I would appreciate any and all asistance
Thanks
Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies
2. UNIX for Dummies Questions & Answers
All,
I am new to unix and i have the following requirement.
I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file..
Could... (35 Replies)
Discussion started by: ksrams
35 Replies
3. Shell Programming and Scripting
Hi All,
In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'.
I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance?
... (1 Reply)
Discussion started by: HemaV
1 Replies
4. Shell Programming and Scripting
How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited
I need append 3 files as above: File1:
P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01
File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies
5. UNIX for Dummies Questions & Answers
Friends,
I need help with the following in UNIX.
Merge all csv files in one folder considering only 1 header row and ignoring header of all other files.
FYI - All files are in same format and contains same headers.
Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies
6. UNIX for Dummies Questions & Answers
:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux.
I am working in Linux terminal.
I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies
7. Shell Programming and Scripting
Hi All,
I am trying to merge all the text files into one file using below snippet
cat /home/Temp/Test/Log/*.txt >> all.txt
But it seems it is not working.
I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt
I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies
8. Shell Programming and Scripting
Hello expert friends,
I'm writing a script to capture stats using sar and stuck up at report generation.
I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies
9. Shell Programming and Scripting
Hi all,
Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),,
How can I merge them to one singe file "X" in shell scripting
Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies
10. UNIX for Beginners Questions & Answers
Hi Experts
Need help...
I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp
e.g.
Header
row1
row2
row3
Trailer
The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies
audit(4) Kernel Interfaces Manual audit(4)
NAME
audit - audit trail format and other information for auditing
DESCRIPTION
Audit records are generated when users make security-relevant system calls, as well as by self-auditing processes that call (see aud-
write(2)). Access to the auditing system is restricted to super-user.
Each audit record consists of an audit record header and a record body. The record header is comprised of sequence number, process ID,
event type, and record body length. The sequence number gives relative order of all records; the process ID belongs to the process being
audited; the event type is a field identifying the type of audited activity; the length is the record body length expressed in bytes.
The record body is the variable-length component of an audit record containing more information about the audited activity. For records
generated by system calls, the body contains the time the audited event completes in either success or failure, and the parameters of the
system calls; for records generated by self-auditing processes, the body consists of the time audwrite(2) writes the records and the high-
level description of the event (see audwrite(2)).
The records in the audit trail are compressed to save file space. When a process is audited the first time, a pid identification record
(PIR) is written into the audit trail containing information that remains constant throughout the lifetime of the process. This includes
the parent's process ID, audit tag, real user ID, real group ID, effective user ID, effective group ID, group ID list, effective, permit-
ted, and retained privileges, compartment ID, and the terminal ID (tty). The PIR is entered only once per process per audit trail.
Information accumulated in an audit trail is analyzed and displayed by (see audisp(1M)).
AUTHOR
was developed by HP.
SEE ALSO
audsys(1M), audevent(1M), audisp(1M), audomon(1M), audwrite(2), audit(5), compartments(5), privileges(5).
audit(4)