Sponsored Content
Top Forums Shell Programming and Scripting Merge text files while combining the multiple header/trailer records into one each. Post 302259204 by oordonez on Monday 17th of November 2008 12:31:38 PM
Old 11-17-2008
Merge text files while combining the multiple header/trailer records into one each.

Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to reduce its run-time. This works fine, however, we're faced with multiple interface files that need to be merged prior transferring to the vendor.


Some Details on the File:
The file generated has a header and a trailer record, and the trailer record has pertinent total values (i.e., employee count, records approved, etc). There are no field separators -- these are fixed length fields.

Predicament in Detail:
We'd like to concatenate the files -- that's the easy part. What makes this difficult is that we need to eliminate the multiple header records and retain only the first one. Also, we need to eliminate the multiple trailer records, but we need to add all the value totals from each trailer into the one trailer record we'll retain at the end.

As you might have surmised by now, I've written some UNIX scripts, but lack some key knowledge related to individual record and field manipulation within a text file. In particular, I'd like to know how I can define specific fields when I read each record -- these are the fields for the trailer records I need to keep a rolling total on. Also, I'd like to know how I can delete individual records.

Any assistance will be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies

2. UNIX for Dummies Questions & Answers

Copy all the files with time stamp and remove header,trailer from file

All, I am new to unix and i have the following requirement. I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file.. Could... (35 Replies)
Discussion started by: ksrams
35 Replies

3. Shell Programming and Scripting

improve performance - replace $\| with $#@ and remove header and trailer records

Hi All, In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'. I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance? ... (1 Reply)
Discussion started by: HemaV
1 Replies

4. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

5. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

6. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

7. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

8. Shell Programming and Scripting

Need to merge multiple text files vertically and place comma between fields

Hello expert friends, I'm writing a script to capture stats using sar and stuck up at report generation. I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies

9. Shell Programming and Scripting

Merge multiple files with common header

Hi all, Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),, How can I merge them to one singe file "X" in shell scripting Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies

10. UNIX for Beginners Questions & Answers

Append pipe | at the end of all the rows except header n trailer for all the files under a directory

Hi Experts Need help... I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp e.g. Header row1 row2 row3 Trailer The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies
audit(4)						     Kernel Interfaces Manual							  audit(4)

NAME
audit - audit trail format and other information for auditing DESCRIPTION
Audit records are generated when users make security-relevant system calls, as well as by self-auditing processes that call (see aud- write(2)). Access to the auditing system is restricted to super-user. Each audit record consists of an audit record header and a record body. The record header is comprised of sequence number, process ID, event type, and record body length. The sequence number gives relative order of all records; the process ID belongs to the process being audited; the event type is a field identifying the type of audited activity; the length is the record body length expressed in bytes. The record body is the variable-length component of an audit record containing more information about the audited activity. For records generated by system calls, the body contains the time the audited event completes in either success or failure, and the parameters of the system calls; for records generated by self-auditing processes, the body consists of the time audwrite(2) writes the records and the high- level description of the event (see audwrite(2)). The records in the audit trail are compressed to save file space. When a process is audited the first time, a pid identification record (PIR) is written into the audit trail containing information that remains constant throughout the lifetime of the process. This includes the parent's process ID, audit tag, real user ID, real group ID, effective user ID, effective group ID, group ID list, effective, permit- ted, and retained privileges, compartment ID, and the terminal ID (tty). The PIR is entered only once per process per audit trail. Information accumulated in an audit trail is analyzed and displayed by (see audisp(1M)). AUTHOR
was developed by HP. SEE ALSO
audsys(1M), audevent(1M), audisp(1M), audomon(1M), audwrite(2), audit(5), compartments(5), privileges(5). audit(4)
All times are GMT -4. The time now is 06:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy