Sponsored Content
Top Forums Shell Programming and Scripting Merge text files while combining the multiple header/trailer records into one each. Post 302259204 by oordonez on Monday 17th of November 2008 12:31:38 PM
Old 11-17-2008
Merge text files while combining the multiple header/trailer records into one each.

Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to reduce its run-time. This works fine, however, we're faced with multiple interface files that need to be merged prior transferring to the vendor.


Some Details on the File:
The file generated has a header and a trailer record, and the trailer record has pertinent total values (i.e., employee count, records approved, etc). There are no field separators -- these are fixed length fields.

Predicament in Detail:
We'd like to concatenate the files -- that's the easy part. What makes this difficult is that we need to eliminate the multiple header records and retain only the first one. Also, we need to eliminate the multiple trailer records, but we need to add all the value totals from each trailer into the one trailer record we'll retain at the end.

As you might have surmised by now, I've written some UNIX scripts, but lack some key knowledge related to individual record and field manipulation within a text file. In particular, I'd like to know how I can define specific fields when I read each record -- these are the fields for the trailer records I need to keep a rolling total on. Also, I'd like to know how I can delete individual records.

Any assistance will be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies

2. UNIX for Dummies Questions & Answers

Copy all the files with time stamp and remove header,trailer from file

All, I am new to unix and i have the following requirement. I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file.. Could... (35 Replies)
Discussion started by: ksrams
35 Replies

3. Shell Programming and Scripting

improve performance - replace $\| with $#@ and remove header and trailer records

Hi All, In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'. I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance? ... (1 Reply)
Discussion started by: HemaV
1 Replies

4. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

5. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

6. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

7. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

8. Shell Programming and Scripting

Need to merge multiple text files vertically and place comma between fields

Hello expert friends, I'm writing a script to capture stats using sar and stuck up at report generation. I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies

9. Shell Programming and Scripting

Merge multiple files with common header

Hi all, Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),, How can I merge them to one singe file "X" in shell scripting Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies

10. UNIX for Beginners Questions & Answers

Append pipe | at the end of all the rows except header n trailer for all the files under a directory

Hi Experts Need help... I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp e.g. Header row1 row2 row3 Trailer The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies
audit(2)																  audit(2)

NAME
audit - write a record to the audit log SYNOPSIS
cc [ flag ... ] file ... -lbsm -lsocket -lnsl [ library... ] #include <sys/param.h> #include <bsm/libbsm.h> int audit(caddr_t record, int length); The audit() function is used to write a record to the system audit log. The data pointed to by record is written to the log after a mini- mal consistency check, with the length parameter specifying the size of the record in bytes. The data should be a well-formed audit record as described by audit.log(4). The kernel validates the record header token type and length, and sets the time stamp value before writing the record to the audit log. The kernel does not do any preselection for user-level generated events. If the audit policy is set to include sequence or trailer tokens, the kernel will append them to the record. Upon successful completion, 0 is returned. Otherwise, -1 is returned and errno is set to indicate the error. The audit() function will fail if: EFAULT The record argument points outside the process's allocated address space. EINVAL The record header token ID is invalid or the length is either less than the header token size or greater than MAXAUDIT- DATA. EPERM The {PRIV_PROC_AUDIT} privilege is not asserted in the effective set of the calling process. USAGE
Only privileged processes can successfully execute this call. See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Interface Stability |Stable | +-----------------------------+-----------------------------+ |MT-Level |MT-Safe | +-----------------------------+-----------------------------+ bsmconv(1M), auditd(1M), auditon(2), auditsvc(2), getaudit(2), audit.log(4), attributes(5), privileges(5) The functionality described in this man page is available only if the Basic Security Module (BSM) has been enabled. See bsmconv(1M) for more information. 31 Mar 2005 audit(2)
All times are GMT -4. The time now is 01:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy