Sponsored Content
Top Forums Shell Programming and Scripting Merge text files while combining the multiple header/trailer records into one each. Post 302259204 by oordonez on Monday 17th of November 2008 12:31:38 PM
Old 11-17-2008
Merge text files while combining the multiple header/trailer records into one each.

Situation:
Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to reduce its run-time. This works fine, however, we're faced with multiple interface files that need to be merged prior transferring to the vendor.


Some Details on the File:
The file generated has a header and a trailer record, and the trailer record has pertinent total values (i.e., employee count, records approved, etc). There are no field separators -- these are fixed length fields.

Predicament in Detail:
We'd like to concatenate the files -- that's the easy part. What makes this difficult is that we need to eliminate the multiple header records and retain only the first one. Also, we need to eliminate the multiple trailer records, but we need to add all the value totals from each trailer into the one trailer record we'll retain at the end.

As you might have surmised by now, I've written some UNIX scripts, but lack some key knowledge related to individual record and field manipulation within a text file. In particular, I'd like to know how I can define specific fields when I read each record -- these are the fields for the trailer records I need to keep a rolling total on. Also, I'd like to know how I can delete individual records.

Any assistance will be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies

2. UNIX for Dummies Questions & Answers

Copy all the files with time stamp and remove header,trailer from file

All, I am new to unix and i have the following requirement. I have file(s) landing into input directory with timestamp, first i want to copy all these files into seperate directory then i want to rename these files without timestamp and also remove header,trailer from that file.. Could... (35 Replies)
Discussion started by: ksrams
35 Replies

3. Shell Programming and Scripting

improve performance - replace $\| with $#@ and remove header and trailer records

Hi All, In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'. I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance? ... (1 Reply)
Discussion started by: HemaV
1 Replies

4. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

5. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

6. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

7. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

8. Shell Programming and Scripting

Need to merge multiple text files vertically and place comma between fields

Hello expert friends, I'm writing a script to capture stats using sar and stuck up at report generation. I have around 10 files in a directory and need to merge them all vertically based on the time value of first column (output file should have only one time value) and insert comma after... (6 Replies)
Discussion started by: prvnrk
6 Replies

9. Shell Programming and Scripting

Merge multiple files with common header

Hi all, Say i have multiple files x1 x2 x3 x4, all with common header (date, time, year, age),, How can I merge them to one singe file "X" in shell scripting Thanks for your suggestions. (2 Replies)
Discussion started by: msarguru
2 Replies

10. UNIX for Beginners Questions & Answers

Append pipe | at the end of all the rows except header n trailer for all the files under a directory

Hi Experts Need help... I am looking for a Unix script to append pipe | at the end of all the rows (except header and trailer)in all the files placed under the directory /interfaces/Temp e.g. Header row1 row2 row3 Trailer The script should read all the files under... (3 Replies)
Discussion started by: phani333
3 Replies
GIT-INTERPRET-TRAI(1)                                               Git Manual                                               GIT-INTERPRET-TRAI(1)

NAME
git-interpret-trailers - add or parse structured information in commit messages SYNOPSIS
git interpret-trailers [options] [(--trailer <token>[(=|:)<value>])...] [<file>...] git interpret-trailers [options] [--parse] [<file>...] DESCRIPTION
Help parsing or adding trailers lines, that look similar to RFC 822 e-mail headers, at the end of the otherwise free-form part of a commit message. This command reads some patches or commit messages from either the <file> arguments or the standard input if no <file> is specified. If --parse is specified, the output consists of the parsed trailers. Otherwise, this command applies the arguments passed using the --trailer option, if any, to the commit message part of each input file. The result is emitted on the standard output. Some configuration variables control the way the --trailer arguments are applied to each commit message and the way any existing trailer in the commit message is changed. They also make it possible to automatically add some trailers. By default, a <token>=<value> or <token>:<value> argument given using --trailer will be appended after the existing trailers only if the last trailer has a different (<token>, <value>) pair (or if there is no existing trailer). The <token> and <value> parts will be trimmed to remove starting and trailing whitespace, and the resulting trimmed <token> and <value> will appear in the message like this: token: value This means that the trimmed <token> and <value> will be separated by ': ' (one colon followed by one space). By default the new trailer will appear at the end of all the existing trailers. If there is no existing trailer, the new trailer will appear after the commit message part of the output, and, if there is no line with only spaces at the end of the commit message part, one blank line will be added before the new trailer. Existing trailers are extracted from the input message by looking for a group of one or more lines that (i) is all trailers, or (ii) contains at least one Git-generated or user-configured trailer and consists of at least 25% trailers. The group must be preceded by one or more empty (or whitespace-only) lines. The group must either be at the end of the message or be the last non-whitespace lines before a line that starts with ---. Such three minus signs start the patch part of the message. When reading trailers, there can be whitespaces after the token, the separator and the value. There can also be whitespaces inside the token and the value. The value may be split over multiple lines with each subsequent line starting with whitespace, like the "folding" in RFC 822. Note that trailers do not follow and are not intended to follow many rules for RFC 822 headers. For example they do not follow the encoding rules and probably many other rules. OPTIONS
--in-place Edit the files in place. --trim-empty If the <value> part of any trailer contains only whitespace, the whole trailer will be removed from the resulting message. This applies to existing trailers as well as new trailers. --trailer <token>[(=|:)<value>] Specify a (<token>, <value>) pair that should be applied as a trailer to the input messages. See the description of this command. --where <placement>, --no-where Specify where all new trailers will be added. A setting provided with --where overrides all configuration variables and applies to all --trailer options until the next occurrence of --where or --no-where. --if-exists <action>, --no-if-exists Specify what action will be performed when there is already at least one trailer with the same <token> in the message. A setting provided with --if-exists overrides all configuration variables and applies to all --trailer options until the next occurrence of --if-exists or --no-if-exists. --if-missing <action>, --no-if-missing Specify what action will be performed when there is no other trailer with the same <token> in the message. A setting provided with --if-missing overrides all configuration variables and applies to all --trailer options until the next occurrence of --if-missing or --no-if-missing. --only-trailers Output only the trailers, not any other parts of the input. --only-input Output only trailers that exist in the input; do not add any from the command-line or by following configured trailer.* rules. --unfold Remove any whitespace-continuation in trailers, so that each trailer appears on a line by itself with its full content. --parse A convenience alias for --only-trailers --only-input --unfold. CONFIGURATION VARIABLES
trailer.separators This option tells which characters are recognized as trailer separators. By default only : is recognized as a trailer separator, except that = is always accepted on the command line for compatibility with other git commands. The first character given by this option will be the default character used when another separator is not specified in the config for this trailer. For example, if the value for this option is "%=$", then only lines using the format <token><sep><value> with <sep> containing %, = or $ and then spaces will be considered trailers. And % will be the default separator used, so by default trailers will appear like: <token>% <value> (one percent sign and one space will appear between the token and the value). trailer.where This option tells where a new trailer will be added. This can be end, which is the default, start, after or before. If it is end, then each new trailer will appear at the end of the existing trailers. If it is start, then each new trailer will appear at the start, instead of the end, of the existing trailers. If it is after, then each new trailer will appear just after the last trailer with the same <token>. If it is before, then each new trailer will appear just before the first trailer with the same <token>. trailer.ifexists This option makes it possible to choose what action will be performed when there is already at least one trailer with the same <token> in the message. The valid values for this option are: addIfDifferentNeighbor (this is the default), addIfDifferent, add, replace or doNothing. With addIfDifferentNeighbor, a new trailer will be added only if no trailer with the same (<token>, <value>) pair is above or below the line where the new trailer will be added. With addIfDifferent, a new trailer will be added only if no trailer with the same (<token>, <value>) pair is already in the message. With add, a new trailer will be added, even if some trailers with the same (<token>, <value>) pair are already in the message. With replace, an existing trailer with the same <token> will be deleted and the new trailer will be added. The deleted trailer will be the closest one (with the same <token>) to the place where the new one will be added. With doNothing, nothing will be done; that is no new trailer will be added if there is already one with the same <token> in the message. trailer.ifmissing This option makes it possible to choose what action will be performed when there is not yet any trailer with the same <token> in the message. The valid values for this option are: add (this is the default) and doNothing. With add, a new trailer will be added. With doNothing, nothing will be done. trailer.<token>.key This key will be used instead of <token> in the trailer. At the end of this key, a separator can appear and then some space characters. By default the only valid separator is :, but this can be changed using the trailer.separators config variable. If there is a separator, then the key will be used instead of both the <token> and the default separator when adding the trailer. trailer.<token>.where This option takes the same values as the trailer.where configuration variable and it overrides what is specified by that option for trailers with the specified <token>. trailer.<token>.ifexists This option takes the same values as the trailer.ifexists configuration variable and it overrides what is specified by that option for trailers with the specified <token>. trailer.<token>.ifmissing This option takes the same values as the trailer.ifmissing configuration variable and it overrides what is specified by that option for trailers with the specified <token>. trailer.<token>.command This option can be used to specify a shell command that will be called to automatically add or modify a trailer with the specified <token>. When this option is specified, the behavior is as if a special <token>=<value> argument were added at the beginning of the command line, where <value> is taken to be the standard output of the specified command with any leading and trailing whitespace trimmed off. If the command contains the $ARG string, this string will be replaced with the <value> part of an existing trailer with the same <token>, if any, before the command is launched. If some <token>=<value> arguments are also passed on the command line, when a trailer.<token>.command is configured, the command will also be executed for each of these arguments. And the <value> part of these arguments, if any, will be used to replace the $ARG string in the command. EXAMPLES
o Configure a sign trailer with a Signed-off-by key, and then add two of these trailers to a message: $ git config trailer.sign.key "Signed-off-by" $ cat msg.txt subject message $ cat msg.txt | git interpret-trailers --trailer 'sign: Alice <alice@example.com>' --trailer 'sign: Bob <bob@example.com>' subject message Signed-off-by: Alice <alice@example.com> Signed-off-by: Bob <bob@example.com> o Use the --in-place option to edit a message file in place: $ cat msg.txt subject message Signed-off-by: Bob <bob@example.com> $ git interpret-trailers --trailer 'Acked-by: Alice <alice@example.com>' --in-place msg.txt $ cat msg.txt subject message Signed-off-by: Bob <bob@example.com> Acked-by: Alice <alice@example.com> o Extract the last commit as a patch, and add a Cc and a Reviewed-by trailer to it: $ git format-patch -1 0001-foo.patch $ git interpret-trailers --trailer 'Cc: Alice <alice@example.com>' --trailer 'Reviewed-by: Bob <bob@example.com>' 0001-foo.patch >0001-bar.patch o Configure a sign trailer with a command to automatically add a 'Signed-off-by: ' with the author information only if there is no 'Signed-off-by: ' already, and show how it works: $ git config trailer.sign.key "Signed-off-by: " $ git config trailer.sign.ifmissing add $ git config trailer.sign.ifexists doNothing $ git config trailer.sign.command 'echo "$(git config user.name) <$(git config user.email)>"' $ git interpret-trailers <<EOF > EOF Signed-off-by: Bob <bob@example.com> $ git interpret-trailers <<EOF > Signed-off-by: Alice <alice@example.com> > EOF Signed-off-by: Alice <alice@example.com> o Configure a fix trailer with a key that contains a # and no space after this character, and show how it works: $ git config trailer.separators ":#" $ git config trailer.fix.key "Fix #" $ echo "subject" | git interpret-trailers --trailer fix=42 subject Fix #42 o Configure a see trailer with a command to show the subject of a commit that is related, and show how it works: $ git config trailer.see.key "See-also: " $ git config trailer.see.ifExists "replace" $ git config trailer.see.ifMissing "doNothing" $ git config trailer.see.command "git log -1 --oneline --format="%h (%s)" --abbrev-commit --abbrev=14 $ARG" $ git interpret-trailers <<EOF > subject > > message > > see: HEAD~2 > EOF subject message See-also: fe3187489d69c4 (subject of related commit) o Configure a commit template with some trailers with empty values (using sed to show and keep the trailing spaces at the end of the trailers), then configure a commit-msg hook that uses git interpret-trailers to remove trailers with empty values and to add a git-version trailer: $ sed -e 's/ Z$/ /' >commit_template.txt <<EOF > ***subject*** > > ***message*** > > Fixes: Z > Cc: Z > Reviewed-by: Z > Signed-off-by: Z > EOF $ git config commit.template commit_template.txt $ cat >.git/hooks/commit-msg <<EOF > #!/bin/sh > git interpret-trailers --trim-empty --trailer "git-version: $(git describe)" "$1" > "$1.new" > mv "$1.new" "$1" > EOF $ chmod +x .git/hooks/commit-msg SEE ALSO
git-commit(1), git-format-patch(1), git-config(1) GIT
Part of the git(1) suite Git 2.17.1 10/05/2018 GIT-INTERPRET-TRAI(1)
All times are GMT -4. The time now is 06:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy