Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merging files with AWK filtering and counting lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging files with AWK filtering and counting lines
# 1  
Old 01-28-2008
Merging files with AWK filtering and counting lines
Hi there,

I have a couple of files I need to merge. I can do a simple merge by concatenating them into one larger file.

But then I need to filter the file to get a desired result.

The output looks like this:
Code:
TRNH 0000000010941
ORDH
OADR
OADR
ORDL
ENDT    1116399         000000003 000000001
TRLR 0000000010941 000000003 000000001
TRNH 0000000010942
ORDH
OADR
OADR
ORDL
ENDT    1116400         000000003 000000001
TRLR 0000000010942 000000003 000000001
TRNH 0000000010943
ORDH
OADR
OMSG
ORDL
ORDL
ENDT    1116399         000000004 000000001
TRLR 0000000010943 000000003 000000001
TRNH 0000000010944
ORDH
OADR
OADR
ORDL
ENDT    1116400         000000003 000000001
ORDH                                       
OADR                                       
OADR
ORDL                                       
ORDL                                       
ENDT    1116400         000000004 000000001
TRLR 0000000010944 000000007 000000002

and the filtering should leave the first and the last line with the TRNH and TRLR (giving the last TRLR the same sequence as the first TRNH). The rest of the TRNH and TRLR lines need to be omitted.

Then the final TRLR should represent the amount of ORDH lines and the amount of OADR, OMSG and ORDL lines.

I haven't gotten to deleting the extra TRNH and TRLR lines yet; this is the filter I have so far:

Code:
BEGIN {
#	 define two counters 
	ordh_cnt = 0;
	ordl_total_cnt = 0;
}

#  Start filter 

#  if line start with ORDH add 1 to counters 
$1 == "ORDH" {
	ordh_cnt++;
}

#  if line starts with TRLR, adjust line to reflect new count of ORDH in order
$1 == "TRLR" {
	printf "%s%9.9d%s\n", substr($0, 0, 31), ordh_cnt, substr($0, 39);
#	 line has been printed, next rule 
	next;
}

#  if line start with ORDL add 1 to counters 
$1 == "ORDL" {
	ordl_total_cnt++;
}

#  if line start with OADR add 1 to counters 
$1 == "OADR" {
	ordl_total_cnt++;
}

#  if line start with OMSG add 1 to counters 
$1 == "OMSG" {
	ordl_total_cnt++;
}

#  if line starts with TRLR, adjust line to reflect new total ORDL, OADR and OMSG in complete file 
$1 == "TRLR" {
	printf "%s%9.9d%s\n", substr($0, 0, 19), ordl_total_cnt, substr($0, 29);
#	 line has been printed, next rule 
	next;
}

#  Line has not changed, print normal line 
{
	print $0;
}

Now the amount of ORDH lines is output to my new file, so that seems to work. Yet the amount of OADR, OMSG and ORDL lines isn't corrected in the output.

The final result should look like this:

Code:
TRNH 0000000010941
ORDH
OADR
OADR
ORDL
ENDT    1116399         000000003 000000001
ORDH
OADR
OADR
ORDL
ENDT    1116400         000000003 000000001
ORDH
OADR
OMSG
ORDL
ORDL
ENDT    1116399         000000004 000000001
ORDH
OADR
OADR
ORDL
ENDT    1116400         000000003 000000001
ORDH                                       
OADR                                       
OADR
ORDL                                       
ORDL                                       
ENDT    1116400         000000004 000000001
TRLR 0000000010941 000000017 000000005

Any help would be greatly appreciated Smilie
# 2  
Old 01-28-2008
Code:
awk 'NR == 1 { trnh = $2; print }
!/^TR(NH|LR)/ { 
  if ($1 == "ORDH") 
    ordh ++ 
  if ($1 ~ /^O(ADR|MSG|RDL)/) 
    ordl ++
  print 
} END {
printf "TRLR %s %.9d %.9d\n", trnh, ordl, ordh
}' filename

# 3  
Old 01-28-2008
Thank you very much, works like a charm! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging two lines into one (awk)

Hi, I am attempting to merge the following lines which run over two lines using awk. INITIAL OUTPUT 2019 Sep 28 10:47:24.695 hkaet9612 last message repeated 1 time 2019 Sep 28 10:47:24.695 hkaet9612 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interfa ce Ethernet1/45 is down (Interface removed)... (10 Replies)
Discussion started by: sand1234
10 Replies

2. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

3. Shell Programming and Scripting

Counting lines in a file using awk

I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it. Does anyone know of a way? Thanks a lot. (7 Replies)
Discussion started by: guitarist684
7 Replies

4. Shell Programming and Scripting

Awk/sed : help on:Filtering multiple lines to one:

Experts Good day, I want to filter multiple lines of same error of same day , to only 1 error of each day, the first line from the log. Here is the file: May 26 11:29:19 cmihpx02 vmunix: NFS write failed for server cmiauxe1: error 5 (RPC: Timed out) May 26 11:29:19 cmihpx02 vmunix: NFS... (4 Replies)
Discussion started by: rveri
4 Replies

5. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

6. Shell Programming and Scripting

Counting lines of code in a directory with awk

I've never toyed with awk, but it seems every time I present an elegant 2- to 8-line script, someone comes back with an awk 1-liner. I just came up with this to count all the lines of source code in a directory. How would I do it in awk? LINES=0 for n in $(wc -l *.cpp *.h | cut -b-7); do ... (2 Replies)
Discussion started by: KenJackson
2 Replies

7. UNIX for Dummies Questions & Answers

Awk counting lines with field match

Hi, Im trying to create a script that reads throught every line in a file and then counts how many lines there with a certain field that matches a input, and also ausing another awk it has to do the same as the above but to then use sort anduniq to get rid of all the unique lines with another... (8 Replies)
Discussion started by: fredted40x
8 Replies

8. Shell Programming and Scripting

awk - Counting number of similar lines

Hi All I have the input file OMAK_11. OMAK 000002EXCLUDE 1341 OMAK 000002EXCLUDE 1341 OMAK 000002EXCLUDE 1341 OMAK 000003EXCLUDE 1341 OMAK 000003EXCLUDE 1341 OMAK 000003EXCLUDE ... (8 Replies)
Discussion started by: dhanamurthy
8 Replies

9. Shell Programming and Scripting

Merging lines using AWK

Hi, Anybody help on this. :( I want to merge the line with previous line, if the line starts with 7. Otherwise No change in the line. Example file aa.txt is like below 122122 222222 333333 734834 702923 389898 790909 712345 999999 My output should be written in another file... (6 Replies)
Discussion started by: senthil_is
6 Replies

10. UNIX for Dummies Questions & Answers

Counting lines and files

Hi experts, First of all thanks for all your help. How can i count the lines within a text file and send this number to another text file? And by the way how can i count the number of files inside a tape ("/dev/rtp") that as one pattern (Ex. "/CTA/") and send this number to a text file? I... (6 Replies)
Discussion started by: jorge.ferreira
6 Replies
Login or Register to Ask a Question