Sponsored Content
Top Forums Shell Programming and Scripting Sort log files based on numeric value in the filename Post 302747807 by Don Cragun on Sunday 23rd of December 2012 02:18:59 AM
Old 12-23-2012
If I understood your requirements correctly, I believe the following script does everything you want. I tried it with a bunch of log files containing zero to four lines matching your pattern, with filenames matching and not matching your naming conventions. For files that have more than one line matching your pattern, it reports the contents of the line two lines after the last line matching your pattern:
Code:
#!/bin/ksh
awk -F% '
# Variables used:
# d1:   Diagnostic message 1 (! 3 _ in lf || np[4] fails to match ID[0-9].log)
# d2:   Diagnostic message 2 (np[3] > 0030 || np[3] not four decimal digits)
# d3:   Diagnostic message 3 (pattern not matched or matched more than once)
# fl:   Found Line (Contents of line 2 lines after last line matching
#       pattern.)
# flpc: Found Line % Count
# lf:   Last FILENAME (We cannot know that we have the last match in a
#       file until we hit the 1st line in the next file.  We save the
#       last value of FILENAME until we can print the results for this
#       file.)
# ln:   Line Number of line 2 lines after last line matching pattern.
# mc:   Match Count (Number of times we have found a line 2 lines after
#       a line matching pattern.)
# np[]: lf split by _ (There should always be 4 elements in this array.)
# ps:   Printable seq (np[3] if np[3] is four decimal digits; otherwise
#       "????".)
# seq:  Sequence number (Used to print lines noting missing sequence
#       numbers in the values of np[3] while 1 <= seq <= 30.)
function pr() {
# Usage: pr()
#    Verify the last FILENAME, print missing sequence numbers, and print
#    results of processing the last file.
        # Skip 1st call when we have not processed a file yet...
        if(lf == "") return
        # Verify filename format...
        if(split(lf, np, /_/) != 4)
                d1 = "  Filename should contain 3 underscores."
        else if(match(np[4], /^ID[0-9].log$/) != 1)
                d1 = "  " np[4] " does not match \"IDx.log\"."
        ps = (match(np[3], /^[0-9][0-9][0-9][0-9]$/) != 1) ? "????" : np[3]
        if(RSTART == 1) {
                # Print headers sequence numbers with no matching log files...`
                while(seq < np[3] && seq <= 30) printf("%04d -\n", seq++)
                seq = np[3] + 1
                if(np[3] > 30) d2 = "  " np[3] " out of range."
        } else  d2 = "  " np[3] " not four decimal digits."
        if(mc != 1) d3 = "  Pattern matched " mc " times."
        printf("%s - %-40s  %-80s  %4d%s%s%s\n", ps, lf, fl, flpc, d1, d2, d3)
        # Clear diagnostic messages, found line, and % count...
        d1 = d2 = d3 = fl = ""
        flpc = 0
}
BEGIN { seq = 1 }       # Set starting sequence #.
FNR == 1 {              # We have the first line of a new file.
        pr()            # Print results from previous file.
        lf = FILENAME   # Save FILENAME of current file.
        mc = 0          # Clear match count for this file.
}
/Percentage Completed/ {# We have a match.
        ln = FNR + 2    # Set line # for 2 lines later.
}
FNR == ln {             # We have a line 2 lines after a match.
        mc++            # Increment match count.
        fl = $0         # Save the line.
        flpc = NF - 1   # Save the number of % characters on this line.
        ln = 0          # Clear the line number.
}
END {   pr()            # Print results from last file processed.
}' $(ls *.log | sort -t_ -k3n -k4)      # Sort the log files to be processed.

On Solaris systems, use /usr/xpg4/bin/awk or nawk instead of awk.

Last edited by Don Cragun; 12-23-2012 at 03:22 AM.. Reason: Warn about which awk to use on Solaris systems.
This User Gave Thanks to Don Cragun For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort files by Date-timestamps available in filename & pick the sortedfiles one by one

Hi, I am new to Unix shell scripting. Can you please help me with this immediate requirement to code.. The requirement is as given below. In a directory say Y, I have files like this. PP_100000_28062006_122731_746.dat PP_100000_28062006_122731_745.dat PP_100000_28062006_122734_745.dat... (4 Replies)
Discussion started by: Chindhu
4 Replies

2. UNIX for Dummies Questions & Answers

mv files based on filename

I have a directory of about 30,000 image files. The file names are all yearmonthday.jpg however some of the files have yearmonthday-snapshot.jpg i would like to move all files that contain the phrase -snapshot to their own directory. Any assistance with the proper commands would be much... (1 Reply)
Discussion started by: jdblank
1 Replies

3. UNIX for Dummies Questions & Answers

sort files by numeric filename

dear all, i have .dat files named as: 34.dat 2.dat 16.dat 107.dat i would like to sort them by their filenames as: 2.dat 16.dat 34.dat 107.dat i have tried numerous combinations of sort and ls command (in vain) to obtain : 107.dat 16.dat 2.dat 34.dat (1 Reply)
Discussion started by: chen.xiao.po
1 Replies

4. Shell Programming and Scripting

sort the files based on timestamp and execute sorted files in order

Hi I have a requirement like below I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory My files looks like this PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies

5. Shell Programming and Scripting

Sort files by date in filename

Hi, I am a newbie to shell programming and I need some help in sorting a list of files in ascending order of date in the filenames. The file format is always : IGL01_AC_D_<YYYYMMDD>_N01_01 For example, in a directory MyDirectory I have the following files: IGL01_AC_D_20110712_N01_01.dat... (11 Replies)
Discussion started by: Yuggy
11 Replies

6. Shell Programming and Scripting

URGENT!!! bash script to sort files into folder according to a string in the filename

Hi all. I am very new to linux scripting and i have a task i can only solve with a script. I need to sort files base on the date string in their filenames and create a folder using the same date string then move the files to their respective folders. Scenario: Folder Path:... (1 Reply)
Discussion started by: ace47
1 Replies

7. Shell Programming and Scripting

Bash script to sort files into folder according to a string in the filename

Hi all. I am very new to linux scripting and i have a task i can only solve with a script. I need to sort files base on the date string in their filenames and create a folder using the same date string then move the files to their respective folders. Scenario: Folder Path:... (1 Reply)
Discussion started by: ace47
1 Replies

8. Shell Programming and Scripting

Using bash to separate files files based on parts of a filename

Hey guys, Sorry for the basic question but I have a lot of files that I want to separate into groups based on filenames which I can then cat together. Eg I have: (a_b_c.txt) WB34_2_SLA8.txt WB34_1_SLA8.txt WB34_1_DB10.txt WB34_2_DB10.txt WB34_1_SLA8.txt WB34_2_SLA8.txt 77_1_SLA8.txt... (1 Reply)
Discussion started by: Breentax
1 Replies

9. Shell Programming and Scripting

Help with sort word and general numeric sort at the same time

Input file: 100%ABC2 3.44E-12 USA A2M%H02579 0E0 UK 100%ABC2 5.34E-8 UK 100%ABC2 3.25E-12 USA A2M%H02579 5E-45 UK Output file: 100%ABC2 3.44E-12 USA 100%ABC2 3.25E-12 USA 100%ABC2 5.34E-8 UK A2M%H02579 0E0 UK A2M%H02579 5E-45 UK Code try: sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies
All times are GMT -4. The time now is 10:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy