Sponsored Content
Top Forums Shell Programming and Scripting Sort log files based on numeric value in the filename Post 302747627 by Don Cragun on Saturday 22nd of December 2012 01:16:21 AM
Old 12-22-2012
Quote:
Originally Posted by alex2005
Hi,
I have a list of log files as follows:
Code:
name_date_0001_ID0.log
name_date_0001_ID2.log
name_date_0001_ID1.log
name_date_0002_ID2.log
name_date_0004_ID0.log
name_date_0005_ID0.log
 
name_date_0021_ID0.log
 
name_date_0025_ID0.log
 
.......................................
name_date_0028_ID0.log
name_date_0029_ID0.log
name_date_0030_ID0.log

I need to generate numbers from 1 to 30 and match them with their equivalent extracted from the filename (if duplicates are found, than multiply the rows with the numbers of duplicates) and then search for a pattern inside the log file.
With help from internet I managed to complete each operation, but I do not know to how to merge them in a single command.
Generate numbers from 1 to 30 using :

Code:
awk 'BEGIN { for (i = 1; i <= 30; i++) printf "%06d\n", i }'




Now I need to match

Code:
0001 - name_date_0001_ID0.log
0001 - name_date_0001_ID2.log
0002 - name_date_0001_ID2.log
0003 - 
......
0030 - name_date_0030_ID0.log




Having those numbers generated first will help me to spot if a file is missing (like name_date_0003_ID0.log)

To extract the numeric value and see how many duplicates are I have used this command:

Code:
 ls -1 |awk -F '_' '{print $3}' | awk -F '.' '{print $1}' | awk '{a[$0]++}END{for(i in a){print i, a[i]}}'
 



with the following results:

Code:
 
0001 3
0002 1
.......
0030 1

The last command that I'm using it is to search for a pattern in all the log files, print the 2nd line after matching, and count number of “%”

Code:
 
awk '/pattern/ {c[NR+2]++}  c[NR]  {n=split($0,a,"%"); printf("%-40s%2s%-80s%2s%4s\n",FILENAME,"  ",$0,"  ",n-1)}' *.log




The final result should look like:

Code:
0001 - name_date_0001_ID0.log    “2nd line after the pattern is found” “number of “%” signs” 
0001 - name_date_0001_ID2.log    “2nd line after the pattern is found” “number of “%” signs” 
0002 - name_date_0001_ID2.log    “2nd line after the pattern is found” “number of “%” signs”
0003 -    
......
0030 - name_date_0030_ID0.log    “2nd line after the pattern is found” “number of “%” signs”
 




Also if possible I would like to check for the length of ID0, ID1 .... IDn and if less then 3 characters then display a warning after “number of “%” signs”

Please help me merging all the codes to achieve the above results.
Alex
I'm not sure what you want to get out of this.

You show a printf format string of "%06d\n", but all of the data you show in your examples show a zero-filled four decimal digit string; not six decimal digits. Do you want four digits or six digits?

The last printf format that you show:
Code:
printf("%-40s%2s%-80s%2s%4s\n",FILENAME,"  ",$0,"  ",n-1)

does not match the output you say you want to see:
Code:
0001 - name_date_0001_ID0.log    “2nd line after the pattern is found” “number of “%” signs”

because the leading four decimal digit sequence and the following " - " are not in the printf format string. Do you want the output to match the format string you gave, or do you want the output to match the example shown?

You code has an underlying assumption that the pattern you're looking for will appear on the same line in every file. (The assumption is hidden by the fact that the array c is not cleared when you start reading a new file.) Do you always want to display a specific line number in each file, or do you want to display the 2nd line after a line that matches a given pattern?

If the pattern appears more than one line in a file, do you want the second line after each match and the count of %s on that line to be printed for each match, or only for the first match in each file?

Will you guarantee that each filename matched by *.log contains exactly three underscore characters? Is the error message that you want to be printed supposed to be a check that the characters between the 3rd underscore and the .log at the end of the filename is ID followed by a single decimal digit, or is it intended to verify that there are exactly three underscores, exactly four (or six) decimal digits between the 2nd and 3rd underscore and ID followed by a single decimal digit followed by .log after the 3rd underscore?

Please give us a sample of the start of the content of one of your log files (in CODE tags) and show us the pattern you want to match.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort files by Date-timestamps available in filename & pick the sortedfiles one by one

Hi, I am new to Unix shell scripting. Can you please help me with this immediate requirement to code.. The requirement is as given below. In a directory say Y, I have files like this. PP_100000_28062006_122731_746.dat PP_100000_28062006_122731_745.dat PP_100000_28062006_122734_745.dat... (4 Replies)
Discussion started by: Chindhu
4 Replies

2. UNIX for Dummies Questions & Answers

mv files based on filename

I have a directory of about 30,000 image files. The file names are all yearmonthday.jpg however some of the files have yearmonthday-snapshot.jpg i would like to move all files that contain the phrase -snapshot to their own directory. Any assistance with the proper commands would be much... (1 Reply)
Discussion started by: jdblank
1 Replies

3. UNIX for Dummies Questions & Answers

sort files by numeric filename

dear all, i have .dat files named as: 34.dat 2.dat 16.dat 107.dat i would like to sort them by their filenames as: 2.dat 16.dat 34.dat 107.dat i have tried numerous combinations of sort and ls command (in vain) to obtain : 107.dat 16.dat 2.dat 34.dat (1 Reply)
Discussion started by: chen.xiao.po
1 Replies

4. Shell Programming and Scripting

sort the files based on timestamp and execute sorted files in order

Hi I have a requirement like below I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory My files looks like this PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies

5. Shell Programming and Scripting

Sort files by date in filename

Hi, I am a newbie to shell programming and I need some help in sorting a list of files in ascending order of date in the filenames. The file format is always : IGL01_AC_D_<YYYYMMDD>_N01_01 For example, in a directory MyDirectory I have the following files: IGL01_AC_D_20110712_N01_01.dat... (11 Replies)
Discussion started by: Yuggy
11 Replies

6. Shell Programming and Scripting

URGENT!!! bash script to sort files into folder according to a string in the filename

Hi all. I am very new to linux scripting and i have a task i can only solve with a script. I need to sort files base on the date string in their filenames and create a folder using the same date string then move the files to their respective folders. Scenario: Folder Path:... (1 Reply)
Discussion started by: ace47
1 Replies

7. Shell Programming and Scripting

Bash script to sort files into folder according to a string in the filename

Hi all. I am very new to linux scripting and i have a task i can only solve with a script. I need to sort files base on the date string in their filenames and create a folder using the same date string then move the files to their respective folders. Scenario: Folder Path:... (1 Reply)
Discussion started by: ace47
1 Replies

8. Shell Programming and Scripting

Using bash to separate files files based on parts of a filename

Hey guys, Sorry for the basic question but I have a lot of files that I want to separate into groups based on filenames which I can then cat together. Eg I have: (a_b_c.txt) WB34_2_SLA8.txt WB34_1_SLA8.txt WB34_1_DB10.txt WB34_2_DB10.txt WB34_1_SLA8.txt WB34_2_SLA8.txt 77_1_SLA8.txt... (1 Reply)
Discussion started by: Breentax
1 Replies

9. Shell Programming and Scripting

Help with sort word and general numeric sort at the same time

Input file: 100%ABC2 3.44E-12 USA A2M%H02579 0E0 UK 100%ABC2 5.34E-8 UK 100%ABC2 3.25E-12 USA A2M%H02579 5E-45 UK Output file: 100%ABC2 3.44E-12 USA 100%ABC2 3.25E-12 USA 100%ABC2 5.34E-8 UK A2M%H02579 0E0 UK A2M%H02579 5E-45 UK Code try: sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies
All times are GMT -4. The time now is 10:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy