Processing of log files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Processing of log files
# 1  
Old 07-11-2015
Processing of log files

Hi,

I have typical logs file something of these formats -

Code:
fn2013.12.13.log
fn2013.12.13_a.log
fn2013.12.13_b.log

suffix part is after the underscore ( ie a.log or b.log )

I need to process the files in ascending date order, but descending suffix order, and check if the file exist and is greater than zero, and on failure an appropriate message should be displayed like "file is of zero size".

Coming from just a normal Unix knowledge background, it is difficult for me to process the file based on the sorting logic as described above. Could someone help me on this. ?

I will really appreciate if someone could help me this or guide me through.

Thanks.
Moderator's Comments:
Mod Comment Please use CODE tags for all sample input, sample output, and code segments.

Last edited by Don Cragun; 07-11-2015 at 10:29 PM.. Reason: Add CODE AND ICODE tags.
# 2  
Old 07-11-2015
How would above three sample files be sorted? Where would the no-suffix file fn2013.12.13.log end up?

Last edited by RudiC; 07-11-2015 at 05:15 PM..
# 3  
Old 07-11-2015
Thanks for the reply!

Well if it does have the suffix, it will be just sorted out of date - the first parameter. I need to pull date (2013.12.13) and suffix (a.log or b.log) -- if present, from the file names.

Last edited by Don Cragun; 07-11-2015 at 10:30 PM.. Reason: Add ICODE tags.
# 4  
Old 07-11-2015
That doesn't answer the question... If you have the following files:
Code:
fn2013.12.13.log
fn2013.12.13_a.log
fn2013.12.13_b.log
fn2013.12.14.log
fn2013.12.14_a.log
fn2013.12.15.log
fn2013.12.16.log
fn2013.12.17.log
fn2013.12.17_a.log
fn2013.12.17_b.log
fn2013.12.17_c.log

exactly what are you trying to do with these files? And, exactly what output(s) are you trying to produce with this set of files?
# 5  
Old 07-12-2015
Hi Don,

Basically, the requirement is to process the files in ascending date
Code:
2013.12.13

order, but descending suffix order
Code:
a.log

, and for every date and suffix, we need to check if the file exists and is greater than 0 size. Also, on failure, we need to output an appropriate message.

So far, I have tried this, but does give me expected sorted result, could someone point where I am doing the mistake :

Code:
for file in `cat file.txt | sort -k1,1nr -k2,2 -t _`; do echo $file; done

# Here is the file.txt :

Code:
ski2015.12.15.12.30.23.log
ski2015.12.15.12.30.24_a.log
ski2015.12.15.12.30.25_b.doc
ski2015.12.15.12.30.25_b.doc
ski2015.12.15.12.30.25_b.log
ski2015.12.15.12.30.25_b.log
ski2015.12.15.12.30.25_b.log
ski2015.12.14.12.30.24_a.log
ski2015.12.14.12.30.25_b.doc
ski2015.12.14.12.30.25_b.doc
ski2015.12.14.12.30.13_b.log
ski2015.12.14.12.29.25_b.log
ski2015.12.14.12.30.25_b.log
ski2015.12.14.12.30.23.log
ski2015.12.14.12.29.25_a.log
ski2015.12.15.12.30.23_a.log
ski2015.12.14.12.29.25_b.log
ski2015.12.14.12.29.25_c.log


Thanks.

Last edited by scott_apc; 07-12-2015 at 01:34 AM..
# 6  
Old 07-12-2015
This thread is titled "Processing of log files".

I don't want "Basically, the requirement is to process...". I gave you an explicit list of .log files in post #4 and asked what output you want when given that list of .log files.

Instead of answering that question you showed us an unsorted list of .log and .doc files (with some of those of files included in the list twice). Are .doc files now considered log files?

How did you get a list of files with duplicate names in that list? If you ran a find command to produce a combined list of files from multiple directories and stripped off the directory names, how do you expect a script to be able to guess at which directory a file came from to determine its size?

You have said you want to process the files with a suffix a.log and b.log and you imply that a file named ski2015.12.15.12.30.23.log doesn't have a suffix because there is no _ in that name. Please take the list I gave you or the list you gave us in post #5 and show us the EXACT output that you want to produce from that list (and explain why some files, if there are any, are not included in the output).

It is easy to see that a sort that sorts a list in reverse numeric order (when the field being sorted is not numeric) and for matching fields on that key sorts in increasing alphanumeric order on a second key doesn't give you a list in the order you have described. What is not easy is trying to guess what files are supposed to be excluded from your output and the order you do want for files that are not excluded if some of those files do not contain an _ character.

Why do you need a list of "log" files other than that produced by letting the shell expand *.log as a filename matching pattern in the directory where these files are located? Or, if you only want files with a suffix (as you have described it), why not just use *_*.log to get the list?
This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 07-12-2015
The structure of the entries in your file.txt doesn't fit those in post#1, and it doesn't represent dates. It vaguely reminds me of date/time entries; should the time be considered in the sort?

Please be way more specific and precise; post a sample output that you need!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Processing multiple files

Hello I have a program cfxfrwb which is designed to remove headers from reports files. The cfxfrwb is located in the following directory /u01/efin/v40/live/bin I run the program against a single report file in the temp directory and it does it's job../cfxfrwb... (2 Replies)
Discussion started by: captainrhodes
2 Replies

2. Shell Programming and Scripting

script: processing files

Hi All, I have a log file which is attached here: ACCTXN.ALLOC, ACCTXN.IDX, ACCTXN.UDX, AE_COMS, AGENCY are different file names present in my input file (it could be 000s files and all names are starting with ). Now, I am interested in getting the files which are having " Predicted... (1 Reply)
Discussion started by: rishav
1 Replies

3. Shell Programming and Scripting

Processing files using awk

Hi I have files in our UNIX directory like the below -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK.idx0 -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK.idx0 -rw-r--r-- 1 devinfo devsupp 432 Sep 14... (7 Replies)
Discussion started by: rbmuruga
7 Replies

4. UNIX and Linux Applications

Processing batches of files at a time

Hi I would like to run my script to process as many as 50 files at a time. Currently my script is being called like so: ./import.sh -f filename so I want to call my script in this way and it must execute every file in the directory /var/local/dsx/import (1 Reply)
Discussion started by: ladyAnne
1 Replies

5. Shell Programming and Scripting

get all files from a directory and pass the files for processing

Hi All, I have a directory in which there will be several files. i want to get all the files and pass it to a piece of code for processing on the files. This is the piece of code which does the processing. tr "\n" "|" < (log file name) | tr "$" "\n" > output echo ' ' >>output while... (1 Reply)
Discussion started by: suresh_kb211
1 Replies

6. Shell Programming and Scripting

Processing Multiple Files

Hello Everyone, I am new to scripting and confused with how to do this efficiently. I am trying to use AWK to do this. I have a lot of files in a folder which has the data of my throughput measurements in two columns i.e. Serial # and Throughput. like this 177.994 847.9 178.996 ... (1 Reply)
Discussion started by: hakim
1 Replies

7. Shell Programming and Scripting

Processing files

Hi I have the folowing input file, the file looks like below sftp> . .. archive x001_ameint*.zip x001_ameint_1.zip x001_ameint_2.zip x001_REPORTS*.zip x001_REPORTS_1.zip sftp> I want my output to look like this x001_ameint*.zip x001_ameint_1.zip x001_ameint_2.zip (1 Reply)
Discussion started by: ramky79
1 Replies

8. Shell Programming and Scripting

Two files processing

I have the following two files: 1st File: 1:1:100 2:101:400 3:401:450 4:451:600 5:601:980 6:981:1500 7:1501:1600 8:1601:1800 9:1801:2000 2nd File: 30 50 80 700 (2 Replies)
Discussion started by: moutaz1983
2 Replies

9. UNIX for Advanced & Expert Users

FTP and Files processing

I want to do the following thing and I am newbie to Unix: 1) FTP files and save them in a folder 2) Pick the oldest file and also its name among the above FTPed files 3) Copy it to another folder X and also move the file to folder Y I know how to do the first task i.e., FTP the files and... (1 Reply)
Discussion started by: harish409
1 Replies

10. Shell Programming and Scripting

Processing files within a directory one by one

Hi How to create a shell script which takes in to account all the files present within a directory DIR one by one e.g. suppose i have a directory named DIR where there are files with the extension .ABC i want to create shell script which processes all these files one by one. ... (1 Reply)
Discussion started by: skyineyes
1 Replies
Login or Register to Ask a Question