Processing of log files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Processing of log files
# 8  
Old 07-12-2015
Hi Don/Rudi,

I regret if I could not present it properly, as there are couple of file types which appears until third character of every files ( ie fn, ski etc ).

Here is my sample file :

Code:
sn2015.12.14.12.29.25_a.log
sn2015.12.14.12.29.25_c.log
sn2015.12.14.12.29.25_b.log
sn2015.12.14.12.29.25.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.24_b.log

#Expected Output

Code:
sn2015.12.14.12.29.25.log
sn2015.12.14.12.29.25.a.log
sn2015.12.14.12.29.25.b.log
sn2015.12.14.12.29.25.c.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.24_b.log

Again thank you for your responses.
# 9  
Old 07-12-2015
Quote:
Originally Posted by scott_apc
.
.
.
#Expected Output

Code:
sn2015.12.14.12.29.25.log
sn2015.12.14.12.29.25.a.log
sn2015.12.14.12.29.25.b.log
sn2015.12.14.12.29.25.c.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.24_b.log

Again thank you for your responses.

How come sn2015.12.14.12.30.13_b.log is sorted below sn2015.12.14.12.30.23.log ? What is the suffix in sn2015.12.14.12.29.25.a.log ?

Do you really expect a reasonable solution based on a dubious specification as given in several posts even though clear questions were asked?
# 10  
Old 07-12-2015
My Bad -

Here is the expected output :

Code:
sn2015.12.14.12.29.25.log
sn2015.12.14.12.29.25_a.log
sn2015.12.14.12.29.25_b.log
sn2015.12.14.12.29.25_c.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.24_b.log

# 11  
Old 07-12-2015
Sorry - I can't help. I came up with a solution but it doesn't match your expected output.
# 12  
Old 07-12-2015
Quote:
Originally Posted by scott_apc
My Bad -

Here is the expected output :

Code:
sn2015.12.14.12.29.25.log
sn2015.12.14.12.29.25_a.log
sn2015.12.14.12.29.25_b.log
sn2015.12.14.12.29.25_c.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.24_b.log

OK. So your sample input contains duplicates. (And you refuse to tell us why.) Getting rid of duplicates is easy.

You tell us that dates are to be in ascending order and suffixes are to be in descending order, but your expected output above has dates in a seemingly random order (assuming that the timestamp is part of what you are calling the date) and suffixes in increasing order (not decreasing). If the timestamp is to be ignored when sorting the data, then the suffices you say you want are in random order in your list above.

And, you say: "I regret if I could not present it properly, as there are couple of file types which appears until third character of every files ( ie fn, ski etc )." Is this supposed to mean something in the way the output is to be sorted??? You have not shown us any input or output that has more than a single sequence of two or three letters that appear in every file in each list you have shown us.

I am sorry, but I am unable to ascertain any pattern in the output you want. And, even if we assume that the line marked in red above is a typo, the output you have shown us doesn't come close to what you said you want.

I give up.
# 13  
Old 07-13-2015
So can I calrify your requirements? For each numerical section on the file name, you want to sort ascending and then for each suffix you want to sort descending, as in from z to a? That doesn't match your output as far as I can see (although I haven't scrutinised it too closely.

Could you clearly paste (not just make up) some sample input and then manually work out the required output getting all the possibilities ironed out.

It might be possible with a single sort command specifying the numeric bit as a primary sort ascending and the suffix as a secondary descending if that's what you need.

Can you tell me the order that these files should process that you listed recently?
Code:
sn2015.12.14.12.29.25.log
sn2015.12.14.12.29.25_a.log
sn2015.12.14.12.29.25_b.log
sn2015.12.14.12.29.25_c.log
sn2015.12.14.12.30.23.log
sn2015.12.14.12.30.13_b.log
sn2015.12.14.12.30.24_a.log
sn2015.12.14.12.30.25_b.log
sn2015.12.15.12.30.23.log
sn2015.12.15.12.30.23_a.log
sn2015.12.15.12.30.24_b.log

It seems that these are sorted, but with the suffix being ascending as well (a to z) so it's just a simple sort without parameters.
Code:
sort file.txt

Indeed if this file file.txt is built from the output from ls in some way, I'm wondering why your output is not sorted already. Does this work:-
Code:
:
:
for file in fn*.log       # Specify matching pattern, i.e. exclude .doc files
do
    : # process $file
done
:
:

If there are too many files and the command fails for being too long, you could try:-
Code:
:
:
for file in `find . -name "fn*.log" | sort`
do
   : # process $file
done
:
:



You need to be very clear about your need else we're just hoping that your "My car doesn't work" type description might be all sorts of serious things, or perhaps you are out of fuel.

We need accurate input to work on and meticulously checked required output to be certain we are going to be helpful and using our time well - we do all give it freely.


I do hope that we can still help you.

Robin

Last edited by rbatte1; 07-13-2015 at 09:45 AM.. Reason: Spelling
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Processing multiple files

Hello I have a program cfxfrwb which is designed to remove headers from reports files. The cfxfrwb is located in the following directory /u01/efin/v40/live/bin I run the program against a single report file in the temp directory and it does it's job../cfxfrwb... (2 Replies)
Discussion started by: captainrhodes
2 Replies

2. Shell Programming and Scripting

script: processing files

Hi All, I have a log file which is attached here: ACCTXN.ALLOC, ACCTXN.IDX, ACCTXN.UDX, AE_COMS, AGENCY are different file names present in my input file (it could be 000s files and all names are starting with ). Now, I am interested in getting the files which are having " Predicted... (1 Reply)
Discussion started by: rishav
1 Replies

3. Shell Programming and Scripting

Processing files using awk

Hi I have files in our UNIX directory like the below -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_1_0_FK.idx0 -rw-r--r-- 1 devinfo devsupp 872 Sep 14 02:09 IMGBTREE27309_12272_11_0_0_PK.idx0 -rw-r--r-- 1 devinfo devsupp 432 Sep 14... (7 Replies)
Discussion started by: rbmuruga
7 Replies

4. UNIX and Linux Applications

Processing batches of files at a time

Hi I would like to run my script to process as many as 50 files at a time. Currently my script is being called like so: ./import.sh -f filename so I want to call my script in this way and it must execute every file in the directory /var/local/dsx/import (1 Reply)
Discussion started by: ladyAnne
1 Replies

5. Shell Programming and Scripting

get all files from a directory and pass the files for processing

Hi All, I have a directory in which there will be several files. i want to get all the files and pass it to a piece of code for processing on the files. This is the piece of code which does the processing. tr "\n" "|" < (log file name) | tr "$" "\n" > output echo ' ' >>output while... (1 Reply)
Discussion started by: suresh_kb211
1 Replies

6. Shell Programming and Scripting

Processing Multiple Files

Hello Everyone, I am new to scripting and confused with how to do this efficiently. I am trying to use AWK to do this. I have a lot of files in a folder which has the data of my throughput measurements in two columns i.e. Serial # and Throughput. like this 177.994 847.9 178.996 ... (1 Reply)
Discussion started by: hakim
1 Replies

7. Shell Programming and Scripting

Processing files

Hi I have the folowing input file, the file looks like below sftp> . .. archive x001_ameint*.zip x001_ameint_1.zip x001_ameint_2.zip x001_REPORTS*.zip x001_REPORTS_1.zip sftp> I want my output to look like this x001_ameint*.zip x001_ameint_1.zip x001_ameint_2.zip (1 Reply)
Discussion started by: ramky79
1 Replies

8. Shell Programming and Scripting

Two files processing

I have the following two files: 1st File: 1:1:100 2:101:400 3:401:450 4:451:600 5:601:980 6:981:1500 7:1501:1600 8:1601:1800 9:1801:2000 2nd File: 30 50 80 700 (2 Replies)
Discussion started by: moutaz1983
2 Replies

9. UNIX for Advanced & Expert Users

FTP and Files processing

I want to do the following thing and I am newbie to Unix: 1) FTP files and save them in a folder 2) Pick the oldest file and also its name among the above FTPed files 3) Copy it to another folder X and also move the file to folder Y I know how to do the first task i.e., FTP the files and... (1 Reply)
Discussion started by: harish409
1 Replies

10. Shell Programming and Scripting

Processing files within a directory one by one

Hi How to create a shell script which takes in to account all the files present within a directory DIR one by one e.g. suppose i have a directory named DIR where there are files with the extension .ABC i want to create shell script which processes all these files one by one. ... (1 Reply)
Discussion started by: skyineyes
1 Replies
Login or Register to Ask a Question