Anyone like a challenge?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Anyone like a challenge?
# 1  
Old 02-11-2015
Anyone like a challenge?

I have searched through google, and this forum to try and find the answer, but alas, nothing quite hits the whole answer.

I am trying to read the last line (or lines) of some log files. I do this often.

The files are named sequentially, using the date as part of the file name, and appending the iteration so like this:

Code:
$ls -ltr |tail -6
h20150208.log.001
h20150209.log.001
h20150210.log.001
h20150211.log.001
h20150211.log.002
h20150211.log.003

And I want generally to tail -1 each of the files to see the end result

There may be 33 files - or may be 3000 files - different each time - I don't want to have to manually specify the filenames - but "*log.0??" works for a "filter". Aged files are compressed (.Z) or gzipped (.gz), and obviously I don't want to read these files.

However.
All of the methods I have tried , using find - don't sort the files into the correct order - I need it to be in chronological order, and I also need to see the filename - preferably on the same line as the resultant output - which is effectively succeeded or failure.
All the attempts I have made using ls -ltr error:
Code:
tail -1 `ls -ltr *.log.0??`
ls -ltr *.log.0??|xargs tail -f

Just to make this a little more challenging - this will be across multiple different O/Ss, including HP-UX, SunOS, RHEL and more.

I would prefer a solution which doesn't need to create a script file - but any amount of typing (or pasting) at the command prompt is acceptable.

The result I'm looking to achieve is rather like:

Code:
h20150208.log.001 Output from last line of file
h20150209.log.001 Output from last line of file
h20150210.log.001 Output from last line of file
h20150211.log.001 Output from last line of file
h20150211.log.002 Output from last line of file
h20150211.log.003 Output from last line of file

So far, the closest I have to what I need is:

Code:
for f in $(find . -name '*.log.0??'); do printf $f && cat $f | tail -1; done

But this fails because it isn't sorted in the correct way, and it concatenates the filename and timestamp (with no gap) like so:

Code:
./h20150131.log.00123:23:06  completed successfully
./h20150201.log.00123:01:47  completed successfully
./h20150208.log.00104:47:18  completed successfully
./h20150205.log.00102:55:49  completed successfully
./h20150126.log.00123:52:02  completed successfully
./h20150207.log.00105:50:30 JOBFAILURE: COMPLETED WITH ERRORS RC=201

If anyone has any fabulous ideas or can point me in the right direction, I'd be most grateful...


Background:
For anyone who's interested, I'm a DBA currently working in 2nd line support - looking after about 8000 database instances over 3500 hosts. This particular query is to look at backup log files, and try & determine where it all went wrong....
------
Regards,



Ian
# 2  
Old 02-11-2015
Try:
Code:
find . -name '*.log.0??' |
sort -t. -k1.2,1n -k2 |
while read f
do
  printf "%s " "$f"
  tail -1 "$f"
done

Note: Using *.log.0?? would limit the result to the first 99 iterations for any particular day. Perhaps a better alternative would be to use: *.log.[0-9][0-9][0-9]

Last edited by Scrutinizer; 02-11-2015 at 08:41 PM..
These 2 Users Gave Thanks to Scrutinizer For This Post:
# 3  
Old 02-11-2015
Damn - I've been trying to do that for days!

If you have the time - I'd be most grateful for an explanation -I get the first line! Smilie
Also, can we exclude the timestamp?
And the "./" if possible?

------
Regards,


Ian

Last edited by BatterBits; 02-11-2015 at 08:44 PM.. Reason: Addition
# 4  
Old 02-11-2015
Given that the filenames contain the date and a sequence number within that date, I don't see the need for sorting by time in ls, and unless there are files in multiple directories, there is no need for find. Doesn't this do what you need?:
Code:
for lf in $(ls *.log.0??|tail -n 6);do printf '%s ' "$lf";tail -n 1 "$lf";done

# 5  
Old 02-11-2015
Hi Don,

Thanks - but that doesn't work - the output is "unsorted", still contains the timestamp which I don't need.

Code:
h20150204.log.001 05:10:30 completed successfully
h20150207.log.001 05:50:30 JOBFAILURE: COMPLETED WITH ERRORS RC=201
h20150210.log.001 00:57:23 JOBFAILURE: COMPLETED WITH ERRORS RC=201
h20150211.log.003 00:35:39 OTHER MESSAGE
h20150127.log.001 02:32:00 completed successfully

-----
Regards,


Ian

---------- Post updated at 01:07 AM ---------- Previous update was at 12:59 AM ----------

Quote:
Originally Posted by Scrutinizer
Try:
Note: Using *.log.0?? would limit the result to the first 99 iterations for any particular day. Perhaps a better alternative would be to use: *.log.[0-9][0-9][0-9]
Yes, I get that, thanks. Generally, there won't be more than 99 files - but I know how to deal with it if there are - the syntax I used, because some of the files are compressed, so I couldn't use *log.*
# 6  
Old 02-11-2015
Hi,
Code:
find . -name '*.log.[0-9][0-9][0-9]' |    # get a the names of all the hosts
sort -t. -k1.2,1n -k2 |                   # sort this using a "." as a field separator; sort
                                          # numerically with the 2nd character to the last
                                          # character of the first field and for the second
                                          # key use a regular sort from filed 2 onwards.
while read f
do
  printf "%s " "$f"
  tail -1 "$f"
done

To get rid of the ./ and the timestamp, aa quick fix would be:
Code:
...
do
   printf "%s" "${f##*/}"
   tail -1 "$f" | awk '{$1=x}1'
done

Or a bit more efficiently if the file name do not contain spaces, perhaps
Code:
...
do
   printf "%s " "${f##*/}"
   tail -1 "$f" 
done | awk '{$2=x}1'

This User Gave Thanks to Scrutinizer For This Post:
# 7  
Old 02-11-2015
Perfect, Scrutinizer, thank you so much

-----
Regards,


Ian

---------- Post updated at 01:17 AM ---------- Previous update was at 01:13 AM ----------

Bizarrely, my previous post is off to be moderated! Smilie


Anyway - both methods are perfect - thanks so much.

---
Regards,



Ian
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Touch Challenge

I've been given a directory full of subdirectories full of logfiles of the same name: /logfiles/day1/file1/blockednodes.csv day1-14 file1-48 The above is the actual directory structure for 14 days worth of a logfile that is generated every 30 minutes. It's been done this way to preserve the... (15 Replies)
Discussion started by: Cludgie
15 Replies

2. AIX

openssh connectivity challenge

Running a Power 5 Blade on AIX, with remote connectivity issues via putty. AIX V 6.1.00-02 openssh V5.2.0.5300 openssl V0.9.8.1103 Intermittent remote connections. Seems to connect every other time I try via my putty client. Using hosts.allow and hosts.deny to filter IP Addrss... (2 Replies)
Discussion started by: FrankM
2 Replies

3. Shell Programming and Scripting

PS1 challenge

Ok then i Have a challenge for you : Give me PS1 so that it always display the least 2 levels of directory (except if i am above of course) I want it this way : so if i go to / /home/ /home/user /home/user/whatever /home/user/whatever1/whatever2 my PS1 should respectively... (12 Replies)
Discussion started by: ctsgnb
12 Replies

4. Shell Programming and Scripting

Geo Weather Challenge

Hi everybody, I'm new to these forums and this is my first post. A couple days ago I was trying to find a simple script that would return an individual's local weather conditions using I.P. based geolocation. After many failed search attempts, I began my quest to create this for myself. I have to... (0 Replies)
Discussion started by: o0110o
0 Replies

5. Shell Programming and Scripting

regex challenge

Here's a regex substitution operation that has stumped me with sed: How do you convert lines like this: first.key ?{x.y.z} second.key ?{xa.ys.zz.s} third.key ?{xa.k} to: first.key ?{x_y_z} second.key ?{xa_ys_zz_s} third.key ?{xa_k} So i'm basically converting all the... (11 Replies)
Discussion started by: neked
11 Replies

6. Shell Programming and Scripting

sed xml challenge

I have a web xml file that looks like this: <allinfo> <info> <a>Name1<\a> <b>address1<\b> <c>phone1<c> <\info> <info> <a>Name2<\a> <b>address2<\b> <c>phone2<c> <\info> <\allinfo> I want to use sed to... (2 Replies)
Discussion started by: katrvu
2 Replies

7. Shell Programming and Scripting

sed replacement, challenge one!!!!

Hi all, Thanks in advanced. This question really bothered me much. What i want is to replace any times of repeated 'TB' to 'T', below is example. It can be fullfil by AWK and perl, but my desire is using SED to realize it. So here means we treat TB as a whole part, which means 's/TB*/T/'... (4 Replies)
Discussion started by: summer_cherry
4 Replies

8. Shell Programming and Scripting

AWK Challenge

I have the following text Microsoft iSCSI Initiator version 2.0 Build 3497 Targets List: iqn.2001-05.com.equallogic:0-8a0906-daef43402-138000002a4477ba-grsrv12-extra iqn.2001-05.com.equallogic:0-8a0906-986f43402-520000002b447951-exchange ... (9 Replies)
Discussion started by: netmedic
9 Replies

9. UNIX for Advanced & Expert Users

safeword challenge

Hi, there are some servers here at work which issue a Safeword challenge after I login. Can anyone tell me exactly how the challenge/response system works? In particular, how are the valid keys decided? (2 Replies)
Discussion started by: blowtorch
2 Replies

10. UNIX for Advanced & Expert Users

X25 Address - A challenge...

Okay - I've been searching near and far for the answer to this seemly simple question..... how do I find the X25 address for a server. Is there some sort of dump or ping or even a config which would tell me the address. I can find nothing on the web and my colleagues can't help either. (1 Reply)
Discussion started by: peter.herlihy
1 Replies
Login or Register to Ask a Question