Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk last n lines of file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk last n lines of file
# 15  
Old 06-18-2014
@Don Cragun
Allright, I will drop that part of head -c. I am learning.
Surely there is some faster way to produce that uplog.txt with the sufficent lines.
So here is what the script zetzwo.sh displays on the screen:
Code:
sandy@jarbo3:~/Desktop$ ./zetzwo.sh
24 not yet
sandy
uptime sandy   Today at: jarbo3   3:20

And here comes the thirty entries of the uplog.txt
Seeing it now, I would like just to hang on to awk. For several reasons. Recently it helped me to find duplicates in a huge file much faster than any bash-command. It was about a file with more than 3 million entries. And to advance a bit in (g)awk.
As you asked for the last thirty lines or entries, here they are.

Code:
09:04:40 up 33 min, 2 users, load average: 0.00, 0.11, 0.10 18.06.2014
09:04:42 up 33 min, 2 users, load average: 0.00, 0.10, 0.10 18.06.2014
09:04:43 up 33 min, 2 users, load average: 0.00, 0.10, 0.10 18.06.2014
09:05:26 up 33 min, 2 users, load average: 0.08, 0.11, 0.10 18.06.2014
09:05:27 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:27 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:28 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:28 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:28 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:29 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:29 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:29 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:30 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:30 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:30 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:31 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:31 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:32 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:33 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:33 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
11:47:25 up 3:15, 2 users, load average: 0.05, 0.10, 0.05 18.06.2014
11:47:26 up 3:15, 2 users, load average: 0.05, 0.10, 0.05 18.06.2014
11:47:27 up 3:15, 2 users, load average: 0.05, 0.10, 0.05 18.06.2014
11:47:28 up 3:15, 2 users, load average: 0.05, 0.10, 0.05 18.06.2014
11:47:29 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:30 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:30 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:31 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:31 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:36 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:52:10 up 3:20, 2 users, load average: 0.00, 0.03, 0.02 18.06.2014

# 16  
Old 06-18-2014
And what 7 and 30 line sums and averages do you want your script to produce from these 30 lines from uplog.txt???

It appears that the last field in your output from zetzwo.sh:
Code:
uptime sandy   Today at: jarbo3   3:20

comes from the third field in the last line of uplog.txt:
Code:
09:04:40 up 33 min, 2 users, load average: 0.00, 0.11, 0.10 18.06.2014
09:04:42 up 33 min, 2 users, load average: 0.00, 0.10, 0.10 18.06.2014
... ... ...
11:47:36 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:52:10 up 3:20, 2 users, load average: 0.00, 0.03, 0.02 18.06.2014

Is that the field you want to sum and average? We can see from this that the output from uptime on your system varies in this field depending on how long your system has been running. Presumably it uses
Code:
n min

for up times less than one hour and h:mm for up times of one hour or more. We saw something in an earlier post where what I assume was this value was 61:52, so I assume that the format doesn't change for larger numbers of hours. Is this correct? (On my system, the format changes when you get to 24 hours:
Code:
10:47  up 5 days, 13:21, 8 users, load averages: 1.32 1.36 2.49

which I'm guessing your system does not do.) We can deal with issues like this if we know what input we'll be getting. But, if we don't know the format of the data we'll be processing, we can't successfully convert your system's up time to a number of minutes (an integer value) or a number of hours (a floating point value with minutes as a fractional part of an hour). And, we need to know the ranges of values we're going to be handling to determine whether we should be using integer or floating point values. (Assuming you're using a system with at least a 32-bit signed long int, and that your average system up times will always be less than 135 years, we can use integer arithmetic.)
# 17  
Old 06-18-2014
Probably the next generation of ssd and nand-chips gets close to that value of MTBF of a 135 years and my digital inheritance does not matter to nobody. I do understand your point of view about the kind of value to be calculated. So I keep going with my ambition to solve it in awk. Thanks a lot really. Regards.
# 18  
Old 06-18-2014
We'll be happy to help you with the awk code (including how to trim the output to a given number of lines from the end of the input file) if you'll just show us what 7 and 30 entry sums and averages you're trying to produce (and verify the formats that the uptime utility produces on your system for various amounts of time the machine has been running).
# 19  
Old 06-18-2014
Okay, I keep it up in this thread, because the original request or task remains unsolved.
Having now a neat database as shown above, I want to use awk to
1. count the file just up to the line 200 and write to /dev/null above that number of lines.
2. sort it upside down (what equals in bash tac, sort -r) for there is no constant number of lines.
3. pick up the first seven lines for the calculation of sum and sum/NR (the average value).
4. repeat step 3 for the first thirty lines for the same calculation.
5 print the result of step 3 and step 4 to stdout or a file (in case I want to use later with xmessage).

My first attempt is to limit the number of lines or NR already written to that file uplog.txt. After doing so I think it gets easier to handle it. Or just to cut it down to the right size.
What I know about awk-structure is, it needs this:

Code:
pattern { action statements }
  function name(parameter list) { statements }

Step one and my attempt just runs into an syntax error.
The code is the following:

Code:
awk 'END {for NR > 181 printf > "/dev/null"}' /home/uplog.txt;

Giving me
Code:
 END {for NR > 181 printf > "/dev/null"}
awk: commandline  :1:          ^ syntax error
awk: commandline  :1: END {for NR > 181 printf > "/dev/null"}
awk: commandline  :1:                   ^ syntax error

I'd say pattern is "file has been read until the END" and the {action statement} in this case is to print all line higher than 180 to /dev/null. I see that the counter is missing.

And here follows a sample of the last entries to that uplog.txt. In the beginning of this thread it was about field number five, what turns out to be the first three now. Never mind the values, I changed the computer.

Code:
21:46:35 up 3:42, 2 users, load average: 0,19, 0,22, 0,29 18.06.2014
21:47:29 up 3:43, 2 users, load average: 0,08, 0,18, 0,27 18.06.2014

Any hints? This time I need to learn, to advance a bit. Thanks in advance.
# 20  
Old 06-18-2014
I repeat: If these are the last 7 lines in uplog.txt:
Code:
09:05:31 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
09:05:33 up 33 min, 2 users, load average: 0.07, 0.10, 0.10 18.06.2014
11:47:30 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:31 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:47:31 up 3:15, 2 users, load average: 0.04, 0.10, 0.05 18.06.2014
11:52:10 up 3:20, 2 users, load average: 0.00, 0.03, 0.02 18.06.2014
22:15:16 up 61:52, 2 users, load average: 0.23, 0.44, 0.30 20.06.2014

EXACTLY what output do you want for the sum and average. Do you want the results as an average number of minutes; hours and minutes; hours, minutes, and seconds; days, hours, minutes, and seconds? If you refuse to show us the format of the output you want, how can we write code that will give you what you want? Please help us help you. Show us what you want!

Other than x mins (as in 33 mins) and h:mm (as in 3:20 and 61:52), is there any other format that your version of uptime produces for the values you want to sum and average? If you won't tell us what format the data is in that we are processing, how can we process that data? Please help us help you. Confirm what data formats appear in the input you want to process!

PLEASE ANSWER THE QUESTIONS I'M ASKING! Let us help you! I think I understand which lines you want to process. And, tac would be a waste of time and sort would not even come close to giving you what you want (1st because the data format for the data you're sorting is not consistent and 2nd because sort will give you the wrong lines to process if your machine is rebooted at any time during the period covered by the 200 lines you want to keep in uplog.txt).
# 21  
Old 06-19-2014
Code:
06:06:33 up 25 min, 2 users, load average: 0,07, 0,14, 0,20 19.06.2014
06:06:34 up 25 min, 2 users, load average: 0,07, 0,14, 0,20 19.06.2014
06:06:35 up 25 min, 2 users, load average: 0,06, 0,13, 0,20 19.06.2014
06:06:36 up 25 min, 2 users, load average: 0,06, 0,13, 0,20 19.06.2014

The first four lines of todays uplog.txt-file and a simple test in awk shows what I'd like to have as a result.
My source-code for awk this time is
Code:
awk 'NR==5 { print; exit }' /home/uplog.txt

What is giving me
Code:
24 not yet
webster
uptime webster   Today at: jarbo3   27 mi
uptime webster at jarbo3   42 mi   28.05.2014

That is correct, because in this older uplog.txt-file user webster began his first attempts with that project and line five of the file says exactly 42 mi in field 5 on may the 28th, 2014.
Code:
uptime webster an jarbo3   42 mi   28.05.2014
uptime webster an jarbo3   50 mi   28.05.2014
uptime webster an jarbo3   50 mi   28.05.2014
uptime webster an jarbo3   1:39   28.05.2014
uptime webster an jarbo3   1:42   28.05.2014

NF 5 right in this sample above is the output format that shall be used for the calculation. nn mi before reaching the full hour and nn:nn after completing the hour.
As I mentioned above, I first want to count the file to a maximum of 200 lines.
So while using the above awk 'NR==5 {print;exit}' there's not even a format that I send. I suppose it is just calculating that values, as you mentioned integer, 32bits, floating-point and so on, putting the right format itself. I put in here a sample of that first lines of that uplog.txt-file. So by being written into that very file the format is already given. The latter awk-command just picks up the former input. This is just a test, to see once per day a successfull run of the script. After all the mishits, it keeps me going on. So, answering your question about the output-format, I should be nn mi, if it is less than an hour, and if it is more than 60 min just like above, e.g. 3:53.

---------- Post updated at 07:22 AM ---------- Previous update was at 07:16 AM ----------

ups, I misunderstood that request for the exaxt output format. Reading the total unix-time here, it is not about the NASA or anything scientific. With less than 60 minutes, just like above nn mi, reaching the total hour in triple-digits summing up 7 days and later on 30 days. For example 129:45. That would mean just 129 hours on air plus 45 minutes. I squeeze a 168 hours into one week.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk to average matching lines in file

The awk below executes and is close (producing the first 4 columns in desired). However, when I add the sum of $7, I get nothing returned. Basically, I am trying to combine all the matching $4 in f1 and output them with the average of $7 in each match. Thank you :). f1 ... (2 Replies)
Discussion started by: cmccabe
2 Replies

2. Shell Programming and Scripting

awk to reorder lines in file

The output of an awk script is the below file. Line 1,3 that starts with the Ion... need to be under line 2,4 that starts with R_. The awk runs but no output results. Thank you :). file IonXpress_007 MEV37 R_2016_09_20_12_47_36_user_S5-00580-7-Medexome IonXpress_007 MEV40... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

awk remove/grab lines from file with pattern from other file

Sorry for the weird title but i have the following problem. We have several files which have between 10000 and about 500000 lines in them. From these files we want to remove lines which contain a pattern which is located in another file (around 20000 lines, all EAN codes). We also want to get... (28 Replies)
Discussion started by: SDohmen
28 Replies

4. Shell Programming and Scripting

Counting lines in a file using awk

I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it. Does anyone know of a way? Thanks a lot. (7 Replies)
Discussion started by: guitarist684
7 Replies

5. Shell Programming and Scripting

Read a file using awk for a given no of lines.

Hi, how do i read a file using awk for a given no of line? e.g 1. read only first 50 line. 2. read starting from line 20 to line 60.. thanks in advance. -alva (8 Replies)
Discussion started by: alvagenesis
8 Replies

6. Shell Programming and Scripting

Reducing file lines in awk

Hi, Here i have to check first record $3 $4 with second record $1 $2 respectively. If match found, then check first record $2 == second record $4 , if it equals , then reduce two records to single record like as desired output. Input_file 1 1 2 1 2 1 3 1 3 1 4 1 3 1 3 2 desired... (3 Replies)
Discussion started by: vasanth.vadalur
3 Replies

7. Shell Programming and Scripting

awk print lines in a file

Dear All, a.txt A 1 Z A 1 ZZ B 2 Y B 2 AA how can i use awk one line to achieve the result: A Z|ZZ B Y|AA Thanks (5 Replies)
Discussion started by: jimmy_y
5 Replies

8. UNIX for Dummies Questions & Answers

How do you subtotal lines in a file? Awk?

I have a file with 8 fields. I need the subtotals for fields 7 & 8 when field 5 changes. cat wk1 01/02/2011/18AB/17/18/000000071/000000033 01/02/2011/18AB/17/18/000000164/000000021 01/02/2011/18AB/17/18/000000109/000000023 01/02/2011/28FB/04/04/000000000/000000000... (2 Replies)
Discussion started by: MS75001
2 Replies

9. Shell Programming and Scripting

Select some lines from a txt file and create a new file with awk

Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created... (4 Replies)
Discussion started by: capnino
4 Replies

10. UNIX for Advanced & Expert Users

Help with splitting lines in a file using awk

I have a file which is one big long line of text about 10Kb long. Can someone provide a way using awk to introduce carriage returns every 40 chars in this file. Any other solutions would also be welcome. Thank you in advance. (5 Replies)
Discussion started by: martinbarretto
5 Replies
Login or Register to Ask a Question