I keep getting lost. You have shown several different bits and pieces of your code and commented that you have made other changes without showing them to us. The code below makes some guesses at what you want based on my understanding from re-reading all of the posts in this thread.
You still refuse to show us the format you want for the sums and averages of the times the system has been running for the last 7 and 30 entries in the log file. So, I have arbitrarily decided to print those values in the following format:
Code:
n line sum: h:mm; average: h:mm:ss
where n is the number of lines used from the end of the file to perform the calculations. (If there are less than 7 or 30 lines in the log file, the output will tell you how many lines were actually used. The sums are shown in hours and minutes. The averages are shown in hours, minutes, and seconds.)
There are several things that still make absolutely no sense to me, but I think the script below does what you asked for (even if it doesn't make sense to me).
First some notes before we get into the code:
If there is any chance that a script can be run close to midnight, it is dangerous to make multiple calls to the date utility to gather bits and pieces of various date components. The code below only invokes date once and extracts all of the date information used in the script from that one invocation.
Even if a machine is just up for 4 days and 4 hours, the substring extraction you're using from the uptime output will not work. (Output in this case from uptime could be something like 105:19 (for 4 days, 9 hours, and 19 minutes) and your current code would put 105:1 in your log file.) The code below puts just the minutes (without the "min" or "mi") or the complete hours and minutes in log file entries.
I repeat that using /home/uplog.txt instead of $HOME/uplog.txt will fail if more than one user on a system runs this script unless they are running with root priviledges.
I repeat that the number you are extracting from the uptime output says absolutely nothing about how long a given user has been working on a project, how long they have been logged in, or anything else related to any particular user. The number you are using just tells you how long it has been since the machine you are using was rebooted. I see absolutely no reason why adding or averaging 7 or 30 of these values would be important to anyone!
This script uses echo to add a line to the end of your log file. The awk script then processes that log file. If, and only if, the log file contains more than 200 (or maxrec) lines, the awk script will rewrite the log file in place throwing away all but the last maxrec lines. Until you are confident that this works correctly, make a backup copy of your log file between the echo that adds a line to the log file and the awk script that rewrites the log file and writes the averages file.
This awk script should work for the up time formats you have described for your version of the uptime utility. (The BSD and OS X versions of the uptime utility provide considerably more complex ouput formats than you have described for your version of uptime. This script will not work correctly with the BSD or OS X versions of uptime.)
This awk script uses two circular buffers. The array l[] holds up to maxrec lines from the log file. After maxrec lines have been read, the next input line overwrites the oldest input line so when we get to the end of file, l[] holds only the last maxrec lines that were in the file. The array mins[] holds up to 30 up time values (after converting them to minutes) from the last 30 lines read from the log file.
The 7 and 30 for the number of lines to be averaged and the 200 for the number of lines to be kept in the log file are parameterized, so you just need to change the number used to initialize nl_av1, nl_av2, and maxrec, respectively, to have the script use whatever values you want. It will automatically adjust such that nl_av1 and nl_av2 will never be greater than maxrec.
The averages are saved in the file /home/averages.txt.
The input and output file pathnames are also parameterized.
Now for the code:
Code:
#!/bin/bash
# Set script related variables:
# Input and output files:
input="/home/uplog.txt" # Input log file (this file will be
# updated in place if it contains more
# than "maxrec" records).
output_averages="/home/averages.txt" # Output averages file.
# Record counts:
maxrec=200 # Maximum # of lines to keep in log file.
nl_av1=7 # # of lines to be used for 1st average.
nl_av2=30 # # of lines to be used for 2nd average.
# Time and date stuff:
read V R <<-EOF
$(date '+%x %A %d/%m/%y die %V.week')
EOF
T=$((86400/3600)) # Hours in a day (the hard way)
# Data for log entry:
machine=$(uname -n) # Node name.
# Get time since reboot from uptime:
IFS=' ,' read junk junk H_or_HM junk <<-EOF
$(uptime)
EOF
echo $T "not yet" # ???
echo $USER # shows me the actual user
echo "today is" $R
# Add entry to log file:
echo "uptime $USER an $machine $H_or_HM $V" >> "$input"
# Process log file:
awk -v maxrec="$maxrec" -v nl_av1="$nl_av1" -v nl_av2="$nl_av2" \
-v avoutf="$output_averages" '
# Normalize script variables:
BEGIN { # Do not accept averages for more lines than we are saving...
# Do not average more input lines than we are keeping in the output.
if(nl_av1 > maxrec) {
printf("nl_av1 reduced from %d to %d.\n", nl_av1, maxrec)
nl_av1 = maxrec
}
if(nl_av2 > maxrec) {
printf("nl_av2 reduced from %d to %d.\n", nl_av2, maxrec)
nl_av2 = maxrec
}
# Set size of uptime data circular buffer:
m = nl_av1 > nl_av2 ? nl_av1 : nl_av2
}
# Save input file pathname in case we need to trim it to maxrec lines.
NR == 1 {
inf = FILENAME
}
# Process input log file data:
{ # Save last maxrec lines in input circular buffer (l[]):
l[NR % maxrec] = $0
# Convert uptime "x mins," or "hours:minutes," to minutes and save it in
# another circular buffer (mins[]).
nf = split($5, hm, /[:,]/)
if (nf == 1) {
# We just have minutes.
mins[NR % m] = hm[1]
} else {# We have hours and minutes separated by a ":".
mins[NR % m] = hm[1] * 60 + hm[2]
}
printf("mins[%d] = %d from %s\n", NR % m, mins[NR % m], $5)
}
# Function to calculate sum and average uptime from last "n" entries in the
# mins[] circular buffer.
# Returns number of lines actually used in calculations.
function av(n, i) {
if(n > NR) {
# Calculate sum and average based on all saved lines.
n = NR
}
t = 0
for(i = NR - n + 1; i <= NR; i++)
t += mins[i % m]
a = t / n
# printf("av(%d): total=%d minutes, average=%.2f minutes\n", n, t, a)
return n
}
# Function to convert "m" minutes to "H" hours, "M" minutes, and "S" seconds.
function m2hms(m) {
S = (m % 1) * 60 # seconds
M = int(m % 60) # minutes
H = int(m / 60) # hours
}
# We have hit EOF on input, rewrite input file if we found more than "maxrec"
# lines, calculate and print both sums and averages.
END { # If we had more than "maxrec" input lines, trim input file to "maxrec"
# lines.
if(NR > maxrec) {
for(i = NR - maxrec + 1; i <= NR; i++)
print l[i % maxrec] > inf
printf("Input file %s trimmed from %d lines to %d lines\n",
inf, NR, maxrec)
} else printf("Input file %s: %d lines processed.\n", inf, NR)
# Print averages.
L = av(nl_av1)
m2hms(t)
printf("%d line sum: %d:%02d; ", L, H, M) > avoutf
m2hms(a)
printf("average: %d:%02d:%02.0f\n", H, M, S) > avoutf
L = av(nl_av2)
m2hms(t)
printf("%d line sum: %d:%02d; ", L, H, M) > avoutf
m2hms(a)
printf("average: %d:%02d:%02.0f\n", H, M, S) > avoutf
}' "$input"
There is nothing bash specific about this script. It will also work with ksh or any other shell that meets basic POSIX shell syntax requirements.
I know that you said that your version of uptime only produces up time data in two formats: xx mins and h:mm, but to be sure that you're getting what you think you're getting, please run the command:
Code:
while sleep 58
do uptime >> uptimelog.txt
done &
for a little over an hour. The goal here is to be sure that your version of uptime doesn't use a different format (such as xx hours instead of xx:00) at integral numbers of hours since the system was rebooted.
If there are special cases like this, you'll have to make adjustments to the script I provided to handle them.
First I beg your pardon for being offline three consecutive days since my provider dropped my line and returned to work just today . That turned me into countless tries about the awk-statement, without a result that is close to my target.
Second, I never said the uptime is linked to any user being logged in or using the computer. Though the given output above shows three users, it is me,myself and I that uses this computer. I always said the uptime just as it is, the computer being switched on and probably connected to the net. The output of time as a sum or an average should be hhh:mm, nothing more. That means for example 123 (hours) : 23 (minutes). Or 123:23. Does this collide with my target to measure the n times before? Your fourth point matters to me, because I said above that my aim is to switch the MAC and perhaps make a redial after a certain time.
Actually now the bash is running your given command and I'll wait to respond to pass you that output as well as my desired output. Meanwhile I will study your source-code. Thanks for your persistence!
---------- Post updated at 08:50 PM ---------- Previous update was at 07:39 PM ----------
So here it is, as I hope the way you expected it. This is my version, setting the date at the end. The last three lines.
What I want is to sum up the third row, the format should be hhh:mm. After trying this, it stumbled upon double digits in that row and stopped at something like 7:56, but that surely is not the maximum of that row.
That is what it writes until now, my own script. So I ran your code to see the result and return the output of your last command. As it is the first twelve lines
Code:
19:13:44 up 8:15, 2 users, load average: 0.12, 0.17, 0.10
19:14:42 up 8:16, 2 users, load average: 0.04, 0.14, 0.09
19:15:40 up 8:17, 2 users, load average: 0.18, 0.16, 0.10
19:16:38 up 8:18, 2 users, load average: 0.15, 0.16, 0.10
19:17:36 up 8:18, 2 users, load average: 0.06, 0.13, 0.09
19:18:34 up 8:19, 2 users, load average: 0.14, 0.13, 0.09
19:19:32 up 8:20, 2 users, load average: 0.10, 0.12, 0.09
19:20:30 up 8:21, 2 users, load average: 0.16, 0.13, 0.09
19:21:28 up 8:22, 2 users, load average: 0.06, 0.10, 0.09
19:22:26 up 8:23, 2 users, load average: 0.52, 0.22, 0.12
19:23:24 up 8:24, 2 users, load average: 0.26, 0.21, 0.12
19:24:22 up 8:25, 2 users, load average: 0.14, 0.18, 0.12
the last five lines of your given command
Code:
20:32:03 up 9:33, 3 users, load average: 0.00, 0.04, 0.06
20:33:01 up 9:34, 3 users, load average: 0.00, 0.03, 0.06
20:33:59 up 9:35, 3 users, load average: 0.03, 0.04, 0.06
20:34:57 up 9:36, 3 users, load average: 0.01, 0.03, 0.05
20:35:55 up 9:37, 3 users, load average: 0.00, 0.02, 0.04
and the last lines of the update to the uplog.txt file
Code:
15:52:02 up 4:53, 2 users, load average: 0.00, 0.01, 0.04 23.06.2014
15:54:09 up 4:55, 2 users, load average: 0.67, 0.22, 0.11 23.06.2014
15:54:31 up 4:55, 2 users, load average: 0.48, 0.20, 0.10 23.06.2014
uptime sandy an jarbo3 9:42 23.06.2014
Code:
mins[4] = 0 from users,
mins[5] = 0 from users,
mins[6] = 582 from 9:42
Input file /home/sandy/logger/uplog.txt trimmed from 216 lines to 200 lines
First I beg your pardon for being offline three consecutive days since my provider dropped my line and returned to work just today . That turned me into countless tries about the awk-statement, without a result that is close to my target.
Second, I never said the uptime is linked to any user being logged in or using the computer. Though the given output above shows three users, it is me,myself and I that uses this computer. I always said the uptime just as it is, the computer being switched on and probably connected to the net. The output of time as a sum or an average should be hhh:mm, nothing more. That means for example 123 (hours) : 23 (minutes). Or 123:23. Does this collide with my target to measure the n times before? Your fourth point matters to me, because I said above that my aim is to switch the MAC and perhaps make a redial after a certain time.
Actually now the bash is running your given command and I'll wait to respond to pass you that output as well as my desired output. Meanwhile I will study your source-code. Thanks for your persistence!
---------- Post updated at 08:50 PM ---------- Previous update was at 07:39 PM ----------
So here it is, as I hope the way you expected it. This is my version, setting the date at the end. The last three lines.
What I want is to sum up the third row, the format should be hhh:mm. After trying this, it stumbled upon double digits in that row and stopped at something like 7:56, but that surely is not the maximum of that row.
That is what it writes until now, my own script. So I ran your code to see the result and return the output of your last command. As it is the first twelve lines
Code:
19:13:44 up 8:15, 2 users, load average: 0.12, 0.17, 0.10
19:14:42 up 8:16, 2 users, load average: 0.04, 0.14, 0.09
19:15:40 up 8:17, 2 users, load average: 0.18, 0.16, 0.10
19:16:38 up 8:18, 2 users, load average: 0.15, 0.16, 0.10
19:17:36 up 8:18, 2 users, load average: 0.06, 0.13, 0.09
19:18:34 up 8:19, 2 users, load average: 0.14, 0.13, 0.09
19:19:32 up 8:20, 2 users, load average: 0.10, 0.12, 0.09
19:20:30 up 8:21, 2 users, load average: 0.16, 0.13, 0.09
19:21:28 up 8:22, 2 users, load average: 0.06, 0.10, 0.09
19:22:26 up 8:23, 2 users, load average: 0.52, 0.22, 0.12
19:23:24 up 8:24, 2 users, load average: 0.26, 0.21, 0.12
19:24:22 up 8:25, 2 users, load average: 0.14, 0.18, 0.12
the last five lines of your given command
Code:
20:32:03 up 9:33, 3 users, load average: 0.00, 0.04, 0.06
20:33:01 up 9:34, 3 users, load average: 0.00, 0.03, 0.06
20:33:59 up 9:35, 3 users, load average: 0.03, 0.04, 0.06
20:34:57 up 9:36, 3 users, load average: 0.01, 0.03, 0.05
20:35:55 up 9:37, 3 users, load average: 0.00, 0.02, 0.04
and the last lines of the update to the uplog.txt file
mins[4] = 0 from users,
mins[5] = 0 from users,
mins[6] = 582 from 9:42
Input file /home/sandy/logger/uplog.txt trimmed from 216 lines to 200 lines
What do you mean by the text marked in red above? I thought you wanted to get the sum and average values of the 3rd FIELD (not ROW) from the last 7 and 30 lines??? And, this is the first you have said about wanting a maximum instead of a sum and average???
When I asked you to run uptime for a couple of hours, what we need to see is what happens around multiples of one hour and around an integral multiple of one day of up time. So, instead of skipping over the data between what you showed us:
show us what is on the lines starting with 19:58, 19:59, 20:00, 20:01, and 20:02. And we need to see what happens around the time when the system has been running for an even multiple of 24 hours. (At least on OS X, the format on one hour multiples, is something like 9 hours, 17 secs, (instead of 9:00,) and on 24 hour multiples the change is even more pronounced.)
If you got the output:
Code:
mins[4] = 0 from users,
mins[5] = 0 from users,
mins[6] = 582 from 9:42
Input file /home/sandy/logger/uplog.txt trimmed from 216 lines to 200 lines
from my script, I obviously left in a debugging statement that I forgot about. Note also that the output here says that that the 1st 16 lines from that file were discarded (thereby keeping the last 200 lines) as you requested. Is the last line in your log file:
Code:
uptime sandy an jarbo3 9:42 23.06.2014
in the format you want?
Obviously, the format I used to write the sums and averages in /home/averages.txt (or whatever pathname you changed it to) is not in the format you want. But, as I said before, I made a wild guess at a reasonable output format based on the data being summed and averaged. If you will actually show us the last few lines of your log file AND show us exactly what the corresponding sum and average output should be for those lines, we might be able to produce data in the format you want. Did you even look in this file to see the output my script produced?
PLEASE DON'T TELL ME I WROTE THE WRONG SUMS AND AVERAGES IF YOU REFUSE TO SHOW US WHAT YOU WANT!
tells you absolutely nothing about the data on the last 7 or on the last 30 lines in your log file; it is looking for the maximum value in all 200 lines in your log file. And, it is doing a string comparison (not a numeric comparison of hours and minutes); so 7:56 is greater than 22:30, but even more importantly $3 seems to be what you want for looking at the output from uptime not the data in your log file where I hope the up time is in field 5:
Code:
uptime sandy an jarbo3 9:42 23.06.2014
shown in red here; not field 3 shown in orange. Or, have you changed the data format in your log file again!
Okay, I will run that uptime-command for some time, but due to the heat in this particular room I cannot leave the computer running after 11:00 am. May you won't believe it, but 107.6 °Fahrenheit is simply to hot, even with a fan directed to it and an air-condition on. Let me put it this way, after re-reading your source-code.
what you wrote sure is this famous ring-buffer and I appreciate your persistence for pushing this newbie, for some people like me have to learn the hard way. But nonetheless your source-code is a fiendishly prank, throughout the last days with my provider working again, I came to a conclusion. One is given in your source-code, bash-variables used again in awk, secondly outsourcing, and third the structure I recently used for finding duplicates on a large file. Like this
1. I will use or re-use the date-variable from the bash for a comparison,
2. outsourcing both calculations to /tmp and
3. re-read from there, since a few kilobyte won't kill any ram.
For sure there a Mozarts and Chopins on any keyboard handling this in a one-liner. What matters to me, that I understand what is written and that should be simple. Simplicity works. So I'll be back here as soon as my source-code improves significantly. Yesterday I even dared to create my first array, but I have to improve the output at stdout. Thats why I'd rather send it without an array to /tmp. But I'll be back, not diving away.
#!/bin/bash
# name of script zetzwo.sh #
# from IBM-tutorial, set a silent exit, if a condition is not true #
# unless it is part of a sequence of commands #
# http://www.ibm.com/developerworks/aix/library/au-usingtraps/ #
set -e
set -u
machine="Today at: ";
machine=$(printf "%s %s" "$machine" "$(uname -n)");
R=`date +%A' der '%d'/'%m'/'%y' the '%V'.'Week`;
V=`date +%x`; #will be used later on, Locale (e.g. 31.12.1999)
T=$((86400/3600)); #24 hours, the hard way
echo $T "not yet";
# shows the actual user
echo $USER;
if (( "$T" < 24 ))
then
echo "today is" $R
else :
fi;
stringZ=`uptime`
stringZ=$(printf "%s %s" "$stringZ" $V)
# Variable bucket and bucket2 for the two little databases to work with #
bucket=(/home/$USER/logger/uplog.txt)
bucket2=(/home/$USER/logger/uplog-two.txt)
echo "uptime" $USER " " $machine " " ${stringZ:13:5} && echo ${stringZ} >> $bucket |
awk ' {NR~/^1$/ NR~/^30$/} END { OFMT="%4f"; ORS ":"; print ENVIRON["USER"]"\t" $3,$11 } ;'
$bucket >> $bucket2;
awk ' {max = 0} {if ($2>max) max=$2} END {OFMT="%6f"; ORS ":"; print max};' $bucket2;
awk ' {sum=sum+$2} END {OFMT="%2f"; ORS ":"; print sum/NR};' $bucket2; #average
While this conception works properly, creating a second database named bucket2 I can go on for the calculation in awk and delete the whole file after 30 days with some bash like this
Code:
find /path/to/files* -type f -mtime +30 -exec rm '{}' '+'
and setting "wait" to create a new one, just because I only wanted to make that calculation for that period.
@Don Cragun, all I could find about awk-printing format to fit in this was the up above.
Sure there is one weakness in that. If I switch on an off my computer several times per day, that would mean to create for each day one seperate file.
This is the output of bucket2, the one to be calculated.
The awk below executes and is close (producing the first 4 columns in desired). However, when I add the sum of $7, I get nothing returned. Basically, I am trying to combine all the matching $4 in f1 and output them with the average of $7 in each match. Thank you :).
f1
... (2 Replies)
The output of an awk script is the below file. Line 1,3 that starts with the Ion... need to be under line 2,4 that starts with R_. The awk runs but no output results. Thank you :).
file
IonXpress_007 MEV37
R_2016_09_20_12_47_36_user_S5-00580-7-Medexome
IonXpress_007 MEV40... (6 Replies)
Sorry for the weird title but i have the following problem.
We have several files which have between 10000 and about 500000 lines in them. From these files we want to remove lines which contain a pattern which is located in another file (around 20000 lines, all EAN codes). We also want to get... (28 Replies)
I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it.
Does anyone know of a way?
Thanks a lot. (7 Replies)
Hi,
how do i read a file using awk for a given no of line?
e.g
1. read only first 50 line.
2. read starting from line 20 to line 60..
thanks in advance.
-alva (8 Replies)
Hi,
Here i have to check first record $3 $4 with second record $1 $2 respectively. If match found, then check first record $2 == second record $4 , if it equals , then reduce two records to single record like as desired output.
Input_file
1 1 2 1
2 1 3 1
3 1 4 1
3 1 3 2
desired... (3 Replies)
I have a file with 8 fields. I need the subtotals for fields 7 & 8 when field 5 changes.
cat wk1
01/02/2011/18AB/17/18/000000071/000000033
01/02/2011/18AB/17/18/000000164/000000021
01/02/2011/18AB/17/18/000000109/000000023
01/02/2011/28FB/04/04/000000000/000000000... (2 Replies)
Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created... (4 Replies)
I have a file which is one big long line of text about 10Kb long.
Can someone provide a way using awk to introduce carriage returns every 40 chars in this file.
Any other solutions would also be welcome.
Thank you in advance. (5 Replies)
