Ok, this is really beyond my scripting skill level so I'm hoping somebody can help me out with this. I have a trace file in the following format:
Code:
<timestame> <devicenum> <sector address> <size in sectors> <0 or 1 (write or read)>
Here is what I need to do. I need to use the <sector address>, <size in sectors>, and the <0 or 1> fields.
I need to first check that the last field is a 0. If it is 0, I will need to check more fields on this line. If it is 1, I can skip it and go on to the next line.
So, if the last field is 0, I need to calculate the "pages" that are in this line. My requirements for pages are:
1) A page will be made up of 4 sectors.
2) A page must start off at a <sector number> that is evenly divisible by 4. If it does not, the <sector address> should rounded DOWN to the nearest sector number that is evenly divisible by 4.
3) Additionally, since a page is 4 sectors, if the <size in sectors> is less than a multiple of 4, it will need to be rounded UP to the closest multiple of 4. In other words, if there are only 3 sectors in the <size in sectors>, that still takes up at least 1 page.
What I want to do is find the top 100 pages that are the most popular in terms of writes (the last column is 0) in a trace file.
For the 1st line, there will be 2 pages: the 1st page starts at 12 and the 2nd page starts at 16.
For the 2nd line, there will also be 2 pages: 1st page starts at 12 and 2nd page starts at 16. Note that both of these pages are actually the same pages from line 1.
The 3rd line is ignored because the last column is a 1 (read request).
For the 4th line, there will be 3 pages: the 1st page starts at 4, the 2nd page starts at 8, and the last page starts at 12. Note that the page starting at 12 is the same as the page in lines 1 and 2.
So for this small example, I want to have a printout similar to this. It should be sorted by the 2nd column in descending order so I can see the most popular files.
And if I haven't already asked you for the world...the faster it runs, the better! I will have to run this on several million lines, so speed is important. I already have awk or perl installed so hopefully it will be one of those. Perl seems to be much faster.
Thank you so much in advance! You guys are awesome!
A longer example of the trace is below for testing:
Well, it sounds like you should not try to do too much at once! You have a sector start and length, so convert to sector range first last, and then to page range first last, and then iterate the pages in that inclusive range. For instance, 123 45 is 123 - 167 is 30 - 41 is 30, 31, 32, ..., 41. This assumes a zero-based sector and page numbering.
Use grep to filter out just the writes, do the conversions, and send the page writes to:
Code:
sort | uniq -c | sort -nr | head -100
If your conversion is too slow in bash/ksh/sh, you can move it to PERL/JAVA/C++/C. If your sort set gets too big, in place of "sort | uniq -c", you can use my aggsx.c in memory aggregator -l option: https://www.unix.com/shell-programmin...roup-unix.html If you want it really fast, do it all in C using a big array of int or long long. You might even pipe the trace to it, and print out periodic reports. Even better would be to decontruct your trace and integrate part of it, so you can work all in int/long, not string numbers.
Last edited by DGPickett; 04-27-2011 at 01:49 PM..
Ok, I got it working. I split up the problem and I was able to figure it out...just took me a while. Here it is for those who are interested. I left all of the pages instead of just keeping the top 100.
Thanks!
Code:
#!/bin/bash
#===============================================================================
#
#===============================================================================
#@| SYNOPSIS
# |
#-| Filename: findPopularPages.sh
#-| Version: 1.0
# |
#-| Purpose:
#-| This script finds the popularity of pages of a disksim
#-| formatted trace file.
#-|
#-| Inputs: Real world trace file in the DiskSim format
#-|
#-| Outputs: Prints trace statistics to screen
#-|
#------------------------------------------------------------------------------
#@| REQUIREMENTS / DEPENDENCIES / ASSUMPTIONS
# |
#------------------------------------------------------------------------------
#@| REVISION HISTORY
# |
#-| Date Created/Revised: 4/30/2011 Author/Revisor: Jonathan Tjioe
#-| STR #: Bug ID: Description:
#-| 1.0 Jonathan Tjioe (4/30/2011)
# |
#===============================================================================
FILE=$1
SCRIPT=findPopularPages.sh
PWD=`pwd`
printf "\nStarted $SCRIPT on $FILE at:\t\t `date`"
#Only print the Write requests ($5==0)
removeReads() {
cat $FILE | awk '$5==0 {print $0}' > TEMP1
}
#The LBA must start at a block address that is divisible by 4
#The size must be in multipes of 4
adjustBlockAndSize() {
cat TEMP1 | awk '$3=(int($3/4)*4)' |awk '{
if (($4%4)!=0) {print $1" "$2" "$3" "((int($4/4)*4)+4)" "$5}
else {print $0} }' > TEMP2
}
#Need to save a list of all of the pages for each line
createPageListing() {
cat TEMP2 | awk '{for (i=$3; i<($3+$4); i+=4) {print i}}' > TEMP3
}
#Count how many times a page was written to
countUniquePages() {
cat TEMP3 | awk '{count[$1]++} END {for(i in count) print i, count[i]}' > TEMP4
}
#Sort the pages by the most popular in descending order
sortPopularity() {
cat TEMP4 | sort -nr -k2 > ${FILE}_mostPopularPages
}
#Delete temporary files
cleanUp() {
rm TEMP1 TEMP2 TEMP3 TEMP4
}
############## Start of Main ##############
removeReads
adjustBlockAndSize
createPageListing
countUniquePages
sortPopularity
cleanUp
############## End of Main ################
printf "\nFinished $SCRIPT on $FILE at:\t\t `date`\n\n"
exit 0
awk '$5==0 {$3=int($3/4)*4; $4=($4%4)?(int($4/4)*4)+4:$4; for (i=$3;i<=$3+$4;i+=4) a[i]++}
END{for (i in a) print i,a[i]|"sort -nr -k2|head -100"}' $FILE
I have a folder structure with multiple sub directories
MAIN
FOLDER1
SUBFOLDER1
files......
FOLDER2
SUBFOLDER1
files......
etc
and I want to find a way to create an output of every files first 20 lines. I've been searching and testing and failing. I can do it in a... (2 Replies)
I wanted to know how to find the memory taken by a process using top command. The output of the top command is as follows as an example:
Mem: 13333364k total, 13238904k used, 94460k free, 623640k buffers
Swap: 25165816k total, 112k used, 25165704k free, 4572904k cached
PID USER ... (6 Replies)
For the first 4 users only that are currently logged in output their effective user id.
It's not important the order in which each logged in i just want to have the top 4.
Same question as here...... (0 Replies)
OFFSET=100;
PAGESIZE=4096;
int dummy_last;
TOPSTACK = (caddr_t)(&dummy_last - OFFSET);
TOPSTACK = (caddr_t)((unsigned long)TOPSTACK -
((unsigned long)TOPSTACK % PAGESIZE));
this i a code to find the top of the stack, but not able to figure it out. can... (2 Replies)
Hi all,
I am looking to find the size of the database by counting all the used pages.
1. I have a file which reads like below
16384 4750850
32768 165
The first column is the pagesize and the second column is the number of pages... (6 Replies)
i have some 1000 files in my dir and i want to find top 100 files and move them to some other location:
below the 2 commands i used, but it is not working
ls -ltr | grep ^- | head -100 | xargs mv destination - _________>not working
ls -ltr | grep ^- | head -100 | xargs mv {}... (3 Replies)
Just joined after using the site as a guest.. (Very Good Stuff in here.. thanks folks.)
I am in the process of hardening a Solaris 10 server using JASS. I also must use DISA Security Checklists (SRR) scripts to test for things that did not get hardened to DISA standards.
One of the things... (5 Replies)
I'm using bash on cygwin/windows.
I'm trying to use find and exclude the directory /cygdrive/c/System\ Volume\ Information. When I try to use the command below I get the error "rm: cannot remove `/cygdrive/c/System Volume Information': Is a directory.
Can someone tell me what I am doing... (3 Replies)
Hi everybody,
I want to know if there is any posibility to find out - on an AIX system - which are the the users who consume most space or at least a posibility to obtain a list with all the users and how much space are they consuming ?
Trying to use du command was useless. Any idea?... (5 Replies)
04-27-2011
