Sponsored Content
Top Forums Shell Programming and Scripting "Elements per page"... seeking ideas... Post 302498038 by treesloth on Friday 18th of February 2011 07:20:58 PM
Old 02-18-2011
"Elements per page"... seeking ideas...

I work for a web hosting company uses Apache. We like to come up with composite models of what our customers do so that we can tailor our servers to what they need. One question we like to answer is, "For a given page downloaded from our customer's virtual server, what is the mean number of elements on that page?" An "element", roughly defined, is a transfer that appears in the Apache log in order to populate a page requested by a customer. The most common "element" type, of course, is images.

So, we'd like to have some reasonable way to determine the mean and dstandard deviation of number of "elements" per page. Possibly of help is that we are just building a general model, so some helpful assumptions may tend to even out over large numbers of log files. And I should mention that effectively our only source of information about this is the Apache logs from customers' virtual servers.

How would you approach this problem? We're certainly not helped by the fact that Apache logs really weren't designed for this. For that matter, neither was HTTP. Even so, without prejudicing your various brains toward one approach, here's a thought...

We know already that almost all pages served by our servers are transferred in less than 6 seconds. That's the HTML source page (or whatever dynamic page type it may be...) and the elements it calls. So, suppose we were to say that all log entries with a certain client IP address appearing within 6 seconds of each other are likely to be associated with a single customer page request. Then we could just record the IP and associated times and look for "clusters" of 6 seconds or less and count the number of elements in that grouping. But I'm uncertain of how to code that sort of "sliding window". I tend to do these things best in awk, and I'm not seeing how to do that.

Any thoughts? Like I said, that's just one approach, and certainly not necessarily the best. Thanks in advance.
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. Shell Programming and Scripting

Compare file names and select correct elements to include in "for each loop"

Hi everyone, I`ll try to be most clear I can explaining my help request. I have 2 folders Folder A-->This folder receives files through FTP constantly Folder B-->The files from Folder A are unzipped and then processed in Folder B Sometimes Folder A doesn`t contain all... (2 Replies)
Discussion started by: cgkmal
2 Replies

3. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

4. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

5. Web Development

New "Page Not Found" (404) Page

Made some changes to the forum, so when a page is not found and generates a 404 error, the site redirects to "Today's Posts" page and added a "Not Found" message: <?php header('HTTP/1.0 404 Not Found', true, 404); header("Location: https://www.unix.com/search.php?do=getdaily&redirect=404");... (0 Replies)
Discussion started by: Neo
0 Replies

6. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

7. Web Development

Quick Fix for Google Search Console "Page is not mobile friendly"

Over the past 10 plus years, we have countless posts where the user did not use CODE tags or they used ICODE tags incorrectly. This has has the results of this site penalized by Google for having pages which are "not mobile friendly". So, working quietly in the background, in the thankless... (0 Replies)
Discussion started by: Neo
0 Replies

8. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies
MOD_APPARMOR(8)                                                      AppArmor                                                      MOD_APPARMOR(8)

NAME
mod_apparmor - fine-grained AppArmor confinement for Apache DESCRIPTION
An AppArmor profile applies to an executable program; if a portion of the program needs different access permissions than other portions, the program can "change hats" via aa_change_hat(2) to a different role, also known as a subprofile. The mod_apparmor Apache module uses the aa_change_hat(2) mechanism to offer more fine-grained confinement of dynamic elements within Apache such as individual php and perl scripts, while still allowing the performance benefits of using mod_php and mod_perl. To use mod_apparmor with Apache, ensure that mod_apparmor is configured to be loaded into Apache, either via a2enmod, yast or manual editing of the apache2(8)/httpd(8) configuration files, and restart Apache. Make sure that apparmor is also functioning. Once mod_apparmor is loaded within Apache, all requests to Apache will cause mod_apparmor to attempt to change into a hat named by the URI (e.g. /app/some.cgi). If no such hat is found, it will fall back to attempting to use the hat DEFAULT_URI; if that also does not exist, it will fall back to using the global Apache profile. Most static web pages can simply make use of the DEFAULT_URI hat. Additionally, before any requests come in to Apache, mod_apparmor will attempt to change hat into the HANDLING_UNTRUSTED_INPUT hat. mod_apparmor will attempt to use this hat while Apache is doing the initial parsing of a given http request, before its given to a specific handler (like mod_php) for processing. Because defining hats for every URI/URL often becomes tedious, mod_apparmor provides the AAHatName and AADefaultHatName Apache configuration options. AAHatName AAHatName allows you to specify a hat to be used for a given Apache <Directory>, <DirectoryMatch>, <Location> or <LocationMatch> directive (see the Apache documenation for more details). Note that mod_apparmor behavior can become confused if <Directory*> and <Location*> directives are intermingled and it is recommended to use one type of directive. If the hat specified by AAHatName does not exist in the Apache profile, then it falls back to the behavior described above. AADefaultHatName AADefaultHatName allows you to specify a default hat to be used for virtual hosts and other Apache server directives, so that you can have different defaults for different virtual hosts. This can be overridden by the AAHatName directive and is checked for only if there isn't a matching AAHatName or hat named by the URI. If the AADefaultHatName hat does not exist, it falls back to the DEFAULT_URI hat if it exists (as described above). URI REQUEST SUMMARY
When profiling with mod_apparmor, it is helpful to keep the following order of operations in mind: On each URI request, mod_apparmor will first aa_change_hat(2) into ^HANDLING_UNTRUSTED_INPUT, if it exists. Then, after performing the initial parsing of the request, mod_apparmor will: 1. try to aa_change_hat(2) into a matching AAHatName hat if it exists and applies, otherwise it will 2. try to aa_change_hat(2) into the URI itself, otherwise it will 3. try to aa_change_hat(2) into an AADefaultHatName hat if it has been defined for the server/vhost, otherwise it will 4. try to aa_change_hat(2) into the DEFAULT_URI hat, if it exists, otherwise it will 5. fall back to the global Apache policy BUGS
mod_apparmor() currently only supports apache2, and has only been tested with the prefork MPM configuration -- threaded configurations of Apache may not work correctly. There are likely other bugs lurking about; if you find any, please report them at <http://https://bugs.launchpad.net/apparmor/+filebug>. SEE ALSO
apparmor(7), subdomain.conf(5), apparmor_parser(8), aa_change_hat(2) and <http://wiki.apparmor.net>. AppArmor 2.7.103 2012-06-28 MOD_APPARMOR(8)
All times are GMT -4. The time now is 04:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy