Sponsored Content
Top Forums Shell Programming and Scripting AWK script to detect webpages from file Post 302406446 by malcomex999 on Tuesday 23rd of March 2010 02:00:15 AM
Old 03-23-2010
Code:
awk '{count[$0]++}END{for(c in count) print c" X"count[c]}' infile

Or...

Code:
sort infile | uniq -c

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Unix script to detect new file entry in directory

Hi All, I want to detect each new file coming / getting created in unix directory. When every new file came to directory, i have to get its details like its size , date and time stamp and store it into another file. Could any one please tell me , how i can achieve that? Thanks. (13 Replies)
Discussion started by: james_1984
13 Replies

2. UNIX for Dummies Questions & Answers

how to detect my script is already running

I have a script which must not be run more than once at any given time. THis script will be scheduled to run every 20 mins as a cron job. In my script can i have logic to say if this script is already running from the previous cron, then exit. How do i go about doing that. If you describe the... (11 Replies)
Discussion started by: rmulchandani
11 Replies

3. Shell Programming and Scripting

AWK script to detect webpages from file

Hi guys I'm very new to unix and I have to create an awk script that detects webpage addresses from a file/webpage and outputs how many times each webpage was detected.e.g. if my file was: (Note: The symbol " was added to stop them being created into links) "www.google.com"... (1 Reply)
Discussion started by: ROFL
1 Replies

4. Shell Programming and Scripting

script to detect a file from inserted usb and puts into a Variable

There is a same named log file that I have on my 2 different android phones. When I plug it into my computer, it appears in the media folder, For example the first android phone: /media/F6BA-0AF5/folder/A.log I want to put that into a variable to be manipulated.... (3 Replies)
Discussion started by: tobenguyen
3 Replies

5. Shell Programming and Scripting

Script to detect dynamic ip change and update to config file

Hi All, I am newbie here and request your assistance. I have a service running on public ip, but since I have a dynamic IP it keeps on changing and every time I need to manually get the new ip and add to the config file and restart the service. This has become bit time consuming. Hence, I... (4 Replies)
Discussion started by: Shaan_Shaan
4 Replies

6. Shell Programming and Scripting

How to detect empty field in awk ?

Hi ! programmers I have a need of detecting empty field in file my file looks like this 40.900|-71.600|1.6|20|1|1961|21.00|3.700||1|US|28035|10029370|31 40.900|-71.600|5.7|20|1|1961|21.00|3.700||1|US|28035|10029370|31 40.900|-71.600|7.8|20|1|1961|21.00|3.700||1|US|28035|10029370|31... (7 Replies)
Discussion started by: Dona Clara
7 Replies

7. Shell Programming and Scripting

Help with detect with regex and move script

Hi all, I am needing some help with a script that will search for a video file by known extensions and then do a pattern search (I'm guessing via regex) and then based on a match of one type of another move the file to an assigned directory. I would like to do this with either a shell script... (7 Replies)
Discussion started by: Simplify
7 Replies

8. Shell Programming and Scripting

How to detect awk and nawk?

I have 10 scripts which use awk/nawk extensively. I have to always change awk to nawk and nawk to awk when i deploy all my scripts to different types of servers as some support nawk while other support awk. Can you propose a solution that without having to tweak my 10 scripts at several... (8 Replies)
Discussion started by: mohtashims
8 Replies

9. Shell Programming and Scripting

awk script to detect specific string in a log file and count it

Hello, can someone guide me on this? I don't know what is the best approach, (awk script, shell script) I am using RedHat Linux version 6.5. There is a third party application deployed on that server. This app by default generates 5 log files and each file is 20MB. These log rollover... (5 Replies)
Discussion started by: ktisbest
5 Replies

10. Shell Programming and Scripting

How to detect url in use in a script?

Hello, I have a small script and it runs from web application in below format: pipe:///path_to_myscript.sh url1 url2 url3 myscript.sh: #!/bin/bash count=0 while do count=$((count+1)) exec 3>&1 ((ffmpeg -i $1 ...... -f mpegts pipe:1 2>/dev/null 1>&3 ) 2>&1 | \ while read LINE; do echo... (9 Replies)
Discussion started by: baris35
9 Replies
Smokeping_probes_WebProxyFilter(3)				     SmokePing					Smokeping_probes_WebProxyFilter(3)

NAME
Smokeping::probes::WebProxyFilter - tests webproxy filter performance and function. OVERVIEW
This probe tests if your filtering webproxy is working properly. Drawing from a list of forbidden websites, it tries to establish a connection to each one of them and registers a 'loss' when it suceeeds. If you want to test availability of a website, use the EchoPingHttp probe. SYNOPSIS
*** Probes *** +WebProxyFilter forks = 5 maxsize = 2000 min_interval = 0.1 offset = 50% step = 300 timeout = 15 useragent = SmokePing/2.x (WebProxyFilter Probe) # The following variables can be overridden in each target section deny_re = Access Prohibited more_hosts = www.playboy.com, www.our-competition.com pings = 5 timeout = 2 # [...] *** Targets *** probe = WebProxyFilter # if this should be the default probe # [...] + mytarget # probe = WebProxyFilter # if the default probe is something else host = my.host deny_re = Access Prohibited more_hosts = www.playboy.com, www.our-competition.com pings = 5 timeout = 2 DESCRIPTION
The probe uses the LWP::UserAgent module to retreive a series of wepages. It expects to get the firewalls 'site-prohibited' page. Any other response (or a real loss) gets logged as a loss and can be used to trigger an alarm. The probe tries to be nice to the firewall and waits at least X seconds between starting filetransfers, where X is the value of the probe specific `min_interval' variable (1 by default). Many variables can be specified either in the probe or in the target definition, the target-specific variable will override the prove- specific variable. VARIABLES
Supported probe-specific variables: forks Run this many concurrent processes at maximum Example value: 5 Default value: 5 maxsize How much of the webpage should be retreived. Default value: 2000 min_interval The minimum interval between each starting GETs in seconds. Example value: 0.1 Default value: 1 offset If you run many probes concurrently you may want to prevent them from hitting your network all at the same time. Using the probe- specific offset parameter you can change the point in time when each probe will be run. Offset is specified in % of total interval, or alternatively as 'random', and the offset from the 'General' section is used if nothing is specified here. Note that this does NOT influence the rrds itself, it is just a matter of when data acqusition is initiated. (This variable is only applicable if the variable 'concurrentprobes' is set in the 'General' section.) Example value: 50% step Duration of the base interval that this probe should use, if different from the one specified in the 'Database' section. Note that the step in the RRD files is fixed when they are originally generated, and if you change the step parameter afterwards, you'll have to delete the old RRD files or somehow convert them. (This variable is only applicable if the variable 'concurrentprobes' is set in the 'General' section.) Example value: 300 timeout How long a single 'ping' takes at maximum Example value: 15 Default value: 5 useragent The web browser we claim to be, just in case the FW is interested Default value: SmokePing/2.x (WebProxyFilter Probe) Supported target-specific variables: deny_re Regular expression, matching the 'deny' response from the firewall Example value: Access Prohibited more_hosts A comma separated list of banned websites to test in addition to the one specified in the host variable. The websites will be tested one after the other in one round, this means that while normal probes do run the same test serveral times in a row, this one will alter the webpage with each round. The reason for this is, that eventhough we try to retreive remote webpages, the answer will come from the firewall everytime, so we kill two birds in one go. First we test the firewalls latency and second we make sure its filter works properly. Example value: www.playboy.com, www.our-competition.com pings How many pings should be sent to each target, if different from the global value specified in the Database section. Note that the number of pings in the RRD files is fixed when they are originally generated, and if you change this parameter afterwards, you'll have to delete the old RRD files or somehow convert them. Example value: 5 timeout Timeout in seconds for the test complete. Example value: 2 Default value: 2 AUTHORS
Tobias Oetiker <tobi@oetiker.ch> sponsored by Virtela BUGS
This probe is somewhat unortodox, since it regards the sucessful retrieval of a banned webpage as a loss. 2.6.8 2013-03-17 Smokeping_probes_WebProxyFilter(3)
All times are GMT -4. The time now is 10:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy