09-08-2012
How to use GREP to extract URL from file
Hi All ,
Here is what I want to do:
Given a line:
98.70.217.222 - - [08/Jul/2012:09:14:29 +0000] "GET /liveupdate-aka.symantec.com/1340071490jtun_nav2k8enn09m25.m25?h=abcdefgh HTTP/1.1" 200 159229484 "-" "hBU1OhDsPXknMepDBJNScBj4BQcmUz5TwAAAAA" "-"
1. Get the URL component: "/liveupdate-aka.symantec.com/1340071490jtun_nav2k8enn09m25.m25?h=abcdefgh"
2. Check if the URL component excluding query string which is "/liveupdate-aka.symantec.com/1340071490jtun_nav2k8enn09m25.m25" is greater than 800 characters
I am using the command below to do this but it is checking if the entire URL component is greater than 800 characters and not excluding the Query string:
gunzip -c * |cut -d ' ' -f7|sort -n|uniq -c|grep '^.*\/[^?]*'|grep '.\{800,\}'
3. Also check is each component in the URL is greater than 200 characters
For eg. Is "/liveupdate-aka.symantec.com/1340071490jtun_nav2k8enn09m25.m25" greater than 800 characters and
is "liveupdate-aka.symantec.com" geater than 200 characters
is "1340071490jtun_nav2k8enn09m25.m25" greater than 200 characters.
It would be good if we can do all of the above in one command if possible.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hello everybody.
I have lines that looks something like this:
<done16=""118"" done18=""$ title=""thisisatitle"" href=""/JoeBanana" alt=""Joe""><done16=""118"" done18=""$ title=""thisisatitle"" href=""/GeraldGiraffe" alt=""Gerald"">
What kind of SED command would I need to use to extract... (4 Replies)
Discussion started by: digi
4 Replies
2. Shell Programming and Scripting
Hi All,
I am new to shell scripting,Could any of you help me on this below :confused:
--------------------
I need to write a shell script where i have open an url and grep for a particular word in the url and want to display it in a flatfile.
Eg: Want to open yahoo.com and grep for yahoo... (1 Reply)
Discussion started by: sumithra
1 Replies
3. UNIX for Advanced & Expert Users
Hi all,
I would like to ask whether in Unix shell/perl have any functions or command to allow grep/cat/read a file inside compressed .tgz without extract it?
I know we can tar tvf a compressed tgz but this only allow we read the path/filename contained inside the tarball. If we want to read... (3 Replies)
Discussion started by: mayshy
3 Replies
4. Shell Programming and Scripting
Hi,
I have following data file;
<outline title="Matt Cutts" type="rss" version="RSS" xmlUrl="http://www.mattcutts.com/blog/feed/" htmlUrl="http://www.mattcutts.com/blog"/>
<outline title="Stone" text="Stone" type="rss" version="RSS" xmlUrl="http://feeds.feedburner.com/STC-Art"... (8 Replies)
Discussion started by: fahdmirza
8 Replies
5. Shell Programming and Scripting
for example, I have an html file, contain
<a href="http://awebsite" id="awebsite" class="first">website</a>and sometime a line contains more then one link, for example
<a href="http://awebsite" id="awebsite" class="first">website</a><a href="http://bwebsite" id="bwebsite"... (36 Replies)
Discussion started by: 14th
36 Replies
6. Shell Programming and Scripting
Hello All,
I need to extract lines from a file that contains ALPHANUMERIC and the length of Alphanumeric is set to 16. I have pasted the sample of the lines from the text file that I have created.
My problem is that sometimes 16 appears in other part of the line. I'm only interested to... (14 Replies)
Discussion started by: mnassiri
14 Replies
7. Shell Programming and Scripting
I need to write a script, which takes the input a log file and create output file with acct no's line by line from selected records with text like (in red) :
18:51:18 | 217863|Acct 0110855565|RC 17608| 16 Subs| 1596 UsgRecs| 2 Secs| 430 CPUms| prmis2:26213 <MoveUsage d
aemon needs to run... (7 Replies)
Discussion started by: rkrish
7 Replies
8. Shell Programming and Scripting
Hello,
I want extract multi values from multi url source to a csv text. Thank you very much for help.
my curl code : curl "http://www.web.com/cities//city.html
Source code:
div class="clear"></div>
<table class="listing-details">
<tr>
... (1 Reply)
Discussion started by: hoo
1 Replies
9. Shell Programming and Scripting
Hello,
Am very new to perl , please help me here !!
I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file.
below is the script which i have written so far ,
#!/usr/bin/perl
use LWP::UserAgent;
use... (2 Replies)
Discussion started by: scott_cog
2 Replies
10. Shell Programming and Scripting
All,
I require a script that grabs some text from the gitHub API and will grep (or other function) for a string a characters that starts with (") quotes followed by two letters, may contain a pipe |, and ending with ) . What i have so far is below but it's not returning anything.
... (4 Replies)
Discussion started by: ChocoTaco
4 Replies