Login or Register to Ask a Question and Join Our Community


Homework & Coursework Questions

How to use xargs to repeat as a loop to grab date string?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Homework and Emergencies Homework & Coursework Questions How to use xargs to repeat as a loop to grab date string?
# 1  
Old 11-24-2014
How to use xargs to repeat as a loop to grab date string?
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!

1. The problem statement, all variables and given/known data:
My goal to find how many requests in 14 days from weblog server. I know to cat a weblog file to wc -l to find the total of all requests have send to server. From there on, I can use a loop of 14 days to go through each line and do comparison with Date String until the string are different I would increment the date until it reach 14th date. I was thinking to seq 14 and xargs commands to go through 14 days cycle.
I would like to know how to do a string compare between date?

Thanks in advance,

Scopiop

2. Relevant commands, code, scripts, algorithms:
Code:
192.192.1.1 - - [10/June/2013...]
192.192.1.2 - - [10/June/2013...]
192.192.1.3 - - [11/June/2013..]
192.192.1.4 - - [12/June/2013..]
........

output
Code:
The last 14 day's requests are: 100


3. The attempts at a solution (include all code and scripts):

4. Complete Name of School (University), City (State), Country, Name of Professor, and Course Number (Link to Course):
City College/Aaron Brick/160B

Note: Without school/professor/course information, you will be banned if you post here! You must complete the entire template (not just parts of it).
# 2  
Old 11-24-2014
I don't think xargs is quite applicable here.

What do the lines really look like? Obscure what you must, but as is they're a little too generic.
# 3  
Old 11-24-2014
Code:
79.114.31.152 - - [10/Jun/2013:07:43:07 -0700] "GET /~otangdec/cnit132/images/chic_dance.gif HTTP/1.1" 200 556572 "http://filelist.ro/details.php?id=23714" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36"
220.181.108.175 - - [10/Jun/2013:07:43:12 -0700] "GET / HTTP/1.1" 200 96 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
107.198.90.65 - - [10/Jun/2013:07:43:13 -0700] "GET /~otangdec/cnit132/images/chic_dance.gif HTTP/1.1" 200 556572 "http://us-mg6.mail.yahoo.com/neo/launch?.rand=39q9ldbqb22hh" "Mozilla/5.0 (Windows NT 6.0; rv:21.0) Gecko/20100101 Firefox/21.0"
93.104.214.107 - - [10/Jun/2013:07:43:16 -0700] "GET /~otangdec/cnit132/guestbook.html HTTP/1.1" 200 12054649 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
93.104.214.107 - - [10/Jun/2013:07:43:19 -0700] "GET /~otangdec/cnit132/addguest.html HTTP/1.1" 200 3510 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
93.104.214.107 - - [10/Jun/2013:07:43:19 -0700] "POST /~otangdec/cnit132/guestbook.pl HTTP/1.1" 200 979 "http://hills.ccsf.cc.ca.us/~otangdec/cnit132/addguest.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
66.249.73.225 - - [10/Jun/2013:07:43:22 -0700] "GET /robots.txt HTTP/1.1" 404 208 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.225 - - [10/Jun/2013:07:43:23 -0700] "GET /~cking1/cnit132/hw7.html HTTP/1.1" 200 7052 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
178.238.236.213 - - [10/Jun/2013:07:43:06 -0700] "GET /~otangdec/cnit132/guestbook.html HTTP/1.1" 200 12054649 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"
178.238.236.213 - - [10/Jun/2013:07:43:40 -0700] "GET /~otangdec/cnit132/addguest.html HTTP/1.1" 200 3510 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"

I want to do a string compare between 10 vs 11 to keep track 14 days cycle
Code:
 [10/Jun/2013:23:59:09]
 [11/Jun/2013:00:00:18]

I want to do a string compare between [
# 4  
Old 11-24-2014
Easier said than done!

I'm deliberately posting the following command for various reasons:
  1. To prove that OP's idea might work under certain/ideal circumstances (non-strict accuracy eg. (roughly) to a day, it's the second half of the month etc.)
  2. To show the OP that the logic is flawed
  3. The professor won't accept this anyway
Code:
seq 24 -1 10 | xargs -I '{}' grep -e '\[{}/Nov/2014:' access.log

This works and it's counting lines for timestamps from 24/Nov/2014 to 10/Nov/2014 (last 14 days).
Now imagine it's Nov 5 ...


Following hints should help you to do it right and to gain an accuracy to a second:

1. Get the last line of access.log (newest entry)
2. Cut the date-time-timezone stamp (10/Jun/2013:07:43:40 -0700)
3. Reformat above string to something you can feed to GNU date, e.g. 2013-06-10 07:43:40 OR 2013/06/10 07:43:40 OR 20130610 07:43:40 OR whatever
4. Using GNU date, convert the reformatted string to seconds since 1970-01-01
4.1 Subtract 60×60×24×14 seconds (14 days) from the result and put the result in a variable, eg. twoweeksago
6. In a while loop you then parse the access.log file line by line
6.1. You repeat the steps 2, 3 and 4 here
6.2. Pseudo code: if seconds greater than/equal to twoweeksago then increment counter
7. echo "$counter requests"

Hope this helps.
This User Gave Thanks to junior-helper For This Post:
scopiop
# 5  
Old 11-25-2014
If you rearrange the date into YYYY/MM/DD HH:MM:SS, it sorts alphabetically and can be compared by simple > >= < <= in awk.
Code:
$ cat dcomp.awk

BEGIN {
        FS="[ \t\\[\\]/:]+"
        split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", A);
        for(X in A)     MON[A[X]]=sprintf("%02d", X);
}

function ymd() {
        return(sprintf("%s/%s/%s %02d:%02d:%02d", $6, MON[$5], $4, $7, $8, $9));
}

(ymd() >= YYYYMMDD" 00:00:00") && (ymd() <= YYYYMMDD" 23:59:59") { C++; }

END {   print C+0       }

$ awk -f dcomp.awk YYYYMMDD="2013/06/10" datafile

10

$

This User Gave Thanks to Corona688 For This Post:
scopiop
# 6  
Old 11-28-2014
I have tried the approach to grep the date of the last line and cut out the Date field and I have problem of getting rid of the square bracket.

Code:
$cat /etc/httpd/logs/access_log | cut -d' ' -f4 
> [25/Nov/2014:12:00:01

How do I remove the square bracket from most left? I have tried to use grep but
Unsuccessful to get rid of the bracket. After removed the bracket I think I can use
date "14 days ago" to set a stop point where I can count for all requests between those
date. I hope I got the right approach from here.

Thank you for your help,
Scopiop
Last edited by scopiop; 11-28-2014 at 05:40 PM..
# 7  
Old 11-28-2014
Quote:
> I have tried the approach to grep the date of the last line
You don't need to cat the whole file to get the last line. Try
Code:
tail -n1 /etc/httpd/logs/access_log | cut -d' ' -f4

It will retrieve only the very last line and apply the cut command to it.

Quote:
> and I have problem of getting rid of the square bracket.
> How do I remove the square bracket from most left?
You could simply add | cut -c2- to the above command. It will cut from the second character on until the end of the line, thus removing the square bracket.

In a loop, the above really should be done by making use of bash's built-ins for better performance.

Code:
       ${parameter#word}
       ${parameter##word}
              Remove matching prefix pattern.

       ${parameter%word}
       ${parameter%%word}
              Remove matching suffix pattern.

Example:
Code:
$ dat=$(tail -n1 access.log)
$ echo "$dat"
127.0.0.1 - - [24/Nov/2014:07:13:55 -0800] "GET /something.php HTTP/1.1" 200 315 "-" "Foo browser/1.0 (<some operating system>) <some user agent>"
$ #remove everything up to the first square bracket
$ dat="${dat#*\[}"
$ echo "$dat"
24/Nov/2014:07:13:55 -0800] "GET /something.php HTTP/1.1" 200 315 "-" "Foo browser/1.0 (<some operating system>) <some user agent>"
$ #remove everything from the first space until the end of string/line
$ dat="${dat%% *}"
$ echo "$dat"
24/Nov/2014:07:13:55
$

Quote:
> I think I can use date "14 days ago" to set a stop point
Yes, you can. Consult this wiki page for the exact strftime format (https://en.wikipedia.org/wiki/Common_Log_Format), but I'm not sure how you plan to do the (exact) comparison.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl to adjust coordinates based on repeat string

In the file below I am trying to count the given repeats of A,T,C,G in each string of letters. Each sequence is below the > and it is possible for a string of repeats to wrap from the line above. For example, in the first line the last letter is a T and the next lines has 3 more. I think the below... (10 Replies)
Discussion started by: cmccabe
10 Replies

2. Shell Programming and Scripting

Single parse to grab two chunks of string

Hi all, I'm struggling with this task and have done alot of googling but not found the solution or atleast not found a way to combine them, i'm hoping someone here can help me out. I have a file that contains many line of config "below is two lines for an example" 323 => 1111,Terry berry... (3 Replies)
Discussion started by: 3sparky
3 Replies

3. Homework & Coursework Questions

How to use loop to repeat task?

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. How can i use loop to repeat task. 2.shirt=15 black=13.50 echo "how many shirt you want" read num echo echo "Please enter a choice" echo "1 ---> normal... (5 Replies)
Discussion started by: Tauatioti
5 Replies

4. Shell Programming and Scripting

Replace string, grab files, rename and move

Hello there! I'm having a lot of trouble writing a script. The script is supposed to: 1) Find all files with the name "Object.mtl" within each folder in the directory: /Users/username/Desktop/convert/Objects 2) Search and replace the string ".bmp" with ".tif" (without the quotations) 3)... (1 Reply)
Discussion started by: Blue Solo
1 Replies

5. Shell Programming and Scripting

Repeat using for loop

I have a file like this 2011-10-10 10:46:00,1-1-13-1-1,151510,ALCLA0A84D2C 2011-10-10 10:46:00,1-1-13-1-1,151520,65537 2011-10-10 10:46:00,1-1-13-1-1,151515,46932 2011-10-10 10:46:00,1-1-13-1-1,151521,32769 2011-10-10 10:46:00,1-1-13-1-1,151522,32769 2011-10-10... (4 Replies)
Discussion started by: LavanyaP
4 Replies

6. Shell Programming and Scripting

Matching 2 chars of a string that repeat

Hello Unix gurus, I have a gzipped file where each line contains 2 street addresses in the US. What I want to do is get a count for each state that does not match. What I have so far is: $ gzcat matched_10_09.txt.gz |cut -c 106-107,184-185 | head -5 CTCT CTNY CTCT CTFL CTMA This cuts... (5 Replies)
Discussion started by: sitney
5 Replies

7. Shell Programming and Scripting

Grab first or second line after a search string

In a shell script, I need to grab the first or second line after a search string in a file. For example: File.out: Random Info Manufacturer: XYZPDQ System Info Manufacturer: Hewlett-Packard Product Name: ProLiant I search for the word FILE, I want to be able to grab the line... (1 Reply)
Discussion started by: jwk1230
1 Replies

8. Homework & Coursework Questions

Date comparison with 'string date having slashes and time zone' in Bash only

1. The problem statement, all variables and given/known data: I have standard web server log file. It contains different columns (like IP address, request result code, request type etc) including a date column with the format . I have developed a log analysis command line utility that displays... (1 Reply)
Discussion started by: TariqYousaf
1 Replies

9. Shell Programming and Scripting

case loop... repeat on bad input?

I'm trying to get a case statement to start over if an undefined option is selected... But I am ata loss on how to actually do it. Here is a quick example of what I have. Echo "1) do this/n 2) Do that/n 3) Quit/n Make a selection/n" Read answer Case answer in 1) Dothid;; 2) Dothat;;... (3 Replies)
Discussion started by: trey85stang
3 Replies

10. Shell Programming and Scripting

String substitution on find results inside exec/xargs

What I'm trying to do is perform a copy, well a ditto actually, on the results of a find command, but some inline string substitution needs to happen. So if I run this code find ./ -name "*.tif" I get back these results. .//1234567.tif .//abcdefg.tif Now the action from exec or xargs I... (2 Replies)
Discussion started by: myndcraft
2 Replies
Login or Register to Ask a Question