Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

SH script to split squid log by date

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting SH script to split squid log by date
# 1  
Old 01-24-2009
Question SH script to split squid log by date
Hi,

I really need your help to make a script to split a large squid's log file into a multiple files, each of them containing the log entries for every logged date.

To achieve the result I planned to use the function "date" with the first log field as a parameter using this syntax:

date -d @(ten number squid timestamp field) +%F

to obtain the desired filename for splitted logs.

Example:
# date -d @1232163496 +%F
will return: 2009-01-16

The idea is to analize the first field of each line in the log file and append whole line to the corresponding file.

Any help would be apreciated,

I'm a newbie ¿Someone can help me?
Last edited by _MCRH_; 01-28-2009 at 05:02 PM.. Reason: Before close it
# 2  
Old 01-25-2009
How does your file looke like and what is the desired output? Do you get the epoch time from a file?

Regards
# 3  
Old 01-26-2009
Hi Franklin52

I want to analize squid's log file "access.log", it look like this:

1232935790.356 1795 192.168.0.10 TCP_MISS/200 551 GET http://www.cubachat ...
1232935790.383 1645 192.168.0.55 TCP_MISS/302 619 GET http://images.imagef ...
1232935791.271 888 192.168.0.55 TCP_MISS/302 618 GET http://images.imagef ...
1232935796.064 1444 192.168.0.55 TCP_MISS/302 619 GET http://images.imagef ...
1232935797.477 1412 192.168.0.55 TCP_MISS/302 618 GET http://images.imagef ...
1232955307.426 527 192.168.0.10 TCP_CLIENT_REFRESH_MISS/407 2116 GET Free Download Manager - absolutely free download accelerator and manager ...
1232955307.703 804 192.168.0.10 TCP_MISS/407 2072 GET http://webmail.rimed ...
1232955315.375 1266 192.168.0.10 TCP_MISS/407 1614 GET http://webmail.rimed ...
1232955322.071 1633 192.168.0.10 TCP_MISS/407 1614 GET http://webmail.rimed ...

The first field of the log is the date and time in UNIX format (seconds since 1/1/1970). Analizing this field (as proposed in my first post), two dates as registered: jan 25 2009 and jan 26 2009, thus two files must be generated:

2009-01-25 containing:
1232935790.356 1795 192.168.0.10 TCP_MISS/200 551 GET http://www.cubachat ...
1232935790.383 1645 192.168.0.55 TCP_MISS/302 619 GET http://images.imagef ...
1232935791.271 888 192.168.0.55 TCP_MISS/302 618 GET http://images.imagef ...
1232935796.064 1444 192.168.0.55 TCP_MISS/302 619 GET http://images.imagef ...
1232935797.477 1412 192.168.0.55 TCP_MISS/302 618 GET http://images.imagef ...

and
2009-01-26 containing:
1232955307.426 527 192.168.0.10 TCP_CLIENT_REFRESH_MISS/407 2116 GET Free Download Manager - absolutely free download accelerator and manager ...
1232955307.703 804 192.168.0.10 TCP_MISS/407 2072 GET http://webmail.rimed ...
1232955315.375 1266 192.168.0.10 TCP_MISS/407 1614 GET http://webmail.rimed ...
1232955322.071 1633 192.168.0.10 TCP_MISS/407 1614 GET http://webmail.rimed ...


Any help would be apreciated,

Thanks
# 4  
Old 01-27-2009
Try one of these solutions (not tested). If you have gawk:

Code:
gawk -F"." '{d=strftime("%Y-%m-%d", $1);print > d}' file

Otherwise:

Code:
awk -F"." '{"date -d @" $1 " +%F" | getline d;print > d;close("date -d @" $1 " +%F")}' file

Regards
Last edited by Franklin52; 01-28-2009 at 04:41 PM.. Reason: typo format strftime
# 5  
Old 01-28-2009
MySQL It works Ok!!, Thanks!!
Hi, Smilie

Thank you so much!!

It works very good.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

I need a Script to read Log string and check date at the same time

I need to check 1 log file, which is logging: 2014-08-18T09:10:39+02:00 user: XXXXX START FEATURE 2014-08-18T09:10:39+02:00 user: XXXXX FINISH FEATURE I first need to check that the START FEATURE starts and finish on the same time/date for the same user, which is different each time START... (2 Replies)
Discussion started by: TheBest43
2 Replies

2. Shell Programming and Scripting

Squid proxy shell script

i need squid proxy log file daily auto generate by shell script (1 Reply)
Discussion started by: sanjeetkumar086
1 Replies

3. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

4. UNIX for Advanced & Expert Users

Change Log Path for Squid on Centos 6

I searched and did not find a useful answer. Can someone please tell me the best practice to change the log directory from /var/log to /opt/squid? (4 Replies)
Discussion started by: glev2005
4 Replies

5. IP Networking

Squid vs iptables = no Squid access.log?

Hello, I have a pretty useless satellite link at home (far from any civilization), so I wanted to set up caching in order to speed things up. My Squid 2.6 runs "3128 transparent" and is set up quite well on a separate machine. I also have my dd-wrt router to move all port 80 traffic through... (0 Replies)
Discussion started by: theWojtek
0 Replies

6. Shell Programming and Scripting

Split Date/time data to two

Hi all, I have logs(in a log file) with the following structure 20100916011501559;0.812;null;TRUE;;FALSE;0.812;0;0;;19 20100916011504762;0.015;null;TRUE;;FALSE;0;4|4;0.015;;4 20100916011504762;0;null;TRUE;;FALSE;0;0;0;;4 20100916011501731;3.343;null;TRUE;;FALSE;3.156;131|65;0.172;;11... (14 Replies)
Discussion started by: babai
14 Replies

7. Shell Programming and Scripting

Filtering the yesterdays date from log files via script.

hi All, I have this sample text file - access.log: Jan 18 21:34:29 root 209.151.232.70 Jan 18 21:34:40 root 209.151.232.70 Jan 18 21:34:43 root 209.151.232.70 Jan 18 21:34:56 root 209.151.232.70 Jan 18 21:35:10 root 209.151.232.70 Jan 18 21:35:23 root 209.151.232.70 Jan 18 21:36:04 root... (2 Replies)
Discussion started by: linuxgeek
2 Replies

8. Shell Programming and Scripting

Split the file based on date value

Hi frnds, I have flat file as . Say : output-file1.txt Output-file2.txt (1 Reply)
Discussion started by: Gopal_Engg
1 Replies

9. Shell Programming and Scripting

Script to copy log files with today's date

I am a newbie to scripting. I need a korn shell script to copy log files of current day to archive folder and rename with current days date stamp. I would really appreciate your help. File structure is as follows. Everyday files get overwritten, so I need copy to a archive directory and... (3 Replies)
Discussion started by: mdncan
3 Replies

10. Shell Programming and Scripting

Split the access.log based on date

I am trying to get the content of Apache access.log file for the current date for viewing purposes. I can get it with the following sed command sed -n '/09\/Oct\/2008/,/09\/Oct\/2008/p' access.log | less now I want to enhance it such that it will automatically take current date instead of... (4 Replies)
Discussion started by: ysprathap
4 Replies
Login or Register to Ask a Question