Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

parsing multi-date text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting parsing multi-date text file
# 1  
Old 07-29-2010
parsing multi-date text file
Hi all:

Trying to parse a log file of rsync activity to get the amount of date being transferred. The log file contains multiple dates and what I am trying to do is get the file sizes for the current date.

What I have been trying to do is pipe it through awk but I am having trouble retrieving the output.

What I am wanting, the Total file size: for the current date. So for today, it will parse out the rsync entries for today (date +%A %Y/%b/%d) get the file size and add each for a total.

I have been trying to use the BEGIN and set the RS = "=". No joy. Is there a better way? Or am I on right track but my awk skills are insufficient?

Many thanks!


Example log file entries below:
Moderator's Comments:
Mod Comment Please, for code and listings like this, use code tags

Code:
===================================
Saturday  2010/Jun/12  16:01:24
INPUT=/incoming/check_bro
INFILE=incoming
OUTPUT=/incoming/check_bro
DEST=/incoming
SRC=/incoming/check_bro

Virus check OK

receiving file list ... done

Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 13
Total bytes sent: 8
Total bytes received: 21

sent 8 bytes  received 21 bytes  58.00 bytes/sec
total size is 0  speedup is 0.00

===================================
Saturday  2010/Jun/12  16:17:12
INPUT=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip
INFILE=incoming
OUTPUT=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip
DEST=/incoming
SRC=/incoming/collect-CX0825BVA00141-2010-06-12-05-06-38.zip

Virus check OK

receiving file list ... done
collect-CX0825BVA00141-2010-06-12-05-06-38.zip

Number of files: 1
Number of files transferred: 1
Total file size: 85776271 bytes
Total transferred file size: 85776271 bytes
Literal data: 85776271 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.332 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 85786932

sent 42 bytes  received 85786932 bytes  24510564.00 bytes/sec
total size is 85776271  speedup is 1.00

===================================
Thursday  2010/Jul/29  09:08:30
INPUT=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz
INFILE=incoming
OUTPUT=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz
DEST=/incoming/state_farm
SRC=/incoming/state_farm/DXiStats-CX0934BVA00518-20100729-070024.tar.gz

Virus check OK

receiving file list ... done
DXiStats-CX0934BVA00518-20100729-070024.tar.gz

Number of files: 1
Number of files transferred: 1
Total file size: 96257450 bytes
Total transferred file size: 96257450 bytes
Literal data: 96257450 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.369 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 96269391

sent 42 bytes  received 96269391 bytes  27505552.29 bytes/sec
total size is 96257450  speedup is 1.00

===================================
Thursday  2010/Jul/29  09:28:50
INPUT=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip
INFILE=incoming
OUTPUT=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip
DEST=/incoming
SRC=/incoming/collect-CX0851BVA00415-2010-07-29-09-42-05.zip

Virus check OK

receiving file list ... done
collect-CX0851BVA00415-2010-07-29-09-42-05.zip

Number of files: 1
Number of files transferred: 1
Total file size: 227957285 bytes
Total transferred file size: 227957285 bytes
Literal data: 227957285 bytes
Matched data: 0 bytes
File list size: 127
File list generation time: 0.854 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 42
Total bytes received: 227985302

sent 42 bytes  received 227985302 bytes  30398045.87 bytes/sec
total size is 227957285  speedup is 1.00

Last edited by pludi; 07-29-2010 at 01:58 PM..
# 2  
Old 07-29-2010
Hi,

Try this.
filesize.pl

Code:
#!/usr/bin/perl

$/="===================================";
$curr_date=`date '+%A %Y\\/%b\\/%d'`;
chop ($curr_date);
while (<>) {
if (/$curr_date(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.*)\n(.+?):\s(.+?)\sbytes/){
$total = $total + ${17};
}
}
print "Total file size ",$total," Bytes for today"

Code:
perl filesize.pl inputfile

This User Gave Thanks to pravin27 For This Post:
raggmopp
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing text file

Hi Friends, I am back for the second round today - :D My input text file is this way Home friends friendship meter Tools Mirrors Downloads My Data About Us Help My own results BLAT Search Results ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO STRAND ... (7 Replies)
Discussion started by: jacobs.smith
7 Replies

2. Shell Programming and Scripting

Parsing text file

I'm totally stumped with how to handle this huge text file I'm trying to deal with. I really need some help! Here is what is looks like: ab1ba67c331a3d731396322fad8dd71a3b627f89359827697645c806091c40b9 0.2 812a3c3684310045f1cb3157bf5eebc4379804e98c82b56f3944564e7bf5dab5 0.6 0.6... (3 Replies)
Discussion started by: comp8765
3 Replies

3. Programming

Parsing a Text file using C++

I was trying to parse the text file, which will looks like this ###XYZABC#### ############ int = 4 char = 1 float = 1 . . ############ like this my text file will contains lots of entries and I need to store these entries in the map eg. map.first = int and map.second = 4 same way I... (5 Replies)
Discussion started by: agupta2
5 Replies

4. Shell Programming and Scripting

Parsing Log File Based on Date & Error

I'm still up trying to figure this out and it is driving me nuts. I have a log file which has a basic format of this... 2010-10-10 22:25:42 Init block 'UA Deployment Date': Dynamic refresh of repository scope variables has failed. The ODBC function has returned an error. The database... (4 Replies)
Discussion started by: k1ko
4 Replies

5. Shell Programming and Scripting

Need help parsing a text file

I have a text file: router1#sh ip blah blah | incl --- Gi2/8 10.60.4.181 --- 10.60.123.175 11 0000 0000 355K Gi2/8 10.60.83.28 --- 224.10.10.26 11 F9FF 3840 154K Gi2/8 10.60.83.198 --- ... (1 Reply)
Discussion started by: streetfighter2
1 Replies

6. UNIX for Dummies Questions & Answers

Parsing Date to a file name

Hi All There is a file "apple_2008-08-15.log". I have to use grep on this file to collect my test log. There are 45 such files. Is there a command that I can use to dynamically substitute the daily date as part of the file name? As of now Iam renaming the file to the new date and running my... (6 Replies)
Discussion started by: pk_eee
6 Replies

7. Shell Programming and Scripting

Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names

I'm trying to write a script that will look in an /exports folder for the oldest export file and move it to a /staging folder. "Oldest" in this case is actually determined by date information embedded in the file names themselves. Also, the script should only move a file from /exports to... (6 Replies)
Discussion started by: nikosey
6 Replies

8. Shell Programming and Scripting

Parsing text from file

Any ideas? 1)loop through text file 2)extract everything between SOL and EOL 3)output files, for example: 123.txt and 124.txt for the file below So far I have: sed -n "/SOL/,/EOL/{p;/EOL/q;}" file Here is an example of my text file. SOL-123.go something goes here something goes... (0 Replies)
Discussion started by: ndnkyd
0 Replies

9. Shell Programming and Scripting

Text File Parsing

Hey Guys.I am a newbie on Bash Shell Scripting and Perl.And I have a question about file parsing. I have a log file which contains reports about a communication device.I need to take some of the reports from the log file.Its hard to explain the issue.but shortly I can say that, the reports has a... (2 Replies)
Discussion started by: Djlethal
2 Replies

10. Shell Programming and Scripting

parsing a system log file via the 'date' command

Hello, I'm trying to update some scripts here that parse our system logs daily. They report information just fine... but they just report too much info. Specifically, if there's been some failed login attempts on several different days (say Monday and Tuesday), when I get the report from... (5 Replies)
Discussion started by: cjones
5 Replies
Login or Register to Ask a Question