Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extracting data from rsync log

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting data from rsync log
# 1  
Old 06-03-2015
Extracting data from rsync log
Hi

I have a daily rsync that runs and i am trying to find a easy way of extracting the start time and end time of the sync and extract the details of how much data was copied.

I would like to use this information to see if i can increase the amount of rsyncs that run in a day.

so the log will reside in the normal place /var/log/rsync.log

I like to extract the information and cat to another file. The idea is i could gather the data daily over the course of a week.

Thanks in advance
Treds
# 2  
Old 06-03-2015
Please give us a sample input and expected output. What have you already tried?
# 3  
Old 06-05-2015
hi Jim

start of a log

Code:
2015/06/05 21:14:06 [32390] building file list
2015/06/05 21:14:06 [32395] building file list
2015/06/05 21:14:06 [32400] building file list
2015/06/05 21:14:06 [32405] building file list
2015/06/05 21:14:06 [32410] building file list
2015/06/05 21:14:18 [32390] .d..t...... STUDIOS/HM_MASTERS/2012_MASTERS/
2015/02/07 21:14:18 [32390] .d..t...... STUDIOS/HM_MASTERS/2012_MASTERS/1202_SEASON_START/Files from HQ/ME/
2015/02/07 21:14:18 [32390] .d..t...... STUDIOS/HM_MASTERS/2012_MASTERS/1202_SEASON_START/Files from HQ/ME/KSA/

End of log
Code:
LR_CRP_ASSET/
2015/06/05 10:18:27 [27914] .d..t...... TIER2/TAG_REPOSITORY_JOB_ASSETS/97/04/47/97044710/JOB_ASSETS/RGB_ASSET/
2015/06/05 10:18:27 [27914] .d..t...... TIER2/TAG_REPOSITORY_JOB_ASSETS/97/04/47/97044710/JOB_ASSETS/RGB_CRP_ASSET/
2015/06/05 10:18:27 [27914] Number of files: 9172
2015/06/05 10:18:27 [27914] Number of files transferred: 0
2015/06/05 10:18:27 [27914] Total file size: 0 bytes
2015/06/05 10:18:27 [27914] Total transferred file size: 0 bytes
2015/06/05 10:18:27 [27914] Literal data: 0 bytes
2015/06/05 10:18:27 [27914] Matched data: 0 bytes
2015/06/05 10:18:27 [27914] File list size: 242568
2015/06/05 10:18:27 [27914] File list generation time: 4.261 seconds
2015/06/05 10:18:27 [27914] File list transfer time: 0.000 seconds
2015/06/05 10:18:27 [27914] Total bytes sent: 247360
2015/06/05 10:18:27 [27914] Total bytes received: 4797
2015/06/05 10:18:27 [27914] sent 247360 bytes  received 4797 bytes  16268.19 bytes/sec
2015/06/05 10:18:27 [27914] total size is 0  speedup is 0.00

I have tried using sed but to be honest my scripting skills are limited.
thanks
treds
Last edited by vbe; 06-05-2015 at 08:41 AM.. Reason: Code tags please
# 4  
Old 06-05-2015
Not sure if I understood the request correctly. Try
Code:
awk 'NR==1 {print $1,$2} END {print $1, $2} /bytes sent/ {print $NF}' /var/log/rsync.log
2015/06/05 21:14:06
247360
2015/06/05 10:18:27

Redirect to taste.
The start and end times seem a bit odd. Sure they're from one log?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Error while extracting data from log file

I am running awk command to extract data from log file to calculate last 15 minutes log using below command and now i am getting bellow error: awk '$0>=$from' from=$(`date -u +"####<%d-%b-%Y %H:%M:%S o'clock GMT>"-15min`) test.log Error: date: 0551-402 Invalid character in date/time... (8 Replies)
Discussion started by: oberoi1403
8 Replies

2. Shell Programming and Scripting

Tail over rsync data

Hi, I have written a shell script (that will be called in every 5 seconds) that used rsync to copy logs of our production server from server A(production) to server B (Dev), because we don't allow team members to hit server A. The shell script is working fine. But we are facing a problem that... (2 Replies)
Discussion started by: rish_max
2 Replies

3. Shell Programming and Scripting

Extracting Delimiter 'TAG' Data From log files

Hi I am trying to extract data from within a log file and output format to a new file for further manipulation can someone provide script to do this? For example I have a file as below and just want to extract all delimited variances of tag 32=* up to the delimiter "|" and output to a new file... (2 Replies)
Discussion started by: Buddyluv
2 Replies

4. Shell Programming and Scripting

Extracting the last 10mins worth of data in a log file

Hi all, Hope someone here will be able to help me. Our system has some scripts that are run from a cron job every ten mins and is used to see how many error there are in that time frame. Problem is that in the scripts grep is used to find the data, but as the day goes on these log file grow to a... (7 Replies)
Discussion started by: Goldengreen
7 Replies

5. Shell Programming and Scripting

extracting data

I have a txt file of the following format >ab_ qwerty >rt_ hfjkil >Ty2 hglashglkasghkf; >P2 aklhfklflkkgfgkfl >ui_ vnllkdskkkffkfkkf >we32 vksksjksj;lslsf'sk's's .... ..... I want to split this big file based on the header (>) (5 Replies)
Discussion started by: Lucky Ali
5 Replies

6. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

7. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

8. UNIX for Advanced & Expert Users

rsync data corruption

Dear Peoples I am using the following command to transfer the files inbetween two servers, i am using this command in a crontab for doing it in every 1 hour on all days. rsync --stats --archive --verbose --compress --force --rsh=ssh --exclude-from=/root/cfg/mkt_scn.exclude... (1 Reply)
Discussion started by: thameema
1 Replies

9. UNIX for Dummies Questions & Answers

Extracting Data Using SED

Given the following text in a file named extract.txt: listenPort:=25 smtpDestination:=2 enableSSL:= I am trying to extract only the value 2 following smtpDestination:= Someone had suggested I use: sed -e "s/^smtpDestination:=\(.*\)$/\1/" extract.txt but this returns: listenPort:=25 2 ... (2 Replies)
Discussion started by: cleanden
2 Replies

10. Shell Programming and Scripting

extracting data from files..

frnds, I m having prob woth doing some 2-3 task simultaneously... what I want is... I have lots ( lacs ) of files in a dir... I want.. these info from arround 2-3 months files filename convention is - abc20080403sdas.xyz ( for todays files ) I want 1. total no of files for 1 dec... (1 Reply)
Discussion started by: clx
1 Replies
Login or Register to Ask a Question