Extract values from log file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract values from log file
# 1  
Old 08-08-2007
Extract values from log file

I would like to write a shell script that will parse through a file similar to the sample below. The data in the file is redirected from rsync into a log file. I would like to call a shell script to parse through and pick out the number beside the percent sign inside the parentheses in the last occurrence in the file. So in the sample text below, I would like it to pick out the "0.1" in "(21, 0.1% of 38220)".

What is the best way to accomplish this? Using sed? Can I search for the last occurrence explicitly or do I need to do several pipe operations to accomplish this?

Here is a section of the file that I want to be able to parse:

Documents/Birthday Invites/Birthday Invite.psd
32768 0% 1.04MB/s 0:00:20
131072 0% 60.18kB/s 0:06:08
4521984 20% 950.50kB/s 0:00:18
8716288 39% 1.07MB/s 0:00:12
12910592 57% 1.12MB/s 0:00:08
17104896 76% 1.23MB/s 0:00:04
21299200 95% 1.13MB/s 0:00:00
22298611 100% 1.13MB/s 0:00:18 (17, 0.1% of 38220)
Documents/Birthday Invites/._Birthday Invite.psd
25190 43% 0.00kB/s 0:00:00
57958 100% 10.42MB/s 0:00:00 (18, 0.1% of 38220)
Documents/Birthday Invites/Invites1.pdf
32768 0% 27.63kB/s 0:04:35
65536 0% 55.08kB/s 0:02:17
4489216 58% 1.23MB/s 0:00:02
7659360 100% 1.91MB/s 0:00:03 (19, 0.1% of 38220)
Documents/Birthday Invites/Invites2.pdf
32768 0% 0.00kB/s 0:00:00
4489216 58% 1.90MB/s 0:00:01
7662562 100% 2.81MB/s 0:00:02 (20, 0.1% of 38220)
Documents/Birthday Invites/Invites3.pdf
32768 0% 0.00kB/s 0:00:00
4489216 58% 1.76MB/s 0:00:01
7657907 100% 2.65MB/s 0:00:02 (21, 0.1% of 38220)
Documents/Birthday Invites/Invites4.pdf
32768 0% 0.00kB/s 0:00:00
4489216 58% 1.20MB/s 0:00:02
# 2  
Old 08-08-2007
Code:
grep '(.*)' input_file | tail -1 | sed 's/.*(.* \(.*\)%.*/\1/'

# 3  
Old 08-08-2007
Thanks, I will give that a try.
# 4  
Old 08-09-2007
Thank you for your help. Because there is the possibility of error messages which have parentheses in them, I needed to modify your shell script a little. But essentially, it's the same thing and it works perfectly.

Code:
egrep '([0-9]+, [0-9.]+% of [0-9]+)' input_file | tail -1 | sed 's/.*(.* \(.*\)%.*/\1/'

This way I am guaranteed only to filter out the progress percentages. I will deal with error messages separately. Could you clarify how you select out the number before the percentage sign in sed? I am assuming it has something to do with the \(.*\)%. Why do you use escaped parentheses here?
# 5  
Old 08-10-2007
answer and post out my concern,pls help me

Follow is my code, but i do not know why awk can only locate to the above line of the last line, but can not locate the last line. Pls help me.

Code:
sed '/(/!d' a > c
line=`cat c | wc -l`
cat c | awk 'BEGIN{print "'"$line"'"}
{
print NR
if (NR=="'"$line"'"-1)
print $0
}' > ff
cat ff | sed 's/.*,\(.*\)%.*/\1/'

output::
5
Code:
1
2
3
4
 0.1
5

Anyone pls help me why i can only locate to the second-last line not the last line.
Any suggestion will be appreciate!
# 6  
Old 08-10-2007
try this
parameter1=`grep "(.*)" inputfile |tail -1|cut -f6 -d" "`
parameter2=${parameter1%"%"}
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

To extract values after the maximum value in a txt file

Hello, I'm new to scripting and I need to write a bash script. Here is example of file on which I'm working: 0.3092381 0.3262799 0.3425480 0.3578379 0.3719490 0.3846908 0.3958855 0.4053738 0.4130160 0.4186991 0.4223357 ... (1 Reply)
Discussion started by: jeo_fb
1 Replies

2. Shell Programming and Scripting

Extract values from xml file script

Hi, please help on this. I want extract values of xml file structure and print in determined way. <ProjectName> --> only appears once <StructList> --> is the top node <Struct> node --> could be more than 1 NameID, STX, STY, PRX, PRY --> appears only 1 time within each <Struct> node... (10 Replies)
Discussion started by: Ophiuchus
10 Replies

3. Shell Programming and Scripting

Find and extract values from one file and update other

Gents, Kindly can you help me to update a file extracting the data from other file. I have: file1 The key in this file is substr($0,4,21), and I need to update the columns 6 and 7 using the information for file2. S 21133.00 21535.00 1 0 919088.8 1843754.5 ... (2 Replies)
Discussion started by: jiam912
2 Replies

4. Shell Programming and Scripting

Extract values froma line in file

p.txt T|DCNT=100|RECCHK=22222.2|PERCHK=32323|# I want to extract the value of 100 22222.2 and 32323 and assign it to variable x1,y1,z1 x=`cut -f2 -d "=" p.txt` x1=`echo $x | cut -f1 -d "|" ` y=`cut -f3 -d "=" p.txt` y1=`echo $y | cut -f1 -d "|" ` z=`cut -f4 -d "=" p.txt` z1=`echo $z... (3 Replies)
Discussion started by: w020637
3 Replies

5. Shell Programming and Scripting

Gawk - to extract values from multi lined file -I

Hi, Request your help in getting help with the below text formatting using awk. I am still learning awk and your help here is appreciated. Thanks in advance. Desireoutput ---------------- Virtual Pool Destination Profile Profile Profile Profile 1. virtual-1 pool-1 212.254.110.174:https... (2 Replies)
Discussion started by: pratheeshp
2 Replies

6. Shell Programming and Scripting

Gawk - to extract values from multi lined file

Hi, I am new to awk and trying to extract some specific fields from the a large file. Can you please help me to write gawk code displaying the out put in the below format: Desired Output: name fallback_ip member member www-trymps.extlb.plstry.com-pool-1 180.254.112.50 ... (4 Replies)
Discussion started by: pratheeshp
4 Replies

7. Shell Programming and Scripting

Extract values from an XML File

Hi, I need to capture all the attributes with delete next to it. The source XML file is attached. The output should contain something like this below: Attributes = legacyExchangeDN Action = Delete Username = Hero Joker Loginid = joker09 OU =... (4 Replies)
Discussion started by: prvnrk
4 Replies

8. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26... (5 Replies)
Discussion started by: simonsimon
5 Replies

9. Shell Programming and Scripting

to extract specific values twice in a file

Hi Friends, I have a file with the following values.. xyz.txt,12345.xml abc.txt,04567.xml cde.txt,12134.xml I would like to extract all the 2nd column values twice as shown in the example like 12345,12345.xml 04567,04567.xml 12134,12134.xml Please advice!! In the formus one of... (7 Replies)
Discussion started by: techmoris
7 Replies

10. Shell Programming and Scripting

I need to extract last column of a file and compare the values

Hi, I am new to unix and I need help in solving below mentioned issue, really appreciate ur help. I have a file sam, john, 2324, 07142007 tom, thomson, 2343, 07142007 john, scott, 2478, 07142007 its a comma delimited file, I need to extract the last column from each line and this... (4 Replies)
Discussion started by: vukkusila
4 Replies
Login or Register to Ask a Question