Extract section of file based on word in section


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract section of file based on word in section
# 1  
Old 09-20-2010
Extract section of file based on word in section

I have a list of Servers in no particular order as follows:

Code:
virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"

And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection).

Code:
9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

For each SERVERNAME in ${virtualMachines} I want to extract each section beginning with "Normal backup" and ending with "Duration" to a file.

The Server name is a constant and will always appears in the "Normal backup/Duration" block of output (typically as "Copying $SERVERNAME" but this can change if it has an error but is always there). So it's kind of like a grep of the $SERVERNAME that also grabs the "Normal backup -> Duration" section that the $SERVERNAME exists within. This also needs to append to the file as sometimes the backup of each server might run more than once and produce more than 1 "Normal backup -> Duration" section.

Another way to describe it would be...

Output only "Normal backup -> Duration" section where $SERVERNAME = IIBSBS and append output to /tmp/vdrlog-IIBSBS.txt

e.g. it would look something like this. My script is just to try and illustrate what I want and does not work.

Code:
#!/bin/bash

virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
backupLog="/tmp/backup.log"

for i in ${virtualMachines} ; do
        grep $i ${backupLog} | awk
                '$4 == "Normal" && $5 == "backup"{f=1; c++}
                f{print >> "/tmp/vdrlog-"'"$i"'"".txt}
                $4 ~ /Duration/{f=0}'
done

Code:
cat /tmp/vdrlog-IIBSBS.txt
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

Code:
cat /tmp/vdrlog-IIBVICDMS01.txt
9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)

Code:
cat /tmp/vdrlog-IIBVICMA01.txt
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)

I previously had great help from Franklink52 in this thread which is what was using to grab the sections but I am updating the script so it accounts for situations where sometimes the backup will run more than once or if errors occur in the backup.
# 2  
Old 09-20-2010
Code:
#!/usr/bin/env ruby 
vmachines=%w(IIBVICDMS01 IIBVICMA01)
#vmachines=%w(IIBSBS IIBVICDMS01 IIBVICMA01)
s=""
while line=gets
  host=line.split[-1] if line =~ /Copying/
  if line.match("Duration")
    setter = 0 and s << line
    if vmachines.include?(host)
      File.open("/tmp/vdrlog-"+host+".txt","w") {|x| x.puts s }
    end
    s=""
  elsif line.match("Normal")
    setter=1
  end
  s <<line if setter
end

Code:
<your command/script that generate the output> ... | ruby myscript.rb

# 3  
Old 09-20-2010
If you don't need the "Normal backup" line then this can do:
Code:
virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
for i in ${virtualMachines}; do awk "/$i/,/Duration/" /tmp/backup.log > vdrlog-$i; done

In case that this line is important for you, let me know I'll come up with some Perl.
This User Gave Thanks to bartus11 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copying section of file based on search criteria

Hi Guru's, I am new to unix scripting. I have a huge file with user details in it(file2) and I have another file with a list of users(file1). Script has to search a user from file1 and get all the associated lines from file2. Example: fiel1: cn=abc cn=DEF cn=xyx File 2: dn:... (10 Replies)
Discussion started by: Samingla
10 Replies

2. Shell Programming and Scripting

Extract record from file based on section.

input file output file (1 Reply)
Discussion started by: lathigara
1 Replies

3. Shell Programming and Scripting

Fetch a section from a file

Hi, I have a file like... $cat file1 +++++++++++++++++++ client1 +++++++++++++++++++++++++++++ col1 col2 col3 ------ ----- ----- (0 rows affected) ========================================================= +++++++++++++++++++ client1 +++++++++++++++++++++++++++++ col1 col2 col3... (6 Replies)
Discussion started by: sam05121988
6 Replies

4. Shell Programming and Scripting

Prepend first line of section to each line until the next section header

I have searched in a variety of ways in a variety of places but have come up empty. I would like to prepend a portion of a section header to each following line until the next section header. I have been using sed for most things up until now but I'd go for a solution in just about anything--... (7 Replies)
Discussion started by: pagrus
7 Replies

5. Shell Programming and Scripting

Deleting a section based on search from other file

Hi Everyone, I need some help to accomplish the below. help is highly appriciated. I have a 45 mb file with ldap entries. Each user entry is separated by a string # entry-id: 1 and so on. Some of the entries has a string xyz: true. I want to delete the section if the user section has xyz: true... (6 Replies)
Discussion started by: Samingla
6 Replies

6. Shell Programming and Scripting

Format output from the file to extract "date" section

Hello Team , I have to extract date section from the below file output. The output of the file is as shown below. I have to extract the "" this section from the above output of the file. can anyone please let me know how can we acheive this? (4 Replies)
Discussion started by: coolguyamy
4 Replies

7. Shell Programming and Scripting

Selecting A Section of A File Based On the Time Within It

Hi, I have a file formated like this: John 7.22 2010-01-25_17:01:36 George 8.22 2010-01-25_17:02:36 Bob 9.62 2010-01-25_17:04:36 Jane 10.11 2010-01-25_17:05:36 Emma 4.52 2010-01-25_17:01:36 What I want to do is cut out only the entries that have... (2 Replies)
Discussion started by: Donkey25
2 Replies

8. Shell Programming and Scripting

Search and extract by section from configuration

Hi, I understand either AWK or SED can do this, but I not sure how to extract the following configuration in section. Meaning when I need to find code with " ip helper-address 192.168.11.2" , it would start from "interface Serial0/0" and "interface FastEthernet0/1". Only displaying both section... (2 Replies)
Discussion started by: haphazard
2 Replies

9. UNIX for Dummies Questions & Answers

how can i extract text section via grep

hi, i have a very long text file. i need to extract with grep command a certain part. for example text file include 1ooo rows: 1.... 2... 3... . . . 1000 i want to view with grep only rows 50-100. any ideas will be appreciated thanks... (8 Replies)
Discussion started by: meny
8 Replies

10. Shell Programming and Scripting

sed & awk--get section of file based 2 params

I need to get a section of a file based on 2 params. I want the part of the file between param 1 & 2. I have tried a bunch of ways and just can't seem to get it right. Can someone please help me out.....its much appreciated. Here is what I have found that looks like what I want....but doesn't... (12 Replies)
Discussion started by: Andy Cook
12 Replies
Login or Register to Ask a Question