Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract section of file based on word in section

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract section of file based on word in section
# 1  
Old 09-20-2010
Extract section of file based on word in section
I have a list of Servers in no particular order as follows:

Code:
virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"

And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection).

Code:
9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

For each SERVERNAME in ${virtualMachines} I want to extract each section beginning with "Normal backup" and ending with "Duration" to a file.

The Server name is a constant and will always appears in the "Normal backup/Duration" block of output (typically as "Copying $SERVERNAME" but this can change if it has an error but is always there). So it's kind of like a grep of the $SERVERNAME that also grabs the "Normal backup -> Duration" section that the $SERVERNAME exists within. This also needs to append to the file as sometimes the backup of each server might run more than once and produce more than 1 "Normal backup -> Duration" section.

Another way to describe it would be...

Output only "Normal backup -> Duration" section where $SERVERNAME = IIBSBS and append output to /tmp/vdrlog-IIBSBS.txt

e.g. it would look something like this. My script is just to try and illustrate what I want and does not work.

Code:
#!/bin/bash

virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
backupLog="/tmp/backup.log"

for i in ${virtualMachines} ; do
        grep $i ${backupLog} | awk
                '$4 == "Normal" && $5 == "backup"{f=1; c++}
                f{print >> "/tmp/vdrlog-"'"$i"'"".txt}
                $4 ~ /Duration/{f=0}'
done

Code:
cat /tmp/vdrlog-IIBSBS.txt
9/17/2010 8:00:02 PM: Normal backup using VDRBACKUPS (Execution unit 10)
9/17/2010 8:00:55 PM: Copying IIBSBS
9/17/2010 8:01:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS-flat.vmdk" using "Network"
9/17/2010 8:58:37 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_1-flat.vmdk" using "Network"
9/17/2010 9:47:28 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBSBS/IIBSBS_2-flat.vmdk" using "Network"
9/17/2010 9:49:05 PM: Execution completed successfully

9/17/2010 9:49:05 PM: Completed: 9 files, 514.6 GB
9/17/2010 9:49:05 PM: Performance: 4879.8 MB/minute
9/17/2010 9:49:05 PM: Duration: 01:49:01 (00:01:02 idle/loading/preparing)

Code:
cat /tmp/vdrlog-IIBVICDMS01.txt
9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS (Execution unit 12)
9/17/2010 8:01:49 PM: Copying IIBVICDMS01
9/17/2010 8:01:56 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01-000002-flat.vmdk" using "Network"
9/17/2010 8:06:06 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICDMS01/IIBVICDMS01_1-000002-flat.vmdk" using "Network"
9/17/2010 8:19:00 PM: Execution completed successfully

9/17/2010 8:19:00 PM: Completed: 7 files, 790.1 GB
9/17/2010 8:19:00 PM: Performance: 47446.3 MB/minute
9/17/2010 8:19:00 PM: Duration: 00:18:54 (00:01:50 idle/loading/preparing)

Code:
cat /tmp/vdrlog-IIBVICMA01.txt
9/17/2010 8:00:04 PM: Normal backup using VDRBACKUPS (Execution unit 11)
9/17/2010 8:01:06 PM: Copying IIBVICMA01
9/17/2010 8:01:14 PM: Performing incremental back up of disk "[IIBVICSAN01-SAS1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:26:23 PM: Performing incremental back up of disk "[IIBVICSAN01-SATA1-1] IIBVICMA01/IIBVICMA01-flat.vmdk" using "Network"
9/17/2010 8:36:39 PM: Execution completed successfully

9/17/2010 8:36:39 PM: Completed: 7 files, 362.1 GB
9/17/2010 8:36:39 PM: Performance: 10461.5 MB/minute
9/17/2010 8:36:39 PM: Duration: 00:36:35 (00:01:09 idle/loading/preparing)

I previously had great help from Franklink52 in this thread which is what was using to grab the sections but I am updating the script so it accounts for situations where sometimes the backup will run more than once or if errors occur in the backup.
# 2  
Old 09-20-2010
Code:
#!/usr/bin/env ruby 
vmachines=%w(IIBVICDMS01 IIBVICMA01)
#vmachines=%w(IIBSBS IIBVICDMS01 IIBVICMA01)
s=""
while line=gets
  host=line.split[-1] if line =~ /Copying/
  if line.match("Duration")
    setter = 0 and s << line
    if vmachines.include?(host)
      File.open("/tmp/vdrlog-"+host+".txt","w") {|x| x.puts s }
    end
    s=""
  elsif line.match("Normal")
    setter=1
  end
  s <<line if setter
end

Code:
<your command/script that generate the output> ... | ruby myscript.rb

# 3  
Old 09-20-2010
If you don't need the "Normal backup" line then this can do:
Code:
virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"
for i in ${virtualMachines}; do awk "/$i/,/Duration/" /tmp/backup.log > vdrlog-$i; done

In case that this line is important for you, let me know I'll come up with some Perl.
This User Gave Thanks to bartus11 For This Post:
jelloir
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copying section of file based on search criteria

Hi Guru's, I am new to unix scripting. I have a huge file with user details in it(file2) and I have another file with a list of users(file1). Script has to search a user from file1 and get all the associated lines from file2. Example: fiel1: cn=abc cn=DEF cn=xyx File 2: dn:... (10 Replies)
Discussion started by: Samingla
10 Replies

2. Shell Programming and Scripting

Extract record from file based on section.

input file output file (1 Reply)
Discussion started by: lathigara
1 Replies

3. Shell Programming and Scripting

Fetch a section from a file

Hi, I have a file like... $cat file1 +++++++++++++++++++ client1 +++++++++++++++++++++++++++++ col1 col2 col3 ------ ----- ----- (0 rows affected) ========================================================= +++++++++++++++++++ client1 +++++++++++++++++++++++++++++ col1 col2 col3... (6 Replies)
Discussion started by: sam05121988
6 Replies

4. Shell Programming and Scripting

Prepend first line of section to each line until the next section header

I have searched in a variety of ways in a variety of places but have come up empty. I would like to prepend a portion of a section header to each following line until the next section header. I have been using sed for most things up until now but I'd go for a solution in just about anything--... (7 Replies)
Discussion started by: pagrus
7 Replies

5. Shell Programming and Scripting

Deleting a section based on search from other file

Hi Everyone, I need some help to accomplish the below. help is highly appriciated. I have a 45 mb file with ldap entries. Each user entry is separated by a string # entry-id: 1 and so on. Some of the entries has a string xyz: true. I want to delete the section if the user section has xyz: true... (6 Replies)
Discussion started by: Samingla
6 Replies

6. Shell Programming and Scripting

Format output from the file to extract "date" section

Hello Team , I have to extract date section from the below file output. The output of the file is as shown below. I have to extract the "" this section from the above output of the file. can anyone please let me know how can we acheive this? (4 Replies)
Discussion started by: coolguyamy
4 Replies

7. Shell Programming and Scripting

Selecting A Section of A File Based On the Time Within It

Hi, I have a file formated like this: John 7.22 2010-01-25_17:01:36 George 8.22 2010-01-25_17:02:36 Bob 9.62 2010-01-25_17:04:36 Jane 10.11 2010-01-25_17:05:36 Emma 4.52 2010-01-25_17:01:36 What I want to do is cut out only the entries that have... (2 Replies)
Discussion started by: Donkey25
2 Replies

8. Shell Programming and Scripting

Search and extract by section from configuration

Hi, I understand either AWK or SED can do this, but I not sure how to extract the following configuration in section. Meaning when I need to find code with " ip helper-address 192.168.11.2" , it would start from "interface Serial0/0" and "interface FastEthernet0/1". Only displaying both section... (2 Replies)
Discussion started by: haphazard
2 Replies

9. UNIX for Dummies Questions & Answers

how can i extract text section via grep

hi, i have a very long text file. i need to extract with grep command a certain part. for example text file include 1ooo rows: 1.... 2... 3... . . . 1000 i want to view with grep only rows 50-100. any ideas will be appreciated thanks... (8 Replies)
Discussion started by: meny
8 Replies

10. Shell Programming and Scripting

sed & awk--get section of file based 2 params

I need to get a section of a file based on 2 params. I want the part of the file between param 1 & 2. I have tried a bunch of ways and just can't seem to get it right. Can someone please help me out.....its much appreciated. Here is what I have found that looks like what I want....but doesn't... (12 Replies)
Discussion started by: Andy Cook
12 Replies
Login or Register to Ask a Question