Including EOL in egrep pattern for multiple lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Including EOL in egrep pattern for multiple lines
# 1  
Old 11-09-2012
Including EOL in egrep pattern for multiple lines

Hi all

I need your help to get a high-performance solution.
I am working on a extensive script to automate file restores using the bprestore tool on a Solaris 5.10 server (bash 3.00). I will only paste the needed parts of the script to avoid any confusion.
To use the script the user has to create a file "filelist" within his home directory, including the request number of the file restore on the first line, followed by all filenames which should be restored line by line, first.
The script gets these filenames by a for loop:
Code:
fcntr=0
flnms=("")
for flnm in `cat $HOME/filelist`
  do    
    if [ ${fcntr} -eq 0 ]
      then
        rnmbr=${flnm}
        fcntr=$((fcntr+1))
      else
        flnms=(${flnms[@]} ${flnm}) 
    fi
  done

Next step is to get all absolute paths for each file using the bplist tool:
Code:
sudo /usr/openv/netbackup/bin/bplist -l -B -R -C ${servername} -b -s ${startdate} -e ${enddate} ${stdsrchpth} | while read filename
  do 
    for file in ${flnms[@]}
      do
        echo ${filename} | grep  "/${file}$" >> ${tmplst}
      done
  done

or
Code:
for file in ${flnms[@]}
  do
    sudo /usr/openv/netbackup/bin/bplist -l -B -R -C ${servername} -b -s  ${startdate} -e ${enddate} ${stdsrchpth} | grep "/${file}$" | while read filename
      do
        echo ${filename} >> ${tmplst}
      done
  done

These would be solutions which work as they should... but:
bplist returns more than six million lines which means a very time consuming process.

I tried to use egrep:
Code:
sudo /usr/openv/netbackup/bin/bplist -l -B -R -C ${servername} -b -s  ${startdate} -e ${enddate} ${stdsrchpth} | egrep "\/file1\$|file2\$|etc.." >> ${tmplst}

Without the \$ part it worked quite fast, but I have to include \$ because the script should not match "file1.data.cdi" if it is searching for "file1.data". I wonder if there is a way to do it like this (including $ as EOL, not as EOF like it happens with the syntax above):
Code:
sudo /usr/openv/netbackup/bin/bplist -l -B -R -C ${servername} -b -s  ${startdate} -e ${enddate} ${stdsrchpth} | egrep ${flnms[@]} >> ${tmplst}

The script should go through the list of files from bplist only once and directly gets all absolute paths for the files within the array ${flnms[@]} using "\/file1\$|\/file1\$|etc..." as syntax.

Many thanks for your help.

Last edited by Anonym; 11-09-2012 at 11:49 AM..
# 2  
Old 11-09-2012
You could try:
Code:
sudo /usr/openv/netbackup/bin/bplist ...  | grep -f <(sed 's|^|/|; s|$|$|' "$HOME/filelist") > "${tmplst}"

# 3  
Old 11-09-2012
Your suggestion will not work. Let my explain why:
There are not only the filenames listet in ${HOME}/filelist (I will update the main post too). The first line of filelist contains the request number of the file restore which is needed to create a appropriate directory where the files should be restored to. There is no general syntax for the request numbers which could be captured by regex.
Code:
fcntr=0
flnms=("")
for flnm in `cat $HOME/filelist`
  do    
    if [ ${fcntr} -eq 0 ]
      then
        rnmbr=${flnm}
        fcntr=$((fcntr+1))
      else
        flnms=(${flnms[@]} ${flnm}) 
    fi
  done

I could create a new temporary file and use it as you have mentioned above:
Code:
myfls="${stddestpath}/myfls.txt"
for flnm in ${flnms[@]}
  do
    echo -e "${flnm}" >> ${myfls}
  done

sudo /usr/openv/netbackup/bin/bplist ...  | grep -f <(sed 's|^|/|; s|$|$|' "${myfls}") > "${tmplst}"

But it does not seem to be a very good solution because I want to avoid any sources of errors. The script should create as few temporary files as possible (two are needed by bprestore, one to change the destination path of restore and one which contains a list of all absolute paths).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

2. Shell Programming and Scripting

Exclude lines in a file with matches with multiple Strings using egrep

Hi I have a txt file and I would like to use egrep without using -v option to exclude the lines which matches with multiple Strings. Let's say I have some text in the txt file. The command should not fetch lines if they have strings something like CAT MAT DAT The command should fetch me... (4 Replies)
Discussion started by: Sathwik
4 Replies

3. Shell Programming and Scripting

Match Pattern and print pattern and multiple lines into one line

Hello Experts , require help . See below output: File inputs ------------------------------------------ Server Host = mike id rl images allocated last updated density vimages expiration last read <------- STATUS ------->... (4 Replies)
Discussion started by: tigerhills
4 Replies

4. Shell Programming and Scripting

Merging two tables including multiple ocurrence of column identifiers and unique lines

I would like to merge two tables based on column 1: File 1: 1 today 1 green 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new 4 a lot 5 sometimes 6 at work (4 Replies)
Discussion started by: BSP
4 Replies

5. Shell Programming and Scripting

Delete to eol after pattern with sed

working on interactive ftp script saving log and want to remove the password from log so "quote PASS foofoo" would become "quote PASS XXXXXXX" replacing or removing the password is of no matter have tried several sed commands but have only been successful with character matching not pattern ... (2 Replies)
Discussion started by: ppaprota
2 Replies

6. Shell Programming and Scripting

Search for Pattern and Print including Lines in between

Gurus, I have a big file that needs to be sorted out and I cant figure out what to do. The file name is as below: Name: xxxx yyyy nnnn Description: dfffgs sdgsgsf hsfhhs afgghhjdgj fjklllll gsfhfh Updated: jafgadsgg gsg Corrected: date today The file consists of line like these. ... (13 Replies)
Discussion started by: The One
13 Replies

7. Shell Programming and Scripting

How to egrep multiple pattern

Hi everyone i want to write a script to grep multiple pattern from all the file from a dir. for example I want to get all the record number from XML file who's last name is asd, smith, dfrt,gokul,and sinha. I tried egrep('sinha'|'gokul'|'asd') but it is not working also i tried saving... (2 Replies)
Discussion started by: revertback
2 Replies

8. Shell Programming and Scripting

sed: Find start of pattern and extract text to end of line, including the pattern

This is my first post, please be nice. I have tried to google and read different tutorials. The task at hand is: Input file input.txt (example) abc123defhij-E-1234jslo 456ujs-W-abXjklp From this file the task is to grep the -E- and -W- strings that are unique and write a new file... (5 Replies)
Discussion started by: TestTomas
5 Replies

9. Shell Programming and Scripting

Pattern search in multiple lines

Hi, I have to search those statements from the file which starts from "shanky"(only shanky, shanky09 or 09shanky is not allowed) and ends with ");". These two string can be in a same line or different line. And also i have to negate those lines which starts with #. Can any one please give me... (2 Replies)
Discussion started by: shanky09
2 Replies

10. Shell Programming and Scripting

removing pattern which is spread in multiple lines

I have several huge files wich contains oracle table creation scripts as follows: I would need to remove the pattern colored in red above. Any sed/awk/pearl code will be of much help. Thanks (2 Replies)
Discussion started by: sabyasm
2 Replies
Login or Register to Ask a Question