Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash script to extract paragraph with globs in it

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script to extract paragraph with globs in it
# 8  
Old 02-27-2017
Hi drysdalk,

I modified your script and ran it again. I have a doubt that it is to do something with my permissions or selinux or something. I created the temp file manually under /tmp but when trying to create it through the script, it gave me an error. As a result, I removed all the substitutions and put in the file names directly, and this time they worked. But this is really stupid and confusing if I cant do that.

Code:
#!/bin/bash

input=/home/dsiddiqui/basic_bash/scripts/example.txt
#tmp=`/tmp/script.tmp`

temp=/home/dsiddiqui/basic_bash/scripts/tempfile
while read -r line
do
        case "$line" in
                "******* BEGIN MESSAGE *******")
                        echo "$line" > /home/dsiddiqui/basic_bash/scripts/tempfile
                        ;;
                "******* END MESSAGE *******")
                        echo "$line" >> /home/dsiddiqui/basic_bash/scripts/tempfile

                        if /bin/grep ^Original\ presentment\ Not\ Found\ \!$ /home/dsiddiqui/basic_bash/scripts/tempfile >/dev/null 2>/dev/null
                        then
                                /bin/cat /home/dsiddiqui/basic_bash/scripts/tempfile
                                echo
                        fi
                        ;;
                *)
                        echo "$line" >> /home/dsiddiqui/basic_bash/scripts/tempfile
                        ;;
        esac

done < $input


Regarding your suggestions, one thing for sure is BEGIN MESSAGE and the END MESSAGE will always occur as a block. No other lines will have the same words again as these are log messages for an application and it is not programmed like that.
# 9  
Old 02-27-2017
Hi,

OK, great - glad to know you've got a working script now.

On the topic of your own script and the variables not working, I think it's because you've not actually defined the variable you're using.

You seem to have commented out tmp and defined temp instead, but unless you also changed every occurrence of $tmp in the script to $temp (or even better just don't bother defining temp at all and modify the tmp definition instead), then that wouldn't actually work, since you'd be trying to use an un-defined variable.

---------- Post updated at 02:49 PM ---------- Previous update was at 02:45 PM ----------

Also just noticed: you wouldn't want to write:

tmp=`/tmp/script.tmp`

That's using backticks, so again would try to run an external binary or script called /tmp/script.tmp which almost certainly does not exist.

Instead, you don't want backticks here, and just a simple:

tmp=/tmp/script.tmp

will suffice.
This User Gave Thanks to drysdalk For This Post:
dsid
# 10  
Old 02-27-2017
Hi drysdalk,

This is so stupid of me. I have totally lost it, it seems Smilie . Made the changes that you suggested in your earlier post, and everything is working fine.

Code:
input=/home/dsiddiqui/basic_bash/scripts/example.txt
tmp=/tmp/tmpfile.scpt

while read -r line
do
        case "$line" in
                "******* BEGIN MESSAGE *******")
                        echo "$line" > "$tmp"
                        ;;
                "******* END MESSAGE *******")
                        echo "$line" >> "$tmp"

                        if /bin/grep ^Original\ presentment\ Not\ Found\ \!$ "$tmp" >/dev/null 2>/dev/null
                        then
                                /bin/cat "$tmp"
                                echo
                        fi
                        ;;
                *)
                        echo "$line" >> "$tmp"
                        ;;
        esac

done < $input

Thanks a lot for your patience.
# 11  
Old 02-27-2017
Hi.
Quote:
Originally Posted by drysdalk
One last version, this time avoiding the use of grep (which may make things run a little faster if you have a great deal of data to get through):


Code:

#!/bin/bash

input=example.txt
tmp=`/bin/tempfile`

while read -r line
do
case "$line" in
*BEGIN\ MESSAGE*)
echo "$line" > "$tmp"
;;
*END\ MESSAGE*)
echo "$line" >> "$tmp"

if /bin/grep ^Original\ presentment\ Not\ Found\ \!$ "$tmp" >/dev/null 2>/dev/null
then
/bin/cat "$tmp"
echo
fi
;;
*)
echo "$line" >> "$tmp"
;;
esac
done < "$input"
And yet grep is still there ... cheers, drl
# 12  
Old 02-27-2017
Ah, yes. I was meaning without the use of if...grep evaluation rather than case (which for some reason I didn't think to keep with the first time I re-wrote it). Still, one grep is better than three Smilie
# 13  
Old 02-27-2017
Hi.

For comparison, here is a single command of the grep family which extracts bounded blocks containing the required string. It does not use temporary files. Ignoring the scaffolding and supporting code, this is the single command that obtains the results:
Code:
cgrep -D -w 'BEGIN MESSAGE' +w 'END MESSAGE' 'Original presentment Not Found' $FILE

Here is the demonstation script and the resulting solution:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate bounded block extraction, cgrep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C specimen cgrep dixf

FILE=${1-data1}

pl " Input data file $FILE:"
specimen $FILE

pl " Results:"
cgrep -D -w 'BEGIN MESSAGE' +w 'END MESSAGE' 'Original presentment Not Found' $FILE

pl " Details for utility cgrep:"
dixf cgrep

exit 0

which produces:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.7 (jessie) 
bash GNU bash 4.3.30
specimen (local) 1.17
cgrep ATT cgrep 8.15
dixf (local) 1.42

-----
 Input data file data1:
Edges: 5:0:5 of 85 lines in file "data1"
******* BEGIN MESSAGE *******

       Station / User:  129   800013   Batch Processing
 SDate / Time / PDate:  26.02.2017 17:07:05   26.02.2017
       Current System:  XXXXXX Production System       
   ---
Time: [17:06:59]

003','040

******* END MESSAGE *******

-----
 Results:
******* BEGIN MESSAGE *******

       Station / User:  129   800013   Batch Processing
 SDate / Time / PDate:  26.02.2017 17:07:05   26.02.2017
       Current System:  XXXXXX Production System       
   Institution Number:  00000043
Application / Version:  abc-inw   30.66.36   Release A (OMNI)
        Function Name:  FindOriginalPresentment

Warning !

Original presentment Not Found !

Institution No (Original Tran): [00000043]
Charge back slip: [70527509216]
Acquirer Reference: [85470355344549150697093]
Presentment Slip: [N/A]
Transaction Class: [002 - Clearing transactions]
Transaction Category: [001 - Presentments]
File Institution No: [00000043]
File No: [00041926]

******* END MESSAGE *******
******* BEGIN MESSAGE *******

       Station / User:  129   800013   Batch Processing
 SDate / Time / PDate:  26.02.2017 17:07:05   26.02.2017
       Current System:  XXXXXX Production System       
   Institution Number:  00000043
Application / Version:  abc-inw   30.66.36   Release A (OMNI)
        Function Name:  FindOriginalPresentment

Warning !

Original presentment Not Found !

Institution No (Original Tran): [00000043]
Charge back slip: [70527509216]
Acquirer Reference: [85470355344549150697093]
Presentment Slip: [N/A]
Transaction Class: [002 - Clearing transactions]
Transaction Category: [001 - Presentments]
File Institution No: [00000043]
File No: [00041926]

******* END MESSAGE *******

-----
 Details for utility cgrep:
cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYSV ...)
Home    : http://sourceforge.net/projects/cgrep/

I have installed cgrep on numerous systems, and, while a c compiler is needed, the compilation is a single step.

I have also benchmarked cgrep and it is as fast (in broad terms) as the fastest grep instances available.

Of course, if you do not wish to obtain and compile the code, or you do not do this kind of task frequently, then you are better off using the other suggestions.

Best wishes ... cheers, drl
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to extract a paragraph containing a given string?

Hello: Have a very annoying problem: Need to extract paragraphs with a specific string in them from a very large file with a repeating record separator. Example data: a file called test.out CREATE VIEW view1 AS something FROM table1 ,table2 as A, table3 (something FROM table4) FROM... (15 Replies)
Discussion started by: delphys
15 Replies

2. UNIX for Dummies Questions & Answers

Extract paragraph that contains a value x<-30

I am using OSX. I have a multi-mol2 file (text file with coordinates and info for several molecules). An example of two molecules in the file is given below for molecule1 and molecule 2. The total file contains >50,000 molecules. I would like to extract out and write to another file only the... (2 Replies)
Discussion started by: Egy
2 Replies

3. Shell Programming and Scripting

how to write bash script that will automatically extract zip file

i'm trying to write a bash script that that will automatically extract zip files after the download. i writed this script #!/bin/bash wget -c https://github.com/RonGokhle/kernel-downloader/zipball/master CURRENDIR=/home/kernel-downloader cd $CURRENDIR rm $CURRENDIR/zipfiles 2>/dev/null ... (2 Replies)
Discussion started by: ron gokhle
2 Replies

4. Shell Programming and Scripting

How to extract multiple line in a paragraph? Please help.

Hi all, The following lines are taken from a long paragraph: Labels of output orbitals: RY* RY* RY* RY* RY* RY* 1\1\GINC-COMPUTE-1-3\SP\UB3LYP\6-31G\C2H5Cr1O1(1+,5)\LIUZHEN\19-Jan-20 10\0\\# ub3lyp/6-31G pop=(nbo,savenbo) gfprint\\E101GECP\\1,5\O,0,-1.7 ... (1 Reply)
Discussion started by: liuzhencc
1 Replies

5. UNIX for Dummies Questions & Answers

Bash script to extract spf records

Hello I am trying to generate a script to run on worldwide firewalls. I need the spf block for large sites like google, etc so I can essentially whitelist google sites for users. (Google here is just an example...) Right now I am just testing Bash oneliners to see how I can isolate the... (1 Reply)
Discussion started by: mbubb
1 Replies

6. Shell Programming and Scripting

script to list out the output in one paragraph

Hi All, I want to run 5 `ps -ef | grep ` cmds in one script and i want the script to give me return code 0 if everything is OK. If it notices one of the processes is not there, it will prompt me the process name and advice me to check it. I've wrote a script that separates the output but I want... (2 Replies)
Discussion started by: fara_aris
2 Replies

7. Linux

Extract a paragraph

Hi , Unix.com has been life saver for me I admit :) I am trying to extract a paragraph based on matching pattern "CREATE TABLE " from a ddl file . The paragraphs are seperated by blank line . Input file is #cat zip.20080604.sql1 CONNECT TO TST103 SET SESSION_USER OPSDM002 ... (2 Replies)
Discussion started by: capri_drm
2 Replies

8. Shell Programming and Scripting

how to extract paragraphs from file in BASH script followed by prefix ! , !! and !!!

I]hi all i am in confusion since last 2 days :( i posted thraed yesterday and some friends did help but still i couldnt get solution to my problem let it be very clear i have a long log file of alkatel switch and i have to seperate the minor major and critical alarms shown by ! , !! and !!!... (6 Replies)
Discussion started by: nabmufti
6 Replies

9. Shell Programming and Scripting

script for a 3 line paragraph

i would like to ask how to make a script that in evry 3 lines of my paragraph(below) it would appear like this: $ cat myparagraph this is line 1 this is line 2 this is line3 this is line 4 this is 5 this 6 this is 7 this 8 ==================================================== $ cat... (2 Replies)
Discussion started by: invinzin21
2 Replies

10. Shell Programming and Scripting

*.pm globs without quoting, *.pl doesn't.

Can someone explain the following? I can use find on *.pm without quotes, but find on *.pl makes on error, I need quotes for the second version. What's up with that? $find -name *.pm ./tieProxyStatus/Status.pm $find -name *.pl find: paths must precede expression Usage: find $find... (2 Replies)
Discussion started by: tphyahoo
2 Replies
Login or Register to Ask a Question