Need Help with a Data Extract script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need Help with a Data Extract script
# 1  
Old 09-14-2011
Need Help with a Data Extract script

hi,

i'am trying to write a shell program which takes a data element
from one file(var/tmp/usr1/pools)and search that data element
in a zipped file in archive location/usr01/archive/PSta*.Z).

Code:
cat var/tmp/usr1/pools
FC5173
FI5178
BE5221
FE5229
ST1604

Code:
ls /usr01/archive/PSta*.Z
/usr01/archive/PStat.dat.021711.001609.Z
/usr01/archive/PStat.dat.021811.001608.Z
/usr01/archive/PStat.dat.021911.001608.Z
/usr01/archive/PStat.dat.022311.001608.Z
/usr01/archive/PStat.dat.022411.001608.Z
/usr01/archive/PStat.dat.022511.001608.Z
/usr01/archive/PStat.dat.022611.001608.Z
/usr01/archive/PStat.dat.022811.190155.Z

below program writes that zipped filename into a temp file(/var/tmp/usr1/fcount) if it finds the occurence.

Code:
#!/usr/bin/sh
fl='/usr01/archive/'
for i in `cat /var/tmp/usr1/pools`
do
for j in `ls ${fl}/*.Z | tr -s ' '| cut -d" " -f9 | cut -d\/ -f7`
do
count=`zcat $fl/$j | grep "$i" | wc -l` 
if [ $count -eq 1 ] ; then
echo ${j} | grep "PSta*" >> /var/tmp/usr1/fcount
fi
done
done

lets consider that the data element found two files having it's occurence and wrote
two filenames to the temp file, Now my goal is to find the latest filename having that
occurence(data element).
the program is getting all the file names into /var/tmp/usr1/fcount where i 'am not able to get the latest
file for each data element
i've been trying without any success..will appreciate if any one help me out here.
thanks
kmr023


Moderator's Comments:
Mod Comment Please use code tags!

Last edited by radoulov; 09-14-2011 at 05:00 PM..
# 2  
Old 09-14-2011
That's a useless use of cat and useless use of backticks, inefficient and potentially dangerous since there's a limit to how much data can be put into one variable.

That's also a useless use of ls *. Just do for j in ${fl}/*.

'for j in *' has the same size limit as the backticks problem. If you have more than a dozen or two files, or the number of files continually grows, this may eventually become a problem.

I don't quite understand the purpose of for j in `ls ${fl}/*.Z | tr -s ' '| cut -d" " -f9 | cut -d\/ -f7`. Are you rewriting your own basename? I don't see any purpose for basename here either.

I don't think grep "PSta*" does what you think it does. grep uses regular expressions, not shell globs, so "a*" here means "zero or more 'a' characters". So it'd match any line containing PSt at all. Just the expression 'PSta' should suffice if you want to find any line with those four letters.

You don't need to pipe grep into wc -l to measure whether grep found any results. grep's return code tells you whether it found anything, which you can feed directly into an ordinary if statement:
Code:
if grep pattern file > /dev/null
then
        echo 'Found'
else
        echo 'not found'
fi

grep can do nearly your entire program by itself, using -f and -F to tell it to read patterns from files instead of the commandline.

My version:
Code:
for FILE in /usr01/archive/PStat*.Z
do
        # Pipe the decompressed file into grep.  Use a pattern file (-f var/tmp/usr1/pools),
        # match fixed strings instead of regexes (-F), and stop on the
        # first match(-m 1) so scanning the entire file isn't always necessary.
        if zcat "$FILE" | grep -m 1 -F -f var/tmp/usr1/pools > /dev/null
        then
                # Print the filename if a match is found.  All goes into /tmp/$$
                echo "$FILE"
        fi
done > /tmp/$$


# "echo a b c | xargs ls" is equivalent to "ls a b c".
#
# List all the files in /tmp/$$, sort by time (-t), take only the most
# recent( | head -n 1)
xargs ls -t < /tmp/$$ | head -n 1
# delete the temp file
rm -f /tmp/$$

---------- Post updated at 09:56 AM ---------- Previous update was at 09:52 AM ----------

You could dispense with the temp file completely actually.

Code:
for FILE in /usr01/archive/PStat*.Z
do
        # Pipe the decompressed file into grep.  Use a pattern file (-f var/tmp/usr1/pools),
        # match fixed strings instead of regexes (-F), and stop on the
        # first match(-m 1) so scanning the entire file isn't always necessary.
        if zcat "$FILE" | grep -m 1 -F -f var/tmp/usr1/pools > /dev/null
        then
                # Print the filename if a match is found.  All goes into xargs
                echo "$FILE"
        fi
done | xargs ls -t | head -n 1


Last edited by Corona688; 09-14-2011 at 01:02 PM.. Reason: fixed paths
# 3  
Old 09-14-2011
i've tried the snippet above, but no luck, getting below error

Code:
grep: illegal option -- m
Usage: grep -hblcnsviw pattern file . . .


Last edited by radoulov; 09-14-2011 at 05:00 PM.. Reason: Code tags!
# 4  
Old 09-14-2011
This is why you should always say what your system is in your opening post... Your system doesn't have some options I was using.

What kind of rotten grep doesn't even have -f, though! Smilie That's going to make the code many, many times slower...

If your system has egrep, you could try using that in the same code. Otherwise:

Code:
for FILE in /usr01/archive/PStat*.Z
do
        zcat "$FILE" > /tmp/$$

        while read STR
        do
                if grep -l "$STR" "/tmp/$$" > /dev/null
                then
                        echo "$FILE"
                        break
                fi
        done < var/tmp/usr1/pools
done | xargs ls -t | head -n 1

rm -f /tmp/$$

# 5  
Old 09-14-2011
you can ignore "-m 1" from the grep command and still use the code.

--ahamed
# 6  
Old 09-14-2011
Quote:
Originally Posted by ahamed101
you can ignore "-m 1" from the grep command and still use the code.
Worth a shot I guess, but it doesn't say it has -f or -F...
# 7  
Old 09-14-2011
no luck, even i took off "-m 1".
i'am using solaris 8. so, any thing compatable help be greatly appreciated
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script to extract data in a file

I have this 5GB file, and i want to extract from the file particulars pattern. this is my script: // count=`grep -wc "MSISDN" file_name` k=1 >OUTPUT >OUTPUT_Final while do cat file_name | awk -F":" -v var="$k" '$1=="MSISDN" {m++}m==var{print; exit}' >> OUTPUT cat file_name |awk -F":"... (33 Replies)
Discussion started by: gillesi
33 Replies

2. Shell Programming and Scripting

Shell script to extract data from csv file

Hi everyone, I have a csv file which has data with different heading and column names as below. Static Data Ingested ,,,,,,,,,,,,Known Explained Rejections Column_1,column_2,Column_3,Column_4,,Column_6,Column_7,,% Column_8,,Column_9 ,Column_10 ,... (14 Replies)
Discussion started by: Vivekit82
14 Replies

3. Shell Programming and Scripting

help with data extract script

hi, i'am trying to write a script which takes a data element from one file and searches in a certain file path and after it finds it it redirects the file names to a new file...Now i have to pick the up the last modified file's Timestamp(MMDDYY) from the files collected for each data element. ... (2 Replies)
Discussion started by: kmr023
2 Replies

4. Shell Programming and Scripting

Script to extract certain data

Hi Not to good with data extraction . I have some output that looks like this 1301925447 1 2 2 mtlhqmst 16215531 0 0 *NULL* bpduplicate Initiating optimized duplication from @aaagc to @aaaeG 1301925484 1 2 4 snccprd6 16215531 0 0 hwd-rg-f04b bpdm begin copying backup id... (5 Replies)
Discussion started by: bombcan1
5 Replies

5. Shell Programming and Scripting

Script to extract data

Hi, Pleae find the attached files. I want to extract all the table names that has got * mark within its last column data. The doc file will give you a clear picture of what I am trying to do. Tables names in oval should be extracted in other file because it has got * mark in its last column data (5 Replies)
Discussion started by: db2cap
5 Replies

6. Shell Programming and Scripting

need a shell script to extract data from a log file.

If I have a log like : Mon Jul 19 05:07:34 2010; TCP; eth3; 52 bytes; from abc to def Mon Jul 19 05:07:35 2010; UDP; eth3; 46 bytes; from aaa to bbb Mon Jul 19 05:07:35 2010; TCP; eth3; 52 bytes; from def to ghi I will need an output like this : Time abc to def... (1 Reply)
Discussion started by: hitha87
1 Replies

7. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

8. Shell Programming and Scripting

Perl script for extract data from xml files

Hi All, Prepare a perl script for extracting data from xml file. The xml data look like as AC StartTime="1227858839" ID="88" ETime="1227858837" DSTFlag="false" Type="2" Duration="303" /> <AS StartTime="1227858849" SigPairs="119 40 98 15 100 32 128 18 131 23 70 39 123 20 120 27 100 17 136 12... (3 Replies)
Discussion started by: allways4u21
3 Replies

9. Shell Programming and Scripting

extract data from xml- shell script using awk

Hi, This is the xml file that i have. - <front-servlet platform="WAS4.0" request-retriever="SiteMinder-aware" configuration-rescan-interval="60000"> <concurrency-throttle maximum-concurrency="50" redirect-page="/jsp/defaulterror.jsp" /> - <loggers> <instrumentation... (5 Replies)
Discussion started by: nishana
5 Replies

10. Shell Programming and Scripting

How to extract data using UNIX shell script?

Hello All, I am starting with UNIX. Any help is highly appreciated. How to extract data using UNIX shell script? And how do you export data using UNIX shell scripts into Microsoft Excel format? Thank you. (3 Replies)
Discussion started by: desiondarun
3 Replies
Login or Register to Ask a Question