grep or awk? Post: 1394

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers grep or awk? Post 1394 by on Friday 2nd of March 2001 03:42:53 PM

03-02-2001

Guest

Here's my problem:
1. I have a file that contains say for this example, three records, each twenty bytes long:
CustNum CustName
1111111111abcdefghij
2222222222abcdefghij
3333333333abcdefghij

2. I have two other very large files (over 500,000 records) one is 500 bytes, the other is 200 bytes long. These two files contain the CustNum from the first file as well as MANY more that I don't want.

3. I want to extract out the CustNum from the first file, then do a loop thru the other two files, matching the CustNum and only writting out those matching records all 500 or 200 bytes.
Essentially, I want to reduce the 100,000 record file to a managable amount.

I have tried cut to extract the records out to a variable, then loop with grep, but the results produce a file of one continuous record. No newline? Should I use awk instead?

Any help would be appreciated......
Here is my code:

#! /bin/ksh
# get_bids_autorenew.sh

#== Local Variables ==#
datadir="/ias/users/app4dxh/data"
driver="$datadir/driver.dat"
file_1="tcpcsm2.data"
file_2="tcpvrm2.data"
num=0
stat=0

Get_Bids()
{
cd $biddir

if [ -s $driver ]
then
bids=`cut -c1-10 $driver`
stat=$?
else
stat=$?
echo "Function: $0 - No data found in $driver or file does not exist."
echo "Aborting script with a status of $stat"
exit 1
}

Match_Files()
{
#== For each bid picked up, check each of the CP and ==#
#== SM files for a match and just write those records. ==#

for i in $bids
do
match=`grep -s $i $file`
stat=$?
while :
do
case "$stat" in
0) echo $match >> file_$num.new
echo Status is $stat
break;;
1) break;;
2) echo "Function: $0 - The file $file is not accessible - grep status is $stat"
break;;
esac
done
done

}

#==Main==#
Get_Bids

#== For each File, execute the Match_Files function ==#
cd $datadir
for data in "$file_1" "$file_2"
do
set $data
file=$1
let num="$num + 1"
Match_Files
done
exit 0

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[grep awk cut] > awk

Hi, I'm very new to scripting. grep $s $filename | awk '{print $2}' | cut -c 1-8 How can I optimize this using a single awk? I tried: awk '/$s/ {print $2}' $filename | cut -c 1-8 However didn't work, I think the awk is not recognizing $s and the verbal is something else.

2. UNIX for Dummies Questions & Answers

grep and awk

I have grep MHz psrinfo-v.out it gives The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, how to get instead of these 4 lines: CPU speed: 3000 MHz i.e. CPU...

3. Shell Programming and Scripting

MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else

Hi Guys, I need to set the value of $7 to zero in case $7 is NULL. I've tried the below command but doesn't work. Any ideas. thanks guys. MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else { print $7}}' ` Harby.

4. Shell Programming and Scripting

Is it better to grep and pipe to awk, or to seach with awk itself

This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what...

5. Shell Programming and Scripting

Read content between xml tags with awk, grep, awk or what ever...

Hello, I trying to extract text that is surrounded by xml-tags. I tried this cat tst.xml | egrep "<SERVER>.*</SERVER>" |sed -e "s/<SERVER>$.*$<\/SERVER>/\1/"|tr "|" " " which works perfect, if the start-tag and the end-tag are in the same line, e.g.: <tag1>Hello Linux-Users</tag1> ...

6. Shell Programming and Scripting

AWK/GREP: grep only lines starting with integer

I have an input file 12.4 1.72849432773174e+01 -7.74784188610632e+01 12.5 9.59432114416327e-01 -7.87018212757537e+01 15.6 5.20139995965960e-01 -5.61612429666624e+01 29.3 3.76696387248366e+00 -7.42896194101892e+01 32.1 1.86899877018077e+01 -7.56508762501408e+01 35 6.98857157014640e+00...

7. Shell Programming and Scripting

awk / grep

how do I change this line to use the awk command RC19=`grep -c "Broken pipe" $FTP_OUT` Code tags please

8. UNIX for Dummies Questions & Answers

Piping grep into awk, read the next line using grep

Hi, I have a number of files containing the information below. """"" Fundallinfo 6.3950 14.9715 14.0482 """"" I would like to grep for Fundallinfo and use it to read the next line? I ideally would like to read the three numbers that follow in the next line and...

9. Shell Programming and Scripting

sed and awk usage to grep a pattern 1 and with reference to this grep a pattern 2 and pattern 3

Hi , I have a file where i have modifed certain things compared to original file . The difference of the original file and modified file is as follows. # diff mir_lex.c.modified mir_lex.c.orig 3209c3209 < if(yy_current_buffer -> yy_is_our_buffer == 0) { --- >...

10. UNIX for Beginners Questions & Answers

Using awk instead of grep -f

Hi Guys. I am trying to count occurances of patterns(occurance can be anywhere in file2) from file1 in file2. file1 is god god pod rod file2 is iamgod iamgod podrod 123rod456 output should be god 2 god 2 pod 1 rod 2 I am not good at awk but i figured out this command.it doesnt...

LEARN ABOUT DEBIAN

aoe-stat

aoe-stat(8)						      System Manager's Manual						       aoe-stat(8)

NAME

       aoe-stat - print aoe device status report

SYNOPSIS

       modprobe aoe
       aoe-stat
       env sysfs_dir=/sys aoe-stat

DESCRIPTION

       The aoe-stat script collects information on ATA over Ethernet devices from sysfs.

       For each AoE device the kernel has discovered, there is one row in the script's output.	Each row has the following columns.

       devicename
	      The device name is of the form eX.Y,
	       with X being the AoE device shelf address, and Y being the AoE slot address.

       size   The size of the AoE device is in gigabytes (billions of bytes).

       ifname The network interface name is printed in the third column.

       payload
	      The  number  of  bytes  read  from  or written to the storage target in each AoE packet appears in the fourth column, unless the aoe
	      driver does not export this information.

       status The device status is in the last column.	Possible values are  up,  down, and  down,closewait.  The "up" status means the aoe driver
	      considers  this  device  ready for I/O.  The "down" status means the opposite.  The "down,closewait" status means that some software
	      still has the device open, and when this straggler closes the device, it will enter the "down" state.

ENVIRONMENT VARIABLES

       If the sysfs_dir variable is set in the environment, it will override the default location where  aoe-stat  will  look  for  sysfs,  namely
       /sys.

WARNINGS

       If the minor device number of a device node does not match that of its namesake, aoe-stat will print a warning as shown below.

	      nai:~# aoe-stat
		    e0.3	 0.104GB   eth0 up
		    e0.4      4398.046GB   eth0 up
		   e20.0      1000.215GB   eth0 up
		   e42.0      2000.431GB   eth0 up
	      aoe-stat Warning: device node /dev/etherd/e45.1 has wrong minor device number
		   e45.1      1152.874GB   eth0 up

       Using  such  a  device  node  is dangerous, because its name doesn't match the actual device that you would be reading from and writing to.
       Such a broken device node should be removed.  Device nodes are created by udev or (on systems without udev) by aoe-mkdevs.

EXAMPLE

       In this example, the root user on a host named nai loads the aoe driver module and then prints a list of all  the  available  aoe  devices.
       Then  he  remembers to bring up the storage network interfaces, does an AoE discovery, and prints the list again.  This time the list shows
       all the devices in shelf seven.

	      nai:~# modprobe aoe
	      nai:~# aoe-stat
	      nai:~# ifconfig eth3 up
	      nai:~# aoe-discover
	      nai:~# aoe-stat
		    e0.0     10995.116GB   eth0 up
		    e0.1     10995.116GB   eth0 up
		    e0.2     10995.116GB   eth0 up
		    e1.0      1152.874GB   eth0 up
		    e7.0       370.566GB   eth0 up
	      nai:~#

SEE ALSO

       aoe-discover(8), aoe-interfaces(8), aoe-mkdevs(8), aoe-mkshelf(8), aoetools(8), udev(7).

AUTHOR

       Ed L. Cashin (ecashin@coraid.com)

																       aoe-stat(8)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[grep awk cut] > awk

Discussion started by: firdousamir

2. UNIX for Dummies Questions & Answers

grep and awk

Discussion started by: melanie_pfefer

3. Shell Programming and Scripting

MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else

Discussion started by: hariza

4. Shell Programming and Scripting

Is it better to grep and pipe to awk, or to seach with awk itself

Discussion started by: DeCoTwc