Sponsored Content
Full Discussion: grep or awk?
Top Forums UNIX for Dummies Questions & Answers grep or awk? Post 1394 by on Friday 2nd of March 2001 03:42:53 PM
Old 03-02-2001
Here's my problem:
1. I have a file that contains say for this example, three records, each twenty bytes long:
CustNum CustName
1111111111abcdefghij
2222222222abcdefghij
3333333333abcdefghij

2. I have two other very large files (over 500,000 records) one is 500 bytes, the other is 200 bytes long. These two files contain the CustNum from the first file as well as MANY more that I don't want.

3. I want to extract out the CustNum from the first file, then do a loop thru the other two files, matching the CustNum and only writting out those matching records all 500 or 200 bytes.
Essentially, I want to reduce the 100,000 record file to a managable amount.

I have tried cut to extract the records out to a variable, then loop with grep, but the results produce a file of one continuous record. No newline? Should I use awk instead?

Any help would be appreciated......
Here is my code:

#! /bin/ksh
# get_bids_autorenew.sh

#== Local Variables ==#
datadir="/ias/users/app4dxh/data"
driver="$datadir/driver.dat"
file_1="tcpcsm2.data"
file_2="tcpvrm2.data"
num=0
stat=0



Get_Bids()
{
cd $biddir

if [ -s $driver ]
then
bids=`cut -c1-10 $driver`
stat=$?
else
stat=$?
echo "Function: $0 - No data found in $driver or file does not exist."
echo "Aborting script with a status of $stat"
exit 1
}



Match_Files()
{
#== For each bid picked up, check each of the CP and ==#
#== SM files for a match and just write those records. ==#

for i in $bids
do
match=`grep -s $i $file`
stat=$?
while :
do
case "$stat" in
0) echo $match >> file_$num.new
echo Status is $stat
break;;
1) break;;
2) echo "Function: $0 - The file $file is not accessible - grep status is $stat"
break;;
esac
done
done

}

#==Main==#
Get_Bids

#== For each File, execute the Match_Files function ==#
cd $datadir
for data in "$file_1" "$file_2"
do
set $data
file=$1
let num="$num + 1"
Match_Files
done
exit 0


 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[grep awk cut] > awk

Hi, I'm very new to scripting. grep $s $filename | awk '{print $2}' | cut -c 1-8 How can I optimize this using a single awk? I tried: awk '/$s/ {print $2}' $filename | cut -c 1-8 However didn't work, I think the awk is not recognizing $s and the verbal is something else. (6 Replies)
Discussion started by: firdousamir
6 Replies

2. UNIX for Dummies Questions & Answers

grep and awk

I have grep MHz psrinfo-v.out it gives The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, how to get instead of these 4 lines: CPU speed: 3000 MHz i.e. CPU... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies

3. Shell Programming and Scripting

MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else

Hi Guys, I need to set the value of $7 to zero in case $7 is NULL. I've tried the below command but doesn't work. Any ideas. thanks guys. MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else { print $7}}' ` Harby. (4 Replies)
Discussion started by: hariza
4 Replies

4. Shell Programming and Scripting

Is it better to grep and pipe to awk, or to seach with awk itself

This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what... (4 Replies)
Discussion started by: DeCoTwc
4 Replies

5. Shell Programming and Scripting

Read content between xml tags with awk, grep, awk or what ever...

Hello, I trying to extract text that is surrounded by xml-tags. I tried this cat tst.xml | egrep "<SERVER>.*</SERVER>" |sed -e "s/<SERVER>\(.*\)<\/SERVER>/\1/"|tr "|" " " which works perfect, if the start-tag and the end-tag are in the same line, e.g.: <tag1>Hello Linux-Users</tag1> ... (5 Replies)
Discussion started by: Sebi0815
5 Replies

6. Shell Programming and Scripting

AWK/GREP: grep only lines starting with integer

I have an input file 12.4 1.72849432773174e+01 -7.74784188610632e+01 12.5 9.59432114416327e-01 -7.87018212757537e+01 15.6 5.20139995965960e-01 -5.61612429666624e+01 29.3 3.76696387248366e+00 -7.42896194101892e+01 32.1 1.86899877018077e+01 -7.56508762501408e+01 35 6.98857157014640e+00... (2 Replies)
Discussion started by: chrisjorg
2 Replies

7. Shell Programming and Scripting

awk / grep

how do I change this line to use the awk command RC19=`grep -c "Broken pipe" $FTP_OUT` Code tags please (5 Replies)
Discussion started by: mbmarciniak
5 Replies

8. UNIX for Dummies Questions & Answers

Piping grep into awk, read the next line using grep

Hi, I have a number of files containing the information below. """"" Fundallinfo 6.3950 14.9715 14.0482 """"" I would like to grep for Fundallinfo and use it to read the next line? I ideally would like to read the three numbers that follow in the next line and... (2 Replies)
Discussion started by: Paul Moghadam
2 Replies

9. Shell Programming and Scripting

sed and awk usage to grep a pattern 1 and with reference to this grep a pattern 2 and pattern 3

Hi , I have a file where i have modifed certain things compared to original file . The difference of the original file and modified file is as follows. # diff mir_lex.c.modified mir_lex.c.orig 3209c3209 < if(yy_current_buffer -> yy_is_our_buffer == 0) { --- >... (5 Replies)
Discussion started by: breezevinay
5 Replies

10. UNIX for Beginners Questions & Answers

Using awk instead of grep -f

Hi Guys. I am trying to count occurances of patterns(occurance can be anywhere in file2) from file1 in file2. file1 is god god pod rod file2 is iamgod iamgod podrod 123rod456 output should be god 2 god 2 pod 1 rod 2 I am not good at awk but i figured out this command.it doesnt... (7 Replies)
Discussion started by: ahfze
7 Replies
STAT(5) 							File Formats Manual							   STAT(5)

NAME
stat, wstat - inquire or change file attributes SYNOPSIS
Tstat tag[2] fid[2] Rstat tag[2] fid[2] stat[116] Twstat tag[2] fid[2] stat[116] Rwstat tag[2] fid[2] DESCRIPTION
The stat transaction inquires about the file identified by fid. The reply will contain a 116-byte (DIRLEN in <libc.h>) machine-independent directory entry laid out as follows: name[28] file name; must be / if the file is the root directory of the server uid[28] owner name gid[28] group name qid.path[4] the file server's identification for the file qid.vers[4] version number for given path mode[4] permissions and flags atime[4] last access time mtime[4] last modification time length[8] length of file in bytes type[2] for kernel use dev[2] for kernel use Integers in this encoding are in little-endian order (least significant byte first). The convM2D and convD2M routines (see fcall(2)) con- vert between directory entries and C structs. This encoding may be turned into a machine dependent Dir structure (see stat(2)) using routines defined in fcall(2). The mode contains permission bits as described in intro(5) and the following: 0x80000000 (this file is a directory), 0x40000000 (append only), 0x20000000 (exclusive use). Writes to append-only files always place their data at the end of the file; the offset in the read or write message is ignored, as is the OTRUNC bit in an open. Exclusive use files may be open for I/O by only one fid at a time across all clients of the server. If a second open is attempted, it draws an error. Servers may implement a timeout on the lock on an exclusive use file: if the fid holding the file open has been unused for an extended period (of order at least minutes), it is reasonable to break the lock and deny the initial fid further I/O. The two time fields are measured in seconds since the epoch (Jan 1 00:00 1970 GMT). The mtime field reflects the time of the last change of content. For a plain file, mtime is the time of the most recent create, open with truncation, or write; for a directory it is the time of the most recent remove, create, or wstat of a file in the directory. Similarly, the atime field records the last read of the contents; also it is set whenever mtime is set. In addition, for a directory, it is set by an attach, walk, or create, all whether successful or not. The length records the number of bytes in the file. Directories and most files representing devices have a conventional length of 0. The stat request requires no special permissions. The wstat request can change some of the file status information. The name can be changed by anyone with write permission in the parent directory; it is an error to change the name to that of an existing file. The mode and mtime can be changed by the owner of the file or the group leader of the file's current group. The directory bit cannot be changed by a wstat; the other defined permission and mode bits can. The gid can be changed: by the owner if also a member of the new group; or by the group leader of the file's current group if also leader of the new group (see intro(5) for more information about permissions and users(6) for users and groups). None of the other data can be altered by a wstat. In particular, there is no way to change the owner of a file. A read of a directory yields an integral number of directory entries in the machine independent encoding given above (see read(5)). ENTRY POINTS
Stat messages are generated by fstat and stat. Wstat messages are generated by fwstat and wstat. STAT(5)
All times are GMT -4. The time now is 08:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy