03-02-2001
Here's my problem:
1. I have a file that contains say for this example, three records, each twenty bytes long:
CustNum CustName
1111111111abcdefghij
2222222222abcdefghij
3333333333abcdefghij
2. I have two other very large files (over 500,000 records) one is 500 bytes, the other is 200 bytes long. These two files contain the CustNum from the first file as well as MANY more that I don't want.
3. I want to extract out the CustNum from the first file, then do a loop thru the other two files, matching the CustNum and only writting out those matching records all 500 or 200 bytes.
Essentially, I want to reduce the 100,000 record file to a managable amount.
I have tried cut to extract the records out to a variable, then loop with grep, but the results produce a file of one continuous record. No newline? Should I use awk instead?
Any help would be appreciated......
Here is my code:
#! /bin/ksh
# get_bids_autorenew.sh
#== Local Variables ==#
datadir="/ias/users/app4dxh/data"
driver="$datadir/driver.dat"
file_1="tcpcsm2.data"
file_2="tcpvrm2.data"
num=0
stat=0
Get_Bids()
{
cd $biddir
if [ -s $driver ]
then
bids=`cut -c1-10 $driver`
stat=$?
else
stat=$?
echo "Function: $0 - No data found in $driver or file does not exist."
echo "Aborting script with a status of $stat"
exit 1
}
Match_Files()
{
#== For each bid picked up, check each of the CP and ==#
#== SM files for a match and just write those records. ==#
for i in $bids
do
match=`grep -s $i $file`
stat=$?
while :
do
case "$stat" in
0) echo $match >> file_$num.new
echo Status is $stat
break;;
1) break;;
2) echo "Function: $0 - The file $file is not accessible - grep status is $stat"
break;;
esac
done
done
}
#==Main==#
Get_Bids
#== For each File, execute the Match_Files function ==#
cd $datadir
for data in "$file_1" "$file_2"
do
set $data
file=$1
let num="$num + 1"
Match_Files
done
exit 0
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi, I'm very new to scripting.
grep $s $filename | awk '{print $2}' | cut -c 1-8
How can I optimize this using a single awk?
I tried:
awk '/$s/ {print $2}' $filename | cut -c 1-8
However didn't work, I think the awk is not recognizing $s and the verbal is something else. (6 Replies)
Discussion started by: firdousamir
6 Replies
2. UNIX for Dummies Questions & Answers
I have grep MHz psrinfo-v.out
it gives
The i386 processor operates at 3000 MHz,
The i386 processor operates at 3000 MHz,
The i386 processor operates at 3000 MHz,
The i386 processor operates at 3000 MHz,
how to get instead of these 4 lines:
CPU speed: 3000 MHz
i.e. CPU... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies
3. Shell Programming and Scripting
Hi Guys,
I need to set the value of $7 to zero in case $7 is NULL. I've tried the below command but doesn't work. Any ideas. thanks guys.
MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else { print $7}}' `
Harby. (4 Replies)
Discussion started by: hariza
4 Replies
4. Shell Programming and Scripting
This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what... (4 Replies)
Discussion started by: DeCoTwc
4 Replies
5. Shell Programming and Scripting
Hello,
I trying to extract text that is surrounded by xml-tags. I tried this
cat tst.xml | egrep "<SERVER>.*</SERVER>" |sed -e "s/<SERVER>\(.*\)<\/SERVER>/\1/"|tr "|" " "
which works perfect, if the start-tag and the end-tag are in the same line, e.g.:
<tag1>Hello Linux-Users</tag1>
... (5 Replies)
Discussion started by: Sebi0815
5 Replies
6. Shell Programming and Scripting
I have an input file
12.4 1.72849432773174e+01 -7.74784188610632e+01
12.5 9.59432114416327e-01 -7.87018212757537e+01
15.6 5.20139995965960e-01 -5.61612429666624e+01
29.3 3.76696387248366e+00 -7.42896194101892e+01
32.1 1.86899877018077e+01 -7.56508762501408e+01
35 6.98857157014640e+00... (2 Replies)
Discussion started by: chrisjorg
2 Replies
7. Shell Programming and Scripting
how do I change this line to use the awk command
RC19=`grep -c "Broken pipe" $FTP_OUT`
Code tags please (5 Replies)
Discussion started by: mbmarciniak
5 Replies
8. UNIX for Dummies Questions & Answers
Hi,
I have a number of files containing the information below.
"""""
Fundallinfo
6.3950 14.9715 14.0482
"""""
I would like to grep for Fundallinfo and use it to read the next line? I ideally would like to read the three numbers that follow in the next line and... (2 Replies)
Discussion started by: Paul Moghadam
2 Replies
9. Shell Programming and Scripting
Hi ,
I have a file where i have modifed certain things compared to original file . The difference of the original file and modified file is as follows.
# diff mir_lex.c.modified mir_lex.c.orig
3209c3209
< if(yy_current_buffer -> yy_is_our_buffer == 0) {
---
>... (5 Replies)
Discussion started by: breezevinay
5 Replies
10. UNIX for Beginners Questions & Answers
Hi Guys.
I am trying to count occurances of patterns(occurance can be anywhere in file2) from file1 in file2.
file1 is
god
god
pod
rod
file2 is
iamgod
iamgod
podrod
123rod456
output should be
god 2
god 2
pod 1
rod 2
I am not good at awk but i figured out this command.it doesnt... (7 Replies)
Discussion started by: ahfze
7 Replies
STAT(5) File Formats Manual STAT(5)
NAME
stat, wstat - inquire or change file attributes
SYNOPSIS
Tstat tag[2] fid[2]
Rstat tag[2] fid[2] stat[116]
Twstat tag[2] fid[2] stat[116]
Rwstat tag[2] fid[2]
DESCRIPTION
The stat transaction inquires about the file identified by fid. The reply will contain a 116-byte (DIRLEN in <libc.h>) machine-independent
directory entry laid out as follows:
name[28] file name; must be / if the file is the root directory of the server
uid[28] owner name
gid[28] group name
qid.path[4] the file server's identification for the file
qid.vers[4] version number for given path
mode[4] permissions and flags
atime[4] last access time
mtime[4] last modification time
length[8] length of file in bytes
type[2] for kernel use
dev[2] for kernel use
Integers in this encoding are in little-endian order (least significant byte first). The convM2D and convD2M routines (see fcall(2)) con-
vert between directory entries and C structs.
This encoding may be turned into a machine dependent Dir structure (see stat(2)) using routines defined in fcall(2).
The mode contains permission bits as described in intro(5) and the following: 0x80000000 (this file is a directory), 0x40000000 (append
only), 0x20000000 (exclusive use). Writes to append-only files always place their data at the end of the file; the offset in the read or
write message is ignored, as is the OTRUNC bit in an open. Exclusive use files may be open for I/O by only one fid at a time across all
clients of the server. If a second open is attempted, it draws an error. Servers may implement a timeout on the lock on an exclusive use
file: if the fid holding the file open has been unused for an extended period (of order at least minutes), it is reasonable to break the
lock and deny the initial fid further I/O.
The two time fields are measured in seconds since the epoch (Jan 1 00:00 1970 GMT). The mtime field reflects the time of the last change
of content. For a plain file, mtime is the time of the most recent create, open with truncation, or write; for a directory it is the time
of the most recent remove, create, or wstat of a file in the directory. Similarly, the atime field records the last read of the contents;
also it is set whenever mtime is set. In addition, for a directory, it is set by an attach, walk, or create, all whether successful or
not.
The length records the number of bytes in the file. Directories and most files representing devices have a conventional length of 0.
The stat request requires no special permissions.
The wstat request can change some of the file status information. The name can be changed by anyone with write permission in the parent
directory; it is an error to change the name to that of an existing file. The mode and mtime can be changed by the owner of the file or
the group leader of the file's current group. The directory bit cannot be changed by a wstat; the other defined permission and mode bits
can. The gid can be changed: by the owner if also a member of the new group; or by the group leader of the file's current group if also
leader of the new group (see intro(5) for more information about permissions and users(6) for users and groups). None of the other data
can be altered by a wstat. In particular, there is no way to change the owner of a file.
A read of a directory yields an integral number of directory entries in the machine independent encoding given above (see read(5)).
ENTRY POINTS
Stat messages are generated by fstat and stat.
Wstat messages are generated by fwstat and wstat.
STAT(5)