Visit Our UNIX and Linux User Community


grep or awk?


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers grep or awk?
# 1  
Old 03-02-2001
Here's my problem:
1. I have a file that contains say for this example, three records, each twenty bytes long:
CustNum CustName
1111111111abcdefghij
2222222222abcdefghij
3333333333abcdefghij

2. I have two other very large files (over 500,000 records) one is 500 bytes, the other is 200 bytes long. These two files contain the CustNum from the first file as well as MANY more that I don't want.

3. I want to extract out the CustNum from the first file, then do a loop thru the other two files, matching the CustNum and only writting out those matching records all 500 or 200 bytes.
Essentially, I want to reduce the 100,000 record file to a managable amount.

I have tried cut to extract the records out to a variable, then loop with grep, but the results produce a file of one continuous record. No newline? Should I use awk instead?

Any help would be appreciated......
Here is my code:

#! /bin/ksh
# get_bids_autorenew.sh

#== Local Variables ==#
datadir="/ias/users/app4dxh/data"
driver="$datadir/driver.dat"
file_1="tcpcsm2.data"
file_2="tcpvrm2.data"
num=0
stat=0



Get_Bids()
{
cd $biddir

if [ -s $driver ]
then
bids=`cut -c1-10 $driver`
stat=$?
else
stat=$?
echo "Function: $0 - No data found in $driver or file does not exist."
echo "Aborting script with a status of $stat"
exit 1
}



Match_Files()
{
#== For each bid picked up, check each of the CP and ==#
#== SM files for a match and just write those records. ==#

for i in $bids
do
match=`grep -s $i $file`
stat=$?
while :
do
case "$stat" in
0) echo $match >> file_$num.new
echo Status is $stat
break;;
1) break;;
2) echo "Function: $0 - The file $file is not accessible - grep status is $stat"
break;;
esac
done
done

}

#==Main==#
Get_Bids

#== For each File, execute the Match_Files function ==#
cd $datadir
for data in "$file_1" "$file_2"
do
set $data
file=$1
let num="$num + 1"
Match_Files
done
exit 0


# 2  
Old 03-02-2001
CPU & Memory

Hi, did yo use the comm command?
Try this with the sort command first to order the files.
I hope to help you.
See you!!
# 3  
Old 03-03-2001
Here is one Perl Script that could solve Your problem (if i am not mistaken your problem). use it as "scriptname Custnofile serachinfile outputfile".


#!/bin/perl

unless ($ARGV[2]) { print "Usage scriptname: <inputfile1-CustNo> <inputfile2:Search in> <outputfile> \n"; exit;}

$CustNoFile=$ARGV[0];
$SearchIn=$ARGV[1];
$NewFile=$ARGV[2];

$DataDir="/ias/users/app4dxh/data";

open (DATA, "$DataDir/$CustNoFile") || die ("Can't open $CustNoFile Reson $!");
@DataArray = <DATA>;
close (DATA);

foreach $Record(@DataArray){
$L= substr($Record, 0, 10);
$Key .= "$L\n";
}
@Keys = split(/\n/, $Key);

open (FILE, "$DataDir/$SearchIn") || die ("Can't open $SearchIn Reason $!");
@Check = <FILE>;
close (FILE);

open (NEW, ">>$DataDir/$NewFile") || die ("can't open $NewFile Reson $!");

foreach $Line(@Check){
chomp ($Line);
$count=0;
foreach $key(@Keys) {
chomp ($key);
if ($Line =~ $key ) { print NEW "$Line\n"; $Found++; next;}
}
}

close (NEW);
print "$Found Keys Found and copied to $NewFile\n";






[Edited by mib on 03-03-2001 at 08:34 AM]
 

Previous Thread | Next Thread
Test Your Knowledge in Computers #71
Difficulty: Easy
Linux, macOS, Android, iOS, Chrome OS, Orbis OS used on the PlayStation 4, whatever firmware is running on your router, all of these operating systems are generally referred to as a Unix-like operating system.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Using awk instead of grep -f

Hi Guys. I am trying to count occurances of patterns(occurance can be anywhere in file2) from file1 in file2. file1 is god god pod rod file2 is iamgod iamgod podrod 123rod456 output should be god 2 god 2 pod 1 rod 2 I am not good at awk but i figured out this command.it doesnt... (7 Replies)
Discussion started by: ahfze
7 Replies

2. Shell Programming and Scripting

sed and awk usage to grep a pattern 1 and with reference to this grep a pattern 2 and pattern 3

Hi , I have a file where i have modifed certain things compared to original file . The difference of the original file and modified file is as follows. # diff mir_lex.c.modified mir_lex.c.orig 3209c3209 < if(yy_current_buffer -> yy_is_our_buffer == 0) { --- >... (5 Replies)
Discussion started by: breezevinay
5 Replies

3. UNIX for Dummies Questions & Answers

Piping grep into awk, read the next line using grep

Hi, I have a number of files containing the information below. """"" Fundallinfo 6.3950 14.9715 14.0482 """"" I would like to grep for Fundallinfo and use it to read the next line? I ideally would like to read the three numbers that follow in the next line and... (2 Replies)
Discussion started by: Paul Moghadam
2 Replies

4. Shell Programming and Scripting

awk / grep

how do I change this line to use the awk command RC19=`grep -c "Broken pipe" $FTP_OUT` Code tags please (5 Replies)
Discussion started by: mbmarciniak
5 Replies

5. Shell Programming and Scripting

AWK/GREP: grep only lines starting with integer

I have an input file 12.4 1.72849432773174e+01 -7.74784188610632e+01 12.5 9.59432114416327e-01 -7.87018212757537e+01 15.6 5.20139995965960e-01 -5.61612429666624e+01 29.3 3.76696387248366e+00 -7.42896194101892e+01 32.1 1.86899877018077e+01 -7.56508762501408e+01 35 6.98857157014640e+00... (2 Replies)
Discussion started by: chrisjorg
2 Replies

6. Shell Programming and Scripting

Read content between xml tags with awk, grep, awk or what ever...

Hello, I trying to extract text that is surrounded by xml-tags. I tried this cat tst.xml | egrep "<SERVER>.*</SERVER>" |sed -e "s/<SERVER>\(.*\)<\/SERVER>/\1/"|tr "|" " " which works perfect, if the start-tag and the end-tag are in the same line, e.g.: <tag1>Hello Linux-Users</tag1> ... (5 Replies)
Discussion started by: Sebi0815
5 Replies

7. Shell Programming and Scripting

Is it better to grep and pipe to awk, or to seach with awk itself

This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what... (4 Replies)
Discussion started by: DeCoTwc
4 Replies

8. Shell Programming and Scripting

MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else

Hi Guys, I need to set the value of $7 to zero in case $7 is NULL. I've tried the below command but doesn't work. Any ideas. thanks guys. MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else { print $7}}' ` Harby. (4 Replies)
Discussion started by: hariza
4 Replies

9. UNIX for Dummies Questions & Answers

grep and awk

I have grep MHz psrinfo-v.out it gives The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, The i386 processor operates at 3000 MHz, how to get instead of these 4 lines: CPU speed: 3000 MHz i.e. CPU... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies

10. Shell Programming and Scripting

[grep awk cut] > awk

Hi, I'm very new to scripting. grep $s $filename | awk '{print $2}' | cut -c 1-8 How can I optimize this using a single awk? I tried: awk '/$s/ {print $2}' $filename | cut -c 1-8 However didn't work, I think the awk is not recognizing $s and the verbal is something else. (6 Replies)
Discussion started by: firdousamir
6 Replies

Featured Tech Videos