UNIX for Advanced & Expert Users

Here is my file : (pipe delimited file)

abc|abc|11112|ffff|3333|oo
abc|abc|11112|ffff|3333|oo
abc|11112|ffff|2222|oo
abc|abc|11112|ffff|3333|oo
abc|22222|ffff|1111|oo

I need 2 things :
1. Need to grep for all rows where in 5th column is 3333. (i can do this using awk.. but my actual file has around 18 million rows... so awk takes more than an hour as 'awk' reads the file line by line).. so i am looking for a solution using grep.
2. Need to grep for alll the lines having the pattern 'abc' in column 1 and 'ffff' in column 4th.

please let me know whether this looks feasible .. thx a lot for helping me out !!!

What makes you think that grep does not read the file 'line by line'?
It's feasible and 'awk' would be one of tools to use.
Please post your 'awk' code - let's see what you've got.

try this instead

Code:

#!/usr/bin/perl

$i=0;
while ( $i eq 0 ){
$line=<STDIN>;
$line=~ s/\n//g;
$line=~ s/\r//g;
($a, $b, $c, $d, $e, $f) = split ("|", $line);
if ( "$d" eq "ffff" ){
if ( "$a" eq "abc" ){
print "$line";
}
}

}

-----------

Or with grep, mabey try

Code:

cat $yourfile|grep -i -e "ffff"|grep -i -e "abc"

this will show all lines only containing both ffff and abc, the perl script I gave you is more controled.

hope it helps


Quick question, Why do you want to return all the information that is duplicated? are you grabbing information from another field?

Code:

cat $yourfile|grep -i -e "ffff"|grep -i -e "abc"

Why exactly do you need a 'cat' here?
Why do you need '-e' for 'grep'?

cat dumps entire file to output

| is a pipe to push into another application's stdin

grep -i -e

-i removes case

-e "string to search for"

I pipe it into another instance of grep to search for the second string, this will make sure all lines output have both instances inside of it. As far as i know, grep doesn't use an internal field separator, so use perl,python or even c if you want something faster then awk.

The perl script I gave you should work with little modification to what you acctually want. I made that in reguards to what I thought you needed.

Once again WHY do you 'cat'? Read this url
I believe '-e' is for the Extended regular expression, which you don't have - loose the '-e'.
You don't need 2 greps piping to each other - instead use ONE 'sed':

Code:

sed '/ffff/!d; /abc/!d'