Hi,
I am trying to find out patterns in file 1 using patterns stored in file 2. Following is the code
FILE1=inputfilename
FILE2=blacklist
blacklist
Code:
1203
97715555
20afEOF
Code:
egrep -f $FILE2 $FILE1
but the above code is not working either using egrep or grep. Just for your information file2 has more than 2000 different patterns with no blank line at the end & file1 has more than 100million records
Last edited by Scrutinizer; 03-28-2013 at 02:52 PM..
Reason: code tags
No wonder it does not work. That should take literally hours to complete. -- Don't do that. Split file1 into a bunch of smaller files - less than 20 or so items in each file
The unix split command will do this for you. BE SURE they have a unique set of characters
Assume you have 100 files of 20 lines each: xx1 .... xx100
This will run much faster - probably less than 1 hour. DO NOT bump the "10" up too high. If you do it will actually make it take longer due to cpu and I/O contention. If you are on linux the system will probably cache most of the giant file in memory, so things will go much faster per small file after you complete one block of 10 jobs.
I tried using awk as well but that too failed. I read the second file & then started comparing the 5th & 6th column which is the actual requirement for checking.
Code:
while read line
do
BLACK[$counter]=$line
counter = `expr $counter + 1`
done <FILE2
there is file 1(cloumn 5,6) from which new file3 needs to be created with patterns not matching from file2. So i used first grep -f to find out the records having the same & then using -v i was trying to create file3 which failed(code tag already given at first instance). Then i tried awk which also didnt gave required output
Code Tag 1 using grep:
Code:
grep -vf $FILE2 $FILE1 > $FILE3
jim thanks for the suggestion but breaking such huge file will be an issue for me & then collating the same again will be cumbersome.
So i tried awk in single output as suggested but didnt worked either, please help
Code:
awk 'NR=FNR {
BLACK[$1]=1 ; next }
{for (count=5;count<=6;count++)
$count in BLACK
}
1
' $FILE2 $FILE1
Thanks corona it was typo mistake. Actually i want to check for 5th & 6th column separated by comma. If 5th or 6th comma matches in BLACK then it should print only the non matching records. Please refer the example above
Code:
awk ' NR==FNR {BLACK[$1]=1;next}
{ for (i=5;i<=6;i++)
if (!($i in BLACK)) print $0
}
' FS="," OFS="," FILE2 FILE1.
my file is below
REREGISTER is something to Failed to create the request
Failed to create the request in not easy
I know how REREGISTERcommand i run is
egrep 'REREGISTER|Failed|to|create|the|request' test1
expected output
REREGISTER is something to Failed to create the request
i should... (2 Replies)
Hi All,
I am using the following script to run some sql on database but i am not getting the result. When i tried the same by removing "-s" option it is working fine but getting other things as well with my input as shown below. Can anyone please suggest why "-s" option is not working in AIX and... (2 Replies)
In solaris, i m trying to find the files having a particulat extension and then from the list i want to exclude those files which is present in a file.
But it seems the -f and -v option are not working
find $source -type f -name $extn | /usr/xpg4/bin/grep -F -v -f $exclude | while read... (7 Replies)
Hi There,
---------
file1
-------
~c asd@ac.com
--------------
Now i am using below command
cat file1|mailx -s " testing" -r " My Name" abc@tech.com (3 Replies)
Hi,
I'm trying to validate if a string matches a regular expression, but it is not working. Am I missing something? Do I need to scape any of the characters?
if echo 'en-GB' | egrep '({1,8})(-{1,8})*' >/dev/null; then
echo Valid value
fi
Thanks in advance (6 Replies)
Running HP 11.31 on a HP3600. But when I log in as a user the who command works but if I use an option like "who -m" I get nothing. Any thoughts on what is causing this problem. (11 Replies)
:confused: In the following, when I enter a valid, existing directory,why am I getting Not Found when the Dir DOES exist?
read auser
echo "You entered $auser"
wait 2
cd /home
if
then
echo "Dir Exists"
sleep 2
else
echo "/home/$auser Not Found"
sleep... (2 Replies)
How can I use the 'ps' command to view current sessions but only for a given process/user, with the -u parm?
In older versions of Unix, this used to work, but not in Sun Solaris.
Thanks (4 Replies)
:cool: fedora core 2 version 2.6.8-1.521 gnu/linux
the last version of redhat that I was working with linux 8.0 a special version that came with a book.. on this version and on spider tools linux 0.9 the second version I worked with.. when i envoked egrep -e from file1 to file2 I would get the... (4 Replies)
