Search values between ranges in File1 within File2


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search values between ranges in File1 within File2
# 1  
Old 01-31-2009
Search values between ranges in File1 within File2

Hi people,


I have 2 files, one with a list of non consecutive ranges (File1.txt), where each range begins with the value in column 1
and finishes with the value in column 2 in the same line, as can be seen above.

Code:
215312581156279  	215312581166279
215312582342558  	215312582357758
215312584725516  	215312584735016
215312589480032  	215312589490032
215312589491032  	215312589501032
215312589592032  	215312589602032
215312589614332  	215312589624332
215312589636632  	215312589646632
215312589658932  	215312589668932
215312589681232  	215312589691232
215312589703532  	215312589713532

In the 2nd file (File2.txt), I have less than 4 millions of lines and I need to search only those lines that contain the values
within the specified ranges in File1.txt and send the matching lines to a new file (File3.txt).


I know how to do with awk or even grep for a few values in the way showed below


Code:
awk '/215312581156279/||/215312581156280/||/215312581156281/' File2.txt > File3.txt

The above command would be for the first 3 values of the first range, but every range in File1.txt contain
easy 10,000 values and sometimes more.

How can I say to awk that I need to search from first value up to last value for each range?

Thanks in advance for any help on this.

Best regards.
# 2  
Old 01-31-2009
Things like awk and grep use string matching, which are great for matching strings. Matching a range of numbers is not a string matching problem, since it means needing to actually process and compare the value of the numbers, not the strings representing them.

I'm sure better perl coders than me could shrink this program down to three lines of garbage, but this seems to work at least.

Code:
#!/usr/bin/perl

my $arr=[];
open FILE1, "file1";
# Get a list of number ranges, store it in an array of arrays
while($line=<FILE1>)
{
        if($line =~ /([0-9]+)[ \t]*([0-9]+)/)
        {
                $arr[++$#arr]=[$1, $2];
        }
}

close FILE1;

# Read in lines from file2
open FILE2, "file2";
while($line=<FILE2>)
{
        $match=0;

# Extract all numbers from $line
        while(($match == 0) && ($line =~ /([0-9]+)/g))
        {
# check each number against the ranges given
                for($n=0; ($n<$#arr)&&($match==0); $n++)
                {
# If it matches, stop hunting and print the line.
                        if(($arr[$n][0] <= $1) &&
                                ($arr[$n][1] >= $1))
                        {
                                print $line;
                                $match=1;
                        }
                }
        }
}

exit 0;

# 3  
Old 01-31-2009
Another one with awk, assuming you have only numbers in the second file, otherwise, replace $0 in the if statement with the desired field:

Code:
awk 'NR==FNR{a[++i]=$1;b[i]=$2;next}
{ for(j=1;j<=i;j++) {
    if($0>=a[j] && $0<=b[j]) {print;break}
  }
}' file1 file2

Regards
# 4  
Old 01-31-2009
How large a value can awk use? These ranges are far higher values than the 32-bit max, I used perl specifically because it can handle arbitrarily large numbers.
# 5  
Old 01-31-2009
Hey Corona688,

Many thanks for take your time to help me, I think it works but I receive an error when I run it, syntax error '}', IŽll try
to find out why this happens when I run some perl scripts in the emulator I used, it could be some libraries are missing.
IŽll try to test in other system, but really thanks.


Hi Franklin again,

Thanks a lot for the help, it works nice, nice when I replaced $0 with the desired field because you anticipated very good the
kind of file, the 2nd file it has numbers and letters. Using $0 didnŽt show correct info, but when I replace with $1 or $2
worked perfectly man!!!.

Again thank both for your valuable help.Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Mapping the values of ids of two columns of file1 from file2

I have of two space separated files: ==> File1 <== PT|np_496075.1 st|K92748.1 st|K89648.1 PT|np_001300561.1 PT|np_497284.1 st|K90752.1 st|K90279.1 PT|np_740775.1 PT|np_497749.1 st|K90752.1 st|K92038.1 PT|np_490856.1 PT|np_497284.1 st|K90752.1 st|K88095.1 PT|np_494764.1 ==> File 2 <==... (2 Replies)
Discussion started by: sammy777888
2 Replies

2. Shell Programming and Scripting

awk to search field2 in file2 using range of fields file1 and using match to another field in file1

I am trying to use awk to find all the $2 values in file2 which is ~30MB and tab-delimited, that are between $2 and $3 in file1 which is ~2GB and tab-delimited. I have just found out that I need to use $1 and $2 and $3 from file1 and $1 and $2of file2 must match $1 of file1 and be in the range... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

Search within file1 numbers from list in file2

Hello to all, I hope somebody could help me with this: I have this File1 (real has 5 million of lines): Number Category --------------- -------------------------------------- 8734060355 3 8734060356 ... (6 Replies)
Discussion started by: Ophiuchus
6 Replies

4. UNIX for Dummies Questions & Answers

if matching strings in file1 and file2, add column from file1 to file2

I have very limited coding skills but I'm wondering if someone could help me with this. There are many threads about matching strings in two files, but I have no idea how to add a column from one file to another based on a matching string. I'm looking to match column1 in file1 to the number... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

5. Shell Programming and Scripting

search from file1 and replace into file2

I have 2 files: file1.txt: 1|15|XXXXXX||9630716||0096000||30/04/2012|E|O|X||||20120525135617-30.04.2012|PAT66OLM|STA||||00001|STA_0096000_YYYPPPXTMEX00_20120525135617_02_P.pdf|... (2 Replies)
Discussion started by: pparthiv
2 Replies

6. Shell Programming and Scripting

Remove lines in file1 with values from file2

Hello, I have two data files: file1 12345 aa bbb cccc 98765 qq www uuuu 76543 pp rrr bbbbb 34567 nn ccc sssss 87654 qq ppp rrrrr file2 98765 34567 I need to remove the lines from file1 if the first field contains a value that appears in file2: output 12345 aa bbb cccc 76543 pp... (2 Replies)
Discussion started by: palex
2 Replies

7. Shell Programming and Scripting

Get values from different columns from file2 when match values of file1

Hi everyone, I have file1 and file2 comma separated both. file1 is: Header1,Header2,Header3,Header4,Header5,Header6,Header7,Header8,Header9,Header10 Code7,,,,,,,,, Code5,,,,,,,,, Code3,,,,,,,,, Code9,,,,,,,,, Code2,,,,,,,,,file2... (17 Replies)
Discussion started by: cgkmal
17 Replies

8. Shell Programming and Scripting

Search & replace fields from file1 to file2

hi, I have two xml files with the name source.xml and tobe_replaced.xml. Sample data: source.xml contains: <?xml version="1.0"?> <product description="prod1" product_info="some/info"> <product description="prod2" product_info="xyz/allinfo"> <product description="abc/partialinfo"... (2 Replies)
Discussion started by: dragon.1431
2 Replies

9. Shell Programming and Scripting

AWK: read values from file1; search for values in file2

I have read another post about this issue and am wondering how to adapt it to my own, much simpler, issue. I have a file of user IDs like so: 333333 321321 546465 ...etc I need to take each number and use it to print records wherein the 5th field matches the user ID pulled from the... (2 Replies)
Discussion started by: Bubnoff
2 Replies

10. Shell Programming and Scripting

Read each word from File1 and search each file in file2

file1: has all words to be searched. 100007 200999 299997 File2: has all file names to be searched. C:\search1.txt C:\search2.txt C:\search3.txt C:\search4.txt Outfile: should have all found lines. Logic: Read each word in file1 and search each file in the list of File2; if the... (8 Replies)
Discussion started by: clem2610
8 Replies
Login or Register to Ask a Question