Best search technique Post: 302408321

Sponsored Content

Top Forums Shell Programming and Scripting Best search technique Post 302408321 by chakrapani on Monday 29th of March 2010 07:53:48 AM

03-29-2010

Registered User

Best search technique - Need help

I have a snippet file with the shown below:

data file

Code:

Actual file has approx 50 Millions such lines with bigger number

Now I need to find a way so that I can pull out all uniq sets
example

Case satisfying conditions
3 2 is uniq
as first column has 3 and 2nd column is "2" only no other 2nd number
also 5 and 1 and it appears only once.

Cases does not work
1,2 etc WHERE 1 has other numbers in 2nd column.

Already tried:
1. Tried database mysql.. Does not work
2. grep and awk : very slow .. My script is running for more than 3 days
3. Sort column and comparing with scnd column ... Need help on any unix/linux tool to do this
4. comm commands also seems to be scared of too much data ...
5. Tried perl with BINMODE and reading a BLOCK etc etc ... Slower than grep and egrep.

Any ideas on how to get this details.....

Last edited by chakrapani; 03-29-2010 at 08:59 AM..

chakrapani

View Public Profile for chakrapani

Find all posts by chakrapani

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Difference Technique's???

Is there any better way of doing this? I only want to find a status of a diff, ie diff the file and return to me whether it is different or not or non-existant. This example works, however I think it could be less messier: workd=`pwd`;find $workd -name "*.sum" | while read line ; do...

2. UNIX for Dummies Questions & Answers

FORK/EXEC technique

Hi! Can someone explain me exactly this technique? Why a process (PARENT) creates a copy of itself with FORK (CHILD)? What's the reason of this behaviour? Sorry, but I cannot understand the logic behind it. Thanks.

3. Shell Programming and Scripting

Perl - use search keywords from array and search a file and print 3rd field when matched

Hi , I have been trying to write a perl script to do this job. But i am not able to achieve the desired result. Below is my code. my $current_value=12345; my @users=("bob","ben","tom","harry"); open DBLIST,"<","/var/tmp/DBinfo"; my @input = <DBLIST>; foreach (@users) { my...

4. Shell Programming and Scripting

Password Obscuring Technique

Hi, We have a unix shell script which tries login to database. The user name and password to connect to database is stored in a file connection.sql. Now connection.sql has contents def ora_user =&1 CONNECT A_PROXY/abc123@DEV01 When on UNIX server we connect to database and set spool on...

5. Linux

Best Compression technique ?

Hi all, I am working on a sample backup code, where i read the files per 7200 bytes and send it to server. Before sending to server, i compress each 7200 bytes using zlib compression algorithm using dictionary max length of 1.5 MB . I find zlib is slow. Can anyone recommend me a...

6. Shell Programming and Scripting

Perl - start search by using search button or by pressing the enter key

#Build label and text box $main->Label( -text => "Input string below:" )->pack(); $main->Entry( -textvariable => \$text456 )->pack(); $main->Button( -text => "Search", -command => sub { errchk ($text456) ...

7. Shell Programming and Scripting

Search pattern on logfile and search for day/dates and skip duplicate lines if any

Hi, I've written a script to search for an Oracle ORA- error on a log file, print that line and the .trc file associated with it as well as the dateline of when I assumed the error occured. In most it is the first dateline previous to the error. Unfortunately, this is not a fool proof script....

8. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo

LEARN ABOUT HPUX

wait

wait(1) 						      General Commands Manual							   wait(1)

NAME

       wait - await process completion

SYNOPSIS

       [pid]

DESCRIPTION

       If  no argument is specified, waits until all processes (started with of the current shell have completed, and reports on abnormal termina-
       tions.  If a numeric argument pid is given and is the process ID of a background process, waits until that process has  completed.   Other-
       wise, if pid is not a background process, exits without waiting for any processes to complete.

       Because the system call must be executed in the parent process, the shell itself executes without creating a new process (see wait(2)).

   Command-Line Arguments
       supports the following command line arguments:

	      The unsigned decimal integer process
			  ID of a command, whose termination is to wait for.

WARNINGS

       Some processes in a 2-or-more-stage pipeline may not be children of the shell, and thus cannot be waited for.

SEE ALSO

       csh(1), ksh(1), sh-posix(1), sh(1), wait(2).

STANDARDS CONFORMANCE

																	   wait(1)