Sponsored Content
Full Discussion: Best search technique
Top Forums Shell Programming and Scripting Best search technique Post 302408321 by chakrapani on Monday 29th of March 2010 07:53:48 AM
Old 03-29-2010
Best search technique - Need help

I have a snippet file with the shown below:

data file
Code:
1 2
1 3
1 3
4 2
3 2
2 1
2 2
5 1
3 2
3 2
2 3
1 4

Actual file has approx 50 Millions such lines with bigger number

Now I need to find a way so that I can pull out all uniq sets
example

Case satisfying conditions
3 2 is uniq
as first column has 3 and 2nd column is "2" only no other 2nd number
also 5 and 1 and it appears only once.

Cases does not work
1,2 etc WHERE 1 has other numbers in 2nd column.

Already tried:
1. Tried database mysql.. Does not work
2. grep and awk : very slow .. My script is running for more than 3 days
3. Sort column and comparing with scnd column ... Need help on any unix/linux tool to do this
4. comm commands also seems to be scared of too much data ...
5. Tried perl with BINMODE and reading a BLOCK etc etc ... Slower than grep and egrep.

Any ideas on how to get this details.....

Last edited by chakrapani; 03-29-2010 at 08:59 AM..
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Difference Technique's???

Is there any better way of doing this? I only want to find a status of a diff, ie diff the file and return to me whether it is different or not or non-existant. This example works, however I think it could be less messier: workd=`pwd`;find $workd -name "*.sum" | while read line ; do... (1 Reply)
Discussion started by: Shakey21
1 Replies

2. UNIX for Dummies Questions & Answers

FORK/EXEC technique

Hi! Can someone explain me exactly this technique? Why a process (PARENT) creates a copy of itself with FORK (CHILD)? What's the reason of this behaviour? Sorry, but I cannot understand the logic behind it. Thanks. (4 Replies)
Discussion started by: marshmallow
4 Replies

3. Shell Programming and Scripting

Perl - use search keywords from array and search a file and print 3rd field when matched

Hi , I have been trying to write a perl script to do this job. But i am not able to achieve the desired result. Below is my code. my $current_value=12345; my @users=("bob","ben","tom","harry"); open DBLIST,"<","/var/tmp/DBinfo"; my @input = <DBLIST>; foreach (@users) { my... (11 Replies)
Discussion started by: chidori
11 Replies

4. Shell Programming and Scripting

Password Obscuring Technique

Hi, We have a unix shell script which tries login to database. The user name and password to connect to database is stored in a file connection.sql. Now connection.sql has contents def ora_user =&1 CONNECT A_PROXY/abc123@DEV01 When on UNIX server we connect to database and set spool on... (7 Replies)
Discussion started by: Gangadhar Reddy
7 Replies

5. Linux

Best Compression technique ?

Hi all, I am working on a sample backup code, where i read the files per 7200 bytes and send it to server. Before sending to server, i compress each 7200 bytes using zlib compression algorithm using dictionary max length of 1.5 MB . I find zlib is slow. Can anyone recommend me a... (3 Replies)
Discussion started by: selvarajvss
3 Replies

6. Shell Programming and Scripting

Perl - start search by using search button or by pressing the enter key

#Build label and text box $main->Label( -text => "Input string below:" )->pack(); $main->Entry( -textvariable => \$text456 )->pack(); $main->Button( -text => "Search", -command => sub { errchk ($text456) ... (4 Replies)
Discussion started by: popeye
4 Replies

7. Shell Programming and Scripting

Search pattern on logfile and search for day/dates and skip duplicate lines if any

Hi, I've written a script to search for an Oracle ORA- error on a log file, print that line and the .trc file associated with it as well as the dateline of when I assumed the error occured. In most it is the first dateline previous to the error. Unfortunately, this is not a fool proof script.... (2 Replies)
Discussion started by: newbie_01
2 Replies

8. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies
COMM(1) 							   User Commands							   COMM(1)

NAME
comm - compare two sorted files line by line SYNOPSIS
comm [OPTION]... FILE1 FILE2 DESCRIPTION
Compare sorted files FILE1 and FILE2 line by line. With no options, produce three-column output. Column one contains lines unique to FILE1, column two contains lines unique to FILE2, and column three contains lines common to both files. -1 suppress column 1 (lines unique to FILE1) -2 suppress column 2 (lines unique to FILE2) -3 suppress column 3 (lines that appear in both files) --check-order check that the input is correctly sorted, even if all input lines are pairable --nocheck-order do not check that the input is correctly sorted --output-delimiter=STR separate columns with STR --help display this help and exit --version output version information and exit Note, comparisons honor the rules specified by 'LC_COLLATE'. EXAMPLES
comm -12 file1 file2 Print only lines present in both file1 and file2. comm -3 file1 file2 Print lines in file1 not in file2, and vice versa. GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report comm translation bugs to <http://translationproject.org/team/> AUTHOR
Written by Richard M. Stallman and David MacKenzie. COPYRIGHT
Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
join(1), uniq(1) The full documentation for comm is maintained as a Texinfo manual. If the info and comm programs are properly installed at your site, the command info coreutils 'comm invocation' should give you access to the complete manual. GNU coreutils 8.22 June 2014 COMM(1)
All times are GMT -4. The time now is 07:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy