03-22-2012
remove duplicates based on a field and criteria
Hi,
I have a file with fields like below:
A;XYZ;102345;222
B;XYZ;123243;333
C;ABC;234234;444
D;MNO;103345;222
E;DEF;124243;333
desired output:
C;ABC;234234;444
D;MNO;103345;222
E;DEF;124243;333
ie, if the 4rth field is a duplicate.. i need only those records where the 3rd field value is greater .. or infact, more specificcally.. need those where the 2nd and 3rd digits of the 3rd field are greater.. Can we do this with awk?
Pls help in finding a solution where i can quickly process a file of around 50000 records
Thanks a lot in advance
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi guys,
Please guide for Solution.
PART-I
INPUT FILE (has 2 columns ID and score)
TC5584_1 93.9
DV161411_2 79.5
BP132435_5 46.8
EB682112_1 34.7
BP132435_4 29.5
TC13860_2 10.1
OUTPUT FILE (It shudn't contain the line ' BP132435_4 29.5 ' as BP132435 is repeated... (2 Replies)
Discussion started by: smriti_shridhar
2 Replies
2. Shell Programming and Scripting
I have a file containing date/time sorted data of the form
...
2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1
2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1
2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0
2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies
3. Shell Programming and Scripting
Can some one provide me a shell script.
I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique.
Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies
4. Shell Programming and Scripting
Hello to all,
I have this file
2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90
2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70
2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies
5. Shell Programming and Scripting
Hi All,
I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file.
Input :
Input.txt
1234,0,x
1234,1,y
5678,10,z
9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies
6. Shell Programming and Scripting
Hello,
I am new to shell scripting. I have a huge file with multiple columns for example:
I have 5 columns below.
HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG
HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL
HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies
7. Shell Programming and Scripting
Hi Folks -
I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below:
aaaaaa|a12|45|56
abbbbaaa|a12|45|56
bbaabb|b1|51|45
bbbbbabbb|b2|51|45
aaabbbaaaa|a11|45|56
... (3 Replies)
Discussion started by: asyed
3 Replies
8. Shell Programming and Scripting
Hi
Description of input file I have:
-------------------------
1) CSV with double quotes for string fields.
2) Some string fields have Comma as part of field value.
3) Have Duplicate lines
4) Have 200 columns/fields
5) File size is more than 10GB
Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies
9. Shell Programming and Scripting
Hi All,
I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example:
Input file:
12345a rerere.rerere len=23
11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies
10. Shell Programming and Scripting
I am trying to see if I can use awk to remove duplicates from a file. This is the file:
-==> Listvol <==
deleting /vol/eng_rmd_0941
deleting /vol/eng_rmd_0943
deleting /vol/eng_rmd_0943
deleting /vol/eng_rmd_1006
deleting /vol/eng_rmd_1012
rearrange /vol/eng_rmd_0943
... (6 Replies)
Discussion started by: newbie2010
6 Replies
LEARN ABOUT PHP
cubrid_seq_put
CUBRID_SEQ_PUT(3) 1 CUBRID_SEQ_PUT(3)
cubrid_seq_put - Update the element value of sequence type column using OID
SYNOPSIS
bool cubrid_seq_put (resource $conn_identifier, string $oid, string $attr_name, int $index, string $seq_element)
DESCRIPTION
The cubrid_seq_put(3) function is used to update the content of the requested element in a sequent type attribute using OID.
PARAMETERS
o $conn_identifier
-Connection identifier.
o $oid
-OID of the instance you want to work with.
o $attr_name
-Name of the attribute that you want to update an element.
o $index
-Index (1-based) of the element that you want to update.
o $seq_element
-New content that you want to use for the update.
RETURN VALUES
TRUE, when process is successful.
FALSE, when process is unsuccessful.
EXAMPLES
Example #1
cubrid_seq_put(3) example
<?php
$conn = cubrid_connect("localhost", 33000, "demodb", "dba");
@cubrid_execute($conn, "DROP TABLE foo");
cubrid_execute($conn, "CREATE TABLE foo(a int AUTO_INCREMENT, b set(int), c sequence(int), d char(10))");
cubrid_execute($conn, "INSERT INTO foo(a, b, c, d) VALUES(1, {1,2,3}, {11,22,33,333}, 'a')");
$req = cubrid_execute($conn, "SELECT * FROM foo", CUBRID_INCLUDE_OID);
cubrid_move_cursor($req, 1, CUBRID_CURSOR_FIRST);
$oid = cubrid_current_oid($req);
$attr = cubrid_col_get($conn, $oid, "c");
var_dump($attr);
cubrid_seq_put($conn, $oid, "c", 1, "111");
$attr = cubrid_col_get($conn, $oid, "c");
var_dump($attr);
cubrid_close_request($req);
cubrid_disconnect($conn);
?>
The above example will output:
array(4) {
[0]=>
string(2) "11"
[1]=>
string(2) "22"
[2]=>
string(2) "33"
[3]=>
string(3) "333"
}
array(4) {
[0]=>
string(3) "111"
[1]=>
string(2) "22"
[2]=>
string(2) "33"
[3]=>
string(3) "333"
}
SEE ALSO
cubrid_seq_drop(3), cubrid_seq_insert(3).
PHP Documentation Group CUBRID_SEQ_PUT(3)