Sponsored Content
Top Forums Shell Programming and Scripting Remove lines with duplicate pairs where AB is equal to BA Post 302922671 by Akshay Hegde on Monday 27th of October 2014 03:11:34 PM
Old 10-27-2014
Small modification to Corona's solution

Code:
awk '!(SEEN[$1,$4]++) && !(($4,$1) in SEEN)'  infile

This User Gave Thanks to Akshay Hegde For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

4. Shell Programming and Scripting

Need to remove the duplicate lines from a log!!

Hello Folks, Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's. Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies

5. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

6. Shell Programming and Scripting

remove duplicate lines with condition

hi to all Does anyone know if there's a way to remove duplicate lines which we consider the same only if they have the first and the second column the same? For example I have : us2333 bbb 5 us2333 bbb 3 us2333 bbb 2 and I want to get us2333 bbb 10 The thing is I cannot... (2 Replies)
Discussion started by: vlm
2 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

So I have a bunch of files that look like this >gi|33332323 MMKCRGVIMVVEKVMKRDGRIVPFDESRIRWAVQ--- >gi|45235353 MMKCR----VEKMRDVFFDESIRWAVQ They go on...sequences are much longer but all in two line (fasta) format. I want to remove duplicate pairs of ID(GI) number and sequence. I tried... (12 Replies)
Discussion started by: bakere19
12 Replies

8. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

9. Shell Programming and Scripting

Remove from a file all lines with value equal to 0

Dear All, I have a file containing 1134 columns and 20825 rows, tabulated as follow -4000 -3900 -3800 -3700 -3600 -3500 -3400 NR_033530 0 0 0 0 0 0 0 NM_001162375 0 0 0 0 0 0 0 NM_007669 0 0 0 0 0 0 328,98 NM_008104 0 388,94 388,94 388,94 0 0 0 NM_010472 0 0 0 0 0 0 0... (7 Replies)
Discussion started by: paolo.kunder
7 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
wstring(3C)						   Standard C Library Functions 					       wstring(3C)

NAME
wstring, wscasecmp, wsncasecmp, wsdup, wscol - Process Code string operations SYNOPSIS
#include <widec.h> int wscasecmp(const wchar_t *s1, const wchar_t *s2); int wsncasecmp(const wchar_t *s1, const wchar_t *s2, int n); wchar_t *wsdup(const wchar_t *s); int wscol(const wchar_t *s); DESCRIPTION
These functions operate on Process Code strings terminated by wchar_t null characters. During appending or copying, these routines do not check for an overflow condition of the receiving string. In the following, s, s1, and s2 point to Process Code strings terminated by a wchar_t null. wscasecmp(), wsncasecmp() The wscasecmp() function compares its arguments, ignoring case, and returns an integer greater than, equal to, or less than 0, depending upon whether s1 is lexicographically greater than, equal to, or less than s2. It makes the same comparison but compares at most n Process Code characters. The four Extended Unix Code (EUC) codesets are ordered from lowest to highest as 0, 2, 3, 1 when characters from different codesets are compared. wsdup() The wsdup() function returns a pointer to a new Process Code string, which is a duplicate of the string pointed to by s. The space for the new string is obtained using malloc(3C). If the new string cannot be created, a null pointer is returned. wscol() The wscol() function returns the screen display width (in columns) of the Process Code string s. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |MT-Level |MT-Safe | +-----------------------------+-----------------------------+ SEE ALSO
malloc(3C), string(3C), wcstring(3C), attributes(5) SunOS 5.10 29 Dec 1996 wstring(3C)
All times are GMT -4. The time now is 05:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy