Sponsored Content
Top Forums Shell Programming and Scripting how to remove duplicate lines Post 302099681 by fredao on Wednesday 13th of December 2006 10:46:46 AM
Old 12-13-2006
how to remove duplicate lines

I have following file content (3 fields each line):

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
df dag 10.0.0.5
dfd dfdas 10.0.0.5
dfd dfd 10.0.0.6
daf nfd 10.0.0.6
...

as can be seen, that the third field is ip address and sorted. but there is duplilcate ip addresses. I want to kill line 6,7 and 9, so the new file will be

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
dfd dfd 10.0.0.6
...

please tell me how to do this, thanks!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

4. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

Need to remove the duplicate lines from a log!!

Hello Folks, Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's. Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies

6. Shell Programming and Scripting

remove duplicate lines with condition

hi to all Does anyone know if there's a way to remove duplicate lines which we consider the same only if they have the first and the second column the same? For example I have : us2333 bbb 5 us2333 bbb 3 us2333 bbb 2 and I want to get us2333 bbb 10 The thing is I cannot... (2 Replies)
Discussion started by: vlm
2 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
DIAL(2) 							System Calls Manual							   DIAL(2)

NAME
dial, hangup, announce, listen, accept, reject, netmkaddr - make and break network connections SYNOPSIS
#include <u.h> #include <libc.h> int dial(char *addr, char *local, char *dir, int *cfdp) int hangup(int ctl) int announce(char *addr, char *dir) int listen(char *dir, char *newdir) int accept(int ctl, char *dir) int reject(int ctl, char *dir, char *cause) char* netmkaddr(char *addr, char *defnet, char *defservice) DESCRIPTION
For these routines, addr is a network address of the form network!netaddr!service, network!netaddr, or simply netaddr. Network is any directory listed in /net or the special token, net. Net is a free variable that stands for any network in common between the source and the host netaddr. Netaddr can be a host name, a domain name, a network address, or a meta-name of the form $attribute, which is replaced by value from the value-attribute pair attribute=value most closely associated with the source host in the network data base (see ndb(6)). If a connection attempt is successful and dir is non-zero, the path name of a line directory that has files for accessing the connection is copied into dir. The path name is guaranteed to be less than 40 bytes long. One line directory exists for each possible connection. The data file in the line directory should be used to communicate with the destination. The ctl file in the line directory can be used to send commands to the line. See dk(3) and ip(3) for messages that can be written to the ctl file. The last close of the data or ctl file will close the connection. Dial makes a call to destination addr on a multiplexed network. If the network in addr is net, dial will try in succession all networks in common between source and destination until a call succeeds. It returns a file descriptor open for reading and writing the data file in the line directory. The addr file in the line directory contains the address called. If the network allows the local address to be set, as is the case with UDP and TCP port numbers, and local is non-zero, the local address will be set to local. If cfdp is non-zero, *cfdp is set to a file descriptor open for reading and writing the control file. Hangup is a means of forcing a connection to hang up without closing the ctl and data files. Announce and listen are the complements of dial. Announce establishes a network name to which calls can be made. Like dial, announce returns an open ctl file. The listen routine takes as its first argument the dir of a previous announce. When a call is received, listen returns an open ctl file for the line the call was received on. It sets newdir to the path name of the new line directory. Accept accepts a call received by listen, while reject refuses the call because of cause. Accept returns a file descriptor for the data file opened ORDWR. Netmkaddr makes an address suitable for dialing or announcing. It takes an address along with a default network and service to use if they are not specified in the address. It returns a pointer to static data holding the actual address to use. EXAMPLES
Make a call and return an open file descriptor to use for communications: int callkremvax(void) { return dial("kremvax", 0, 0, 0); } Call the local authentication server: int dialauth(char *service) { return dial(netmkaddr("$auth", 0, service), 0, 0, 0); } Announce as kremvax on Datakit and loop forever receiving calls and echoing back to the caller anything sent: int bekremvax(void) { int dfd, acfd, lcfd; char adir[40], ldir[40]; int n; char buf[256]; afd = announce("dk!kremvax", adir); if(afd < 0) return -1; for(;;){ /* listen for a call */ lcfd = listen(adir, ldir); if(lcfd < 0) return -1; /* fork a process to echo */ switch(fork()){ case -1: perror("forking"); close(lcfd); break; case 0: /* accept the call and open the data file */ dfd = accept(lcfd, ldir); if(dfd < 0) return -1; /* echo until EOF */ while((n = read(dfd, buf, sizeof(buf))) > 0) write(dfd, buf, n); exits(0); default: close(lcfd); break; } } } SOURCE
/sys/src/libc/port SEE ALSO
auth(2), dk(3), ip(3), stream(3), ndb(8) DIAGNOSTICS
Dial, announce, and listen return -1 if they fail. Hangup returns nonzero if it fails. DIAL(2)
All times are GMT -4. The time now is 11:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy