Sponsored Content
Top Forums Shell Programming and Scripting how to remove duplicate lines Post 302099681 by fredao on Wednesday 13th of December 2006 10:46:46 AM
Old 12-13-2006
how to remove duplicate lines

I have following file content (3 fields each line):

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
df dag 10.0.0.5
dfd dfdas 10.0.0.5
dfd dfd 10.0.0.6
daf nfd 10.0.0.6
...

as can be seen, that the third field is ip address and sorted. but there is duplilcate ip addresses. I want to kill line 6,7 and 9, so the new file will be

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
dfd dfd 10.0.0.6
...

please tell me how to do this, thanks!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

4. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

Need to remove the duplicate lines from a log!!

Hello Folks, Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's. Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies

6. Shell Programming and Scripting

remove duplicate lines with condition

hi to all Does anyone know if there's a way to remove duplicate lines which we consider the same only if they have the first and the second column the same? For example I have : us2333 bbb 5 us2333 bbb 3 us2333 bbb 2 and I want to get us2333 bbb 10 The thing is I cannot... (2 Replies)
Discussion started by: vlm
2 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
FILEDESC(9)						   BSD Kernel Developer's Manual					       FILEDESC(9)

NAME
filedesc, dupfdopen, falloc, fd_getfile, fdalloc, fdcheckstd, fdclear, fdclone, fdcloseexec, fdcopy, fdexpand, fdfree, fdinit, fdrelease, fdremove, fdshare, fdunshare -- file descriptor tables and operations SYNOPSIS
#include <sys/file.h> #include <sys/filedesc.h> int falloc(struct lwp *l, struct file **resultfp, int *resultfd); struct file * fd_getfile(struct filedesc *fdp, int fd); int dupfdopen(struct lwp *l, int indx, int dfd, int mode, int error); int fdalloc(struct proc *p, int want, int *result); int fdcheckstd(struct lwp *l); void fdclear(struct lwp *l); int fdclone(struct lwp *l, struct file *fp, int fd, int flag, const struct fileops *fops, void *data); void fdcloseexec(struct lwp *l); struct filedesc * fdcopy(struct proc *p); void fdexpand(struct proc *p); void fdfree(struct lwp *l); struct filedesc * fdinit(struct proc *p); int fdrelease(struct lwp *l, int fd); void fdremove(struct filedesc *fdp, int fd); void fdshare(struct proc *p1, struct proc *p2); void fdunshare(struct lwp *l); DESCRIPTION
For user processes, all I/O is done through file descriptors. These file descriptors represent underlying objects supported by the kernel and are created by system calls specific to the type of object. In NetBSD, six types of objects can be represented by file descriptors: data files, pipes, sockets, event queues, crypto, and miscellaneous. The kernel maintains a descriptor table for each process which is used to translate the external representation of a file descriptor into an internal representation. The file descriptor is merely an index into this table. The file descriptor table maintains the following informa- tion: o the number of descriptors allocated in the file descriptor table; o approximate next free descriptor; o a reference count on the file descriptor table; and o an array of open file entries. On creation of the file descriptor table, a fixed number of file entries are created. It is the responsibility of the file descriptor opera- tions to expand the available number of entries if more are required. Each file entry in the descriptor table contains the information nec- essary to access the underlying object and to maintain common information. See file(9) for details of operations on the file entries. New file descriptors are generally allocated by falloc() and freed by fdrelease(). File entries are extracted from the file descriptor table by fd_getfile(). Most of the remaining functions in the interface are purpose specific and perform lower-level file descriptor operations. FUNCTIONS
The following functions are high-level interface routines to access the file descriptor table for a process and its file entries. falloc(p, *resultfp, *resultfd) Create a new open file entry and allocate a file descriptor for process p. This operation is performed by invoking fdalloc() to allocate the new file descriptor. The credential on the file entry are inherited from process p. The falloc() function is respon- sible for expanding the file descriptor table when necessary. A pointer to the file entry is returned in *resultfp and the file descriptor is returned in *resultfd. The falloc() function returns zero on success, otherwise an appropriate error is returned. fd_getfile(fdp, fd) Get the file entry for file descriptor fd in the file descriptor table fdp. The file entry is returned if it is valid and useable, otherwise NULL is returned. dupfdopen(l, indx, dfd, mode, error) Duplicate file descriptor dfd for lwp l. The following functions operate on the file descriptor table for a process. fdalloc(p, want, *result) Allocate a file descriptor want for process p. The resultant file descriptor is returned in *result. The fdalloc() function returns zero on success, otherwise an appropriate error is returned. fdcheckstd(l) Check the standard file descriptors 0, 1, and 2 and ensure they are referencing valid file descriptors. If they are not, create references to /dev/null. This operation is necessary as these file descriptors are given implicit significance in the Standard C Library and it is unsafe for setuid(2) and setgid(2) processes to be started with these file descriptors closed. fdclear(l) Clear the descriptor table for lwp l. This operation is performed by invoking fdinit() to initialise a new file descriptor table to replace the old file descriptor table and invoking fdfree() to release the old one. fdclone(l, fp, fd, flag, fops, data) This function is meant to be used by devices which allocate a file entry upon open. fdclone() fills fp with the given parameters. It always returns the in-kernel errno value EMOVEFD, which is meant to be returned from the device open routine. This special return value is interpreted by the caller of the device open routine. fdcloseexec(l) Close any files for process p that are marked ``close on exec''. This operation is performed by invoking fdunshare() for the process and invoking fdrelease() on the appropriate file descriptor. fdcopy(p) Copy the file descriptor table from process p and return a pointer to the copy. The returned file descriptor is guaranteed to have a reference count of one. All file descriptor state is maintained. The reference counts on each file entry referenced by the file descriptor table is incremented accordingly. fdexpand(p) Expand the file descriptor table for process p by allocating memory for additional file descriptors. fdfree(l) Decrement the reference count on the file descriptor table for lwp l and release the file descriptor table if the reference count drops to zero. fdinit(p) Create a file descriptor table using the same current and root directories of process p. The returned file descriptor table is guaranteed to have a reference count of one. fdrelease(l, fd) Remove file descriptor fd from the file descriptor table of lwp l. The operation is performed by invoking closef(). fdremove(fdp, fd) Unconditionally remove the file descriptor fd from file descriptor table fdp. fdshare(p1, p2) Share the file descriptor table belonging to process p1 with process p2. Process p2 is assumed not to have a file descriptor table already allocated. The reference count on the file descriptor table is incremented. This function is used by fork1(9). fdunshare(l) Ensure that lwp l does not share its file descriptor table. If its file descriptor table has more than one reference, the file descriptor table is copied by invoking fdcopy(). The reference count on the original file descriptor table is decremented. RETURN VALUES
Successful operations return zero. A failed operation will return a non-zero return value. Possible values include: [EBADF] Bad file descriptor specified. [EMFILE] Cannot exceed file descriptor limit. [ENOSPC] No space left in file descriptor table. CODE REFERENCES
The framework for file descriptor handling is implemented within the file sys/kern/kern_descrip.c. SEE ALSO
file(9) BSD
July 24, 2006 BSD
All times are GMT -4. The time now is 06:40 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy