Sponsored Content
Top Forums Shell Programming and Scripting how to remove duplicate lines Post 302099681 by fredao on Wednesday 13th of December 2006 10:46:46 AM
Old 12-13-2006
how to remove duplicate lines

I have following file content (3 fields each line):

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
df dag 10.0.0.5
dfd dfdas 10.0.0.5
dfd dfd 10.0.0.6
daf nfd 10.0.0.6
...

as can be seen, that the third field is ip address and sorted. but there is duplilcate ip addresses. I want to kill line 6,7 and 9, so the new file will be

23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
dfd dfd 10.0.0.6
...

please tell me how to do this, thanks!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

4. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

Need to remove the duplicate lines from a log!!

Hello Folks, Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's. Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies

6. Shell Programming and Scripting

remove duplicate lines with condition

hi to all Does anyone know if there's a way to remove duplicate lines which we consider the same only if they have the first and the second column the same? For example I have : us2333 bbb 5 us2333 bbb 3 us2333 bbb 2 and I want to get us2333 bbb 10 The thing is I cannot... (2 Replies)
Discussion started by: vlm
2 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
MKTEMP(3)						   BSD Library Functions Manual 						 MKTEMP(3)

NAME
mktemp, mkstemp, mkdtemp, mkdtempat_np, mkstemps, mkstempsat_np, mkostemp, mkostemps, mkostempsat_np -- make temporary file name (unique) LIBRARY
Standard C Library (libc, -lc) SYNOPSIS
#include <unistd.h> char * mktemp(char *template); int mkstemp(char *template); char * mkdtemp(char *template); char * mkdtempat_np(int dfd, char *template); int mkstemps(char *template, int suffixlen); int mkstempsat_np(int dfd, char *template, int suffixlen); int mkostemp(char *template, int oflags); int mkostemps(char *template, int suffixlen, int oflags); int mkostempsat_np(int dfd, char *template, int suffixlen, int oflags); DESCRIPTION
The mktemp() function takes the given file name template and overwrites a portion of it to create a file name. This file name is guaranteed not to exist at the time of function invocation and is suitable for use by the application. The template may be any file name with some num- ber of 'Xs' appended to it, for example /tmp/temp.XXXXXX. The trailing 'Xs' are replaced with a unique alphanumeric combination. The number of unique file names mktemp() can return depends on the number of 'Xs' provided; six 'Xs' will result in mktemp() selecting one of 56800235584 (62 ** 6) possible temporary file names. The mkstemp() function makes the same replacement to the template and creates the template file, mode 0600, returning a file descriptor opened for reading and writing. This avoids the race between testing for a file's existence and opening it for use. The mkdtemp() function makes the same replacement to the template as in mktemp() and creates the template directory, mode 0700. The mkostemp() function is like mkstemp() but allows specifying additional open(2) flags (defined in <fcntl.h>). The permitted flags are O_APPEND, O_SHLOCK, O_EXLOCK and O_CLOEXEC. The mkstemps() and mkostemps() functions act the same as mkstemp() and mkostemp() respectively, except they permit a suffix to exist in the template. The template should be of the form /tmp/tmpXXXXXXsuffix. The mkstemps() and mkostemps() functions are told the length of the suf- fix string. The mkdtempat_np(), mkstempsat_np(), and mkostempsat_np() functions act the same as mkdtemp(), mkstemps(), and mkostemps() respectively, except in the case where the path specifies a relative path. In this case, path is treated as relative to the directory associated with the file descriptor fd instead of the current working directory. RETURN VALUES
The mktemp() and mkdtemp() functions return a pointer to the template on success and NULL on failure. The mkstemp(), mkostemp(), mkstemps() and mkostemps() functions return -1 if no suitable file could be created. If either call fails an error code is placed in the global vari- able errno. ERRORS
The mkstemp(), mkostemp(), mkstemps(), mkostemps(), and mkdtemp() functions may set errno to one of the following values: [ENOTDIR] The pathname portion of the template is not an existing directory. The mkostemp() and mkostemps() functions may also set errno to the following value: [EINVAL] The oflags argument is invalid. The mkstemp(), mkostemp(), mkstemps(), mkostemps() and mkdtemp() functions may also set errno to any value specified by the stat(2) function. The mkstemp(), mkostemp(), mkstemps() and mkostemps() functions may also set errno to any value specified by the open(2) function. The mkdtemp() function may also set errno to any value specified by the mkdir(2) function. NOTES
A common problem that results in a core dump is that the programmer passes in a read-only string to mktemp(), mkstemp(), mkstemps() or mkdtemp(). This is common with programs that were developed before ISO/IEC 9899:1990 (``ISO C90'') compilers were common. For example, calling mkstemp() with an argument of "/tmp/tempfile.XXXXXX" will result in a core dump due to mkstemp() attempting to modify the string con- stant that was given. LEGACY SYNOPSIS
#include <unistd.h> The include file <unistd.h> is necessary and sufficient for all functions. SEE ALSO
chmod(2), getpid(2), mkdir(2), open(2), stat(2), compat(5) STANDARDS
The mkstemp() and mkdtemp() functions are expected to conform to IEEE Std 1003.1-2008 (``POSIX.1''). The mktemp() function is expected to conform to IEEE Std 1003.1-2001 (``POSIX.1'') and is not specified by IEEE Std 1003.1-2008 (``POSIX.1''). The mkostemp(), mkstemps() and mkostemps() functions do not conform to any standard. HISTORY
A mktemp() function appeared in Version 7 AT&T UNIX. The mkstemp() function appeared in 4.4BSD. The mkdtemp() function first appeared in OpenBSD 2.2, and later in FreeBSD 3.2. The mkstemps() function first appeared in OpenBSD 2.4, and later in FreeBSD 3.4. The mkostemp() and mkostemps() functions appeared in OS X 10.12. BUGS
This family of functions produces filenames which can be guessed, though the risk is minimized when large numbers of 'Xs' are used to increase the number of possible temporary filenames. This makes the race in mktemp(), between testing for a file's existence (in the mktemp() function call) and opening it for use (later in the user application) particularly dangerous from a security perspective. Whenever it is possible, mkstemp() or mkostemp() should be used instead, since it does not have the race condition. If mkstemp() cannot be used, the filename created by mktemp() should be created using the O_EXCL flag to open(2) and the return status of the call should be tested for fail- ure. This will ensure that the program does not continue blindly in the event that an attacker has already created the file with the inten- tion of manipulating or reading its contents. BSD
December 18, 2015 BSD
All times are GMT -4. The time now is 01:41 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy