How to delete duplicate entries without using awk command? Post: 302743069

10 More Discussions You Might Find Interesting

1. Ubuntu

delete duplicate rows with awk files

Hi every body I have some text file with a lots of duplicate rows like this: 165.179.568.197 154.893.836.174 242.473.396.153 165.179.568.197 165.179.568.197 165.179.568.197 154.893.836.174 how can I delete the repeated rows? Thanks Saeideh

2. Shell Programming and Scripting

Counting duplicate entries in a file using awk

Hi, I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d The file looks like 10.1.1.1 10.1.1.1 10.1.1.1 10.1.2.4 10.1.2.4 12.1.5.6 . . . . and so on.... There are duplicate/multiple entries for some IP...

3. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance

4. Shell Programming and Scripting

How can i delete the content between all the occurences of two strings using sed or awk command

Hi. I have to delete the content between all the occurrences of the xml tags in a single file. For example: * The tags <script>.....................</script> occurs more than once in the same file. * It follows tagging rules meaning a start tag will be followed by an end tag. Will not have...

5. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks

6. Shell Programming and Scripting

Help with removing duplicate entries with awk or Perl

Hi, I have a file which looks like:ke this : chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11130990 11131025 chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11131583...

7. UNIX for Dummies Questions & Answers

Need an awk command to delete a line

Hi all, As of now am using an awk command to check the number of columns in a file that has 10 lakh rows. Is it possible to remove that particular line having an extra column and copy the remaining lines to a new file ? YOUR HELP IS HIGHLY APPRECIATED. THANKS IN ADVANCE

8. Shell Programming and Scripting

Delete duplicate row

Hi all, how can delete duplicate files in file form, e.g. $cat file1 aaa 123 234 345 456 bbb 345 345 657 568 ccc 345 768 897 456 aaa 123 234 345 456 ddd 786 784 234 263 ccc 345 768 897 456 aaa 123 234 345 456 ccc 345 768 897 456 then i need ouput file1 some,

9. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same...

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ...

LEARN ABOUT CENTOS

truncate

TRUNCATE(1)							   User Commands						       TRUNCATE(1)

NAME

       truncate - shrink or extend the size of a file to the specified size

SYNOPSIS

       truncate OPTION... FILE...

DESCRIPTION

       Shrink or extend the size of each FILE to the specified size

       A FILE argument that does not exist is created.

       If  a  FILE  is	larger than the specified size, the extra data is lost.  If a FILE is shorter, it is extended and the extended part (hole)
       reads as zero bytes.

       Mandatory arguments to long options are mandatory for short options too.

       -c, --no-create
	      do not create any files

       -o, --io-blocks
	      treat SIZE as number of IO blocks instead of bytes

       -r, --reference=RFILE
	      base size on RFILE

       -s, --size=SIZE
	      set or adjust the file size by SIZE bytes

       --help display this help and exit

       --version
	      output version information and exit

       SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units are K, M, G, T, P, E, Z, Y (powers of  1024)  or  KB,  MB,  ...
       (powers of 1000).

       SIZE  may also be prefixed by one of the following modifying characters: '+' extend by, '-' reduce by, '<' at most, '>' at least, '/' round
       down to multiple of, '%' round up to multiple of.

       GNU  coreutils  online  help:  <http://www.gnu.org/software/coreutils/>	Report	truncate  translation  bugs   to   <http://translationpro-
       ject.org/team/>

AUTHOR

       Written by Padraig Brady.

COPYRIGHT

       Copyright (C) 2013 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       dd(1), truncate(2), ftruncate(2)

       The  full  documentation  for truncate is maintained as a Texinfo manual.  If the info and truncate programs are properly installed at your
       site, the command

	      info coreutils 'truncate invocation'

       should give you access to the complete manual.

GNU coreutils 8.22						     June 2014							       TRUNCATE(1)

10 More Discussions You Might Find Interesting

1. Ubuntu

delete duplicate rows with awk files

Discussion started by: sashtari

2. Shell Programming and Scripting

Counting duplicate entries in a file using awk

Discussion started by: sajal.bhatia

3. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Discussion started by: cola

4. Shell Programming and Scripting

How can i delete the content between all the occurences of two strings using sed or awk command

Discussion started by: satheeshkumar