Counting duplicate entries in a file using awk Post: 302462283

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Counting The Number Of Duplicate Lines In a File

Hello. First time poster here. I have a huge file of IP numbers. I am trying to output only the class b of the IPs and rank them by most common and output the total # of duplicate class b's before the class b. An example is below: 12.107.1.1 12.107.9.54 12.108.3.89 12.109.109.4 12.109.6.3 ...

2. Programming

Counting duplicate chars in C

Hi, im trying to create a C program that will count the number of characters, duplicate characters and non duplicate characters in a file and output this to the screen. Here is my code so far: #include <stdio.h> int main( void ) { char c; int duplicate = 0; int nonduplicate = 0; int...

3. Shell Programming and Scripting

Counting multiple entries in a file using awk

Hi, I have a big file (~960MB) having epoch time values (~50 million entries) which looks like 897393601 897393601 897393601 897393601 897393602 897393602 897393602 897393602 897393602 897393603 897393603 897393603 897393603 and so on....each time stamp has more than one...

4. Shell Programming and Scripting

AWK Command to duplicate lines in a file?

Hi, I have a file with date in it like: UserString1 UserString2 UserString3 UserString4 UserString5 I need two entries for each line so it reads like UserString1 UserString1 UserString2 UserString2 etc. Can someone help me with the awk command please? Thanks

5. Shell Programming and Scripting

counting particular record format in a file using AWK

I am trying to count records of particular format from a file and assign it to a variable. I tried below command br_count=wc -l "inputfile.dat"| awk -F"|" '{if (NF != "14") print }' but I amnot able to get it done. Please share me some idea how to get it done. Thanks in advance

6. Shell Programming and Scripting

Counting entries in a file

Hi, I have a very large two column log file in the following format: # Epoch Time IP Address 899726401 112.254.1.0 899726401 112.254.1.0 899726402 154.162.38.0 899726402 160.114.12.0 899726402 165.161.7.0 899726403 ...

7. Shell Programming and Scripting

Help with removing duplicate entries with awk or Perl

Hi, I have a file which looks like:ke this : chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11130990 11131025 chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11131583...

8. Shell Programming and Scripting

How to delete duplicate entries without using awk command?

Hello.. I am trying to remove the duplicate entries in a log files and used the the below shell script to do the same. awk '!x++' <filename> Can I do without using the awk command and the regex? I do not want to start the search from the beginning of the line in the log file as it contains...

9. Shell Programming and Scripting

Shell script with awk command for counting in a file

Hi, I hope you can help me with the awk command in shell scripting. I want to do the following, but it doesn't work. for i in $REF1 $REF2 $REF3; do awk '{if($n>=0 && $n<=50000){count+=1}} END{print count}' ${DIR}${i} >${DIR}${i}_count.txt done REF1 to REF3 are only variables for .txt...

10. Shell Programming and Scripting

Counting lines in a file using awk

I want to count lines of a file using AWK (only) and not in the END part like this awk 'END{print FNR}' because I want to use it. Does anyone know of a way? Thanks a lot.

LEARN ABOUT DEBIAN

uniq

UNIQ(1) 							   User Commands							   UNIQ(1)

NAME

       uniq - report or omit repeated lines

SYNOPSIS

       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION

       Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).

       With no options, matching lines are merged to the first occurrence.

       Mandatory arguments to long options are mandatory for short options too.

       -c, --count
	      prefix lines by the number of occurrences

       -d, --repeated
	      only print duplicate lines

       -D, --all-repeated[=delimit-method]
	      print all duplicate lines delimit-method={none(default),prepend,separate} Delimiting is done with blank lines

       -f, --skip-fields=N
	      avoid comparing the first N fields

       -i, --ignore-case
	      ignore differences in case when comparing

       -s, --skip-chars=N
	      avoid comparing the first N characters

       -u, --unique
	      only print unique lines

       -z, --zero-terminated
	      end lines with 0 byte, not newline

       -w, --check-chars=N
	      compare no more than N characters in lines

       --help display this help and exit

       --version
	      output version information and exit

       A field is a run of blanks (usually spaces and/or TABs), then non-blank characters.  Fields are skipped before chars.

       Note:  'uniq'  does  not  detect  repeated  lines unless they are adjacent.  You may want to sort the input first, or use `sort -u' without
       `uniq'.	Also, comparisons honor the rules specified by `LC_COLLATE'.

AUTHOR

       Written by Richard M. Stallman and David MacKenzie.

REPORTING BUGS

       Report uniq bugs to bug-coreutils@gnu.org
       GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
       General help using GNU software: <http://www.gnu.org/gethelp/>
       Report uniq translation bugs to <http://translationproject.org/team/>

COPYRIGHT

       Copyright (C) 2011 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       comm(1), join(1)

       The full documentation for uniq is maintained as a Texinfo manual.  If the info and uniq programs are properly installed at your site,  the
       command

	      info coreutils 'uniq invocation'

       should give you access to the complete manual.

GNU coreutils 8.12.197-032bb					  September 2011							   UNIQ(1)