Extract and count number of Duplicate rows


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract and count number of Duplicate rows
Prev   Next
# 1  
Old 03-08-2013
Extract and count number of Duplicate rows

Hi All,

I need to extract duplicate rows from a file and write these bad records into another file. And need to have a count of these bad records.
i have a command
Code:
awk '
{s[$0]++}
END {
  for(i in s) {
    if(s[i]>1) {
      print i
    }
  }
}' ${TMP_DUPE_RECS}>>${TMP_BAD_DATA_DUPE_RECS}

but this doesnt solve my problem.
HTML Code:
Input:
A
  A
  A
  B
  B
  C
HTML Code:
Desired Output:
  
A
  A
  B
Count of bad records=3
But when i run my script i get out put as:
A
B
Count of bad records=2. Which is not true.
As always any help appreciated.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reseting row count every given number of rows

I have a file with 48 rows. I am counting 6 rows and adding 6 to that number and repeating the operation, and then output the value in column 1. For the second column, I would like to get sort of a binary output (1s and 2s) every 3rd row. This is what I have: awk '{print ++src +... (1 Reply)
Discussion started by: Xterra
1 Replies

2. Shell Programming and Scripting

Extract and exclude rows based on duplicate values

Hello I have a file like this: > cat examplefile ghi|NN603762|eee mno|NN607265|ttt pqr|NN613879|yyy stu|NN615002|uuu jkl|NN607265|rrr vwx|NN615002|iii yzA|NN618555|ooo def|NN190486|www BCD|NN628717|ppp abc|NN190486|qqq EFG|NN628717|aaa HIJ|NN628717|sss > I can sort the file by... (5 Replies)
Discussion started by: CHoggarth
5 Replies

3. Shell Programming and Scripting

Extract duplicate rows with conditions

Gents Can you help please. Input file 5490921425 1 7 1310342 54909214251 5490921425 2 1 1 54909214252 5491120937 1 1 3 54911209371 5491120937 3 1 1 54911209373 5491320785 1 ... (4 Replies)
Discussion started by: jiam912
4 Replies

4. UNIX for Dummies Questions & Answers

Script to count number of rows

Hi, I need a solaris shell script to read multiple files and count number of unique name rows(strings) from those files. The input and output should be like this Input: file 1 abc cde abc ... (9 Replies)
Discussion started by: ssk250
9 Replies

5. Shell Programming and Scripting

How to extract duplicate rows

Hi! I have a file as below: line1 line2 line2 line3 line3 line3 line4 line4 line4 line4 I would like to extract duplicate lines (not unique, triplicate or quadruplicate lines). Output will be as below: line2 line2 I would appreciate if anyone can help. Thanks. (4 Replies)
Discussion started by: chromatin
4 Replies

6. UNIX for Dummies Questions & Answers

count number of rows based on other column values

Could anybody help with this? I have input below ..... david,39 david,39 emelie,40 clarissa,22 bob,42 bob,42 tim,32 bob,39 david,38 emelie,47 what i want to do is count how many names there are with different ages, so output would be like this .... david,2 emelie,2 clarissa,1... (3 Replies)
Discussion started by: itsme999
3 Replies

7. Shell Programming and Scripting

how to add the number of row and count number of rows

Hi experts a have a very large file and I need to add two columns: the first one numbering the incidence of records and the another with the total count The input file: 21 2341 A 21 2341 A 21 2341 A 21 2341 C 21 2341 C 21 2341 C 21 2341 C 21 4567 A 21 4567 A 21 4567 C ... (6 Replies)
Discussion started by: juelillo
6 Replies

8. UNIX for Dummies Questions & Answers

how to count number of rows and sum of column using awk

Hi All, I have the following input which i want to process using AWK. Rows,NC,amount 1,1202,0.192387 2,1201,0.111111 3,1201,0.123456 i want the following output count of rows = 3 ,sum of amount = 0.426954 Many thanks (2 Replies)
Discussion started by: pistachio
2 Replies

9. Shell Programming and Scripting

How to extract duplicate rows

I have searched the internet for duplicate row extracting. All I have seen is extracting good rows or eliminating duplicate rows. How do I extract duplicate rows from a flat file in unix. I'm using Korn shell on HP Unix. For.eg. FlatFile.txt ======== 123:456:678 123:456:678 123:456:876... (5 Replies)
Discussion started by: bobbygsk
5 Replies

10. Shell Programming and Scripting

Extract duplicate fields in rows

I have a input file with formating: 6000000901 ;36200103 ;h3a01f496 ; 2000123605 ;36218982 ;heefa1328 ; 2000273132 ;36246985 ;h08c5cb71 ; 2000041207 ;36246985 ;heef75497 ; Each fields is seperated by semi-comma. Sometime, the second files is... (6 Replies)
Discussion started by: anhtt
6 Replies
Login or Register to Ask a Question
DELETE(7)							   SQL Commands 							 DELETE(7)

NAME
DELETE - delete rows of a table SYNOPSIS
DELETE FROM [ ONLY ] table [ WHERE condition ] INPUTS table The name (optionally schema-qualified) of an existing table. condition This is an SQL selection query which returns the rows which are to be deleted. Refer to the SELECT statement for further description of the WHERE clause. OUTPUTS DELETE count Message returned if items are successfully deleted. The count is the number of rows deleted. If count is 0, no rows were deleted. DESCRIPTION
DELETE removes rows which satisfy the WHERE clause from the specified table. If the condition (WHERE clause) is absent, the effect is to delete all rows in the table. The result is a valid, but empty table. Tip: TRUNCATE [truncate(7)] is a PostgreSQL extension which provides a faster mechanism to remove all rows from a table. By default DELETE will delete tuples in the table specified and all its sub-tables. If you wish to only update the specific table men- tioned, you should use the ONLY clause. You must have write access to the table in order to modify it, as well as read access to any table whose values are read in the condition. USAGE
Remove all films but musicals: DELETE FROM films WHERE kind <> 'Musical'; SELECT * FROM films; code | title | did | date_prod | kind | len -------+---------------------------+-----+------------+---------+------- UA501 | West Side Story | 105 | 1961-01-03 | Musical | 02:32 TC901 | The King and I | 109 | 1956-08-11 | Musical | 02:13 WD101 | Bed Knobs and Broomsticks | 111 | | Musical | 01:57 (3 rows) Clear the table films: DELETE FROM films; SELECT * FROM films; code | title | did | date_prod | kind | len ------+-------+-----+-----------+------+----- (0 rows) COMPATIBILITY
SQL92 SQL92 allows a positioned DELETE statement: DELETE FROM table WHERE CURRENT OF cursor where cursor identifies an open cursor. Interactive cursors in PostgreSQL are read-only. SQL - Language Statements 2002-11-22 DELETE(7)