Sponsored Content
Top Forums Shell Programming and Scripting How to : Find duplicate number from file? with bash Post 302268323 by joeyg on Monday 15th of December 2008 11:32:05 AM
Old 12-15-2008
Hammer & Screwdriver Without seeing your data, here is one possible solution

Code:
> cat file97
101
102
103
104
105
106
107
108
109
110
105
109
> sort -n file97 | uniq -c | awk '$1>1 {print $2}'
105
109

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Counting The Number Of Duplicate Lines In a File

Hello. First time poster here. I have a huge file of IP numbers. I am trying to output only the class b of the IPs and rank them by most common and output the total # of duplicate class b's before the class b. An example is below: 12.107.1.1 12.107.9.54 12.108.3.89 12.109.109.4 12.109.6.3 ... (2 Replies)
Discussion started by: crunchtime
2 Replies

2. Shell Programming and Scripting

Bash Script duplicate file names

I am trying to write a housekeeping bash script. Part of it involves searching all of my attached storage media for photographs and moving them into a single directory. The problem occurs when files have duplicate names, obviously a file called 001.jpg will get overwritten with another file... (6 Replies)
Discussion started by: stumpyuk
6 Replies

3. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

4. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

5. Shell Programming and Scripting

Find the number of non-duplicate names recursively.

Hi, here comes another newbie question: How to find the number of non-duplicate names recursively? For example, my files are stored in the folders like: If I do find . -depth -name "*.txt" | wc -l This will gives out a result "4". One .txt file named "1.txt" in folder "1", and... (2 Replies)
Discussion started by: jiapei100
2 Replies

6. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

7. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

8. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance! (4 Replies)
Discussion started by: vlm
4 Replies

9. Shell Programming and Scripting

Bash script to find the number of files and identify which ones are 0 bytes.

I am writing a bash script to find out all the files in a directory which are empty. I am running into multiple issues. I will really appreciate if someone can please help me. #!/bin/bash DATE=$(date +%m%d%y) TIME=$(date +%H%M) DIR="/home/statsetl/input/civil/test" ... (1 Reply)
Discussion started by: monasharma13
1 Replies

10. Shell Programming and Scripting

Bash Script to find/sort/move images/duplicate images from USB drive

Ultimately, I'm looking to create a script that allows me to plug in a usb drive with lots of jpegs on it & copy them over to a folder on my hard drive. So in the process of copying I am looking to hash check them, record dupes to a file, copy only 1 of the identical files (if it doesn't exsist... (1 Reply)
Discussion started by: JonaQuinn
1 Replies
ascii(5)							File Formats Manual							  ascii(5)

NAME
ascii - Octal, hexadecimal, and decimal ASCII character sets DESCRIPTION
The octal character set is: 000 nul 020 dle 040 sp 060 0 100 @ 120 P 140 ` 160 p 001 soh 021 dc1 041 ! 061 1 101 A 121 Q 141 a 161 q 002 stx 022 dc2 042 " 062 2 102 B 122 R 142 b 162 r 003 etx 023 dc3 043 # 063 3 103 C 123 S 143 c 163 s 004 eot 024 dc4 044 $ 064 4 104 D 124 T 144 d 164 t 005 enq 025 nak 045 % 065 5 105 E 125 U 145 e 165 u 006 ack 026 syn 046 & 066 6 106 F 126 V 146 f 166 v 007 bel 027 etb 047 ' 067 7 107 G 127 W 147 g 167 w 010 bs 030 can 050 ( 070 8 110 H 130 X 150 h 170 x 011 ht 031 em 051 ) 071 9 111 I 131 Y 151 i 171 y 012 nl 032 sub 052 * 072 : 112 J 132 Z 152 j 172 z 013 vt 033 esc 053 + 073 ; 113 K 133 [ 153 k 173 { 014 np 034 fs 054 , 074 < 114 L 134 154 l 174 | 015 cr 035 gs 055 - 075 = 115 M 135 ] 155 m 175 } 016 so 036 rs 056 . 076 > 116 N 136 ^ 156 n 176 ~ 017 si 037 us 057 / 077 ? 117 O 137 _ 157 o 177 del The hexadecimal character set is: 00 nul 10 dle 20 sp 30 0 40 @ 50 P 60 ` 70 p 01 soh 11 dc1 21 ! 31 1 41 A 51 Q 61 a 71 q 02 stx 12 dc2 22 " 32 2 42 B 52 R 62 b 72 r 03 etx 13 dc3 23 # 33 3 43 C 53 S 63 c 73 s 04 eot 14 dc4 24 $ 34 4 44 D 54 T 64 d 74 t 05 enq 15 nak 25 % 35 5 45 E 55 U 65 e 75 u 06 ack 16 syn 26 & 36 6 46 F 56 V 66 f 76 v 07 bel 17 etb 27 ' 37 7 47 G 57 W 67 g 77 w 08 bs 18 can 28 ( 38 8 48 H 58 X 68 h 78 x 09 ht 19 em 29 ) 39 9 49 I 59 Y 69 i 79 y 0a nl 1a sub 2a * 3a : 4a J 5a Z 6a j 7a z 0b vt 1b esc 2b + 3b ; 4b K 5b [ 6b k 7b { 0c np 1c fs 2c , 3c < 4c L 5c 6c l 7c | 0d cr 1d gs 2d - 3d = 4d M 5d ] 6d m 7d } 0e so 1e rs 2e . 3e > 4e N 5e ^ 6e n 7e ~ 0f si 1f us 2f / 3f ? 4f O 5f _ 6f o 7f del The decimal character set is: 0 nul 16 dle 32 sp 48 0 64 @ 80 P 96 ` 112 p 1 soh 17 dc1 33 ! 49 1 65 A 81 Q 97 a 113 q 2 stx 18 dc2 34 " 50 2 66 B 82 R 98 b 114 r 3 etx 19 dc3 35 # 51 3 67 C 83 S 99 c 115 s 4 eot 20 dc4 36 $ 52 4 68 D 84 T 100 d 116 t 5 enq 21 nak 37 % 53 5 69 E 85 U 101 e 117 u 6 ack 22 syn 38 & 54 6 70 F 86 V 102 f 118 v 7 bel 23 etb 39 ' 55 7 71 G 87 W 103 g 119 w 8 bs 24 can 40 ( 56 8 72 H 88 X 104 h 120 x 9 ht 25 em 41 ) 57 9 73 I 89 Y 105 i 121 y 10 nl 26 sub 42 * 58 : 74 J 90 Z 106 j 122 z 11 vt 27 esc 43 + 59 ; 75 K 91 [ 107 k 123 { 12 np 28 fs 44 , 60 < 76 L 92 108 l 124 | 13 cr 29 gs 45 - 61 = 77 M 93 ] 109 m 125 } 14 so 30 rs 46 . 62 > 78 N 94 ^ 110 n 126 ~ 15 si 31 us 47 / 63 ? 79 O 95 _ 111 o 127 del ascii(5)
All times are GMT -4. The time now is 07:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy