02-25-2019
And, pls add a definition of what makes a "duplicate" - a common file name? Common meta data as e.g. size, time stamps? Identical contents / check sum?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I want to find duplicates in file on 2nd field i wrote this code:
nawk '{a++} END{for i in a {if (a>1) print}}' temp
Could not find whats wrong with this.
Appreciate help (5 Replies)
Discussion started by: pinnacle
5 Replies
2. Shell Programming and Scripting
I have a folder which in turn has numerous sub folders all containing pdf files with same file named in different ways.
So I need a script if it can be written to find and print the duplicate files (That is files with same size) along with the respective paths.
So I assume here that same file... (5 Replies)
Discussion started by: deaddevil
5 Replies
3. Shell Programming and Scripting
Hi All,
I am Oracle Apps Tech guy, I have a requirement to find 777 permission is there or not for all Folders and Sub-folders
Under APPL_TOP (Folder/directory) with below conditions
i) the directory names should start with xx..... (like xxau,xxcfi,xxcca...etc)
and exclude the directory... (11 Replies)
Discussion started by: gagan4599
11 Replies
4. Shell Programming and Scripting
Hello,
My text file has input of the form
abc dft45.xml
ert rt653.xml
abc ert57.xml
I need to write a perl script/shell script to find duplicates in the first column and write it into a text file of the form...
abc dft45.xml
abc ert57.xml
Can some one help me plz? (5 Replies)
Discussion started by: gameboy87
5 Replies
5. UNIX for Dummies Questions & Answers
example data
5666700842511TAfmoham03151008075205999900000001000001000++
5666700843130MAfmoham03151008142606056667008390315100005001
6666666663130MAfmoham03151008142606056667008390315100005001
I'd like to sort on position 10-14 where the characters are eq "130MA".
Then based on positions... (0 Replies)
Discussion started by: mmarshall
0 Replies
6. UNIX for Dummies Questions & Answers
Hi all,
Using grep command, i want to find the pattern of text in all directories and sub-directories.
e.g: if i want to search for a pattern named "parmeter", i used the command
grep -i "param" ../*
is this correct? (1 Reply)
Discussion started by: vinothrajan55
1 Replies
7. Shell Programming and Scripting
these are numeric ids..
222932017099186177
222932014385467392
222932017371820032
222932017409556480
I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way..
sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies
8. Shell Programming and Scripting
Hi All,
Input.txt
123,ABC,XYZ1,A01,IND,I68,IND,NN
123,ABC,XYZ1,A01,IND,I67,IND,NN
998,SGR,St,R834,scot,R834,scot,NN
985,SGR0399,St,R180,T15,R180,T1,YY
985,SGR0399,St,R180,T15,R180,T1,NN
985,SGR0399,St,R180,T15,R180,T1,NN
2943,SGR?99,St,R68,Scot,R77,Scot,YY... (2 Replies)
Discussion started by: unme
2 Replies
9. Shell Programming and Scripting
with below given format,
I have been trying to find out all IDs for those entries with duplicate names in 2nd and 3rd columns and their count like how many time duplication happened for any name if any,
0.237788 Aaban Aahva
0.291066 Aabheer Aahlaad
0.845814 Aabid Aahan
0.152208 Aadam... (6 Replies)
Discussion started by: busyboy
6 Replies
10. UNIX for Beginners Questions & Answers
Hello All,
This is a noob question. I tried searching for the answer but the answer found did not help me .
I have a file that can have duplicates.
100
200
300
400
100
150
the number 100 is duplicated twice. I want to find the duplicate along with the line number.
expected... (4 Replies)
Discussion started by: vatigers
4 Replies
LEARN ABOUT CENTOS
hardlink
hardlink(1) General Commands Manual hardlink(1)
NAME
hardlink - Consolidate duplicate files via hardlinks
SYNOPSIS
hardlink [-c] [-n] [-v] [-vv] [-h] directory1 [ directory2 ... ]
DESCRIPTION
This manual page documents hardlink, a program which consolidates duplicate files in one or more directories using hardlinks.
hardlink traverses one or more directories searching for duplicate files. When it finds duplicate files, it uses one of them as the mas-
ter. It then removes all other duplicates and places a hardlink for each one pointing to the master file. This allows for conservation of
disk space where multiple directories on a single filesystem contain many duplicate files.
Since hard links can only span a single filesystem, hardlink is only useful when all directories specified are on the same filesystem.
OPTIONS
-c Compare only the contents of the files being considered for consolidation. Disregards permission, ownership and other differ-
ences.
-f Force hardlinking across file systems.
-n Do not perform the consolidation; only print what would be changed.
-v Print summary after hardlinking.
-vv Print every hardlinked file and bytes saved. Also print summary after hardlinking.
-h Show help.
AUTHOR
hardlink was written by Jakub Jelinek <jakub@redhat.com>.
Man page written by Brian Long.
Man page updated by Jindrich Novy <jnovy@redhat.com>
BUGS
hardlink assumes that its target directory trees do not change from under it. If a directory tree does change, this may result in hardlink
accessing files and/or directories outside of the intended directory tree. Thus, you must avoid running hardlink on potentially changing
directory trees, and especially on directory trees under control of another user.
hardlink(1)