Fastest way to delete duplicates from a large filelist..... Post: 302537331

Sponsored Content

Top Forums Shell Programming and Scripting Fastest way to delete duplicates from a large filelist..... Post 302537331 by agama on Thursday 7th of July 2011 09:52:24 PM

07-07-2011

Registered User

First off, it is wise to avoid 'loading' variables with cat -- in your case, with a million filenames/pathnames, you are likely to exceed the amount that can be stuffed into a variable. Something like this would allow you to do the same thing without issues:

Code:

while read filename
do
    echo $filename
done <file-list-file

That said, you are correct your approach isn't efficient. I interpreted your requirements to be that you need a list of files from 'filewithpath.txt' that are NOT listed in the duplicate list file. If that is the case, this should work for you:

Code:

sed 's!.*/!!' FileWithPath.txt | sort -u >/tmp/f1      # strip pathname and sort removing any dups
sort -u DuplicateFiles.txt >/tmp/f2                       # both files must be sorted for comm, remove dups just in case
comm -23 /tmp/f1 /tmp/f2 >ListWithoutDups.txt
rm /tmp/f1 /tmp/f2

I have always found the options to comm to be difficult to understand and have to read the man page nearly every time I use it. In this case, comm reads both files in parallel (thus they must be sorted) and keeps the records that are unique to the first file (not listed in the second file).

Last edited by agama; 07-07-2011 at 10:54 PM.. Reason: added comments

agama

View Public Profile for agama

Find all posts by agama

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

fastest way to remove duplicates.

I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it. Currently, I am using: sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins. Is there any other faster way...

2. UNIX for Dummies Questions & Answers

Fastest way to traverse through large directories

Hi! I have thousands of sub-directories, and hundreds of thousands of files in them. What is the fast way to find out which files are older than a certain date? Is the "find" command the fastest? Or is there some other way? Right now I have a C script that traverses through and checks...

3. Shell Programming and Scripting

how to delete/remove directory in fastest way

hello i need help to remove directory . The directory is not empty ., it contains several sub directories and files inside that.. total number of files in one directory is 12,24,446 . rm -rf doesnt work . it is prompting for every file .. i want to delete without prompting and...

4. Shell Programming and Scripting

An interactive way to delete duplicates

1)I am trying to write a script that works interactively lists duplicated records on certain field/column and asks user to delete one or more. And finally it deletes all the records the used has asked for. I have an idea to store those line numbers in an array, not sure how to do this in...

5. Shell Programming and Scripting

how can I delete duplicates in the log?

I have a log file and I am trying to run a script against it to search for key issues such as invalid users, errors etc. In one part, I grep for session closed and get a lot of the same thing,, ie. root username etc. I want to remove the multiple root and just have it do a count, like wc -l ...

6. Shell Programming and Scripting

Fastest way to delete line

I have a 5 GB text file(log/debug) I want to delete all lines containing 'TRACE' Command used sed -i '/TRACE/d' mylog.txt Is there any other fastest way to do this?

7. Shell Programming and Scripting

Delete duplicates via script?

Hello, i have the following problem: there are two folders with a lot of files. Example: FolderA contains AAA, BBB, CCC FolderB contains DDD, EEE, AAA How can i via script identify AAA as duplicate in Folder B and delete it there? So that only DDD and EEE remain, in Folder B? Thank you...

8. Shell Programming and Scripting

Delete duplicates in CA bundle

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates. The alignment of the certificate within the bundle seems to be different. Example: Cert 1...

9. Shell Programming and Scripting

Delete only if duplicates found in each record

Hi, i have another problem. I have been trying to solve it by myself but failed. inputfile ;; ID T08578 NAME T08578 SBASE 30696 EBASE 32083 TYPE P func just test func chronology func cholesterol func null INT 30765-37333 INT 37154-37318 Link 5546 Link 8142

10. Shell Programming and Scripting

To Delete the duplicates using Part of File Name

I am using the below script to delete duplicate files but it is not working for directories with more than 10k files "Argument is too long" is getting for ls -t. Tried to replace ls -t with find . -type f \( -iname "*.xml" \) -printf '%T@ %p\n' | sort -rg | sed -r 's/* //' | awk...

LEARN ABOUT DEBIAN

gpiv_fasty

GPIV_FASTY(1)						      General Commands Manual						     GPIV_FASTY(1)

NAME

       gpiv_fasty - Simple data manipulation tool for PIV data.

SYNOPSIS

       gpiv_fasty [-h | --help] [-p | --print] [-v | --version] [filename] < stdin > stdout

DESCRIPTION

       gpiv_fasty  returns fast running y-positions of data that are originally written with fast running x-positions. All other variables (parti-
       cle displacements, snr, peak#) are at the new positions as well.

       The parameters read from the configuration resources (containing the key POST) may be overruled by the command line options,  as  explained
       below.

Options
       -h | --help
	      On-line help.

       -p | --print
	      Prints  parameters, command line options and input and output filenames to stdout. The output is identic of filename.par, in case -f
	      is used.

       -v | --version
	      Print version information on standard output, then exits successfully.

       filename
	      Input Piv data file. Overrides stdin and stdout. Output will be written to  filename.ma.piv.  The  parameters  will  be  written	or
	      appended to filename.par and may be used for future use by including them in ./gpivrc.

       SEE ALSO
	      manipiv, gpivtools

AUTHOR

       Gerber Van der Graaf

								  8 November 2006						     GPIV_FASTY(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

fastest way to remove duplicates.

Discussion started by: radhika

2. UNIX for Dummies Questions & Answers

Fastest way to traverse through large directories

Discussion started by: sreedharange

3. Shell Programming and Scripting

how to delete/remove directory in fastest way

Discussion started by: getdpg

4. Shell Programming and Scripting

An interactive way to delete duplicates

Discussion started by: chvs2000