Sponsored Content
Top Forums Shell Programming and Scripting Unique files in a given directory Post 302544631 by cue on Thursday 4th of August 2011 10:39:32 AM
Old 08-04-2011
Thanks for the great help DGPickett. Could you please explain what some of the switches do and their importance; for example -n99 in xargs and -u +0 -1 in sort. The reason I ask is that I rewrote this using parallel and I'm wondering if my script will lead to some pitfalls that I've overlooked.

Code:
#!/bin/bash
#cdupes.sh
MasterL=Master.ck
CompareL=Compare.ck

#Parallel
find "$1" -type f | parallel cksum > "$MasterL"
find "$2" -type f | parallel cksum | sort > "$CompareL"

join "$CompareL" <(cut -d' ' -f1 "$MasterL" | sort -u) | cut -d' ' -f3-


Last edited by cue; 08-06-2011 at 04:12 PM.. Reason: Fixed script for unique checksums only for empty files.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Directory Inode Number Not Unique

Hi, I know that inode for each file is unique, but is it the for the directory? So far I found different directories has the same inode nubmer when you do ls -i, could some one explain why? Thanks a lot. (9 Replies)
Discussion started by: nj302
9 Replies

2. UNIX for Dummies Questions & Answers

To get unique numbers from two files

here i have two files: file 1 1 2 3 4 5 5 6 7 8 9 file 2 4 5 6 6 8 8 (6 Replies)
Discussion started by: i.scientist
6 Replies

3. Shell Programming and Scripting

Unique Directory and Folder Deletion Script

Ok, so I just got charged with the task of deleting some 300 user folders in a FTP server to free up some space. I managed to grep and cut the list of user folders to delete into a list of one user folder per line. Example: bob00 jane01 sue03 In the home folder, there are folders a-z, and... (5 Replies)
Discussion started by: b4sher
5 Replies

4. Shell Programming and Scripting

Find all images, append unique prefix to name and move to different directory

Hi, I have a directory with Multiple subdirectories and 1000s of pictures (jpg) in each directory. The problem is that each directory has a 001.jpg in them. I want to append a unique name (the directory_name)would be fine. and then move them to one main backup directory once they have been... (1 Reply)
Discussion started by: kmaq7621
1 Replies

5. UNIX for Dummies Questions & Answers

getting unique lines from 2 files

hi i have used comm -13 <(sort 1.txt) <(sort 2.txt) option to get the unique lines that are present in file 2 but not in file 1. but some how i am getting the entire file 2. i would expect few but not all uncommon lines fro my dat. is there anything wrong with the way i used the command? my... (1 Reply)
Discussion started by: anurupa777
1 Replies

6. Shell Programming and Scripting

Looping through entire directory and count unique values

Hello, I`m a complete newbie to coding, please help with this problem. I have multiple files in a directory, I have to loop through the contents of each file and extract number of unique isoforms in that file. Each file is tab delimited and only the line with the first parent (column 3)... (1 Reply)
Discussion started by: ritakadm
1 Replies

7. Shell Programming and Scripting

Extract unique files

In a incoming folder i have list of files like below,i want to pick the unique files to process the job. if same file contain more than one then it should pick latest date modified file to process. drwxrwsrwx 2 n308799 infagrp 256 May 20 17:42 Final_Working drwxrwsrwx 2... (1 Reply)
Discussion started by: katakamvivek
1 Replies

8. Shell Programming and Scripting

Add unique identifier from file to filetype in directory

I am trying to add a unique identifier to two file extensions .bam and .vcf in a directory located at /home/cmccabe/Desktop/index/R_2016_09_21_14_01_15_user_S5-00580-9-Medexome. The identifier is in $2 of the input file. What the code below is attempting to do is strip off the last portion... (21 Replies)
Discussion started by: cmccabe
21 Replies

9. Shell Programming and Scripting

Directory containing files,Print names of the files in the directory that are exactly same content.

Given a directory containing say a few thousand files, please output a list of all the names of the files in the directory that are exactly the same, i.e. have the same contents. func(a_directory_name) output -> {“matches”: , ... ]} e.g. func(“/home/my/files”) where the directory... (7 Replies)
Discussion started by: anuragpgtgerman
7 Replies

10. UNIX for Beginners Questions & Answers

Print number of lines for files in directory, also print number of unique lines

I have a directory of files, I can show the number of lines in each file and order them from lowest to highest with: wc -l *|sort 15263 Image.txt 16401 reference.txt 40459 richtexteditor.txt How can I also print the number of unique lines in each file? 15263 1401 Image.txt 16401... (15 Replies)
Discussion started by: spacegoose
15 Replies
PARPORT_REGISTER_POR(9) 				       Parallel Port Devices					   PARPORT_REGISTER_POR(9)

NAME
parport_register_port - register a parallel port SYNOPSIS
struct parport * parport_register_port(unsigned long base, int irq, int dma, struct parport_operations * ops); ARGUMENTS
base base I/O address irq IRQ line dma DMA channel ops pointer to the port driver's port operations structure DESCRIPTION
When a parallel port (lowlevel) driver finds a port that should be made available to parallel port device drivers, it should call parport_register_port. The base, irq, and dma parameters are for the convenience of port drivers, and for ports where they aren't meaningful needn't be set to anything special. They can be altered afterwards by adjusting the relevant members of the parport structure that is returned and represents the port. They should not be tampered with after calling parport_announce_port, however. If there are parallel port device drivers in the system that have registered themselves using parport_register_driver, they are not told about the port at this time; that is done by parport_announce_port. The ops structure is allocated by the caller, and must not be deallocated before calling parport_remove_port. If there is no memory to allocate a new parport structure, this function will return NULL. COPYRIGHT
Kernel Hackers Manual 2.6. July 2010 PARPORT_REGISTER_POR(9)
All times are GMT -4. The time now is 04:28 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy