Filtering F-Dupes


 
Thread Tools Search this Thread
# 1  
Filtering F-Dupes

Is there an easy way to tell FDupes what filetypes to look at or ignore?
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove dupes in a large file

I have a large file 1.5 gb and want to sort the file. I used the following AWK script to do the job !x++ The script works but it is very slow and takes over an hour to do the job. I suspect this is because the file is not sorted. Any solution to speed up the AWk script or a Perl script would... (4 Replies)
Discussion started by: gimley
4 Replies

2. Shell Programming and Scripting

Modify script to remove dupes with two delimiters

Hello, I have a script which removes duplicates in a database with a single delimiter = The script is given below: # script to remove dupes from a row with structure word=word BEGIN{FS="="} {for(i=1;i<=NF;i++){a++;}for(i in a){b=b"="i}{sub("=","",b);$0=b;b="";delete a}}1 How do I modify... (6 Replies)
Discussion started by: gimley
6 Replies

3. Shell Programming and Scripting

Help with Perl script for identifying dupes in column1

Dear all, I have a large dictionary database which has the following structure source word=target word e.g. book=livre Since the database is very large in spite of all the care taken, it so happens that at times the source word is repeated e.g. book=livre book=tome Since I want to... (7 Replies)
Discussion started by: gimley
7 Replies

4. Shell Programming and Scripting

Identifying dupes within a database and creating unique sub-sets

Hello, I have a database of name variants with the following structure: variant=variant=variant The number of variants can be as many as thirty to forty. Since the database is quite large (at present around 60,000 lines) duplicate sets of variants creep in. Thus John=Johann=Jon and... (2 Replies)
Discussion started by: gimley
2 Replies

5. Shell Programming and Scripting

Removing dupes within 2 delimited areas in a large dictionary file

Hello, I have a very large dictionary file which is in text format and which contains a large number of sub-sections. Each sub-section starts with the following header : #DATA #VALID 1 and ends with a footer as shown below #END The data between the Header and the Footer consists of... (6 Replies)
Discussion started by: gimley
6 Replies

6. Shell Programming and Scripting

Merging dupes on different lines in a dictionary

I am working on a homonym dictionary of names i.e. names which are clustered together according to their “sound-alike” pronunciation: An example will make this clear: Since the dictionary is manually constructed it often happens that inadvertently two sets of “homonyms” which should be grouped... (2 Replies)
Discussion started by: gimley
2 Replies

7. Shell Programming and Scripting

deleting dupes in a row

Hello, I have a large database in which name homonyms are arranged in a row. Since the database is large and generated by hand, very often dupes creep in. I want to remove the dupes either using an awk or perl script. An input is given below The expected output is given below: As can be... (2 Replies)
Discussion started by: gimley
2 Replies

8. Shell Programming and Scripting

Help in modifying existing Perl Script to produce report of dupes

Hello, I have a large amount of data with the following structure: Word=Transliterated word I have written a Perl Script (reproduced below) which goes through the full file and identifies all dupes on the right hand side. It creates successfully a new file with two headers: Singletons and Dupes.... (5 Replies)
Discussion started by: gimley
5 Replies

9. Shell Programming and Scripting

Script for identifying and deleting dupes in a line

I am compiling a synonym dictionary which has the following structure Headword=Synonym1,Synonym2 and so on, with each synonym separated by a comma. As is usual in such cases manual preparation of synonyms results in repeating the synonym which results in dupes as in the example below:... (3 Replies)
Discussion started by: gimley
3 Replies

10. Shell Programming and Scripting

Using an awk script to identify dupes in two files

Hello, I have two files. File1 or the master file contains two columns separated by a delimiter: a=b b=d e=f g=h File 2 which is the file to be processed has only a single column a h c b What I need is an awk script to identify unique names from file 2 which are not found in the... (6 Replies)
Discussion started by: gimley
6 Replies
funtools(7)							SAORD Documentation						       funtools(7)

NAME
Funtools - FITS Users Need Tools SYNOPSIS
This document is the Table of Contents for Funtools. DESCRIPTION
Funtools, is a "minimal buy-in" FITS library and utility package developed at the the High Energy Astrophysics Division of SAO. The Fun- tools library provides simplified access to a wide array of file types: standard astronomical FITS images and binary tables, raw arrays and binary event lists, and even tables of ASCII column data. A sophisticated region filtering library (compatible with ds9) filters images and tables using boolean operations between geometric shapes, support world coordinates, etc. Funtools also supports advanced capabilities such as optimized data searching using index files. The main goal of the Funtools project has been to develop a minimal buy-in FITS library for researchers who are occasional (but serious) coders. In this case, "minimal buy-in" means "easy to learn, easy to use, and easy to re-learn next month". We have tried to achieve this goal by emphasizing two essential capabilities. The first is the ability to develop FITS programs without knowing much about FITS, i.e., without having to deal with the arcane rules for generating a properly formatted FITS file. The second is to support the use of already- familiar C/Unix facilities, especially C structs and Unix stdio. Taken together, these two capabilities should allow researchers to lever- age their existing programming expertise while minimizing the need to learn new and complex coding rules. Choose from the following topics: o Funtools User Programs o funcalc: Funtools calculator (for binary tables) [funcalc(1)] o funcen: find centroid (for binary tables) [funcen(1)] o funcnts: count photons in specified regions [funcnts(1)] o funcone: cone search on RA, Dec columns [funcone(1)] o fundisp: display data in a Funtools data file [fundisp(1)] o funhead: display a header in a Funtools file [funhead(1)] o funhist: create a 1D histogram of a column [funhist(1)] o funimage: create a FITS image from a Funtools data file [funimage(1)] o funindex: create an index on a column in a binary table [funindex(1)] o funjoin: join two or more FITS binary tables on specified columns [funjoin(1)] o funmerge: merge one or more Funtools table files [funmerge(1)] o funsky: convert between image and sky coordinates, using WCS info from a FITS header [funsky(1)] o funtable: copy selected rows from a Funtools file to a FITS binary table [funtable(1)] o funtbl: extract a table from Funtools ASCII output [funtbl(1)] o funtools and ds9 image display [funds9(7)] o Funtools Programming o Funtools Programming Summary [funlib(3)] o Funtools Programming Tutorial [funlib(3)] o A Short Digression on Subroutine Order [funlib(3)] o Compiling and Linking [funlib(3)] o The Funtools Reference Handle [funlib(3)] o The Funtools Programming Reference Manual o FunOpen: open a Funtools file [funopen(3)] o FunImageGet: retrieve image data [funimageget(3)] o FunImagePut: output image data [funimageput(3)] o FunImageRowGet: retrieve image data by row [funimagerowget(3)] o FunImageRowPut: output image data by row [funimagerowput(3)] o FunTableRowGet: retrieve rows from a table [funtablerowget(3)] o FunTableRowPut: output rows to a table [funtablerowput(3)] o FunColumnSelect: select columns in a table for access [funcolumnselect(3)] o FunColumnActivate: activate columns in a table for read/write [funcolumnactivate(3)] o FunColumnLookup: lookup info about the columns in a table [funcolumnlookup(3)] o FunInfoGet: get info about an image or table [funinfoget(3)] o FunInfoPut: put info about an image or table [funinfoput(3)] o FunParamGet: get header param [funparamget(3)] o FunParamPut: put header param [funparamput(3)] o FunFlush: flush I/O in a Funtools file [funflush(3)] o FunClose: close a Funtools file [funclose(3)] o Funtools Programming Examples [funlib(3)] o evmerge: merge new columns with existing columns o evcols: add column and rows to binary tables o imblank: blank out image values below a threshold o Funtools Data Files [funfiles(7)] o Supported Data Formats o FITS File and Extensions o Non-FITS Raw Event Files o Non-FITS Array Files o Column-based Text (ASCII) Files o Database Views of Tables o Image Sections and Blocking o Binning FITS Binary Tables and Non-FITS Event Files o Disk Files and Other Supported File Types o Funtools Data Filtering o Table Filtering [funfilters(7)] o Fast Table Filtering using Indexes [funidx(7)] o Spatial Region Filtering [funregions(7)] o Region Geometry [reggeometry(7)] o Region Algebra [regalgebra(7)] o Region Coordinates [regcoords(7)] o Region Boundaries [regbounds(7)] o Differences Between Funtools and IRAF Regions [regdiff(7)] o Combining Table and Region Filters [funcombine(7)] o Miscellaneous o Funtools Environment Variables [funenv(7)] o Funtools ChangeLog version 1.4.2 January 2, 2008 funtools(7)

Featured Tech Videos