Sponsored Content
Top Forums Shell Programming and Scripting fastest way to remove duplicates. Post 76073 by pixelbeat on Friday 24th of June 2005 10:07:42 AM
Old 06-24-2005
It's already in a database!
Just do add a sort by in the select clause and
index the appropriate fields.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to delete/remove directory in fastest way

hello i need help to remove directory . The directory is not empty ., it contains several sub directories and files inside that.. total number of files in one directory is 12,24,446 . rm -rf doesnt work . it is prompting for every file .. i want to delete without prompting and... (6 Replies)
Discussion started by: getdpg
6 Replies

2. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

3. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

4. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

5. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

6. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

7. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

8. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

9. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

10. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
DBD::XBase(3pm) 					User Contributed Perl Documentation					   DBD::XBase(3pm)

NAME
DBD::XBase - DBI driver for XBase compatible database files SYNOPSIS
use DBI; my $dbh = DBI->connect("DBI:XBase:/directory/subdir") or die $DBI::errstr; my $sth = $dbh->prepare("select MSG from test where ID != 1") or die $dbh->errstr(); $sth->execute() or die $sth->errstr(); my @data; while (@data = $sth->fetchrow_array()) { ## further processing } $dbh->do('update table set name = ? where id = 45', {}, 'krtek'); DESCRIPTION
DBI compliant driver for module XBase. Please refer to DBI(3) documentation for how to actually use the module. In the connect call, specify the directory containing the dbf files (and other, memo, etc.) as the third part of the connect string. It defaults to the current directory. Note that with dbf, there is no database server that the driver would talk to. This DBD::XBase calls methods from XBase.pm module to read and write the files on the disk directly, so any limitations and features of XBase.pm apply to DBD::XBase as well. DBD::XBase basically adds SQL, DBI compliant interface to XBase.pm. The DBD::XBase doesn't make use of index files at the moment. If you really need indexed access, check XBase(3) for notes about support for variour index types. SUPPORTED SQL COMMANDS
The SQL commands currently supported by DBD::XBase's prepare are: select select fields_or_expressions from table [ where condition ] [ order by field ] Fields_or_expressions is a comma separated list of fields or arithmetic expressions, or a "*" for all fields from the table. The "where" condition specifies which rows will be returned, you can have arbitrary arithmetic and boolean expression here, compare fields and constants and use "and" and "or". Match using "like" is also supported. Examples: select * from salaries where name = "Smith" select first,last from people where login = "ftp" or uid = 1324 select id,first_name,last_name from employ where last_name like 'Ki%' order by last_name select id + 1, substr(name, 1, 10) from employ where age > 65 select id, name from employ where id = ? You can use bind parameters in the where clause, as the last example shows. The actual value has to be supplied via bind_param or in the call to execute or do, see DBI(3) for details. To check for NULL values in the "where" expression, use "id is null" and "id is not null", not "id == null". Please note that you can only select from one table, joins are not supported and are not planned to be supported. If you need them, get a real RDBMS (or send me a patch). In the arithmetic expressions you can use a couple of SQL functions -- currently supported are concat, substr (and substring), trim, ltrim and rtrim, length. I do not have an exact idea of which and how many functions I want to support. It's easy to write them in a couple of minutes now the interface is there (check the XBase::SQL module if you want to send a patch containing support for more), it's just that I do not really need them and sometimes it's hard to tell what is useful and what is SQL92 compatible. Comment welcome. The select command may contain and order by clause. Only one column is supported for sorting at the moment, patches are welcome. The group by clause is not supported (and I do not plan them), nor are the aggregate functions. delete delete from table [ where condition ] The "where" condition is the same as for select. Examples: delete from jobs ## emties the table delete from jobs where companyid = "ISW" delete from jobs where id < ? insert insert into table [ ( fields ) ] values ( list of values ) Here fields is a (optional) comma separated list of fields to set, list of values is a list of constants to assign. If the fields are not specified, sets the fields in the natural order of the table. You can use bind parameters in the list of values. Examples: insert into accounts (login, uid) values ("guest", 65534) insert into accounts (login, uid) values (?, ?) insert into passwd values ("user","*",4523,100,"Nice user", "/home/user","/bin/bash") update update table set field = new value [ , set more fields ] [ where condition ] Example: update passwd set uid = 65534 where login = "guest" update zvirata set name = "Jezek", age = 4 where id = 17 Again, the value can also be specified as bind parameter. update zvirata set name = ?, age = ? where id = ? create table create table table name ( columns specification ) Columns specification is a comma separated list of column names and types. Example: create table rooms ( roomid int, cat char(10), balcony boolean ) The allowed types are char num numeric int integer float boolean blob memo date time datetime Some of them are synonyms. They are of course converted to appropriate XBase types. drop table drop table table name Example: drop table passwd ATTRIBUTES
Besides standard DBI attribudes, DBD::XBase supports database handle attribute xbase_ignorememo: $dbh->{'xbase_ignorememo'} = 1; Setting it to 1 will cause subsequent tables to be opened while ignoring the memo files (dbt, fpt). So you can read dbf files for which you don't have (you have lost them, for example) the memo files. The memo fields will come out as nulls. VERSION
1.03 AVAILABLE FROM
http://www.adelton.com/perl/DBD-XBase/ AUTHOR
(c) 1997--2011 Jan Pazdziora. Contact the author at jpx dash perl at adelton dot com. SEE ALSO
perl(1); DBI(3), XBase(3); dbish(1) Translation into Japanese (older version) at http://member.nifty.ne.jp/hippo2000/perltips/DBD/XBase.htm by Kawai Takanori. perl v5.12.4 2011-08-31 DBD::XBase(3pm)
All times are GMT -4. The time now is 06:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy