Sponsored Content
Full Discussion: Sort large file
Top Forums UNIX for Dummies Questions & Answers Sort large file Post 302284033 by jim mcnamara on Wednesday 4th of February 2009 02:40:29 PM
Old 02-04-2009
If the file is somewhat redundant and the key is not too huge try awk. Let's say the first two words of the record make the key:
Code:
awk '!arr[$1 $2]++' hugefile > uniquefile

This reads thru the file once, no sorting. Obviously there has to be enough disk space for the outputfile. It uses associative arrays - hash tables is another name.
 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

Strange difference in file size when copying LARGE file..

Hi, Im trying to take a database backup. one of the files is 26 GB. I am using cp -pr to create a backup copy of the database. after the copying is complete, if i do du -hrs on the folders i saw a difference of 2GB. The weird fact is that the BACKUP folder was 2 GB more than the original one! ... (1 Reply)
Discussion started by: 0ktalmagik
1 Replies

2. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

3. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

4. Shell Programming and Scripting

Script to search a large file with a list of terms in another file

Hi- I am trying to search a large file with a number of different search terms that are listed one per line in 3 different files. Most importantly I need to be able to do a case insensitive search. I have tried just using egrep -f but it doesn't seam to be able to handle the -i option when... (3 Replies)
Discussion started by: dougzilla
3 Replies

5. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

6. Shell Programming and Scripting

Script to sort large file with frequency

Hello, I have a very large file of around 2 million records which has the following structure: I have used the standard awk program to sort: # wordfreq.awk --- print list of word frequencies { # remove punctuation #gsub(/_]/, "", $0) for (i = 1; i <= NF; i++) freq++ } END { for (word... (3 Replies)
Discussion started by: gimley
3 Replies

7. Shell Programming and Scripting

Sort help: How to sort collected 'file list' by date stamp :

Hi Experts, I have a filelist collected from another server , now want to sort the output using date/time stamp filed. - Filed 6, 7,8 are showing the date/time/stamp. Here is the input: #---------------------------------------------------------------------- -rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies

8. UNIX for Advanced & Expert Users

Help optimizing sort of large files

I'm doing a hobby project that has me sorting huge files with sort of monotonous keys. It's very slow -- the current file is about 300 GB and has been sorting for a day. I know that sort has this --batch-size and --buffer-size parameters, but I'd like a jump start if possible to limit the... (42 Replies)
Discussion started by: kogorman3
42 Replies

9. Linux

Split a large textfile (one file) into multiple file to base on ^L

Hi, Anyone can help, I have a large textfile (one file), and I need to split into multiple file to break each file into ^L. My textfile ========== abc company abc address abc contact ^L my company my address my contact my skills ^L your company your address ========== (3 Replies)
Discussion started by: fspalero
3 Replies

10. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
Net::LDAP::Control::Sort(3)				User Contributed Perl Documentation			       Net::LDAP::Control::Sort(3)

NAME
Net::LDAP::Control::Sort - Server Side Sort (SSS) control object SYNOPSIS
use Net::LDAP::Control::Sort; use Net::LDAP::Constant qw(LDAP_CONTROL_SORTRESULT); $sort = Net::LDAP::Control::Sort->new( order => "cn -phone" ); $mesg = $ldap->search( @args, control => [ $sort ]); ($resp) = $mesg->control( LDAP_CONTROL_SORTRESULT ); print "Results are sorted " if $resp and !$resp->result; DESCRIPTION
"Net::LDAP::Control::Sort" is a sub-class of Net::LDAP::Control. It provides a class for manipulating the LDAP Server Side Sort (SSS) request control 1.2.840.113556.1.4.473 as defined in RFC-2891 If the server supports sorting, then the response from a search operation will include a sort result control. This control is handled by Net::LDAP::Control::SortResult. CONSTRUCTOR ARGUMENTS
order A string which defines how entries may be sorted. It consists of multiple directives, spearated by whitespace. Each directive describes how to sort entries using a single attribute. If two entries have identical attributes, then the next directive in the list is used. Each directive specifies a sorting order as follows -attributeType:orderingRule The leading "-" is optional, and if present indicates that the sorting order should be reversed. "attributeType" is the attribute name to sort by. "orderingRule" is optional and indicates the rule to use for the sort and should be valid for the given "attributeType". Any one attributeType should only appear once in the sorting list. Examples "cn" sort by cn using the default ordering rule for the cn attribute "-cn" sort by cn using the reverse of the default ordering rule "age cn" sort by age first, then by cn using the default ordering rules "cn:1.2.3.4" sort by cn using the ordering rule defined as 1.2.3.4 METHODS
As with Net::LDAP::Control each constructor argument described above is also available as a method on the object which will return the current value for the attribute if called without an argument, and set a new value for the attribute if called with an argument. SEE ALSO
Net::LDAP, Net::LDAP::Control::SortResult, Net::LDAP::Control, http://www.ietf.org/rfc/rfc2891.txt AUTHOR
Graham Barr <gbarr@pobox.com> Please report any bugs, or post any suggestions, to the perl-ldap mailing list <perl-ldap@perl.org> COPYRIGHT
Copyright (c) 1999-2004 Graham Barr. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.16.3 2013-06-07 Net::LDAP::Control::Sort(3)
All times are GMT -4. The time now is 05:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy