10-17-2010
Sort a big data file
Hello,
I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field.
I'm trying to sort with "sort" command and it brings me 6 minutes.
I have tried with some transformation methods in perl but it results "Out of memory". I was wondering to find any way (perl or unix shell script) to perform the fastest sort method of a big data file?.
Thanks,
bye.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
1 . Thanks everyone who read the post first.
2 . I have a log file which size is 143M , I can not use vi open it .I can not use xedit open it too.
How to view it ?
If I want to view 200-300 ,how can I implement it
3 . Thanks (3 Replies)
Discussion started by: chenhao_no1
3 Replies
2. Shell Programming and Scripting
Morning guys. Another day another question. :rolleyes:
I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution:
for results in `grep "^\
... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies
3. Shell Programming and Scripting
How to cut data from big file
my file around 30 gb
I tried "head -50022172 filename > newfile.txt ,and tail -5454283 newfile.txt. It's slowy.
afer that I tried sed -n '46467831,50022172p' filename > newfile.txt ,also slow
Please recommend me , faster command to cut some data from... (4 Replies)
Discussion started by: almanto
4 Replies
4. UNIX for Dummies Questions & Answers
I have a 5000 line config.log file with several "maybe" errors. Any reccomendations on finding solvable problems? (2 Replies)
Discussion started by: NeedLotsofHelp
2 Replies
5. Shell Programming and Scripting
Hi,
I have 2 files
format of file 1 is:
a1
b2
a2
c2
d1
f3
format of file 2 is (tab delimited):
a1 1.2 0.5 0.06 0.7 0.9 1 0.023
a3 0.91 0.007 0.12 0.34 0.45 1 0.7
a2 1.05 2.3 0.25 1 0.9 0.3 0.091
b1 1 5.4 0.3 9.2 0.3 0.2 0.1
b2 3 5 7 0.9 1 9 0 1
b3 0.001 1 2.3 4.6 8.9 10 0 1 0... (10 Replies)
Discussion started by: Lucky Ali
10 Replies
6. Shell Programming and Scripting
Hi. I am not sure the title gives an optimal description of what I want to do. Also, I tried to post this in the "UNIX for Dummies Questions & Answers", but it seems no-one was able to help out.
I have several text files that contain data in many columns. All the files are organized the same... (14 Replies)
Discussion started by: JamesT
14 Replies
7. UNIX for Advanced & Expert Users
I have a text file and each field is separated by semicolon ( ; ). Field number 7 is internally separated by comma ( , ) and pipe ( | ) symbol. I want to sort file based on three different fields which are marked in BOLD.
Here first BOLD field will have numbers upto the length of 9 characters,... (6 Replies)
Discussion started by: jnrohit2k
6 Replies
8. Shell Programming and Scripting
Hello,
I'm trying to sort a large data file by the 3rd column so that all of the first words in the 3rd column that are in all uppercase appear before (or after) the non uppercase words. For example,
Data file:
xxx 12345 Rat in the house
xxx 12345 CAT in the hat
xxx 12345 Dog in the... (4 Replies)
Discussion started by: palex
4 Replies
9. Shell Programming and Scripting
I have to sort below output in text file in unix bash
20170308
DA,I,113
20170308
PM,I,123
20170308
DA,U,22
20170308
PM,U,123
20170309
DA,I,11
20170309
PM,I,23
20170309
DA,U,123
20170309
PM,U,233 (8 Replies)
Discussion started by: Adfire
8 Replies
10. Shell Programming and Scripting
I have a string of pre defined ip address list which will always remain constant their order will never change like in below sample:
iplist=8.8.5.19,9.7.5.14,12.9.9.23,8.8.8.14,144.1.113
In the above example i m considering only 5 ips but there could be many more.
Now i have a file which... (15 Replies)
Discussion started by: mohtashims
15 Replies
LEARN ABOUT MOJAVE
net::ldap::control::sort5.18
Net::LDAP::Control::Sort(3) User Contributed Perl Documentation Net::LDAP::Control::Sort(3)
NAME
Net::LDAP::Control::Sort - Server Side Sort (SSS) control object
SYNOPSIS
use Net::LDAP::Control::Sort;
use Net::LDAP::Constant qw(LDAP_CONTROL_SORTRESULT);
$sort = Net::LDAP::Control::Sort->new(
order => "cn -phone"
);
$mesg = $ldap->search( @args, control => [ $sort ]);
($resp) = $mesg->control( LDAP_CONTROL_SORTRESULT );
print "Results are sorted
" if $resp and !$resp->result;
DESCRIPTION
"Net::LDAP::Control::Sort" is a sub-class of Net::LDAP::Control. It provides a class for manipulating the LDAP Server Side Sort (SSS)
request control 1.2.840.113556.1.4.473 as defined in RFC-2891
If the server supports sorting, then the response from a search operation will include a sort result control. This control is handled by
Net::LDAP::Control::SortResult.
CONSTRUCTOR ARGUMENTS
order
A string which defines how entries may be sorted. It consists of multiple directives, separated by whitespace. Each directive describes
how to sort entries using a single attribute. If two entries have identical attributes, then the next directive in the list is used.
Each directive specifies a sorting order as follows
-attributeType:orderingRule
The leading "-" is optional, and if present indicates that the sorting order should be reversed. "attributeType" is the attribute name
to sort by. "orderingRule" is optional and indicates the rule to use for the sort and should be valid for the given "attributeType".
Any one attributeType should only appear once in the sorting list.
Examples
"cn" sort by cn using the default ordering rule for the cn attribute
"-cn" sort by cn using the reverse of the default ordering rule
"age cn" sort by age first, then by cn using the default ordering rules
"cn:1.2.3.4" sort by cn using the ordering rule defined as 1.2.3.4
METHODS
As with Net::LDAP::Control each constructor argument described above is also available as a method on the object which will return the
current value for the attribute if called without an argument, and set a new value for the attribute if called with an argument.
SEE ALSO
Net::LDAP, Net::LDAP::Control::SortResult, Net::LDAP::Control, http://www.ietf.org/rfc/rfc2891.txt
AUTHOR
Graham Barr <gbarr@pobox.com>
Please report any bugs, or post any suggestions, to the perl-ldap mailing list <perl-ldap@perl.org>
COPYRIGHT
Copyright (c) 1999-2004 Graham Barr. All rights reserved. This program is free software; you can redistribute it and/or modify it under the
same terms as Perl itself.
perl v5.18.2 2013-07-21 Net::LDAP::Control::Sort(3)