02-04-2009
Sort large file
I was wondering how sort works.
Does file size and time to sort increase geometrically?
I have a 5.3 billion line file I'd like to use with sort -u I'm wondering if that'll take forever because of a geometric expansion?
If it takes 100 hours that's fine but not 100 days.
Thanks so much.
10 More Discussions You Might Find Interesting
1. Filesystems, Disks and Memory
Hi, Im trying to take a database backup. one of the files is 26 GB. I am using cp -pr to create a backup copy of the database. after the copying is complete, if i do du -hrs on the folders i saw a difference of 2GB.
The weird fact is that the BACKUP folder was 2 GB more than the original one!
... (1 Reply)
Discussion started by: 0ktalmagik
1 Replies
2. Shell Programming and Scripting
I have one large file, after every 200 line i have to split the file and the add header and footer to each small file?
It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies
3. Shell Programming and Scripting
Hello Gurus,
We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .
Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies
4. Shell Programming and Scripting
Hi-
I am trying to search a large file with a number of different search terms that are listed one per line in 3 different files. Most importantly I need to be able to do a case insensitive search.
I have tried just using egrep -f but it doesn't seam to be able to handle the -i option when... (3 Replies)
Discussion started by: dougzilla
3 Replies
5. UNIX for Advanced & Expert Users
Hello all -
I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies
6. Shell Programming and Scripting
Hello,
I have a very large file of around 2 million records which has the following structure:
I have used the standard awk program to sort:
# wordfreq.awk --- print list of word frequencies
{
# remove punctuation
#gsub(/_]/, "", $0)
for (i = 1; i <= NF; i++)
freq++
}
END {
for (word... (3 Replies)
Discussion started by: gimley
3 Replies
7. Shell Programming and Scripting
Hi Experts,
I have a filelist collected from another server , now want to sort the output using date/time stamp filed.
- Filed 6, 7,8 are showing the date/time/stamp.
Here is the input:
#----------------------------------------------------------------------
-rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies
8. UNIX for Advanced & Expert Users
I'm doing a hobby project that has me sorting huge files with sort of monotonous keys. It's very slow -- the current file is about 300 GB and has been sorting for a day. I know that sort has this --batch-size and --buffer-size parameters, but I'd like a jump start if possible to limit the... (42 Replies)
Discussion started by: kogorman3
42 Replies
9. Linux
Hi,
Anyone can help, I have a large textfile (one file), and I need to split into multiple file to break each file into ^L.
My textfile
==========
abc company
abc address
abc contact
^L
my company
my address
my contact
my skills
^L
your company
your address
========== (3 Replies)
Discussion started by: fspalero
3 Replies
10. UNIX for Beginners Questions & Answers
Dear Users,
Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file
input file.txt
scaffold1 928 929 C/T +
scaffold1 942 943 G/C +
scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
LEARN ABOUT DEBIAN
cz::sort
Cz::Sort(3pm) User Contributed Perl Documentation Cz::Sort(3pm)
NAME
Cz::Sort - Czech sort
SYNOPSIS
use Cz::Sort;
my $result = czcmp("_x j&a", "_&p");
my @sorted = czsort qw(plachta plaoka Planieka planieka plani);
print "@sorted
";
DESCRIPTION
Implements czech sorting conventions, indepentent on current locales in effect, which are often bad. Does the four-pass sort. The idea and
the base of the conversion table comes from Petr Olsak's program csr and the code is as compliant with CSN 97 6030 as possible.
The basic function provided by this module, is czcmp. If compares two scalars and returns the (-1, 0, 1) result. The function can be called
directly, like
my $result = czcmp("_x j&a", "_&p");
But for convenience and also because of compatibility with older versions, there is a function czsort. It works on list of strings and
returns that list, hmm, sorted. The function is defined simply like
sub czsort
{ sort { czcmp($a, $b); } @_; }
standard use of user's function in sort. Hashes would be simply sorted
@sorted = sort { czcmp($hash{$a}, $hash{$b}) }
keys %hash;
Both czcmp and czsort are exported into caller's namespace by default, as well as cscmp and cssort that are just aliases.
This module comes with encoding table prepared for ISO-8859-2 (Latin-2) encoding. If your data come in different one, you might want to
check the module Cstocs which can be used for reencoding of the list's data prior to calling czsort, or reencode this module to fit your
needs.
VERSION
0.68
SEE ALSO
perl(1), Cz::Cstocs(3).
AUTHOR
(c) 1997--2000 Jan Pazdziora <adelton@fi.muni.cz>, http://www.fi.muni.cz/~adelton/
at Faculty of Informatics, Masaryk University, Brno
perl v5.10.1 2000-05-16 Cz::Sort(3pm)