Shell Programming and Scripting

I don't think we identified your O/S beyond it being a Linux. There is much variation.

It is possible that your "sort" command does not have a "-y" switch or other switch to pre-allocate memory. Have you checked "man sort" or perhaps "info sort"?

Maybe you are a non-root user and have a memory quota which is too low to do this large sort?

Perhaps you have a basic kernel and the sort is trying to open more files than is allowed?

Have you checked the directory where you expect to find the sort workfiles? Are they there? Is there enough disc space in that filesystem?

Is the running sort using CPU? I'm starting to wonder if your "sort" program is faulty.

Afterthought. We assume that this is a unix standard format text file with each line terminated with a line-feed character (only) and that it has not come from a Microsoft platform.

linux version:
Linux version 2.6.18-164.6.1.el5 (mockbuild@ls20-bc2-14.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46))

SORT:
sort (GNU coreutils) 5.97

it does not have -y but it has -S (buffersize) and it does not help neither!

also my sort program is fine! and it does use CPU!

Googling your version of "sort" and the symptoms uncovered a can of worms.

For example: If your locale is anything other than "C" the performance of sort can be atrocious. There are other variants on this theme including the program ignoring the buffer parameter.

What is the output from the "locale" command ?

Suggest you take up the issue with your software supplier in case a fixed version is available.

locale: LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

so you guess it's because of my version?

I think we have found the problem.

UTF-8 is cited as the worst possible character set to sort or grep because it can't be sorted as a simple binary key.
If your data is actually US ASCII then I'd try the sort with with the locale set to "C".

On my system (yours has more values):

Code:

LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=

If you must sort in UTF-8 order some recommend using the sort command in "perl". The "sort" command in "perl" is three times slower than unix "sort" for LANG=C , but ten times faster for LANG=en_US.UTF-8 .


Rant: The UTF-8 issue arises from the latest Posix standards. It has given me grief with mixed platform XML too.

Hi. If you don't want to create your own perl utility, you may be interested in:

Code:

msort - utility for sorting records in complex ways

...

       msort fully supports Unicode. The text to be sorted, and all
       specifications, should be in UTF-8 Unicode. (If you have plain ASCII
       text, this is not a problem as ASCII is a subset of Unicode.) Full
       Unicode case-folding is available, in Turkic and non-Turkic variants.
       Unicode normalization is performed before sorting.

-- excerpt from man msort, q.v

http://billposer.org/Software/msort.html

Good luck ... cheers, drl