Sponsored Content
Full Discussion: sort takes a long time
Top Forums Shell Programming and Scripting sort takes a long time Post 302494711 by methyl on Tuesday 8th of February 2011 06:31:49 AM
Old 02-08-2011
What Operating System and version are you using?
How many records?
How long does the process take?
Do you have spare memory and disc space which you could give to this sort process.

If your sort has a "-o filename" parameter , use this to specify the output file not a Shell redirect (> filename). It will be much much faster.

If set, what is the value of $TMPDIR ? Can it be set to point to a fast filesystem with at least twice the space of the size of the unsorted file?

You can get a dramatic improvement in the performance of unix sort by tuning the "-y kmem" parameter. It is very important that you start with enough memory allocated to do some serious sorting on the first pass.

Off topic: If you have a database engine it is often quicker to load a large file into a database table with suitable keys, then write the file out in the required order.
 

10 More Discussions You Might Find Interesting

1. Programming

fwrite takes extremely long time

After my previous thread, I think I found out what causes the long delays. I run this program on several Linux computers, and the sometimes (after the file with the arrays becomes big) the fwrite takes between 100 ms to 900 ms. This is very bad for me, as I want a timer to halt each 30 ms.... ... (5 Replies)
Discussion started by: inna
5 Replies

2. Shell Programming and Scripting

Killing a process that takes too long

Hello, I have a C program that takes anywhere from 5 to 100 arguments and I'd like to run it from a script that makes sure it doesnt take too long to execute. If the C program takes more than 5 seconds to execute, i would like the shell script to kill it and return a short message to the user. ... (3 Replies)
Discussion started by: WeezelDs
3 Replies

3. Shell Programming and Scripting

shell script takes long time to complete

Hi all, I wrote this shell script to validate filed numbers for input file. But it take forever to complete validation on a file. The average speed is like 9mins/MB. Can anyone tell me how to improve the performance of a shell script? Thanks (12 Replies)
Discussion started by: ozzman
12 Replies

4. Linux

it takes long time to login on server

Hi, I am trying to login using ssh on Red Hat Linux 5 server, The password appears immediately but after I enter the password it takes about 90 seconds to login completely. Please suggest what changes require? Regards, Manoj (4 Replies)
Discussion started by: manoj.solaris
4 Replies

5. UNIX for Dummies Questions & Answers

time how long it takes to load a module

Hello, like the title says, how can i measure the time it takes to load a module in Linux, and how how can i measure the time it takes to load a statically compiled module. /Best Regards Olle ---------- Post updated at 01:13 PM ---------- Previous update was at 11:54 AM ---------- For... (0 Replies)
Discussion started by: ollebanan
0 Replies

6. UNIX for Dummies Questions & Answers

Changing Password process takes a long time

We are running unix. After a reboot of the server we have found that changing password takes a long time. if type in passwd "username" you can type in the 1st instance of the password , press enter , then it will wait for about 3 minutes before bringing up the confirm password line typing it in... (4 Replies)
Discussion started by: AIXlewis
4 Replies

7. UNIX and Linux Applications

database takes long time to process

Hi, we currently having a issue where when we send jobs to the server for the application lawson, it is taking a very long time to complete. here are the last few lines of the database log. 2012-09-18-10.35.55.707279-240 E244403536A576 LEVEL: Warning PID : 950492 ... (1 Reply)
Discussion started by: techy1
1 Replies

8. Shell Programming and Scripting

Wget takes a long time to complete

Hi, I wish to check the return value for wget $url. However, some urls are designed to take 45 minutes or more to return. All i need to check if the URL can be reached or not using wget. How can i get wget to return the value in a few seconds ? (8 Replies)
Discussion started by: mohtashims
8 Replies

9. Shell Programming and Scripting

Find command takes long

Hi, I am trying to search for a Directory called "mont" under a directory path "/opt/app/var/dumps" Although "mont" is in the very parent directory called "dumps" i.e "/opt/app/var/dumps/mont" and it can never be inside any Sub-Directory of "dumps"; my below find command which also checks... (5 Replies)
Discussion started by: mohtashims
5 Replies

10. Shell Programming and Scripting

Transfer file from a server takes long time

It takes 6 hrs for a 90 GB zip file that i am copying / transferring from serverA onto serverB. scp user1@serverA:/opt/setup/cash.zip . Output: cash.zip 21% 19GB 4.7MB/s 4:11:46 ETA uname -a SunOS serverB 5.11 11.2 sun4v sparc sun4vCan you please suggest if i could do... (11 Replies)
Discussion started by: mohtashims
11 Replies
SORT(1) 						    BSD General Commands Manual 						   SORT(1)

NAME
sort -- sort or merge text files SYNOPSIS
sort [-bcdfHilmnrSsu] [-k field1[,field2]] [-o output] [-R char] [-T dir] [-t char] [file ...] DESCRIPTION
The sort utility sorts text files by lines. Comparisons are based on one or more sort keys extracted from each line of input, and are per- formed lexicographically. By default, if keys are not given, sort regards each input line as a single field. The following options are available: -c Check that the single input file is sorted. If the file is not sorted, sort produces the appropriate error messages and exits with code 1; otherwise, sort returns 0. sort -c produces no output. -H Ignored for compatibility with earlier versions of sort. -m Merge only; the input files are assumed to be pre-sorted. -o output The argument given is the name of an output file to be used instead of the standard output. This file can be the same as one of the input files. -S Don't use stable sort. Default is to use stable sort. -s Use stable sort, keeps records with equal keys in their original order. This is the default. Provided for compatibility with other sort implementations only. -T dir Use dir as the directory for temporary files. The default is the value specified in the environment variable TMPDIR or /tmp if TMPDIR is not defined. -u Unique: suppress all but one in each set of lines having equal keys. If used with the -c option, check that there are no lines with duplicate keys. The following options override the default ordering rules. When ordering options appear independent of key field specifications, the requested field ordering rules are applied globally to all sort keys. When attached to a specific key (see -k), the ordering options over- ride all global ordering options for that key. -d Only blank space and alphanumeric characters are used in making comparisons. -f Considers all lowercase characters that have uppercase equivalents to be the same for purposes of comparison. -i Ignore all non-printable characters. -l Sort by the string length of the field, not by the field itself. -n An initial numeric string, consisting of optional blank space, optional minus sign, and zero or more digits (including decimal point) is sorted by arithmetic value. (The -n option no longer implies the -b option.) -r Reverse the sense of comparisons. The treatment of field separators can be altered using these options: -b Ignores leading blank space when determining the start and end of a restricted sort key. A -b option specified before the first -k option applies globally to all -k options. Otherwise, the -b option can be attached independently to each field argument of the -k option (see below). Note that the -b option has no effect unless key fields are specified. -t char char is used as the field separator character. The initial char is not considered to be part of a field when determining key offsets (see below). Each occurrence of char is significant (for example, ``charchar'' delimits an empty field). If -t is not specified, the default field separator is a sequence of blank-space characters, and consecutive blank spaces do not delimit an empty field; further, the initial blank space is considered part of a field when determining key offsets. -R char char is used as the record separator character. This should be used with discretion; -R <alphanumeric> usually produces undesir- able results. The default record separator is newline. -k field1[,field2] Designates the starting position, field1, and optional ending position, field2, of a key field. The -k option replaces the obso- lescent options +pos1 and -pos2. The following operands are available: file The pathname of a file to be sorted, merged, or checked. If no file operands are specified, or if a file operand is -, the standard input is used. A field is defined as a minimal sequence of characters followed by a field separator or a newline character. By default, the first blank space of a sequence of blank spaces acts as the field separator. All blank spaces in a sequence of blank spaces are considered as part of the next field; for example, all blank spaces at the beginning of a line are considered to be part of the first field. Fields are specified by the -k field1[,field2] argument. A missing field2 argument defaults to the end of a line. The arguments field1 and field2 have the form m.n and can be followed by one or more of the letters b, d, f, i, l, n, and r, which correspond to the options discussed above. A field1 position specified by m.n (m, n > 0) is interpreted as the nth character in the mth field. A miss- ing .n in field1 means '.1', indicating the first character of the mth field; if the -b option is in effect, n is counted from the first non- blank character in the mth field; m.1b refers to the first non-blank character in the mth field. A field2 position specified by m.n is interpreted as the nth character (including separators) of the mth field. A missing .n indicates the last character of the mth field; m = 0 designates the end of a line. Thus the option -k v.x,w.y is synonymous with the obsolescent option +v-1.x-1-w-1.y; when y is omitted, -k v.x,w is synonymous with +v-1.x-1-w+1.0. The obsolescent +pos1 -pos2 option is still supported, except for -w.0b, which has no -k equivalent. ENVIRONMENT
If the following environment variable exists, it is used by sort. TMPDIR sort uses the contents of the TMPDIR environment variable as the path in which to store temporary files. FILES
/tmp/sort.* Default temporary files. outputNUMBER Temporary file which is used for output if output already exists. Once sorting is finished, this file replaces output (via link(2) and unlink(2)). EXIT STATUS
Sort exits with one of the following values: 0 Normal behavior. 1 On disorder (or non-uniqueness) with the -c option 2 An error occurred. SEE ALSO
comm(1), join(1), uniq(1), qsort(3), radixsort(3) HISTORY
A sort command appeared in Version 5 AT&T UNIX. This sort implementation appeared in 4.4BSD and is used since NetBSD 1.6. BUGS
Posix requires the locale's thousands separator be ignored in numbers. It may be faster to sort very large files in pieces and then explic- itly merge them. NOTES
This sort has no limits on input line length (other than imposed by available memory) or any restrictions on bytes allowed within lines. To protect data sort -o calls link(2) and unlink(2), and thus fails on protected directories. Input files should be text files. If file doesn't end with record separator (which is typically newline), the sort utility silently supplies one. The current sort uses lexicographic radix sorting, which requires that sort keys be kept in memory (as opposed to previous versions which used quick and merge sorts and did not.) Thus performance depends highly on efficient choice of sort keys, and the -b option and the field2 argument of the -k option should be used whenever possible. Similarly, sort -k1f is equivalent to sort -f and may take twice as long. BSD
December 18, 2010 BSD
All times are GMT -4. The time now is 08:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy