sort split merge -u unique Post: 302551977

Sponsored Content

Top Forums Shell Programming and Scripting sort split merge -u unique Post 302551977 by drl on Thursday 1st of September 2011 09:24:23 AM

09-01-2011

Registered User

Hi.

Most file sorting programs will follow the design noted in External sorting - Wikipedia, the free encyclopedia -- i.e. they will create smaller temporary sorted files, then do as many merges as necessary to accomplish a final sort. If you think about it, this is very reasonable: you sort a memory-load of data (as an array, using one of any number of algorithms), write it out to a temporary file until all input files are read, then you read chunks from each sorted temporary to produce a final file. If there too many temporary files, you might need to do an intermediate merge, etc.

I just glanced though a few thousand lines of GNU sort ( part of coreutils, coreutils-8.5/src/sort.c ) and it seems to do the sorting into temporary files.

I think the most you could do to help a system sort is to provide one or more directories for the temporary files on devices different from the input file:

Code:

       -T, --temporary-directory=DIR
              use DIR for temporaries, not $TMPDIR or /tmp; multiple options
              specify multiple directories

-- excerpt from Linux man sort

You could experiment with your own splitting by splitting a sample of your data, using utility split, and then sorting those files. I'd guess that would be slower than sort's internal splitting, but it might be worth a try, at least it would be educational.

You could also use a trace facility to look at the sort while it is going on. That would probably show scratch files being opened and child processes being created to help with the overall process: strace - trace system calls and signals, for example.

Best wishes ... cheers, drl

drl

View Public Profile for drl

Find all posts by drl

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort and Unique in Perl

Hi, May I know, if a pipe separated File is large, what is the best method to calculate the unique row count of 3rd column and get a list of unique value of the 3rdcolum? Thanks in advance!

2. UNIX for Dummies Questions & Answers

split a file with unique sets

This may sound like a trivial problem, but I still need some help: I have a file with ids and I want to split it 'n' ways (could be any number) into files: 1 1 1 2 2 3 3 4 5 5 Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may...

3. Shell Programming and Scripting

Awk sort and unique

4. Shell Programming and Scripting

How to merge columns into lines, using unique keys?

I would really appreciate a sulution for this : invoice# client# 5929 231 4358 231 2185 231 6234 231 1166 464 1264 464 3432 464 1720 464 9747 464 1133 791 4930 791 5496 791 6291 791 8681 989 3023 989

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,...

6. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks.

7. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber...

8. Shell Programming and Scripting

Sort unique

Hi, I have an input file that I have sorted in a previous stage by $1 and $4. I now need something that will take the first record from each group of data based on the key being $1 Input file 1000AAA|"ZZZ"|"Date"|"1"|"Y"|"ABC"|""|AA 1000AAA|"ZZZ"|"Date"|"2"|"Y"|"ABC"|""|AA...

9. UNIX for Beginners Questions & Answers

Merge 4 bim files by keeping only the overlapping variants (unique rs values )

Dear community, I am facing a problem and I kindly ask your help: I have 4 different data sets consisted from 3 different types of array. On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets: x2014: 1 rs3094315...

LEARN ABOUT REDHAT

sort

SORT(1) 							   User Commands							   SORT(1)

NAME

       sort - sort lines of text files

SYNOPSIS

       sort [OPTION]... [FILE]...

DESCRIPTION

       Write sorted concatenation of all FILE(s) to standard output.

       Ordering options:

       Mandatory arguments to long options are mandatory for short options too.

       -b, --ignore-leading-blanks ignore leading blanks

       -d, --dictionary-order
	      consider only blanks and alphanumeric characters

       -f, --ignore-case
	      fold lower case to upper case characters

       -g, --general-numeric-sort
	      compare according to general numerical value

       -i, --ignore-nonprinting
	      consider only printable characters

       -M, --month-sort
	      compare (unknown) < `JAN' < ... < `DEC'

       -n, --numeric-sort
	      compare according to string numerical value

       -r, --reverse
	      reverse the result of comparisons

       Other options:

       -c, --check
	      check whether input is sorted; do not sort

       -k, --key=POS1[,POS2]
	      start a key at POS1, end it at POS 2 (origin 1)

       -m, --merge
	      merge already sorted files; do not sort

       -o, --output=FILE
	      write result to FILE instead of standard output

       -s, --stable
	      stabilize sort by disabling last-resort comparison

       -S, --buffer-size=SIZE
	      use SIZE for main memory buffer

       -t, --field-separator=SEP use SEP instead of non- to whitespace transition

       -T, --temporary-directory=DIR
	      use DIR for temporaries, not $TMPDIR or /tmp multiple options specify multiple directories

       -u, --unique
	      with -c: check for strict ordering
	      otherwise: output only the first of an equal run

       -z, --zero-terminated
	      end lines with 0 byte, not newline

       --help display this help and exit

       --version
	      output version information and exit

       POS  is	F[.C][OPTS],  where  F	is the field number and C the character position in the field.	OPTS is one or more single-letter ordering
       options, which override global ordering options for that key.  If no key is given, use the entire line as the key.

       SIZE may be followed by the following multiplicative suffixes: % 1% of memory, b 1, K 1024 (default), and so on for M, G, T, P, E, Z, Y.

       With no FILE, or when FILE is -, read standard input.

       *** WARNING *** The locale specified by the environment affects sort order.  Set LC_ALL=C to get  the  traditional  sort  order	that  uses
       native byte values.

AUTHOR

       Written by Mike Haertel and Paul Eggert.

REPORTING BUGS

       Report bugs to <bug-coreutils@gnu.org>.

COPYRIGHT

       Copyright (C) 2002 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
       LAR PURPOSE.

SEE ALSO

       The full documentation for sort is maintained as a Texinfo manual.  If the info and sort programs are properly installed at your site,  the
       command

	      info sort

       should give you access to the complete manual.

sort (coreutils) 4.5.3						   October 2002 							   SORT(1)