sort split merge -u unique Post: 302551977

Sponsored Content

Top Forums Shell Programming and Scripting sort split merge -u unique Post 302551977 by drl on Thursday 1st of September 2011 09:24:23 AM

09-01-2011

Registered User

Hi.

Most file sorting programs will follow the design noted in External sorting - Wikipedia, the free encyclopedia -- i.e. they will create smaller temporary sorted files, then do as many merges as necessary to accomplish a final sort. If you think about it, this is very reasonable: you sort a memory-load of data (as an array, using one of any number of algorithms), write it out to a temporary file until all input files are read, then you read chunks from each sorted temporary to produce a final file. If there too many temporary files, you might need to do an intermediate merge, etc.

I just glanced though a few thousand lines of GNU sort ( part of coreutils, coreutils-8.5/src/sort.c ) and it seems to do the sorting into temporary files.

I think the most you could do to help a system sort is to provide one or more directories for the temporary files on devices different from the input file:

Code:

       -T, --temporary-directory=DIR
              use DIR for temporaries, not $TMPDIR or /tmp; multiple options
              specify multiple directories

-- excerpt from Linux man sort

You could experiment with your own splitting by splitting a sample of your data, using utility split, and then sorting those files. I'd guess that would be slower than sort's internal splitting, but it might be worth a try, at least it would be educational.

You could also use a trace facility to look at the sort while it is going on. That would probably show scratch files being opened and child processes being created to help with the overall process: strace - trace system calls and signals, for example.

Best wishes ... cheers, drl

drl

View Public Profile for drl

Find all posts by drl

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort and Unique in Perl

Hi, May I know, if a pipe separated File is large, what is the best method to calculate the unique row count of 3rd column and get a list of unique value of the 3rdcolum? Thanks in advance!

2. UNIX for Dummies Questions & Answers

split a file with unique sets

This may sound like a trivial problem, but I still need some help: I have a file with ids and I want to split it 'n' ways (could be any number) into files: 1 1 1 2 2 3 3 4 5 5 Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may...

3. Shell Programming and Scripting

Awk sort and unique

4. Shell Programming and Scripting

How to merge columns into lines, using unique keys?

I would really appreciate a sulution for this : invoice# client# 5929 231 4358 231 2185 231 6234 231 1166 464 1264 464 3432 464 1720 464 9747 464 1133 791 4930 791 5496 791 6291 791 8681 989 3023 989

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,...

6. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks.

7. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber...

8. Shell Programming and Scripting

Sort unique

Hi, I have an input file that I have sorted in a previous stage by $1 and $4. I now need something that will take the first record from each group of data based on the key being $1 Input file 1000AAA|"ZZZ"|"Date"|"1"|"Y"|"ABC"|""|AA 1000AAA|"ZZZ"|"Date"|"2"|"Y"|"ABC"|""|AA...

9. UNIX for Beginners Questions & Answers

Merge 4 bim files by keeping only the overlapping variants (unique rs values )

Dear community, I am facing a problem and I kindly ask your help: I have 4 different data sets consisted from 3 different types of array. On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets: x2014: 1 rs3094315...

LEARN ABOUT MINIX

wrap-and-sort

WRAP-AND-SORT(1)					      General Commands Manual						  WRAP-AND-SORT(1)

NAME

       wrap-and-sort - wrap long lines and sort items in Debian packaging files

SYNOPSIS

       wrap-and-sort [options]

DESCRIPTION

       wrap-and-sort  wraps the package lists in Debian control files. By default the lists will only split into multiple lines if the entries are
       longer than 80 characters. wrap-and-sort sorts the package lists in Debian control files and all .install files. Beside that  wrap-and-sort
       removes trailing spaces in these files.

       This  script should be run in the root of a Debian package tree. It searches for control, control.in, copyright, copyright.in, install, and
       *.install in the debian directory.

OPTIONS

       -h, --help
	      Show this help message and exit.

       -a, --wrap-always
	      Wrap all package lists in the Debian control file even if the entries are shorter than 80 characters and could fit in one line line.

       -s, --short-indent
	      Only indent wrapped lines by one space (default is in-line with the field name).

       -b, --sort-binary-packages
	      Sort binary package paragraphs by name.

       -k, --keep-first
	      When sorting binary package paragraphs, leave the first one at the top.  Unqualified debhelper(7) configuration files are applied to
	      the first package.

       -n, --no-cleanup
	      Do not remove trailing whitespaces.

       -d path, --debian-directory=path
	      Location of the debian directory (default: ./debian).

       -f file, --file=file
	      Wrap  and sort only the specified file.  You can specify this parameter multiple times.  All supported files will be processed if no
	      files are specified.

       -v, --verbose
	      Print all files that are touched.

AUTHORS

       wrap-and-sort and this manpage have been written by Benjamin Drung <bdrung@debian.org>.

       Both are released under the ISC license.

DEBIAN
								 Debian Utilities						  WRAP-AND-SORT(1)