How to sort in memory


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to sort in memory
# 8  
Old 11-10-2008
sort responds the the TMPDIR environment variable. As long as you have room to write the new file on disk, and room for a temp file, you should be able to sort it. use TMPDIR to put temp storage elsewhere

And, no you may not have enough memory for an in-memory sort. What does ulimit show?

Code:
# /mytmp is a direcotry on a filesystem with enough free space 184GB
# /dest is a directory on a disk with enough free space.  184G
export TMPDIR=/mytmp
sort -k 1.1,1.20 -o /dest/INITIATE_FINAL_ALL_810804405.dat \
     INITIATE_FINAL_ALL_810804405.dat

Did you try sort -y [some big number]


This also shows you why creating super large files is a poor design choice. Thre is almost always another way.....

I created a dummy keydef for the below example:

Your other option is a file-based radix-sort. joeyg posted something like this earlier.
You still need just as much disk space as above. Use the starting letter of your key to
write, say 26 files (A.txt - Z.txt). Then sort each one. use cat to combine them back into a big file
assume your key starts in position 1 in the file.
Code:
awk '{ file=substr($0,1,1); print $0 > file".txt"} INITIATE_FINAL_ALL_810804405.dat


Last edited by jim mcnamara; 11-10-2008 at 04:50 PM..
# 9  
Old 11-10-2008
Hammer & Screwdriver Here is an approach in perl

Below is my input file, the output from my script, and the actual script.

Perhaps this might get you to a way to sort.

Code:
> cat rawfile
Joe_________12_Main_St__A001________LX
Benny_______5_Spring____A002________LX
Will________2_A_St______A003________LX
Carl________15_X_Way____A004________LX
Henry_______85_Alpo_____B007________MX
Cal_________923_V_St____A006________RX
Holly_______62_US_______A005________TX
Will________9_Strong____A007________SX
Kate________101_Smile___C002________SX

> cat rawsorted 
Joe_________12_Main_St__A001________LX
Benny_______5_Spring____A002________LX
Kate________101_Smile___C002________SX
Will________2_A_St______A003________LX
Carl________15_X_Way____A004________LX
Holly_______62_US_______A005________TX
Cal_________923_V_St____A006________RX
Henry_______85_Alpo_____B007________MX
Will________9_Strong____A007________SX

> cat psff
#! /usr/bin/perl
# start of perl
# use strict;

# set vars
 $source = "rawfile";
 $outfile = ">rawsorted";

# open the source file
open(SOURCE,$source);
@rawdata=<SOURCE>;
# open the output file
open(OUTFILE,$outfile);

# sort at position 25 (note the 24) for length of 10
# sort in reverse order, hence the b before a in sort{ cmp } statement
# sort in forward order, hence the a before b in sort{ cmp } statement
@sorted = map{$_->[0]}
               sort {$a->[1] cmp $b->[1]}
               map {[$_,substr($_,24,10)]} @rawdata;

# write the sorted file to disk
print OUTFILE @sorted;

close SOURCE;
close OUTFILE;

# 10  
Old 11-10-2008
Hi.

The sort utility is also a merge utility. So you can carve-up input files into arbitrary-sized pieces, say with split or in any fashion you like, sort the resultant files onto themselves (with "-o" option), and then merge the bunch into a single output file.

Also, I agree with Jim: super-sized files are generally not a good choice ... cheers, drl
# 11  
Old 11-10-2008
As all records have the same size..
Can't you simply memory map the whole file (mmap()) and then do qsort() to sort the whole file.
If I remember will, mmap will prevent the whole file being loaded into RAM.

If you run into problems with not enough virtual memory :
If you have a root access:
Modify the memory limits of the process with setrlimit (RLIMIT_AS, RLIMIT_DATA)
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort help: How to sort collected 'file list' by date stamp :

Hi Experts, I have a filelist collected from another server , now want to sort the output using date/time stamp filed. - Filed 6, 7,8 are showing the date/time/stamp. Here is the input: #---------------------------------------------------------------------- -rw------- 1 root ... (3 Replies)
Discussion started by: rveri
3 Replies

2. Shell Programming and Scripting

Help with sort word and general numeric sort at the same time

Input file: 100%ABC2 3.44E-12 USA A2M%H02579 0E0 UK 100%ABC2 5.34E-8 UK 100%ABC2 3.25E-12 USA A2M%H02579 5E-45 UK Output file: 100%ABC2 3.44E-12 USA 100%ABC2 3.25E-12 USA 100%ABC2 5.34E-8 UK A2M%H02579 0E0 UK A2M%H02579 5E-45 UK Code try: sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Discussion started by: perl_beginner
2 Replies

3. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

4. UNIX for Advanced & Expert Users

Script to sort the files and append the extension .sort to the sorted version of the file

Hello all - I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
Discussion started by: pankaj80
3 Replies

5. Shell Programming and Scripting

Is this "Out of Memory!" error due to sort() of perl how to resolve it?

I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program... (3 Replies)
Discussion started by: lilili07
3 Replies

6. Shell Programming and Scripting

How to Sort Floating Numbers Using the Sort Command?

Hi to all. I'm trying to sort this with the Unix command sort. user1:12345678:3.5:2.5:8:1:2:3 user2:12345679:4.5:3.5:8:1:3:2 user3:12345687:5.5:2.5:6:1:3:2 user4:12345670:5.5:2.5:5:3:2:1 user5:12345671:2.5:5.5:7:2:3:1 I need to get this: user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Discussion started by: daniel.gbaena
7 Replies

7. Solaris

restrcit physical memory with zone.max-locked-memory

Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Discussion started by: fugitive
1 Replies

8. Programming

How to deal with lots of data in memory in order not to run out of memory

Hi, I'm trying to learn how to manage memory when I have to deal with lots of data. Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at once (yep!) but of course the... (1 Reply)
Discussion started by: emitrax
1 Replies

9. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

10. AIX

ulimits max locked memory virtual memory

Hi, Would any one be so kind to explain me : are ulimits defined for each user seperately ? When ? Specialy what is the impact of : max locked memory and virtual memory on performance of applications for a user. Many thanks. PS : this is what I can see in MAN : ulimit ] ... (5 Replies)
Discussion started by: big123456
5 Replies
Login or Register to Ask a Question