sort responds the the TMPDIR environment variable. As long as you have room to write the new file on disk, and room for a temp file, you should be able to sort it. use TMPDIR to put temp storage elsewhere
And, no you may not have enough memory for an in-memory sort. What does ulimit show?
Did you try sort -y [some big number]
This also shows you why creating super large files is a poor design choice. Thre is almost always another way.....
I created a dummy keydef for the below example:
Your other option is a file-based radix-sort. joeyg posted something like this earlier.
You still need just as much disk space as above. Use the starting letter of your key to
write, say 26 files (A.txt - Z.txt). Then sort each one. use cat to combine them back into a big file
assume your key starts in position 1 in the file.
Last edited by jim mcnamara; 11-10-2008 at 04:50 PM..
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,288
Thanks Given: 430
Thanked 480 Times in 395 Posts
Hi.
The sort utility is also a merge utility. So you can carve-up input files into arbitrary-sized pieces, say with split or in any fashion you like, sort the resultant files onto themselves (with "-o" option), and then merge the bunch into a single output file.
Also, I agree with Jim: super-sized files are generally not a good choice ... cheers, drl
As all records have the same size..
Can't you simply memory map the whole file (mmap()) and then do qsort() to sort the whole file.
If I remember will, mmap will prevent the whole file being loaded into RAM.
If you run into problems with not enough virtual memory :
If you have a root access:
Modify the memory limits of the process with setrlimit (RLIMIT_AS, RLIMIT_DATA)
Hi Experts,
I have a filelist collected from another server , now want to sort the output using date/time stamp filed.
- Filed 6, 7,8 are showing the date/time/stamp.
Here is the input:
#----------------------------------------------------------------------
-rw------- 1 root ... (3 Replies)
Input file:
100%ABC2 3.44E-12 USA
A2M%H02579 0E0 UK
100%ABC2 5.34E-8 UK
100%ABC2 3.25E-12 USA
A2M%H02579 5E-45 UK
Output file:
100%ABC2 3.44E-12 USA
100%ABC2 3.25E-12 USA
100%ABC2 5.34E-8 UK
A2M%H02579 0E0 UK
A2M%H02579 5E-45 UK
Code try:
sort -k1,1 -g -k2 -r input.txt... (2 Replies)
Hi Experts,
Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing.
Why memory utilization on solaris always looks high?
I have statement about memory on solaris, is this true:
Memory in solaris is used for... (4 Replies)
Hello all -
I am to this forum and fairly new in learning unix and finding some difficulty in preparing a small shell script. I am trying to make script to sort all the files given by user as input (either the exact full name of the file or say the files matching the criteria like all files... (3 Replies)
I am running a program written in perl script and it is stopped with "Out of memory!" error. This is very strange because at the time then the program is aborted, it only used 4GB RAM and there are still 30GB free physical memory left in the system. I check the perl script and found the program... (3 Replies)
Hi to all.
I'm trying to sort this with the Unix command sort.
user1:12345678:3.5:2.5:8:1:2:3
user2:12345679:4.5:3.5:8:1:3:2
user3:12345687:5.5:2.5:6:1:3:2
user4:12345670:5.5:2.5:5:3:2:1
user5:12345671:2.5:5.5:7:2:3:1
I need to get this:
user3:12345687:5.5:2.5:6:1:3:2... (7 Replies)
Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Hi,
I'm trying to learn how to manage memory when I have to deal with lots of data.
Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that
holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at
once (yep!) but of course the... (1 Reply)
Hi,
Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:-
1. Total Physical Memory in the system(total RAM).
2. Available Physical Memory(i.e. RAM Usage)
3. Total (Logical) Memory in the system
4. Available (Logical) Memory.
I know... (4 Replies)
Hi,
Would any one be so kind to explain me :
are ulimits defined for each user seperately ? When ?
Specialy what is the impact of :
max locked memory
and
virtual memory
on performance of applications for a user.
Many thanks.
PS :
this is what I can see in MAN :
ulimit ]
... (5 Replies)