File size limitation of unix sort command.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File size limitation of unix sort command.
# 1  
Old 10-05-2005
File size limitation of unix sort command.

hi ,

iam trying to sort millions of records which is delimited and i cant able to
use sort command more than 60 million..if i try to do so i got an message stating that "File size limit exceeded",Is there any file size limit for using sort command..
How can i solve this problem.

thanks

cskumar
# 2  
Old 10-05-2005
what version of unix/sort exactly?
gnu sort for example supports files > 2GB,
and efficiently sorts/merges files > RAM size
# 3  
Old 10-05-2005
Hi Pixelbeat,

Thanks for your quick response. We are trying this on HP UX 11. We are generating input file for sorting through c program and calling unix sort command through c only, as follows::

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

FILE *fout;
int main(int argc,char *argv[])
{
int loopcnt=0;
unsigned long int TTRECS = 60000000;

if((fout = fopen("SORTINPUT","w")) == NULL)
{
printf("\nUnable to open the output_file file\n");
exit(1);
}
for (loopcnt=TTRECS;loopcnt>=1;loopcnt--)
{
fprintf(fout,"%d|%s%d|%s|\n",loopcnt,"Dummy",loopcnt,"Dummy Description");
}
fflush(fout);
printf("Executing::sort -t '|' -k 1,1 SORTINPUT> sorttestoutput\n");

system("echo Sort Start Time;date");
system("sort -t '|' -k 1,1 SORTINPUT > sorttestoutput");
system("echo status returned::$?");

system("echo Sort End Time;date");
printf("Total Records Configured:: %lu\n", TTRECS);
return 0;
}


We are trying this test just to make sure that unix sort command is not having any limitations on file size. But what we observed is when file is packed with more than 60 million records [ around 2 Gb size], we are getting an error stating "File size limit exceeded" on console. Can u kindly state,

1] whether we got any maximum file size limit in HP UX while creating a file through C program.

2] Can unix sort command can handle file size > 2 GB without any failure.

3] Is there any restriction for length of each record [each line] or each field[fields are separated by a delimeter].

Thanks,
# 4  
Old 10-06-2005
Quote:
Originally Posted by cskumar
Hi Pixelbeat,


1] whether we got any maximum file size limit in HP UX while creating a file through C program.
I think it's quite possible that you have such a file size limitation.
If I were you I'd try to edit your C code to see if the error occurs before the sort
command is called or not. I'd say it's before it, because I think the error might
begin by "sort: " if the error was caused by sort.
# 5  
Old 10-06-2005
thanks for ur reply chlorine,

Yes as u said error is before sort only as we couldn't create a file of size more than 2 GB through c.
[1] Can u suggest a way to get rid of that size problem.

[2] Can u let us know whether we got any such file size limitation for sort command.
# 6  
Old 10-06-2005
I have no idea if it is feasible to change the filesize limit.
For sort, the man page should tell you exactly the limitations. If the man page doesn't mention any limitations then there should be none.
# 7  
Old 10-06-2005
Ah OK, you need to use interfaces
in your program that allow you to have
files > 2GB.

For linux (glibc) you can do it explicitly
in the source by defining _LARGEFILE64_SOURCE
and then calling open(,O_LARGEFILE), stat64() etc.

You can also get everything working implicitly
in 64 bit mode by just defining _FILE_OFFSET_BITS=64
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

File size limitation in the EST 2012 x86_64 GNU/Linux

Hello Friends, I tried to take tar backup in my server, but it ended with an error. It said that: /home/back/pallava_backup/fbackup_backup/stape_config /home/back/romam_new.tar.gz tar: /home/backup/back.tar.gz: Cannot write: No space left on device tar: Error is not recoverable: exiting... (10 Replies)
Discussion started by: siva3492
10 Replies

2. Linux

File size limitation in Linux

Hi friends, I tried to take a backup of my PC using tar command. But it ended with an error tar: /home/backup/back.tar.gz: Cannot write: No space left on device tar: Error is not recoverable: exiting now But i checked the disk space and there is enough space is available. ]# df Filesystem... (11 Replies)
Discussion started by: siva3492
11 Replies

3. Solaris

How to extend 2 GB file size limitation

Hello All, I am using a SunOS machine. My application creates output files for the downstream systems. However output files are restricted to 2GB of file size in SunOS due to which I am forced to create multiple files which is not supported by the downstream due to some limitations. Is... (5 Replies)
Discussion started by: pasupuleti81
5 Replies

4. UNIX for Advanced & Expert Users

Find command -size option limitation ?

Hi All, I ran code in test environment to find the files more than 1TB given below is a snippet from code: FILE_SYSTEM=/home/arun MAX_FILE_LIMIT=1099511627776 find $FILE_SYSTEM -type f -size +"$MAX_FILE_LIMIT"c -ls -xdev 2>/dev/null | while read fname do echo "File larger than... (3 Replies)
Discussion started by: Arunprasad
3 Replies

5. Shell Programming and Scripting

fetchmail - log file size limitation

Hi, I am using fetchmail in my application so as to download mails to the localhost where the application is hosted from the mailserver.Fetchmail is configured as as to run as a daemon polling mails during an interval of 1sec. So my concern here is, during each 2sec it is writing two... (10 Replies)
Discussion started by: DILEEP410
10 Replies

6. Shell Programming and Scripting

Size limitation in Tar command

Hi to every body there, I am new this forum and this is my first post. I am a new user of Unix, is there any size limitation of files while creating tar file. Thanks in advance (4 Replies)
Discussion started by: Manvar Khan
4 Replies

7. UNIX for Dummies Questions & Answers

Unix -- Space problem -- File number limitation?

Dear all Recently I cant touch file in one mount point (which is not full, 78% full only), it says can't write to device, obviously it means it's full, I deleted some files and I can write some files only. I wonder is there any file number limitation in a mount point and how can I check or how... (2 Replies)
Discussion started by: shanemcmahon
2 Replies

8. Linux

File size limitation for rcp

Hi I am trying to rcp a file from Solaris box to Linux. When the file size is 2,205,255,047, the rcp fails with the message Jan 10 01:11:53 hqsas167 rsh: pam_authenticate: error Authentication failed However when I rcp a file with smaller size - 9,434,477 - the rcp completes with... (2 Replies)
Discussion started by: schoubal
2 Replies

9. HP-UX

HP-UX 11i - File Size Limitation And Number Of Folders Limitation

Hi All, Can anyone please clarify me the following questions: 1. Is there any file size limitation in HP-UX 11i, that I can able to create upto certain size of file (say 2 GB) and not more then that???? 2. At max. how many files we can able to keep inside a folder???? 3. How many... (2 Replies)
Discussion started by: sundeep_mohanty
2 Replies

10. Solaris

unix group file limitation

Does anyone know how to get around the unix group file limitation whereby you have a limit of 1024 characters when adding users to a unix group? (3 Replies)
Discussion started by: asmillie
3 Replies
Login or Register to Ask a Question