Sponsored Content
Top Forums Shell Programming and Scripting File size limitation of unix sort command. Post 85409 by cskumar on Wednesday 5th of October 2005 09:40:49 AM
Old 10-05-2005
Hi Pixelbeat,

Thanks for your quick response. We are trying this on HP UX 11. We are generating input file for sorting through c program and calling unix sort command through c only, as follows::

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

FILE *fout;
int main(int argc,char *argv[])
{
int loopcnt=0;
unsigned long int TTRECS = 60000000;

if((fout = fopen("SORTINPUT","w")) == NULL)
{
printf("\nUnable to open the output_file file\n");
exit(1);
}
for (loopcnt=TTRECS;loopcnt>=1;loopcnt--)
{
fprintf(fout,"%d|%s%d|%s|\n",loopcnt,"Dummy",loopcnt,"Dummy Description");
}
fflush(fout);
printf("Executing::sort -t '|' -k 1,1 SORTINPUT> sorttestoutput\n");

system("echo Sort Start Time;date");
system("sort -t '|' -k 1,1 SORTINPUT > sorttestoutput");
system("echo status returned::$?");

system("echo Sort End Time;date");
printf("Total Records Configured:: %lu\n", TTRECS);
return 0;
}


We are trying this test just to make sure that unix sort command is not having any limitations on file size. But what we observed is when file is packed with more than 60 million records [ around 2 Gb size], we are getting an error stating "File size limit exceeded" on console. Can u kindly state,

1] whether we got any maximum file size limit in HP UX while creating a file through C program.

2] Can unix sort command can handle file size > 2 GB without any failure.

3] Is there any restriction for length of each record [each line] or each field[fields are separated by a delimeter].

Thanks,
 

10 More Discussions You Might Find Interesting

1. Solaris

unix group file limitation

Does anyone know how to get around the unix group file limitation whereby you have a limit of 1024 characters when adding users to a unix group? (3 Replies)
Discussion started by: asmillie
3 Replies

2. HP-UX

HP-UX 11i - File Size Limitation And Number Of Folders Limitation

Hi All, Can anyone please clarify me the following questions: 1. Is there any file size limitation in HP-UX 11i, that I can able to create upto certain size of file (say 2 GB) and not more then that???? 2. At max. how many files we can able to keep inside a folder???? 3. How many... (2 Replies)
Discussion started by: sundeep_mohanty
2 Replies

3. Linux

File size limitation for rcp

Hi I am trying to rcp a file from Solaris box to Linux. When the file size is 2,205,255,047, the rcp fails with the message Jan 10 01:11:53 hqsas167 rsh: pam_authenticate: error Authentication failed However when I rcp a file with smaller size - 9,434,477 - the rcp completes with... (2 Replies)
Discussion started by: schoubal
2 Replies

4. UNIX for Dummies Questions & Answers

Unix -- Space problem -- File number limitation?

Dear all Recently I cant touch file in one mount point (which is not full, 78% full only), it says can't write to device, obviously it means it's full, I deleted some files and I can write some files only. I wonder is there any file number limitation in a mount point and how can I check or how... (2 Replies)
Discussion started by: shanemcmahon
2 Replies

5. Shell Programming and Scripting

Size limitation in Tar command

Hi to every body there, I am new this forum and this is my first post. I am a new user of Unix, is there any size limitation of files while creating tar file. Thanks in advance (4 Replies)
Discussion started by: Manvar Khan
4 Replies

6. Shell Programming and Scripting

fetchmail - log file size limitation

Hi, I am using fetchmail in my application so as to download mails to the localhost where the application is hosted from the mailserver.Fetchmail is configured as as to run as a daemon polling mails during an interval of 1sec. So my concern here is, during each 2sec it is writing two... (10 Replies)
Discussion started by: DILEEP410
10 Replies

7. UNIX for Advanced & Expert Users

Find command -size option limitation ?

Hi All, I ran code in test environment to find the files more than 1TB given below is a snippet from code: FILE_SYSTEM=/home/arun MAX_FILE_LIMIT=1099511627776 find $FILE_SYSTEM -type f -size +"$MAX_FILE_LIMIT"c -ls -xdev 2>/dev/null | while read fname do echo "File larger than... (3 Replies)
Discussion started by: Arunprasad
3 Replies

8. Solaris

How to extend 2 GB file size limitation

Hello All, I am using a SunOS machine. My application creates output files for the downstream systems. However output files are restricted to 2GB of file size in SunOS due to which I am forced to create multiple files which is not supported by the downstream due to some limitations. Is... (5 Replies)
Discussion started by: pasupuleti81
5 Replies

9. Linux

File size limitation in Linux

Hi friends, I tried to take a backup of my PC using tar command. But it ended with an error tar: /home/backup/back.tar.gz: Cannot write: No space left on device tar: Error is not recoverable: exiting now But i checked the disk space and there is enough space is available. ]# df Filesystem... (11 Replies)
Discussion started by: siva3492
11 Replies

10. Linux

File size limitation in the EST 2012 x86_64 GNU/Linux

Hello Friends, I tried to take tar backup in my server, but it ended with an error. It said that: /home/back/pallava_backup/fbackup_backup/stape_config /home/back/romam_new.tar.gz tar: /home/backup/back.tar.gz: Cannot write: No space left on device tar: Error is not recoverable: exiting... (10 Replies)
Discussion started by: siva3492
10 Replies
poll(7d)																  poll(7d)

NAME
poll - driver for fast poll on many file descriptors SYNOPSIS
#include <sys/devpoll.h> int fd = open("/dev/poll", O_RDWR); ssize_t n = write(int fd, struct pollfd buf[], int bufsize); int n = ioctl(int fd, DP_POLL, struct dvpoll* arg); int n = ioctl(int fd, DP_ISPOLLED, struct pollfd* pfd); fd Open file descriptor that refers to the /dev/poll driver. path /dev/poll buf Array of pollfd structures. bufsize Size of buf in bytes. arg Pointer to pollcall structure. pfd Pointer to pollfd structure. Note - The /dev/poll device, associated driver and corresponding manpages may be removed in a future Solaris release. For similar func- tionality in the event ports framework, see port_create(3C). The /dev/poll driver is a special driver that enables you to monitor multiple sets of polled file descriptors. By using the /dev/poll driver, you can efficiently poll large numbers of file descriptors. Access to the /dev/poll driver is provided through open(2), write(2), and ioctl(2) system calls. Writing an array of pollfd struct to the /dev/poll driver has the effect of adding these file descriptors to the monitored poll file descriptor set represented by the fd. To monitor multiple file descriptor sets, open the /dev/poll driver multiple times. Each fd corre- sponds to one set. For each pollfd struct entry (defined in sys/poll.h): struct pollfd { int fd; short events; short revents; } The fd field specifies the file descriptor being polled. The events field indicates the interested poll events on the file descriptor. If a pollfd array contains multiple pollfd entries with the same fd field, the "events" field in each pollfd entry is OR'ed. A special POLLRE- MOVE event in the events field of the pollfd structure removes the fd from the monitored set. The revents field is not used. Write returns the number of bytes written successfully or -1 when write fails. The DP_POLL ioctl is used to retrieve returned poll events occured on the polled file descriptors in the monitored set represented by fd. arg is a pointer to the devpoll structures which are defined as follows: struct dvpoll { struct pollfd* dp_fds; int dp_nfds; int dp_timeout; } The dp_fds points to a buffer that holds an array of returned pollfd structures. The dp_nfds field specifies the size of the buffer in terms of the number of pollfd entries it contains. The dp_nfds field also indicates the maximum number of file descriptors from which poll information can be obtained. If there is no interested events on any of the polled file descriptors, the DP_POLL ioctl call will wait dp_timeout milliseconds before returning. If dp_timeout is 0, the ioctl call returns immediately. If dp_timeout is -1, the call blocks until an interested poll events is available or the call is interrupted. Upon return, if the ioctl call has failed, -1 is returned. The memory content pointed by dp_fds is not modified. A return value 0 means the ioctl is timed out. In this case, the memory content pointed by dp_fds is not modified. If the call is successful, it returns the number of valid pollfd entries in the array pointed by dp_fds; the contents of the rest of the buffer is undefined. For each valid pollfd entry, the fd field indicates the file desciptor on which the polled events happened. The events field is the user specified poll events. The revents field contains the events occurred. -1 is returned if the call fails. DP_ISPOLLED ioctl allows you to query if a file descriptor is already in the monitored set represented by fd. The fd field of the pollfd structure indicates the file descriptor of interest. The DP_ISPOLLED ioctl returns 1 if the file descriptor is in the set. The events field contains 0. The revents field contains the currently polled events. The ioctl returns 0 if the file descriptor is not in the set. The pollfd structure pointed by pfd is not modified. The ioctl returns a -1 if the call fails. The following example shows how /dev/poll may be used. { ... /* * open the driver */ if ((wfd = open("/dev/poll", O_RDWR)) < 0) { exit(-1); } pollfd = (struct pollfd* )malloc(sizeof(struct pollfd) * MAXBUF); if (pollfd == NULL) { close(wfd); exit(-1); } /* * initialize buffer */ for (i = 0; i < MAXBUF; i++) { pollfd[i].fd = fds[i]; pollfd[i].events = POLLIN; pollfd[i].revents = 0; } if (write(wfd, &pollfd[0], sizeof(struct pollfd) * MAXBUF) != sizeof(struct pollfd) * MAXBUF) { perror("failed to write all pollfds"); close (wfd); free(pollfd); exit(-1); } /* * read from the devpoll driver */ dopoll.dp_timeout = -1; dopoll.dp_nfds = MAXBUF; dopoll.dp_fds = pollfd; result = ioctl(wfd, DP_POLL, &dopoll); if (result < 0) { perror("/dev/poll ioctl DP_POLL failed"); close (wfd); free(pollfd); exit(-1); } for (i = 0; i < result; i++) { read(dopoll.dp_fds[i].fd, rbuf, STRLEN); } ... } The following example is part of a test program which shows how DP_ISPOLLED() ioctl may be used. { ... loopcnt = 0; while (loopcnt < ITERATION) { rn = random(); rn %= RANGE; if (write(fds[rn], TESTSTRING, strlen(TESTSTRING)) != strlen(TESTSTRING)) { perror("write to fifo failed."); close (wfd); free(pollfd); error = 1; goto out1; } dpfd.fd = fds[rn]; dpfd.events = 0; dpfd.revents = 0; result = ioctl(wfd, DP_ISPOLLED, &dpfd); if (result < 0) { perror("/dev/poll ioctl DP_ISPOLLED failed"); printf("errno = %d0, errno); close (wfd); free(pollfd); error = 1; goto out1; } if (result != 1) { printf("DP_ISPOLLED returned incorrect result: %d.0, result); close (wfd); free(pollfd); error = 1; goto out1; } if (dpfd.fd != fds[rn]) { printf("DP_ISPOLLED returned wrong fd %d, expect %d0, dpfd.fd, fds[rn]); close (wfd); free(pollfd); error = 1; goto out1; } if (dpfd.revents != POLLIN) { printf("DP_ISPOLLED returned unexpected revents %d0, dpfd.revents); close (wfd); free(pollfd); error = 1; goto out1; } if (read(dpfd.fd, rbuf, strlen(TESTSTRING)) != strlen(TESTSTRING)) { perror("read from fifo failed"); close (wfd); free(pollfd); error = 1; goto out1; } loopcnt++; } EACCES A process does not have permission to access the content cached in /dev/poll. EINTR A signal was caught during the execution of the ioctl(2) function. EFAULT The request argument requires a data transfer to or from a buffer pointed to by arg, but arg points to an illegal address. EINVAL The request or arg parameter is not vald for this device, or field of the dvpoll struct pointed by arg is not valid (for example, when using write/pwrite dp_nfds is greater than {OPEN_MAX}, or when using the DPPOLL ioctl dp_nfds is greater than or equal to {OPEN_MAX}}. ENXIO The O_NONBLOCK flag is set, the named file is a FIFO, the O_WRONLY flag is set, and no process has the file open for read- ing; or the named file is a character special or block special file and the device associated with this special file does not exist. See attributes(5) for a description of the following attributes: +-----------------------------+-------------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |Architecture |SPARC, | |Availability |SUNWcar ( Solaris) | | |SUNWcsr, SUNWcsu (Solaris on ) | | |SUNWhea (header files) | |Interface Stability |Obsolete | |MT-Level |Safe | +-----------------------------+-------------------------------+ open(2), poll(2), write(2), attributes(5) The /dev/poll API is particularly beneficial to applications that poll a large number of file descriptors repeatedly. Applications will exhibit the best performance gain if the polled file descriptor list rarely change. When using the /dev/poll driver, you should remove a closed file descriptor from a monitored poll set. Failure to do so may result in a POLLNVAL revents being returned for the closed file descriptor. When a file descriptor is closed but not removed from the monitored set, and is reused in subsequent open of a different device, you will be polling the device associated with the reused file descriptor. In a multithreaded application, careful coordination among threads doing close and DP_POLL ioctl is recommended for consistent results. The /dev/poll driver caches a list of polled file descriptors, which are specific to a process. Therefore, the /dev/poll file descriptor of a process will be inherited by its child process, just like any other file descriptors. But the child process will have very limited access through this inherited /dev/poll file descriptor. Any attempt to write or do ioctl by the child process will result in an EACCES error. The child process should close the inherited /dev/poll file descriptor and open its own if desired. The /dev/poll driver does not yet support polling. Polling on a /dev/poll file descriptor will result in POLLERR being returned in the revents field of pollfd structure. 14 Sep 2005 poll(7d)
All times are GMT -4. The time now is 10:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy