Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

read() wont allow me to read files larger than 2 gig (on a 64bit)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users read() wont allow me to read files larger than 2 gig (on a 64bit)
# 1  
Old 12-23-2009
read() wont allow me to read files larger than 2 gig (on a 64bit)
Hi the following c-code utilizing the 'read()' man 2 read method cant read in files larger that 2gig.
Hi I've found a strange problem on ubuntu64bit, that limits the data you are allowed to allocate on a 64bit platform using the c function 'read()'

The following program wont allow to allocate more that 2.1gig memory.

PHP Code:
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>
#include <sysexits.h>
#include <unistd.h>
#include <sys/stat.h>

// get bytesize of file
size_t fsize(const char* fname){
  struct stat st ;
  stat(fname,&st);
  return st.st_size;
}

int main() {
  const char *infile = "bigfile.dat";
  int fd;
  size_t bytes_read, bytes_expected = fsize(infile);
  char *data;
 printf("\nLONG_MAX:%lu\n",LONG_MAX);

  if ((fd = open(infile,O_RDONLY)) < 0)
    err(EX_NOINPUT, "%s", infile);

  if ((data =(char *) malloc(bytes_expected)) == NULL)
    err(EX_OSERR, "data malloc");


  bytes_read = read(fd, data, bytes_expected);

  if (bytes_read != bytes_expected)
    err(EX_DATAERR, "Read only %lu of %lu bytes",bytes_read, bytes_expected);

  /* ... operate on data ... */

  free(data);

  exit(EX_OK);
} 
Code:
./a.out
LONG_MAX:9223372036854775807
a.out: Read only 2147479552 of 2163946253 bytes: Success

According to man 2 read, the maximum is limited by SSIZE_MAX
which is defined in

/usr/include/bits/posix1_lim.h
# define SSIZE_MAX LONG_MAX

And LONG_MAX is defined in /usr/include/limits.h as
# if __WORDSIZE == 64
# define LONG_MAX 9223372036854775807L
# else
# define LONG_MAX 2147483647L
# endif
# define LONG_MIN (-LONG_MAX - 1L)

Either this is a bug in the ubuntu buildsystem,
or my build system is broken.

Can anyone with a 64 try to run the above program.

Thanks

edit:

by the way
Code:
readelf -h ./a.out
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x400750
  Start of program headers:          64 (bytes into file)
  Start of section headers:          5312 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 34

Code:
ldd ./a.out
        linux-vdso.so.1 =>  (0x00007fff689ff000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ffee433e000)
        libm.so.6 => /lib/libm.so.6 (0x00007ffee40ba000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007ffee3ea3000)
        libc.so.6 => /lib/libc.so.6 (0x00007ffee3b34000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffee464e000)

thanks in advance
# 2  
Old 12-23-2009
ssize_t largest 32 bit value is ~2.1GB, which you seem to know. I had ubuntu about several years ago, and there was a largefile option you had to enable:

Code:
cc -D_FILE_OFFSET_BITS=64 myfile.c -o myfile

# 3  
Old 12-23-2009
Quote:
Originally Posted by jim mcnamara
ssize_t largest 32 bit value is ~2.1GB, which you seem to know. I had ubuntu about several years ago, and there was a largefile option you had to enable:

Code:
cc -D_FILE_OFFSET_BITS=64 myfile.c -o myfile

Is that necessary even with a 64-bit compile in Ubuntu? The OP's "readelf" output, and the results of his LONG_MAX output show he's running a 64-bit binary.
# 4  
Old 12-23-2009
As you can see in the code I supplied
PHP Code:
size_t bytes_expected = fsize(infile);
  if ((data =(char *) malloc(bytes_expected)) == NULL)
    err(EX_OSERR, "data malloc"); 
The problem is not related to the size_t, part but solely to the 'read' function.
If it was a problem with the size_t the malloc would complain.

The max size_t on a ubuntu64 is 18446744073709551615.

The compiler flag didn't work.
# 5  
Old 12-23-2009
Hello ,
I suppose ext3 filesystem doesnt support reading file sizes more than 2 GB
2GB Filesize Limit
# 6  
Old 12-23-2009
Of cause ext3 support larger than 2gig files on a 64bit platform

try
dd if=/dev/zero of=bigfile bs=1024 count=5145728
# 7  
Old 12-24-2009
Just a sentence. Probably there is an error here that is alike the one described: "Problems of 64-bit code in real programs: FreeBSD".
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[Bash] Read History function & Read Arrowkeys

Hi. How can I create a history function? (By "read" command or so) & How can I configure a read command so that the arrow keys are not displayed so funny? (^[[A) Thanks in advance. (4 Replies)
Discussion started by: sinnlosername
4 Replies

2. Shell Programming and Scripting

Read from file and execute the read command

Hi, I am facing issues with the below: I have a lookup file say lookup.lkp.This lookup.lkp file contains strings delimited by comma(,). Now i want to read this command from file and execute it. So my code below is : Contents in the lookup.lkp file is : c_e,m,a,`cd $BOX | ls cef_*|tail... (7 Replies)
Discussion started by: vital_parsley
7 Replies

3. Shell Programming and Scripting

Help with Bash piped while-read and a read user input at the same time

Hi I am new to writing script and want to use a Bash Piped while-read and read from user input. if something happens on server.log then do while loop or if something happend on user input then do while loop. Pseudocode something like: tail -n 3 -f server.log | while read serverline || read... (8 Replies)
Discussion started by: MyMorris
8 Replies

4. Shell Programming and Scripting

How to read log files from last read

Hi i am looking a way to look at a log file(log.txt) from the last time I've read it. However after some days the main log file(log.txt) is rename to (log.txt.1). So now i will have two log files as below. log.txt.1 log.txt Now, i have to read the log from the point where i have left... (3 Replies)
Discussion started by: sumitsks
3 Replies

5. UNIX for Dummies Questions & Answers

When reading a csv file, counter to read 20 lines and wait for minute then read next 20 till end

Hello All, i am a newbie and need some help when reading a csv file in a bourne shell script. I want to read 10 lines, then wait for a minute and then do a reading of another 10 lines and so on in the same way. I want to do this till the end of file. Any inputs are appreciated ... (3 Replies)
Discussion started by: victor.s
3 Replies

6. UNIX for Dummies Questions & Answers

Read statement within while read loop

hi, this is my script #!/bin/ksh cat temp_file.dat | while read line do read test if ]; then break else echo "ERROR" fi done when i execute this code , the script does wait for the user input . it directly prints "ERROR" and terminates after the no. of times as there... (3 Replies)
Discussion started by: siva1612
3 Replies

7. Shell Programming and Scripting

Read Embedded Newline characters with read (builtin) in KSH93

Hi Guys, Happy New Year to you all! I have a requirement to read an embedded new-line using KSH's read builtin. Here is what I am trying to do: run_sql "select guestid, address, email from guest" | while read id addr email do ## Biz logic goes here done I can take care of any... (6 Replies)
Discussion started by: a_programmer
6 Replies

8. Programming

Cannot read a file with read(fd, buffer, buffersize) function

# include <stdio.h> # include <fcntl.h> # include <stdlib.h> # include <sys/stat.h> int main(int argc, char *argv) { int fRead, fPadded, padVal; int btRead; int BUFFSIZE = 512; char buff; if (argc != 4) { printf ("Please provide all of the... (3 Replies)
Discussion started by: naranja18she
3 Replies

9. Shell Programming and Scripting

Can I use read to read content of a variable

Can I use the read command to read the contents of a variable? I'm trying by using the following code and getting nothing back. I'm in a Linux environment. #!/bin/ksh IFS=~ VAR1=1~2~3~4 echo $VAR1 | read a b c d print "$a $b $c $d" (9 Replies)
Discussion started by: nmalencia
9 Replies
Login or Register to Ask a Question