I'm rotating huge image rasters (1+ GB) using a 3-shear algorithm. I rotate 'in-place' after reallocating the raster. A necessary first step is reorganizing the image in the new raster dimensions that will exist after the rotation. RGBA pixels byte packed into uint32.
The following code runs fine on rasters < 4 Gb. No issues.
When I run it on rasters > 4 Gb, memmove() crashes.
I was under the impression that 64 bit Linux could take on all size data structures.
Any ideas out there. I am at my wits end on this.
Any replies greatly appreciated.
Imagtek
imagtek.com
Moderator's Comments:
Please use code tags
Last edited by jim mcnamara; 08-24-2014 at 06:04 PM..
Reason: CODE TAGS
4294967296 is 4GB; 4294967295 is usually the limit to the size of what an uint32_t pointer can reference. Change your datatype.
However, consider:
Your object must exceed that value or you have exhausted heap memory or exceeded the process memory limit if one is set. What exact error did you get? Fix your code to display what the error is, i.e.
Did you check: ulimit -a for the process. What does free show? While the code is running?
There are other factors that can also limit allocating giant chunks of memory - heap fragmentation. I do not know how to report that on RH or CentOS from the command line. I do know you can get heap information by writing C code to look at anonymous memory in /proc system-wide.
Heap framentation can occur when you make a bunch of calls to malloc or realloc in succession where the size continually doubles. And there are other process that tie up substantial heap memory or shared memory. You get to a point where the contiguous size chunk you want is not available but the total free memory (made of lots of smaller chunks) exceeds what you need. java is notorious for that - so much so that one of the startup parms allows you to force java to get max possible heap when it first starts.
Start by finding out the exact error, you are assuming ENOMEM.
Last edited by jim mcnamara; 08-24-2014 at 06:28 PM..
What are the values of Rcols and Rrows when realloc() is failing? What type are Rcols and Rrows?
Making a wild guess that longu is an unsigned long and Rcols and Rrows are ints (signed or unsigned), you may be overflowing type int in the calculation (Rcols*Rrows*4) before casting the result to a long. Have you tried:
instead of:
so the calculation will be performed using longs instead of ints?
As an example:
In the future, please show us the declarations for all of the variables we're looking at and any non-standard typedefs used in those declarations. Evaluating problems like this is much easier if we can see full details about the variables being used instead of having to hypothesize about possible problems and guessing at types that would lead to those problems.
PS: Jim: (uint32_t *)realloc(...) on a 64-bit compile casts the void * pointer returned by realloc() to a (64-bit) pointer to a 32-bit integer, not a 32-bit pointer. Note that if the code had used sizeof(uint32_t) instead of 4, this problem would disappear if my guess about integer overflow was correct.
Last edited by Don Cragun; 08-24-2014 at 08:59 PM..
Reason: Add PS
@Don - good point - but my assumptions are different. Here is why:
CentOS ( -Therefore probably RH. Solaris 10 sparc as well, my platform at work) defaults to gcc -m32 unless you tell it otherwise. My assumption is RH defaults to that and the OP blew past 4GB. There are two gcc libraries - one for 32 and one for 64.
Dunno if that is a good assumption.
The OP seems to be silent. But if he posts let's ask about the -m64 option to gcc. Or posting output from the command file compiled_image_file
Hi Jim,
Agreed. We have two plausible explanations for the reported problems and no information to determine if either (or both) of us guessed correctly.
Many sincere thanks. Very useful info. I admit that I have been rather cavalier slinging gigabytes of image memory around on a 64 bit machine with 8 Gb RAM, simply because I've been getting away with it, not due to deep understanding.
I've been assuming that (longu)(a * b * c) would cast every int32 in the expression to 64 bit long BEFORE the calculation. If that is the problem, how embarrassing...I thought I knew how to code.
I will pursue these leads and get back with the resolution. I think one of you probably nailed it. I want to be able to rotate an image of ANY size, machine capable of course.
Thanks again guys.
imagtek
---------- Post updated at 10:01 AM ---------- Previous update was at 08:52 AM ----------
OK. The problem is fixed. I was overflowing the expression used to determine the reallocated raster size.
The array dimensions Rcols, Rrows were int32. Plenty big for
any realistic raster dimension, but computationally vulnerable.
I assumed that casting a C/C++ expression such as
(long)(a * b * c)
had the effect of
( (long)a * (long)b * (long)c )
it does NOT.
I misunderstood the language spec. In 32 bit environment this hid in the grass. In 64 bit, it is a very big deal. I read the language spec over and over but somehow missed this subtlety.
The variables inside the expression are promoted to the largest type INSIDE the expression before computation. Any cast outside the expression effects only the expression result.
Images are getting big enough now that it finally bit me. Simple fix in about a dozen routines vulnerable to this.
Now my 4 Gb image rotates nicely to a 5 Gb image. Just takes awhile... Need to revisit some vulnerable code. Now I don't understand why some things did NOT blow up, but not an issue for here.
You're never to old to learn something you should have known a long time ago.
Thanks Guys! Bugs based upon misconceptions are the worst.
Hello everyone.
I bought a dell laptop (XPS M1330) online which came without a hard drive. There is a Windows Vista Ultimate OEMAct sticker with product key at the bottom case. I checked dell website (here) for this model and it says this model supports both 32 and 64-bit version of Windows... (4 Replies)
Hi All,
I am new to linux and i am configuring DNS server but while i restart service it shows below given error. Please help to solve this error.
# /etc/init.d/named restart
Stopping named:
Generating /etc/rndc.key:
... (2 Replies)
Hi,
Here is the issue. From the program snippet I have Base: 0x1800000000, Size: 0x3FFE7FFFFFFFF which are of 40 and 56 bits. SO I used use bignum to do the math but summing them up I always failed having correct result.
perl interpreter info,
perl, v5.8.8 built for... (0 Replies)
Hi,
I am using DB2 9.7 , configured odbc.ini file and exported environment variables but I am getting the following error.
error while loading shared libraries: libodbc.so.1: wrong ELF class: ELFCLASS64
Kindly help. (4 Replies)
In bash:
for i in 1 2 3 4 5
do
tmp_${i}=$i
echo $i
done
It gives error on executing tmp_1=1, and so on...
Why? If I executed this in prompt, not from script, it shouldn't make a subshell should it?
please use "code" tags! thanks... (2 Replies)
Besides syntax-related issues, is there anything else I should be aware of when using memmove() ? If misued, can it create runtime errors ? (3 Replies)