MPSImage(3) MetalPerformanceShaders.framework MPSImage(3)
NAME
MPSImage
SYNOPSIS
#import <MPSImage.h>
Inherits NSObject.
Inherited by MPSTemporaryImage.
Instance Methods
(nonnull instancetype) - initWithDevice:imageDescriptor:
(nonnull instancetype) - initWithParentImage:sliceRange:featureChannels:
(nonnull instancetype) - initWithTexture:featureChannels:
(nonnull instancetype) - init
(MPSImageBatch *__nonnull) - batchRepresentationWithSubRange:
(MPSImageBatch *__nonnull) - batchRepresentation
(MPSImage *__nonnull) - subImageWithFeatureChannelRange:
(NSUInteger) - resourceSize
(MPSPurgeableState) - setPurgeableState:
(void) - readBytes:dataLayout:bytesPerRow:region:featureChannelInfo:imageIndex:
(void) - writeBytes:dataLayout:bytesPerRow:region:featureChannelInfo:imageIndex:
(void) - readBytes:dataLayout:bytesPerRow:bytesPerImage:region:featureChannelInfo:imageIndex:
(void) - writeBytes:dataLayout:bytesPerRow:bytesPerImage:region:featureChannelInfo:imageIndex:
(void) - readBytes:dataLayout:imageIndex:
(void) - writeBytes:dataLayout:imageIndex:
(void) - synchronizeOnCommandBuffer:
Class Methods
(nonnull id< MPSImageAllocator >) + defaultAllocator
Properties
id< MTLDevice > device
NSUInteger width
NSUInteger height
NSUInteger featureChannels
NSUInteger numberOfImages
MTLTextureType textureType
MTLPixelFormat pixelFormat
NSUInteger precision
MTLTextureUsage usage
size_t pixelSize
id< MTLTexture > texture
NSString * label
MPSImage * parent
Detailed Description
This depends on Metal.framework A MPSImage object describes a MTLTexture that may have more than 4 channels. Some image types, such as
those found in convolutional neural networks (CNN) differ from a standard texture in that they may have more than 4 channels per image.
While the channels could hold RGBA data, they will more commonly hold a number of structural permutations upon a multi-channel image as the
neural network progresses. It is not uncommon for each pixel to have 32 or 64 channels in it.
A standard MTLTexture may have no more than 4 channels. The additional channels are stored in slices of 2d texture array (i.e. texture type
is MTLTextureType2DArray) such that 4 consecutive channels are stored in each slice of this array. If the number of feature channels is N,
number of array slices needed is (N+3)/4. E.g. a CNN image with width 3 and height 2 with 9 channels will be stored as
slice 0 RGBA RGBA RGBA
RGBA RGBA RGBA
slice 1 RGBA RGBA RGBA
RGBA RGBA RGBA (ASCII art /diagonal offset/ intended to show a Z dimension)
slice 2 R??? R??? R???
R??? R??? R???
The width and height of underlying 2d texture array is the same as the width and height of the MPSImage. The array length is equal to
(featureChannels + 3) / 4. Channels marked with ? are just for padding and should not contain NaNs or Infs.
A MPSImage can be container of multiple CNN images for batch processing. In order to create a MPSImage that contains N images, create
MPSImageDescriptor with numberOfImages set to N.
Although a MPSImage can contain numberOfImages > 1, the actual number of images among these processed by MPSCNNKernel is controlled by z-
dimension of the clipRect. A MPSCNNKernel processes n=clipRect.size.depth images from this collection. The starting source image index to
process is given by offset.z. The starting index of the destination image is given by clipRect.origin.z. The MPSCNNKernel takes
n=clipRect.size.depth images from tje source at indices [offset.z, offset.z+n], processes each independently and stores the result in the
destination at indices [clipRect.origin.z, clipRect.origin.z+n] respectively. Offset.z+n should be <= [src numberOfImage] and
clipRect.origin.z+n should be <= [dest numberOfImages] and offset.z must be >= 0.
Example: Suppose MPSCNNConvolution takes an input image with 8 channels and outputs an image with 16 channels. The number of slices needed
in the source 2d texture array is 2 and the number of slices needed in the destination 2d array is 4. Suppose the source batch size is 5
and destination batch size is 4. (Multiple N-channel images can be processed concurrently in a batch.) The number of source slices will be
2*5=10 and number of destination slices will be 4*4=16. If you want to process just images 2 and 3 of the source and store the result at
index 1 and 2 in the destination, you may achieve this by setting offset.z=2, clipRect.origin.z=1 and clipRect.size.depth=2.
MPSCNNConvolution will take, in this case, slice 4 and 5 of source and produce slices 4 to 7 of destination. Similarly, slices 6 and 7 will
be used to produce slices 8 to 11 of destination.
All MPSCNNKernels process images within each batch independently. That is, calling a MPSCNNKernel on an batch is formally the same as
calling it on each image in the batch one at a time. However, quite a lot of CPU and GPU overhead will be avoided if batch processing is
used. This is especially important for better performance on small images.
If the number of feature channels is <= 4 and numberOfImages = 1 i.e. only one slice is needed to represent a MPSImage, the underlying
metal texture type will be MTLTextureType2D rather than MTLTextureType2DArray.
There are also MPSTemporaryImages, intended for use for very short-lived image data that are produced and consumed immediately in the same
MTLCommandBuffer. They are a useful way to minimize CPU-side texture allocation costs and greatly reduce the amount of memory used by your
image pipeline.
Creation of the underlying texture may in some cases occur lazily. You should in general avoid calling MPSImage.texture except when
unavoidable to avoid materializing memory for longer than necessary. When possible, use the other MPSImage properties to get information
about the MPSImage instead.
Most MPSImages of 4 or fewer feature channels can generate quicklooks output in Xcode for easy visualization of image data in the object.
MPSTemporaryImages can not.
Method Documentation
- (MPSImageBatch * __nonnull) batchRepresentation
Make a MPSImageBatch that points to the individual images in the MPSImage If the original is a temporary image, the result will be a
temporary image. It will hold 1 readCount on the original. When the readCount drops to 0, it will decrement the appropriate counter on the
parent.
Returns:
A MPSImageBatch aliasing the texel storage in the original batch image
- (MPSImageBatch * __nonnull) batchRepresentationWithSubRange: (NSRange) subRange
Make a representation of a MPSImage (batch) as a MPSImageBatch Before the MPSImageBatch was introduced, several images could be
concatenated into a MPSImage as multiple image slices in a MTLTexture2DArray to make a image batch. If the image contained more than 4
feature channels, then each image would have round_up( feature channels / 4) slices and the total number of slices in the MPSImage would be
slices * number of images.
Because many devices can operate on texture arrays of no more than 2048 slices, storage in this format is limited. For example in
InceptionV3, 2048 feature channels at its widest point, the largest batch size that can be processed in this way is 4, well under commonly
accepted practice for training. Consequently, the older batching style is deprecated and the MPSImageBatch is introduced. It is also easier
to manage sub-batches and to concatenate sub-batches for memory management with the MPSImageBatch, so this format is favored going forward.
To facilitate forward migration, this method will prepare an array of MPSImages that each point to the appropriate set of slices in storage
for the original image. Since they share storage, writes to the parent will alter the content of the children, and vice versa.
If the original is a temporary image, the result will be a temporary image. It will hold 1 readCount on the original. When the readCount
drops to 0, it will decrement the appropriate counter on the parent.
This is a much cheaper form of the slice operator, and should be used instead when the slice operator does not need to operate out of
place.
Parameters:
subRange The range of images in the original image from which the batch will be derived.
Returns:
A MPSImageBatch referencing a subregion of the original batch image.
+ (nonnull id <MPSImageAllocator>) defaultAllocator
Get a well known MPSImageAllocator that makes MPSImages
Reimplemented in MPSTemporaryImage.
- (nonnull instancetype) init
- (nonnull instancetype) initWithDevice: (nonnull id< MTLDevice >) device(const MPSImageDescriptor *__nonnull) imageDescriptor
Initialize an empty image object
Parameters:
device The device that the image will be used. May not be NULL.
imageDescriptor The MPSImageDescriptor. May not be NULL.
Returns:
A valid MPSImage object or nil, if failure. Storage to store data needed is allocated lazily on first use of MPSImage or when
application calls MPSImage.texture
Reimplemented in MPSTemporaryImage.
- (nonnull instancetype) initWithParentImage: (MPSImage *__nonnull) parent(NSRange) sliceRange(NSUInteger) featureChannels
Use -batchRepresentation or -subImageWithFeatureChannelRange instead Generally, you should call -batchRepresentation or
-subImageWithFeatureChannelRange instead because they are safer. This is provided so that these interfaces will work with your MPSImage
subclass.
Parameters:
parent The parent image that owns the texture. It may be a sub-image.
sliceRange The range of MTLTexture2dArray slices to be included in the sub-image
featureChannels The number of feature channels in the new image. The number of images is inferred.
Returns:
A MPSImage that references a subregion of the texel storage in parent instead of using its own storage.
- (nonnull instancetype) initWithTexture: (nonnull id< MTLTexture >) texture(NSUInteger) featureChannels
Initialize an MPSImage object using Metal texture. Metal texture has been created by user for specific number of feature channels and
number of images.
Parameters:
texture The MTLTexture allocated by the user to be used as backing for MPSImage.
featureChannels Number of feature channels this texture contains.
Returns:
A valid MPSImage object or nil, if failure. Application can let MPS framework allocate texture with properties specified in
imageDescriptor using initWithDevice:MPSImageDescriptor API above. However in memory intensive application, you can save memory (and
allocation/deallocation time) by using MPSTemporaryImage where MPS framework aggressively reuse memory underlying textures on same
command buffer. See MPSTemporaryImage class for details below. However, in certain cases, application developer may want more control
on allocation, placement, reusing/recycling of memory backing textures used in application using Metal Heaps API. In this case,
application can create MPSImage from pre-allocated texture using initWithTexture:featureChannels.
MTLTextureType of texture can be MTLTextureType2D ONLY if featureChannels <= 4 in which case MPSImage.numberOfImages is set to 1. Else it
should be MTLTextureType2DArray with arrayLength == numberOfImage * ((featureChannels + 3)/4). MPSImage.numberOfImages is set to
texture.arrayLength / ((featureChannels + 3)/4).
For MTLTextures containing typical image data which application may obtain from MetalKit or other libraries such as that drawn from a JPEG
or PNG, featureChannels should be set to number of valid color channel e.g. for RGB data, even thought MTLPixelFormat will be
MTLPixelFormatRGBA, featureChannels should be set to 3.
Reimplemented in MPSTemporaryImage.
- (void) readBytes: (void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) bytesPerRow(NSUInteger) bytesPerImage(MTLRegion)
region(MPSImageReadWriteParams) featureChannelInfo(NSUInteger) imageIndex
readBytes Get the values inside MPSImage and put them in the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
bytesPerRow Bytes to stride to point to next row(pixel just below current one) in the user buffer.
bytesPerImage Bytes to stride to point to next slice in the user buffer.
featureChannelInfo information user fills in to write to a set of feature channels in the image
imageIndex Image index in MPSImage to write to.
region region of the MPSImage to read from. A region is a structure with the origin in the Image from which to start reading values and
a size which represents the width and height of the rectangular region to read from. The z direction denotes the number of images, thus
for 1 image, origin.z = 0 and size.depth = 1 Use the enum to set data is coming in with what order. The data type will be determined
by the pixelFormat defined in the Image Descriptor.
- (void) readBytes: (void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) bytesPerRow(MTLRegion) region(MPSImageReadWriteParams)
featureChannelInfo(NSUInteger) imageIndex
readBytes Get the values inside MPSImage and put them in the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
bytesPerRow Bytes to stride to point to next row(pixel just below current one) in the user buffer.
featureChannelInfo information user fills in to write to a set of feature channels in the image
imageIndex Image index in MPSImage to write to.
region region of the MPSImage to read from. A region is a structure with the origin in the Image from which to start reading values and
a size which represents the width and height of the rectangular region to read from. The z direction denotes the number of images, thus
for 1 image, origin.z = 0 and size.depth = 1 Use the enum to set data is coming in with what order. The data type will be determined
by the pixelFormat defined in the Image Descriptor.
- (void) readBytes: (void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) imageIndex
readBytes Get the values inside MPSImage and put them in the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
imageIndex Image index in MPSImage to read from. Use the enum to set data is coming in with what order. The data type will be
determined by the pixelFormat defined in the Image Descriptor. Region is full image, buffer width and height is same as MPSImage width
and height.
- (NSUInteger) resourceSize
Get the number of bytes used to allocate underyling MTLResources This is the size of the backing store of underlying MTLResources. It does
not include all storage used by the object, for example the storage used to hold the MPSImage instantiation and MTLTexture is not included.
It only measures the size of the allocation used to hold the texels in the image. This value is subject to change between different devices
and operating systems.
Except when -initWithTexture:featureChannels: is used, most MPSImages (including MPSTemporaryImages) are allocated without a backing store.
The backing store is allocated lazily when it is needed, typically when the .texture property is called. Consequently, in most cases, it
should be inexpensive to make a MPSImage to see how much memory it will need, and release it if it is too large.
This method may fail in certain circumstances, such as when the MPSImage is created with -initWithTexture:featureChannels:, in which case 0
will be returned. 0 will also be returned if it is a sub-image or sub-batch (.parent is not nil).
- (MPSPurgeableState) setPurgeableState: (MPSPurgeableState) state
Set (or query) the purgeability state of a MPSImage Usage is per [MTLResource setPurgeableState:], except that the MTLTexture might be
MPSPurgeableStateAllocationDeferred, which means there is no texture to mark volatile / nonvolatile. Attempts to set purgeability on
MTLTextures that have not been allocated will be ignored.
- (MPSImage * __nonnull) subImageWithFeatureChannelRange: (NSRange) range
- (void) synchronizeOnCommandBuffer: (__nonnull id< MTLCommandBuffer >) commandBuffer
Flush the underlying MTLTexture from the device's caches, and invalidate any CPU caches if needed. This will call [id <MTLBlitEncoder>
synchronizeResource: ] on the image's MTLTexture, if any. This is necessary for all MTLStorageModeManaged resources. For other resources,
including temporary resources (these are all MTLStorageModePrivate), and textures that have not yet been allocated, nothing is done. It is
more efficient to use this method than to attempt to do this yourself with the texture property.
Parameters:
commandBuffer The commandbuffer on which to synchronize
- (void) writeBytes: (const void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) bytesPerRow(NSUInteger) bytesPerImage(MTLRegion)
region(MPSImageReadWriteParams) featureChannelInfo(NSUInteger) imageIndex
writeBytes Set the values inside MPSImage with the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
bytesPerRow Bytes to stride to point to next row(pixel just below current one) in the user buffer.
bytesPerImage Bytes to stride to point to next slice in the user buffer.
region region of the MPSImage to write to. A region is a structure with the origin in the Image from which to start writing values and
a size which represents the width and height of the rectangular region to write in. The z direction denotes the number of images, thus
for 1 image, origin.z = 0 and size.depth = 1
featureChannelInfo information user fills in to read from a set of feature channels in the image
imageIndex Image index in MPSImage to write to. Use the enum to set data is coming in with what order. The data type will be
determined by the pixelFormat defined in the Image Descriptor.
- (void) writeBytes: (const void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) bytesPerRow(MTLRegion)
region(MPSImageReadWriteParams) featureChannelInfo(NSUInteger) imageIndex
writeBytes Set the values inside MPSImage with the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
bytesPerRow Bytes to stride to point to next row(pixel just below current one) in the user buffer.
region region of the MPSImage to write to. A region is a structure with the origin in the Image from which to start writing values and
a size which represents the width and height of the rectangular region to write in. The z direction denotes the number of images, thus
for 1 image, origin.z = 0 and size.depth = 1
featureChannelInfo information user fills in to read from a set of feature channels in the image
imageIndex Image index in MPSImage to write to. Use the enum to set data is coming in with what order. The data type will be
determined by the pixelFormat defined in the Image Descriptor.
- (void) writeBytes: (const void *__nonnull) dataBytes(MPSDataLayout) dataLayout(NSUInteger) imageIndex
writeBytes Set the values inside MPSImage with the Buffer passed in.
Parameters:
dataBytes The array allocated by the user to be used to put data from MPSImage, the length should be imageWidth * imageHeight *
numberOfFeatureChannels and dataType should be inferred from pixelFormat defined in the Image Descriptor.
dataLayout The enum tells how to layout MPS data in the buffer.
imageIndex Image index in MPSImage to write to. Use the enum to set data is coming in with what order. The data type will be
determined by the pixelFormat defined in the Image Descriptor. Region is full image, buffer width and height is same as MPSImage width
and height.
Property Documentation
- device [read], [nonatomic], [retain]
The device on which the MPSImage will be used
- featureChannels [read], [nonatomic], [assign]
The number of feature channels per pixel.
- height [read], [nonatomic], [assign]
The formal height of the image in pixels.
- label [read], [write], [atomic], [copy]
A string to help identify this object.
- numberOfImages [read], [nonatomic], [assign]
numberOfImages for batch processing
- (MPSImage*) parent [read], [nonatomic], [retain]
The MPSImage from which this MPSImage was derived. Otherwise nil. This will point to the original image if this image was created using
-batchRepresentation, -batchRepresentationWithRange: or -subImageWithRange:.
- pixelFormat [read], [nonatomic], [assign]
The MTLPixelFormat of the underlying texture
- pixelSize [read], [nonatomic], [assign]
Number of bytes from the first byte of one pixel to the first byte of the next pixel in storage order. (Includes padding.)
- precision [read], [nonatomic], [assign]
The number of bits of numeric precision available for each feature channel. This is precision, not size. That is, float is 24 bits, not
32. half precision floating-point is 11 bits, not 16. SNorm formats have one less bit of precision for the sign bit, etc. For formats like
MTLPixelFormatB5G6R5Unorm it is the precision of the most precise channel, in this case 6. When this information is unavailable, typically
compressed formats, 0 will be returned.
- texture [read], [nonatomic], [assign]
The associated MTLTexture object. This is a 2D texture if numberOfImages is 1 and number of feature channels <= 4. It is a 2D texture array
otherwise. To avoid the high cost of premature allocation of the underlying texture, avoid calling this property except when strictly
necessary. [MPSCNNKernel encode...] calls typically cause their arguments to become allocated. Likewise, MPSImages initialized with
-initWithTexture: featureChannels: have already been allocated.
- textureType [read], [nonatomic], [assign]
The type of the underlying texture, typically MTLTextureType2D or MTLTextureType2DArray
- usage [read], [nonatomic], [assign]
Description of texture usage.
- width [read], [nonatomic], [assign]
The formal width of the image in pixels.
Author
Generated automatically by Doxygen for MetalPerformanceShaders.framework from the source code.
Version MetalPerformanceShaders-100 Thu Feb 8 2018 MPSImage(3)