Unreal Engine and custom data textures
Recently we needed to add custom rendering capabilities to Unreal Engine. Such a work requires a deep dive into dark areas of the heavily abstracted UE4 RHI ecosystem, especially given the fact that Epic does not provide extensive documentation on the internal core APIs of the engine. Several good articles have been written on custom rendering implementation in UE4 by Matt Hoffman. Major steps had been also covered pretty well in this blog post(the link contains the content mirror as the blog appears to be dead). Yet, you will have to figure out most of the things by yourself as none of the above mentioned articles present a step by step tutorial. This article is not another ‘introduction” into custom UE4 rendering, nor it will attempt to teach you a complete tutorial how to incorporate your own rendering pipeline. In this post I want to talk about data textures and their integration into a custom rendering pass.
What are the ‘custom’ data textures?
First, all the textures used in modern real-time rendering are basically data textures. The storage of the textures is usually a block of memory residing in main memory, or in GPU (Video) memory. The data itself can be expressed via one of the multiple formats available to the developer during texture setup phase. Direct3D, Vulkan, OpenGL — all these APIs come with a long list of available texture formats. One would normally select RGB,RGBA,BGRA,BGR,RG, or even single channel R (gray scale textures) 8bit per channel format for pixel data storage. Additionally, as in case of UE4, image data imported into the engine is transcoded into one of the compressed texture formats such as DXT1 or DXT5 (aka DDS), which is the default format for external image data. HDR images usually require RGB F16 format where each channel is a half float (16bit) value . In complex real life rendering scenarios very often textures are used to store a data unrelated to pixels. Here are examples of such cases:
- A specific shader stage generates an arbitrary data such as physics simulation, procedural geometry, distance fields etc. and stores results in texture memory .This technique is actually employed by most of the CUDA based ML framework where texture arrays are used much more frequently than buffers.
- There is a need to feed the shaders with an additional external data to perform complex custom shading such as BRDF LUT maps required for image based lighting calculation(IBL) in PBR rendering. Some GPU based vector shape rendering techniques use textures to fetch 2d shape data such as lines and curves for rasterization in a pixel shader.
Texture formats in Unreal Engine.
Texture formats in UE4 are exposed to developer via editor interface when one imports an image, creates cube map or render target, which is also texture. I won’t list here all the available formats, but just note that UE4 cover most of the possible use cases when it comes to pixel data.
An arbitrary data of float or integer type usually requires a format such as 2 or 4 bytes float, signed or unsigned 32bit integer. There also cases where each data entry in a texture is going to occupy 2 or even 4 byte (float or integer) per channel. In OpenGL. for example, this would be GL_RGBA32F, GL_RGBA32UI for float and unsigned integer textures respectively (4 bytes per channel). UE4 editor has only an option for F16 floating format which is single channel half float. This is one of those cases where you have to use UE4 C++ API. Texture2D creation interface gives us access to many more formats unavailable in the editor. When you create UE4 texture via CreateTransient() function you have EPixelFormat enumerator, where UE4 keeps most of the texture formats supported by modern GPUs. The generic EPixelFormat setup happens in RHI.cpp where we already can learn something about each UE4 format type.
The problem here is ,again, the documentation. We needed to create two textures:
- Half float (2 bytes) per channel RGBA texture.
- Short integer (2 bytes) per channel RGBA texture.
EPixelFormat enum list is not documented and we were not sure, for example, if EPixelFormat::PF_FloatRGBA corresponds to GL_RGBA16F or GL_RGBA32F, though there is also EPixelFormat::PF_A32B32G32R32F,which is definitely 4 bytes per channel ABGR.
It is hard to dig deeper into the low level rendering API (RHI) via UTexture2D::CreateTransient() as it doesn’t invoke RHI related routines directly. The function is made to be called from the game thread, and the actual RHI resource creation happens on the rendering thread. In our use case we decided to use the RHI resources directly as the custom rendering logic was implemented in a regular non UObject class since they were not supposed to be exposed to the blueprints or the application developers. Once you create a texture using raw RHI API you can figure out a lot of “under the hood” stuff via tracing the call stack. Here is what happens when you create a texture FTexture2DRHIRef object (must be called on the rendering thread):
//Our inhouse method to create 'raw' FTexture2DRHI object
static FTexture2DRHIRef CreateRHITextureResource(unsigned int width,unsigned int height,EPixelFormat pixelFormat){FRHIResourceCreateInfo CreateInfo;FTexture2DRHIRef Texture = RHICreateTexture2D(width, height, pixelFormat, 1, 1, TexCreate_ShaderResource, CreateInfo);return Texture;}
Where for pixelFormat we passed EPixelFormat::PF_FloatRGBA assuming it stands for RGBA half float per channel. To verify our assumption we had to do the following:
RHICommandList.h
FORCEINLINE FTexture2DRHIRef RHICreateTexture2D(uint32 SizeX, uint32 SizeY, uint8 Format, uint32 NumMips, uint32 NumSamples, ETextureCreateFlags Flags, FRHIResourceCreateInfo& CreateInfo);Calls:
FORCEINLINE FTexture2DRHIRef RHICreateTexture2D(uint32 SizeX, uint32 SizeY, uint8 Format, uint32 NumMips, uint32 NumSamples, ETextureCreateFlags Flags, ERHIAccess InResourceState, FRHIResourceCreateInfo& CreateInfo)Calls:
FTexture2DRHIRef FDynamicRHI::RHICreateTexture2D_RenderThread(class FRHICommandListImmediate& RHICmdList, uint32 SizeX, uint32 SizeY, uint8 Format, uint32 NumMips, uint32 NumSamples, ETextureCreateFlags Flags, ERHIAccess InResourceState, FRHIResourceCreateInfo& CreateInfo);Calls:
GDynamicRHI->RHICreateTexture2D(SizeX, SizeY, Format, NumMips, NumSamples, Flags, InResourceState, CreateInfo);Boom! Here is where the things start getting interesting. GDynamicRHI presents a polymorphic interface which in real-time implements the chosen low level rendering API backend. That’s, In UE4 there is DynamicRHI interface implementation for DirectX 11/12, Vulkan, Metal and OpenGL. When Ctrl + Left Mouse clicking the above method in Visual Studio, one gets the intellisense proposing all the different flavors of RHICreateTexture2D by related RHI. Let’s pick OpenGL driver and keep submersing further:
FTexture2DRHIRef FOpenGLDynamicRHI::RHICreateTexture2D(uint32 SizeX,uint32 SizeY,uint8 Format,uint32 NumMips,uint32 NumSamples,ETextureCreateFlags Flags, ERHIAccess InResourceState,FRHIResourceCreateInfo& Info);Calls a pretty much self explanatory:
(FRHITexture2D*)CreateOpenGLTexture(SizeX,SizeY,false,false,false,Format,NumMips,NumSamples,1,Flags,Info.ClearValueBinding,Info.BulkData);Calls:
FRHITexture* Texture = CreateOpenGLRHITextureOnly(SizeX, SizeY, bCubeTexture, bArrayTexture, bIsExternal, Format, NumMips, NumSamples, ArraySize, Flags, InClearValue, BulkData);which calls a function:
InitializeGLTexture(Texture, SizeX, SizeY, bCubeTexture, bArrayTexture, bIsExternal, Format, NumMips, NumSamples, ArraySize, Flags, InClearValue, BulkData);This method is a good example of the state of the art multiple rendering API abstraction mechanism in action. At this point, ‘Format’ parameter arrives as uint8 numeric and not as EPixelFormat. This is the place where Unreal Engine must figure out what native texture format (OpenGL in this case) the provided EPixelFormat corresponds to. Unreal Engine keeps a lookup table in a form of array per RHI type. The size of the array corresponds to the size of EPixelFormat enumerator and each entry in this array keeps native texture format information that corresponds to each value in EPixelFormat. And here is how it works in practice in case of OpenGL RHI:
InitializeGLTexture() retrieves related texture format in this line:
const FOpenGLTextureFormat& GLFormat = GOpenGLTextureFormats[Format];So the actual EPixelFormat to OpenGL texture format mapping happens via indexing into GOpenGLTextureFormats array consisting of FOpenGLTextureFormat type,which contains extra fields, specific to OpenGL API which describe chosen texture format during creation. OpenGLDevice.cpp contains the following method:
static void InitRHICapabilitiesForGL()It generates GOpenGLTextureFormats array and that’s actually our final stop. Here we can learn about every OpenGL format and how it is mapped to the generic EPixelFormat:
We can see, for example, that PF_FloatRGBA maps to GL_RGBA16F, half float per channel. As I already mentioned, in our use case the second texture format was short integer (2 bytes) per channel RGBA. Looking at the code above we can see that EPixelFormat::PF_R16G16B16A16_UINT maps to GL_RGBA16, which is GL_UNSIGNED_SHORT. This is exactly what we needed.
Additional info.
UE4 is not well documented when it comes to the areas like RHI. Yet, among different source files one can stumble upon locations such as NoExportType.h header, which provides some interesting comments for the fields in EPixelFormat as well as brief explanation how to add new pixel format types.
Congratulations! You have been able to take a look at a very tiny part of the very complex low level rendering infrastructure in UE4. The ability to freely access and learn the source code of Unreal Engine makes it formidable differentiator compared to other commercial game engines. It makes it possible to learn about industrial rendering engine architecture and with the right skills it is totally possible to modify and extend any part of the engine to specific needs.
In the next article I will show how to use these textures in a custom render pass.
