diff --git a/3rdparty/d3d12memalloc/CHANGELOG.md b/3rdparty/d3d12memalloc/CHANGELOG.md index 9e69431e1f..bc027d3bf8 100644 --- a/3rdparty/d3d12memalloc/CHANGELOG.md +++ b/3rdparty/d3d12memalloc/CHANGELOG.md @@ -1,3 +1,23 @@ -# 1.0.0 (2019-09-02) +# 2.0.1 (2022-04-05) + +A maintenance release with some bug fixes and improvements. There are no changes in the library API. + +- Fixed an assert failing when detailed JSON dump was made while a custom pool was present with specified string name (#36, thanks @rbertin-aso). +- Fixed image height calculation in JSON dump visualization tool "GpuMemDumpVis.py" (#37, thanks @rbertin-aso). +- Added JSON Schema for JSON dump format - see file "tools\GpuMemDumpVis\GpuMemDump.schema.json". +- Added documentation section "Resource reference counting". + +# 2.0.0 (2022-03-25) + +So much has changed since the first release that it doesn’t make much sense to compare the differences. Here are the most important features that the library now provides: + +- Powerful custom pools, which give an opportunity to not only keep certain resources together, reserve some minimum or limit the maximum amount of memory they can take, but also to pass additional allocation parameters unavailable to simple allocations. Among them, probably the most interesting is `POOL_DESC::HeapProperties`, which allows you to specify parameters of a custom memory type, which may be useful on UMA platforms. Committed allocations can now also be created in custom pools. +- The API for statistics and budget has been redesigned - see structures `Statistics`, `Budget`, `DetailedStatistics`, `TotalStatistics`. +- The library exposes its core allocation algorithm via the “virtual allocator” interface. This can be used to allocate pieces of custom memory or whatever you like, even something completely unrelated to graphics. +- The allocation algorithm has been replaced with the new, more efficient TLSF. +- Added support for defragmentation. +- Objects of the library can be used with smart pointers designed for COM objects. + +# 1.0.0 (2019-09-02) First published version. diff --git a/3rdparty/d3d12memalloc/README.md b/3rdparty/d3d12memalloc/README.md index 65a9ff59ca..81c6be18aa 100644 --- a/3rdparty/d3d12memalloc/README.md +++ b/3rdparty/d3d12memalloc/README.md @@ -41,7 +41,7 @@ Additional features: - Statistics: Obtain brief or detailed statistics about the amount of memory used, unused, number of allocated heaps, number of allocations etc. - globally and per memory heap type. Current memory usage and budget as reported by the system can also be queried. - Debug annotations: Associate custom `void* pPrivateData` and debug `LPCWSTR pName` with each allocation. - JSON dump: Obtain a string in JSON format with detailed map of internal state, including list of allocations, their string names, and gaps between them. -- Convert this JSON dump into a picture to visualize your memory using attached Python script. +- Convert this JSON dump into a picture to visualize your memory. See [tools/GpuMemDumpVis](tools/GpuMemDumpVis/README.md). - Virtual allocator - an API that exposes the core allocation algorithm to be used without allocating real GPU memory, to allocate your own stuff, e.g. sub-allocate pieces of one large buffer. # Prerequisites @@ -104,10 +104,12 @@ For more information see [NOTICES.txt](NOTICES.txt). # Software using this library - **[The Forge](https://github.com/ConfettiFX/The-Forge)** - cross-platform rendering framework. Apache License 2.0. +- **[Wicked Engine](https://github.com/turanszkij/WickedEngine)** - 3D engine with modern graphics [Some other projects on GitHub](https://github.com/search?q=D3D12MemAlloc.h&type=Code) and some game development studios that use DX12 in their games. # See also +- **[Vcpkg](https://github.com/Microsoft/vcpkg)** dependency manager from Microsoft offers a port of this library that is easy to install. - **[Vulkan Memory Allocator](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator/)** - equivalent library for Vulkan. License: MIT. - **[TerraFX.Interop.D3D12MemoryAllocator](https://github.com/terrafx/terrafx.interop.d3d12memoryallocator)** - interop bindings for this library for C#, as used by [TerraFX](https://github.com/terrafx/terrafx). License: MIT. diff --git a/3rdparty/d3d12memalloc/include/D3D12MemAlloc.h b/3rdparty/d3d12memalloc/include/D3D12MemAlloc.h index 4efbb50115..4e87bf0277 100644 --- a/3rdparty/d3d12memalloc/include/D3D12MemAlloc.h +++ b/3rdparty/d3d12memalloc/include/D3D12MemAlloc.h @@ -24,9 +24,9 @@ /** \mainpage D3D12 Memory Allocator -Version 2.0.0-development (2021-07-26) +Version 2.1.0-development (2023-07-05) -Copyright (c) 2019-2022 Advanced Micro Devices, Inc. All rights reserved. \n +Copyright (c) 2019-2023 Advanced Micro Devices, Inc. All rights reserved. \n License: MIT Documentation of all members: D3D12MemAlloc.h @@ -36,6 +36,7 @@ Documentation of all members: D3D12MemAlloc.h - \subpage quick_start - [Project setup](@ref quick_start_project_setup) - [Creating resources](@ref quick_start_creating_resources) + - [Resource reference counting](@ref quick_start_resource_reference_counting) - [Mapping memory](@ref quick_start_mapping_memory) - \subpage custom_pools - \subpage defragmentation @@ -50,7 +51,7 @@ Documentation of all members: D3D12MemAlloc.h - [Thread safety](@ref general_considerations_thread_safety) - [Versioning and compatibility](@ref general_considerations_versioning_and_compatibility) - [Features not supported](@ref general_considerations_features_not_supported) - + \section main_see_also See also - [Product page on GPUOpen](https://gpuopen.com/gaming-product/d3d12-memory-allocator/) @@ -58,9 +59,18 @@ Documentation of all members: D3D12MemAlloc.h */ // If using this library on a platform different than Windows PC or want to use different version of DXGI, -// you should include D3D12-compatible headers before this library on your own and define this macro. +// you should include D3D12-compatible headers before this library on your own and define +// D3D12MA_D3D12_HEADERS_ALREADY_INCLUDED. +// Alternatively, if you are targeting the open sourced DirectX headers, defining D3D12MA_USING_DIRECTX_HEADERS +// will include them rather the ones provided by the Windows SDK. #ifndef D3D12MA_D3D12_HEADERS_ALREADY_INCLUDED - #include + #if defined(D3D12MA_USING_DIRECTX_HEADERS) + #include + #include + #else + #include + #endif + #include #endif @@ -132,6 +142,18 @@ If providing your own implementation, you need to implement a subset of std::ato // Forward declaration if ID3D12ProtectedResourceSession is not defined inside the headers (older SDK, pre ID3D12Device4) struct ID3D12ProtectedResourceSession; +// Define this enum even if SDK doesn't provide it, to simplify the API. +#ifndef __ID3D12Device1_INTERFACE_DEFINED__ +typedef enum D3D12_RESIDENCY_PRIORITY +{ + D3D12_RESIDENCY_PRIORITY_MINIMUM = 0x28000000, + D3D12_RESIDENCY_PRIORITY_LOW = 0x50000000, + D3D12_RESIDENCY_PRIORITY_NORMAL = 0x78000000, + D3D12_RESIDENCY_PRIORITY_HIGH = 0xa0010000, + D3D12_RESIDENCY_PRIORITY_MAXIMUM = 0xc8000000 +} D3D12_RESIDENCY_PRIORITY; +#endif + namespace D3D12MA { class D3D12MA_API IUnknownImpl : public IUnknown @@ -144,7 +166,7 @@ public: protected: virtual void ReleaseThis() { delete this; } private: - D3D12MA_ATOMIC_UINT32 m_RefCount{1}; + D3D12MA_ATOMIC_UINT32 m_RefCount = {1}; }; } // namespace D3D12MA @@ -226,8 +248,6 @@ enum ALLOCATION_FLAGS /** Create allocation only if additional memory required for it, if any, won't exceed memory budget. Otherwise return `E_OUTOFMEMORY`. - - \warning Currently this feature is not fully implemented yet. */ ALLOCATION_FLAG_WITHIN_BUDGET = 0x4, @@ -237,7 +257,6 @@ enum ALLOCATION_FLAGS */ ALLOCATION_FLAG_UPPER_ADDRESS = 0x8, - /** Set this flag if the allocated memory will have aliasing resources. Use this when calling D3D12MA::Allocator::CreateResource() and similar to @@ -306,7 +325,6 @@ struct ALLOCATION_DESC /** \brief Custom pool to place the new resource in. Optional. When not NULL, the resource will be created inside specified custom pool. - It will then never be created as committed. */ Pool* CustomPool; /// Custom general-purpose pointer that will be stored in D3D12MA::Allocation. @@ -570,7 +588,6 @@ private: UINT64 m_Size; UINT64 m_Alignment; ID3D12Resource* m_Resource; - UINT m_CreationFrameIndex; void* m_pPrivateData; wchar_t* m_Name; @@ -637,7 +654,7 @@ private: AllocHandle GetAllocHandle() const; NormalBlock* GetBlock(); template - void SetResource(ID3D12Resource* resource, const D3D12_RESOURCE_DESC_T* pResourceDesc); + void SetResourcePointer(ID3D12Resource* resource, const D3D12_RESOURCE_DESC_T* pResourceDesc); void FreeName(); D3D12MA_CLASS_NO_COPY(Allocation) @@ -838,6 +855,14 @@ enum POOL_FLAGS */ POOL_FLAG_ALGORITHM_LINEAR = 0x1, + /** \brief Optimization, allocate MSAA textures as committed resources always. + + Specify this flag to create MSAA textures with implicit heaps, as if they were created + with flag D3D12MA::ALLOCATION_FLAG_COMMITTED. Usage of this flags enables pool to create its heaps + on smaller alignment not suitable for MSAA textures. + */ + POOL_FLAG_MSAA_TEXTURES_ALWAYS_COMMITTED = 0x2, + // Bit mask to extract only `ALGORITHM` bits from entire set of flags. POOL_FLAG_ALGORITHM_MASK = POOL_FLAG_ALGORITHM_LINEAR }; @@ -895,6 +920,29 @@ struct POOL_DESC Valid only if ID3D12Device4 interface is present in current Windows SDK! */ ID3D12ProtectedResourceSession* pProtectedSession; + /** \brief Residency priority to be set for all allocations made in this pool. Optional. + + Set this parameter to one of the possible enum values e.g. `D3D12_RESIDENCY_PRIORITY_HIGH` + to apply specific residency priority to all allocations made in this pool: + `ID3D12Heap` memory blocks used to sub-allocate for placed resources, as well as + committed resources or heaps created when D3D12MA::ALLOCATION_FLAG_COMMITTED is used. + This can increase/decrease chance that the memory will be pushed out from VRAM + to system RAM when the system runs out of memory, which is invisible to the developer + using D3D12 API while it can degrade performance. + + Priority is set using function `ID3D12Device1::SetResidencyPriority`. + It is performed only when `ID3D12Device1` interface is defined and successfully obtained. + Otherwise, this parameter is ignored. + + This parameter is optional. If you set it to `D3D12_RESIDENCY_PRIORITY(0)`, + residency priority will not be set for allocations made in this pool. + + There is no equivalent parameter for allocations made in default pools. + If you want to set residency priority for such allocation, you need to do it manually: + allocate with D3D12MA::ALLOCATION_FLAG_COMMITTED and call + `ID3D12Device1::SetResidencyPriority`, passing `allocation->GetResource()`. + */ + D3D12_RESIDENCY_PRIORITY ResidencyPriority; }; /** \brief Custom memory pool @@ -1009,6 +1057,14 @@ enum ALLOCATOR_FLAGS Only avaiable if `ID3D12Device8` is present. Otherwise, the flag is ignored. */ ALLOCATOR_FLAG_DEFAULT_POOLS_NOT_ZEROED = 0x4, + + /** \brief Optimization, allocate MSAA textures as committed resources always. + + Specify this flag to create MSAA textures with implicit heaps, as if they were created + with flag D3D12MA::ALLOCATION_FLAG_COMMITTED. Usage of this flags enables all default pools + to create its heaps on smaller alignment not suitable for MSAA textures. + */ + ALLOCATOR_FLAG_MSAA_TEXTURES_ALWAYS_COMMITTED = 0x8, }; /// \brief Parameters of created Allocator object. To be used with CreateAllocator(). @@ -1144,7 +1200,26 @@ public: Allocation** ppAllocation, REFIID riidResource, void** ppvResource); -#endif // #ifdef __ID3D12Device4_INTERFACE_DEFINED__ +#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + /** \brief Similar to Allocator::CreateResource2, but there are initial layout instead of state and + castable formats list + + It internally uses `ID3D12Device10::CreateCommittedResource3` or `ID3D12Device10::CreatePlacedResource2`. + + To work correctly, `ID3D12Device10` interface must be available in the current system. Otherwise, `E_NOINTERFACE` is returned. + */ + HRESULT CreateResource3(const ALLOCATION_DESC* pAllocDesc, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_BARRIER_LAYOUT InitialLayout, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + UINT32 NumCastableFormats, + DXGI_FORMAT* pCastableFormats, + Allocation** ppAllocation, + REFIID riidResource, + void** ppvResource); +#endif // #ifdef __ID3D12Device10_INTERFACE_DEFINED__ /** \brief Allocates memory without creating any resource placed in it. @@ -1201,6 +1276,41 @@ public: REFIID riidResource, void** ppvResource); +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + /** \brief Similar to Allocator::CreateAliasingResource, but supports new structure `D3D12_RESOURCE_DESC1`. + + It internally uses `ID3D12Device8::CreatePlacedResource1`. + + To work correctly, `ID3D12Device8` interface must be available in the current system. Otherwise, `E_NOINTERFACE` is returned. + */ + HRESULT CreateAliasingResource1(Allocation* pAllocation, + UINT64 AllocationLocalOffset, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_RESOURCE_STATES InitialResourceState, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + REFIID riidResource, + void** ppvResource); +#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + /** \brief Similar to Allocator::CreateAliasingResource1, but there are initial layout instead of state and + castable formats list + + It internally uses `ID3D12Device10::CreatePlacedResource2`. + + To work correctly, `ID3D12Device10` interface must be available in the current system. Otherwise, `E_NOINTERFACE` is returned. + */ + HRESULT CreateAliasingResource2(Allocation* pAllocation, + UINT64 AllocationLocalOffset, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_BARRIER_LAYOUT InitialLayout, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + UINT32 NumCastableFormats, + DXGI_FORMAT* pCastableFormats, + REFIID riidResource, + void** ppvResource); +#endif // #ifdef __ID3D12Device10_INTERFACE_DEFINED__ + /** \brief Creates custom pool. */ HRESULT CreatePool( @@ -1223,7 +1333,7 @@ public: - `pNonLocalBudget` returns the budget of the system memory available for D3D12 resources. - When IsUMA() `== TRUE` (integrated graphics chip): - `pLocalBudget` returns the budget of the shared memory available for all D3D12 resources. - All memory is considered "local". + All memory is considered "local". - `pNonLocalBudget` is not applicable and returns zeros. This function is called "get" not "calculate" because it is very fast, suitable to be called @@ -1246,8 +1356,9 @@ public: */ void CalculateStatistics(TotalStatistics* pStats); - /// Builds and returns statistics as a string in JSON format. - /** @param[out] ppStatsString Must be freed using Allocator::FreeStatsString. + /** \brief Builds and returns statistics as a string in JSON format. + * + @param[out] ppStatsString Must be freed using Allocator::FreeStatsString. @param DetailedMap `TRUE` to include full list of allocations (can make the string quite long), `FALSE` to only return statistics. */ void BuildStatsString(WCHAR** ppStatsString, BOOL DetailedMap) const; @@ -1559,9 +1670,9 @@ to be passed along with `D3D12_RESOURCE_DESC` and other parameters for created resource. This structure describes parameters of the desired memory allocation, including choice of `D3D12_HEAP_TYPE`. -The function also returns a new object of type D3D12MA::Allocation, created along -with usual `ID3D12Resource`. It represents allocated memory and can be queried -for size, offset, `ID3D12Resource`, and `ID3D12Heap` if needed. +The function returns a new object of type D3D12MA::Allocation. +It represents allocated memory and can be queried for size, offset, `ID3D12Heap`. +It also holds a reference to the `ID3D12Resource`, which can be accessed by calling D3D12MA::Allocation::GetResource(). \code D3D12_RESOURCE_DESC resourceDesc = {}; @@ -1580,7 +1691,6 @@ resourceDesc.Flags = D3D12_RESOURCE_FLAG_NONE; D3D12MA::ALLOCATION_DESC allocationDesc = {}; allocationDesc.HeapType = D3D12_HEAP_TYPE_DEFAULT; -D3D12Resource* resource; D3D12MA::Allocation* allocation; HRESULT hr = allocator->CreateResource( &allocationDesc, @@ -1588,15 +1698,16 @@ HRESULT hr = allocator->CreateResource( D3D12_RESOURCE_STATE_COPY_DEST, NULL, &allocation, - IID_PPV_ARGS(&resource)); + IID_NULL, NULL); + +// Use allocation->GetResource()... \endcode -You need to remember both resource and allocation objects and destroy them -separately when no longer needed. +You need to release the allocation object when no longer needed. +This will also release the D3D12 resource. \code allocation->Release(); -resource->Release(); \endcode The advantage of using the allocator instead of creating committed resource, and @@ -1619,6 +1730,65 @@ they can be kept together. By using this library, you don't need to handle this manually. +\section quick_start_resource_reference_counting Resource reference counting + +`ID3D12Resource` and other interfaces of Direct3D 12 use COM, so they are reference-counted. +Objects of this library are reference-counted as well. +An object of type D3D12MA::Allocation remembers the resource (buffer or texture) +that was created together with this memory allocation +and holds a reference to the `ID3D12Resource` object. +(Note this is a difference to Vulkan Memory Allocator, where a `VmaAllocation` object has no connection +with the buffer or image that was created with it.) +Thus, it is important to manage the resource reference counter properly. + +The simplest use case is shown in the code snippet above. +When only D3D12MA::Allocation object is obtained from a function call like D3D12MA::Allocator::CreateResource, +it remembers the `ID3D12Resource` that was created with it and holds a reference to it. +The resource can be obtained by calling `allocation->GetResource()`, which doesn't increment the resource +reference counter. +Calling `allocation->Release()` will decrease the resource reference counter, which is = 1 in this case, +so the resource will be released. + +Second option is to retrieve a pointer to the resource along with D3D12MA::Allocation. +Last parameters of the resource creation function can be used for this purpose. + +\code +D3D12MA::Allocation* allocation; +ID3D12Resource* resource; +HRESULT hr = allocator->CreateResource( + &allocationDesc, + &resourceDesc, + D3D12_RESOURCE_STATE_COPY_DEST, + NULL, + &allocation, + IID_PPV_ARGS(&resource)); + +// Use resource... +\endcode + +In this case, returned pointer `resource` is equal to `allocation->GetResource()`, +but the creation function additionally increases resource reference counter for the purpose of returning it from this call +(it actually calls `QueryInterface` internally), so the resource will have the counter = 2. +The resource then need to be released along with the allocation, in this particular order, +to make sure the resource is destroyed before its memory heap can potentially be freed. + +\code +resource->Release(); +allocation->Release(); +\endcode + +More advanced use cases are possible when we consider that an D3D12MA::Allocation object can just hold +a reference to any resource. +It can be changed by calling D3D12MA::Allocation::SetResource. This function +releases the old resource and calls `AddRef` on the new one. + +Special care must be taken when performing defragmentation. +The new resource created at the destination place should be set as `pass.pMoves[i].pDstTmpAllocation->SetResource(newRes)`, +but it is moved to the source allocation at end of the defragmentation pass, +while the old resource accessible through `pass.pMoves[i].pSrcAllocation->GetResource()` is then released. +For more information, see documentation chapter \ref defragmentation. + + \section quick_start_mapping_memory Mapping memory The process of getting regular CPU-side pointer to the memory of a resource in @@ -1892,9 +2062,20 @@ You can perform the defragmentation incrementally to limit the number of allocat in each pass, e.g. to call it in sync with render frames and not to experience too big hitches. See members: D3D12MA::DEFRAGMENTATION_DESC::MaxBytesPerPass, D3D12MA::DEFRAGMENTATION_DESC::MaxAllocationsPerPass. -It is also safe to perform the defragmentation asynchronously to render frames and other Direct3D 12 and %D3D12MA +Thread safety: +It is safe to perform the defragmentation asynchronously to render frames and other Direct3D 12 and %D3D12MA usage, possibly from multiple threads, with the exception that allocations returned in D3D12MA::DEFRAGMENTATION_PASS_MOVE_INFO::pMoves shouldn't be released until the defragmentation pass is ended. +During the call to D3D12MA::DefragmentationContext::BeginPass(), any operations on the memory pool +affected by the defragmentation are blocked by a mutex. + +What it means in practice is that you shouldn't free any allocations from the defragmented pool +since the moment a call to `BeginPass` begins. Otherwise, a thread performing the `allocation->Release()` +would block for the time `BeginPass` executes and then free the allocation when it finishes, while the allocation +could have ended up on the list of allocations to move. +A solution to freeing allocations during defragmentation is to find such allocation on the list +`pass.pMoves[i]` and set its operation to D3D12MA::DEFRAGMENTATION_MOVE_OPERATION_DESTROY instead of +calling `allocation->Release()`, or simply deferring the release to the time after defragmentation finished. Mapping is out of scope of this library and so it is not preserved after an allocation is moved during defragmentation. You need to map the new resource yourself if needed. diff --git a/3rdparty/d3d12memalloc/src/D3D12MemAlloc.cpp b/3rdparty/d3d12memalloc/src/D3D12MemAlloc.cpp index 896cceeb61..21c178269f 100644 --- a/3rdparty/d3d12memalloc/src/D3D12MemAlloc.cpp +++ b/3rdparty/d3d12memalloc/src/D3D12MemAlloc.cpp @@ -27,6 +27,7 @@ #include #include #include +#include #include // for _aligned_malloc, _aligned_free #ifndef _WIN32 #include @@ -106,6 +107,16 @@ especially to test compatibility with D3D12_RESOURCE_HEAP_TIER_1 on modern GPUs. #define D3D12MA_DEFAULT_BLOCK_SIZE (64ull * 1024 * 1024) #endif +#ifndef D3D12MA_DEBUG_LOG + #define D3D12MA_DEBUG_LOG(format, ...) + /* + #define D3D12MA_DEBUG_LOG(format, ...) do { \ + wprintf(format, __VA_ARGS__); \ + wprintf(L"\n"); \ + } while(false) + */ +#endif + #endif // _D3D12MA_CONFIGURATION //////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////// @@ -117,6 +128,10 @@ especially to test compatibility with D3D12_RESOURCE_HEAP_TIER_1 on modern GPUs. #define D3D12MA_IID_PPV_ARGS(ppType) __uuidof(**(ppType)), reinterpret_cast(ppType) +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + #define D3D12MA_CREATE_NOT_ZEROED_AVAILABLE 1 +#endif + namespace D3D12MA { static constexpr UINT HEAP_TYPE_COUNT = 4; @@ -133,10 +148,18 @@ static const WCHAR* const HeapTypeNames[] = L"READBACK", L"CUSTOM", }; +static const WCHAR* const StandardHeapTypeNames[] = +{ + L"DEFAULT", + L"UPLOAD", + L"READBACK", +}; static const D3D12_HEAP_FLAGS RESOURCE_CLASS_HEAP_FLAGS = D3D12_HEAP_FLAG_DENY_BUFFERS | D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES | D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES; +static const D3D12_RESIDENCY_PRIORITY D3D12_RESIDENCY_PRIORITY_NONE = D3D12_RESIDENCY_PRIORITY(0); + #ifndef _D3D12MA_ENUM_DECLARATIONS // Local copy of this enum, as it is provided only by , so it may not be available. @@ -377,7 +400,15 @@ template static T RoundDiv(T x, T y) { return (x + (y / (T)2)) / y; } template static T DivideRoundingUp(T x, T y) { return (x + y - 1) / y; } - + +static WCHAR HexDigitToChar(UINT8 digit) +{ + if(digit < 10) + return L'0' + digit; + else + return L'A' + (digit - 10); +} + /* Performs binary search and returns iterator to first element that is greater or equal to `key`, according to comparison `cmp`. @@ -427,26 +458,29 @@ static IterT BinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& va return end; } -static UINT HeapTypeToIndex(D3D12_HEAP_TYPE type) +static UINT StandardHeapTypeToIndex(D3D12_HEAP_TYPE type) { switch (type) { case D3D12_HEAP_TYPE_DEFAULT: return 0; case D3D12_HEAP_TYPE_UPLOAD: return 1; case D3D12_HEAP_TYPE_READBACK: return 2; - case D3D12_HEAP_TYPE_CUSTOM: return 3; default: D3D12MA_ASSERT(0); return UINT_MAX; } } -static D3D12_HEAP_TYPE IndexToHeapType(UINT heapTypeIndex) +static D3D12_HEAP_TYPE IndexToStandardHeapType(UINT heapTypeIndex) { - D3D12MA_ASSERT(heapTypeIndex < 4); - // D3D12_HEAP_TYPE_DEFAULT starts at 1. - return (D3D12_HEAP_TYPE)(heapTypeIndex + 1); + switch(heapTypeIndex) + { + case 0: return D3D12_HEAP_TYPE_DEFAULT; + case 1: return D3D12_HEAP_TYPE_UPLOAD; + case 2: return D3D12_HEAP_TYPE_READBACK; + default: D3D12MA_ASSERT(0); return D3D12_HEAP_TYPE_CUSTOM; + } } -static UINT64 HeapFlagsToAlignment(D3D12_HEAP_FLAGS flags) +static UINT64 HeapFlagsToAlignment(D3D12_HEAP_FLAGS flags, bool denyMsaaTextures) { /* Documentation of D3D12_HEAP_DESC structure says: @@ -459,6 +493,9 @@ static UINT64 HeapFlagsToAlignment(D3D12_HEAP_FLAGS flags) https://docs.microsoft.com/en-us/windows/desktop/api/d3d12/ns-d3d12-d3d12_heap_desc */ + if (denyMsaaTextures) + return D3D12_DEFAULT_RESOURCE_PLACEMENT_ALIGNMENT; + const D3D12_HEAP_FLAGS denyAllTexturesFlags = D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES | D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES; const bool canContainAnyTextures = @@ -888,6 +925,7 @@ class Vector public: using value_type = T; using iterator = T*; + using const_iterator = const T*; // allocationCallbacks externally owned, must outlive this object. Vector(const ALLOCATION_CALLBACKS& allocationCallbacks); @@ -904,13 +942,10 @@ public: iterator begin() { return m_pArray; } iterator end() { return m_pArray + m_Count; } - iterator rend() { return begin() - 1; } - iterator rbegin() { return end() - 1; } - - const iterator cbegin() const { return m_pArray; } - const iterator cend() const { return m_pArray + m_Count; } - const iterator crbegin() const { return cend() - 1; } - const iterator crend() const { return cbegin() - 1; } + const_iterator cbegin() const { return m_pArray; } + const_iterator cend() const { return m_pArray + m_Count; } + const_iterator begin() const { return cbegin(); } + const_iterator end() const { return cend(); } void push_front(const T& src) { insert(0, src); } void push_back(const T& src); @@ -1177,6 +1212,7 @@ public: void AddNewLine() { Add(L'\n'); } void AddNumber(UINT num); void AddNumber(UINT64 num); + void AddPointer(const void* ptr); private: Vector m_Data; @@ -1221,6 +1257,22 @@ void StringBuilder::AddNumber(UINT64 num) while (num); Add(p); } + +void StringBuilder::AddPointer(const void* ptr) +{ + WCHAR buf[21]; + uintptr_t num = (uintptr_t)ptr; + buf[20] = L'\0'; + WCHAR *p = &buf[20]; + do + { + *--p = HexDigitToChar((UINT8)(num & 0xF)); + num >>= 4; + } + while (num); + Add(p); +} + #endif // _D3D12MA_STRING_BUILDER_FUNCTIONS #endif // _D3D12MA_STRING_BUILDER @@ -1264,6 +1316,7 @@ public: // Posts next part of an open string. The number is converted to decimal characters. void ContinueString(UINT num); void ContinueString(UINT64 num); + void ContinueString_Pointer(const void* ptr); // Posts next part of an open string. Pointer value is converted to characters // using "%p" formatting - shown as hexadecimal number, e.g.: 000000081276Ad00 // void ContinueString_Pointer(const void* ptr); @@ -1449,6 +1502,12 @@ void JsonWriter::ContinueString(UINT64 num) m_SB.AddNumber(num); } +void JsonWriter::ContinueString_Pointer(const void* ptr) +{ + D3D12MA_ASSERT(m_InsideString); + m_SB.AddPointer(ptr); +} + void JsonWriter::EndString(LPCWSTR pStr) { D3D12MA_ASSERT(m_InsideString); @@ -1511,61 +1570,63 @@ void JsonWriter::AddAllocationToObject(const Allocation& alloc) break; default: D3D12MA_ASSERT(0); break; } + WriteString(L"Size"); WriteNumber(alloc.GetSize()); + WriteString(L"Usage"); + WriteNumber((UINT)alloc.m_PackedData.GetResourceFlags()); + + void* privateData = alloc.GetPrivateData(); + if (privateData) + { + WriteString(L"CustomData"); + BeginString(); + ContinueString_Pointer(privateData); + EndString(); + } + LPCWSTR name = alloc.GetName(); if (name != NULL) { WriteString(L"Name"); WriteString(name); } - if (alloc.m_PackedData.GetResourceFlags()) - { - WriteString(L"Flags"); - WriteNumber((UINT)alloc.m_PackedData.GetResourceFlags()); - } if (alloc.m_PackedData.GetTextureLayout()) { WriteString(L"Layout"); WriteNumber((UINT)alloc.m_PackedData.GetTextureLayout()); } - if (alloc.m_CreationFrameIndex) - { - WriteString(L"CreationFrameIndex"); - WriteNumber(alloc.m_CreationFrameIndex); - } } void JsonWriter::AddDetailedStatisticsInfoObject(const DetailedStatistics& stats) { BeginObject(); + WriteString(L"BlockCount"); WriteNumber(stats.Stats.BlockCount); - WriteString(L"AllocationCount"); - WriteNumber(stats.Stats.AllocationCount); - WriteString(L"UnusedRangeCount"); - WriteNumber(stats.UnusedRangeCount); WriteString(L"BlockBytes"); WriteNumber(stats.Stats.BlockBytes); + WriteString(L"AllocationCount"); + WriteNumber(stats.Stats.AllocationCount); WriteString(L"AllocationBytes"); WriteNumber(stats.Stats.AllocationBytes); + WriteString(L"UnusedRangeCount"); + WriteNumber(stats.UnusedRangeCount); - WriteString(L"AllocationSize"); - BeginObject(true); - WriteString(L"Min"); - WriteNumber(stats.AllocationSizeMin); - WriteString(L"Max"); - WriteNumber(stats.AllocationSizeMax); - EndObject(); - - WriteString(L"UnusedRangeSize"); - BeginObject(true); - WriteString(L"Min"); - WriteNumber(stats.UnusedRangeSizeMin); - WriteString(L"Max"); - WriteNumber(stats.UnusedRangeSizeMax); - EndObject(); - + if (stats.Stats.AllocationCount > 1) + { + WriteString(L"AllocationSizeMin"); + WriteNumber(stats.AllocationSizeMin); + WriteString(L"AllocationSizeMax"); + WriteNumber(stats.AllocationSizeMax); + } + if (stats.UnusedRangeCount > 1) + { + WriteString(L"UnusedRangeSizeMin"); + WriteNumber(stats.UnusedRangeSizeMin); + WriteString(L"UnusedRangeSizeMax"); + WriteNumber(stats.UnusedRangeSizeMax); + } EndObject(); } @@ -2806,7 +2867,7 @@ struct AllocationRequest UINT64 sumFreeSize; // Sum size of free items that overlap with proposed allocation. UINT64 sumItemSize; // Sum size of items to make lost that overlap with proposed allocation. SuballocationList::iterator item; - BOOL zeroInitialized; + BOOL zeroInitialized = FALSE; // TODO Implement proper handling in TLSF and Linear, using ZeroInitializedRange class. }; #endif // _D3D12MA_ALLOCATION_REQUEST @@ -2930,11 +2991,13 @@ public: virtual void AddStatistics(Statistics& inoutStats) const = 0; virtual void AddDetailedStatistics(DetailedStatistics& inoutStats) const = 0; virtual void WriteAllocationInfoToJson(JsonWriter& json) const = 0; + virtual void DebugLogAllAllocations() const = 0; protected: const ALLOCATION_CALLBACKS* GetAllocs() const { return m_pAllocationCallbacks; } UINT64 GetDebugMargin() const { return IsVirtual() ? 0 : D3D12MA_DEBUG_MARGIN; } + void DebugLogAllocation(UINT64 offset, UINT64 size, void* privateData) const; void PrintDetailedMap_Begin(JsonWriter& json, UINT64 unusedBytes, size_t allocationCount, @@ -2962,11 +3025,28 @@ BlockMetadata::BlockMetadata(const ALLOCATION_CALLBACKS* allocationCallbacks, bo D3D12MA_ASSERT(allocationCallbacks); } +void BlockMetadata::DebugLogAllocation(UINT64 offset, UINT64 size, void* privateData) const +{ + if (IsVirtual()) + { + D3D12MA_DEBUG_LOG(L"UNFREED VIRTUAL ALLOCATION; Offset: %llu; Size: %llu; PrivateData: %p", offset, size, privateData); + } + else + { + D3D12MA_ASSERT(privateData != NULL); + Allocation* allocation = reinterpret_cast(privateData); + + privateData = allocation->GetPrivateData(); + LPCWSTR name = allocation->GetName(); + + D3D12MA_DEBUG_LOG(L"UNFREED ALLOCATION; Offset: %llu; Size: %llu; PrivateData: %p; Name: %s", + offset, size, privateData, name ? name : L"D3D12MA_Empty"); + } +} + void BlockMetadata::PrintDetailedMap_Begin(JsonWriter& json, UINT64 unusedBytes, size_t allocationCount, size_t unusedRangeCount) const { - json.BeginObject(); - json.WriteString(L"TotalBytes"); json.WriteNumber(GetSize()); @@ -2974,10 +3054,10 @@ void BlockMetadata::PrintDetailedMap_Begin(JsonWriter& json, json.WriteNumber(unusedBytes); json.WriteString(L"Allocations"); - json.WriteNumber(allocationCount); + json.WriteNumber((UINT64)allocationCount); json.WriteString(L"UnusedRanges"); - json.WriteNumber(unusedRangeCount); + json.WriteNumber((UINT64)unusedRangeCount); json.WriteString(L"Suballocations"); json.BeginArray(); @@ -2993,13 +3073,11 @@ void BlockMetadata::PrintDetailedMap_Allocation(JsonWriter& json, if (IsVirtual()) { - json.WriteString(L"Type"); - json.WriteString(L"ALLOCATION"); json.WriteString(L"Size"); json.WriteNumber(size); if (privateData) { - json.WriteString(L"PrivateData"); + json.WriteString(L"CustomData"); json.WriteNumber((uintptr_t)privateData); } } @@ -3032,7 +3110,6 @@ void BlockMetadata::PrintDetailedMap_UnusedRange(JsonWriter& json, void BlockMetadata::PrintDetailedMap_End(JsonWriter& json) const { json.EndArray(); - json.EndObject(); } #endif // _D3D12MA_BLOCK_METADATA_FUNCTIONS #endif // _D3D12MA_BLOCK_METADATA @@ -3682,6 +3759,7 @@ public: void AddStatistics(Statistics& inoutStats) const override; void AddDetailedStatistics(DetailedStatistics& inoutStats) const override; void WriteAllocationInfoToJson(JsonWriter& json) const override; + void DebugLogAllAllocations() const override; private: /* @@ -3820,7 +3898,7 @@ bool BlockMetadata_Linear::Validate() const { if (!IsVirtual()) { - D3D12MA_VALIDATE((UINT64)alloc->GetAllocHandle() == suballoc.offset); + D3D12MA_VALIDATE(GetAllocationOffset(alloc->GetAllocHandle()) == suballoc.offset); D3D12MA_VALIDATE(alloc->GetSize() == suballoc.size); } sumUsedSize += suballoc.size; @@ -3862,7 +3940,7 @@ bool BlockMetadata_Linear::Validate() const { if (!IsVirtual()) { - D3D12MA_VALIDATE((UINT64)alloc->GetAllocHandle() == suballoc.offset); + D3D12MA_VALIDATE(GetAllocationOffset(alloc->GetAllocHandle()) == suballoc.offset); D3D12MA_VALIDATE(alloc->GetSize() == suballoc.size); } sumUsedSize += suballoc.size; @@ -3896,7 +3974,7 @@ bool BlockMetadata_Linear::Validate() const { if (!IsVirtual()) { - D3D12MA_VALIDATE((UINT64)alloc->GetAllocHandle() == suballoc.offset); + D3D12MA_VALIDATE(GetAllocationOffset(alloc->GetAllocHandle()) == suballoc.offset); D3D12MA_VALIDATE(alloc->GetSize() == suballoc.size); } sumUsedSize += suballoc.size; @@ -4638,6 +4716,19 @@ void BlockMetadata_Linear::WriteAllocationInfoToJson(JsonWriter& json) const PrintDetailedMap_End(json); } +void BlockMetadata_Linear::DebugLogAllAllocations() const +{ + const SuballocationVectorType& suballocations1st = AccessSuballocations1st(); + for (auto it = suballocations1st.begin() + m_1stNullItemsBeginCount; it != suballocations1st.end(); ++it) + if (it->type != SUBALLOCATION_TYPE_FREE) + DebugLogAllocation(it->offset, it->size, it->privateData); + + const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd(); + for (auto it = suballocations2nd.begin(); it != suballocations2nd.end(); ++it) + if (it->type != SUBALLOCATION_TYPE_FREE) + DebugLogAllocation(it->offset, it->size, it->privateData); +} + Suballocation& BlockMetadata_Linear::FindSuballocation(UINT64 offset) const { const SuballocationVectorType& suballocations1st = AccessSuballocations1st(); @@ -4649,31 +4740,31 @@ Suballocation& BlockMetadata_Linear::FindSuballocation(UINT64 offset) const // Item from the 1st vector. { - const SuballocationVectorType::iterator it = BinaryFindSorted( - suballocations1st.cbegin() + m_1stNullItemsBeginCount, - suballocations1st.cend(), + const SuballocationVectorType::const_iterator it = BinaryFindSorted( + suballocations1st.begin() + m_1stNullItemsBeginCount, + suballocations1st.end(), refSuballoc, SuballocationOffsetLess()); - if (it != suballocations1st.cend()) + if (it != suballocations1st.end()) { - return *it; + return const_cast(*it); } } if (m_2ndVectorMode != SECOND_VECTOR_EMPTY) { // Rest of members stays uninitialized intentionally for better performance. - const SuballocationVectorType::iterator it = m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER ? - BinaryFindSorted(suballocations2nd.cbegin(), suballocations2nd.cend(), refSuballoc, SuballocationOffsetLess()) : - BinaryFindSorted(suballocations2nd.cbegin(), suballocations2nd.cend(), refSuballoc, SuballocationOffsetGreater()); - if (it != suballocations2nd.cend()) + const SuballocationVectorType::const_iterator it = m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER ? + BinaryFindSorted(suballocations2nd.begin(), suballocations2nd.end(), refSuballoc, SuballocationOffsetLess()) : + BinaryFindSorted(suballocations2nd.begin(), suballocations2nd.end(), refSuballoc, SuballocationOffsetGreater()); + if (it != suballocations2nd.end()) { - return *it; + return const_cast(*it); } } D3D12MA_ASSERT(0 && "Allocation not found in linear allocator!"); - return *suballocations1st.crbegin(); // Should never occur. + return const_cast(suballocations1st.back()); // Should never occur. } bool BlockMetadata_Linear::ShouldCompact1st() const @@ -4964,6 +5055,7 @@ public: void AddStatistics(Statistics& inoutStats) const override; void AddDetailedStatistics(DetailedStatistics& inoutStats) const override; void WriteAllocationInfoToJson(JsonWriter& json) const override; + void DebugLogAllAllocations() const override; private: // According to original paper it should be preferable 4 or 5: @@ -5195,7 +5287,7 @@ bool BlockMetadata_TLSF::CreateAllocationRequest( // Round up to the next block UINT64 sizeForNextList = allocSize; - UINT64 smallSizeStep = SMALL_BUFFER_SIZE / (IsVirtual() ? 1 << SECOND_LEVEL_INDEX : 4); + UINT16 smallSizeStep = SMALL_BUFFER_SIZE / (IsVirtual() ? 1 << SECOND_LEVEL_INDEX : 4); if (allocSize > SMALL_BUFFER_SIZE) { sizeForNextList += (1ULL << (BitScanMSB(allocSize) - SECOND_LEVEL_INDEX)); @@ -5603,11 +5695,22 @@ void BlockMetadata_TLSF::WriteAllocationInfoToJson(JsonWriter& json) const if (block->IsFree()) PrintDetailedMap_UnusedRange(json, block->offset, block->size); else - PrintDetailedMap_Allocation(json, block->size, block->offset, block->PrivateData()); + PrintDetailedMap_Allocation(json, block->offset, block->size, block->PrivateData()); } PrintDetailedMap_End(json); } +void BlockMetadata_TLSF::DebugLogAllAllocations() const +{ + for (Block* block = m_NullBlock->prevPhysical; block != NULL; block = block->prevPhysical) + { + if (!block->IsFree()) + { + DebugLogAllocation(block->offset, block->size, block->PrivateData()); + } + } +} + UINT8 BlockMetadata_TLSF::SizeToMemoryClass(UINT64 size) const { if (size > SMALL_BUFFER_SIZE) @@ -5796,7 +5899,7 @@ protected: const UINT64 m_Size; const UINT m_Id; - HRESULT Init(ID3D12ProtectedResourceSession* pProtectedSession); + HRESULT Init(ID3D12ProtectedResourceSession* pProtectedSession, bool denyMsaaTextures); private: ID3D12Heap* m_Heap = NULL; @@ -5828,7 +5931,7 @@ public: BlockVector* GetBlockVector() const { return m_BlockVector; } // 'algorithm' should be one of the *_ALGORITHM_* flags in enums POOL_FLAGS or VIRTUAL_BLOCK_FLAGS - HRESULT Init(UINT32 algorithm, ID3D12ProtectedResourceSession* pProtectedSession); + HRESULT Init(UINT32 algorithm, ID3D12ProtectedResourceSession* pProtectedSession, bool denyMsaaTextures); // Validates all data structures inside this object. If not valid, returns false. bool Validate() const; @@ -5911,11 +6014,144 @@ struct CommittedAllocationParameters D3D12_HEAP_PROPERTIES m_HeapProperties = {}; D3D12_HEAP_FLAGS m_HeapFlags = D3D12_HEAP_FLAG_NONE; ID3D12ProtectedResourceSession* m_ProtectedSession = NULL; + bool m_CanAlias = false; + D3D12_RESIDENCY_PRIORITY m_ResidencyPriority = D3D12_RESIDENCY_PRIORITY_NONE; bool IsValid() const { return m_List != NULL; } }; #endif // _D3D12M_COMMITTED_ALLOCATION_PARAMETERS +// Simple variant data structure to hold all possible variations of ID3D12Device*::CreateCommittedResource* and ID3D12Device*::CreatePlacedResource* arguments +struct CREATE_RESOURCE_PARAMS +{ + CREATE_RESOURCE_PARAMS() = delete; + CREATE_RESOURCE_PARAMS( + const D3D12_RESOURCE_DESC* pResourceDesc, + D3D12_RESOURCE_STATES InitialResourceState, + const D3D12_CLEAR_VALUE* pOptimizedClearValue) + : Variant(VARIANT_WITH_STATE) + , pResourceDesc(pResourceDesc) + , InitialResourceState(InitialResourceState) + , pOptimizedClearValue(pOptimizedClearValue) + { + } +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + CREATE_RESOURCE_PARAMS( + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_RESOURCE_STATES InitialResourceState, + const D3D12_CLEAR_VALUE* pOptimizedClearValue) + : Variant(VARIANT_WITH_STATE_AND_DESC1) + , pResourceDesc1(pResourceDesc) + , InitialResourceState(InitialResourceState) + , pOptimizedClearValue(pOptimizedClearValue) + { + } +#endif +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + CREATE_RESOURCE_PARAMS( + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_BARRIER_LAYOUT InitialLayout, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + UINT32 NumCastableFormats, + DXGI_FORMAT* pCastableFormats) + : Variant(VARIANT_WITH_LAYOUT) + , pResourceDesc1(pResourceDesc) + , InitialLayout(InitialLayout) + , pOptimizedClearValue(pOptimizedClearValue) + , NumCastableFormats(NumCastableFormats) + , pCastableFormats(pCastableFormats) + { + } +#endif + + enum VARIANT + { + VARIANT_INVALID = 0, + VARIANT_WITH_STATE, + VARIANT_WITH_STATE_AND_DESC1, + VARIANT_WITH_LAYOUT + }; + + VARIANT Variant = VARIANT_INVALID; + + const D3D12_RESOURCE_DESC* GetResourceDesc() const + { + D3D12MA_ASSERT(Variant == VARIANT_WITH_STATE); + return pResourceDesc; + } + const D3D12_RESOURCE_DESC*& AccessResourceDesc() + { + D3D12MA_ASSERT(Variant == VARIANT_WITH_STATE); + return pResourceDesc; + } + const D3D12_RESOURCE_DESC* GetBaseResourceDesc() const + { + // D3D12_RESOURCE_DESC1 can be cast to D3D12_RESOURCE_DESC by discarding the new members at the end. + return pResourceDesc; + } + D3D12_RESOURCE_STATES GetInitialResourceState() const + { + D3D12MA_ASSERT(Variant < VARIANT_WITH_LAYOUT); + return InitialResourceState; + } + const D3D12_CLEAR_VALUE* GetOptimizedClearValue() const + { + return pOptimizedClearValue; + } + +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + const D3D12_RESOURCE_DESC1* GetResourceDesc1() const + { + D3D12MA_ASSERT(Variant >= VARIANT_WITH_STATE_AND_DESC1); + return pResourceDesc1; + } + const D3D12_RESOURCE_DESC1*& AccessResourceDesc1() + { + D3D12MA_ASSERT(Variant >= VARIANT_WITH_STATE_AND_DESC1); + return pResourceDesc1; + } +#endif + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + D3D12_BARRIER_LAYOUT GetInitialLayout() const + { + D3D12MA_ASSERT(Variant >= VARIANT_WITH_LAYOUT); + return InitialLayout; + } + UINT32 GetNumCastableFormats() const + { + D3D12MA_ASSERT(Variant >= VARIANT_WITH_LAYOUT); + return NumCastableFormats; + } + DXGI_FORMAT* GetCastableFormats() const + { + D3D12MA_ASSERT(Variant >= VARIANT_WITH_LAYOUT); + return pCastableFormats; + } +#endif + +private: + union + { + const D3D12_RESOURCE_DESC* pResourceDesc; +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + const D3D12_RESOURCE_DESC1* pResourceDesc1; +#endif + }; + union + { + D3D12_RESOURCE_STATES InitialResourceState; +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + D3D12_BARRIER_LAYOUT InitialLayout; +#endif + }; + const D3D12_CLEAR_VALUE* pOptimizedClearValue; +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + UINT32 NumCastableFormats; + DXGI_FORMAT* pCastableFormats; +#endif +}; + #ifndef _D3D12MA_BLOCK_VECTOR /* Sequence of NormalBlock. Represents memory blocks allocated for a specific @@ -5938,12 +6174,17 @@ public: bool explicitBlockSize, UINT64 minAllocationAlignment, UINT32 algorithm, - ID3D12ProtectedResourceSession* pProtectedSession); + bool denyMsaaTextures, + ID3D12ProtectedResourceSession* pProtectedSession, + D3D12_RESIDENCY_PRIORITY residencyPriority); ~BlockVector(); + D3D12_RESIDENCY_PRIORITY GetResidencyPriority() const { return m_ResidencyPriority; } const D3D12_HEAP_PROPERTIES& GetHeapProperties() const { return m_HeapProps; } + D3D12_HEAP_FLAGS GetHeapFlags() const { return m_HeapFlags; } UINT64 GetPreferredBlockSize() const { return m_PreferredBlockSize; } UINT32 GetAlgorithm() const { return m_Algorithm; } + bool DeniesMsaaTextures() const { return m_DenyMsaaTextures; } // To be used only while the m_Mutex is locked. Used during defragmentation. size_t GetBlockCount() const { return m_Blocks.size(); } // To be used only while the m_Mutex is locked. Used during defragmentation. @@ -5966,26 +6207,11 @@ public: UINT64 size, UINT64 alignment, const ALLOCATION_DESC& allocDesc, - const D3D12_RESOURCE_DESC& resourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE *pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource); -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ - HRESULT CreateResource2( - UINT64 size, - UINT64 alignment, - const ALLOCATION_DESC& allocDesc, - const D3D12_RESOURCE_DESC1& resourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE *pOptimizedClearValue, - Allocation** ppAllocation, - REFIID riidResource, - void** ppvResource); -#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ - void AddStatistics(Statistics& inoutStats); void AddDetailedStatistics(DetailedStatistics& inoutStats); @@ -6001,7 +6227,9 @@ private: const bool m_ExplicitBlockSize; const UINT64 m_MinAllocationAlignment; const UINT32 m_Algorithm; + const bool m_DenyMsaaTextures; ID3D12ProtectedResourceSession* const m_ProtectedSession; + const D3D12_RESIDENCY_PRIORITY m_ResidencyPriority; /* There can be at most one allocation that is completely empty - a hysteresis to avoid pessimistic case of alternating creation and destruction of a ID3D12Heap. */ @@ -6082,7 +6310,7 @@ private: D3D12MA_ATOMIC_UINT64 m_BlockBytes[DXGI_MEMORY_SEGMENT_GROUP_COUNT] = {}; D3D12MA_ATOMIC_UINT64 m_AllocationBytes[DXGI_MEMORY_SEGMENT_GROUP_COUNT] = {}; - D3D12MA_ATOMIC_UINT32 m_OperationsSinceBudgetFetch{0}; + D3D12MA_ATOMIC_UINT32 m_OperationsSinceBudgetFetch = {0}; D3D12MA_RW_MUTEX m_BudgetMutex; UINT64 m_D3D12Usage[DXGI_MEMORY_SEGMENT_GROUP_COUNT] = {}; UINT64 m_D3D12Budget[DXGI_MEMORY_SEGMENT_GROUP_COUNT] = {}; @@ -6317,13 +6545,16 @@ class AllocatorPimpl friend class Allocator; friend class Pool; public: - std::atomic_uint32_t m_RefCount{1}; + std::atomic_uint32_t m_RefCount = {1}; CurrentBudgetData m_Budget; AllocatorPimpl(const ALLOCATION_CALLBACKS& allocationCallbacks, const ALLOCATOR_DESC& desc); ~AllocatorPimpl(); ID3D12Device* GetDevice() const { return m_Device; } +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ + ID3D12Device1* GetDevice1() const { return m_Device1; } +#endif #ifdef __ID3D12Device4_INTERFACE_DEFINED__ ID3D12Device4* GetDevice4() const { return m_Device4; } #endif @@ -6364,32 +6595,24 @@ public: UINT HeapPropertiesToMemorySegmentGroup(const D3D12_HEAP_PROPERTIES& heapProps) const; UINT64 GetMemoryCapacity(UINT memorySegmentGroup) const; - HRESULT CreateResource( - const ALLOCATION_DESC* pAllocDesc, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE *pOptimizedClearValue, - Allocation** ppAllocation, + HRESULT CreatePlacedResourceWrap( + ID3D12Heap *pHeap, + UINT64 HeapOffset, + const CREATE_RESOURCE_PARAMS& createParams, REFIID riidResource, void** ppvResource); -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ - HRESULT CreateResource2( + HRESULT CreateResource( const ALLOCATION_DESC* pAllocDesc, - const D3D12_RESOURCE_DESC1* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE *pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource); -#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ HRESULT CreateAliasingResource( Allocation* pAllocation, UINT64 AllocationLocalOffset, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE *pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, REFIID riidResource, void** ppvResource); @@ -6408,14 +6631,16 @@ public: // Allocation object must be deleted externally afterwards. void FreeHeapMemory(Allocation* allocation); - void SetCurrentFrameIndex(UINT frameIndex); + void SetResidencyPriority(ID3D12Pageable* obj, D3D12_RESIDENCY_PRIORITY priority) const; - void CalculateStatistics(TotalStatistics& outStats); + void SetCurrentFrameIndex(UINT frameIndex); + // For more deailed stats use outCustomHeaps to access statistics divided into L0 and L1 group + void CalculateStatistics(TotalStatistics& outStats, DetailedStatistics outCustomHeaps[2] = NULL); void GetBudget(Budget* outLocalBudget, Budget* outNonLocalBudget); void GetBudgetForHeapType(Budget& outBudget, D3D12_HEAP_TYPE heapType); - void BuildStatsString(WCHAR** ppStatsString, BOOL DetailedMap); + void BuildStatsString(WCHAR** ppStatsString, BOOL detailedMap); void FreeStatsString(WCHAR* pStatsString); private: @@ -6423,12 +6648,20 @@ private: const bool m_UseMutex; const bool m_AlwaysCommitted; + const bool m_MsaaAlwaysCommitted; + bool m_DefaultPoolsNotZeroed = false; ID3D12Device* m_Device; // AddRef +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ + ID3D12Device1* m_Device1 = NULL; // AddRef, optional +#endif #ifdef __ID3D12Device4_INTERFACE_DEFINED__ ID3D12Device4* m_Device4 = NULL; // AddRef, optional #endif #ifdef __ID3D12Device8_INTERFACE_DEFINED__ ID3D12Device8* m_Device8 = NULL; // AddRef, optional +#endif +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + ID3D12Device10* m_Device10 = NULL; // AddRef, optional #endif IDXGIAdapter* m_Adapter; // AddRef #if D3D12MA_DXGI_1_4 @@ -6460,19 +6693,9 @@ private: HRESULT AllocateCommittedResource( const CommittedAllocationParameters& committedAllocParams, UINT64 resourceSize, bool withinBudget, void* pPrivateData, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE *pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource); -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ - HRESULT AllocateCommittedResource2( - const CommittedAllocationParameters& committedAllocParams, - UINT64 resourceSize, bool withinBudget, void* pPrivateData, - const D3D12_RESOURCE_DESC1* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE *pOptimizedClearValue, - Allocation** ppAllocation, REFIID riidResource, void** ppvResource); -#endif - // Allocates and registers new heap without any resources placed in it, as dedicated allocation. // Creates and returns Allocation object. HRESULT AllocateHeap( @@ -6514,6 +6737,7 @@ private: AllocatorPimpl::AllocatorPimpl(const ALLOCATION_CALLBACKS& allocationCallbacks, const ALLOCATOR_DESC& desc) : m_UseMutex((desc.Flags & ALLOCATOR_FLAG_SINGLETHREADED) == 0), m_AlwaysCommitted((desc.Flags & ALLOCATOR_FLAG_ALWAYS_COMMITTED) != 0), + m_MsaaAlwaysCommitted((desc.Flags & ALLOCATOR_FLAG_MSAA_TEXTURES_ALWAYS_COMMITTED) != 0), m_Device(desc.pDevice), m_Adapter(desc.pAdapter), m_PreferredBlockSize(desc.PreferredBlockSize != 0 ? desc.PreferredBlockSize : D3D12MA_DEFAULT_BLOCK_SIZE), @@ -6532,7 +6756,7 @@ AllocatorPimpl::AllocatorPimpl(const ALLOCATION_CALLBACKS& allocationCallbacks, { m_CommittedAllocations[i].Init( m_UseMutex, - (D3D12_HEAP_TYPE)(D3D12_HEAP_TYPE_DEFAULT + i), + IndexToStandardHeapType(i), NULL); // pool } @@ -6546,12 +6770,30 @@ HRESULT AllocatorPimpl::Init(const ALLOCATOR_DESC& desc) desc.pAdapter->QueryInterface(D3D12MA_IID_PPV_ARGS(&m_Adapter3)); #endif +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ + m_Device->QueryInterface(D3D12MA_IID_PPV_ARGS(&m_Device1)); +#endif + #ifdef __ID3D12Device4_INTERFACE_DEFINED__ m_Device->QueryInterface(D3D12MA_IID_PPV_ARGS(&m_Device4)); #endif #ifdef __ID3D12Device8_INTERFACE_DEFINED__ m_Device->QueryInterface(D3D12MA_IID_PPV_ARGS(&m_Device8)); + + if((desc.Flags & ALLOCATOR_FLAG_DEFAULT_POOLS_NOT_ZEROED) != 0) + { + D3D12_FEATURE_DATA_D3D12_OPTIONS7 options7 = {}; + if(SUCCEEDED(m_Device->CheckFeatureSupport(D3D12_FEATURE_D3D12_OPTIONS7, &options7, sizeof(options7)))) + { + // DEFAULT_POOLS_NOT_ZEROED both supported and enabled by the user. + m_DefaultPoolsNotZeroed = true; + } + } +#endif + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + m_Device->QueryInterface(D3D12MA_IID_PPV_ARGS(&m_Device10)); #endif HRESULT hr = m_Adapter->GetDesc(&m_AdapterDesc); @@ -6583,9 +6825,11 @@ HRESULT AllocatorPimpl::Init(const ALLOCATOR_DESC& desc) D3D12_HEAP_FLAGS heapFlags; CalcDefaultPoolParams(heapProps.Type, heapFlags, i); -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ - if (desc.Flags & ALLOCATOR_FLAG_DEFAULT_POOLS_NOT_ZEROED) +#if D3D12MA_CREATE_NOT_ZEROED_AVAILABLE + if(m_DefaultPoolsNotZeroed) + { heapFlags |= D3D12_HEAP_FLAG_CREATE_NOT_ZEROED; + } #endif m_BlockVectors[i] = D3D12MA_NEW(GetAllocs(), BlockVector)( @@ -6597,8 +6841,10 @@ HRESULT AllocatorPimpl::Init(const ALLOCATOR_DESC& desc) SIZE_MAX, // maxBlockCount false, // explicitBlockSize D3D12MA_DEBUG_ALIGNMENT, // minAllocationAlignment - 0, // Default algorithm - NULL); // pProtectedSession + 0, // Default algorithm, + m_MsaaAlwaysCommitted, + NULL, // pProtectedSession + D3D12_RESIDENCY_PRIORITY_NONE); // residencyPriority // No need to call m_pBlockVectors[i]->CreateMinBlocks here, becase minBlockCount is 0. } @@ -6611,12 +6857,18 @@ HRESULT AllocatorPimpl::Init(const ALLOCATOR_DESC& desc) AllocatorPimpl::~AllocatorPimpl() { +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + SAFE_RELEASE(m_Device10); +#endif #ifdef __ID3D12Device8_INTERFACE_DEFINED__ SAFE_RELEASE(m_Device8); #endif #ifdef __ID3D12Device4_INTERFACE_DEFINED__ SAFE_RELEASE(m_Device4); #endif +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ + SAFE_RELEASE(m_Device1); +#endif #if D3D12MA_DXGI_1_4 SAFE_RELEASE(m_Adapter3); #endif @@ -6687,16 +6939,60 @@ UINT64 AllocatorPimpl::GetMemoryCapacity(UINT memorySegmentGroup) const } } +HRESULT AllocatorPimpl::CreatePlacedResourceWrap( + ID3D12Heap *pHeap, + UINT64 HeapOffset, + const CREATE_RESOURCE_PARAMS& createParams, + REFIID riidResource, + void** ppvResource) +{ +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_LAYOUT) + { + if (!m_Device10) + { + return E_NOINTERFACE; + } + return m_Device10->CreatePlacedResource2(pHeap, HeapOffset, + createParams.GetResourceDesc1(), createParams.GetInitialLayout(), + createParams.GetOptimizedClearValue(), createParams.GetNumCastableFormats(), + createParams.GetCastableFormats(), riidResource, ppvResource); + } else +#endif +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE_AND_DESC1) + { + if (!m_Device8) + { + return E_NOINTERFACE; + } + return m_Device8->CreatePlacedResource1(pHeap, HeapOffset, + createParams.GetResourceDesc1(), createParams.GetInitialResourceState(), + createParams.GetOptimizedClearValue(), riidResource, ppvResource); + } else +#endif + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE) + { + return m_Device->CreatePlacedResource(pHeap, HeapOffset, + createParams.GetResourceDesc(), createParams.GetInitialResourceState(), + createParams.GetOptimizedClearValue(), riidResource, ppvResource); + } + else + { + D3D12MA_ASSERT(0); + return E_INVALIDARG; + } +} + + HRESULT AllocatorPimpl::CreateResource( const ALLOCATION_DESC* pAllocDesc, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE* pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource) { - D3D12MA_ASSERT(pAllocDesc && pResourceDesc && ppAllocation); + D3D12MA_ASSERT(pAllocDesc && createParams.GetBaseResourceDesc() && ppAllocation); *ppAllocation = NULL; if (ppvResource) @@ -6704,17 +7000,69 @@ HRESULT AllocatorPimpl::CreateResource( *ppvResource = NULL; } - D3D12_RESOURCE_DESC finalResourceDesc = *pResourceDesc; - D3D12_RESOURCE_ALLOCATION_INFO resAllocInfo = GetResourceAllocationInfo(finalResourceDesc); + CREATE_RESOURCE_PARAMS finalCreateParams = createParams; + D3D12_RESOURCE_DESC finalResourceDesc; +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + D3D12_RESOURCE_DESC1 finalResourceDesc1; +#endif + D3D12_RESOURCE_ALLOCATION_INFO resAllocInfo; + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE) + { + finalResourceDesc = *createParams.GetResourceDesc(); + finalCreateParams.AccessResourceDesc() = &finalResourceDesc; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc); + } +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + else if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE_AND_DESC1) + { + if (!m_Device8) + { + return E_NOINTERFACE; + } + finalResourceDesc1 = *createParams.GetResourceDesc1(); + finalCreateParams.AccessResourceDesc1() = &finalResourceDesc1; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc1); + } +#endif +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + else if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_LAYOUT) + { + if (!m_Device10) + { + return E_NOINTERFACE; + } + finalResourceDesc1 = *createParams.GetResourceDesc1(); + finalCreateParams.AccessResourceDesc1() = &finalResourceDesc1; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc1); + } +#endif + else + { + D3D12MA_ASSERT(0); + return E_INVALIDARG; + } D3D12MA_ASSERT(IsPow2(resAllocInfo.Alignment)); D3D12MA_ASSERT(resAllocInfo.SizeInBytes > 0); BlockVector* blockVector = NULL; CommittedAllocationParameters committedAllocationParams = {}; bool preferCommitted = false; - HRESULT hr = CalcAllocationParams(*pAllocDesc, resAllocInfo.SizeInBytes, - pResourceDesc, - blockVector, committedAllocationParams, preferCommitted); + + HRESULT hr; +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + if (createParams.Variant >= CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE_AND_DESC1) + { + hr = CalcAllocationParams(*pAllocDesc, resAllocInfo.SizeInBytes, + createParams.GetResourceDesc1(), + blockVector, committedAllocationParams, preferCommitted); + } + else +#endif + { + hr = CalcAllocationParams(*pAllocDesc, resAllocInfo.SizeInBytes, + createParams.GetResourceDesc(), + blockVector, committedAllocationParams, preferCommitted); + } if (FAILED(hr)) return hr; @@ -6724,16 +7072,14 @@ HRESULT AllocatorPimpl::CreateResource( { hr = AllocateCommittedResource(committedAllocationParams, resAllocInfo.SizeInBytes, withinBudget, pAllocDesc->pPrivateData, - &finalResourceDesc, InitialResourceState, pOptimizedClearValue, - ppAllocation, riidResource, ppvResource); + finalCreateParams, ppAllocation, riidResource, ppvResource); if (SUCCEEDED(hr)) return hr; } if (blockVector != NULL) { hr = blockVector->CreateResource(resAllocInfo.SizeInBytes, resAllocInfo.Alignment, - *pAllocDesc, finalResourceDesc, - InitialResourceState, pOptimizedClearValue, + *pAllocDesc, finalCreateParams, ppAllocation, riidResource, ppvResource); if (SUCCEEDED(hr)) return hr; @@ -6742,83 +7088,13 @@ HRESULT AllocatorPimpl::CreateResource( { hr = AllocateCommittedResource(committedAllocationParams, resAllocInfo.SizeInBytes, withinBudget, pAllocDesc->pPrivateData, - &finalResourceDesc, InitialResourceState, pOptimizedClearValue, - ppAllocation, riidResource, ppvResource); + finalCreateParams, ppAllocation, riidResource, ppvResource); if (SUCCEEDED(hr)) return hr; } return hr; } -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ -HRESULT AllocatorPimpl::CreateResource2( - const ALLOCATION_DESC* pAllocDesc, - const D3D12_RESOURCE_DESC1* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE* pOptimizedClearValue, - Allocation** ppAllocation, - REFIID riidResource, - void** ppvResource) -{ - D3D12MA_ASSERT(pAllocDesc && pResourceDesc && ppAllocation); - - *ppAllocation = NULL; - if (ppvResource) - { - *ppvResource = NULL; - } - if (m_Device8 == NULL) - { - return E_NOINTERFACE; - } - - D3D12_RESOURCE_DESC1 finalResourceDesc = *pResourceDesc; - D3D12_RESOURCE_ALLOCATION_INFO resAllocInfo = GetResourceAllocationInfo(finalResourceDesc); - D3D12MA_ASSERT(IsPow2(resAllocInfo.Alignment)); - D3D12MA_ASSERT(resAllocInfo.SizeInBytes > 0); - - BlockVector* blockVector = NULL; - CommittedAllocationParameters committedAllocationParams = {}; - bool preferCommitted = false; - HRESULT hr = CalcAllocationParams(*pAllocDesc, resAllocInfo.SizeInBytes, - pResourceDesc, - blockVector, committedAllocationParams, preferCommitted); - if (FAILED(hr)) - return hr; - - const bool withinBudget = (pAllocDesc->Flags & ALLOCATION_FLAG_WITHIN_BUDGET) != 0; - hr = E_INVALIDARG; - if (committedAllocationParams.IsValid() && preferCommitted) - { - hr = AllocateCommittedResource2(committedAllocationParams, - resAllocInfo.SizeInBytes, withinBudget, pAllocDesc->pPrivateData, - &finalResourceDesc, InitialResourceState, pOptimizedClearValue, - ppAllocation, riidResource, ppvResource); - if (SUCCEEDED(hr)) - return hr; - } - if (blockVector != NULL) - { - hr = blockVector->CreateResource2(resAllocInfo.SizeInBytes, resAllocInfo.Alignment, - *pAllocDesc, finalResourceDesc, - InitialResourceState, pOptimizedClearValue, - ppAllocation, riidResource, ppvResource); - if (SUCCEEDED(hr)) - return hr; - } - if (committedAllocationParams.IsValid() && !preferCommitted) - { - hr = AllocateCommittedResource2(committedAllocationParams, - resAllocInfo.SizeInBytes, withinBudget, pAllocDesc->pPrivateData, - &finalResourceDesc, InitialResourceState, pOptimizedClearValue, - ppAllocation, riidResource, ppvResource); - if (SUCCEEDED(hr)) - return hr; - } - return hr; -} -#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ - HRESULT AllocatorPimpl::AllocateMemory( const ALLOCATION_DESC* pAllocDesc, const D3D12_RESOURCE_ALLOCATION_INFO* pAllocInfo, @@ -6862,16 +7138,53 @@ HRESULT AllocatorPimpl::AllocateMemory( HRESULT AllocatorPimpl::CreateAliasingResource( Allocation* pAllocation, UINT64 AllocationLocalOffset, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE* pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, REFIID riidResource, void** ppvResource) { *ppvResource = NULL; - D3D12_RESOURCE_DESC resourceDesc2 = *pResourceDesc; - D3D12_RESOURCE_ALLOCATION_INFO resAllocInfo = GetResourceAllocationInfo(resourceDesc2); + CREATE_RESOURCE_PARAMS finalCreateParams = createParams; + D3D12_RESOURCE_DESC finalResourceDesc; +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + D3D12_RESOURCE_DESC1 finalResourceDesc1; +#endif + D3D12_RESOURCE_ALLOCATION_INFO resAllocInfo; + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE) + { + finalResourceDesc = *createParams.GetResourceDesc(); + finalCreateParams.AccessResourceDesc() = &finalResourceDesc; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc); + } +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + else if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE_AND_DESC1) + { + if (!m_Device8) + { + return E_NOINTERFACE; + } + finalResourceDesc1 = *createParams.GetResourceDesc1(); + finalCreateParams.AccessResourceDesc1() = &finalResourceDesc1; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc1); + } +#endif +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + else if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_LAYOUT) + { + if (!m_Device10) + { + return E_NOINTERFACE; + } + finalResourceDesc1 = *createParams.GetResourceDesc1(); + finalCreateParams.AccessResourceDesc1() = &finalResourceDesc1; + resAllocInfo = GetResourceAllocationInfo(finalResourceDesc1); + } +#endif + else + { + D3D12MA_ASSERT(0); + return E_INVALIDARG; + } D3D12MA_ASSERT(IsPow2(resAllocInfo.Alignment)); D3D12MA_ASSERT(resAllocInfo.SizeInBytes > 0); @@ -6887,14 +7200,7 @@ HRESULT AllocatorPimpl::CreateAliasingResource( return E_INVALIDARG; } - return m_Device->CreatePlacedResource( - existingHeap, - newOffset, - &resourceDesc2, - InitialResourceState, - pOptimizedClearValue, - riidResource, - ppvResource); + return CreatePlacedResourceWrap(existingHeap, newOffset, finalCreateParams, riidResource, ppvResource); } void AllocatorPimpl::FreeCommittedMemory(Allocation* allocation) @@ -6936,6 +7242,17 @@ void AllocatorPimpl::FreeHeapMemory(Allocation* allocation) m_Budget.RemoveBlock(memSegmentGroup, allocSize); } +void AllocatorPimpl::SetResidencyPriority(ID3D12Pageable* obj, D3D12_RESIDENCY_PRIORITY priority) const +{ +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ + if (priority != D3D12_RESIDENCY_PRIORITY_NONE && m_Device1) + { + // Intentionally ignoring the result. + m_Device1->SetResidencyPriority(1, &obj, &priority); + } +#endif +} + void AllocatorPimpl::SetCurrentFrameIndex(UINT frameIndex) { m_CurrentFrameIndex.store(frameIndex); @@ -6945,7 +7262,7 @@ void AllocatorPimpl::SetCurrentFrameIndex(UINT frameIndex) #endif } -void AllocatorPimpl::CalculateStatistics(TotalStatistics& outStats) +void AllocatorPimpl::CalculateStatistics(TotalStatistics& outStats, DetailedStatistics outCustomHeaps[2]) { // Init stats for (size_t i = 0; i < HEAP_TYPE_COUNT; i++) @@ -6953,6 +7270,11 @@ void AllocatorPimpl::CalculateStatistics(TotalStatistics& outStats) for (size_t i = 0; i < DXGI_MEMORY_SEGMENT_GROUP_COUNT; i++) ClearDetailedStatistics(outStats.MemorySegmentGroup[i]); ClearDetailedStatistics(outStats.Total); + if (outCustomHeaps) + { + ClearDetailedStatistics(outCustomHeaps[0]); + ClearDetailedStatistics(outCustomHeaps[1]); + } // Process default pools. 3 standard heap types only. Add them to outStats.HeapType[i]. if (SupportsResourceHeapTier2()) @@ -7003,8 +7325,13 @@ void AllocatorPimpl::CalculateStatistics(TotalStatistics& outStats) pool->AddDetailedStatistics(tmpStats); AddDetailedStatistics( outStats.HeapType[heapTypeIndex], tmpStats); + + UINT memorySegment = HeapPropertiesToMemorySegmentGroup(poolHeapProps); AddDetailedStatistics( - outStats.MemorySegmentGroup[HeapPropertiesToMemorySegmentGroup(poolHeapProps)], tmpStats); + outStats.MemorySegmentGroup[memorySegment], tmpStats); + + if (outCustomHeaps) + AddDetailedStatistics(outCustomHeaps[memorySegment], tmpStats); } } @@ -7016,7 +7343,7 @@ void AllocatorPimpl::CalculateStatistics(TotalStatistics& outStats) AddDetailedStatistics( outStats.HeapType[heapTypeIndex], tmpStats); AddDetailedStatistics( - outStats.MemorySegmentGroup[StandardHeapTypeToMemorySegmentGroup(IndexToHeapType(heapTypeIndex))], tmpStats); + outStats.MemorySegmentGroup[StandardHeapTypeToMemorySegmentGroup(IndexToStandardHeapType(heapTypeIndex))], tmpStats); } // Sum up memory segment groups to totals. @@ -7106,155 +7433,329 @@ void AllocatorPimpl::GetBudgetForHeapType(Budget& outBudget, D3D12_HEAP_TYPE hea } } -void AllocatorPimpl::BuildStatsString(WCHAR** ppStatsString, BOOL DetailedMap) +void AllocatorPimpl::BuildStatsString(WCHAR** ppStatsString, BOOL detailedMap) { StringBuilder sb(GetAllocs()); { - JsonWriter json(GetAllocs(), sb); - Budget localBudget = {}, nonLocalBudget = {}; GetBudget(&localBudget, &nonLocalBudget); TotalStatistics stats; - CalculateStatistics(stats); + DetailedStatistics customHeaps[2]; + CalculateStatistics(stats, customHeaps); - json.BeginObject(); - - json.WriteString(L"Total"); - json.AddDetailedStatisticsInfoObject(stats.Total); - for (size_t heapType = 0; heapType < HEAP_TYPE_COUNT; ++heapType) - { - json.WriteString(HeapTypeNames[heapType]); - json.AddDetailedStatisticsInfoObject(stats.HeapType[heapType]); - } - - json.WriteString(L"Budget"); + JsonWriter json(GetAllocs(), sb); json.BeginObject(); { - json.WriteString(L"Local"); - WriteBudgetToJson(json, localBudget); - json.WriteString(L"NonLocal"); - WriteBudgetToJson(json, nonLocalBudget); - } - json.EndObject(); - - if (DetailedMap) - { - json.WriteString(L"DetailedMap"); + json.WriteString(L"General"); json.BeginObject(); + { + json.WriteString(L"API"); + json.WriteString(L"Direct3D 12"); + + json.WriteString(L"GPU"); + json.WriteString(m_AdapterDesc.Description); + + json.WriteString(L"DedicatedVideoMemory"); + json.WriteNumber((UINT64)m_AdapterDesc.DedicatedVideoMemory); + json.WriteString(L"DedicatedSystemMemory"); + json.WriteNumber((UINT64)m_AdapterDesc.DedicatedSystemMemory); + json.WriteString(L"SharedSystemMemory"); + json.WriteNumber((UINT64)m_AdapterDesc.SharedSystemMemory); + + json.WriteString(L"ResourceHeapTier"); + json.WriteNumber(static_cast(m_D3D12Options.ResourceHeapTier)); + + json.WriteString(L"ResourceBindingTier"); + json.WriteNumber(static_cast(m_D3D12Options.ResourceBindingTier)); + + json.WriteString(L"TiledResourcesTier"); + json.WriteNumber(static_cast(m_D3D12Options.TiledResourcesTier)); + + json.WriteString(L"TileBasedRenderer"); + json.WriteBool(m_D3D12Architecture.TileBasedRenderer); + + json.WriteString(L"UMA"); + json.WriteBool(m_D3D12Architecture.UMA); + json.WriteString(L"CacheCoherentUMA"); + json.WriteBool(m_D3D12Architecture.CacheCoherentUMA); + } + json.EndObject(); + } + { + json.WriteString(L"Total"); + json.AddDetailedStatisticsInfoObject(stats.Total); + } + { + json.WriteString(L"MemoryInfo"); + json.BeginObject(); + { + json.WriteString(L"L0"); + json.BeginObject(); + { + json.WriteString(L"Budget"); + WriteBudgetToJson(json, IsUMA() ? localBudget : nonLocalBudget); // When UMA device only L0 present as local + + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.MemorySegmentGroup[!IsUMA()]); + + json.WriteString(L"MemoryPools"); + json.BeginObject(); + { + if (IsUMA()) + { + json.WriteString(L"DEFAULT"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.HeapType[0]); + } + json.EndObject(); + } + json.WriteString(L"UPLOAD"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.HeapType[1]); + } + json.EndObject(); + + json.WriteString(L"READBACK"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.HeapType[2]); + } + json.EndObject(); + + json.WriteString(L"CUSTOM"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(customHeaps[!IsUMA()]); + } + json.EndObject(); + } + json.EndObject(); + } + json.EndObject(); + if (!IsUMA()) + { + json.WriteString(L"L1"); + json.BeginObject(); + { + json.WriteString(L"Budget"); + WriteBudgetToJson(json, localBudget); + + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.MemorySegmentGroup[0]); + + json.WriteString(L"MemoryPools"); + json.BeginObject(); + { + json.WriteString(L"DEFAULT"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(stats.HeapType[0]); + } + json.EndObject(); + + json.WriteString(L"CUSTOM"); + json.BeginObject(); + { + json.WriteString(L"Stats"); + json.AddDetailedStatisticsInfoObject(customHeaps[0]); + } + json.EndObject(); + } + json.EndObject(); + } + json.EndObject(); + } + } + json.EndObject(); + } + + if (detailedMap) + { + const auto writeHeapInfo = [&](BlockVector* blockVector, CommittedAllocationList* committedAllocs, bool customHeap) + { + D3D12MA_ASSERT(blockVector); + + D3D12_HEAP_FLAGS flags = blockVector->GetHeapFlags(); + json.WriteString(L"Flags"); + json.BeginArray(true); + { + if (flags & D3D12_HEAP_FLAG_SHARED) + json.WriteString(L"HEAP_FLAG_SHARED"); + if (flags & D3D12_HEAP_FLAG_ALLOW_DISPLAY) + json.WriteString(L"HEAP_FLAG_ALLOW_DISPLAY"); + if (flags & D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER) + json.WriteString(L"HEAP_FLAG_CROSS_ADAPTER"); + if (flags & D3D12_HEAP_FLAG_HARDWARE_PROTECTED) + json.WriteString(L"HEAP_FLAG_HARDWARE_PROTECTED"); + if (flags & D3D12_HEAP_FLAG_ALLOW_WRITE_WATCH) + json.WriteString(L"HEAP_FLAG_ALLOW_WRITE_WATCH"); + if (flags & D3D12_HEAP_FLAG_ALLOW_SHADER_ATOMICS) + json.WriteString(L"HEAP_FLAG_ALLOW_SHADER_ATOMICS"); +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + if (flags & D3D12_HEAP_FLAG_CREATE_NOT_RESIDENT) + json.WriteString(L"HEAP_FLAG_CREATE_NOT_RESIDENT"); + if (flags & D3D12_HEAP_FLAG_CREATE_NOT_ZEROED) + json.WriteString(L"HEAP_FLAG_CREATE_NOT_ZEROED"); +#endif + + if (flags & D3D12_HEAP_FLAG_DENY_BUFFERS) + json.WriteString(L"HEAP_FLAG_DENY_BUFFERS"); + if (flags & D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES) + json.WriteString(L"HEAP_FLAG_DENY_RT_DS_TEXTURES"); + if (flags & D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES) + json.WriteString(L"HEAP_FLAG_DENY_NON_RT_DS_TEXTURES"); + + flags &= ~(D3D12_HEAP_FLAG_SHARED + | D3D12_HEAP_FLAG_DENY_BUFFERS + | D3D12_HEAP_FLAG_ALLOW_DISPLAY + | D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER + | D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES + | D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES + | D3D12_HEAP_FLAG_HARDWARE_PROTECTED + | D3D12_HEAP_FLAG_ALLOW_WRITE_WATCH + | D3D12_HEAP_FLAG_ALLOW_SHADER_ATOMICS); +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ + flags &= ~(D3D12_HEAP_FLAG_CREATE_NOT_RESIDENT + | D3D12_HEAP_FLAG_CREATE_NOT_ZEROED); +#endif + if (flags != 0) + json.WriteNumber((UINT)flags); + + if (customHeap) + { + const D3D12_HEAP_PROPERTIES& properties = blockVector->GetHeapProperties(); + switch (properties.MemoryPoolPreference) + { + default: + D3D12MA_ASSERT(0); + case D3D12_MEMORY_POOL_UNKNOWN: + json.WriteString(L"MEMORY_POOL_UNKNOWN"); + break; + case D3D12_MEMORY_POOL_L0: + json.WriteString(L"MEMORY_POOL_L0"); + break; + case D3D12_MEMORY_POOL_L1: + json.WriteString(L"MEMORY_POOL_L1"); + break; + } + switch (properties.CPUPageProperty) + { + default: + D3D12MA_ASSERT(0); + case D3D12_CPU_PAGE_PROPERTY_UNKNOWN: + json.WriteString(L"CPU_PAGE_PROPERTY_UNKNOWN"); + break; + case D3D12_CPU_PAGE_PROPERTY_NOT_AVAILABLE: + json.WriteString(L"CPU_PAGE_PROPERTY_NOT_AVAILABLE"); + break; + case D3D12_CPU_PAGE_PROPERTY_WRITE_COMBINE: + json.WriteString(L"CPU_PAGE_PROPERTY_WRITE_COMBINE"); + break; + case D3D12_CPU_PAGE_PROPERTY_WRITE_BACK: + json.WriteString(L"CPU_PAGE_PROPERTY_WRITE_BACK"); + break; + } + } + } + json.EndArray(); + + json.WriteString(L"PreferredBlockSize"); + json.WriteNumber(blockVector->GetPreferredBlockSize()); + + json.WriteString(L"Blocks"); + blockVector->WriteBlockInfoToJson(json); + + json.WriteString(L"DedicatedAllocations"); + json.BeginArray(); + if (committedAllocs) + committedAllocs->BuildStatsString(json); + json.EndArray(); + }; json.WriteString(L"DefaultPools"); json.BeginObject(); - - if (SupportsResourceHeapTier2()) { - for (size_t heapType = 0; heapType < STANDARD_HEAP_TYPE_COUNT; ++heapType) + if (SupportsResourceHeapTier2()) { - json.WriteString(HeapTypeNames[heapType]); - json.BeginObject(); - - json.WriteString(L"Blocks"); - - BlockVector* blockVector = m_BlockVectors[heapType]; - D3D12MA_ASSERT(blockVector); - blockVector->WriteBlockInfoToJson(json); - - json.EndObject(); // heap name - } - } - else - { - for (size_t heapType = 0; heapType < STANDARD_HEAP_TYPE_COUNT; ++heapType) - { - for (size_t heapSubType = 0; heapSubType < 3; ++heapSubType) + for (uint8_t heapType = 0; heapType < STANDARD_HEAP_TYPE_COUNT; ++heapType) { - static const WCHAR* const heapSubTypeName[] = { - L" + buffer", - L" + texture", - L" + texture RT or DS", - }; - json.BeginString(); - json.ContinueString(HeapTypeNames[heapType]); - json.ContinueString(heapSubTypeName[heapSubType]); - json.EndString(); + json.WriteString(StandardHeapTypeNames[heapType]); json.BeginObject(); - - json.WriteString(L"Blocks"); - - BlockVector* blockVector = m_BlockVectors[heapType * 3 + heapSubType]; - D3D12MA_ASSERT(blockVector); - blockVector->WriteBlockInfoToJson(json); - - json.EndObject(); // heap name + writeHeapInfo(m_BlockVectors[heapType], m_CommittedAllocations + heapType, false); + json.EndObject(); } } - } - - json.EndObject(); // DefaultPools - - json.WriteString(L"CommittedAllocations"); - json.BeginObject(); - - for (size_t heapTypeIndex = 0; heapTypeIndex < STANDARD_HEAP_TYPE_COUNT; ++heapTypeIndex) - { - json.WriteString(HeapTypeNames[heapTypeIndex]); - json.BeginArray(); - m_CommittedAllocations[heapTypeIndex].BuildStatsString(json); - json.EndArray(); - } - - json.EndObject(); // CommittedAllocations - - json.WriteString(L"Pools"); - json.BeginObject(); - - for (size_t heapTypeIndex = 0; heapTypeIndex < HEAP_TYPE_COUNT; ++heapTypeIndex) - { - json.WriteString(HeapTypeNames[heapTypeIndex]); - json.BeginArray(); - MutexLockRead mutex(m_PoolsMutex[heapTypeIndex], m_UseMutex); - size_t index = 0; - for (auto* item = m_Pools[heapTypeIndex].Front(); item != nullptr; item = PoolList::GetNext(item)) + else { - json.BeginObject(); - json.WriteString(L"Name"); - if (item->GetName() != nullptr) + for (uint8_t heapType = 0; heapType < STANDARD_HEAP_TYPE_COUNT; ++heapType) { - json.WriteString(item->GetName()); + for (uint8_t heapSubType = 0; heapSubType < 3; ++heapSubType) + { + static const WCHAR* const heapSubTypeName[] = { + L" - Buffers", + L" - Textures", + L" - Textures RT/DS", + }; + json.BeginString(StandardHeapTypeNames[heapType]); + json.EndString(heapSubTypeName[heapSubType]); + + json.BeginObject(); + writeHeapInfo(m_BlockVectors[heapType * 3 + heapSubType], m_CommittedAllocations + heapType, false); + json.EndObject(); + } } - else - { - json.BeginString(); - json.ContinueString(index); - json.EndString(); - } - ++index; - - json.WriteString(L"Blocks"); - item->GetBlockVector()->WriteBlockInfoToJson(json); - - json.WriteString(L"CommittedAllocations"); - json.BeginArray(); - if (item->SupportsCommittedAllocations()) - item->GetCommittedAllocationList()->BuildStatsString(json); - json.EndArray(); - - json.EndObject(); } - json.EndArray(); } + json.EndObject(); - json.EndObject(); // Pools + json.WriteString(L"CustomPools"); + json.BeginObject(); + for (uint8_t heapTypeIndex = 0; heapTypeIndex < HEAP_TYPE_COUNT; ++heapTypeIndex) + { + MutexLockRead mutex(m_PoolsMutex[heapTypeIndex], m_UseMutex); + auto* item = m_Pools[heapTypeIndex].Front(); + if (item != NULL) + { + size_t index = 0; + json.WriteString(HeapTypeNames[heapTypeIndex]); + json.BeginArray(); + do + { + json.BeginObject(); + json.WriteString(L"Name"); + json.BeginString(); + json.ContinueString(index++); + if (item->GetName()) + { + json.ContinueString(L" - "); + json.ContinueString(item->GetName()); + } + json.EndString(); - json.EndObject(); // DetailedMap + writeHeapInfo(item->GetBlockVector(), item->GetCommittedAllocationList(), heapTypeIndex == 3); + json.EndObject(); + } while ((item = PoolList::GetNext(item)) != NULL); + json.EndArray(); + } + } + json.EndObject(); } json.EndObject(); } const size_t length = sb.GetLength(); - WCHAR* result = AllocateArray(GetAllocs(), length + 1); - memcpy(result, sb.GetData(), length * sizeof(WCHAR)); - result[length] = L'\0'; + WCHAR* result = AllocateArray(GetAllocs(), length + 2); + result[0] = 0xFEFF; + memcpy(result + 1, sb.GetData(), length * sizeof(WCHAR)); + result[length + 1] = L'\0'; *ppStatsString = result; } @@ -7274,19 +7775,46 @@ bool AllocatorPimpl::PrefersCommittedAllocation(const D3D12_RESOURCE_DESC_T& res HRESULT AllocatorPimpl::AllocateCommittedResource( const CommittedAllocationParameters& committedAllocParams, UINT64 resourceSize, bool withinBudget, void* pPrivateData, - const D3D12_RESOURCE_DESC* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE* pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource) { D3D12MA_ASSERT(committedAllocParams.IsValid()); + HRESULT hr; + ID3D12Resource* res = NULL; + // Allocate aliasing memory with explicit heap + if (committedAllocParams.m_CanAlias) + { + D3D12_RESOURCE_ALLOCATION_INFO heapAllocInfo = {}; + heapAllocInfo.SizeInBytes = resourceSize; + heapAllocInfo.Alignment = HeapFlagsToAlignment(committedAllocParams.m_HeapFlags, m_MsaaAlwaysCommitted); + hr = AllocateHeap(committedAllocParams, heapAllocInfo, withinBudget, pPrivateData, ppAllocation); + if (SUCCEEDED(hr)) + { + hr = CreatePlacedResourceWrap((*ppAllocation)->GetHeap(), 0, + createParams, D3D12MA_IID_PPV_ARGS(&res)); + if (SUCCEEDED(hr)) + { + if (ppvResource != NULL) + hr = res->QueryInterface(riidResource, ppvResource); + if (SUCCEEDED(hr)) + { + (*ppAllocation)->SetResourcePointer(res, createParams.GetBaseResourceDesc()); + return hr; + } + res->Release(); + } + FreeHeapMemory(*ppAllocation); + } + return hr; + } + if (withinBudget && !NewAllocationWithinBudget(committedAllocParams.m_HeapProperties.Type, resourceSize)) { return E_OUTOFMEMORY; } - ID3D12Resource* res = NULL; /* D3D12 ERROR: * ID3D12Device::CreateCommittedResource: * When creating a committed resource, D3D12_HEAP_FLAGS must not have either @@ -7297,100 +7825,93 @@ HRESULT AllocatorPimpl::AllocateCommittedResource( * * [ STATE_CREATION ERROR #640: CREATERESOURCEANDHEAP_INVALIDHEAPMISCFLAGS] */ - HRESULT hr; -#ifdef __ID3D12Device4_INTERFACE_DEFINED__ - if (m_Device4) + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_LAYOUT) { - hr = m_Device4->CreateCommittedResource1( - &committedAllocParams.m_HeapProperties, - committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, - pResourceDesc, InitialResourceState, - pOptimizedClearValue, committedAllocParams.m_ProtectedSession, D3D12MA_IID_PPV_ARGS(&res)); - } - else -#endif - { - if (committedAllocParams.m_ProtectedSession == NULL) + if (!m_Device10) { - hr = m_Device->CreateCommittedResource( + return E_NOINTERFACE; + } + hr = m_Device10->CreateCommittedResource3( &committedAllocParams.m_HeapProperties, committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, - pResourceDesc, InitialResourceState, - pOptimizedClearValue, D3D12MA_IID_PPV_ARGS(&res)); - } - else - hr = E_NOINTERFACE; - } - - if (SUCCEEDED(hr)) - { - if (ppvResource != NULL) - { - hr = res->QueryInterface(riidResource, ppvResource); - } - if (SUCCEEDED(hr)) - { - const BOOL wasZeroInitialized = TRUE; - Allocation* alloc = m_AllocationObjectAllocator.Allocate(this, resourceSize, pResourceDesc->Alignment, wasZeroInitialized); - alloc->InitCommitted(committedAllocParams.m_List); - alloc->SetResource(res, pResourceDesc); - alloc->SetPrivateData(pPrivateData); - - *ppAllocation = alloc; - - committedAllocParams.m_List->Register(alloc); - - const UINT memSegmentGroup = HeapPropertiesToMemorySegmentGroup(committedAllocParams.m_HeapProperties); - m_Budget.AddBlock(memSegmentGroup, resourceSize); - m_Budget.AddAllocation(memSegmentGroup, resourceSize); - } - else - { - res->Release(); - } - } - return hr; -} - + createParams.GetResourceDesc1(), createParams.GetInitialLayout(), + createParams.GetOptimizedClearValue(), committedAllocParams.m_ProtectedSession, + createParams.GetNumCastableFormats(), createParams.GetCastableFormats(), + D3D12MA_IID_PPV_ARGS(&res)); + } else +#endif #ifdef __ID3D12Device8_INTERFACE_DEFINED__ -HRESULT AllocatorPimpl::AllocateCommittedResource2( - const CommittedAllocationParameters& committedAllocParams, - UINT64 resourceSize, bool withinBudget, void* pPrivateData, - const D3D12_RESOURCE_DESC1* pResourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE* pOptimizedClearValue, - Allocation** ppAllocation, REFIID riidResource, void** ppvResource) -{ - D3D12MA_ASSERT(committedAllocParams.IsValid()); - - if (m_Device8 == NULL) + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE_AND_DESC1) { - return E_NOINTERFACE; + if (!m_Device8) + { + return E_NOINTERFACE; + } + hr = m_Device8->CreateCommittedResource2( + &committedAllocParams.m_HeapProperties, + committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, + createParams.GetResourceDesc1(), createParams.GetInitialResourceState(), + createParams.GetOptimizedClearValue(), committedAllocParams.m_ProtectedSession, + D3D12MA_IID_PPV_ARGS(&res)); + } else +#endif + if (createParams.Variant == CREATE_RESOURCE_PARAMS::VARIANT_WITH_STATE) + { +#ifdef __ID3D12Device4_INTERFACE_DEFINED__ + if (m_Device4) + { + hr = m_Device4->CreateCommittedResource1( + &committedAllocParams.m_HeapProperties, + committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, + createParams.GetResourceDesc(), createParams.GetInitialResourceState(), + createParams.GetOptimizedClearValue(), committedAllocParams.m_ProtectedSession, + D3D12MA_IID_PPV_ARGS(&res)); + } + else +#endif + { + if (committedAllocParams.m_ProtectedSession == NULL) + { + hr = m_Device->CreateCommittedResource( + &committedAllocParams.m_HeapProperties, + committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, + createParams.GetResourceDesc(), createParams.GetInitialResourceState(), + createParams.GetOptimizedClearValue(), D3D12MA_IID_PPV_ARGS(&res)); + } + else + hr = E_NOINTERFACE; + } + } + else + { + D3D12MA_ASSERT(0); + return E_INVALIDARG; } - if (withinBudget && - !NewAllocationWithinBudget(committedAllocParams.m_HeapProperties.Type, resourceSize)) - { - return E_OUTOFMEMORY; - } - - ID3D12Resource* res = NULL; - HRESULT hr = m_Device8->CreateCommittedResource2( - &committedAllocParams.m_HeapProperties, - committedAllocParams.m_HeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS, // D3D12 ERROR: ID3D12Device::CreateCommittedResource: When creating a committed resource, D3D12_HEAP_FLAGS must not have either D3D12_HEAP_FLAG_DENY_NON_RT_DS_TEXTURES, D3D12_HEAP_FLAG_DENY_RT_DS_TEXTURES, nor D3D12_HEAP_FLAG_DENY_BUFFERS set. These flags will be set automatically to correspond with the committed resource type. [ STATE_CREATION ERROR #640: CREATERESOURCEANDHEAP_INVALIDHEAPMISCFLAGS] - pResourceDesc, InitialResourceState, - pOptimizedClearValue, committedAllocParams.m_ProtectedSession, D3D12MA_IID_PPV_ARGS(&res)); if (SUCCEEDED(hr)) { + SetResidencyPriority(res, committedAllocParams.m_ResidencyPriority); + if (ppvResource != NULL) { hr = res->QueryInterface(riidResource, ppvResource); } if (SUCCEEDED(hr)) { - const BOOL wasZeroInitialized = TRUE; - Allocation* alloc = m_AllocationObjectAllocator.Allocate(this, resourceSize, pResourceDesc->Alignment, wasZeroInitialized); + BOOL wasZeroInitialized = TRUE; +#if D3D12MA_CREATE_NOT_ZEROED_AVAILABLE + if((committedAllocParams.m_HeapFlags & D3D12_HEAP_FLAG_CREATE_NOT_ZEROED) != 0) + { + wasZeroInitialized = FALSE; + } +#endif + + Allocation* alloc = m_AllocationObjectAllocator.Allocate( + this, resourceSize, createParams.GetBaseResourceDesc()->Alignment, wasZeroInitialized); alloc->InitCommitted(committedAllocParams.m_List); - alloc->SetResource(res, pResourceDesc); + alloc->SetResourcePointer(res, createParams.GetBaseResourceDesc()); alloc->SetPrivateData(pPrivateData); *ppAllocation = alloc; @@ -7408,7 +7929,6 @@ HRESULT AllocatorPimpl::AllocateCommittedResource2( } return hr; } -#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ HRESULT AllocatorPimpl::AllocateHeap( const CommittedAllocationParameters& committedAllocParams, @@ -7447,7 +7967,16 @@ HRESULT AllocatorPimpl::AllocateHeap( if (SUCCEEDED(hr)) { - const BOOL wasZeroInitialized = TRUE; + SetResidencyPriority(heap, committedAllocParams.m_ResidencyPriority); + + BOOL wasZeroInitialized = TRUE; +#if D3D12MA_CREATE_NOT_ZEROED_AVAILABLE + if((heapDesc.Flags & D3D12_HEAP_FLAG_CREATE_NOT_ZEROED) != 0) + { + wasZeroInitialized = FALSE; + } +#endif + (*ppAllocation) = m_AllocationObjectAllocator.Allocate(this, allocInfo.SizeInBytes, allocInfo.Alignment, wasZeroInitialized); (*ppAllocation)->InitHeap(committedAllocParams.m_List, heap); (*ppAllocation)->SetPrivateData(pPrivateData); @@ -7469,16 +7998,20 @@ HRESULT AllocatorPimpl::CalcAllocationParams(const ALLOCATION_DESC& allocDesc, U outCommittedAllocationParams = CommittedAllocationParameters(); outPreferCommitted = false; + bool msaaAlwaysCommitted; if (allocDesc.CustomPool != NULL) { PoolPimpl* const pool = allocDesc.CustomPool->m_Pimpl; + msaaAlwaysCommitted = pool->GetBlockVector()->DeniesMsaaTextures(); outBlockVector = pool->GetBlockVector(); - outCommittedAllocationParams.m_ProtectedSession = pool->GetDesc().pProtectedSession; - outCommittedAllocationParams.m_HeapProperties = pool->GetDesc().HeapProperties; - outCommittedAllocationParams.m_HeapFlags = pool->GetDesc().HeapFlags; + const auto& desc = pool->GetDesc(); + outCommittedAllocationParams.m_ProtectedSession = desc.pProtectedSession; + outCommittedAllocationParams.m_HeapProperties = desc.HeapProperties; + outCommittedAllocationParams.m_HeapFlags = desc.HeapFlags; outCommittedAllocationParams.m_List = pool->GetCommittedAllocationList(); + outCommittedAllocationParams.m_ResidencyPriority = pool->GetDesc().ResidencyPriority; } else { @@ -7486,10 +8019,12 @@ HRESULT AllocatorPimpl::CalcAllocationParams(const ALLOCATION_DESC& allocDesc, U { return E_INVALIDARG; } + msaaAlwaysCommitted = m_MsaaAlwaysCommitted; outCommittedAllocationParams.m_HeapProperties = StandardHeapTypeToHeapProperties(allocDesc.HeapType); outCommittedAllocationParams.m_HeapFlags = allocDesc.ExtraHeapFlags; - outCommittedAllocationParams.m_List = &m_CommittedAllocations[HeapTypeToIndex(allocDesc.HeapType)]; + outCommittedAllocationParams.m_List = &m_CommittedAllocations[StandardHeapTypeToIndex(allocDesc.HeapType)]; + // outCommittedAllocationParams.m_ResidencyPriority intentionally left with default value. const ResourceClass resourceClass = (resDesc != NULL) ? ResourceDescToResourceClass(*resDesc) : HeapFlagsToResourceClass(allocDesc.ExtraHeapFlags); @@ -7521,14 +8056,18 @@ HRESULT AllocatorPimpl::CalcAllocationParams(const ALLOCATION_DESC& allocDesc, U { outBlockVector = NULL; } - if ((allocDesc.Flags & (ALLOCATION_FLAG_NEVER_ALLOCATE | ALLOCATION_FLAG_CAN_ALIAS)) != 0) + if ((allocDesc.Flags & ALLOCATION_FLAG_NEVER_ALLOCATE) != 0) { outCommittedAllocationParams.m_List = NULL; } + outCommittedAllocationParams.m_CanAlias = allocDesc.Flags & ALLOCATION_FLAG_CAN_ALIAS; - if (resDesc != NULL && !outPreferCommitted && PrefersCommittedAllocation(*resDesc)) + if (resDesc != NULL) { - outPreferCommitted = true; + if (resDesc->SampleDesc.Count > 1 && msaaAlwaysCommitted) + outBlockVector = NULL; + if (!outPreferCommitted && PrefersCommittedAllocation(*resDesc)) + outPreferCommitted = true; } return (outBlockVector != NULL || outCommittedAllocationParams.m_List != NULL) ? S_OK : E_INVALIDARG; @@ -7536,7 +8075,17 @@ HRESULT AllocatorPimpl::CalcAllocationParams(const ALLOCATION_DESC& allocDesc, U UINT AllocatorPimpl::CalcDefaultPoolIndex(const ALLOCATION_DESC& allocDesc, ResourceClass resourceClass) const { - const D3D12_HEAP_FLAGS extraHeapFlags = allocDesc.ExtraHeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS; + D3D12_HEAP_FLAGS extraHeapFlags = allocDesc.ExtraHeapFlags & ~RESOURCE_CLASS_HEAP_FLAGS; + +#if D3D12MA_CREATE_NOT_ZEROED_AVAILABLE + // If allocator was created with ALLOCATOR_FLAG_DEFAULT_POOLS_NOT_ZEROED, also ignore + // D3D12_HEAP_FLAG_CREATE_NOT_ZEROED. + if(m_DefaultPoolsNotZeroed) + { + extraHeapFlags &= ~D3D12_HEAP_FLAG_CREATE_NOT_ZEROED; + } +#endif + if (extraHeapFlags != 0) { return UINT32_MAX; @@ -7610,7 +8159,7 @@ void AllocatorPimpl::CalcDefaultPoolParams(D3D12_HEAP_TYPE& outHeapType, D3D12_H void AllocatorPimpl::RegisterPool(Pool* pool, D3D12_HEAP_TYPE heapType) { - const UINT heapTypeIndex = HeapTypeToIndex(heapType); + const UINT heapTypeIndex = (UINT)heapType - 1; MutexLockWrite lock(m_PoolsMutex[heapTypeIndex], m_UseMutex); m_Pools[heapTypeIndex].PushBack(pool->m_Pimpl); @@ -7618,7 +8167,7 @@ void AllocatorPimpl::RegisterPool(Pool* pool, D3D12_HEAP_TYPE heapType) void AllocatorPimpl::UnregisterPool(Pool* pool, D3D12_HEAP_TYPE heapType) { - const UINT heapTypeIndex = HeapTypeToIndex(heapType); + const UINT heapTypeIndex = (UINT)heapType - 1; MutexLockWrite lock(m_PoolsMutex[heapTypeIndex], m_UseMutex); m_Pools[heapTypeIndex].Remove(pool->m_Pimpl); @@ -7638,7 +8187,14 @@ HRESULT AllocatorPimpl::UpdateD3D12Budget() D3D12_RESOURCE_ALLOCATION_INFO AllocatorPimpl::GetResourceAllocationInfoNative(const D3D12_RESOURCE_DESC& resourceDesc) const { + // This is how new D3D12 headers define GetResourceAllocationInfo function - + // different signature depending on these macros. +#if defined(_MSC_VER) || !defined(_WIN32) return m_Device->GetResourceAllocationInfo(0, 1, &resourceDesc); +#else + D3D12_RESOURCE_ALLOCATION_INFO retVal; + return *m_Device->GetResourceAllocationInfo(&retVal, 0, 1, &resourceDesc); +#endif } #ifdef __ID3D12Device8_INTERFACE_DEFINED__ @@ -7646,13 +8202,22 @@ D3D12_RESOURCE_ALLOCATION_INFO AllocatorPimpl::GetResourceAllocationInfoNative(c { D3D12MA_ASSERT(m_Device8 != NULL); D3D12_RESOURCE_ALLOCATION_INFO1 info1Unused; + + // This is how new D3D12 headers define GetResourceAllocationInfo function - + // different signature depending on these macros. +#if defined(_MSC_VER) || !defined(_WIN32) return m_Device8->GetResourceAllocationInfo2(0, 1, &resourceDesc, &info1Unused); +#else + D3D12_RESOURCE_ALLOCATION_INFO retVal; + return *m_Device8->GetResourceAllocationInfo2(&retVal, 0, 1, &resourceDesc, &info1Unused); +#endif } #endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ template D3D12_RESOURCE_ALLOCATION_INFO AllocatorPimpl::GetResourceAllocationInfo(D3D12_RESOURCE_DESC_T& inOutResourceDesc) const { +#ifdef __ID3D12Device1_INTERFACE_DEFINED__ /* Optional optimization: Microsoft documentation says: https://docs.microsoft.com/en-us/windows/win32/api/d3d12/nf-d3d12-id3d12device-getresourceallocationinfo @@ -7668,6 +8233,7 @@ D3D12_RESOURCE_ALLOCATION_INFO AllocatorPimpl::GetResourceAllocationInfo(D3D12_R AlignUp(inOutResourceDesc.Width, D3D12_DEFAULT_RESOURCE_PLACEMENT_ALIGNMENT), // SizeInBytes D3D12_DEFAULT_RESOURCE_PLACEMENT_ALIGNMENT }; // Alignment } +#endif // #ifdef __ID3D12Device1_INTERFACE_DEFINED__ #if D3D12MA_USE_SMALL_RESOURCE_PLACEMENT_ALIGNMENT if (inOutResourceDesc.Alignment == 0 && @@ -7710,21 +8276,14 @@ void AllocatorPimpl::WriteBudgetToJson(JsonWriter& json, const Budget& budget) { json.BeginObject(); { - json.WriteString(L"BlockCount"); - json.WriteNumber(budget.Stats.BlockCount); - json.WriteString(L"AllocationCount"); - json.WriteNumber(budget.Stats.AllocationCount); - json.WriteString(L"BlockBytes"); - json.WriteNumber(budget.Stats.BlockBytes); - json.WriteString(L"AllocationBytes"); - json.WriteNumber(budget.Stats.AllocationBytes); - json.WriteString(L"UsageBytes"); - json.WriteNumber(budget.UsageBytes); json.WriteString(L"BudgetBytes"); json.WriteNumber(budget.BudgetBytes); + json.WriteString(L"UsageBytes"); + json.WriteNumber(budget.UsageBytes); } json.EndObject(); } + #endif // _D3D12MA_ALLOCATOR_PIMPL #endif // _D3D12MA_ALLOCATOR_PIMPL @@ -7789,14 +8348,14 @@ MemoryBlock::~MemoryBlock() } } -HRESULT MemoryBlock::Init(ID3D12ProtectedResourceSession* pProtectedSession) +HRESULT MemoryBlock::Init(ID3D12ProtectedResourceSession* pProtectedSession, bool denyMsaaTextures) { D3D12MA_ASSERT(m_Heap == NULL && m_Size > 0); D3D12_HEAP_DESC heapDesc = {}; heapDesc.SizeInBytes = m_Size; heapDesc.Properties = m_HeapProps; - heapDesc.Alignment = HeapFlagsToAlignment(m_HeapFlags); + heapDesc.Alignment = HeapFlagsToAlignment(m_HeapFlags, denyMsaaTextures); heapDesc.Flags = m_HeapFlags; HRESULT hr; @@ -7838,6 +8397,10 @@ NormalBlock::~NormalBlock() { if (m_pMetadata != NULL) { + // Define macro D3D12MA_DEBUG_LOG to receive the list of the unfreed allocations. + if (!m_pMetadata->IsEmpty()) + m_pMetadata->DebugLogAllAllocations(); + // THIS IS THE MOST IMPORTANT ASSERT IN THE ENTIRE LIBRARY! // Hitting it means you have some memory leak - unreleased Allocation objects. D3D12MA_ASSERT(m_pMetadata->IsEmpty() && "Some allocations were not freed before destruction of this memory block!"); @@ -7846,9 +8409,9 @@ NormalBlock::~NormalBlock() } } -HRESULT NormalBlock::Init(UINT32 algorithm, ID3D12ProtectedResourceSession* pProtectedSession) +HRESULT NormalBlock::Init(UINT32 algorithm, ID3D12ProtectedResourceSession* pProtectedSession, bool denyMsaaTextures) { - HRESULT hr = MemoryBlock::Init(pProtectedSession); + HRESULT hr = MemoryBlock::Init(pProtectedSession, denyMsaaTextures); if (FAILED(hr)) { return hr; @@ -7970,7 +8533,9 @@ BlockVector::BlockVector( bool explicitBlockSize, UINT64 minAllocationAlignment, UINT32 algorithm, - ID3D12ProtectedResourceSession* pProtectedSession) + bool denyMsaaTextures, + ID3D12ProtectedResourceSession* pProtectedSession, + D3D12_RESIDENCY_PRIORITY residencyPriority) : m_hAllocator(hAllocator), m_HeapProps(heapProps), m_HeapFlags(heapFlags), @@ -7980,7 +8545,9 @@ BlockVector::BlockVector( m_ExplicitBlockSize(explicitBlockSize), m_MinAllocationAlignment(minAllocationAlignment), m_Algorithm(algorithm), + m_DenyMsaaTextures(denyMsaaTextures), m_ProtectedSession(pProtectedSession), + m_ResidencyPriority(residencyPriority), m_HasEmptyBlock(false), m_Blocks(hAllocator->GetAllocs()), m_NextBlockId(0) {} @@ -8117,9 +8684,7 @@ HRESULT BlockVector::CreateResource( UINT64 size, UINT64 alignment, const ALLOCATION_DESC& allocDesc, - const D3D12_RESOURCE_DESC& resourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE* pOptimizedClearValue, + const CREATE_RESOURCE_PARAMS& createParams, Allocation** ppAllocation, REFIID riidResource, void** ppvResource) @@ -8128,12 +8693,10 @@ HRESULT BlockVector::CreateResource( if (SUCCEEDED(hr)) { ID3D12Resource* res = NULL; - hr = m_hAllocator->GetDevice()->CreatePlacedResource( + hr = m_hAllocator->CreatePlacedResourceWrap( (*ppAllocation)->m_Placed.block->GetHeap(), (*ppAllocation)->GetOffset(), - &resourceDesc, - InitialResourceState, - pOptimizedClearValue, + createParams, D3D12MA_IID_PPV_ARGS(&res)); if (SUCCEEDED(hr)) { @@ -8143,7 +8706,7 @@ HRESULT BlockVector::CreateResource( } if (SUCCEEDED(hr)) { - (*ppAllocation)->SetResource(res, &resourceDesc); + (*ppAllocation)->SetResourcePointer(res, createParams.GetBaseResourceDesc()); } else { @@ -8159,60 +8722,6 @@ HRESULT BlockVector::CreateResource( return hr; } -#ifdef __ID3D12Device8_INTERFACE_DEFINED__ -HRESULT BlockVector::CreateResource2( - UINT64 size, - UINT64 alignment, - const ALLOCATION_DESC& allocDesc, - const D3D12_RESOURCE_DESC1& resourceDesc, - D3D12_RESOURCE_STATES InitialResourceState, - const D3D12_CLEAR_VALUE* pOptimizedClearValue, - Allocation** ppAllocation, - REFIID riidResource, - void** ppvResource) -{ - ID3D12Device8* const device8 = m_hAllocator->GetDevice8(); - if (device8 == NULL) - { - return E_NOINTERFACE; - } - - HRESULT hr = Allocate(size, alignment, allocDesc, 1, ppAllocation); - if (SUCCEEDED(hr)) - { - ID3D12Resource* res = NULL; - hr = device8->CreatePlacedResource1( - (*ppAllocation)->m_Placed.block->GetHeap(), - (*ppAllocation)->GetOffset(), - &resourceDesc, - InitialResourceState, - pOptimizedClearValue, - D3D12MA_IID_PPV_ARGS(&res)); - if (SUCCEEDED(hr)) - { - if (ppvResource != NULL) - { - hr = res->QueryInterface(riidResource, ppvResource); - } - if (SUCCEEDED(hr)) - { - (*ppAllocation)->SetResource(res, &resourceDesc); - } - else - { - res->Release(); - SAFE_RELEASE(*ppAllocation); - } - } - else - { - SAFE_RELEASE(*ppAllocation); - } - } - return hr; -} -#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ - void BlockVector::AddStatistics(Statistics& inoutStats) { MutexLockRead lock(m_Mutex, m_hAllocator->UseMutex()); @@ -8254,7 +8763,9 @@ void BlockVector::WriteBlockInfoToJson(JsonWriter& json) json.ContinueString(pBlock->GetId()); json.EndString(); + json.BeginObject(); pBlock->m_pMetadata->WriteAllocationInfoToJson(json); + json.EndObject(); } json.EndObject(); @@ -8507,13 +9018,15 @@ HRESULT BlockVector::CreateBlock( m_HeapFlags, blockSize, m_NextBlockId++); - HRESULT hr = pBlock->Init(m_Algorithm, m_ProtectedSession); + HRESULT hr = pBlock->Init(m_Algorithm, m_ProtectedSession, m_DenyMsaaTextures); if (FAILED(hr)) { D3D12MA_DELETE(m_hAllocator->GetAllocs(), pBlock); return hr; } + m_hAllocator->SetResidencyPriority(pBlock->GetHeap(), m_ResidencyPriority); + m_Blocks.push_back(pBlock); if (pNewBlockIndex != NULL) { @@ -8843,8 +9356,8 @@ bool DefragmentationContextPimpl::IncrementCounters(UINT64 bytes) // Early return when max found if (++m_PassStats.AllocationsMoved >= m_MaxPassAllocations || m_PassStats.BytesMoved >= m_MaxPassBytes) { - D3D12MA_ASSERT(m_PassStats.AllocationsMoved == m_MaxPassAllocations || - m_PassStats.BytesMoved == m_MaxPassBytes && "Exceeded maximal pass threshold!"); + D3D12MA_ASSERT((m_PassStats.AllocationsMoved == m_MaxPassAllocations || + m_PassStats.BytesMoved == m_MaxPassBytes) && "Exceeded maximal pass threshold!"); return true; } return false; @@ -9174,8 +9687,10 @@ PoolPimpl::PoolPimpl(AllocatorPimpl* allocator, const POOL_DESC& desc) desc.MinBlockCount, maxBlockCount, explicitBlockSize, D3D12MA_MAX(desc.MinAllocationAlignment, (UINT64)D3D12MA_DEBUG_ALIGNMENT), - desc.Flags & POOL_FLAG_ALGORITHM_MASK, - desc.pProtectedSession); + (desc.Flags & POOL_FLAG_ALGORITHM_MASK) != 0, + (desc.Flags & POOL_FLAG_MSAA_TEXTURES_ALWAYS_COMMITTED) != 0, + desc.pProtectedSession, + desc.ResidencyPriority); } PoolPimpl::~PoolPimpl() @@ -9300,7 +9815,7 @@ ULONG STDMETHODCALLTYPE IUnknownImpl::Release() { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - const uint32_t newRefCount = --m_RefCount; + const uint32_t newRefCount = --m_RefCount; if (newRefCount == 0) ReleaseThis(); return newRefCount; @@ -9423,7 +9938,7 @@ Allocation::Allocation(AllocatorPimpl* allocator, UINT64 size, UINT64 alignment, m_Size{ size }, m_Alignment{ alignment }, m_Resource{ NULL }, - m_CreationFrameIndex{ allocator->GetCurrentFrameIndex() }, + m_pPrivateData{ NULL }, m_Name{ NULL } { D3D12MA_ASSERT(allocator); @@ -9503,7 +10018,7 @@ NormalBlock* Allocation::GetBlock() } template -void Allocation::SetResource(ID3D12Resource* resource, const D3D12_RESOURCE_DESC_T* pResourceDesc) +void Allocation::SetResourcePointer(ID3D12Resource* resource, const D3D12_RESOURCE_DESC_T* pResourceDesc) { D3D12MA_ASSERT(m_Resource == NULL && pResourceDesc); m_Resource = resource; @@ -9573,20 +10088,20 @@ void Pool::GetStatistics(Statistics* pStats) { D3D12MA_ASSERT(pStats); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->GetStatistics(*pStats); + m_Pimpl->GetStatistics(*pStats); } void Pool::CalculateStatistics(DetailedStatistics* pStats) { D3D12MA_ASSERT(pStats); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->CalculateStatistics(*pStats); + m_Pimpl->CalculateStatistics(*pStats); } void Pool::SetName(LPCWSTR Name) { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->SetName(Name); + m_Pimpl->SetName(Name); } LPCWSTR Pool::GetName() const @@ -9664,7 +10179,12 @@ HRESULT Allocator::CreateResource( return E_INVALIDARG; } D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - return m_Pimpl->CreateResource(pAllocDesc, pResourceDesc, InitialResourceState, pOptimizedClearValue, ppAllocation, riidResource, ppvResource); + return m_Pimpl->CreateResource( + pAllocDesc, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialResourceState, pOptimizedClearValue), + ppAllocation, + riidResource, + ppvResource); } #ifdef __ID3D12Device8_INTERFACE_DEFINED__ @@ -9683,10 +10203,42 @@ HRESULT Allocator::CreateResource2( return E_INVALIDARG; } D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - return m_Pimpl->CreateResource2(pAllocDesc, pResourceDesc, InitialResourceState, pOptimizedClearValue, ppAllocation, riidResource, ppvResource); + return m_Pimpl->CreateResource( + pAllocDesc, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialResourceState, pOptimizedClearValue), + ppAllocation, + riidResource, + ppvResource); } #endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ +HRESULT Allocator::CreateResource3( + const ALLOCATION_DESC* pAllocDesc, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_BARRIER_LAYOUT InitialLayout, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + UINT32 NumCastableFormats, + DXGI_FORMAT* pCastableFormats, + Allocation** ppAllocation, + REFIID riidResource, + void** ppvResource) +{ + if (!pAllocDesc || !pResourceDesc || !ppAllocation) + { + D3D12MA_ASSERT(0 && "Invalid arguments passed to Allocator::CreateResource3."); + return E_INVALIDARG; + } + D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK + return m_Pimpl->CreateResource( + pAllocDesc, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialLayout, pOptimizedClearValue, NumCastableFormats, pCastableFormats), + ppAllocation, + riidResource, + ppvResource); +} +#endif // #ifdef __ID3D12Device10_INTERFACE_DEFINED__ + HRESULT Allocator::AllocateMemory( const ALLOCATION_DESC* pAllocDesc, const D3D12_RESOURCE_ALLOCATION_INFO* pAllocInfo, @@ -9716,9 +10268,66 @@ HRESULT Allocator::CreateAliasingResource( return E_INVALIDARG; } D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - return m_Pimpl->CreateAliasingResource(pAllocation, AllocationLocalOffset, pResourceDesc, InitialResourceState, pOptimizedClearValue, riidResource, ppvResource); + return m_Pimpl->CreateAliasingResource( + pAllocation, + AllocationLocalOffset, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialResourceState, pOptimizedClearValue), + riidResource, + ppvResource); } +#ifdef __ID3D12Device8_INTERFACE_DEFINED__ +HRESULT Allocator::CreateAliasingResource1( + Allocation* pAllocation, + UINT64 AllocationLocalOffset, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_RESOURCE_STATES InitialResourceState, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + REFIID riidResource, + void** ppvResource) +{ + if (!pAllocation || !pResourceDesc || !ppvResource) + { + D3D12MA_ASSERT(0 && "Invalid arguments passed to Allocator::CreateAliasingResource."); + return E_INVALIDARG; + } + D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK + return m_Pimpl->CreateAliasingResource( + pAllocation, + AllocationLocalOffset, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialResourceState, pOptimizedClearValue), + riidResource, + ppvResource); +} +#endif // #ifdef __ID3D12Device8_INTERFACE_DEFINED__ + +#ifdef __ID3D12Device10_INTERFACE_DEFINED__ +HRESULT Allocator::CreateAliasingResource2( + Allocation* pAllocation, + UINT64 AllocationLocalOffset, + const D3D12_RESOURCE_DESC1* pResourceDesc, + D3D12_BARRIER_LAYOUT InitialLayout, + const D3D12_CLEAR_VALUE* pOptimizedClearValue, + UINT32 NumCastableFormats, + DXGI_FORMAT* pCastableFormats, + REFIID riidResource, + void** ppvResource) +{ + if (!pAllocation || !pResourceDesc || !ppvResource) + { + D3D12MA_ASSERT(0 && "Invalid arguments passed to Allocator::CreateAliasingResource."); + return E_INVALIDARG; + } + D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK + return m_Pimpl->CreateAliasingResource( + pAllocation, + AllocationLocalOffset, + CREATE_RESOURCE_PARAMS(pResourceDesc, InitialLayout, pOptimizedClearValue, NumCastableFormats, pCastableFormats), + riidResource, + ppvResource); +} +#endif // #ifdef __ID3D12Device10_INTERFACE_DEFINED__ + HRESULT Allocator::CreatePool( const POOL_DESC* pPoolDesc, Pool** ppPool) @@ -9736,7 +10345,7 @@ HRESULT Allocator::CreatePool( return E_INVALIDARG; } D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - * ppPool = D3D12MA_NEW(m_Pimpl->GetAllocs(), Pool)(this, *pPoolDesc); + * ppPool = D3D12MA_NEW(m_Pimpl->GetAllocs(), Pool)(this, *pPoolDesc); HRESULT hr = (*ppPool)->m_Pimpl->Init(); if (SUCCEEDED(hr)) { @@ -9753,7 +10362,7 @@ HRESULT Allocator::CreatePool( void Allocator::SetCurrentFrameIndex(UINT frameIndex) { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->SetCurrentFrameIndex(frameIndex); + m_Pimpl->SetCurrentFrameIndex(frameIndex); } void Allocator::GetBudget(Budget* pLocalBudget, Budget* pNonLocalBudget) @@ -9763,21 +10372,21 @@ void Allocator::GetBudget(Budget* pLocalBudget, Budget* pNonLocalBudget) return; } D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->GetBudget(pLocalBudget, pNonLocalBudget); + m_Pimpl->GetBudget(pLocalBudget, pNonLocalBudget); } void Allocator::CalculateStatistics(TotalStatistics* pStats) { D3D12MA_ASSERT(pStats); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->CalculateStatistics(*pStats); + m_Pimpl->CalculateStatistics(*pStats); } void Allocator::BuildStatsString(WCHAR** ppStatsString, BOOL DetailedMap) const { D3D12MA_ASSERT(ppStatsString); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->BuildStatsString(ppStatsString, DetailedMap); + m_Pimpl->BuildStatsString(ppStatsString, DetailedMap); } void Allocator::FreeStatsString(WCHAR* pStatsString) const @@ -9785,7 +10394,7 @@ void Allocator::FreeStatsString(WCHAR* pStatsString) const if (pStatsString != NULL) { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->FreeStatsString(pStatsString); + m_Pimpl->FreeStatsString(pStatsString); } } @@ -9816,8 +10425,7 @@ Allocator::~Allocator() BOOL VirtualBlock::IsEmpty() const { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - - return m_Pimpl->m_Metadata->IsEmpty() ? TRUE : FALSE; + return m_Pimpl->m_Metadata->IsEmpty() ? TRUE : FALSE; } void VirtualBlock::GetAllocationInfo(VirtualAllocation allocation, VIRTUAL_ALLOCATION_INFO* pInfo) const @@ -9825,8 +10433,7 @@ void VirtualBlock::GetAllocationInfo(VirtualAllocation allocation, VIRTUAL_ALLOC D3D12MA_ASSERT(allocation.AllocHandle != (AllocHandle)0 && pInfo); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - - m_Pimpl->m_Metadata->GetAllocationInfo(allocation.AllocHandle, *pInfo); + m_Pimpl->m_Metadata->GetAllocationInfo(allocation.AllocHandle, *pInfo); } HRESULT VirtualBlock::Allocate(const VIRTUAL_ALLOCATION_DESC* pDesc, VirtualAllocation* pAllocation, UINT64* pOffset) @@ -9839,7 +10446,7 @@ HRESULT VirtualBlock::Allocate(const VIRTUAL_ALLOCATION_DESC* pDesc, VirtualAllo D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - const UINT64 alignment = pDesc->Alignment != 0 ? pDesc->Alignment : 1; + const UINT64 alignment = pDesc->Alignment != 0 ? pDesc->Alignment : 1; AllocationRequest allocRequest = {}; if (m_Pimpl->m_Metadata->CreateAllocationRequest( pDesc->Size, @@ -9871,7 +10478,7 @@ void VirtualBlock::FreeAllocation(VirtualAllocation allocation) D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->m_Metadata->Free(allocation.AllocHandle); + m_Pimpl->m_Metadata->Free(allocation.AllocHandle); D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); } @@ -9879,7 +10486,7 @@ void VirtualBlock::Clear() { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - m_Pimpl->m_Metadata->Clear(); + m_Pimpl->m_Metadata->Clear(); D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); } @@ -9888,15 +10495,14 @@ void VirtualBlock::SetAllocationPrivateData(VirtualAllocation allocation, void* D3D12MA_ASSERT(allocation.AllocHandle != (AllocHandle)0); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - - m_Pimpl->m_Metadata->SetAllocationPrivateData(allocation.AllocHandle, pPrivateData); + m_Pimpl->m_Metadata->SetAllocationPrivateData(allocation.AllocHandle, pPrivateData); } void VirtualBlock::GetStatistics(Statistics* pStats) const { D3D12MA_ASSERT(pStats); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); + D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); ClearStatistics(*pStats); m_Pimpl->m_Metadata->AddStatistics(*pStats); } @@ -9905,7 +10511,7 @@ void VirtualBlock::CalculateStatistics(DetailedStatistics* pStats) const { D3D12MA_ASSERT(pStats); D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); + D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); ClearDetailedStatistics(*pStats); m_Pimpl->m_Metadata->AddDetailedStatistics(*pStats); } @@ -9916,11 +10522,13 @@ void VirtualBlock::BuildStatsString(WCHAR** ppStatsString) const D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - StringBuilder sb(m_Pimpl->m_AllocationCallbacks); + StringBuilder sb(m_Pimpl->m_AllocationCallbacks); { JsonWriter json(m_Pimpl->m_AllocationCallbacks, sb); D3D12MA_HEAVY_ASSERT(m_Pimpl->m_Metadata->Validate()); + json.BeginObject(); m_Pimpl->m_Metadata->WriteAllocationInfoToJson(json); + json.EndObject(); } // Scope for JsonWriter const size_t length = sb.GetLength(); @@ -9935,7 +10543,7 @@ void VirtualBlock::FreeStatsString(WCHAR* pStatsString) const if (pStatsString != NULL) { D3D12MA_DEBUG_GLOBAL_MUTEX_LOCK - D3D12MA::Free(m_Pimpl->m_AllocationCallbacks, pStatsString); + D3D12MA::Free(m_Pimpl->m_AllocationCallbacks, pStatsString); } }