8.2K Views
July 24, 24
スライド概要
Unreal Fest Gold Coast 2024で講演したスライドです。開発イテレーションに関する内容について新しい機能やメカニズムを紹介しています。
Unreal Engineを開発・提供しているエピック ゲームズ ジャパンによる公式アカウントです。 勉強会や配信などで行った講演資料を公開しています。 公式サイトはこちら https://www.unrealengine.com/ja/
Unreal Fest Gold Coast 2024 Improving Development Iterations: A Deep Dive into Enhanced Feature Ken Kuwano Software Engineer, Developer Relations Epic Games Japan
Note: ● This session will cover the content of UE5.4 (May 2024) ● This session will introduces features that are still under development and may be updated in the future
Summary You will be able to do… ● ● ● ● Gain a better understanding of the features that can improve your development iterations Accelerate routine iterations such as cooking and packaging by applying them to your projects Check it quickly on the target device and save time to adjust the look and feel Consider yourself the best solution for your own project
Why improve development iterations?
Recent Trends Large projects Number of assets increases Increased map size Increased synchronization and load times Storage requirements ● ● Remote-work facilitates Difficult to get to the office Work the same way from anywhere Cloud computing Leverage build, storage, and virtualization environments WorldPartition, Nanite, Lumen, etc… in UE5 Our development environment is getting better, even better content is required
Customer Feedback Packaging takes a long time, so verification on actual equipment is time consuming Project synchronization takes a long time from the beginning Packaging is very long, but I don't know why Cook process takes a long time, but I don't know how to shorten it Long loading times in the editor, can't load assets in the background Even if I change one file, it takes a long time before I can check it on the actual device There is a long waiting time when executing in PIE Increased number of files takes time to transfer files to the actual device for confirmation It is difficult to build for all platforms with multiple build configurations Insufficient disk space because of increasing number of files to handle Long shader compilation time if don not have DDC Tight disk space, including project resources The editor is slow to start up and open maps Slow project sync due to the large number of files OFPA files Long time to build projects, both compiling and linking
Creating better products require further improvement of development iterations
Agenda ● Sync ● Build ● Cook ● Load
● Agenda Sync ○ Virtual Asset ○ Unreal Cloud DDC ● Build ● Cook ● Load
Virtual Assets Virtual Asset Structured Data Production Ready for UE5.3 ● ● ● ● ● Splitting assets into two pieces of data reduce disk space and increase synchronization efficiency The larger project size, the faster and more efficiently data can be synchronized between members Faster synchronization process by reducing the number of synchronizations Works well for projects that use a lot of textures and meshes On Ancient Game, first sync saves x50 disk space Perforce 11000101010 00010101010 0101 FA1C238E //Game/Content Asset .uasset Bulk Data Perforce FA1C238E - payload 1101010101001 0101010100101 1100101010101 10010101111010 1010101010101 0100101010100 110010101010 //Payloads DDC
Asset Structure Virtual Aaaset Texture Asset Texture Asset Structured Data (.uasset) Properties and other data published in the editor Components required as an asset (Required Data) .uasset Bulk Data (.upayload) Pixel-by-pixel rendering data and other data for display Components required for texture (Additional Data) By separating and managing assets, which are smaller data required for updating, according to their use
Virtual Assets Operation Flow //Game/Content //Payloads download //Payloads check out pull //Payloads check in Structured Data (.uasset) Bulk Data (.upayload) Sync PIE Edit Submit
Notes: Virtual Assets ● ● ● ● ● ● ● Network connection required, effective in high-speed network Use Perforce for Source Control Apply to new project, but be careful applying to projects under development Effectiveness varies depending on the project Even if you "only virtualize huge assets", you can benefit from it Less likely to benefit if not dealing with large size assets Small scale, small number of people, and single development are especially difficult to benefit from
Region: US - East Region: EU - West Accelerate Data Synchs Unreal Cloud DDC Production Ready for UE5.4 Bring your data closer to distributed teams • General content addressable storage • Active & passive region replication • Auto-connect to closest region • OIDC login & authentication • Don’t build what has already been built! Region Replication Region: Asia Perforce Sync Cloud DDC Studio Shared DDC • Source and docker image on Github • Production tested on AWS • Instructions available for Azure Local DDC Local DDC Local DDC Remote Users
UnrealCloudDDC document https://github.com/EpicGames/UnrealEngine/tree/5.4/Engine/Source/Programs/UnrealCloudDDC
DDC ● ● ● ● ● Function to cache derived data (real data, etc.) of textures and meshes Accelerates access to large amounts of data by caching them in advance Cache is checked when each asset is loaded, and if it is not there, it is created If there is one, the cache is used to expand the data File access to LocalDDC is faster than SharedDDC
Saved Data in DDC Static Mesh Optimized vertex buffers and index buffers UV, LOD resources, Nanite resources, etc... Skeletal Mesh LOD resources Texture Output of compression tools (DXT binary) Material Compiled shadermap Shader bitecode Audio Streaming audio resources
Agenda ● Sync ● Build ○ Unreal Build Accelerator ● Cook ● Load
Unreal Build Accelerator (UBA) Beta for UE5.4 ● UBA is a local and distributed compilation solution for C++ Compilation used in conjunction with Unreal Build Tool and/or Horde’s Remote Build (compute task) system ● Horde is the coordinator to assign helper resources. UBA transmits local data, executes the remote task and returns the results ● ● Win64 platform + C++ compilation job Recommended for deployment infrastructure on the same physical and private network UBA
Horde Documentation https://github.com/EpicGames/UnrealEngine/blob/5.4/Engine/Source/Programs/Horde/Docs/Tutorials/RemoteCompilation.md
Distribution Build Steps Horde Server Build Machine UBA Local PC Horde Agent UBA Remote PC
Distribution Build Steps Horde Server 1) UBT request a Compute Task to the Horde Server Build Machine Local PC Remote PC
Distribution Build Steps 2) Horde Server allocates helpers to the requesting job (new or existing) Horde Server Build Machine Local PC Horde Agent Remote PC
Distribution Build Steps Build Machine UBA Local PC Horde Agent 3) UBA on Compile Agent uploads itself to Helper Agents Remote PC
4) UBA Helpers connect to UBA on the Compile Agent to assist with the Compile operation Distribution Build Steps Build Machine UBA Local PC UBA Remote PC
UBA Visualizer
Tips https://forums.unrealengine.com/t/unrealbuildaccelerator-uba/1826434
Agenda ● Sync ● Build ● Cook ● ○ Multiprocess Cooking ○ Iterative Cooking ○ Incremental Cooking Load
Cooking in packaging flow Build Cook Load Assets LoadPackage Stage Package Save Assets Serialize Archive
Cooking in packaging flow Build Cook Stage .uasset .uasset .uexp .ubulk Package Archive UAsset Files Package Header data, load first file when the package is loaded Export Files Package archive data, load package import/export data when deploying Bulk Files Stores the actual data of the asset, loads platform-specific data, map data, etc. when needed
Multi-Process Cook Production Ready for UE5.4 Faster, parallel cooks • Leverage available CPU cores and memory resources by directing the cook process across multiple workers • Set the following parameter in Editor.ini [CookSettings] CookProcessCount=4 Or, set the cooking argument -CookProcessCount=4 Cook Director Cook Worker 1 Cook Worker 2 Cook Worker 3 Cook Worker 4
Cooking CitySample with 16 workers Cook Director Cook Worker 1 Cook Worker 2 Cook Worker 3 Cook Worker 4
Multiprocess cook architecture LoadPackage InQueue UnrealEditor-Cmd LoadPackage ForCooking Cook Director WorkerServer0 LoadPackage WorkerClient0 UnrealEditor-Cmd WorkerServer1 WorkerClient1 Load Serialize Cooking UnrealEditor-Cmd WorkerServer2 WorkerClient2 UnrealEditor-Cmd WorkerServer3 Create, Tick, Assign Launch, Snd, Rcv WorkerClient3
Multiprocess cook effectiveness (CitySample)
Cook Iterative Cooking Collect Files ● ● ● ● Iterative-cooking is an alternative to full-cooking Cooking with -iterative arguments Iterative-cooking is parameterized by the results of a previous cook Verify if there are cooked assets in /Saved/Cooked/ directory Requests exists? N Y Cooked Asset already exists? N Cooking Saving AssetRegistry Y
Iterative Cooking x14.0 faster Cooking time reduced from 815 to 58 s (12m37s saved)
Incremental Cooking ● Future optimization aims to incorporate incremental changes to config settings and code, minimizing redundant data re-cooking ● Limited handling of scenarios like partial map file changes, occasional false positives in cooking list ● Future focus on "incremental cooking" to reduce false positives in verification while maintaining iterative guarantees TBD: Iterative Cooking Loading Flow
Agenda ● Sync ● Build ● Cook ● Load ○ Zen Store ○ Zen Storage ○ Zen Loader
Zen Serise ● Zen Store ○ Feature to store cache data ● Zen Storage ○ Location of the cache data ○ Stored here if Zen Store is enabled ● Unreal Editor Zen Server Caching Zen Server ○ Application that manages the cache ○ Supports writing to and reading from Zen Storage Zen Storage Zen Store ● Zen Loader, Zen Cooker, Zen Snapshot, Zen Streaming, etc…
Zen Server ● ● ● Manage multiple data outside of editor Used to manage DDC locally Set the following parameter in Engine.ini [Zen] AutoLaunch=true https://github.com/EpicGames/zen
ZenServer sibling process launched by UnrealEditor UnrealEditor ● UnrealEditor can launch and manage lifetime of a local instance of ZenServer ● Hosts multiple distinct storage services ● Separate codebase with support for all our editor platforms Zen Server
Zen Storage as Local DDC Production Ready for UE5.4 DDC ● Replace use of local filesystem DDC ● Deduplicate DDC contents across workspaces on a machine ● Gives us more control over data caching than when using the filesystem directly UnrealEditor Resource ZenServe r DDC data
Zen Storage as Shared DDC Experimental for UE5.4 ● Replace use of network file shares as DDC storage ● Provide consistency across editor platforms DDC UnrealEditor Resource ZenServe r UnrealEditor UnrealEditor DDC data
Zen as Cooked Output Beta for UE5.4 UnrealEditor Cooked Assets Assets ZenServe r Target Device Cooked Assets Data ● Avoid writing to large numbers of loose files when cooking ● Opens the door to efficient network data streaming to target platforms
Comparison applying Zen Store (on PS5) Launch Application (s) Load Map (s) x1.8 faster x3.1 faster
Zen Loader ● ● ● ● New asset loading system with efficient File I/O and less blocking Shorter time to start and manipulate the Editor More efficient asset loading and faster map loading Set the following parameters in Engine.ini Game Thread Comp lete Wait Load Request Async Loading Thread2 Load IoService Wait Comp lete Read Seek/Read Data .ucas .uasset [/Script/Engine.EditorStreamingSettings] s.ZenLoaderEnabled=True s.AsyncLoadingThreadEnabled=True
Loading WireframeMaterial with Zen Loader is disabled (30.6ms) Blocking load with GameThread
Loading WireframeMaterial with Zen Loader is enabled (12.2ms: approx. x2.5 faster) Asynchronous DDC access
Testing for Zen Loader Launch Editor(s) x1.1 faster Open Map(s) x1.8 faster Start PIE(s) x3.1 faster
Conclusion ● ● ● ● Gain a better understanding of the features that can improve your development iterations Accelerate routine iterations such as cooks and packaging by applying them to your projects Check it quickly on the target device and save time to adjust the look and feel Consider yourself the best solution for your own project
Thank you!