Captain's Diary #56: How we made products rendering 20× faster
- 51 minutes ago
- 11 min read
Welcome everyone to today's edition of Captain's diary. I am Captain Marek and today I will be talking about some serious performance improvements we've done over the past few months. Are you ready for 20x faster products rendering?
Before that, I have an important announcement. Update 4.2 is nearly finished and will be released on the 20th of July! The key new features to look forward to are:
An in-game COI Hub integration for mods, maps, and blueprints.
Universal wagons for trains that can carry all 3 major types of cargo.
More train track pieces like 2-tile s-bends or grade changes.
Train waypoints with allow/deny options.
Ability for trains to autonomously go forward and reverse on bi-directional tracks if that's the shortest path.
New statistics for all products showing detailed usage breakdown.
Performance optimizations described in today's diary.
And more!
We will showcase these features as we get closer to the update launch.
Products rendering optimizations
All products in Captain of Industry flowing through your factory are rendered, either on conveyor belts, in storages, or on vehicles and trains. This is awesome to look at, but performance was taking a hit, especially on weaker machines that were running late-game saves with 100k+ products. Now that computer hardware prices are through the roof, wouldn't it be nice to be able to build even larger factories without needing to upgrade your computer?
From the early days, our products rendering was already quite optimized. Unity's default approach of managing things in the game, Game Objects, is completely unusable for this purpose due to CPU overhead, GPU inefficiencies and also high RAM usage. So we've been using GPU instancing which allows us to render hundreds of thousands of individual products very efficiently. But, as other performance bottlenecks were removed, our previously fast products renderer was taking up more and more of our overall frame time. We did some analysis and found our old approach was not sufficient for the following reasons:
No LODs: Product 3D models were fixed and rendered the same from all distances. Using simplified 3D models at a distance can reduce GPU load with little visual fidelity loss.
No distance culling: If a product was in the frame, it was rendered, even if it was super far away. This used to not be an issue when COI started, our maps were small enough that you could see products across the entire map. Since we now have much larger maps, this is no longer the case.
Too many draw calls: Every storage was rendering products separately, one draw call per storage. We thought that given that a storage renders lots of products at once, this would not be a problem, but our profiling showed that it was, especially for players who like to stockpile resources.
Different handling of all product sources: conveyors, vehicles, and storages all had separate handling of products. This introduced inefficiencies.
Large memory bandwidth: Each product was using 20-64 bytes per frame of CPU=>GPU bandwidth. This may not sound like much, but with 100k products rendering at 60 FPS, that is up to 366 MiB/s transfer rate just for products, which is very significant.
Optimizations in a nutshell
We've done a lot of optimizations, and some are fairly technical, so here is the gist.
1. Unified renderer
To be able to benefit from all optimizations and simplify our code, all three custom product renderers for conveyors, vehicles, and storages were replaced with a unified one.
This alone was a big win for draw calls. For example, every storage used to have its own renderer, costing at least one draw call per storage, even when many storages were storing the same product.
The unified renderer issues just one draw call per unique mesh, no matter where the products are: on a belt, in a storage, or on a vehicle. All product textures live in one shared texture array, so each product instance just carries a small texture index, and products that share a mesh render together even if they look completely different. We even render storage racks as if they were products, which collapsed per-storage rack draw calls into roughly one per rack mesh. The end result: draw calls no longer grow with the number of storages or the variety of products, only with the number of unique meshes. And as you will see below, we made sure there are very few of those.
2. Optimized meshes
The first very significant optimization was to simplify product 3D models. Some products had over 200 triangles which is not acceptable on our scale. Our new target was sub 50. Redoing all products has been a big undertaking, especially with the significant constraints on polygon count. Two 3D artists spent over 8 weeks full-time just working on the new products. Some products went through 5+ iterations just to get them looking great and fitting our polygon budget.
In the end, we have not only optimized all products (some have 90% fewer polygons), but also made them look way nicer. Below is an example of some products and how they evolved.
3. LODs
On the surface, level-of-detail (LOD) optimization sounds straightforward. Render a simplified mesh at distances where the degraded quality is barely noticeable. However, this brings a lot more complexity, since the LOD level for each product has to be determined and rendered.
Until now, we had one unique mesh per product. If we just slapped 2 LODs on each product, say 200 => 50 => 12 triangles, and called it a day, it would not necessarily improve performance due to the draw call overhead.
Here is the catch: The GPU has so much parallel capacity that drawing 1 product or 1000 products takes virtually the same time, because the hardware is nowhere near saturated. Now let's add LODs to those 1000 products. Based on its distance from the camera, each product now uses one of three meshes, so instead of one draw call with 1000 products we need three draw calls with roughly 330 products each. The number of rendered triangles went down, but we now have three draw calls which can easily end up being slower than rendering everything at full detail in a single call. LODs only pay off when each LOD level contains enough products for the saved rendering work to outweigh the overhead of the extra draw calls.
There are multiple ways to address this. Advanced techniques exist to draw multiple different meshes in a single draw call, but they come with limitations that made them a poor fit for us. Instead, we have decided to actually reduce the number of unique meshes. We have categorized all products into a small set of mesh families: three core shapes (box, barrel, and puck) plus a few variants such as crates (which turn into plain boxes at distance), rough and smooth piles for loose products, and molten metals. That is around 11 families in total.
This categorization meant that the look of some products had to be slightly compromised, but we were able to reduce 100+ unique meshes down to just 21. Only around 10 products keep their own unique close-up mesh, but share all the distant LODs. This was a huge win performance-wise.
By the way, this is also good news for modders: mesh families are regular game protos. Modded products can join one of the built-in families to get all the LODs and optimizations for free, bring their own family, or override any individual LOD mesh.
4. Stack LOD
All products in the game can stack up to a quantity of 3. We use this technique to scale conveyor belt throughput and visually fill space with products on vehicles and storages. The drawback is that each stack is 3 separate instances. For example, a single tier 4 unit storage shows 864 individual products when full.
The idea is that from a certain distance, we can fuse the stack of 3 products into a single mesh, saving CPU processing time (3× fewer instances to pack), memory bandwidth (3× fewer instances to transfer), and some GPU time too (fewer triangles to render). Now, due to the quirk of instancing efficiency on low instance counts, this would not have worked on its own, but with shared meshes and shared renderers, we are able to collect enough 3-stacks for this to make a significant difference.
5. Reduction of memory per product
Before our optimizations, a static product needed 20 bytes of data to be rendered, per frame. This is actually not that much: a 3D position with a compressed rotation takes 16 bytes, plus an extra 4 bytes for other data about the product such as scale or texture randomization for loose products. Dynamic products were more expensive: they need two poses to interpolate between, and products on vehicles also need a full rotation (32 bytes per pose), bringing them to 40-64 bytes per product.
This memory footprint hurts in two ways. First, the CPU has to pack and assemble it. So all this memory has to flow through CPU caches and registers. Then, once assembled, it has to be sent to the GPU. Due to quirks of Unity (more on that in a bit), there is one extra CPU=>CPU memory copy of the entire data followed by the CPU=>GPU copy. With 100k products at 60 FPS, this is up to 366 MiB/s CPU=>GPU transfer rate and twice that within the CPU.
This issue has been addressed by uploading all possible poses of all products to the GPU in one large static buffer, and just using a simple index to reference them. This reduced 20-64 bytes per product to 8/16 bytes (static/dynamic). Moreover, we only use 3 bytes for the index and the other 5 hold lots of extra data like stacking noise, stack offset, texture index, etc.
And as a cherry on top, there were a few bits left in the data structure, so instead of using 8+8 bytes for dynamic products (to interpolate between them), we packed the pose index delta there, so now all static AND dynamic products just take 8 bytes per instance.
Playing with bits and reducing a few bytes here and there may sound kind of boring, but memory bandwidth is no joke and this change alone resulted in significant perf gains.
And now the promised Unity quirk. Unity has an API to write instance data directly into GPU-visible memory (LockBufferForWrite), skipping the extra CPU=>CPU copy that the regular SetData API does. Naturally, we used the direct-write path, expecting easy savings. Well, when we measured it, it was a steady ~1.7 ms per frame SLOWER than the good old SetData, about 10% of the frame time in our test scene! I am convinced this is some issue on Unity's side, but without access to their source code I cannot tell for sure. The lesson: never assume, always measure.
6. Zero pop-in culling
An obvious optimization is to not render products that are not on the screen (frustum culling). But there is a catch. Products are packed and culled on the simulation thread which runs at 10 updates per second. So the set of visible products can be up to 100-200 ms behind the current camera. If we culled products naively, a quick camera pan would show empty conveyor belts on the edge of the screen for a few frames, before the products pop in. It may sound minor, but it is very noticeable: products constantly pop in around the screen edges when the camera is moving.
The usual solution is to cull each frame, but that is expensive and complicated given that products have to end up in contiguous buffers for the GPU to render efficiently.
Our solution is to use two data streams. Products inside the camera view are packed into the main stream, as expected. But products outside the view that are still within render distance are not thrown away. Instead, they are packed into a second, "reserve" stream, together with a small directory that records each entry's bounding box in the world. Every render frame, this reserve directory is tested against the live camera, and entries that just became visible are copied over to the main stream. The test is hierarchical, a coarse world grid first and individual entries second, so it usually takes less than 0.1 ms. The result: you can pan and rotate the camera as fast as you like, and products are simply always there.
And when nothing changes (camera is still), all of this per-frame work is skipped entirely, so an idle scene costs virtually nothing.
7. And more...
There have been many more optimizations that I won't be diving into here. Here is a laundry list, and if you'd like to read some low-level details, maybe I can write a deep-dive blog post later.
Optimized products packing on sim thread.
Replay instead of repack for products that did not change.
Two-level indirection to efficiently support products on vehicles and trains.
Per-segment LODs on long conveyor belts.
Reduced update frequency on distant conveyors.
Conveyor belts with congruent shapes share their pose tables (on both CPU and GPU), saving CPU and GPU memory on large maps.
Tiny meshes get several co-located copies baked into one mesh, so a single GPU instance draws multiple products at once.
Results
The results were evaluated across multiple saves and computers. First, let's look at the torture map: ALL products in the game with lots of moving conveyor belts. This map was running at a mere 23 FPS but after all the optimizations we got it to 78 FPS! That's over 3× improvement.
FPS as a metric can be deceiving, so let's compare absolute frame times. With no products present, a frame took 11.1 ms (same for both the old and new renderer). With all the products, the old renderer's frame time was 43.4 ms, so 74% of the frame time was spent on the products. With the new renderer, the frame time went down all the way to 12.8 ms, so just 13% is being spent on products (despite them covering the entire screen!). That means the products rendering itself went from 32.3 ms to just 1.7 ms. That's nearly a 20× speedup!
The speedup also depends on your hardware, especially your GPU. It turns out that on Jeremy's machine (RTX 3xxx) and Matt's computer (GTX 1xxx) the improvements were even more impressive compared to my machine (RTX 4xxx). In general, the worse the computer you have, the more speedup you can expect.
Now let's take a look at some real save games from the community. From our test, you can see overall FPS uplift roughly from 10% to 90%, depending on the map, camera view, and your computer.
And finally, let's take a look at speedups based on the graphics fidelity preset. Many of the described techniques scale based on the preset. We've introduced a new option, "Products texture quality", that controls the resolution of product textures, and it is also folded into our overall presets. The chart below shows that the new renderer is very efficient across all presets.
Bonus: terrain simulation optimization
We are not done with optimizations. By completely slashing products rendering time, other systems became the next contenders to tackle.
For example, we've also optimized terrain simulation. Even if you are not thinking about it, there is always some terraforming going on around the map and these terrain changes do interact with many systems: vehicle path finding (ground or sea), buildings, designations fulfillment, trees, renderers and many more.
By improving how these events flow through all the subsystems, we were able to significantly reduce the time spent processing them, resulting in a 5-10x speedup that cuts down the game lag caused by terrain avalanches.
Wrap-Up
And that's a wrap! This was one of the bigger performance projects we have done, touching everything from GPU buffers and shaders all the way to remaking every product 3D model in the game. The result is up to 20x faster products rendering that benefits everyone: from older laptops running modest factories to megabases.
Big thanks to Amy, Flickerra, JD Plays, and all others from our wonderful community who shared saves with us.
All these optimizations arrive with Update 4.2 on July 20th, so you can put your factory to the test very soon. Captain Marek out!