Development Updates

[Zippy]

3 hours ago, appstrader said:

I saw this thread was 999 posts it would of been rude to scroll past and not make it a 1000 posts,

Keep up the great work jp !

Ya big softy!!!

[bullionbars2011]

Fantastic work there m8 one of a kind for mfme this

[johnparker007]

Tech update:  I've done some rough benchmarking to identify current main problem areas (i.e. the areas eating up the most precious CPU cycles), the main three being:

Lamps (Yet to further optimise)
Even though I've got these doing fancy custom GPU instanced rendering stuff, it's just not enough, I need to go deeper on these...

For context, if you can see 100 machines at once playing their attract modes (in a very large arcade with an open area in the middle), and if we say on average a machine has ~200 lamps, and if we guess that around half of those lamps are on at any moment, then we have 100 machines x 100 lamps, which gives us 10,000 'blended' lamps to draw per frame (in Arcade Sim, every single lamp is actually a blended lamp).  So this is (one of) the main reasons why the CPU usage currently massively ramps up when there are lots of machines on-screen running attracts.  If I can get this 2-3x faster overall, it'll be a real nice big saving  ...more info to follow in a future tech update, I do have an idea...  

Cabinet model geometry (Properly optimised today)
The big job was creating a 'MeshCombiner' - when a machine loads, this takes its linked meshes that share the same material from my 3ds Max export, and combines them to one mesh (with some other shenanigans).  This reduces the work the CPU has to do to render each machine cabinet model. 

Before it was saying: 'Hey GPU, here's the left side panel of the JPM vogue in grey, draw that', 'Hey GPU, here's the right side panel of the JPM vogue in grey, draw that', etc.  Now, for each common material (wood veneer, grey metal etc) per machine, it groups it all into one mesh at load time, to pass to the GPU.

Seven segment / segment alpha displays (Workaround optimised today)
These displays, along with the dot alpha, are just 'first pass' approaches at rendering them well/efficiently.  The dot alphas are insanely bad!   I just needed to get something initially working.  So for those very inefficient dot alphas I already had stuff set up to only render them when you are actually playing on their machine.  Otherwise (in attract mode) their renderers are disabled.  I've copied that workaround fix over to the seven segment displays, and the segment alpha displays.

The long term plan with these is probably going to be based on what I'm hoping to achieve with the lamp renderer rewrite, so then they will render with no dependency on Unity's game object/matrix transformation system (which chews up CPU).

The lamp renderer rewrite - general idea is to leverage the parallel processing of the GPU along with reducing CPU -> GPU traffic - it's currently just a totally experimental idea floating around the old noggin, but I believe this will yield fairly big CPU savings if I can pull it off - so worth a week or two's work   If it doesn't work, hopefully the knowledge gleaned may reveal another approach.

The results of my optimisations/workarounds so far:

Here's a still image from my most recent video, in a demanding spot, showing the FPS/CPU usage:


And since my changes, I ran the build again, and navigated to the same spot, with the same amount of the new colored lights, and here's a grab of that:

Approximate average CPU usage down from ~68% to ~55% - I am on the ale now to celebrate the improvements so far  

Edited May 15, 2021 by johnparker007

[serene02]

Nice optimisations there bud,  enjoy your ale

J

[Zippy]

No drinking on the job!!!!!!!!!!!!!!!!!!!!!!!!!!!

[johnparker007]

Not meant to be working on the arcade today, but really wanted to at least get a start on the experiments for the new lamp rendering idea (that should use much less CPU).

Here's a still of a test, with a standard cube with a texture on it, correctly obscuring the colored test quads behind it (as in the arcade, we'll want the lamps to be obscured if there's a machine partially blocking the view).

The colored test quads ae randomly generated in a large cubic volume - there's one miiiillion of them!  The hope is that I can get all the quad lamps rendering via this new approach.  Counter-intuitively, before I was turning off lamps when they were off, but now I think with the new approach it'll be faster to leave them all on, and just render fully transparent when they're off.  

And here's a pretty hypnotic video of flying through the 'cube cloud', just to show how nice and smooth it is (though youtube compression has made it look a bit jerky and blurry), even with a million quads:

 

[Zippy]

Good job I didn't watch that while I was pissed.

Anyone for a shot of Whiskey this morning?

Edited May 16, 2021 by Big J

[serene02]

 Nice one JP!!   I love the tech updates

J

[johnparker007]

Done a bit more on this, just wanted to see about changing a property of each quad each frame.  In this test I'm changed the red, green, and blue value to random values for every quad, every update  

Had to take it down from a million to 'just' 100,000 - as the code setting deriving 3,000,000 random numbers (1M * 3 for r,g,b)  and writing them per frame was slowing things down a bit!  This isn't a real bottleneck, as with the 'attract mode codec' (not yet developed), then each frame, I will only be writing the lamps that have changed.  Even during flashy attract modes, it's usually only changing about 5 lamps every few frames per machine (so just setting 1 'alpha' value per changed lamp per frame).

So now the quads are 'twinkling' and when they come close you can see their colors all independently changing very quickly.  If you pause the video at any point when you're 'inside the cloud', you can see the quads all just have a random flat color each.  All silky smooth and sharp in real life, but these new test videos are getting destroyed by youtube's compression, so you'll just have to take my work for it   

 

[vectra666]

.... space..... the final frontier 

looks very Trekkie 

[johnparker007]

Worked out a bit more on this new quad rendering technique - got some basic texture data in there now



lol have just realised that since I'm using a '9' lamp image from my old Tuppeny Gambler test, I've accidentally made the '69' universe  

Edited May 16, 2021 by johnparker007

[johnparker007]

Tech update on the new (faster) lamp renderer research  

So I've made some progress, though I've also found a limitation that I'll probably have to work around.

In this test on Eastern Promise, I'm using this atlas texture, that contains all the on lamp images (there's duplicate images on there, that's for future work on blended lamps to sort out):

The texture needs to be like this with all the lamps combined, so then I can hard call the GPU to parallel process draw the output - so it has lots of different cores all drawing lamps at the same time for a speed up.

Then in Unity game engine, I'm bypassing Unity's built-in rendering, and calling the GPU to draw the lamps directly via compute buffers/shader - this is another big speed up over the current system.

And as a test, I have hardcoded a region from that atlas sheet above in the new shader so the GPU can draw a small section, to render a '6' lamp image from the bottom glass, where each lamp is:



The positions seem right, though I'm missing something with the scales - the maths starts getting a bit bonkers with this stuff... hopefully I'll crack that later.

So the limitation I've found is that due to how I'm doing this, the GPU draws things in pretty much random order - so while transparency works to show the unlit backglass behind for any single isolated lamp... the problem is when lamps overlap.  As I have the scale wrong here, more lamps are overlapping, which is handy so at least I can see the problem clearly.

So on the two overlapping lamps outlined in pink, you can see there is a corruption.  The bottom lamp is drawn 'in front' of the top lamp, but is has drawn the unlit background in where they overlap.  In the area outlined in green, I got lucky and the GPU processed them in the order that meant that the mixing between the 'front' lamp and the 'back' lamp is non-corrupted.

In reality, most lamps on layouts don't overlap (though they sometimes may overlap in the 3d machine layout even if they don't in the MFME source layout, due to perspective correction and the fixed rectangular nature of texture coordinates).

As this drawing technique is significantly faster, I'll be sticking at this research.  I'm thinking I may be able to do a workaround of either:

(A) - breaking overlapping lamps into smaller sub rectangles where possible, to eliminate overlapping - in some cases, this will let me remove the overlapping area

(B) something called Bitonic sorting, though that looks pretty arcane, and there are people reporting that when they did manage to pull it off, it was actually quite slow, which defeats the object (as all this code is running on the GPU to get the speed up, which isn't made for sorting)

(C) a simpler approach - for all lamps involved in an overlap, I render those using the standard system I already have, and only use the fast renderer for the lamps which don't overlap

So firstly I'll just try to get it working more, complete with the overlap corruptions.  Then if it's as fast as I think it's going to be (allowing slower CPUs/more machines on screen at once), I'll probably end up doing a combination of (A), then for lamps that can't be done that way (C).

Also there's some blurriness, but I think that'll resolve itself one I have the scales stuff figured out more, and using the correct area of the big atlas texture per lamps (down to mip-mapping).

Edited May 17, 2021 by johnparker007

[johnparker007]

Still early testing stage with the new experimental lamp renderer, but I got it just far enough along that I could give it a stress test, to see if this is all going to be worth doing - and it's performed pretty well for an early hack test 

I'm fairly confident continuing on this path (fixing/working around the various graphical issues I've got left) should yield very worthwhile performance savings - which means a lower bar for entry in terms of what spec PCs can run big arcades with lots of fruities visible at once.

Here's a video of the stress testing, with as many machines on screen as I think is possible to actually squeeze in.  I stop at a couple of machines, so you can see they are all rendering the new test lamps (all permanently on), on top of the unlit backglass  (that still have some issues to sort out, but nothing insurmountable):

It's just running in the editor for now (will actually go a little bit faster in a proper build, as the editor adds some overhead) - frame rate never drops below a solid 60FPS  

CPUGPU usage goes up when I put lots of colored room lamps on (they are expensive to calculate/draw) - but this is far more densely packed that they will be arranged in the finished feature, plus some other tricks, so they won't ultimately cost that much to render.

Edited May 18, 2021 by johnparker007

[johnparker007]

Further big performance gains found from simple switch in rendering - I can use a rendering that is faster for lighting, since we can't have baked lightmaps...

So a shot from video in previous post showing 16 room lights, plus lots of machines with all lamps on.

Before (slower):


...and almost the same shot with the speed up:

After (faster):

Over 25% less CPU, and nearly 50% less GPU !  For this many colored room lights and machines with their lamps on-screen at once, to have such lower numbers - it's all very encouraging   

Edited May 18, 2021 by johnparker007

[slasher]

That's my worst nightmare. A room full of Reflex Eastern Promise

[johnparker007]

2 hours ago, slasher said:

That's my worst nightmare. A room full of Reflex Eastern Promise

That machine was were only picked for testing as it has plenty of lamps and a single texture containing them all, honest   

[Zippy]

Was so tempted to put a Whippy on his post @slasher lmfao

Nice going there though John, just a disco ball now and the party can start

[mr x]

9 hours ago, slasher said:

That's my worst nightmare. A room full of Reflex Eastern Promise

sponsored by bob rudd and keeday

[vectra666]

9 hours ago, slasher said:

That's my worst nightmare. A room full of Reflex Eastern Promise

Could be worse could have a 1000 noels grinning at you, suppose it’s just as bad looking at a 1000 of my layouts at once lol

[Zippy]

1 hour ago, vectra666 said:

Could be worse could have a 1000 noels grinning at you

or a 1000 Vectra666's

[johnparker007]

Another tech update on further (CPU) optimisation research. 

A chunk of the gains from the previous two tests were disabling a load of the 'game object hierarchy' that have been built for each machine instance. 

Now that I have this new way of rendering lamps (not yet finished but showing big speed gains) - I think it can be applied to the 7 segments, segment alphas and dot alphas, potentially reels (all of which have a large hierarchy overhead with their current approach).

Still just testing in the editor, I roughly remade the test scene again, and now I have more significant CPU gains...

Previous test, simulating some reduced hierarchy:

Current test, simulating almost all reduced hierarchy:

I think the investigations can stop for a while now, as I've got to do many hours of redesigning/reprogramming everything so that the machines that load for use are structured like this, with their components being rendered collectively by the new approach.

This many machines with all their lamps on, with 16 room lights visible - at under 20% CPU and 20% GPU, happy with the results of all this research, less happy about all the work involved to change everything to put it into practice!

It's really lowering the barrier for entry though in terms of how powerful a PC will be needed to run it  

For ref, my PC is:

i5 7600K @ 4.3Ghz
NVidia 1060 6gb

Would be faster in a real build, though there's so much hacked in for these tests I can't easily benchmark that.

Edited May 19, 2021 by johnparker007

[Zippy]

11 minutes ago, johnparker007 said:

For ref, my PC is:

i5 7600K @ 4.3Ghz
NVidia 1060 6gb

Friggin show off, bet that cost you a few quid.

Mine is quite paltry,

AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx     2.10 GHz    8GB

Then again I am not doing what you are doing but has got me thinking

 

 

 

[slasher]

4 hours ago, vectra666 said:

Could be worse could have a 1000 noels grinning at you, suppose it’s just as bad looking at a 1000 of my layouts at once lol

That would be a massive improvement unless they're the later noels

[johnparker007]

1 hour ago, Big J said:

Friggin show off, bet that cost you a few quid.

Mine is quite paltry,

AMD Ryzen 5 3500U with Radeon Vega Mobile Gfx     2.10 GHz    8GB

Then again I am not doing what you are doing but has got me thinking

 

 

 

Hard to say how well it'd run on that at this early stage - I think your RAM and CPU would be ok when it's had all the optimisation work... the graphics card may struggle a bit to maintain 60fps, especially with the colored room lights... but I may find more optimisations yet...   

[Zippy]

1 hour ago, johnparker007 said:

Hard to say how well it'd run on that at this early stage - I think your RAM and CPU would be ok when it's had all the optimisation work... the graphics card may struggle a bit to maintain 60fps, especially with the colored room lights... but I may find more optimisations yet...   

Don't worry John I ain't short of a few pennies so if I have to up the ante to play in a dogs bollocks arcade so be it. Don't know if you ever saw that post I made when Wizard was around about the margins between playing MFME and the real thing so your simulator will be no different, it saves money and we get to do all sorts of shite to machines with minimal financial impact.

I'm sure you and @slasher could advise me best when the time comes of course, still having that problem with the Big 50 (graphics on JP) and just playtested another JPM 10 quider and the same but only on the lamp flashes with the JP. 

Edited May 19, 2021 by Big J