We have all heard the importance of performance testing. It's a phase you need to complete in your implementation check list in LCS, and if you engage with FastTrack they will make sure it stays on your radar as a best practice.

Performance testing is often sold and performed under the belief that somehow it's an exercise that will by magic ensure that you will not experience performance problems when you finally turn the ON switch on your new shiny D365F&O system.

But as with everything it's important to understand what it's value is, and not just do it because everyone is shouting about it. And in the case of performance testing you are really going to be throwing your time and money out the window unless you understand what it's good for.

The first thing you need to pay attention to in a D354F&O system is that (best practice) performance testing is always done in what Microsoft calls a sandbox environment. This is a non-production environment hosted by Microsoft and comes in four (4) different levels of performance (Tier 1 can be hosted in your own Azure tenant, aka. cloud hosted environment and not appropriate). Namely:

1. Tier-2 – Standard Acceptance Test 30-40 users or 3,000 transaction lines per hour.

2. Tier-3 – Premier Acceptance Test 40-70 users or 7,000 transaction lines per hour.

3. Tier-4 – Standard Performance Test 70-150 users or 20,000 transaction lines per hour

4. Tier-5 – Premier Performance Test > 150 users or > 20,000 transaction lines per hour

Now here is the first gotcha. Notice that the highest performing tier is designed for over 150 users. If we look at the range covered by the other tiers we should expect that it will max out at 300'ish users and there is no sensible way to extrapolate the maximum number of transactions it should be able to handle, and anyway I have never seen any data to support the user and transaction loads specified by Microsoft.

I recommend using the Performance test form at System administration > Periodic tasks > Run performance test to get an idea of the relative performance of the different tiers, and once you have access to your production environment some idea of how representative your sandbox is of the real world where it really counts. Not perfect but helps. 

In other words even if you accept the provided sizing information as valid, if you have more than 150 users (very common) there is no way to know if Tier-5 is in any way comparable or not to your production environment and to run performance testing (even if 20,000 transactions per hour should be able to handle most workloads). 

BUT... what you really need is to determine if your solution will perform as expected in your production environment, so testing performance on an environment that is anything else other than an exact copy of your production environment may provide an indication, but does not really tell you how it will perform in production.

Note: Multiple people have noticed that not only does Microsoft scale production environments depending on the customer workload (no known or published criteria) but having what appears to be a lower database tier than a Tier-2 sandbox.

In summary, your platform for performance testing is not the same as your production environment and as a matter of fact you have not control of your production environment and Microsoft may and does change it without your knowledge. Performance testing under these conditions is at best indicative. Take it for what it is and don't elude yourself, I have seen many D365F&O production environments with huge performance problems even though performance testing was done by the book.