There’s a great Q and Answer at dijkstra-path-finding-in-c-is-15x-slower-than-c-version which starts with C++ code at 2ms vs C# at 38ms; and finishes with C# at 2.8ms.
On similar lines, the solution to this riddle — why-does-net-core-2-0-perform-worse-than-net-framework-4-6-1 — has nothing to do with the framework; it results from the slower performance of 64bit code compared to 32bit.
Which just goes to confirm what you might have suspected;
VMs aren’t that slow; and
For best performance, you have to understand your VM, and your language, and your O/S, and ultimately your CPU.