Babel is JavaScript. SWC is Rust. esbuild is Go.
If the speed came from Rust, the Go tool should sit with Babel. It sits with SWC. The gap is a native binary plus a redesigned architecture, not the language.