There are at least three separate things that rustc and cargo could do to significantly improve compilation speed, but they are complex engineering efforts that require a lot of development time and engineers working on them full-time for that period.
Off the top of my head they are:
• doing what zig does to compile only items that are reachable, which requires making rustc reentrant to stop after name resolution and cargo more complex to pass used item paths through subsequently to pick up after name resolution for the entire projects dependency tree has been computed,
• leveraging an incremental linker and have rustc only codegen the binary patch instead of the whole binary and have the linker calculate the patch,
• and caching proc-macros and treat them as idempotent (likely as opt-in)
There are of course others, like pre-compiled cratedistribution, better intra-rustc parallelism or feature cfg handling, but those have more open questions and require additional design work before I could claim they're attainable in a reasonable amount of time.