I've never looked before, because I've always had something like MSI Afterburner installed (which shows many things).

Bit of an odd omission, given the plethora of third-party apps that expose the information.

ETA: Afterburner is very GPU-oriented and the configuration UI is complex, as it's the ne plus ultra of overclocking and monitoring tools. It also has the most awful "gamer" UI I've ever seen :D

If you're just looking to monitor temps over time, and you've got an AMD processor, Ryzen Master is a much more approachable proposition.

Also, if you have an AMD processor, consider installing the AMD chipset drivers and selecting the AMD Ryzen Balanced power plan; this will much more aggressively reduce clock speeds and voltages at idle. The difference between this and the stock power plan for me is 10°C for desktop workloads. The thermal cutoff for Ryzen chips is 95°C, so having it run at ~35°C in normal use gives a reassuring amount of headroom.

Context: I have an AMD Ryzen 7 3700X (8C/16T) with the stock Wraith Prism cooler, on a B350 motherboard alongside an AMD Radeon 5700XT.

Side note: you'd have to be off your tits right now to choose an Intel CPU for a (new) desktop system, given the advent of the new AMD chips, which are both faster AND cheaper.