@rmondello if you've got any interest in the history of nuclear physics and the discovery of radiation, Musée Curie in the 5th arrondissement is well worth a visit! It's not huge, so you probably don't need more than a few hours to enjoy it, but I had a great time.
It was longer ago that I visited it, but I also remember enjoying the Henri Cartier Bresson Foundation gallery in the 3rd arrondissement, but besides enjoying the gallery it was long enough ago that I can't remember much else sadly.
People have already mentioned it, but I'd throw another recommendation in for Musée d'Orsay. It's pretty big, but the collection is massive and quite varied, and even the architecture itself grabs my attention every time I've been.
Sadly the Pompidou centre is shut for the next 4 years for renovations, or I'd have suggested there too!