One option is to use a timed mute, this is an option on the web interface and some third party apps.
On the web, mute the person but before you confirm the mute select "show options" and choose how long you mute them for.
(As someone else pointed out in your replies, the unlisted thing doesn't help if you're following the account).