Seeking advice. I have many services that require cert renewal for HTTPS. My usual workflow for nginx is:
1) set up a cron job to call certbot monthly (strictly more often than needed for a 90 day cert but I'm lazy)
2) 91 days later, re-cert fails, realize that there was a bug in my script, fix it
3) 91 days later, re-cert fails, realize there was a bug in my script, fix it
4) at this point it probably now works forever
Is there a good way to test things properly so I skip steps 2 and 3??
@darius if you want to stick with nginx, try https://nginx.org/en/docs/http/ngx_http_acme_module.html which will handle LE certificates directly within nginx. Caddy or Apache do the same if you can/want to change your web server.
As do monitoring, updown.io does check certs and emails you if they are about to expire
@darius is "have you looked at Caddy?" an option?
@darius I might be missing something here but, at step 1, run the script that cron will run in order to do your initial certificate setup. If that works then it should work a month later when cron runs it. The only variable is if you've actually set cron up to run the correct script at the correct time, which is admittedly tricky.
To solve that problem, if you're on debian or a derivative, put your script in /etc/cron.monthly/ and chmod +x it. Debian has a built-in cron rule to run everything that's executable by root in that directory once a month. Then test it by running sudo /etc/cron.monthly/your_script. If it runs ok and sets up the certificate then it should work when cron runs it.
Also, answering the question you didn't ask, iirc letsencrypt actually recommend you run certbot every day. At least on my servers it runs every day and I will have done that for good reason. There is an /etc/cron.daily for that purpose on debian.
@whimsy it is often a permissions issue where cron is running as the wrong user (for example I run the script manually with sudo and it works but then I install it in crontab running as some other user)
@darius I don't think I have any helpful advice then. Hope you find something that works for you 😊
@whimsy it is good to know letsencrypt encourages daily recert! I was worried i was being a bad citizen doing monthly
@darius I can't find the recommendation right now but monthly is definitely ok! You should get the same certificate back every time until the certificate is a certain age so there's no actual renewal most of the time, but it gives the service an opportunity to renew the certificate early if that's necessary for some reason.
@darius do your first certificate deployment with your cronjob script. Add to cron only after it works. (Basically intentional „test in prod“). If you run certificate renewal from a major script which runs bespoke ones: monitor the first one for failure (eg by setting a working email alias for „root“ in your /etc/aliases and setting „set -e“ or other hard fail options in your major script).
@darius
1. Put everything behind Caddy
There is no step 2. I know it doesn't work for every use case, but it's super hand for me
@darius@friend.camp I used the exact same method until I discovered linuxserver swag and I barely have to worry about it
@darius apologies if this is unhelpful but honestly the main reason I switched from nginx to caddy is that caddy jumps straight to step 4 for you
I did not know about out-of-band monitoring services for this! Makes sense that they exist... might have to try one
I monitor from an old phone that I keep around for the games with this:
https://f-droid.org/packages/net.ibbaa.keepitup
@darius I typically pair auto-renewal with an out-of-band monitor via something like Uptime Kuma or Zabbix - either of these can check cert expiry and alert if the expiry date is within some range.
This doesn’t fix script errors, but will help to catch all manner of failure modes.
@darius never make mistakes
in all seriousness though, does letsencrypt have a sandbox environment? or are you actually verifying failures at 30/60/90 days before the observable failure on day 91?
@trwnh I forget this stuff exists until the observable failure happens!
@trwnh I forget this stuff exists until the observable failure happens!
