vdt-update-certs has a problem when their are failures.
After a failure to get the certificates, the script retries once an
hour. Imagine the VDT web site goes down for a while. Various clients
will enter this retry phase. The longer the outage, the greater the
percentage of clients.
When the VDT web server becomes accessible again, all retrying clients
will hit over an over time span. If the web site is unavailable for more
than 23 hours, this would probably be 100% of clients. Problem 1: when
we come back up, we get hit hard.
At this point, they all "reset the clock", and will fetch the new CA
certificates every 24 hours. So we'll get sacked again 24 hours later.
We need to do something to redistribute the times that the clients retry
upon failure so that they don't synchronize. Or we need to have a
different way of recovering from failures.
RT for crt.cs.wisc.edu
