Working on this right now Randy. Thought we found it just a few minutes ago but there is still one issue left. Hopefully in the next few hours.
I posted updates in the "Outage" thread on the situation we were dealing with. We went down late Thursday night EST. We were back up Sunday morning, but then went back down late Sunday night EST. Monday night we were back up again.
The issue was our Certificate Authority updated our root certificate, killing all connections to our server. Certificates update all the time and it's no big deal, but the root Certificate is typically a "once every 20 years" thing. They pushed the update to our server. We were finally able to to square things away, revert to the original working root certificate and that's when all was well again on Sunday. Then they went and pushed the updated root to our server again Sunday night even though we had turned off the auto-update. It turned out there was one additional setting in another part of the system that needed to be turned off to kill the auto-updates on the certificates. Once we found that, turned it off, and put the original certificate back again, everything came back up.
However, in the process of troubleshooting / adjusting / trying things it caused some other APIs to break. There are a lot of them and we have been going through them one by one to turn them back on and fix them.
Please sign in to leave a comment.