...
JEMHCloud validates connectivity with your server prior to retrieving messages. Messages on your mail-server are only marked read if JEMHC completes processing. In such cases no mail will be lost.
JEMHCloud Maintenance
We use the JEMHCloud blog to announce upcoming upgrades and the completion of them. We try to publish information about upcoming upgrades a few days prior to the weekly maintenance so that you can prepare for them if needed.
To minimise interruptionTo minimize interruption, upgrades are usually planned for 12:00hrs on Sundays UTC.
...
For minor updates, we aim to have a fully automated deployment process, the flow is:
These upgrades are usually complete in a 5 minute window, with a weekly release, gives a technical weekly availability of 99.95%, however we try to beat that with the process:
- Updated applications are posted
- New nodes are warm booted, increasing the pool, these will be the 'current' versions, takes ~2m to boot, once available they become part of the load balancers group
- Existing customer connections will continue to work interactively via JIRACloud nodes, but scheduled jobs will notice the update and not progress further message retrieval, marking themselves as 'stopping' in their healthcheck response
- The Loadblancer that fronts all active nodes reacts to nodes marked as 'stopping' via their healthcheck. After four 'fails' (30S apart, 2m total) the nodes are terminated by the infrastructure and new nodes are created. At this point interactive sessions may fail, requiring a browser refresh.
- Meanwhile the loadbalancer farms new interactive user request to the new nodes.
- The new nodes, on launch continue message retrieval for their hosts.
- After 3minutes, the 'active' node count is reduced to the operating normUpdate binaries are staged
- External Attachments are disabled
- Nodes determine new binaries are available, and stop their scheduling of new jobs
- Nodes complete the current jobs that are running, continue to process webhooks from your hosts
- New nodes are launched using new binaries
- When we have enough healthy new nodes in our load balanced pool, we remove the older hosts and wait 30S to allow any HTTP operations to complete
- Old nodes are terminated, leaving the new application binaries in effect
For more complex upgrades, that involve database changes, all nodes may need to be taken down prior to the upgrade.