Posts

Showing posts with the label distributed systems

What the health!? Implementing health probes for highly available, self-healing, global services

Health probes are essential to running a highly available service, yet they are surprisingly tricky to implement without inadvertently making your uptime worse . Even popular frameworks like Spring Boot (until recently 1 ) used unfortunate defaults that may accidentally encourage a hurried developer to fall into some surprising traps that reduce their services' uptime. Over time, my team has come up with a set of rules to avoid these traps . I'll lay them out first succinctly, and then explain them in detail along with the model we use that embodies them. Don't overload health endpoints – favor more over reuse Keep logic out of monitors – put in endpoints Be very conservative when determining if un healthy Run health checks in background – not on demand If your in a hurry , you can skip straight to the summary . This post may use Kubernetes terms like "container" and "probe", but these ideas can be applied to services running on any ...

Asynchronous denormalization and transactional messaging with MongoDB change streams

MongoDB provides a very useful set of abstractions for building a reliable service. I've found in practice, few databases can rival its indexing, resiliency (automated failover, even between regions; sharding) and consistency (primary and secondary reads) characteristics and their corresponding client library support. Of course, for all that, you give up flexible transaction boundaries (among other things). With a traditional RDBMS, you can describe the data you need to fit your business model, and worry about isolating writes among different tables later. With document-stores like MongoDB, you must design your documents around sensible isolation boundaries and query patterns. So, when joins are needed, you're typically advised to denormalize the joined data in order to keep queries to a single document, but at the same time MongoDB provides little facilities for denormalizing data! As of MongoDB 4.0, there are multi-document transactions, though they come at a performance...