SLO assignment to Deployments
Andrew Waters
It's possible to create SLO's and check/uncheck the Deployments you want to apply them to on the Service level.
In the case where you have multiple environment running the same (Dive) Service, you may actually have different SLO's.
Specifically, you may have a tiered set of clusters based on SLA's and the budget and SLO's are therefore impossible to configure. By moving the concept of deployment assignment to the objective itself, this would create a huge amount of flexibility that currently doesn't appear to exist.
William Morgan
We're currently thinking about how to best address the concept of an "environment" in Dive. If you were able to categorize workloads for a given service into one or more environments (e.g. via a K8s annotation, or some other mechanism), and define SLOs per environment for a service, would that satisfy this use case?
Andrew Waters
William Morgan: Well I've given some thought this morning to the whole Git based configuration that we discussed.
I'm currently thinking our ideal workflow would be an annotation on the manifest file that configures the SLO, any (future planned) alerts/notifications and run books for the service/deployment. This keeps my manifest changes tightly coupled with my SLO for cleaner visibility and almost forces the engineer who deploys it to consider the availability of their service.
That's a slightly different direction to this original issue though.
Andrew Waters
I do wonder what the complexity of such annotations would be which leads me back to CRD's
Andrew Waters
William Morgan: my ideal declaration (roughly chopped):
This allows me to only deploy it to production environments too or change my SLO between environments.
(edited: no markdown on Canny so I've added an image instead)
William Morgan
Andrew Waters: Yeah, I like where this is going.
Andrew Waters
William Morgan: I've created a quick PoC repo for how I would approach this using CRD's:
https://github.com/andrew-waters/dive-crds
Two ways this could be implemented:
- Dive inspects events that match the config.dive.co/v1API and takes the appropriate action to the underlying Dive database.
- Provide an API for the resources and either a standalone agent can watch them locally and apply them via the Dive API or bundle the same logic into the current Dive agent.