Some events (rollout) are stuck on "ongoing" | Voters

Some events (rollout) are stuck on "ongoing"

complete

Ran Magen

I'm not yet sure why.

December 10, 2019

William Morgan

marked this post as

complete

William Morgan

Thanks for your patience. We've shipped a fix for this issue. Please let us know if you're still experiencing it!

William Morgan

marked this post as

in progress

Ran Magen

I manually set the replicas for one of the services to 1, it's been 3 minutes and the event is still "ongoing".

Ran Magen

Ok now it's fixed! I guess it polls every n > 3 minutes?

Ran Magen

I think I figured out the problem here. Specifically, this is reproducible in a specific namespace for 4 specific services, these services are different from others in that they have 0 replicas in that namespace.

So we do tell k8s to update their deployment, but not to bring up any replicas.

William Morgan

marked this post as

under review

William Morgan

Rats. We had a bug related to this but I thought we had fixed it. I assume the rollout concluded, in reality?

Ran Magen

William Morgan: Yes, it had. Is there a workaround to make Dive see the event is done?

William Morgan

Ran Magen: Well... sort of. The workaround right now is "we manually fix the database entry" 🙀. We're digging into what happened with these events right now so stay tuned.