Kubernetes Events Can Be Complicated

I stumbled across this the other day and wanted to write it down for posterity.

In Kubernetes, you can—and at this point I’m speaking loosely, but will tighten things up below—subscribe to a stream of WatchEvents that describe interesting things happening to various Kubernetes resources in your Kubernetes cluster.

What is somewhat mind-altering is that one of the kinds of resources whose WatchEvent stream you can subscribe to is the Kubernetes Event kind.  These two things are different.

Whoa.

Now it’s time to get very, very specific with concepts and typography.  If I am speaking about a Kubernetes resource, you’ll see it capitalized in fixed-width type, like so:

Pod

…and I will do my best to prefix it with the word “Kubernetes”:

Kubernetes Pod

If I am just talking semantics, you’ll see the term in a normal typeface without the word “Kubernetes” in front of it.

If you’re programming in Java, and are using the fabric8 Kubernetes client (the only Java Kubernetes client that I’m aware of), you can receive all events from your Kubernetes cluster by following this recipe:

final Watch closeMe = client.events().inAnyNamespace().watch(new Watcher() {
  @Override
  public final void eventReceived(final Action action, final Event resource) {
  
  }

  @Override
  public final void onClose(final KubernetesClientException kubernetesClientException) {

  }
});

(Note that WordPress’s horrid editor sometimes eats the angle brackets necessary to include the generic type parameter of Event that should follow new Watcher above.)  The eventReceived() method will get called asynchronously as the cluster does interesting things, and you can root around in the contents of the received event to see what happened.  The action will be one of ADDED, MODIFIED, DELETED or ERROR.  Simple, right?

So I was messing about with the bleeding-edge service-catalog project, and installing it in minikube and uninstalling it and generally thrashing around breaking things.  I was somewhat surprised to receive an io.fabric8.kubernetes.model.Event in this stream together with an action equal to DELETED (!) that looked like this (I formatted the output for some degree of legibility below):

Event(
  apiVersion=v1,
  count=1,
  firstTimestamp=2017-04-25T23:41:54Z, 
  involvedObject=ObjectReference(
    apiVersion=v1, 
    fieldPath=spec.containers{controller-manager}, 
    kind=Pod,
    name=catalog-catalog-controller-manager-1242994143-ddl0l,
    namespace=catalog,
    resourceVersion=462865,
    uid=11fc24bf-2a05-11e7-a27a-080027117396,
    additionalProperties={}
  ), 
  kind=Event,
  lastTimestamp=2017-04-25T23:41:54Z,
  message=Started container with id 7b51c389f153832e7719a99738706c2ff38aa28b298b80741f439b712f166262, 
  metadata=ObjectMeta(
    annotations=null,
    clusterName=null,
    creationTimestamp=2017-04-25T23:41:54Z,
    deletionGracePeriodSeconds=null,
    deletionTimestamp=null,
    finalizers=[],
    generateName=null,
    generation=null,
    labels=null,
    name=catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, 
    namespace=catalog,
    ownerReferences=[],
    resourceVersion=472706,
    selfLink=/api/v1/namespaces/catalog/events/catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77,
    uid=c3851fae-2a10-11e7-a27a-080027117396,
    additionalProperties={}
  ),
  reason=Started,
  source=EventSource(
    component=kubelet,
    host=minikube,
    additionalProperties={}
  ),
  type=Normal,
  additionalProperties={}
)

So to the naïve eye, this is some sort of event that represents the deletion of something else.  But maybe it also represents the starting of a container (see the bold highlights above)?  And there is a kind=Pod property buried in there, but there’s also a kind=Event, and if this is a deletion, how come the deletionTimestamp property is null?  And if this is a deletion, how come the reason property is Started?

To understand this, we need to go to the source.

First, let’s look at the fabric8 watch machinery and see how it’s calling our Watcher implementation.  You’ll note there that the code is taking delivery of a JSON payload, supplied over WebSockets by the Kubernetes cluster, of the Kubernetes WatchEvent “kind”. So fundamentally the things being received by fabric8’s watch machinery are Java representations of Kubernetes WatchEvents.

OK, fine.  What’s a Kubernetes WatchEvent?  It is a (semantic) event that describes an addition, deletion or modification of a Kubernetes resource.  It has a type field saying whether it’s an addition, deletion or modification, and an object field that holds a representation of the Kubernetes resource that was added, deleted or modified.

OK, so what’s a Kubernetes resource?  A Kubernetes resource is one of its “things” (a Kubernetes Pod, a Kubernetes Deployment, a Kubernetes ReplicaSet, etc. etc.).  You can get a pretty good idea (maybe an exhaustive idea) of what sorts of things we’re talking about by looking at the reference documentation.

Easy so far.

But another kind of Kubernetes resource is a Kubernetes Event.

So it must follow that a Kubernetes WatchEvent can describe the addition, deletion or modification of a Kubernetes Event, because a Kubernetes Event is a kind of Kubernetes resource.

I don’t know about you, but that kind of blew my mind a little bit.  (I also don’t want to think about what happens if Kubernetes WatchEvents are also capable of being watched!)

So now that we know this, we know this too:

The io.fabric8.kubernetes.model.Event that your Watcher implementation is handed, when your Watcher implementation is constructed with the Java code listed earlier in this blog post, is really the Java representation of the JSON present in a Kubernetes WatchEvent‘s object field, and the Action that your Watcher implementation is handed is really the Java representation of the JSON present in a Kubernetes WatchEvent‘s type field.

So the (semantic) event we received reads (semantically) something like this:

Hello! This is a Kubernetes WatchEvent with a type of Deleted informing you that the Kubernetes resource it is talking about, a Kubernetes Event, describing the starting of a particular Kubernetes Pod‘s container, was deleted from the record of such things.

This suggests three interesting things as well (which I haven’t researched, so this may be common knowledge, but it was interesting to me!).

1. The first thing is that Kubernetes Events are capable of being deleted.
2. The second thing is that Kubernetes Events are capable of being stored.
3. The third thing is that therefore Kubernetes Events serve as a persistent record of a Kubernetes resource’s state over time.

To Java programmers (like yours truly) used to thinking of (semantic) events as transient announcements of in-flight state (think Swing), this takes a little mental reorientation.

Once you are successfully mentally reoriented, however, it makes sense that when a Kubernetes resource notionally described by certain (definitionally persistent) Kubernetes Events is deleted, then so too are its describing Kubernetes Events.

And when a Kubernetes resource is created, so too are (definitionally persistent) Kubernetes Events describing its creation.  And so it follows that you can therefore get Kubernetes WatchEvents delivered to you describing not just “normal” resource additions and deletions but also, if you wish, Kubernetes Event additions and deletions.  In fact, these are exactly and the only Kubernetes WatchEvents you will get delivered to you if you type client.events().inAnyNamespace().watch(myWatcher).

This also suggests that the kubectl get events --watch-only output is doing some interesting unpacking and reassembling of things under the covers.

The command basically sets up a watch using the very same REST endpoint that the fabric8 Kubernetes client recipe detailed above ends up talking to, and receives the very same information.  But its output looks like this (depending on how you’re reading this blog post, you’ll probably have to scroll the following to see the (wide) output):

$ kubectl get events --watch-only
LASTSEEN                      FIRSTSEEN                     COUNT NAME    KIND       SUBOBJECT TYPE   REASON            SOURCE                MESSAGE
2017-04-27 10:18:36 -0700 PDT 2017-04-27 10:18:36 -0700 PDT 1     busybox Deployment           Normal ScalingReplicaSet deployment-controller Scaled up replica set busybox-2844454261 to 1

Note how this makes things look (properly!) like a semantic event occurred representing the scaling up of a particular deployment.

But under the covers, things are a little different.

The corresponding fabric8 io.fabric8.kubernetes.model.Event received with a corresponding Action of type ADDED looks like this (when output as a Java String via its toString() method:

Event(
  apiVersion=v1,
  count=1,
  firstTimestamp=2017-04-27T17:18:36Z,
  involvedObject=ObjectReference(
    apiVersion=extensions,
    fieldPath=null,
    kind=Deployment,
    name=busybox,
    namespace=default,
    resourceVersion=556693,
    uid=8c979eeb-2b6d-11e7-a27a-080027117396,
    additionalProperties={}),
  kind=Event,
  lastTimestamp=2017-04-27T17:18:36Z,
  message=Scaled up replica set busybox-2844454261 to 1,
  metadata=ObjectMeta(
    annotations=null,
    clusterName=null,
    creationTimestamp=2017-04-27T17:18:36Z,
    deletionGracePeriodSeconds=null,
    deletionTimestamp=null,
    finalizers=[],
    generateName=null,
    generation=null,
    labels=null,
    name=busybox.14b950bb4d3ec7c1,
    namespace=default,
    ownerReferences=[],
    resourceVersion=556695,
    selfLink=/api/v1/namespaces/default/events/busybox.14b950bb4d3ec7c1,
    uid=8c99063e-2b6d-11e7-a27a-080027117396,
    additionalProperties={}),
  reason=ScalingReplicaSet,
  source=EventSource(
    component=deployment-controller,
    host=null,
    additionalProperties={}),
  type=Normal,
  additionalProperties={}
)

Let us remember that what is being printed here is a Java representation of the Kubernetes Event that was the contents of the object field of a Kubernetes WatchEvent whose type field‘s value was Added.

You can see from the things that I’ve bolded above that:

• The thing being Added is a Kubernetes Event
• Its name is busybox.14b950bb4d3ec7c1
• It “involves” an object (resource) whose kind is a Kubernetes Deployment and whose name (identifier) is busybox
• The KubernetesEvent‘s message is “Scaled up replica set busybox-2844454261 to 1“
• The reason for the Kubernetes Event‘s creation is “ScalingReplicaSet“
• The selfLink referencing the Kubernetes Event being described references a REST endpoint in the /events “space” with the Kubernetes Event‘s name (busybox.14b950bb4d3ec7c1) as the identifier within that space.  Note particularly this is not the identity of the (involved) Kubernetes Deployment, i.e. it is not “busybox“.

I hope this helps you make sense of your Kubernetes event streams!