Monthly Archives: April 2017

Kubernetes Events Can Be Complicated

I stumbled across this the other day and wanted to write it down for posterity.

In Kubernetes, you can—and at this point I’m speaking loosely, but will tighten things up below—subscribe to a stream of WatchEvents that describe interesting things happening to various Kubernetes resources in your Kubernetes cluster.

What is somewhat mind-altering is that one of the kinds of resources whose WatchEvent stream you can subscribe to is the Kubernetes Event kind.  These two things are different.

Whoa.

Now it’s time to get very, very specific with concepts and typography.  If I am speaking about a Kubernetes resource, you’ll see it capitalized in fixed-width type, like so:

Pod

…and I will do my best to prefix it with the word “Kubernetes”:

Kubernetes Pod

If I am just talking semantics, you’ll see the term in a normal typeface without the word “Kubernetes” in front of it.

If you’re programming in Java, and are using the fabric8 Kubernetes client (the only Java Kubernetes client that I’m aware of), you can receive all events from your Kubernetes cluster by following this recipe:

final Watch closeMe = client.events().inAnyNamespace().watch(new Watcher() {
  @Override
  public final void eventReceived(final Action action, final Event resource) {
  
  }

  @Override
  public final void onClose(final KubernetesClientException kubernetesClientException) {

  }
});

(Note that WordPress’s horrid editor sometimes eats the angle brackets necessary to include the generic type parameter of Event that should follow new Watcher above.)  The eventReceived() method will get called asynchronously as the cluster does interesting things, and you can root around in the contents of the received event to see what happened.  The action will be one of ADDED, MODIFIED, DELETED or ERROR.  Simple, right?

So I was messing about with the bleeding-edge service-catalog project, and installing it in minikube and uninstalling it and generally thrashing around breaking things.  I was somewhat surprised to receive an io.fabric8.kubernetes.model.Event in this stream together with an action equal to DELETED (!) that looked like this (I formatted the output for some degree of legibility below):

Event(
  apiVersion=v1,
  count=1,
  firstTimestamp=2017-04-25T23:41:54Z, 
  involvedObject=ObjectReference(
    apiVersion=v1, 
    fieldPath=spec.containers{controller-manager}, 
    kind=Pod,
    name=catalog-catalog-controller-manager-1242994143-ddl0l,
    namespace=catalog,
    resourceVersion=462865,
    uid=11fc24bf-2a05-11e7-a27a-080027117396,
    additionalProperties={}
  ), 
  kind=Event,
  lastTimestamp=2017-04-25T23:41:54Z,
  message=Started container with id 7b51c389f153832e7719a99738706c2ff38aa28b298b80741f439b712f166262, 
  metadata=ObjectMeta(
    annotations=null,
    clusterName=null,
    creationTimestamp=2017-04-25T23:41:54Z,
    deletionGracePeriodSeconds=null,
    deletionTimestamp=null,
    finalizers=[],
    generateName=null,
    generation=null,
    labels=null,
    name=catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, 
    namespace=catalog,
    ownerReferences=[],
    resourceVersion=472706,
    selfLink=/api/v1/namespaces/catalog/events/catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77,
    uid=c3851fae-2a10-11e7-a27a-080027117396,
    additionalProperties={}
  ),
  reason=Started,
  source=EventSource(
    component=kubelet,
    host=minikube,
    additionalProperties={}
  ),
  type=Normal,
  additionalProperties={}
)

So to the naïve eye, this is some sort of event that represents the deletion of something else.  But maybe it also represents the starting of a container (see the bold highlights above)?  And there is a kind=Pod property buried in there, but there’s also a kind=Event, and if this is a deletion, how come the deletionTimestamp property is null?  And if this is a deletion, how come the reason property is Started?

To understand this, we need to go to the source.

First, let’s look at the fabric8 watch machinery and see how it’s calling our Watcher implementation.  You’ll note there that the code is taking delivery of a JSON payload, supplied over WebSockets by the Kubernetes cluster, of the Kubernetes WatchEvent “kind”. So fundamentally the things being received by fabric8’s watch machinery are Java representations of Kubernetes WatchEvents.

OK, fine.  What’s a Kubernetes WatchEvent?  It is a (semantic) event that describes an addition, deletion or modification of a Kubernetes resource.  It has a type field saying whether it’s an addition, deletion or modification, and an object field that holds a representation of the Kubernetes resource that was added, deleted or modified.

OK, so what’s a Kubernetes resource?  A Kubernetes resource is one of its “things” (a Kubernetes Pod, a Kubernetes Deployment, a Kubernetes ReplicaSet, etc. etc.).  You can get a pretty good idea (maybe an exhaustive idea) of what sorts of things we’re talking about by looking at the reference documentation.

Easy so far.

But another kind of Kubernetes resource is a Kubernetes Event.

So it must follow that a Kubernetes WatchEvent can describe the addition, deletion or modification of a Kubernetes Event, because a Kubernetes Event is a kind of Kubernetes resource.

I don’t know about you, but that kind of blew my mind a little bit.  (I also don’t want to think about what happens if Kubernetes WatchEvents are also capable of being watched!)

So now that we know this, we know this too:

The io.fabric8.kubernetes.model.Event that your Watcher implementation is handed, when your Watcher implementation is constructed with the Java code listed earlier in this blog post, is really the Java representation of the JSON present in a Kubernetes WatchEvent‘s object field, and the Action that your Watcher implementation is handed is really the Java representation of the JSON present in a Kubernetes WatchEvent‘s type field.

So the (semantic) event we received reads (semantically) something like this:

Hello! This is a Kubernetes WatchEvent with a type of Deleted informing you that the Kubernetes resource it is talking about, a Kubernetes Event, describing the starting of a particular Kubernetes Pod‘s container, was deleted from the record of such things.

This suggests three interesting things as well (which I haven’t researched, so this may be common knowledge, but it was interesting to me!).

  1. The first thing is that Kubernetes Events are capable of being deleted.
  2. The second thing is that Kubernetes Events are capable of being stored.
  3. The third thing is that therefore Kubernetes Events serve as a persistent record of a Kubernetes resource’s state over time.

To Java programmers (like yours truly) used to thinking of (semantic) events as transient announcements of in-flight state (think Swing), this takes a little mental reorientation.

Once you are successfully mentally reoriented, however, it makes sense that when a Kubernetes resource notionally described by certain (definitionally persistent) Kubernetes Events is deleted, then so too are its describing Kubernetes Events.

And when a Kubernetes resource is created, so too are (definitionally persistent) Kubernetes Events describing its creation.  And so it follows that you can therefore get Kubernetes WatchEvents delivered to you describing not just “normal” resource additions and deletions but also, if you wish, Kubernetes Event additions and deletions.  In fact, these are exactly and the only Kubernetes WatchEvents you will get delivered to you if you type client.events().inAnyNamespace().watch(myWatcher).

This also suggests that the kubectl get events --watch-only output is doing some interesting unpacking and reassembling of things under the covers.

The command basically sets up a watch using the very same REST endpoint that the fabric8 Kubernetes client recipe detailed above ends up talking to, and receives the very same information.  But its output looks like this (depending on how you’re reading this blog post, you’ll probably have to scroll the following to see the (wide) output):

$ kubectl get events --watch-only
LASTSEEN                      FIRSTSEEN                     COUNT NAME    KIND       SUBOBJECT TYPE   REASON            SOURCE                MESSAGE
2017-04-27 10:18:36 -0700 PDT 2017-04-27 10:18:36 -0700 PDT 1     busybox Deployment           Normal ScalingReplicaSet deployment-controller Scaled up replica set busybox-2844454261 to 1

Note how this makes things look (properly!) like a semantic event occurred representing the scaling up of a particular deployment.

But under the covers, things are a little different.

The corresponding fabric8 io.fabric8.kubernetes.model.Event received with a corresponding Action of type ADDED looks like this (when output as a Java String via its toString() method:

Event(
  apiVersion=v1,
  count=1,
  firstTimestamp=2017-04-27T17:18:36Z,
  involvedObject=ObjectReference(
    apiVersion=extensions,
    fieldPath=null,
    kind=Deployment,
    name=busybox,
    namespace=default,
    resourceVersion=556693,
    uid=8c979eeb-2b6d-11e7-a27a-080027117396,
    additionalProperties={}),
  kind=Event,
  lastTimestamp=2017-04-27T17:18:36Z,
  message=Scaled up replica set busybox-2844454261 to 1,
  metadata=ObjectMeta(
    annotations=null,
    clusterName=null,
    creationTimestamp=2017-04-27T17:18:36Z,
    deletionGracePeriodSeconds=null,
    deletionTimestamp=null,
    finalizers=[],
    generateName=null,
    generation=null,
    labels=null,
    name=busybox.14b950bb4d3ec7c1,
    namespace=default,
    ownerReferences=[],
    resourceVersion=556695,
    selfLink=/api/v1/namespaces/default/events/busybox.14b950bb4d3ec7c1,
    uid=8c99063e-2b6d-11e7-a27a-080027117396,
    additionalProperties={}),
  reason=ScalingReplicaSet,
  source=EventSource(
    component=deployment-controller,
    host=null,
    additionalProperties={}),
  type=Normal,
  additionalProperties={}
)

Let us remember that what is being printed here is a Java representation of the Kubernetes Event that was the contents of the object field of a Kubernetes WatchEvent whose type field‘s value was Added.

You can see from the things that I’ve bolded above that:

  • The thing being Added is a Kubernetes Event
  • Its name is busybox.14b950bb4d3ec7c1
  • It “involves” an object (resource) whose kind is a Kubernetes Deployment and whose name (identifier) is busybox
  • The KubernetesEvent‘s message is “Scaled up replica set busybox-2844454261 to 1
  • The reason for the Kubernetes Event‘s creation is “ScalingReplicaSet
  • The selfLink referencing the Kubernetes Event being described references a REST endpoint in the /events “space” with the Kubernetes Event‘s name (busybox.14b950bb4d3ec7c1) as the identifier within that space.  Note particularly this is not the identity of the (involved) Kubernetes Deployment, i.e. it is not “busybox“.

I hope this helps you make sense of your Kubernetes event streams!

Advertisements

MicroBean Launcher

This is the eleventh of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Commons CLI.

This post covers MicroBean Launcher.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Launcher lets you run a Java SE CDI 2.0 application from the command line by specifying Maven artifact coordinates to other bean archives, transitively resolving them and their dependencies, using MicroBean Maven CDI, to your local Maven repository.

To install MicroBean Launcher, place it (and its minimal dependencies) on your classpath.

Here is an example of its usage, assuming it and its dependencies are on your classpath already:

java org.microbean.launcher.main.Main --artifactPath com.foobar:frobnicator:1.0,com.foobar:caturgiator:2.0

This command will download, if necessary, com.foobar‘s frobnicator jar artifact at version 1.0 and com.foobar‘s caturgiator jar artifact at version 2.0.  These files and their transitive compile- and runtime-scoped dependencies will be placed in the current user’s local Maven repository (~/.m2/repository by default, but the user’s ~/.m2/settings.xml file, which can dictate where the local repository is, is honored).  (If the artifacts already exist, then no download happens.)  A classpath will be built out of all of these artifacts and effectively appended to the current classpath, and MicroBean Main will be invoked.

Why is this useful?  For one, specifying a classpath can now be done in terms of Maven artifact coordinates instead of local filesystem references.  Because MicroBean Maven CDI is in charge of dependency resolution using Maven’s own resolution machinery under the covers, you don’t need to be aware of whether the artifacts in question were downloaded or already present.

But more than this, note that in the last ten blog posts the installation instructions have been the same: you place whatever bean archive is being described on your classpath.  This is one of the nice things about CDI: CDI archives are loosely coupled modules that can be discovered.

This means you can compose a Java SE CDI 2.0 application together by simply referring to Maven artifacts.

Specifically, assuming you have MicroBean Launcher and its dependencies on your CLASSPATH, if you write, say, a JAX-RS Application and a root resource class that it exposes, and place them on your classpath, then you can run that application immediately, downloading only what you need and only what you don’t already have, by running a command line similar to the following:

java org.microbean.launcher.main.Main --artifactPath org.microbean:microbean-jersey-container-grizzly2-http-cdi-extension,org.microbean:microbean-jersey-container-grizzly2-http-cdi,org.microbean:microbean-grizzly-http-server-cdi

Note in particular that your application remains standards-compliant, and you selected the server you wanted to use to run it dynamically from the command line.  You wrote only the code you needed to and none other.

Note that the second time you run this all of the artifacts will already be present on your system.

These are early days for these personal projects and I’ll have plenty more to say about them in the future.  Thanks for reading!

MicroBean Commons CLI

This is the tenth of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Maven CDI.  The next post covers MicroBean Launcher.

This post covers MicroBean Commons CLI.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Commons CLI provides a CDI portable extension that sets up a producer method that produces a CommandLine if there is an Options CDI bean instance in the current CDI container, and if something somewhere makes the command-line arguments available as a String[] qualified with @Named("commandLineArguments") (it just so happens that MicroBean Main does this).

To install MicroBean Commons CLI, place it (and its dependencies) on your classpath.

This means that if your application has a producer method in it that returns an Options, and MicroBean Commons CLI is on the classpath, then you can @Inject a CommandLine anywhere you like, which gives you access to command line option processing.  This, in turn, means that your CDI bean can process command line options easily and intelligently, even though in all likelihood you probably didn’t write the public static void main(String[]) method.  See my post on MicroBean Main for more on this general pattern.

The next post covers MicroBean Launcher, which allows you to link and run a CDI 2.0 Java SE application together out of Maven-addressible artifact coordinates.

MicroBean Maven CDI

This is the ninth of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Jersey Container Grizzly2 HTTP CDI Extension.  The next post covers MicroBean Commons CLI.

This post covers MicroBean Maven CDI.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Maven CDI adapts the inner workings of the Maven Artifact Resolver project so that it can be exposed in a CDI 2.0 environment.

To install MicroBean Maven CDI, place it on your classpath.

The Maven Artifact Resolver project is the Maven-as-build-tool-independent “guts” inside of Maven responsible for transitive dependency resolution and management.  (I’ve written before on this topic.)  MicroBean Maven CDI makes this tooling available inside a CDI 2.0 environment (including Java SE CDI 2.0 applications), along with the common Maven conventions of user-specific settings.xml files and local Maven repositories.  Any interesting magic that it performs is really confined to the translation of Plexus annotations such as Component and Requirement to CDI injection points, which it does by virtue of the power of the CDI portable extension SPI.

There are many ways this could be useful.  Consider—from within your CDI bean—taking in a Maven-style groupId:artifactId:version String identifier and having it resolve to a local file automatically if that file is not already present in exactly the same way that Maven resolves it (jokes about “downloading the Internet” are hereby routed to /dev/null, as usually these indicate an ignorance of the (well-documented) updatePolicy element).  As we’ll see, there are even more powerful things you can do with these capabilities.  I’ve touched on some of them earlier.

In the next post, we’ll touch on the integration of the Apache Commons CLI project with CDI 2.0 by way of MicroBean Commons CLI.

MicroBean Jersey Container Grizzly2 HTTP CDI Extension

This is the eighth of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Grizzly HTTP Server CDI Integration.  The next post covers MicroBean Maven CDI.

This post covers MicroBean Jersey Container Grizzly2 HTTP CDI Extension.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Jersey Container Grizzly2 HTTP CDI Extension (another mouthful) uses MicroBean CDI Utilities and Jersey and Grizzly classes to provide an AbstractBlockingExtension that starts up a Jersey server on a configurable port inside a CDI container if an Application instance is found in the same CDI container.

To install MicroBean Jersey Container Grizzly2 HTTP CDI Extension, place it on your classpath.

MicroBean Jersey Container Grizzly2 HTTP CDI Extension does exactly nothing if there is not an HttpServer instance in the CDI container.  You can cause an HttpServer to exist in the CDI container in any way that you like.  One particularly useful way is to use MicroBean Jersey Container Grizzly HTTP CDI Integration (just place it, too, on your classpath).

If there is an HttpServer instance in the CDI container (or actually any number), then MicroBean Jersey Container Grizzly2 HTTP CDI Extension starts it in such a way that the CDI container is politely and legally blocked.  (I’ve written before on the topic of politely blocking the CDI container.)  See the documentation for the AbstractBlockingExtension class for more information.

The net effect of all this is that if you place the following projects on your classpath (and their minimal dependencies) together with your Application instance, you will have an executable Java SE CDI program serving your JAX-RS application on a port of your choosing via Jersey’s GrizzlyHttpServerFactory without having had to write any code other than that of your Application class:

The next post covers MicroBean Maven CDI.  I promise this will continue to be relevant, despite the Maven reference. 😀

MicroBean Grizzly HTTP Server CDI Integration

This is the seventh of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Jersey Container Grizzly HTTP CDI Integration.  The next post will cover MicroBean Jersey Container Grizzly2 HTTP CDI Extension.

This post covers MicroBean Grizzly HTTP Server CDI Integration.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Grizzly HTTP Server CDI Integration is a simple, small CDI-related project that provides producer methods for two simple things:

That’s it.

To install MicroBean Grizzly HTTP Server CDI Integration, place it on your classpath.

This project on its own is, of course, not very interesting.  But it arranges for some of the raw materials of Grizzly to be made available to CDI injection points, mostly with the assumption that they will be used byMicroBean Jersey Container Grizzly HTTP CDI Integration.  Obviously, if you want to go about supplying instances of these objects in some other way, simply do not put MicroBean Grizzly HTTP Server CDI Integration on the classpath and use your own producer methods instead.

In the next post, we’ll look at MicroBean Jersey Container Grizzly2 HTTP CDI Extension, which will tie several of the MicroBean libraries together.

MicroBean Jersey Container Grizzly HTTP CDI Integration

This is the sixth of a series of posts on some of the personal projects I’ve been working on.  As I hope you’ll see, they all fit together.  The previous post covered MicroBean Main.  The next post will cover MicroBean Grizzly HTTP Server CDI Integration.

This post covers MicroBean Jersey Container Grizzly HTTP CDI Integration.  Its website is here, its source code is here and its binaries are available from Maven Central.

MicroBean Jersey Container Grizzly HTTP CDI Integration (what a mouthful) provides CDI producer methods that produce two things:

Jersey’s GrizzlyHttpServerFactory uses a GrizzlyHttpContainer to build an HttpServer.  A GrizzlyHttpContainer can be produced by a ContainerFactory if it is supplied with GrizzlyHttpContainer.class and an Application.

To install MicroBean Jersey Container Grizzly HTTP CDI Integration, place it on the classpath.

By itself, this project doesn’t do much.  As mentioned, it installs some producer methods that integrate some Jersey and Grizzly raw materials, but they themselves don’t do anything unless someone else asks for, say, an HttpServer or a GrizzlyHttpContainer.  You are not likely to ask for these things directly in your own code (via @Inject).  And even if you do, you won’t get anything useful unless you have also placed an Application bean into the same CDI environment.

So merely placing this project on the classpath won’t do anything.

But when it is combined with MicroBean Jersey Container Grizzly2 HTTP CDI Extension, the subject of the next post, things get more interesting.  Specifically, as we’ll see, this combination allows you to simply supply an Application, and thanks to the interaction of MicroBean Jersey Container Grizzly2 HTTP CDI ExtensionMicroBean Jersey Container Grizzly HTTP CDI Integration, MicroBean Grizzly HTTP Server CDI IntegrationMicroBean CDI UtilitiesMicroBean Configuration CDIMicroBean ConfigurationMicroBean Main and MicroBean Configuration API, you can have a Jersey server running on a configured port as part of a CDI container started by a public static void main(String[]) method you didn’t write, simply by placing projects on the classpath.  I’ve written previously on this topic.

(I told you this would all start coming together.)

In the next post, we’ll look at MicroBean Grizzly HTTP Server CDI Integration, another supplier of raw materials for running Jersey inside CDI.