Harald Uebele’s Blog

Test your Github Pages content locally

2021-02-15T00:00:00+00:00

I am using Github Pages, for this blog, and for some workshops, tutorials, etc. Github Pages uses Jekyll to render the pages and there are instructions on how to setup Jekyll locally to test your content before publishing. I never managed to get them to work, I am not a Ruby expert and something was always missing.

On the other hand, using the git commit && git push approach is tedious because Github Pages can take some time before it starts rendering.

I found the perfect solution, at least for me:

Hans Kristian Flaatten (Starefossen) has created a Docker image to solve this problem, instructions are here in his Github repo.

You open a terminal session in the root directory of your local repo and start the Docker image like this:

$ docker run -it --rm -v "$PWD":/usr/src/app -p "4000:4000" starefossen/github-pages

This mounts your current directory into the image and starts Jekyll. Your pages are served under http://localhost:4000.

What’s really cool: you can keep editing your content and whenever you save it, the rendered pages are regenerated. So when you refresh your browser pointing to http://localhost:4000 you immediately see the changes!

Rendering your pages locally offers another useful option:

I added a new directory in the _posts directory and called it unpublished.

I also added unpublished to the .gitignore file. That means everything in the unpublished directory is not pushed into the Github repository itself and hence not published in my official blog. But it is visible when rendered locally using Starefossen’s container image. To finally publish it, simply move it to the correct directory.

Moving my Blog from Wordpress to Github Pages

2021-02-10T00:00:00+00:00

While I was still working as a Developer Advocate at IBM, I have maintained a blog on Wordpress.com. Now that I retired, I don’t blog much. So I decided to let the Wordpress subscription expire by the end of this year, 2021. But I didn’t want to trash all I wrote so I started to play with Github Pages, Jekyll, and other tools. As you can see I have successfully moved my blog to Github Pages, now.

Image by Peggy und Marco Lachmann-Anke on Pixabay

I have used Github Pages before to write the instructions for workshops but have always used one of the Github built-in themes. But they don’t work well for blogs. There are many other, Jekyll-based themes that can be used with Github Pages and work for blogs.

1. Prepare a Github repository

First of all you need a Github public repository named yourgithubusername.github.io.

If the first part of the repository doesn’t exactly match your username, it won’t work, so make sure to get it right.

The full URL of my repository is https://github.com/haralduebele/haralduebele.github.io and Github Pages will serve its content on https://haralduebele.github.io. This is called a user or organisation site.

2. Select a theme for Github Pages

The one I selected is called “Reverie”. I tried it, liked it, modified it and that is what you are looking at right now. The README has great setup instructions.

You need to modify _config.yml, too, before you can see something meaningful.

An important and not too obvious change is the permalink:

permalink: /:year/:month/:day/:title/

This duplicates the URL format for blog posts from Wordpress.com.

Once you commit and push your changes, it will take a moment and then you can view your new site.

3. Pack your crates

You can export your content on Wordpress.com under ‘Tools’ - ‘Export’.

I choose to export all content and export the media library:

Exported content will be a ZIP file with a XML document. The exported media library is a TAR file that contains the images, etc. sorted in folders by year and month.

What do you do with the huge Wordpress XML? Somebody (Will Boyd, lonekorean) already thought of that:

4. Convert Wordpress XML to MarkDown

I found a pretty good tool here.

Using it is pretty straightforward using the instructions in the repository. It requires Node.js 12.14 or later.

Unpack the Wordpress XML from the ZIP file into the root of this repository, run the script node index.js, and answer the questions.

I had it create folders for years and months. Output looks something like this:

index.md is the actual post. If there is an images folder, it will contain all the images the tool was able to grab or scrape from the XML.

5. Complete the conversion

“wordpress-export-to-markdown” does a pretty good job but it does require moving files and some manual touch up to the blog posts.

a. File names

In Jekyll or Reverie respectively, blog entries go into the _posts directory. They need to follow a specific name schema: yyyy-mm-dd-name.md. The conversion tool creates a name like this for the folders but not for the actual md files. They are all called index.md. So you need to rename the files before you copy them over to the _posts directory. I have created year directories under _posts to make them a little easier to organize.

b. Images

The images from the images folders go to the images folder in your new repo. I created year folders and month folders under the year folders to make it manageable. I believe that the XML files didn’t contain all images when I exported/converted. But you always have the media export that should contain all the images.

c. Frontmatter

The exported index.md files contain frontmatter pulled from Wordpress:

---
title: "Serverless and Knative - Part 1: Installing Knative on CodeReady Containers"
date: "2020-06-02"
tags: 
  - "knative"
  - "kubernetes"
  - "serverless"
---

But you must add some more. This is what I usually have there, e.g:

---
layout: post
title: "Serverless and Knative - Part 1: Installing Knative on CodeReady Containers"
date: "2020-06-02"
categories: [Knative,Kubernetes,Serverless]
published: false
---

You need “layout: post” and you can add “categories” which will show up in the post and you can display all your blog entries sorted by categories.

Change published: false to published: true to make the post visible.

d. Image links and subtitles

Image links should look like this:

![description](/images/yyyy/mm/imagename.ext)

This assumes that you also sort your images into years and months folders.

On Wordpress I sometimes used subtitles under images. In the converted blog entries, the subtitles are simply text, which doesn’t really look good. I use a code block like this:

The text is then centered, smaller, and in italics
{:style="color:gray;font-style:italic;font-size:90%;text-align:center;"}

The text is then centered, smaller, and in italics

e. Open external links in new windows/tabs

Github markdown cannot do this at its own. But you can simply add the HTML code ({:target=”_blank”}) to the link:

[Link Text](https://url){:target="_blank"}

f. Syntax highlighting

The Reverie theme uses Pygments/Dracula to highlight code in preformatted sections. I found this to be helpful, especially with quoted YAML.

```sh
$ this would be shell commands
```

```yaml
and:
  this:
    - would:
        be: yaml
```

g. Escape characters

Look out for backslashes \ and remove them, they are not needed.

6. Changes to the Theme

I made modifications to the theme, e.g. I changed the font family in style.scss to IBM Plex because that is my favorite font.

I added “read time” to my posts based on this article.

Instead of the search page that is part of the Reverie theme I created an archive page that lists all my blogs sorted by year. This is based on Rafa Garrido’s answer in this Stackoverflow question.

And some more stuff … you can go over your top once you figured out how Jekyll works.

Update: Comments section

Github Pages uses Jekyll to create a static site. This means you can’t include logic which would be needed to add comments.

I looked at Disqus, the Reverie theme I use is enabled for Disqus. It is an external service and the pages with Disqus added seem to get very heavy and heavily tracked, too.

I read about the idea to use Github Issues to store the comments. I like this idea and looked at several examples. Then I found utterances. It is a Github App that you install in your repository, you do a little configuration, add a piece of code to the post.html. That’s it. It just works. And its Open Source, too. Data is stored as Github Issues, there is no tracking and no ads. So this is what you see below.

(Maybe) Learn something new about Minikube

2021-02-08T00:00:00+00:00

I wrote my first blog that involved Minikube in February 2019. And I still use Minikube a lot.

Recently I tried to figure out how to run Kubernetes exercises on a “memory challenged” notebook (8 GB RAM). I looked into alternatives, namely K3s (a small foorprint Kubernetes distribution) and K3d which uses K3s and runs it on top of Docker and not in a VM. That sounded like a solution to the memory challenge. K3d runs Docker in Docker: a worker node is a Docker container running on your workstation’s Docker instance. The worker node itself runs its own Docker and on this Docker instance all the Kubernetes deployments are running. This is totally cool on Linux since it eliminates the need for virtualization completely since Docker runs native on Linux. On Mac and Windows you use the virtualization that is part of Docker Desktop. So you need virtualization but it is perfectly integrated in your host operating system.

When I looked a little closer into the Minikube documentation I realized that Minikube can use Docker, too. So here is the first thing I learned new:

1. Minikube Docker Driver

The Docker driver became experimental somewhere around Minikube Version 1.8 in early 2020. It is now (Minikube Version 1.17) a preferred driver for Linux, macOS, and Windows.

If you use Minikube a lot, at some point you may have set configuration options, e.g. for the driver. Check with:

minikube config view

For the Docker driver it will show:

- driver: docker

You may have a setting like:

- vm-driver: virtualbox

That is ancient, vm-driver as a parameter has been deprecated for quite some time.

Initial start of a Minikube cluster will take some time because it needs to download the Docker image but consecutive starts should be a lot faster.

Update It seems that on Mac and Windows networking is different because of Docker Desktop:

In many workshops I use NodePorts to access deployed applications. I use $(minikube ip) to determine the worker nodes IP address and kubectl get svc XXX --output 'jsonpath={.spec.ports[*].nodePort} to get the corresponding nodeport.
You should use minikube service XXX to access the service or minikube service XXX --url to get the URL, instead. This seems to use minikube tunnel (see below) under the covers to gain access to the service.

I found more interesting features that I didn’t know before:

2. Minikube Service

The command minikube service makes working with Kubernetes services a lot easier.

Get a list of all services. If the service is of type “NodePort”, display the URL:

 minikube service list

Result (e.g.):

 |----------------------|---------------------------|--------------|---------------------------|
 |      NAMESPACE       |           NAME            | TARGET PORT  |            URL            |
 |----------------------|---------------------------|--------------|---------------------------|
 | default              | kubernetes                | No node port |
 | default              | mysql                     |         3306 | http://192.168.49.2:32423 |
 | default              | todo                      |         3000 | http://192.168.49.2:30675 |
 | kube-system          | kube-dns                  | No node port |
 |----------------------|---------------------------|--------------|---------------------------|

Restrict the list to one namespace:
```
 minikube service list -n default
```
Open a specific service in your default browser, e.g.
```
 minikube service todo
```
will open the URL for the todo service in your browser.

I don’t know what happens if call your service ‘list’, though :-)
Get the URL of a specific service. Helpful in scripts:
```
 minikube service todo --url
```
Result, e.g.:
```
 http://192.168.49.2:30675
```

3. Minikube Tunnel

If you use an Ingress on Minikube, for example Istio Ingress Gateway, you will have noticed that the corresponding service never gets an external IP address because that is simply not possible on Minikube.

Example:

kubectl get svc -n istio-system

Output:

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.106.56.168    <pending>     15021:32561/TCP,80:30169/TCP,443:30629/TCP,31400:30606/TCP,15443:32011/TCP   97m
...

Now, in a separate terminal session execute:

minikube tunnel

minikube tunnel creates a network route on the host to the service using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system.

Note: The command requires root rights (sudo) because it creates a network configuration.

If you check the service now, the result will look similar to:

Output:

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                                      AGE
istio-ingressgateway   LoadBalancer   10.106.56.168    10.106.56.168   15021:32561/TCP,80:30169/TCP,443:30629/TCP,31400:30606/TCP,15443:32011/TCP   4h37m

The IP address 10.106.56.168 will be available on your workstation.

You can then use services like nip.io or xip.io to create dummy DNS entries, like myingress.10.106.56.168.xip.io.

4. Minikube Logviewer

Central logging is important and helpful. But installing something like an ELK stack may be a bit overwhelming for Minikube. A while ago I found the Minikube Logviewer which is quite simple and doesn’t require a lot of resources.

Now I found out that it is available as a Minikube addon, too. Enable it with

minikube addons enable logviewer

For some reason you need to restart your Minikube cluster but after that it works:

I hope you will find this list helpful!

Run your Code and Containers Serverless on IBM Cloud Code Engine

2020-09-21T00:00:00+00:00

IBM Cloud Code Engine is a fully managed, serverless platform that runs your containerized workloads, including web apps, micro-services, event-driven functions, or batch jobs. Code Engine even builds container images for you from your source code. Because these workloads are all hosted within the same Kubernetes infrastructure, all of them can seamlessly work together. The Code Engine experience is designed so that you can focus on writing code and not on the infrastructure that is needed to host it.

I am a big fan of Kubernetes, it is a very powerful tool to manage containerized applications. But if you only want to run a small application without exactly knowing how much traffic it will generate then Kubernetes may be too big, too expensive, and too much effort. A serverless platform would most likely be better suited for this, for example Knative Serving. But it still requires Kubernetes. If you run a Knative instance on your own you probably don’t gain much. This is where something like IBM’s Code Engine comes to play: They run the (multi-tenant) environment, you use a little part of it and in the end pay only what you use. You don’t pay for any idle infrastructure. Code Engine is currently available as a Beta.

Code Engine offers 3 different options:

Applications
Jobs
Container Builds

Applications and jobs are organized in “Projects” which are based on Kubernetes namespaces and act as a kind of folder. Apps and jobs within each folder can communicate over a private network with each other.

Run your code as an application

This is based on Knative Serving. A container image is deployed, it runs and accepts requests until it is terminated by the operator. An example would be a web application that users interact with or a microservice that receives requests from a user or from other microservices. Since it is based on Knative serving it allows scale-to-zero; no resources are used and hence no money is spent when nobody uses the service. If it receives a request, it spins up, serves the request, and goes dormant again after a time-out. If you allow for auto scaling, it spins up more instances if a huge number of requests come in. Knative Serving itself can do this but IBM’s Code Engine offers a nice web-based GUI for this. And some additional features that I describe later.

Run a job

What is the difference between an app and a job? An app runs until it is terminated by an operator, and it can receive requests. A job doesn’t receive requests and it runs to completion, i.e. it runs until the task it has been started for is complete. This is not Knative Serving but Kubernetes knows jobs and in the linked document is an example that computes π to 2000 places and prints it out. Which is a typical example for a job.

This is how the job would look in Code Engine:

There is a Job Configuration, it specifies the container image (perl) and in the Pi example the command (perl) and the 3 arguments to compute π to 2000 places and print it.

Submitting a “jobrun” creates a pod and in the pod’s log we will find π as:

3.14159265358979323846264338327950288419716939937...

The Submit Job is interesting:

This is where a Code Engine job differs from Kubernetes: In this screenshot, Array indices of “1-50” means that Code Engine will start 50 jobs numbered 1 through 50 using the same configuration. It doesn’t really make sense to calculate the number Pi 50 fifty times. (It should render the identical result 50 times, if not, something is seriously wrong.) But imagine a scenario like this: You have a huge sample of sensor data (or images, or voice samples, etc.) that you need to process to create a ML model. Instead of starting one huge job to process all, you could start 50 or 100 or even more smaller jobs that work on subsets of the data in an “embarrassingly parallel” approach. The current limit is a maximum of 1000 job instances at the same time.

Each of the pods for one of these jobs in an array gets an environment variable JOB_INDEX injected. You could then create an algorithm where each job is able to determine which subset of data to work on based on the index number. If one of the jobs fails, e.g. JOB_INDEX=17, you could restart a single job with just this single Array index instead of rerunning all of them.

Build a Container Image

Code Engine can build container images for you. There are 2 “build strategies”: Buildpack and Dockerfile:

Buildpack (or “Cloud Native Buildpack”) is something you may know from Cloud Foundry or Heroku: the Buildpack inspects your code in a source repository, determines the language environment, and then creates a container image. This is of course limited to the supported languages and language enviroments, and it is based on a number of assumptions. So it will not always work but if it does it relieves developers from writing and maintaining Dockerfiles. The Buildpack strategy is based on Paketo, which is a Cloud Foundry project. Paketo in turn is based on Cloud Native Buildpacks which are maintained under Buildpacks.io and are a Cloud Native Computing Foundation (CNCF) sandbox project at the moment. Buildpacks are currently available for Go, Java, Node.js, PHP, and .NET Core. More will probably follow.

The Dockerfile strategy is straightforward: Specify your source repository and the name of the Dockerfile within, then start to create. It is based on Kaniko and builds the container image inside a container in the Kubernetes cluster. The Dockerfile strategy should always work, even when using Buildpack fails.

The container images are stored in an image registry, this can be Docker Hub or the IBM Cloud Container Registry (ICR) or other registries, both public and private. You can safely store the credentials to access private image registries in Code Engine. These secrets can then be used to store images after being build or to retrieve images to deploy a Code Engine app or job.

Of course, you don’t have to build your container images in Code Engine. You can use your existing DevOps toolchains to create the images and store them in a repository and Code Engine can pick them up from there. But its nice that you can build them in a simple and easy way with Code Engine.

Code Engine CLI

There is a Code Engine plugin for the ibmcloud CLI. Currently the Code Engine (CE or ce) CLI has more functionality than the web based UI in the IBM Cloud dashboard. This will most likely change when Code Engine progresses during the Beta and when it becomes generally available later.

You can use the CLI to retrieve the Kubernetes API configuration used by Code Engine. Once this has been done you can also use kubectl and the kn CLI, you do have only limited permissions in the Kubernetes cluster, though. I have made a quick test: kubectl apply -f service.yaml does work, it creates an app in Code Engine. kn service list or kn service describe hello also work. You ar enot limited to the ibmcloud CLI, then.

Networking

Code Engine apps are assigned a URL in the form https://hello.abcdefgh-1234.us-south.codeengine.appdomain.cloud. They are accessible externally using HTTPS/TLS secured by a Let’s Encrypt certificate. If you deploy a workload with multiple services/apps, maybe only one of them needs to be accessed from the Internet, e.g. the backend-for-frontend. You can limit the networking of the other services to private Code Engine internal endpoints with the CLI:

$ ibmcloud ce application create --name myapp --image ibmcom/hello --cluster-local

This is the same you would do with a label in the YAML file of a Knative service.

Code Engine jobs do not need this, they cannot be accessed externally by definition. Jobs can still make external requests, though. And they can call Code Engine apps internally, there is an example in the Code Engine sample git repo at https://github.com/IBM/CodeEngine.

Integrate IBM Cloud services

If you know Cloud Foundry on the IBM Cloud this should be familiar. IBM Cloud services like Cloud Object Storage, Cloudant database, the Watson services, etc. can be “bound” to a Cloud Foundry app. When the Cloud Foundry app is started, an environment variable VCAP_SERVICES is injected into the pod that holds a JSON object with the configuration (URLs, credentials, etc.) of the bound service/s. The application starting in the pod can then retrieve the configuration and configure access to the service/s. The developers of Code Engine have duplicated this method and in addition to the JSON object in VCAP_SERVICES they also inject individual environment variables for a service (for code that struggles with JSON like Bash scripts).

The helloworld example displays the environment variables of the pod it is running in. If you bind a IBM Cloud service to it, you can display the results with it:

This binding of IBM Cloud services is really interesting for Code Engine jobs. Remember that you cannot connect to them and they can by themselves only write to the joblog. With this feature, you can bind for example a Cloud Object Storage (COS) service to the job, place your data into a COS bucket, run an array of jobs that pick “their” data based on their JOB_INDEX number, and when done, place the results back into the COS bucket.

You may have guessed that under the covers, binding an IBM Cloud service to a Code Engine app or job creates a Kubernetes secret automatically.

Conclusion

Keep in mind that at the time of this writing IBM Cloud Code Engine has just started Beta (it was announced last week). It still has beta limitations, some functions are only available in the CLI, not in the Web UI, and during the Beta, price plans are not available yet. But it is already very promising, it is a very easy start for your small apps using serverless technologies. I am sure that there will be more features and functions in Code Engine as it progresses towards general availability.

Application Security from a Platform Perspective

2020-09-03T00:00:00+00:00

We have added an application security example to our pet project Cloud Native Starter.

Picture 1: Application Architecture

The functionality of our sample is this:

A Web-App service serves a Vue.js/Javascript Web-App frontend application running in the browser of a client
This frontend redirects the user to the login page of Keycloak, an open source identity and access management (IAM) system
After successful login, the frontend obtains a JSON Web Token (JWT) from Keycloak
It requests a list of blog articles from the Web-API using the JWT
The Web-API in turn requests the article information from the Articles service, again using the JWT
The Web-API and Articles services use Keycloak to verify the validity of the JWT and authorize the requests

My colleague Niklas Heidloff has blogged about the language specific application security aspects here:

We also created an app security workshop from it, the material is publicly available on Gitbook.

In this article I want to talk about application security from the platform side. This is what we cover in the above mentioned workshop:

Picture 2: Platform view of the Cloud Native Starter security sample

There are two things that I want to write about:

Accessing the application externally using TLS (HTTPS, green arrow)
Internal Istio Service Mesh security using mutual TLS (mTLS, red-brown arrows)

About the architecture

This is a sample setup for a workshop with the main objective to make it as complete as possible while also keeping it as simple as possible. That’s why there are some “short cuts”:

Istio installation is performed with the demo profile.
Istio Pod auto-injection is enabled on the default namespace using the required annotation.
Web-App deployment in the default namespace is part of the Istio service mesh although it doesn’t benefit a lot from it, there is no communication with other services in the mesh. But it allows us to use the Istio Ingress for TLS encrypted HTTPS access. In a production environment I would probably place Web-App outside the mesh, maybe even outside of Kubernetes, it is only a web server.
Keycloak is installed into the default namespace, too. It is an ‘ephemeral’ development install that consists only of a single pod without persistence. By placing it in the default namespace it can be accessed by the Web-App frontend in the browser through the Istio Ingress using TLS/HTTPS which is definitely a requirement for an IAM – you do not want your authentication information flowing unencrypted through the Internet!
Making it part of the Service Mesh itself automatically enables encryption in the communication with the Web-API and Articles services; both call Keycloak to verify the validity of the JWT token passed by the frontend.
In a production setup, Keycloak would likely be installed in its own namespace. You could either make this namespace part of the Istio service mesh, too. Or you could configure the Istio Egress to enable outgoing calls from the Web-API and Articles services to a Keycloak service outside the mesh. Or maybe you even have an existing Keycloak instance running somewhere else. Then you would also use the Istio Egress to get access to it.

We are using Keycloak in our workshop setup, it is open source and widely used. Actually any OpenID Connect (OIDC) compliant IAM service should work. Another good exampe would be the App ID service on IBM Cloud which has the advantage of being a managed service so you dan’t have to manage it.

Accessing the application with TLS

In this example we are using Istio to help secure our application. We will use the Istio Ingress to route external traffic from the Web-App frontend into the application inside the service mesh.

From a Kubernetes networking view, the Istio Ingress is a Kubernetes service of type LoadBalancer. It requires an external IP address to make it accessible from the Internet. And it will also need a DNS entry in order to be able to create a TLS certificate and to configure the Istio Ingress Gateway correctly.

How you do that is dependent on your Kubernetes implementation and your Cloud provider. In our example we use the IBM Cloud and the IBM Cloud Kubernetes Service (IKS). For IKS the process of exposing the Istio Ingress with a DNS name and TLS is documented in this article and here based on the Istio Bookinfo sample.

The documentation is very good, I won’t repeat it here. But a little background may be required: When you issue the command to create a DNS entry for the load-balancer (ibmcloud ks nlb-dns create ...), in the background this command also produces a Let’s Encrypt TLS certificate for this DNS entry and it stores this TLS certificate in a Kubernetes secret in the default namespace. The Istio Ingress is running in the istio-system namespace, it cannot access a secret in default. That is the reason for the intermediate step to export the secret with the certificate and recreate it in istio-system.

So how is storing a TLS certificate in a Kubernetes secret secure, it is only base64 encoded and not encrypted? That is true but there is are two possible solutions:

Use a certificate management system like IBM Certificate Manager: Certificate Manager uses the Hardware Security Module (HSM)-based IBM Key Protect service for storing root encryption keys. Those root encryption keys are used to wrap per-tenant data encryption keys, which are in turn used to encrypt per-certificate keys which are then stored securely within Certificate Manger databases.
Add a Key Management System (KMS) to the IKS cluster on the IBM Cloud. There is even a free option, IBM Key Protect for IBM Cloud, or for the very security conscious there is the IBM Hyper Protect Crypto Service. Both can be used to encrypt the etcd server of the Kubernetes API server and Kubernetes secrets. You would need to manage the TLS certificates yourself, though.

Or use both, the certificate management system to manage your TLS certificates and the KMS for the rest.

We didn’t cover adding a certificate management system or a KMS in our workshop to keep it simple. But there is a huge documentation section on many aspects of protecting sensitive information in your cluster on the IBM Cloud:

Picture 3 (c) IBM Corp.

Istio Security

In my opinion, Istio is a very important and useful addition to Kubernetes when you work with Microservices architectures. It has features for traffic management, security, and observability. The Istio documentation has a very good section on Istio security features.

In our example we set up Istio with “pod auto-injection” enabled for the default namespace. This means that into every pod that is deployed into the default namespace, Istio deploys an additional container, the Envoy proxy. Istio then changes the routing information in the pod so that all other containers in the pod communicate with services in other pods only through this proxy. For example, when the Web-API service calls the REST API of the Articles service, the Web-API container in the Web-API pod connects to the Envoy proxy in the Web-API pod which makes the request to the Envoy proxy in the Articles pod which passes the request to the Articles container. Sounds complicated but it happens automagically.

The Istio control plane contains a certificate authority (CA) that can manage keys and certificates. This Istio CA creates a X.509 certificate for every Envoy proxy and this certificate can be used for encryption and authentication in the service mesh.

Picture 4 (c) istio.io

You can see in Picture 4 that each of our pods is running an Envoy sidecar and each sidecar holds a (X.509) certificate, including the Istio Ingress which is of course part of the service mesh, too.

With the certificates in place in all the pods, all the communication in the service mesh is automatically encrypted using mutual TLS or mTLS. mTLS means that in the case of a client service (e.g. Web-API) calling a server service (e.g. Articles) both sides can verify the authenticity of the other side. When using “simple” TLS, only the client can verify the authenticity of the server, not vice versa.

The Istio CA even performs automatic certificate and key rotation. Imagine what you would need to add to your code to implement this yourself!

You still need to configure the Istio Ingress Gateway. “Gateway” is an Istio configuration resource. This is what its definition looks like

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: default-gateway-ingress
  namespace: default
spec:
  selector:
	istio: ingressgateway
  servers:
  - port:
	  number: 443
	  name: https
	  protocol: HTTPS
	tls:
	  mode: SIMPLE
	  serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
	  privateKey: /etc/istio/ingressgateway-certs/tls.key
	hosts:
	- "harald-uebele-k8s-1234567890-0001.eu-de.containers.appdomain.cloud"

This requires that you followed the instructions that I linked in the previous section “Accessing the application with TLS”. These instructions create the DNS hostname specified in the hosts: variable and the TLS privateKey and serverCertificate in the correct location.

Now you can access the Istio Ingress using the DNS hostname and only (encrypted) HTTPS as protocol. HTTPS is terminated at the Istio Ingress which means the communication is decrypted there, the Ingress has the required keys to do so. The Istio Ingress is part of the Istio Service Mesh so all the communication between the Ingress and any other service in the mesh will be re-encrypted using mTLS. This happens automatically.

We also need to define an Istio VirtualService for the Istio Ingress Gateway to configure the internal routes:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtualservice-ingress
spec:
  hosts:
  - "harald-uebele-k8s-1234567890-0001.eu-de.containers.appdomain.cloud"
  gateways:
  - default-gateway-ingress
  http:
  - match:
    - uri:
        prefix: /auth
    route:
    - destination:
        port:
          number: 8080
        host: keycloak
  - match:
    - uri:
        prefix: /articles
    route:
    - destination:
        port:
          number: 8081
        host: web-api
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: web-app

The DNS hostname is specified in the hosts: variable, again.

There are 3 routing rules in this example:

https://harald-uebele-k8s-1234567890-0001.eu-de.containers.appdomain.cloud/auth will route the request to the Keycloak service, port 8080. If you know Keycloak you will know that 8080 is the unencrypted port!
https://harald-uebele-k8s-1234567890-0001.eu-de.containers.appdomain.cloud/articles to the Web-API service, port 8081.
Calling https://harald-uebele-k8s-1234567890-0001.eu-de.containers.appdomain.cloud without a path sends the request to Web-App service which basically is a Nginx webserver listending on port 80. Again: http only!

Is this secure? Yes, because all involved parties establish their service mesh internal communications via the Envoy proxies and those will encrypt traffic.

Can it be more secure? Yes, because the Istio service mesh is using mTLS in “permissive” mode. So you can still access the services via unencrypted requests. This is done on purpose to allow you to migrate into a Istio service mesh without immediately breaking your application. In our example you could still call the Artictles service using its NodePort which effectively bypasses Istio security.

Switching to STRICT mTLS

STRICT means that mTLS is enforced for communication in the Istio service mesh. No unencrypted and (X.509!) no unauthorized communication is possible. This eliminates pretty much the possibility of man-in-the-middle attacks.

This requires a PeerAuthentication definition:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "default"
spec:
  mtls:
    mode: STRICT

The PeerAuthentication policy can be set mesh wide, for a namespace, or for a workload using a selector. In this example the policy is set for namespace default.

Once this definition is applied, only mTLS encrypted traffic is possible. You cannot access any service running inside the Istio service mesh by calling it on its NodePort. This also means that services running inside the service mesh can not call services outside without going through an Istio Egress Gateway.

You can do even more with Istio without changing a line of your code. The Istio security concepts and security tasks gives a good overview of what is possible.

Knative Example: Deploying a Microservices Application

2020-07-02T00:00:00+00:00

I have written about Knative Installation, Knative Serving, and Knative Eventing. I have used the simple HelloWorld sample application which is perfectly fine to learn Knative. But I wanted to apply what I have learned with an example that is closer to reality. If you have followed my blog, you should know our pet project Cloud Native Starter. It contains sample code that demonstrates how to get started with cloud-native applications and microservice based architectures.

Cloud Native Starter is basically made up of 3 microservices: Web-API, Articles, and Authors. I have used it for an Istio hands-on workshop where one of the objectives is Traffic Management:

A browser-based application requests a list of blog articles from the Web-API via the Istio Ingress.
The Web-API service retrieves a list of blog articles from the Articles services, and for every article it retrieves author details from the Authors service.
There are two versions of the Web-API service.
Container images for all services are available on my Docker Hub repository.

I think this is perfect to exercise my new Knative skills.

For this example I wanted to give Minikube another try. In my first blog about Knative installation I had issues with Minikube together with Knative 0.12 which has specific instructions on how to install it on Minikube. I have now tested Minikube v1.11.0 with Knative Serving 0.15 and Kourier as networking layer using the default Knative 0.15 installation instructions and I am happy to report:

Knative Serving 0.15 works on Minikube!

Here is the experience with Cloud Native Starter and Knative:

Microservice 1: Authors

The simplest service is Authors, I started to deploy it with a simple Knative YAML file:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors
spec:
  template:
    metadata:
      name: authors-v1
    spec:
      containers:
      - image: docker.io/haraldu/authors:1
        env:
        - name: DATABASE
          value: 'local'
        - name: CLOUDANT\_URL
          value: ''

The only additional configuration are the two environment variables, DATABASE and CLOUDANT_URL. With those the service could be configured to use an external Cloudant database to store the author information. With the settings above, authors information is stored in memory (local) only.

When you deploy this on Minikube, it creates a Knative service

$ kn service list
NAME       URL                                         LATEST        AGE     CONDITIONS   READY   REASON
authors    http://authors.default.example.com          authors-v1    12s     3 OK / 3     True    

It shows that the service listens on the URL:

http://authors.default.example.com

This URL cannot be called directly, it is not resolvable via DNS unless you are able to configure your DNS server or use a local hosts file. With a “real” Kubernetes or OpenShift cluster with a real Ingress e.g. provisioned on the IBM Cloud these steps would not be necessary. To be able to call the API, we need the IP address of the Minikube “worker” node:

$ minikube ip
192.168.39.169

And here you can find the NodePort of the Kourier ingress:

kubectl get svc kourier -n kourier-system
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kourier   LoadBalancer   10.109.186.248   <pending>     80:30265/TCP,443:31749/TCP   4d1h

The NodePort that serves HTTP is 30265. The correct Ingress IP address is therefore: 192.168.39.169:30265

A REST API call to the Authors service using ‘curl’ is then build like this:

$ curl -H 'Host: authors.default.example.com' http://192.168.39.169:30265/api/v1/getauthor?name=Harald%20Uebele
{"name":"Harald Uebele","twitter":"@harald_u","blog":"https://haralduebele.blog"}

In this way the Ingress gets the request with the correct host name in the request header.

‘authors.default.example.com’ is an external URL. But the Authors service needs to be called internally only, it shouldn’t be exposed to the outside. A Knative service can be configured as ‘private cluster-local’. This is done by tagging either the Knative service or the route:

$ kubectl label kservice authors serving.knative.dev/visibility=cluster-local
service.serving.knative.dev/authors labeled

Checking the Knative service again:

$ kn service list
NAME       URL                                         LATEST        AGE    CONDITIONS   READY   REASON  
authors    http://authors.default.svc.cluster.local    authors-v1    84m    3 OK / 3     True    

The URL is now cluster-local. We can also accomplish that by adding an annotation to the YAML file. This saves one step but we are no longer able test the API in a simple manner with curl.

Microservice 2: Articles

The Articles Knative service definition is this:

kind: ConfigMap
apiVersion: v1
metadata:
  name: articles-config
data:
  samplescreation: CREATE
  inmemory: USE\_IN\_MEMORY\_STORE
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: articles
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    metadata:
      name: articles-v1
    spec:
      containers:
      - image: docker.io/haraldu/articles:1
        ports:
        - containerPort: 8080
        env:
        - name: samplescreation
          valueFrom:
            configMapKeyRef:
              name: articles-config
              key: samplescreation
        - name: inmemory
          valueFrom:
            configMapKeyRef:
              name: articles-config
              key: inmemory
        livenessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:8080/"\]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:8080/health | grep -q articles"\]
          initialDelaySeconds: 40

Articles uses a ConfigMap which needs to be created, too.

In the spec.containers section, environment variables are pulled from the ConfigMap and also liveness and readiness probes are defined. Articles is already tagged as ‘cluster-local’, it will only be callable from within the cluster.

Deploy and check shows nothing unusual:

$ kn service list
NAME       URL                                         LATEST        AGE    CONDITIONS   READY   REASON
articles   http://articles.default.svc.cluster.local   articles-v1   53s    3 OK / 3     True    
authors    http://authors.default.svc.cluster.local    authors-v1    99m    3 OK / 3     True    

Since Articles is cluster-internal, it can not be tested. You could use another container in the cluster that can be SSHed into, e.g. an otherwise empty Fedora container, and call the API from there. So I think the best practice during development is to tag the service cluster-only via command as explained in the Authors service section and not use the label in the YAML file. That way you can test the API using curl via external URL and switch to cluster-only once you are confident that the service works as expected.

Microservice 3: Web-API

This is the service that caused the most trouble although the YAML to deploy it is quite simple:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: web-api
spec:
  template:
    metadata:
      name: web-api-v1
    spec:
      containers:
      - image: docker.io/haraldu/web-api:1
        ports:
        - containerPort: 9080
        livenessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/"\]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/health | grep -q web-api"\]
          initialDelaySeconds: 40

It uses readiness and liveness probes like Articles, both services are based on MicroProfile and this is to show the MicroProfile Health feature.

This service must reachable from the outside, no tagging for cluster-local is therefore required.

Deploy it and check for the URL:

$ kn service list
NAME       URL                                         LATEST        AGE    CONDITIONS   READY   REASON
articles   http://articles.default.svc.cluster.local   articles-v1   53s    3 OK / 3     True    
authors    http://authors.default.svc.cluster.local    authors-v1    99m    3 OK / 3     True    
web-api    http://web-api.default.example.com          web-api-v1a   4d1h   3 OK / 3     True    

Test it with ‘curl’:

$ curl -H 'Host: web-api.default.example.com' http://192.168.39.169:30265/web-api/v1/getmultiple

Nothing happens, the call seems to hang, it returns an empty object. The error log shows:

[err] com.ibm.webapi.business.getArticles: Cannot connect to articles service

What is wrong? Digging into the code reveals that Web-API issues REST requests to the wrong URL, e.g. for Articles:

static final String BASE_URL = "http://articles:8080/articles/v1/";

Identical situation for Authors:

static final String BASE_URL = "http://authors:3000/api/v1/";

The URLs are correct for Kubernetes, both services run in the same namespace and can be called by simply using their name. And they listen on different ports. For Knative they need to be changed to call http://articles.default.svc.cluster.local/articles/v1/ and http://authors.default.svc.cluster.local/api/v1/, both without port definition because Knative and its Ingress require fully qualified DNS names and expose HTTP on port 80. I have changed the code, recompiled the two versions of Web-API and created Container Images on Docker Hub: docker.io/haraldu/web-api:knative-v1 and docker.io/haraldu/web-api:knative-v2 (which we need later).

Testing with ‘curl’ still gives no result, but checking of the pods shows why:

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
articles-v1-deployment-5ddf9869c7-rslv5 0/2 Running 0 22s
web-api-v1-deployment-ff547b857-pc5ms 2/2 Running 0 2m8s

Articles has been scaled to zero and it is still in the process of starting (READY: 0/2). It is a traditional Java app and takes some time to start. initialDelaySeconds parameters for liveness and readiness probes add some additional delay. Authors has been scaled to zero, too, but as a Node.js app it starts quickly. For Java based microservices that are supposed to be deployed on Knative, Quarkus is definitely a better choice as it reduces startup time dramatically.

Disable Scale-to-Zero

This is the modified YAML for Articles, it includes the cluster-local label and the minScale: "1" that prevents scale to zero:

kind: ConfigMap
apiVersion: v1
metadata:
  name: articles-config
data:
  samplescreation: CREATE
  inmemory: USE\_IN\_MEMORY\_STORE
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: articles
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    metadata:
      name: articles-v1
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - image: docker.io/haraldu/articles:1
        ports:
        - containerPort: 8080
        env:
        - name: samplescreation
          valueFrom:
            configMapKeyRef:
              name: articles-config
              key: samplescreation
        - name: inmemory
          valueFrom:
            configMapKeyRef:
              name: articles-config
              key: inmemory
        livenessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:8080/"\]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:8080/health | grep -q articles"\]
          initialDelaySeconds: 40

And here is the one for Web-API (v1):

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: web-api
spec:
  template:
    metadata:
      name: web-api-v1
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - image: docker.io/haraldu/web-api:knative-v1
        ports:
        - containerPort: 9080
        livenessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/"\]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/health | grep -q web-api"\]
          initialDelaySeconds: 40

Canary Testing

In the architecture diagram at the very beginning of this article you can see two versions of Web-API. Their difference is: Version 1 displays a list of 5 articles, Version 2 displays 10 articles. If you deploy a new version of a microservice you will most likely want to test it first, maybe as a canary deployment on a subset of users using Traffic Management.

This is how you define it:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: web-api
spec:
  template:
    metadata:
      name: web-api-v2
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
      - image: docker.io/haraldu/web-api:knative-v2
        ports:
        - containerPort: 9080
        livenessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/"\]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: \["sh", "-c", "curl -s http://localhost:9080/health | grep -q web-api"\]
          initialDelaySeconds: 40
  traffic:
    - tag: v1
      revisionName: web-api-v1
      percent: 75
    - tag: v2
      revisionName: web-api-v2
      percent: 25

In the image section, the knative-v2 Container image is referenced.

The traffic sections performs a 75% / 25% split between Version 1 and Version 2. If you know Istio you will know where this function comes from. You will also know how much needs to be configured to enable traffic management with Istio: VirtualService, DestinationRule, and entries to the Ingress Gateway configuration.

Conclusion and further information

This was the description of an almost “real life” microservices example on Knative. You have seen that with typical Java based microservices with their long start-up times the serverless scale-to-zero pattern doesn’t work. If you want to use Java together with scale-to-zero, you need to utilize recent developments in Java like Quarkus with its impressively fast start-up.

So is Knative worth the effort and resources? I am not sure about Knative Eventing. But Knative Serving with its easier deployment files and the easy implementation of auto-scaling and traffic management are definitely worth a try. But keep in mind that Knative is not well suited for every workload that you would deploy on Kubernetes.

Additional reading:

Knative documentation, https://knative.dev/docs
Red Hat Knative Tutorial, https://redhat-developer-demos.github.io/knative-tutorial
Deploying serverless apps with Knative, https://cloud.ibm.com/docs/containers?topic=containers-serverless-apps-knative

Serverless and Knative - Part 3: Knative Eventing

2020-06-10T00:00:00+00:00

This is part 3 of my blog series about Serverless and Knative. I covered Installing Knative on CodeReady Containers in part 1 and Knative Serving in part 2.

Knative Eventing allows to pass events from an event producer to an event consumer. Knative events follow the CloudEvents specification.

Event producers can be anything:

“Ping” jobs that periodically send an event
Apache CouchDB sending an event when a record is written, changed, or deleted
Kafka Message Broker
Github repository
Kubernetes API Server emitting cluster events
and many more.

An event consumer is any type of code running on Kubernetes (typically) that is callable. It can be a “classic” Kubernetes deployment and service, and of course in can be a Knative Service.

A good source to learn Knative eventing is the Knative documentation itself and the Red Hat Knative Tutorial. I think, the Red Hat tutorial is better structured and more readable.

There are three usage patterns for Knative Eventing, the first one being the simplest:

Source to Sink

In this case, the source sends a message to a sink, there is no queuing or filtering, it is a one-to-one relationship.

Knative Event Sources are Knative objects. The following sources are installed when Knative is installed:

$ kubectl api-resources --api-group='sources.knative.dev'
NAME               SHORTNAMES   APIGROUP              NAMESPACED   KIND
apiserversources                sources.knative.dev   true         ApiServerSource
pingsources                     sources.knative.dev   true         PingSource
sinkbindings                    sources.knative.dev   true         SinkBinding

There are many more sources, e.g. a Kafka Source or a CouchDB Source, but they need to be installed separately. To get a basic understanding of Knative eventing, the PingSource is sufficient. It creates something comparable to a cron job on Linux that periodically emits a message.

The Source links to the Sink so it is best to define/deploy the Sink first. It is a simple Knative Service, the code snippets are all from the Red Hat Knative Tutorial:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: eventinghello
spec:
  template:
    metadata:
      name: eventinghello-v1
    spec:
      containers:
      - image: quay.io/rhdevelopers/eventinghello:0.0.2

And this is the Source definition:

apiVersion: sources.knative.dev/v1alpha2
kind: PingSource 
metadata:
  name: eventinghello-ping-source
spec: 
  schedule: "*/2 * * * *"
  jsonData: '{"key": "every 2 mins"}'
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: eventinghello

PingSource is one of the default Knative Sources.
The Schedule is typical cron, it defines that the “ping” happens every 2 minutes.
jsonData is the (fixed) message that is transmitted.
sink defines the Knative Service that the Source connects to: eventinghello.

When both elements are deployed we can see that an eventinghello pod is started every two minutes, in its log we can see the message ‘{“key”: “every 2 mins”}’. The pod itself terminates after about 60 to 70 seconds (Knative scale to zero) and another pod is started after the 2 minutes interval of the PingSource are over and the next message is sent.

To recap the Source-to-Sink pattern: it connects an event source with an event sink in a one-to-one relation. In my opinion it is a starting point to understand Knative Eventing terminology but it would be an incredible waste of resources if this were the only available pattern. The next pattern is:

Channel and Subscription

A Knative Channel is a custom resource that can persist events and allows to forward events to multiple destinations (via subscriptions). There are multiple channel implementations: InMemoryChannel, KafkaChannel, NATS Channel, etc.

By default all Knative Channels in a Kubernetes cluster use the InMemoryChannel implementation. The Knative documentation describes InMemoryChannels as “a best effort Channel. They should NOT be used in Production. They are useful for development.” Characteristics are:

No Persistence: When a Pod goes down, messages go with it.
No Ordering Guarantee: There is nothing enforcing an ordering, so two messages that arrive at the same time may go to subscribers in any order. Different downstream subscribers may see different orders.
No Redelivery Attempts: When a subscriber rejects a message, there is no attempts to retry sending it.
Dead Letter Sink: When a subscriber rejects a message, this message is sent to the dead letter sink, if present, otherwise it is dropped.

A lot of restrictions but it is much easier to set up compared to the KafkaChannel where you need to create a Kafka Server first.

Knative Eventing is very configurable here: you can change the cluster wide Channel default and you can change the Channel implementation per namespace. For example you can keep InMemoryChannel as the cluster default but use KafkaChannel in one or two projects (namespaces) with much higher requirements for availability and message delivery.

A Knative Subscription connects (= subscribes) a Sink service to a Channel. Each Sink service needs its own Subscription to a Channel.

Coming from the Source to Sink pattern in the previous section, the Source to Sink relation is now replaced with a Source to Channel relation. One or multiple Sink services subscribe to the Channel:

The Channel and Subscription pattern decouples the event producer (Source) from the event consumer (Sink) and allows for a one to many relation between Source and Sink. Every message / event emitted by the Source is forwarded to one or many Sinks that are subscribed to the Channel.

Brokers and Triggers

The Broker and Trigger pattern extends the Channel and Subscription pattern and is the most interesting scenario. Therefore I won’t go into more detail here but the Red Hat Knative Tutorial has an example for Channel and Subscriber.

A Broker is a Knative custom resource that is composed of at least two distinct objects, an ingress and a filter. Events are sent to the Broker ingress, the filter strips all metadata from the event data that is not part of the CloudEvent. Brokers typically use Knative Channels to deliver the events.

This is the definition of a Knative Broker:

apiVersion: eventing.knative.dev/v1beta1
kind: Broker
metadata:
  name: default
spec:
  # Configuration specific to this broker.
  config:
    apiVersion: v1
    kind: ConfigMap
    name: config-br-default-channel
    namespace: knative-eventing

A Trigger is very similar to a Subscription, it subscribes to events from a specific Broker but the most interesting aspect is that it allows filtering on specific events based on their CloudEvent attributes:

apiVersion: eventing.knative.dev/v1beta1
kind: Trigger
metadata:
  name: my-service-trigger
spec:
  broker: default
  filter:
    attributes:
      type: dev.knative.foo.bar
      myextension: my-extension-value
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-service

I think this is were Knative Eventing gets interesting. Why would you install an overhead of resources (called Knative Eventing) into your Kubernetes cluster to simply send a message / event from one pod to another? But with an event broker that receives a multitude of different events and triggers that filter out a specific event and route that to a specific (micro) service I can see an advantage.

This is the slightly modified example from the Red Hat Knative Tutorial:

To create a default broker requires no YAML. To use the default Broker for a Kubernetes namespace just add a label:

$ kubectl label namespace knativetutorial knative-eventing-injection=enabled

This will automatically create the required resources. To check:

$ kubectl get broker
NAME      READY   REASON   URL                                                       AGE
default   True             http://default-broker.knativetutorial.svc.cluster.local   3d19h

$ kubectl get channel
NAME                                                        READY   REASON   URL                                                                       AGE
inmemorychannel.messaging.knative.dev/default-kne-trigger   True             http://default-kne-trigger-kn-channel.knativetutorial.svc.cluster.local   3d19h

The first command shows the “default” broker is ready and listens to the URL http://default-broker.knativetutorial.svc.cluster.local. The second command shows that our default broker uses the InMemoryChannel implementation.

The example implements 2 services (sinks) to receive events: eventingaloha and eventingbonjour.

aloha-sink.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: eventingaloha
spec:
  template:
    metadata:
      name: eventingaloha-v1
      annotations:
        autoscaling.knative.dev/target: "1"
    spec:
      containers:
      - image: quay.io/rhdevelopers/eventinghello:0.0.2

bonjour-sink.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: eventingbonjour
spec:
  template:
    metadata:
      name: eventingbonjour-v1
      annotations:
        autoscaling.knative.dev/target: "1"
    spec:
      containers:
      - image: quay.io/rhdevelopers/eventinghello:0.0.2

They are exactly the same, they are based on the same container image, only the name is different. The name will help to distinguish which service received an event.

When everything is set up, we will send three different event types to the broker: ‘aloha’, ‘bonjour’, and ‘greetings’. The ‘aloha’ type should go to the eventingaloha service, ‘bonjour’ to the eventingbonjour service, and ‘greetings’ to both. To accomplish this we need triggers.

Triggers have some limitations. First, you can filter on multiple attributes, e.g.:

  filter:
    attributes:
      type: dev.knative.foo.bar
      myextension: my-extension-value

But the attributes are always AND: ‘dev.knative.foo.bar’ AND ‘my-extension-value’. We cannot define a trigger that would filter on ‘aloha’ OR ‘greetings’. We need 2 triggers for that.

Also a trigger can only define a single subscriber (service). We cannot define a trigger for ‘greetings’ with both the eventingaloha service and the eventingbonjour service as subscribers.

This means we will need 4 Trigger configurations:

If you start to seriously work with Knative Triggers, think about a good naming convention for them first. Otherwise troubleshooting could be difficult in case the triggers don’t work as expected: OpenShift Web Console does a very good job at visualizing Knative objects but it ignores Triggers. And this is what you see in the command line:

$ kubectl get trigger
NAME               READY   REASON   BROKER    SUBSCRIBER_URI   AGE
alohaaloha         True             default                    21h
bonjourbonjour     True             default                    21h
greetingsaloha     True             default                    21h
greetingsbonjour   True             default                    21h

Our example now looks like this:

We have the Knative default Broker, 4 Knative Triggers that filter on specific event attributes and pass the events to one or both of the 2 Knative eventing services. We don’t have an event source yet.

A little further up we saw that the broker listens to the URL
http://default-broker.knativetutorial.svc.cluster.local

We will now simply start a pod in our cluster based on a base Fedora image that contains the curl command based on this curler.yaml:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: curler
  name: curler
spec:
  containers:
  - name: curler
    image: fedora:29 
    tty: true

Start with:

$ kubectl -n knativetutorial apply -f curler.yaml

Get a bash shell in the running pod:

$ kubectl -n knativetutorial exec -it curler -- /bin/bash

In the curler pod, we send an event using curl to the broker URL, event type ‘aloha’:

[root@curler /]# curl -v "http://default-broker.knativetutorial.svc.cluster.local" 
> -X POST 
> -H "Ce-Id: say-hello" 
> -H "Ce-Specversion: 1.0" 
> -H "Ce-Type: aloha" 
> -H "Ce-Source: mycurl" 
> -H "Content-Type: application/json" 
> -d '{"key":"from a curl"}'

In the OpenShift Web Console we can see that an eventingaloha pod has been started:

After about a minute this scales down to 0 again. Next test is type ‘bonjour’, again in the curler pod:

[root@curler /]# curl -v "http://default-broker.knativetutorial.svc.cluster.local" 
-X POST 
-H "Ce-Id: say-hello" 
-H "Ce-Specversion: 1.0" 
-H "Ce-Type: bonjour" 
-H "Ce-Source: mycurl" 
-H "Content-Type: application/json" 
-d '{"key":"from a curl"}'

This starts a eventingbonjour pod as expected:

If we are fast enough we can check its logs and see our event has been forwarded:

2020-06-09 08:38:22,348 INFO eventing-hello ce-id=say-hello
2020-06-09 08:38:22,349 INFO eventing-hello ce-source=mycurl
2020-06-09 08:38:22,350 INFO eventing-hello ce-specversion=1.0
2020-06-09 08:38:22,351 INFO eventing-hello ce-time=2020-06-09T08:38:12.512544667Z
2020-06-09 08:38:22,351 INFO eventing-hello ce-type=bonjour
2020-06-09 08:38:22,352 INFO eventing-hello content-type=application/json
2020-06-09 08:38:22,355 INFO eventing-hello content-length=21
2020-06-09 08:38:22,356 INFO eventing-hello POST:{"key":"from a curl"}

In the last test we send the ‘greetings’ type event:

[root@curler /]# curl -v "http://default-broker.knativetutorial.svc.cluster.local" 
-X POST 
-H "Ce-Id: say-hello" 
-H "Ce-Specversion: 1.0" 
-H "Ce-Type: greetings" 
-H "Ce-Source: mycurl" 
-H "Content-Type: applicatio

And as expected we see pods in both services are started:

Using Apache Kafka

I didn’t go through the Knative Kafka Example. But since it is hard to find and also the preferable method of setting up a production scale Broker & Trigger pattern for Knative Eventing, I wanted to have it documented here.

There are actually 2 parts in the Kafka example:

Start with Installing Apache Kafka: This will probably work in OpenShift (and CRC), too. But depending on the OpenShift version I would start to install the Strimzi or the Red Hat AMQ Streams operator from the OperatorHub catalog in the OpenShift Web Console and create a Kafka cluster with the help of the installed operator.
Continue with the Apache Channel Example. This example installs a Kafka Channel and uses it together with the Knative Default Broker. In the end, an Event Sink is created, a Trigger that connects the Sink to the Broker, and an Event Source (that uses the Kubernetes API Server to generate events).

Knative Eventing Recap

I have had a look now at both Knative Serving and Knative Eventing:

I really like Knative Serving, I think it can help a developer be more productive.

I am undecided about Eventing, though. The Broker & Trigger example based on the InMemoryChannel is easy to set up. But using the InMemoryChannel is for testing and learning only, it is not viable for production. And if I set up my cluster with an instance of Apache Kafka I do ask myself why I should take the messaging detour through Eventing and not use Kafka Messaging in my code directly.

Serverless and Knative - Part 2: Knative Serving

2020-06-03T00:00:00+00:00

In the first part of this series I went through the installation of Knative on CodeReady Containers which is basically Red Hat OpenShift 4.4 running on a notebook.

In this second part I will cover Knative Serving, which is responsible for deploying and running containers, also networking and auto-scaling. Auto-scaling allows scale to zero and is probably the main reason why Knative is referred to as Serverless platform.

Before digging into Knative Serving let me share a piece of information from the Knative Runtime Contract which helps to position Knative. It compares Kubernetes workloads (general-purpose containers) with Knative workloads (stateless request-triggered containers):

“In contrast to general-purpose containers, stateless request-triggered (i.e. on-demand) autoscaled containers have the following properties:

Little or no long-term runtime state (especially in cases where code might be scaled to zero in the absence of request traffic).
Logging and monitoring aggregation (telemetry) is important for understanding and debugging the system, as containers might be created or deleted at any time in response to autoscaling.
Multitenancy is highly desirable to allow cost sharing for bursty applications on relatively stable underlying hardware resources.”

Or in other words: Knative sees itself better suited for short running processes. You need to provide central logging and monitoring because the pods come and go. And multi-tenant hardware can be provided large enough to scale for peaks and at the same time make effective use of the resources.

As a developer, I would expect Knative to make my life easier (Knative claims that it is “abstracting away the complex details and enabling developers to focus on what matters”) but instead when coming from Kubernetes it gets more complicated and confusing at first because Knative uses new terminology for its resources. They are:

Service: Responsible for managing the life cycle of an application/workload. Creates and owns the other Knative objects Route and Configuration.
Route: Maps a network endpoint to one or multiple Revisions. Allows Traffic Management.
Configuration: Desired state of the workload. Creates and maintains Revisions.
Revision: A specific version of a code deployment. Revisions are immutable. Revisions can be scaled up and down. Rules can be applied to the Route to direct traffic to specific Revisions.

Did I already mention that this is confusing? We now need to distinguish between Kubernetes services and Knative services. And on OpenShift, between OpenShift Routes and Knative Routes.

Enough complained, here starts the interesting part:

Creating a sample application

I am following this example from the Knative web site which is a simple Hello World type of application written in Node.js. The sample is also available in Java, Go, PHP, Python, Ruby, and some other languages.

Instead of using the Docker build explained in the example I am using an OpenShift Binary build which builds the Container image on OpenShift and stores it as an Image stream in the OpenShift Image Repository. Of course, the Container image could also be on Docker Hub or Quay.io or any other repository that you can access. If you follow the Knative example step by step, you create the Node.js application, a Dockerfile, and some more files. On OpenShift, for the Binary build, we need the application code and the Dockerfile and then create an OpenShift project and the Container image with these commands:

$ oc new-project knativetutorial
$ oc new-build --name helloworld --binary --strategy docker
$ oc start-build helloworld --from-dir=.

Deploying an app as Knative Service

Next I continue with the Knative example. This is the service.yaml file required to deploy the ‘helloworld’ example as a Knative Service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v1
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v1"

If you are familiar with Kubernetes, you have to start to pay close attention to the first line, to see that this is the definition of a Knative Service.

All you need for your deployment are the highlighted lines, specifically the first ‘metadata’.’name’ and the ‘containers’.’images’ specification to tell Kubernetes where to find the Container image.

Line 11 specifies the location of the Container image just like every other Kubernetes deployment description. In this example, the ‘helloworld’ image is the Image stream in the OpenShift internal Image Repository in a project called ‘knativetutorial’. It is the result of the previous section “Creating a sample application”.

Lines 12, 13, and 14 are setting an environment variable and are used to “create” different versions. (In the Hello World code, the variable TARGET represents the “World” part.)

Lines 7 and 8, ‘metadata’ and ‘name’, are optional but highly recommended. They are used to provide arbitrary names for the Revisions. If you omit this second name, Knative will use default names for the Revisions (“helloworld-nodejs-xhz5df”) and if you have more than one version/revision this makes it difficult to distinguish between them.

With CRC and Knative correctly set up, I simply deploy the service using oc:

$ oc apply -f service.yaml
service.serving.knative.dev/helloworld-nodejs created

The reply isn’t very spectacular but if you look around (oc get all) you can see that a lot has happened:

A Kubernetes Pod is created, running two containers: user-container and Envoy
Multiple Kubernetes services are created, one is equipped with an OpenShift route
An OpenShift Route is created
A Kubernetes deployment and a replica-set are created
Knative service, configuration, route, and revision objects are created

It would have taken a YAML file with a lot more definitions and specifications to accomplish all that with plain Kubernetes. I would say that the Knative claim of “abstracting away the complex details and enabling developers to focus on what matters” is definitely true!

Take a look at the OpenShift Console, in the Developer, Topology view:

I really like the way the Red Hat OpenShift developers have visualized Knative objects here.

If you click on the link (Location) of the Route, you will see the helloworld-nodejs response in a browser:

If you wait about a minute or so, the Pod will terminate: “All Revisions are autoscaled to 0”. If you click on the Route location (URL) then, a Pod will be spun up again.

Another good view of the Knative service is available through the kn CLI tool:

$ kn service list
NAME                URL                                                         LATEST                 AGE   CONDITIONS   READY   REASON
helloworld-nodejs   http://helloworld-nodejs-knativetutorial.apps-crc.testing   helloworld-nodejs-v1   13m   3 OK / 3     True  

$ kn service describe helloworld-nodejs
Name:       helloworld-nodejs
Namespace:  knativetutorial
Age:        15m
URL:        http://helloworld-nodejs-knativetutorial.apps-crc.testing

Revisions:  
  100%  @latest (helloworld-nodejs-v1) [1] (15m)
        Image:  image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest (at 53b1b4)

Conditions:  
  OK TYPE                   AGE REASON
  ++ Ready                  15m 
  ++ ConfigurationsReady    15m 
  ++ RoutesReady            15m 

Adding a new revision

I will now create a second version of our app and deploy it as a second Revision using a new file, service-v2.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v2
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v2 -- UPDATED"

I have changed the revision number to ‘-v2’ and modified the environment variable TARGET so that we can see which “version” is called. Apply with:

$ oc apply -f service-v2.yaml
service.serving.knative.dev/helloworld-nodejs configured

Checking with the kn CLI we can see that Revision ‘-v2’ is now used:

$ kn service describe helloworld-nodejs
Name:       helloworld-nodejs
Namespace:  knativetutorial
Age:        21m
URL:        http://helloworld-nodejs-knativetutorial.apps-crc.testing

Revisions:  
  100%  @latest (helloworld-nodejs-v2) [2] (23s)
        Image:  image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest (at 53b1b4)

Conditions:  
  OK TYPE                   AGE REASON
  ++ Ready                  18s 
  ++ ConfigurationsReady    18s 
  ++ RoutesReady            18s 

It is visible in the OpenShift Web Console, too:

Revision 2 has now fully replaced Revision 1.

Traffic Management

What if we want to Canary test Revision 2? It is just a simple modification in the YAML:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v2
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v2 -- UPDATED"
  traffic:
    - tag: v1
      revisionName: helloworld-nodejs-v1
      percent: 75
    - tag: v2
      revisionName: helloworld-nodejs-v2
      percent: 25

This will create a 75% / 25% distribution between revision 1 and 2. Deploy the change and watch in the OpenShift Web Console:

Have you ever used Istio? To accomplish this with Istio requires configuring the Ingress Gateway plus defining a Destination Rule and a Virtual Service. In Knative it is just adding a few lines of code to the Service description. Have you noticed the “Set Traffic Distribution” button in the screen shot of the OpenShift Web Console? Here you can modify the distribution on the fly:

Auto-Scaling

Scale to zero is an interesting feature but without additional tricks (like pre-started containers or pods which aren’t available in Knative) it can be annoying because users have to wait until a new pod is started and ready to receive requests. Or it can lead to problems like time-outs in a microservices architecture if a scaled-to-zero service is called by another service and has to be started first.

On the other hand, if our application / microservice is hit hard with requests, a single pod may not be sufficient to serve them and we may need to scale up. And preferably scale up and down automatically.

Auto-scaling is accomplished by simply adding a few annotation statements to the Knative Service description:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-nodejs
spec:
  template:
    metadata:
      name: helloworld-nodejs-v3
      annotations:
        # the minimum number of pods to scale down to
        autoscaling.knative.dev/minScale: "1"
        # the maximum number of pods to scale up to
        autoscaling.knative.dev/maxScale: "5"
        # Target in-flight-requests per pod.
        autoscaling.knative.dev/target: "1"
    spec:
      containers:
        - image: image-registry.openshift-image-registry.svc:5000/knativetutorial/helloword:latest
          env:
            - name: TARGET
              value: "Node.js Sample v3 -- Scaling"

minScale: “1” prevents scale to zero, there will always be at least 1 pod active.
maxScale: “5” will allow to start a maximum of 5 pods.
target: “1” limits every started pod to 1 concurrent request at a time, this is just to make it easier to demo.

All auto-scale parameters are listed and described here.

Here I deployed the auto-scale example and run a load test using the hey command against it:

$ hey -z 30s -c 50 http://helloworld-nodejs-knativetutorial.apps-crc.testing/

Summary:
  Total:	30.0584 secs
  Slowest:	1.0555 secs
  Fastest:	0.0032 secs
  Average:	0.1047 secs
  Requests/sec:	477.1042
  
  Total data:	501935 bytes
  Size/request:	35 bytes

Response time histogram:
  0.003 [1]	    |
  0.108 [9563]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.214 [3308]	|■■■■■■■■■■■■■■
  0.319 [899]	|■■■■
  0.424 [367]	|■■
  0.529 [128]	|■
  0.635 [42]	|
  0.740 [15]	|
  0.845 [10]	|
  0.950 [5]	    |
  1.056 [3]	    |

Latency distribution:
  10% in 0.0249 secs
  25% in 0.0450 secs
  50% in 0.0776 secs
  75% in 0.1311 secs
  90% in 0.2157 secs
  95% in 0.2936 secs
  99% in 0.4587 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0001 secs, 0.0032 secs, 1.0555 secs
  DNS-lookup:	0.0001 secs, 0.0000 secs, 0.0197 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0079 secs
  resp wait:	0.1043 secs, 0.0031 secs, 1.0550 secs
  resp read:	0.0002 secs, 0.0000 secs, 0.3235 secs

Status code distribution:
  [200]	14341 responses

$ oc get pod
NAME                                               READY   STATUS    RESTARTS   AGE
helloworld-nodejs-v3-deployment-66d7447b76-4dhql   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-pvxqg   2/2     Running   0          29s
helloworld-nodejs-v3-deployment-66d7447b76-qxkbc   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-vhc69   2/2     Running   0          28s
helloworld-nodejs-v3-deployment-66d7447b76-wphwm   2/2     Running   0          2m35s

In the end of the output we see 5 pods are started, one of them for a longer time (2m 35s) than the rest. That is the minScale: “1” pre-started pod.

Jakarta EE Example from Cloud Native Starter

I wanted to see how easy it is to deploy any form of application using Knative Serving.

I used the authors-java-jee microservice that is part of our Cloud Native Starter project and that we use in an exercise of an OpenShift workshop. A Container image of this service is stored on Dockerhub in my colleague Niklas Heidloffs registry as nheidloff/authors:v1

This is the Knative service.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v1
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1

When I deployed this I noticed that it never starts (you need to scroll the following view to the right to see the problem):

$ kn service list
NAME          URL                                                   LATEST   AGE   CONDITIONS   READY     REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing            33s   0 OK / 3     Unknown   RevisionMissing : Configuration "authors-jee" is waiting for a Revision to become ready.

$ oc get pod
NAME                                         READY   STATUS    RESTARTS   AGE
authors-jee-v1-deployment-7dd4b989cf-v9sv9   1/2     Running   0          42s

The user-container in the pod never starts and the Revision never becomes ready. Why is that?

To understand this problem you have to know that there are two versions of the authors service: The first version is written in Node.js and listens on port 3000. The second version is the JEE version we try to deploy here. To make it a drop-in replacement for the Node.js version it is configured to listen on port 3000, too. Very unusual for JEE and something Knative obviously does not pick up from the Docker metadata in the image.

The Knative Runtime Contract has some information about Inbound Network Connectivity, Protocols and Ports:

“The developer MAY specify this port at deployment; if the developer does not specify a port, the platform provider MUST provide a default. Only one inbound containerPort SHALL be specified in the core.v1.Container specification. The hostPort parameter SHOULD NOT be set by the developer or the platform provider, as it can interfere with ingress autoscaling. Regardless of its source, the selected port will be made available in the PORT environment variable.”

I found another piece of information regarding containerPort in the IBM Cloud documentation about Knative:

“By default, all incoming requests to your Knative service are sent to port 8080. You can change this setting by using the containerPort specification.”

I modified the Knative service yaml with ports.containerPort info:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000

Note the Revision ‘-v2’! Check after deployment:

$ kn service list
NAME          URL                                                   LATEST           AGE   CONDITIONS   READY   REASON
authors-jee   http://authors-jee-knativetutorial.apps-crc.testing   authors-jee-v2   11m   3 OK / 3     True    

$ oc get pod
NAME                                        READY   STATUS    RESTARTS   AGE
authors-jee-v2-deployment-997d44565-mhn7w   2/2     Running   0          51s

The authors-java-jee microservice is using Eclipse Microprofile and has implemented specific health checks. They can be used as Kubernetes readiness and liveness probes, the YAML file then looks like this, syntax is exactly the standard Kubernetes syntax:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000
        livenessProbe:
          exec:
            command: ["sh", "-c", "curl -s http://localhost:3000/"]
          initialDelaySeconds: 20
        readinessProbe:
          exec:
            command: ["sh", "-c", "curl -s http://localhost:3000/health | grep -q authors"]
          initialDelaySeconds: 40

Microservices Architectures and Knative private services

So far the examples I tested where all exposed on public URLs using the Kourier Ingress Gateway. This is useful for testing and also for externally accessible microservices, e.g. backend-for-frontend services that serve a browser-based web front end or a REST API for other external applications. The multitude of microservices in a cloud native application will only and should only be called cluster local and not be exposed with an external URL.

The Knative documentation has information on how to label a service cluster-local. You can either add a label to the Knative service or the Knative route. The steps described in the documentation are to 1. deploy the service and then 2. convert it to cluster-local via the label.

You can easily add the label to the YAML file and immediately deploy a cluster-local Knative service. This is the modified Jakarta EE example of the previous section:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: authors-jee
  labels:
    serving.knative.dev/visibility: cluster-local
spec:
  template:
    metadata:
      name: authors-jee-v2
    spec:
      containers:
      - image: docker.io/nheidloff/authors:v1
        ports:
        - containerPort: 3000

When this is deployed to OpenShift, the correct URL shows up in the Route:

Of course you can no longer open the URL in your browser, this address is only available from within the Kubernetes cluster.

Debugging Tips

There are new places to look for information as to why a Knative service doesn’t work. Here is a list of helpful commands and examples:

Display the Knative service:

 $ kn service list
 NAME          URL                                                   LATEST   AGE    CONDITIONS   READY   REASON
 authors-jee   http://authors-jee-knativetutorial.apps-crc.testing            3m7s   0 OK / 3     False   RevisionMissing : Configuration "authors-jee" does not have any ready Revision.

It is normal and to be expected that the revision is not available for some time immediately after the deployment because the application container needs to start first. But in this example the revision isn’t available after over 3 minutes and that is not normal.

You can also display Knative service info using oc instead of kn by using ‘kservice’:

 $ oc get kservice
 NAME          URL                                                   LATESTCREATED    LATESTREADY   READY   REASON
 authors-jee   http://authors-jee-knativetutorial.apps-crc.testing   authors-jee-v2                 False   RevisionMissing

Check the pod:

 $ oc get pod
 No resources found in knativetutorial namespace.

That is bad: no pod means no logs to look at.

Get information about the revision:

   $ oc get revision
   NAME             CONFIG NAME   K8S SERVICE NAME   GENERATION   READY   REASON
   authors-jee-v2   authors-jee                      1            False   ContainerMissing

   $ oc get revision authors-jee-v2 -o yaml
   apiVersion: serving.knative.dev/v1
   kind: Revision
   [...]
   status:
     conditions:
     - lastTransitionTime: "2020-06-03T08:12:49Z"
       message: 'Unable to fetch image "docker.io/nheidloff/authors:1": failed to resolve
         image to digest: failed to fetch image information: GET https://index.docker.io/v2/nheidloff/authors/manifests/1:
       MANIFEST_UNKNOWN: manifest unknown; map[Tag:1]'
     reason: ContainerMissing
     status: "False"
     type: ContainerHealthy
   - lastTransitionTime: "2020-06-03T08:12:49Z"
     message: 'Unable to fetch image "docker.io/nheidloff/authors:1": failed to resolve
       image to digest: failed to fetch image information: GET https://index.docker.io/v2/nheidloff/authors/manifests/1:
       MANIFEST_UNKNOWN: manifest unknown; map[Tag:1]'
     reason: ContainerMissing
     status: "False"
     type: Ready
   - lastTransitionTime: "2020-06-03T08:12:47Z"
     status: Unknown
     type: ResourcesAvailable
 [...]

The conditions under the status topic show that I have (on purpose as a demo) mistyped the Container image tag.

This is a real example:

 $ oc get revision helloworld-nodejs-v1 -o yaml
 [...]
 status:
   conditions:
   - lastTransitionTime: "2020-05-28T06:42:14Z"
     message: The target could not be activated.
     reason: TimedOut
     severity: Info
     status: "False"
     type: Active
   - lastTransitionTime: "2020-05-28T06:40:04Z"
     status: Unknown
     type: ContainerHealthy
   - lastTransitionTime: "2020-05-28T06:40:05Z"
     message: '0/1 nodes are available: 1 Insufficient cpu.'
     reason: Unschedulable
     status: "False"
     type: Ready
   - lastTransitionTime: "2020-05-28T06:40:05Z"
     message: '0/1 nodes are available: 1 Insufficient cpu.'
     reason: Unschedulable
       status: "False"
       type: ResourcesAvailable

These conditions clearly show that the cluster is under CPU pressure and unable to schedule a new pod. This was on my first CRC configuration that used only 6 vCPUs.

In my next blog article in this series I will talk about Knative Eventing.

Serverless and Knative - Part 1: Installing Knative on CodeReady Containers

2020-06-02T00:00:00+00:00

I have worked with Kubernetes for quite some time now, also with Istio Service Mesh. Recently I decided that I want to explore Knative and its possibilities.

Note: This blog post and the following 2 posts on Knative are based on Red Hat OpenShift. The instructions unfortunately don’t seem to work anymore and are also based on a very old version of Knative (v0.12). As I no longer have access to OpenShift (I retired from IBM) I can’t update the blog articles with tested and working instructions. But if you want to have a more current Knative Serving experience based on Minikube, you can test drive a workshop I have created. It is currently based on Knative v0.26.

So what is Knative? The Knative web site describes it as “components build on top of Kubernetes, abstracting away the complex details and enabling developers to focus on what matters.” It has two distinct components, originally it were three:

Knative Build. It is no longer part of Knative, it is now a project of its own: “Tekton”
Knative Serving, responsible for deploying and running containers, also networking and auto-scaling. Auto-scaling allows scale to zero and is the main reason why Knative is referred to as Serverless platform.
Knative Eventing, connecting Knative services (deployed by Knative Serving) with events or streams of events.

This will be a series of blogs about installing Knative, Knative Serving, and Knative Eventing.

In order to explore Knative you need to have access to an instance, of course, and that may require installing it yourself. The Knative documentation (for v0.12) has instructions on how to install it on many different Kubernetes platforms, including Minikube. Perfect, Knative on my notebook.

Installation

I followed the instructions for Minikube and installed it, and started a tutorial. At some point, I finished for the day, and stopped Minikube. The next morning it wouldn’t start again. I tried to find out what went wrong and in the end deleted the Minikube profile, recreated it, and reinstalled Knative again. Just out of curiosity I restarted Minikube and ran into the very same problem. This time I was a little more successful with my investigation and found this issue: https://github.com/knative/eventing/issues/2544. I thought about moving to Knative 0.14 shortly but then decided to test it on OpenShift. If you read some of my previous blogs you may know that I am a fan of CodeReady Containers (CRC).

Knative on Red Hat OpenShift is called OpenShift Serverless. It has been a preview (“beta”) for quite some time but since end of April 2020 it is GA, generally available, no longer preview only. According to the Red Hat OpenShift documentation OpenShift Serverless v1.7.0 is based on Knative 0.13.2 (as of May 1st, 2020) and it is tested on OpenShift 4.3 and 4.4. The CRC version I am currently using (v1.10) is built on top of OpenShift 4.4. So it should work.

The hardware or cluster size requirements for OpenShift Serverless are steep: minimum 10 CPUs and 40 GB of RAM. I only have 8 vCPUs (4 cores) and 32 GB of RAM in my notebook and I do need to run an Operating System besides CRC but I thought I give it a try. I started Knative installation on a CRC config using 6 vCPUs and 20 GB of RAM and so far it seems to work. I have tried it on smaller configurations and got unschedulable pods (Memory and/or CPU pressure).

Installation is accomplished via an OpenShift Serverless Operator and it took me probably less then 20 minutes to have both Knative Serving and Eventing installed by just following the instructions:

Install the OpenShift Serverless operator
Create a namespace for Knative Serving
Create Knative Serving via the Serverless operators API. This also installs Kourier as “an open-source lightweight Knative Ingress based on Envoy.” Kourier is a lightweight replacement for Istio.
Create a namespace for Knative Eventing
Create Knative Eventing via the Serverless operators API.

I have started and stopped CRC many times now and it doesn’t have the issues that Minikube had.

As a future exercise I will test the Knative Add-on for the IBM Cloud Kubernetes Service. This installs Knative 0.14 together with Istio on top of Kubernetes and requires a minimum of 3 worker nodes with 4 CPUs and 16 GB om memory (b3c.4x16 is the machine specification).

In the next blog article I will cover Knative Serving with an example from the Knative documentation.

Two great additions to ‘kubectl’

2020-05-20T00:00:00+00:00

I started to learn Kubernetes in its vanilla form. Almost a year ago I made my first steps on Red Hat OpenShift. From then on, going back to vanilla Kubernetes made me miss the easy way you switch namespaces (aka projects) in OpenShift. With ‘oc project’ it is like switching directories on your notebook. You can do that with ‘kubectl’ somehow but it is not as simple.

Recently I found 2 power tools for kubectl: ‘kubectx’ and ‘kubens’. Ahmet Alp Balkan, a Google Software Engineer, created them and open sourced them (https://github.com/ahmetb/kubectx).

The Github repo has installation instructions for macOS and diferent flavours of Linux. When you install them, also make sure to install ‘fzf’ (“A command-line fuzzy finder”, https://github.com/junegunn/fzf), it is a cool addition.

kubens

‘kubens’ allows you to quickly switch namespaces in Kubernetes. Normally you work in ‘default’ and whenever you need to check something or do something in another namespace you need to add the ‘-n namespace’ parameter to your command.

‘kubens istio-system’ will make ‘istio-system’ your new home and a subsequent ‘kubectl get pod’ or ‘kubectl get svc’ will show the pods and services in istio-system. Thats not all.

‘kubens’ without a parameter will list all namespaces and with ‘fzf’ installed too you have a selectable list:

I think that is even better than ‘oc projects’!

kubectx

‘kubectx’ is really helpful when you work with multiple Kubernetes clusters. I typically work with a Kubernetes cluster on the IBM Cloud (IKS) and then very often start CRC (CodeReady Containers) to try something out on OpenShift. When I log into OpenShift, my connection to the IKS cluster drops. It actually doesn’t drop but the kube context is switched to CRC. With ‘kubectx’ you can switch between them.

In this example I have two contexts, one is CRC, the other IKS (Kubernetes on IBM Cloud):

Not exactly easy to know which one is which, isn’t it? But you can set aliases for the entries like this:

$ kubectx CRC=default/api-crc-testing:6443/kube:admin
$ kubectx IKS=knative/br1td2of0j1q10rc8aj0

And then you get a list with recognizable names:

You can now switch via the list. In addition, with ‘kubectx -‘ you can switch to the previous context.

When you constantly create new kube contexts, e.g. create new CRC or Minikube instances, this list may grow and get unmanageable. But with ‘kubectx -d ' you can delete entries from the list. (They will still be in the kube context, though.)