KubeCon + CloudNativeCon Europe Virtual | August 17-20, 2020 | Don’t Miss Out | Learn more



Extending Kubernetes to an unlimited one through tensile-kube

By | Blog

Member Blog Post

Guest Post from Weidong Cai and Ye Yin of Tencent

Recently, the Tencent Games container team named Tenc has open sourced the Kubernetes (K8s) multi-cluster scheduling project tensile-kube. This blog will briefly introduce the tensile-kube.

Birth of tensile-kube

There are dozens of online K8s clusters on the Tenc computing platform. These clusters have some fragmented resources that cannot be effectively used. A common scenario is that a job requires N resources, but the remaining resources of the existing clusters A, B, C, etc. do not satisfy N, and the total resources of clusters A, B, and C can also satisfy N. In addition, we have another scenario that users have services that need to be published to multiple clusters at the same time.

In fact, the industry has had similar needs for a long time and solutions such as federation have been born. However, these will be a large transformation cost for the existing K8s while adding a lot of complicated CRDs. Therefore, we independently developed a more lightweight solution named tensile-kube based on virtual-kubelet.

Introduction to virtual-kubelet (VK)

VK is a basic library that is open sourced by Microsoft. Its implementation is usually called a provider. The early VK included various providers. At present, the VK community has separated it into three parts: virtual-kubelet, node-cli and provider. The virtual-kubelet provides an abstract capability. By implementing and register the provider in a K8s cluster, users’ pods can be scheduled to this virtual node. The core capability of virtual-kubelet is the life cycle control of Node and Pod.

Architecture of tensile-kube

There are 4 core components of tensile-kube:

  • virtual-node
    It is a K8s provider implemented based on virtual-kubelet and multiple controllers are added to synchronize resources such as ConfigMap, Secret and Service.
  • webhook
    It helps convert the fields in the Pod that may interfere with the scheduling of the upper-layer cluster and write them into the annotation. The virtual-node restores them when creating the Pod to avoid affecting the user’s expected scheduling results.
  • descheduler
    To avoid the impact of resource fragmentation, this component has been redeveloped to make it more suitable for this scenario.
  • multi-scheduler
    It is developed based on scheduling framework and mainly used for connecting the API Server of the lower-level clusters to avoid the impact of resource fragmentation and determine whether the virtual-node resources are sufficient when scheduling.


  • multi-scheduler would be better for small clusters. If you want to use it, you need to replace kube-scheduler in the upper cluster with multi-scheduler.
  • large clusters are recommended to use descheduler while multi-scheduler is too heavy.

Features of tensile-kube

  • No intrusion into the native K8s cluster
    In tensile-kube, the connection between the upper and lower clusters is established through virtual-node. The upper-layer Pod is scheduled to the virtual-node through the scheduler and then the Pod is created to a lower-layer cluster. The entire solution is non-intrusive to K8s.
  • Low cost of K8s upper platform reconstruction
    From the perspective of the upper-level K8s platform, the clusters connected through tensile-kube is a large cluster. The difference can almost be shield, so for the platform, almost no transformation is required.
  • Development based on native virtual-kubelet
    Tensile-kube is based on the native virtual-kubelet without too many extra dependencies. It just depends on the features of native virtual-kubelet which would enable us upgrades the cluster more conveniently.

All of these make tensile-kube achieve the scheduling and management of multiple clusters easily. In addition, it is suitable for most scenarios.

Main abilities of tensile-kube

  • Cluster fragment resource integration
    There are resource fragments existing in clusters. Through tensile-kube, these fragments can be combined into a large resource pool to make full use of idle resources.
  • Ability of multi-cluster scheduling
    Combined with Affinity and etc., Pod’s multi-cluster scheduling can be achieved. For clusters based on K8s 1.16 and above, we can also implement more fine-grained cross-cluster scheduling based on TopologySpreadConstraint.
  • Cross-cluster communication
    Tensile-kube natively provides the ability to communicate across clusters through Service. But the premise is that the network between the different lowers clusters are able to accessed through Pod IP, such as: private clusters share a flannel.
  • Convenient operation and maintenance cluster
    When we upgrade or change a single cluster, we often need to notify business migration, cluster shielding scheduling, etc. Through tensile-kube, we only need to set the virtual-node to be unschedulable.
  • Native kubectl capabilities
    Tensile-kube supports all operations of native kubectl, including: kubectl logs and kubectl exec. This is very convenient for the operators and maintainers to manage the pods on the lower-level clusters without additional learning costs.

Other possible applications

  • Edge computing
  • Hybrid cloud

Feature work

It has been 6 years since Tenc brought K8s into production environment in 2014. Tensile-kube is born from our years of container-related experience. At present, based on tensile-kube, we have expanded the original cluster from two thousand nodes to more ten thousand nodes. Previously, we have submitted many PRs to the VK community to expand VK functions and optimize its performance. In the future, we will continue to improve VK and tensile-kube in the following aspects:

  1. Multi-cluster scheduler optimization;
  2. Real-time optimization of provider status synchronization;
  3. Resource name conflicts in sub clusters.

Project address: https://github.com/virtual-kubelet/tensile-kube


Why the Kubernetes Scheduler Is Not Enough for Your AI Workloads

By | Blog

Member Post

KubeCon + CloudNativeCon sponsor guest post from Alaa Youssef, manager of the Container Cloud Platform at IBM Research

AI Workloads on The Cloud

The use of container clouds orchestrated by Kubernetes, for the execution of AI (Artificial Intelligence) and ML (Machine Learning) workloads has led many to question the suitability and effectiveness of Kubernetes’ resource management and scheduling to meet all the requirements imposed by these workloads.

To begin with, some of the typical machine learning and deep learning frameworks, such as SparkFlinkKubeflowFfDL, etc., require multiple learners or executors to be concurrently scheduled and started in order for the application to be able to run. Also, co-locating these workers on the same node, or placing them on topologically close nodes, is desired to minimize the communication delays of large data sets. More recent trends show tendency towards massively parallel and elastic jobs, where a big number of short running tasks (minutes, seconds, or even less) is spawned to consume the available resources. The more resources can be dedicated to start more parallel tasks the faster the job will finish. In the end the amount of consumed resources is going to be more or less the same, but it is the overall job response time that is going to be impacted negatively if the allocated resources are constrained. In an offline processing mode, the response time is not a critical factor to the end user.

New use cases have also emerged where users increasingly use these frameworks to gain insights interactively, online; performing interactive exploration of data, as well as online processing. For example, using a Jupyter notebook, a data scientist may issue commands to start a ML job to analyze a big data set waiting for the results to start a subsequent processing job, in an interactive session.

Managing Jobs vs Tasks

When dealing with big numbers of AI/ML jobs submitted on daily or even hourly basis, high variability in the demand for resources is created. Some of these jobs may arrive almost simultaneously, or create a backlog of work, and require an arbitration policy that decides which job goes first given the current cloud resources available. Job priorities, classes of service, and user quotas, enable the formulation of meaningful policies, from users’ perspective, and linking them to the charging model. For scarce resource, such as GPUs, the ability to buffer and prioritize jobs, as well as to enforce quotas, becomes more crucial. Note that whole jobs, as opposed to individual tasks, is the subject of queuing and control, since it does not make much sense to focus only on individual tasks, as the partial execution of tasks from multiple jobs may lead to partial deadlocks, and many jobs may be simultaneously active while none is able to proceed to completion.

It’s Only a Cloud!

We often hear the term “the sky is the limit” to refer to unlimited amount of something. Cloud enthusiasts like to think of cloud resources as infinite with no limit! They often praise the elasticity of the cloud and its ability to absorb any resource demand whatsoever. While this is an ideal model to aspire for, one must balance it with the reality that cloud operators face, which is having to operate profitable businesses that offer reasonable pricing of offered resources and services. These prices cannot be reasonable if the cloud provider has to own seemingly infinite resources. In fact, minimizing their cost starts with operating their physical resources (hardware) at higher utilization points. Luckily, the common wisdom phrase did not state that the cloud is the limit but rather that the sky is.

At the cluster level, cluster owners may request to expand or shrink their cluster resources, and all providers offer the ability to achieve that. However, response times vary. Typically, it is on the order of minutes, and depends on how many additional worker nodes are being added. In the case of bare-metal machines with special hardware configuration, like certain GPU type for example, it may take longer to fulfill a cluster scale up request. While cluster scaling is a useful mechanism for relatively longer time-window capacity planning, cluster owners cannot rely on this mechanism to respond to instantaneous demand fluctuations induced by resource hungry AI and ML workloads.

The Evolving Multi-cloud and Multi-cluster Patterns

The typical enterprise today uses services from multiple cloud providers. An organization owning tens of Kubernetes clusters is not uncommon today. Smaller clusters are easier to manage and in the case of failure the blast radius of impacted applications and users is limited. Therefore a sharded multi-cluster architecture appeals to both service providers and operations teams. This creates an additional management decision that needs to be made when running a certain AI workload, as to where to run it; on which cluster should it be placed. A static assignment of users or application sets to clusters is too naïve to efficiently utilize available multi-cluster resources.

The Rise of Hybrid Cloud and Edge Computing

A scenario, which is often cited as an advantage of cloud computing, is the ability to burst from private cloud or private IT infrastructure to public cloud at times of unexpected or seasonal high demands. While with traditional retail applications this seasonality may be well studied and anticipated, on the other hand, with AI and ML workloads in general this need for offloading local resources or bursting to a public cloud may arise more frequently and at any time, as a resource hungry massively parallel job may be started at any moment, necessitating the need to instantaneously burst from private to public cloud in a dynamically managed seamless way.

Advancements in the field of IoT (Internet of Things) and the surge in use of devices and sensors at the edge of the network, has led to the emergence of Edge Computing as a new paradigm. In this paradigm, the compute power is brought near to the sources of data, where initial data analysis and learning steps may take place near the data generation sources, instead of the overhead of having to transfer enormous data streams all the way back to a centralized cloud computing location. In some situations, this data streaming may not be feasible due to lack of availability of fast and reliable network connectivity, and hence the need for Kubernetes clusters at the edge of the network.

The Kubernetes Scheduler Can’t Do Everything

Now that we have covered the landscape of issues, as well as evolving runtime patterns, associated with executing AI workloads in the cloud, we can look at what the Kubernetes scheduler is good at addressing from among these issues and what it is not suited for.

The scheduler is responsible for the placement of PODs (the schedulable unit in Kubernetes; you can think of it as a container potentially together with supporting side-car containers) within a Kubernetes cluster. It is part and parcel of the Kubernetes architecture and it is good for placing PODs on the right worker nodes, within a cluster, while maintaining constraints such as available capacity, resource requests and limits, and affinity to certain nodes. A POD may represent a task in a job, or an executor in which multiple tasks may be executed by the AI/ML framework. Discussing these different execution models for frameworks and the pros and cons of each is beyond the scope of this article.

What the Kubernetes scheduler cannot do is managing jobs holistically. It is obvious in the multi-cluster and hybrid scenarios that a scheduler sitting within a single cluster cannot have the global view to dispatch jobs to multiple clusters. But even in single cluster environments, it lacks the ability to manage jobs, and instead jobs must be broken down, immediately as soon as they are received by the Kubernetes API server, to their constituting PODs and submitted to the scheduler. This may cause a problem of excessive numbers of pending pods, in times of high demand, which may overwhelm the scheduler other controllers, and the underlying etcd persistence layer, which slows down the overall cluster performance significantly.

What is The Solution?

All the above indicates that something is missing in the overall approach to resource management of AI/ML workloads in Kubernetes based environments and suggests that the best way to manage such workloads is to follow a two-level resource management approach that is Kube-native.

Kube-native two-level resource management for AI Workloads.

First, what is Kube-native?
What is meant is that the solution is built as a Kubernetes extension using the extensible frameworks that it offers. Any new resource management component should be realized as a Kubernetes operator (controller). In addition, the API and interaction mechanism exposed to the end users and admins should be a seamless extension of the Kubernetes API. Users and admins should be able to use the standard Kubectl cli, for example, to manage the AI jobs’ life cycle.

Second, what are the two levels of resource management?
The first level is the existing scheduler level. As explained above, a scheduler responsible for the placement of PODs within each Kubernetes cluster is needed for placing PODs, or sometimes groups of PODs together, on the right nodes, while maintaining constraints such as co-location, affinity, co-scheduling (gang scheduling), etc. Your vanilla Kube scheduler does not do all of that, but you can replace it by a scheduler that does all or some of these functions. Several open source Kube schedulers are available, like VolcanoYuniKorn, and Safe Scheduler. Each of these schedulers addresses some of these requirements.

The second level, or the higher-level resource manager is responsible for queuing and dispatching the AI/ML jobs, and enforcement of user quotas. Every job submitted to the system is received and queued for execution by the second level resource manager. It has the responsibility to decide when to release the job to be served, and where that job should be served, i.e., on which cluster. It implements policies that regulate the flow of jobs to the clusters, allocate resources to jobs at coarse granularity level, and realize the SLA differentiation imposed by priorities or classes of service. The important thing to note about it is that it operates on the job level, not the task or POD level, hence the lower lever (first level) schedulers, running in each cluster, are not overwhelmed with pending PODs that should not be executed yet. Also, once a decision is made to execute a job, its corresponding resources, including its PODs, are created only on the selected target cluster, at that moment.

In order to perform its job properly and be able to make these decisions, the second level resource manager needs to tap to the resource monitoring streams on the target clusters. Coupled with the right policy configurations, it is capable of queuing and dispatching jobs properly. Depending on implementation details, the second level manager may require an “agent” on each target cluster to assist it in accomplishing its goals. The agent would receive the dispatched job and create locally its corresponding PODs, in this case, instead of the Dispatcher creating them remotely.

To the contrary of the current state of the art for the first level scheduler where several open as well as proprietary solutions exist, including the Kube default scheduler, not much attention has been paid by the community to the second level resource manager. One open source project — Multi-Cluster App Dispatcher — addresses this important aspect of the solution. Hopefully, we can shed more light on that in future blog posts.

Alaa Youssef manages the Container Cloud Platform team at IBM T.J. Watson Research Center. His research focus is on cloud computing, secure distributed systems, and continuous delivery of large software systems. He has authored and co-authored many technical publications, and holds over a dozen issued patents. Dr. Youssef is a senior architect who has held multiple technical and management positions in IBM Research, and in Services in multiple geographies. He received his PhD in Computer Science from Old Dominion University, Virginia, and his MSc in Computer

Common Kubernetes Config Security Threats

By | Blog

Member Post

Guest post originally published on the Fairwinds blog by Joe Pelletier, VP of strategy at Fairwinds

Kubernetes Config Security Threats

Securing workloads in Kubernetes is an important part of overall cluster security. The overall goal should be to ensure that containers are running with as minimal privileges as possible. This includes avoiding privilege escalation, not running containers as a root user, and using read only file systems wherever possible.

Security vulnerabilities can slip into production because of oversight or inexperience. Speed to delivery vs. critical security safeguards are often at odds as teams attempt to balance the velocity of engineering with the reactionary pace of security. This balancing act can result in messy Kubernetes configurations and unnecessary risk. Problems can arise if workloads are mis-configured by developers through inexperience or neglect. 

Unfortunately, individual application developers often neglect security configuration for each workload. For example, it’s often easier to over-permission a deployment with root access to just get something working. Forcing individual contributors to design their own security configuration all but ensures inconsistency and mistakes.

Common Kubernetes Security Configurations That Lead to Vulnerable Deployments

So how do you quickly and proactively identify Kubernetes security misconfigurations to prevent breaches? In our experience there are common Kubernetes security misconfigurations that lead to vulnerable deployments.

Not identifying and addressing these configurations can have negative business consequences. For example, if a container runs as root but doesn’t necessarily need this level of access, then a malicious container could have the privileges to steal data or cause other damage to the system.

Configuration Severity Description
security.hostIPCSet danger Fails when hostIPC attribute is configured.
security.hostPIDSet danger Fails when hostPID attribute is configured.
security.notReadOnlyRootFilesystem warning Fails when securityContext.readOnlyRootFilesystem is not true.
security.privilegeEscalationAllowed danger Fails when securityContext.allowPrivilegeEscalation is true.
security.runAsRootAllowed danger Fails when securityContext.runAsNonRoot is not true.
security.runAsPrivileged danger Fails when securityContext.privileged is true.
security.insecureCapabilities warning Fails when securityContext.capabilities includes one of the capabilities listed here
security.dangerousCapabilities danger Fails when securityContext.capabilities includes one of the capabilities listed here

To address these common kubernetes security threats, teams need a way to quickly check configurations found in the securityContext attribute for both Kubernetes pods and containers. Using a configuration validation tool that provides visibility into an organization’s Kubernetes security posture by auditing workloads and validating configurations for weaknesses, container vulnerabilities, and misconfigured deployments can help.

There are open source tools and configuration validation software available that can present configuration findings, keep a historical record of the results across all clusters and offer remediation guidance. Look for one that allows you to track and prioritize security, efficiency and reliability issues, collaborate across teams, and apply best practices as applications move from development to production.

Learn more about Fairwinds Insights or the open source tool Polaris free to download from GitHub.


How the Productivity Software Company Nulab Boosted Its Own Productivity with Microservices and Kubernetes

By | Blog

CNCF Staff Post

The company behind the productivity tools Backlog, Cacoo, and Typetalk, Nulab serves 4 million users around the world. “Our mission is to make productivity fun and enjoyable for everyone, and that’s regardless of where they’re working from,” says Bing Han, Head of Marketing. “A lot of our users have different international offices and are using our tools to help them communicate.”

But Nulab found its own productivity was hampered by the monolith legacy Java application for Cacoo that was hosted on AWS. “The technology we were using was too old,” says Software Engineer Jay Chen. A diagramming tool, Cacoo was built seven years ago on Java 6, and every version upgrade would take one engineer two weeks of full-time work and then another week of testing. “It was really difficult to find all the possible bugs in our system since Cacoo was growing too huge,” he says. “It was really scary for us since we had to deploy and see how it goes.”

The application was also difficult to scale up, making it hard to add new features or functionality into the system. “Every small change takes longer,” Chen says.

Plus, Nulab, which was founded in Japan in 2004, has in recent years expanded to offices on three continents. In 2017, Nulab’s technology leader set out to solve how engineers can work globally. The team determined that the solution was to break down the monolith into microservices written in Golang.  “Microservices were absolutely necessary for members to develop the same application across time zones,” says Chen. The migration is about 80% done; there’s still one microservice using Java. “That’s the beauty of microservices,” he says.

For orchestration, the team considered Docker Swarm but decided to go with Kubernetes. “Kubernetes supported more of the functionality that we need,” Chen says. “Kubernetes has the API that’s integrated with all of the client, an user-friendly UI that we can just log in from our credential. And kubectl is a pretty useful command. This is really easy to use.”

The Nulab team rolls its own Kubernetes for Cacoo applications on AWS EC2 with kops. There is also other CNCF technology in the stack: Prometheus, Envoy, and gRPC. 

With Kubernetes, Nulab has seen stellar results. Deploying new code to production is six times faster: 6 minutes, down from 40 minutes. “That was a big difference,” says Chen. “Before microservices, for every small change we had to wait for an hour to deploy.”

Kubernetes has also improved reliability: “Our SRE team doesn’t have to keep logging into our server to check why the server is down, and then restart the server,” he says. “Kubernetes manages those things. It’s self-healing. If the microservice is down, it’s just restarting automatically without any downtime.”

Microservices have, as expected, enabled Nulab’s distributed team to work more efficiently. “We are now able to assign all of the responsibility and ownership for each microservice,” Chen says. “So now we don’t really have the code merge conflict anymore. We can all work smoothly. It’s saving a lot of time for us as well.”

To find out more about Nulab’s cloud native journey, read the full case study.

OpenTelemetry: Future-Proofing Your Instrumentation

By | Blog

Member Blog

Guest post originally published on the New Relic blog by John Watson, lead engineer, and Lavanya Chockalingam, senior product marketing manager at New Relic

As a developer, you care about the performance of your applications, and you know just how challenging it can be to find the right instrumentation to capture telemetry data. One option is to use a proprietary agent, but what if you don’t want to be locked into that agent’s vendor?

You could use one of the open standards like OpenCensus or OpenTracing, but it can be tricky to decide which option is right for your application and workflows. Any option you choose must be able to scale and adapt to your needs as your applications change over time. But it’s also become increasingly important that the instrumentation you choose adheres to standards that can be shared across vendors and projects.

Vendor-neutral open standards matter. And OpenTelemetry, a Cloud Native Computing Foundation (CNCF) project, has created an open specification you can use to standardize how you collect and send telemetry data to backend platforms.

In this post, we’ll explain more about the OpenTelemetry project—how it works, the benefits it offers, and how to get started.

What is OpenTelemetry?

At its inception, OpenTelemetry formed by merging the OpenTracing and OpenCensus projects. OpenTelemetry offers a single set of APIs and libraries that standardize how you collect and transfer telemetry data. OpenTelemetry provides a secure, vendor-neutral specification for instrumentation so that you can send data to distinct backends of your choice, such as New Relic.

The following components make up the OpenTelemetry project:

  • Specifications to drive consistency across all projects
  • APIs that contain interfaces and implementations based on the specifications
  • SDKs (reference implementations of the APIs) created specifically for languages like Java, Python, Go, Erlang, and more
  • Exporters that enable you to send data to a backend of your choice
  • Collectors that offer a vendor-agnostic implementation for processing and exporting

(We’ll dive more into these components in the architecture section below.)

Key terms

If you’re new to OpenTelemetry, these are the key terms you need to know:

  • Traces: A record of activity for a request through a distributed system. A trace is a directed acyclic graph of spans.
  • Spans: Named, timed operations representing a single operation within a trace. Spans can be nested to form a trace tree. Each trace contains a root span, which typically describes the end-to-end latency and, optionally, one or more sub-spans for its sub-operations.
  • Metrics: A raw measurement about a service, captured at runtime. OpenTelemetry defines three metric instruments: counter, measure, and observer. An observer supports an asynchronous API collecting metric data on demand, once per collection interval.
  • Context: A span contains a span context, which is a set of globally unique identifiers that represent the unique request that each span is a part of, representing the data required for moving trace information across service boundaries. OpenTelemetry also supports correlation context which can carry any user defined properties. Correlation context is not required and components may choose not to carry or store this information.
  • Context propagation: The means by which context is bundled and transferred between services, typically via HTTP headers. Context propagation is a key part of the OpenTelemetry system, and has some interesting use cases beyond tracing, for example when doing A/B testing. OpenTelemetry supports multiple protocols for context propagation and to avoid issues, it’s important that you use a single method throughout your application. For example, if you use the W3C specification in one service, you need to use it everywhere in your system.

Benefits of OpenTelemetry

By combining OpenTracing and OpenCensus into a single open standard, OpenTelemetry provides the following benefits:

  • Simplified choice. You don’t need to decide between one standard or the other. Are you currently using OpenTracing or OpenCensus? Don’t worry, OpenTelemetry offers backward compatibility for both projects.
  • Cross-platform. OpenTelemetry supports a variety of languages and backends. It represents a vendor-neutral path to capturing and transmitting telemetry to backends without altering existing instrumentation. This is an important project that satisfies developers looking for this kind of freedom.
  • Streamlined observability. As OpenTelemetry puts it, “Effective observability requires high-quality telemetry.” Expect to see more vendors move toward OpenTelemetry, as it’s far easier to accommodate and test against a single standard.

In short, instead of being bogged down in the nitty-gritty of decision making, you can focus your efforts on building amazing, reliable, solutions-based software. And that’s what it’s all about.

Using OpenTelemetry

The OpenTelemetry APIs and SDKs come with plenty of quick-start guides and documentation, so you can learn how to ingest data immediately. The quick-start guide for Java, for example, outlines how to acquire a tracer, create spans, add attributes, and propagate context across the various spans.

You will also find examples of common use cases to help get you started. These provide practical working code that demonstrates how to instrument HTTP/gRPC servers and clients, database connectors, and more.

After you have instrumented your application with OpenTelemetry trace APIs, you can send the trace data to an observability platform like New Relic or any other backend using one of the pre-built exporters in the OpenTelemetry registry.

Specifications for metrics and logs are still in the development phase, but once launched, they’ll play an important role in achieving OpenTelemetry’s main goal: to ensure libraries and frameworks have all built-in telemetry data types, which allows developers to ingest telemetry data without having to instrument it.

OpenTelemetry architecture components

Since OpenTelemetry is intended to be a cross-language framework for vendors and observability backends, it’s extremely flexible and extensible, yet it’s also quite complex. The architecture of a default implementation of OpenTelemetry breaks down into three components:

  1. The OpenTelemetry API
  2. The OpenTelemetry SDK, consisting of:
    • The Tracer pipeline
    • The Meter pipeline
    • A shared Context layer
  3. The Collector

Let’s take a look at each architecture component.

The OpenTelemetry API

Application developers use the Open Telemetry API to instrument their code, and library authors use it to write instrumentation directly into their libraries. The API does not address operational concerns or how data is sent to vendor backends.

We can break the  API into four parts:

  1. A Tracer API
  2. A Metrics API
  3. A Context API
  4. A set of semantic conventions

The Tracer API

The Tracer API supports the generation of spans, which are named, timed operations representing a contiguous segment of work in a trace. Spans can be assigned a traceId and, optionally, be annotated with time-stamped events. A Tracer tags spans with a name and a version. When looking at your data, the name and version associated with a Tracer allow you to track the instrumentation library that generated the spans.

The Metric API

The Metric API provides access to various types of Metric instruments, such as Counters and  Observers. Counters allow you to count things. Observers allow you to measure values at discrete points in time; for example, you’d use Observers to “observe” values that aren’t taking place in the context of a Span, such as current CPU load or the total number of bytes free on a disk. See the specification for a list of instruments that cover the full range of possible metric use cases.

The Context API

The Context API adds context information—like W3C Trace ContextZipkin B3 headers, or New Relic distributed tracing headers—to spans and traces that use the same “context.” In addition, this API allows you to track how spans propagate within a system. The context is updated as a trace propagates from one process to the next. Metric instruments always have access to the current context.

Semantic conventions

Finally, the OpenTelemetry API contains a set of semantic conventions that includes guidelines and rules for naming spans, attributes, and for associating errors with spans (among other things). By encoding this in the API specification, the OpenTelemetry project ensures that all instrumentation, regardless of author or language, contains the same semantic information. This capability is enormously valuable to vendors looking to provide a consistent APM experience for all users.

The OpenTelemetry SDK

Next, the OpenTelemetry SDK is an implementation of the OpenTelemetry API. The SDK consists of roughly three parts, analogous to the APIs we just covered—a Tracer, a Meter, and a shared Context layer that ties it all together.

Ideally, the SDK should fulfill 99% of standard use cases, but you can customize the SDK if needed. In the Tracer pipeline implementation, for example, you can customize anything aside from how the core implementation interacts with the shared Context layer. For example, within the implementation of the Tracer pipeline, you can customize the sampling algorithm that it uses.

The Tracer pipeline

When configuring the SDK, you associate one or more SpanProcessors with the implementation of the Tracer pipeline. A SpanProcessor watches the lifecycle of spans, and delivers spans to a SpanExporter when appropriate. Built into the SDK is a simple SpanProcessor that forwards finished spans directly to the exporter one at a time.

The SDK also contains a batching implementation that forwards finished spans in batches on a configurable interval. However, since the SpanProcessor implementation accepts plug-ins, you can implement your own and assign it custom behavior. For example, if your telemetry backend supports viewing “in progress” spans, you could create a SpanProcessor implementation that sends spans for export at every change to the span state.

At the end of the Tracer pipeline is the SpanExporter. An exporter’s job is simple: convert the spans from their OpenTelemetry representations to whatever the telemetry backend requires and send them to that backend. Providing a custom SpanExporter is the easiest way for a telemetry vendor to participate in the OpenTelemetry ecosystem.

The Meter pipeline

The Meter pipeline is considerably more complex than the Tracer pipeline, as metrics are more complex than spans.  The following description is based on the Java implementation of the SDK and may vary across languages.

The Meter pipeline creates and maintains metrics instrumentation of various types, including Counters and Observers. Each instance of an instrument will need to be aggregated in some way. By default, Counters are aggregated by summing the values, and Observers are aggregated by taking the last value recorded. All of the instrument types have a default aggregation defined for them.

(As of this writing, the ability to configure custom aggregations for metrics instrumentation is still in the proposal stage.)

Implementations of the Meter Pipeline currently vary across languages, but in all cases, metric aggregations are delivered to the MetricExporter. Again, as in the case of spans, vendors can provide their own exporter to convert the aggregated data generated by the metric aggregators to the types that are needed for the telemetry backend.

OpenTelemetry supports two styles of exporter: “push”-based exporters, where the exporter sends data to the backend on a timed interval, and “pull”-based exporters, where the backend will query for the data when it wants it. New Relic is an example of a push-based backend, and Prometheus is a pull-based backend.

The shared Context layer

The shared Context layer sits between the Tracer and Meter pipelines and enables all non-observer metric recordings to take place in the context of an executing span. You can customize the Context with propagators, which propagate the span context both in and out of the system. As stated earlier, all OpenTelemetry SDK implementations provide an implementation of the W3C Trace Context specification, but you can optionally include Zipkin B3 propagation, or others.

The Collector

The OpenTelemetry Collector is a stand-alone service that can ingest metrics and spans from various sources, including ZipkinJaeger, and OpenCensus. The collector can do tail-based sampling of spans, and enables exporting spans and metrics to a large number of vendor and open-source telemetry systems, including New Relic.

For more information about OpenTelemetry’s architecture, check out the full specification.

Start exploring OpenTelemetry today

Currently, OpenTelemetry is in beta phase. Right now is the ideal time to begin exploring what this project can do for you.

Check out this basic Java example that outlines how you can instrument your code to send metrics and traces to a backend like New Relic.

To find out more about other open source projects New Relic is involved in, visit New Relic Open Source.

Open Sourcing the etcd Security Audit

By | Blog

Guest post from Sahdev Zala and Xiang Li, maintainers for etcd 

We are proud to announce that the etcd team has successfully completed a 3rd party security audit for the etcd latest major release 3.4. The third party security audit was done for etcd v3.4.3 by Trail of Bits. We are thankful to the CNCF for sponsoring this audit. Also our big thanks to all the etcd maintainers, specially to Gyuho Lee, Hitoshi Mitake and Brandon Philips for working along with us during the whole process of auditing work. 

A report from the security audit is available in the etcd community repo. We recommend that you take a look at it for the details. The report covers the process, what has been reviewed, and the issues that have been identified to be addressed. The audit was performed as a mixture of manual and automated review. Automated review consisted of running various static analysis tools, such as errcheck, ineffassign, and go-sec. The google/gofuzz and dvyukov/go-fuzz testing harnesses were developed to test the etcd Write Ahead Log (WAL) implementation. Results were subsequently reviewed and triaged as necessary. Manual review focused on gaining familiarity with the implementation details of etcd in various areas, such as, configuration options, default settings, service discovery, WAL operations, Raft consensus and leader election, proxy and gateway. The various security areas evaluated include data validation, access controls, cryptography, logging, authentication, data exposure, denial of service and configuration. 

We are glad to see that there was no major issue found in the core components of etcd. According to the report summary, overall, the etcd codebase represents a mature and heavily adopted product. From the reported issues there was only one high severity issue which was found in the etcd gateway which is a simple TCP proxy that forwards network data to the etcd cluster. All the issues and severity are explained in great detail in the report. Issues are already addressed with needed code updates, documentation and better logging. The fixes are backported to supported versions of etcd, v3.3 and v3.4. These updated releases are now available. The security advisories are created using the GitHub Security tool to publish information about security vulnerabilities.

It is worth noting that a security audit is part of the graduation criteria for CNCF projects. Specifically, the graduation criteria says:

Have completed an independent and third party security audit with results published of similar scope and quality as the following example (including critical vulnerabilities addressed): https://github.com/envoyproxy/envoy#security-audit and all critical vulnerabilities need to be addressed before graduation.

The next step for the etcd team is to work on the project graduation. 

Security audits are one of the benefits of CNCF projects and we are grateful for them and the analysis performed by Trail of Bits, they were also involved in the detailed Kubernetes Security Audit last year. This analysis has provided some concrete areas we can work to improve and given us confidence in what we have. 

A Guide to Untangling the CNCF Cross-Community Relationships

By | Blog

Ambassador Post

Guest post from Diane Mueller, Director of Community Development at Red Hat

The adoption of CNCF technology and continuous growth in terms of projects, contributors, and end users has created one of the most active, dynamic open source ecosystems in the world. Since March 2018, when the first project, Kubernetes, officially ‘graduated’ as a CNCF project, 10 more projects have ‘Graduated,’ and a boatload of new ones (19) are currently being “Incubated” by the CNCF. It has become a daunting, almost impossible, task to navigate the complexity of the community, even as an active participant since the inception of the CNCF. 

While I am eternally grateful for the filters that are baked into the CNCF Interactive Landscape enabling us to easily navigate to each of the project’s home pages, the Landscape diagram doesn’t help explain in a digestible way the burgeoning complexity of the relationships between projects, participants, and the myriad of repositories that are bursting at the seams with innovations, ideas, and almost continuous release announcements. 

For the past few years at Red Hat, with the help of some open source tooling from Bitergia, I’ve been applying data visualization techniques to help us navigate the uncharted territory of project/participant relationships using network analysis. These interactive visualizations enable us to help nurture, track, and support the multiple CNCF and other open source communities that we participate in. 

Leveraging this data-driven approach has become a bit of a minor obsession of mine that has helped untangle key relationships, identify new entrants (both projects and participants), and facilitate conversations with participants that often were not even aware themselves of the key roles they were playing bridging multiple communities. 

Figure #1: Year to Date Diagram of Developers’ (pink dots) Git Activity in CNCF Graduated and Incubated projects as of July 2020. Note: There is a line (pink) when that developer has contributed to a given project (blue dots). A developer has more than one line, if that developer has contributed to more than one project – these are the connector’ participants. 

Interactive Visualization tools help us gain a better understanding of the underlying relationships between the projects and their participants. One still needs a basic understanding of the roles each of the projects plays in the ecosystem to utilize the information accurately. 

The ‘jellyfish’ diagram represents all of the participants and their relationships with both Graduated and Incubated CNCF projects. The current data set includes 13K+ participants, 250+ repositories, and just goes back to year to date as of July 2020. 

There are quite a few more ancillary repositories that we have not yet incorporated into the data set as it’s a bit of a moving target, as you might imagine. For this visualization, we’ve just included the participants’ code contributions.

Each dot represents a developer that has committed at least one piece of code, logged an issue, or made a pull request to repositories.  All the data points come from publicly available data from Github similar to the data that CNCF’s Devstats dashboards

Untangling the Community Connections

If you take a look at the left side of the ‘jellyfish’ diagram above, you will notice that the Linkerd, TUF, and Falco appear a bit less entangled with the rest of the CNCF projects, but still share participants with Kubernetes. This makes it relatively easy to ‘see’ the ‘connector’ personas that are bridging these communities and (one hopes) facilitating communication. At the very least, the network analysis helps us pinpoint participants with knowledge of both projects that can potentially be tapped to help bridge any gaps. 

In the denser section of the diagram, we see those projects whose participants are highly interconnected with multiple projects. This activity highlights the interdependencies of the communities and the complexity of the communication required to ensure healthy, aligned project releases, and road maps. These connectors are the ones that know how each project works, understand their idiosyncrasy, and are the glue that helps them to be aligned and facilitate the cooperation.

To illustrate, let’s dive into a specific example. When we filtered on just the Kubernetes, Operator Framework, Rook, and Helm, we see more clearly the ‘connector’ participants between the projects. 

Figure #2: Kubernetes in the middle, Rook at the right side, Operator Framework at the top, and HELM at the left side of the chart. 

There appears to be a healthy amount of cross-community collaboration and communication between Operator Framework and Kubernetes (a), Helm and Kubernetes (b), and also signals that there’s an opportunity to encourage more collaboration between Helm and Operator Framework (c). 

Nurturing and Onboarding New Participants

With an average of 350 first time committers each month joining into the CNCF ecosystem. We can watch for their points of entry to see which projects they have engaged previously (if any) with, which ‘connectors’ that might help be able to help them get better engaged and help us ensure that they are plugged into the whole ecosystem. 

Identifying Emerging Projects

An additional insight we gain from this network analysis is the ability to ‘see’ movement over time as participants migrate from one project to the next. Incorporating a time filter to the visualizations can often give us the ability to predict the emergence of new projects and initiatives within and outside of the CNCF ecosystem, often a signal to get ready for a new submission to the CNCF Technical Oversight Committee  for Sandbox status

How to Connect

If you’d like to learn more about applying network analysis to community development, check out this recent presentation “Data Driven Approach to Community Development”  with OpenShift’s Diane Mueller and Bitergia’s Daniel Izquierdo

If you are involved in any of the CNCF projects or initiatives, there’s a great new CNCF Contributor Strategy SIG responsible for contributor experience, sustainability, governance, and openness guidance to help CNCF community groups and projects with their own contributor strategies for a healthy project.

If you are just looking to get started contributing to any of the CNCF projects, check out the CNCF’s Guide to Contributing to the CNCF ecosystem

If you’d like to learn more about Red Hat OpenShift’s Open Source Community Initiatives, please visit OpenShift Commons.




4 Tips for Maximizing Your Virtual KubeCon Experience

By | Blog

Member Post

KubeCon + CloudNativeCon sponsor guest post from Amanda Katona, Cloud Native Community Engagement Director at VMware

We are fast approaching the first ever virtual KubeCon. What started a few years ago with a handful of people in a hotel conference room evolved into thousands crowding convention center hallways. And now, it’s a party of one as we all have our own virtual experience.

At this point we have all attended multiple virtual conferences so far this year. With hundreds of hours of programmed content, it risks becoming an overwhelming (and occasionally numbing) experience. So, we wanted to share some ideas for how you can get the most value from the time you invest in KubeCon this year.

Build your agenda

It’s worth stating the obvious … don’t miss that all-important session (if it’s your first KubeCon, you might start here). Explore the KubeCon agenda and pinpoint the top few sessions you must attend. Create invites on your calendar and protect the time. When attending remote, it’s easy to let other responsibilities infringe upon your session time, so be prepared to defend it from meetings and fires that can smolder for a few days (if you were in Amsterdam, they’d have to wait).

Prepare questions for your top 5 sessions

Dig into the KubeCon agenda and pinpoint the 5 sessions that you find most compelling. Rather than come with a clean slate, reflect on what resonated and write down questions you want to ask the presenter. Either ask them in the session, or follow-up directly with speakers to request more resources or recommendations of peers with whom you should also connect. And for additional perspectives, bring your questions to the CNCF project maintainers at the Project Pavilion.

Take incredible notes

Open up an Evernote file or a moleskin notepad and scratch out everything that makes you think. And don’t stop here, the best note-takers add their own color commentary and opinions. That additional context adds a ton of value when it’s time to circle back through those notes days, weeks or months later and draw insight.

Get your swag on

The expo floor at a physical conference is like Halloween for adults—tech or treat. While you won’t be collecting any new swag this year, it’s a chance to show off your favorite swag from yesteryear. Pull out a vintage KubeCon 2017 t-shirt, don your branded socks and guzzle from your most envy-inspiring water bottle.

Be extra social and extra positive

Since we can’t be there in person to trade stories, we’ll need to be extra diligent about using Slack to engage. Find the Slack channels for your favored KubeCon tracks and hang out there to trade ideas with peers. Highlight standout ideas, important projects and opportunities to contribute. Most importantly, use social to spread positivity and celebrate the incredible effort that goes into nurturing this vibrant community year-round.

Eat french fries with mayonnaise

Cause hey, if we were in Amsterdam, that’s what we SHOULD be doing. You can take the KubeCon out of Amsterdam, but you can’t wrestle that last french fry from me. Put a batch in the oven and bring a little bit of the experience to your home office or kitchen table.

This isn’t what we expected. It’s a remarkably challenging time. And yet, we have little doubt this community will find ways to make the best of it. See you at KubeCon soon!


Kubernetes RBAC 101: Authentication

By | Blog

Member Blog Post

Guest post originally published on the Kublr blog by Oleg Chunikhin

Leveraging Client Certificates and Bearer Tokens to Authenticate in Kubernetes

In part one of this series on Kubernetes RBAC, we introduced authentication and authorization methods. In this article, we’ll dive a little deeper into authentication — a prerequisite for RBAC.

As we saw, there are a few authentication methods including client certificates, bearer tokens, HTTP basic auth, auth proxy, and impersonation. Because HTTP basic auth and statically configured bearer tokens are considered insecure, we won’t cover them here. Instead, we’ll focus on the authentication mechanisms that are viable options for production deployments.

Client certificates

When authenticating through client certificates, the client must first obtain a valid x509 client certificate which the Kubernetes API server will accept as authentication. This usually means that the client certificate must be signed by the cluster CA certificate.

Externally Signed Certificates

The client certificate can be signed by the Kubernetes API server itself, or externally by an administrator or an enterprise PKI. Let’s first look how the certificate is signed externally, outside the Kubernetes API server.

Kubernetes RBAC
Authentication: X509 Client Cert, PKI

  1. The client (user) generates a CSR (certificate signing request) using a personal private key
  2. The client (user) sends the CSR to the signing authority (an administrator or an enterprise PKI)
  3. The signing authority signs a client certificate based on the CSR and the Kubernetes API server CA private key
  4. The signing authority sends the signed certificate to the client
  5. The client can now use the client certificate with the private key to authenticate the API server requests

There is a drawback, however. The server CA private key will be exposed to an external system or administrator. While that may be acceptable with an enterprise PKI, it likely isn’t with manual certificate signatures.

Here is a sequence of signing certificate commands:

User: generate user private key (if not exist):

openssl genrsa -out user1.key 2048

User: generate user CSR:

openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=group1/O=group2"

Admin: sign user client cert:

openssl x509 -req -in user1.csr -CA cluster-ca.crt -CAkey cluster-ca.key \
    -set_serial 101 -extensions client -days 365 -outform PEM -out user1.crt

User: use with kubectl via options or kubeconfig:

kubectl --client-key=user1.key --client-certificate=user1.crt get nodes

kubectl config set-credentials user1 --client-key user1.key --client-certificate user1.crt --embed-certs
kubectl config set-context user1 --cluster demo-rbac --user user1
kubectl --context=user1 get nodes

kubectl config use-context user1
kubectl config get-contexts
kubectl get nodes

Internally Signed Certificates

Alternatively, you can use client certificate authentication directly from the cluster. As a client, you can create certificate signature requests. In this case, the system administrator or external system does not sign it. Instead, it sends it to the Kubernetes cluster which will sign the certificate and return it to the administrator who can now extract the signed certificate from the Kubernetes API and send it back to the client. This is done with a special object in the Kubernetes API called CertificateSigningRequest.

Kubernetes RBAC 2

Authentication: X509 Client Cert, Kubernetes CSR

Here is a sequence of commands:

User: generate user privat key (if not exist):

openssl genrsa -out user2.key 2048

User: generate user CSR:

openssl req -new -key user2.key -out user2.csr -subj "/CN=user2/O=group1/O=group2"

Admin: use Kubernetes API server to sign the CSR:

kubectl apply -f - <<EOF
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
  name: user2
  request: $(cat user2.csr | base64 | tr -d '\n')
  usages: ['digital signature', 'key encipherment',
    'client auth']

Admin (approver): approve or deny the CSR in the Kubernetes API:

kubectl certificate approve user2
kubectl certificate deny user2

Admin: extract the approved and signed certificate from the Kubernetes API:

kubectl get csr user2 -o jsonpath='{.status.certificate}' | \
base64 --decode > user2.crt

User: use with kubectl via options or kubeconfig:

kubectl --client-key=user2.key --client-certificate=user2.crt get nodes

kubectl config set-credentials user2 --client-key user2.key --client-certificate user2.crt --embed-certs
kubectl config set-context user2 --cluster demo-rbac --user user2

Bearer Tokens

Service Account

Instead of client certificates, you can also use bearer tokens to authenticate subjects in Kubernetes. The easiest way to get a token is by creating a service account in the Kubernetes API. The Kubernetes server will then automatically issue a token associated with the service account, and anyone using that token will be identified as using this service account to access the cluster.

RBAC Kubernetes 3

Authentication: Service Account

Here is a sequence of commands you can use to create a service account, get a token from it and use that token to access Kubernetes API:

Create service account:

kubectl create serviceaccount sa1

Get service account token:

kubectl get -o yaml sa sa1
SA_SECRET="$(kubectl get sa sa1 -o jsonpath='{.secrets[0].name}')"

kubectl get -o yaml secret "${SA_SECRET}"
SA_TOKEN="$(kubectl get secret "${SA_SECRET}" -o jsonpath='{.data.token}' | base64 -d)"

Send request:

kubectl "--token=${SA_TOKEN}" get nodes

kubectl config set-credentials sa1 "--token=${SA_TOKEN}"
kubectl config set-context sa1 --cluster demo-rbac --user sa1

Side note:

Please note that generally, when working with kubectl, you won’t specify secrets and credentials via command line instead you will use a kubectl configuration file. You can modify that configuration file using kubectl commands. This allows you to add the token into your kube config file as an additional set of credentials and use it via a new context in the kubeconfig file. Learn more about using kubeconfig files to organize access to different clusters with multiple sets of credentials in the Kubernetes documentation on this subject.

Using a kubeconfig file allows you to run kubectl without specifying any sensitive information in the command line while relying on the current context set within that config file.

Note that you can always operate with your config file using various command-line options.

OIDC Token

Alternatively, you can leverage an external identity provider like OIDC to authenticate through a token. At Kublr, we use Keycloak. We love this identity provider as it’s a powerful, scalable open source tool, supporting all modern standards like SAML, OIDC, XACML, etc. It also integrates with most identity management systems. Kublr uses it by default as a user management system for Kublr and managed Kubernetes clusters, but it can also serve as an identity broker when integrated with enterprise identity management tools, or even used as an identity manager for user applications via its powerful “realms.” Realms are a completely separate domain for users, groups, authentication, federation, etc.

How does authentication with OIDC work?

First, your Kubernetes API server must be configured to talk to an OIDC endpoint / OIDC provider. This is done via Kubernetes API server configuration parameters. The following snippet shows additions in the Kublr cluster specification necessary to setup the Kubernetes API server:

          oidc_client_id: '--oidc-client-id=kubernetes'
          oidc_groups_claim: '--oidc-groups-claim=user_groups'
          oidc_issuer_url: '--oidc-issuer-url=https://***'
          oidc_username_claim: '--oidc-username-claim=preferred_username'

When a client connects to a Kubernetes API, it talks to an identity provider using one of the flows defined in OIDC protocol to get an authentication access and refresh token. The identity provider sends the tokens back for the client to authenticate with the Kubernetes API.

The Kubernetes API server talks directly with the OIDC identity provider via OIDC API to verify if the client provided token is valid. The token provides all information needed for the Kubernetes API server to identify the client. The client, on the other hand, can also refresh that token using a “refresh token.”

Kubernetes RBAC

Authentication: OIDC

Let’s see how this looks in a command-line world. We will use cURL to talk to the identity provider and kubectl to talk to the Kubernetes server. Although in real-life scenarios in most cases this will be hidden under the hood of the framework or client library of your choice.

Login request from the client with visualization of the response:

curl \
  -d "grant_type=password" \
  -d "scope=openid" \
  -d "client_id=kubernetes" \
  -d "client_secret=${CLIENT_SECRET}" \
  -d "username=da-admin" \
  -d "password=${USER_PASSWORD}" \
  https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | jq .

Login – the same request but with response tokens saved in environment variables for use in the followup commands:

eval "$(curl -d "grant_type=refresh_token" -d "client_id=kubernetes" \
  -d "client_secret=${CLIENT_SECRET}" -d "refresh_token=${REFRESH_TOKEN}" \
  https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | \
  jq -r '"REFRESH_TOKEN="+.refresh_token,"TOKEN="+.access_token,"ID_TOKEN="+.id_token')" ; \
  echo ; echo "TOKEN=${TOKEN}" ; echo ; echo "ID_TOKEN=${ID_TOKEN}" ; echo ; \

Refresh token request with visualized response:

curl \
  -d "grant_type=refresh_token" \
  -d "client_id=kubernetes" \
  -d "client_secret=${CLIENT_SECRET}" \
  -d "refresh_token=${REFRESH_TOKEN}" \
  https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | jq -r .

Refresh – the same request with response tokens saved in the environment variables:

eval "$(curl -d "grant_type=refresh_token" -d "client_id=kubernetes" \
-d "client_secret=${CLIENT_SECRET}" -d "refresh_token=${REFRESH_TOKEN}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | \
jq -r '"REFRESH_TOKEN="+.refresh_token,"TOKEN="+.access_token,"ID_TOKEN="+.id_token')" ; \
echo ; echo "TOKEN=${TOKEN}" ; echo ; echo "ID_TOKEN=${ID_TOKEN}" ; echo ; \

Token introspection request:

curl \
  --user "kubernetes:${CLIENT_SECRET}" \
  -d "token=${TOKEN}" \
  https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token/introspect | jq .

kubectl kubeconfig configuration and request:

kubectl config set-credentials da-admin \
   "--auth-provider=oidc" \
   "--auth-provider-arg=idp-issuer-url=https://kcp.kublr-demo.com/auth/realms/demo-app" \
   "--auth-provider-arg=client-id=kubernetes" \
   "--auth-provider-arg=client-secret=${CLIENT_SECRET}" \
   "--auth-provider-arg=refresh-token=${REFRESH_TOKEN}" \

kubectl config set-context da-admin --cluster=demo-rbac --user=da-admin

kubectl --context=da-admin get nodes

Access tokens are usually short-lived, while the refresh tokens have a longer shelf life. You can refresh tokens through the command line sending the ID and refresh token to the identity provider, providing you a set of refreshed tokens.

You can also introspect the token with an identity provider endpoint. That’s essentially an API the Kubernetes API server can use to check who is sending a specific request.

As mentioned above, there are two more ways to provide access to a Kubernetes cluster. One is using an auth proxy, mainly used by vendors to set up different Kubernetes architectures. It assumes that you start a proxy server, which is responsible for authenticating user requests and forwarding them to the Kubernetes API. That proxy can authenticate users and clients anyway it likes and will add user identifications into the request headers for requests that are sent to the Kubernetes API.

Authentication proxy

Authentication: Authenticating Proxy

This allows the Kubernetes API to know who they work with. Kublr, for example, uses this authentication method to proxy dashboard requests, web console requests, and provide a proxy Kubernetes API endpoint.

Lastly, there is authentication through impersonation. If you already have certain credentials providing access to the Kubernetes API, those credentials can be used to “impersonate” users through authorization rules. This will allow the user to send impersonation headers so the Kubernetes API will switch your authentication context to that impersonated user.

authentication impersonation

Authentication: Impersonation

For regular clients and for production purposes, you only really have two options: client certificates or bearer tokens.

Roll up your sleeves

If you want to get your hands dirty, there are some tools you can use to analyze and debug apps connected to a Kubernetes API. cURL is great for experiments with REST APIs. Then, of course, there is kubectl — the Kubernetes CLI. jq command-line JSON processor helps visualize and process JSON data. JSON and YAML, as you may know, are commonly used file formats for Kubernetes and the Kubernetes API.


Here is an example of using cURL in linux to call Kubernetes API:

curl -k -v -XGET -H 'Authorization: Bearer ***' \
  '' | jq -C . | less -R
  • “-k” switch disables checking the server certificate
  • “-v” enables verbose output
  • “-H ‘Authorization: …’ ” adds a token authorization header
  • Sending curl output through “jq -C” command formats and colorizes server JSON output
  • “less -R” allows you to scroll up and down the output


Examples of using kubectl:

kubectl --kubeconfig=kc.yaml get nodes

export KUBECONFIG="$(pwd)/config.yaml"
kubectl get nodes

kubectl get nodes --v=9
  • --kubeconfig” option allows you to specify the location of the kubeconfig file containing information and credentials necessary to locate and authenticate with a Kubernetes API server.
  •  “KUBECONFIG” environment variable can be used to specify a default kubeconfig file location
  • If neither “--kubeconfig” option nor “KUBECONFIG” environment variable are specified, kubectl will look for a kubeconfig file at a default location “$HOME/.kube/config”

Example of a kubeconfig file:

apiVersion: v1
kind: Config
- name: demo-rbac
    certificate-authority-data: ***
- name: demo-rbac-admin-token
    token: ***
- name: demo-rbac
    cluster: demo-rbac
    user: demo-rbac-admin-token
current-context: demo-rbac


Authentication in Kubernetes can be handled in different ways. For production-grade deployments you have some options. You can use client certificates that can be either signed externally or by Kubernetes through the Kubernetes API. Alternatively, you can use a bearer token in Kubernetes by creating a service account or leverage an external identity provider like OIDC. These are all valid approaches. Which route you’ll go will be ultimately determined by your system and application architecture and requirements.

If you’d like to experiment with RBAC, download Kublr and play around with its RBAC feature. The intuitive UI helps speed up the steep learning curve when dealing with complexities of Kubernetes deployment and RBAC YAML files.

Jaeger Project Journey Report: A 917% increase in companies contributing code

By | Blog

Project Post

Today we are excited to release our next project journey report for Jaeger. This is the sixth such report we have compiled for one of our graduated projects. It assesses the state of the Jaeger project and how CNCF has impacted its progress and growth.

Jaeger is an open source, end-to-end distributed tracing platform built to help companies of all sizes monitor and troubleshoot their cloud native architectures. Contributors to Jaeger include many of the world’s largest tech companies, such as Uber, Red Hat, Ryanair, IBM, and Ticketmaster as well as fast-growing mid-size companies like Cloudbees. 

The report highlights the growth of Jaeger since the project joined CNCF on September 13, 2017. Between then and now:

  • The total number of companies contributing code has increased by 917% from 29 to 295. 
  • Jaeger has enjoyed a 603% expansion of individual contributors adding 1,682.
  • the number of authors and companies committing documentation to Jaeger has grown by 496% and 415%, respectively. 

Since joining CNCF Jaeger has added:

  • 1,753 contributors
  • 4,310 code commits
  • 3.2K pull requests
  • 39.3K contributions
  • 302 contributing companies

Be sure to read the full report for more impressive growth stats!

Next month, Jaeger will celebrate its fifth birthday. It has been fantastic to be a part of the project’s journey for three of those years, and we can’t wait to see the milestones to come!


1 2 53