Introduction

When building RESTful APIs with ASP.NET Core, you'll sooner or later find yourself in the place where you need to add versioning to your API. Mostly because you need to introduce breaking changes but need to keep backwards compatibility.

This is where API versioning comes into play, and allows you to evolve your application without breaking existing clients. As you add new features or change data contracts, older versions of the API can remain stable and compatible for consumers who depend on them.

This allows you to introduce improvements safely, manage deprecations in a controlled way, and maintain backward compatibility across mobile apps, integrations, or partner systems.

Let's have a look at the different types of API versioning and compare them to each other. Later, we'll get to see some code.

Types of API versioning

Versioning by URL segment

This is by far the most popular way to version a public API, where the version itself is part of the URL, e.g.

https://api.kloudshift.net/v1/products
https://api.kloudshift.net/v2/customers

Advantages

This approach clearly and explicitly communicates which version a client is using. Also, requiring an explicit service version helps ensure existing clients don't break.

Disadvantage

However, with this way of versioning, it's not possible to select a default API version, when clients don't specify one explicitly. To enable such scenarios, double route registration would be required providing multiple routes for the same controller action, that can clutter your code.

Also, clients must change URLs, when upgrading to a new API version, which increases maintenance friction.

Further, some REST purists might argue, that this approach breaks resource identity. In pure REST, a resource's URI should be stable - adding /v2 means a "new" resource, even if it's semantically the same entity.

When to use

This type of versioning is best suited if you're building a public API with multiple long-lived versions and you want maximum visibility and simplicity for clients.

You should avoid it if you are a REST purist and need strict semantics or prefer transparent evolution. If the later is the case, consider header or media-type versioning...

Versioning by header

When versioning an API by header, clients need to specifiy which version of the API they are talking to by adding a custom header.

curl https://api.kloudshift.net/products -H "X-Api-Version: 1" ...

Advantage

With this approach, the URLs and resource paths remain clean and free of version strings. It aligns well with REST principles in that the URL represents the resource not its version. This also implies that API versioning is decoupled from routing.

Also I like the fact, that default API versions can be controlled backend-side and independelty from the client.

It further allows clients to switch versions withoug having to change URLs - just the header value. This can be beneficial, e.g. when building an SDK.

Disadvantage

However on the downside, I think this approach is less discoverable to clients. Which header should I set? Which versions are supported by this endpoint?

Some APIs just return a 400 Bad Request, without letting you know that a version header was missing in the request. Don't be like that, this is enoying! Luckily Asp.Versioning.Http has some revelation to that, we'll get to that later.

When to use

You should use header-based API versioning when you want to keep your URLs clean and version agnostic, and when your clients (e.g., mobile apps, backends, SDKs) can easily control their requests/headers.

However, when you rely on caching (CDNs, proxies), you should better choose URL based versioning, since caches often key by URL, not headers. Further, when your API is accessed primarily from browsers or forms you are better of with versioning by URL path segment or query strings...

Versioning by query string

When versioning an API by query string, you include the version as a query parameter in the URL, for example:

https://api.kloudshift.net/products?version=1

Advantages

For clients, this approach is very simple to use and understand, it doesn't require modifying headers or complex request formats.

Disadvantages

Just like the URL path segment versioning approach, REST purist might argue it doesn't follow strict REST semantics. The resource should be identified by the URL - the version is more of a representational concern. Including it as a query parameter mixes concerns.

When to use

Query-based versioning works well for internal APIs, low-traffic services, or early-stage projects where simplicity and ease of testing outweigh strict REST compliance or caching concerns.

Versioning by media type

When using API versioning by media type, a client needs to set the requested version in the Accept header that is used by the HTTP content negotiation mechanism.

curl https://api.kloudshift.net/api/products -H "Accept: application/json; charset=utf-8; version=1.3-rc" ...

Advantages

Just like the header-based versioning approach, this comes with the benefit of clean and stable URLs. It keeps the URL free of version information and aligns well with REST principles.

Although being the most complex approach, it also provides the best flexibility, in that it allows versioning per representation, not per endpoint.

Disadvantages

This approach comes with the same disadvantages as the header-based versioning, in that it's not obvious for a client how it needs to select a version and which are available (the Asp.Versioning.Http NuGet package helps us with in this aspect, more later).

Also, it can be seen as some overhead to the client, since the client needs to manage headers carefully making it harder for simple integrations or frontend apps that expect straightforward URLs.

When to use

You should consider media-type versioning if you're building a highly RESTful enterprise-level, or hypermedia-driven API, where representations evolve independently of endpoints. Otherwise, prefer URL or header-based versioning for ease of maintenance.

Versioning with ASP.NET Core

Getting started

To add versioning to your project you need to add the following packages. Since this blog post deals with Minimal APIs only, its sufficient to install Asp.Versioning.Http.

Then add these calls to your Program.cs, where AddProblemDetails adds services required for creation of ProblemDetails for failed requests and AddApiVersioning adds the versioning capabilities.

builder.Services.AddProblemDetails();
builder.Services.AddApiVersioning();

Versioning by URL segment

To enable versioning by URL segment I initialize the ApiVersionReader with an instance of UrlSegmentApiVersionReader.

Also, I create one ApiVersionSet per endpoint, that allows defining which versions are supported. Finally, a route template is required that is defined with /api/v{version:apiVersion}/products and map each endpoint to a specific version.

public static class Program
{
    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);

        builder.Services.AddProblemDetails();
        builder.Services.AddApiVersioning(o =>
        {
            o.ApiVersionReader = new UrlSegmentApiVersionReader();
        });

        var app = builder.Build();

        var products = app.NewApiVersionSet()
            .HasApiVersion(new(1))
            .HasApiVersion(new(2))
            .Build();

        app.MapPost("/api/v{version:apiVersion}/products", (HttpContext ctx, ProductRequest req) =>
            {
                return TypedResults.Ok(new
                {
                    version = ctx.GetRequestedApiVersion()?.ToString(),
                    request = req
                });
            })
            .WithApiVersionSet(products)
            .MapToApiVersion(1);

        app.MapPost("/api/v{version:apiVersion}/products", (HttpContext ctx, ProductRequestV2 req) =>
            {
                return TypedResults.Ok(new
                {
                    version = ctx.GetRequestedApiVersion()?.ToString(),
                    request = req
                });
            })
            .WithApiVersionSet(products)
            .MapToApiVersion(2);

        app.Run();
    }
}

Example request

https://api.kloudshift.net/v2/products

Versioning by header

To enable versioning by header, we initialize the ApiVersionReader with an instance of HeaderApiVersionReader and pass in the expected header name.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
});

Example request

https://api.kloudshift.net/products -H "X-Api-Version: 2"

Versioning by query string

To enable versioning by query string, we use the QueryStringApiVersionReader type and optionally pass a name for the parameter name to use. The default is api-version.

builder.Services.AddApiVersioning(o =>
{
    // defaults to "api-version"
    o.ApiVersionReader = new QueryStringApiVersionReader("version");
});

Example request

https://api.kloudshift.net/products?version=2

Versioning by media-type

To enable versioning by media type we use the MediaTypeApiVersionReader and optionally pass a name for the parameter, which defaults to v.

builder.Services.AddApiVersioning(o =>
{
    // paramter name defaults to "v"
    o.ApiVersionReader = new MediaTypeApiVersionReader();
});

Example request

https://api.kloudshift.net/products -H "Accept: application/json; charset=utf-8; v=1 "

Version discovery

Most likely you want to communicate to a client which versions each endpoint supports. This can be achieved by using the ReportApiVersions option, which adds the api-supported-versions header to the response.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
    o.ReportApiVersions = true;
});

As you can see in the example request/response below, the client requests version 2 by setting the header x-api-version: 2. The client response then contains the api-supported-versions header indicating it supports both version 1, and 2.

POST http://api.kloudshift.net/products HTTP/1.1
X-Api-Version: 2

{
  "Name": "Backend development",
  "Price": 42.4,
  "Description": "Some description"
}
 
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 11 Nov 2025 14:30:39 GMT
Server: Kestrel
Transfer-Encoding: chunked
api-supported-versions: 1, 2

This option can also be set on the ApiVersionSet. In the example, only the products endpoint will advertise available versions.

var products = app.NewApiVersionSet()
    .HasApiVersion(new(1))
    .HasApiVersion(new(2))
    .ReportApiVersions()
    .Build();

var customers = app.NewApiVersionSet()
    .HasApiVersion(new(1))
    .HasApiVersion(new(2))
    .HasApiVersion(new(3))
    .Build();

app.MapPost("/api/products", ...)
   .WithApiVersionSet(products)
   .MapToApiVersion(2);

app.MapPost("/api/customers", ...)
   .WithApiVersionSet(customers)
   .MapToApiVersion(3);

Handling requests without a specified API version

The last feature, that I'd like to highlight is the ApiVersionSelector option. This setting defines the behavior of how an API version is selected by the backend, when a client has not requested an explicit API version and you don't want such calls to fail with a 400 Bad Request response.

There are four types of ApiVersionSelectors, these are:

DefaultApiVersionSelector

This selector always selects the configured DefaultApiVersion regardless of the available API version available.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
    
    o.DefaultApiVersion = new ApiVersion(2);
    o.AssumeDefaultVersionWhenUnspecified = true;
    o.ApiVersionSelector = new DefaultApiVersionSelector(o);
});

ConstantApiVersionSelector

Always selects the defined API version.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
    
    o.AssumeDefaultVersionWhenUnspecified = true;
    o.ApiVersionSelector = new ConstantApiVersionSelector(o);
});

CurrentImplementationApiVersionSelector

Selects the maximum API version available which doesn't have a version status. For example, if the version 1, 2 and 3-alpha are available, then 2 will be selected.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
    
    o.AssumeDefaultVersionWhenUnspecified = true;
    o.ApiVersionSelector = new CurrentImplementationApiVersionSelector(o);
});

LowestImplementedApiVersionSelector

Selects the minimum API version available which does not have a version status. For example, if the version 0.9-beta, 1, 2 and 3-alpha are available, then 1 will be selected.

builder.Services.AddApiVersioning(o =>
{
    o.ApiVersionReader = new HeaderApiVersionReader("X-Api-Version");
    
    o.AssumeDefaultVersionWhenUnspecified = true;
    o.ApiVersionSelector = new LowestImplementedApiVersionSelector(o);
});

Conclusion

• There are four common approaches to version your API, each having it's advantage and disadvantage.
• Selecting the right versioning approach depends on the clients your writing your API for.
• I'd recommend implementing API versioning right from the beginning of your project. Sooner or later you'll need to implement breaking changes, but need to keep backwards compatibility.
• My personal favorite is the header-based versioning, since I like to keep endpoints stable. YMMV.

That's it for today. Happy hacking 😎

Further reading

GitHub - dotnet/aspnet-api-versioning: Provides a set of libraries which add service API versioning to ASP.NET Web API, OData with ASP.NET Web API, and ASP.NET Core.

Provides a set of libraries which add service API versioning to ASP.NET Web API, OData with ASP.NET Web API, and ASP.NET Core. - dotnet/aspnet-api-versioning

GitHubdotnet

Home

Provides a set of libraries which add service API versioning to ASP.NET Web API, OData with ASP.NET Web API, and ASP.NET Core. - dotnet/aspnet-api-versioning

GitHubdotnet

How are REST APIs versioned?

I am currently working on a REST API, and the question was raised, how are, and how should, REST APIs be versioned? Here are the results of my research. It seems that there are a number of people r…

LexicalscopeTim Wood

Introduction

For many small and medium-sized businesses, the cost of running workloads on hyperscalers can be too high, while data sovereignity and control remain non-negotiable priorities - this is especially true for the finance and health-care industry in Switzerland.

At the same time, operating a full Kubernetes cluster—even with a managed control plane—brings more overhead than most teams can justify. Not everybody is running a fully-fledged micro-service application that needs global scaling capabilities.

Integrating a PaaS platform like Azure App Service into your CICD toolchain often adds unncessary complexity and feels more suited for large enterprises than for SMB needs. It doesn't provide the unified Developer Experience (DX) you might be looking for.

If you've been nodding throughout these first paragraphes, and you are looking for a self-hostable alternative to services like Heroku, Vercel, Netliy, Azure App Service, etc. than this article is for you.

I'll compare CapRover, Coolify and Dokploy, three off-the-shelf Internal Developer Platforms, that make it simple for your team to deploy and manage applications and databases with simple effort.

I've deployed and tested these platforms on a VPS running Ubuntu 24.04 As of writing the latest versions where CapRover 1.14.0, Coolify v4.0.0-beta.431, and DokPloy v0.25.4. All solutions are based on Docker & Docker Swarm.

These are the aspects I had a closer look at

• Maturity & community support
• Installation, upgrading & maintenance
• Docker Compose & Docker Swarm support
• API & CLI
• Multi server setup & clustering
• Domain name management & automatic HTTPs support (ACME)
• User, Teams & Permission Management
• Git Push Deployments with GitHub
• Logging, Monitoring & Notifications
• Backup capabilities of platform configuration & volumes
• Multi environment support (PROD, STAGING, DEV, ...)
• Preview deployments

Let's start with an overview describing CapRover, Coolify and Dokploy.

A brief description of the projects

CapRover

CapRover claims to be an extremely easy to use app/database deployment & web server manager for your Node.js, Python, PHP, ASP.NET, Ruby, MySQL, MongoDB, Postgres, WordPress, ... applications.

It's blazingly fast and very robust as it uses Docker, nginx, LetsEncrypt and NetData under the hood behind its simple-to-use interface.

Coolify

The platform carries the claim of "self-hosting with superpowers". An open-source & self-hostable alternative to Vercel, Heroku, Netlify and Railway for easily deploying websites, databases, web applications and 280+ one-click services to your own server.

Dokploy

The web dsite states "Dokploy is a stable, easy-to-use deployment solution designed to simplify the application management process. Think of Dokploy as a free alternative self-hostable solution to platforms like Heroku, Vercel, and Netlify."

Maturity & community support

When deciding for a critical platform solution, we don't want to be left without community support.

So, to get an understanding of how active a GitHub project is, I usually glance at the last commit date, and compare other metrics, e.g. amount of sponsors, stars, repo creation timestamp (created_at) and the number of contributors.

Of course, this is only a high-level view on the activness of a project, but still provides some sense on how mature and allive a code base is.

Here are the numbers.

Installation, upgrading & maintenance

CapRover

Before installing CapRover, you need to manually set up Docker CE, which is a straightforward process. Next, you need to create a wildcard domain, e.g. *.mydomain.com, and install the CLI tool on your developer machine via npm. Once prepared, installation is as simple as running a single docker run command via SSH.

Since CapRover itself runs in a container, in-place upgrades are easy and can be triggered directly from the Web UI. According to the documentation, upgrades cause only a brief interruption of running apps.

CapRover also provides scheduled Docker cleanup tasks, helping keep disk usage under control.

Coolify

To install Coolify, start with a fresh VPS with at least 2 GB RAM and 2 cores, and run the installation script — Docker CE will be installed automatically. After the first login, a user account is created and a setup wizard guides you through the initial configuration.

Coolify supports automatic (scheduleable) and manual updating. A test upgrade from v4.0.0-beta.431 to v4.0.0-beta.432 when smoothly without any interruption.

Another nice feature is that it allows you to patch your servers straight from the Coolify UI (running e.g. apt update , currently experimental)

It also includes a scheduled Docker cleanup feature that is capable of:

• Removing stopped containers managed by Coolify
• Deleting unused images
• Clearing the build cache
• Removing old Coolify helper images
• Optionally removing stale volumes and networks

Dokploy

To set up Dokploy, your VPS should have at least 2 GB RAM and 30 GB disk space. Installation is very simple: just run the script provided in the documentation. It will automatically install Docker CE. After the first login, a user account is created.

Upgrades are handled via a short shell script, though the documentation does not clarify whether they cause downtime.

In addition to scheduled Docker cleanup tasks, Dokploy also supports running custom scripts on a cron-based schedule.

Docker Compose & Docker Swarm support

CapRover

Since CapRover uses a custom format for it's deployments (captain files/one-click templates), it provides only limited Docker Compose support. Only a subselection of Docker Compose parameters can be used.

Coolify

The platform fully supports native Docker Compose syntax and displays environment variables from the Compose file directly in the Web UI.

You can create Docker Compose based Resources either from a public or a private repositoriy. Coolify also allows copying and pasting Docker Compose definitions into a text field.

I liked that in all cases the Docker Compose file serves as the single source of truth, ensuring no unexpected side effects. At the time of writing I encountered a validation logic bug documented here.

[Bug]: Invalid docker-compose file. Undefined array key “volumes” · Issue #6208 · coollabsio/coolify

Error Message and Logs When clicking the “Validate” button I get this error. The docker compose file works, everything is fine and the container is running and accessible but I noticed that in the…

GitHubkevincam3

Coolify supports an experimental Docker Swarm mode: you register a Swarm manager (and optionally workers) with Coolify, allowing it to coordinate deployments across nodes. 

In this setup, your Swarm must use an external container registry so that all worker nodes can pull the images the manager builds.   

Dokploy

Integrates well with Docker Compose and Docker Stack. You can point to a public Git Repo holding a docker-compose.yaml or pull a definition from your private GitHub repo. Alternatively, you can input raw compose definitions into a text field. Feels mature and well integrated.

API & CLI

CapRover

CapRover does not directly expose an API, but it provides experimental API access through the CLI command caprover api. The JavaScript-based CLI allows you to:

• Perform actions to prepare CapRover on a server
• Deploy your app to a specific CapRover machine
• Call a generic API (experimental)

Coolify

It provides an API secured by a Bearer token, which can be generated from the UI and supports different permission levels such as root, write, deploy, read, and read:sensitive. However, it does not include a CLI.

Dokploy

Dokploy exposes a feature-rich API secured with JWT authentication, where API keys can be scoped by organizations. Rate limiting is supported, and a Swagger interface is available. It doesn't provide granular token or API permissions.

In addition Dokploy provides a JavaScript-based CLI-tool via npm, that allows you to create, deploy and manage applications, databases, environments and projects.

Multi server setup & clustering

CapRover

This platform supports clustering through Docker Swarm, though it also requires the use of an external container registry. Nodes can be joined to the swarm as either workers or managers via the UI, which involves generating SSH keys, but in my experience it was simpler to add nodes manually.

One limitation is that the documentation lags behind, making some steps less straightforward than they should be.

Coolify

Coolify allows deploying the same application to multiple servers. You can add a load balancer in front of them to enable HA scenarios (requires manual setup). This feature is currently marked as experimental.

You can also add a dedicated build server to offload the build process from the machine actually hosting the applications. This keeps the load separated, so it doesn't affect the applications performance. This feature requires a container registry where the build server can push his images to.

Dokploy

When installing Dokploy on a single VPS, the same server handles application builds, hosts the applications, and serves the management UI simultaneously. This doesn't scale well for larger setups, why Dokploy supports a multi server setup.

With the multi-server setup, you can separate the management UI from building & hosting. The integration is straight forward, and can be carried out mostly from the UI (besides adding the public SSH key). You won't need an additional container registry.

After successful setup, you can select a server when creating a new service.

I didn't find a way to further separate building from hosting with the multi-server feature.

But Dokploy also offers a clustering feature. The idea of using clusters is to allow each server to host a different application and, using Traefik along with the load balancer, redirect the traffic from the dokploy server to the servers you choose. The reverse proxy remains on the manager node.

It allows to deploy multiple replicas of an application and distribute them across worker nodes (Docker Stack and Applications only, doesn't work for Compose).

For this setup to work, a container registry is required, and ideally the Docker Swarm nodes should communicate over a private network with each other.

For high-availability, you can scale Traefik to multiple replicas and use an external load balancer for that (not tested, maybe for another article)

Domain name management & automatic HTTPs

CapRover

By default every new deployed application is reachable under the wildcard domain, that got created prior installation (e.g. my-app.subdomain.domain.tld). New domains can be bound and configured from within the UI. CapRover comes with builtin support for Let's Encrypt and supports HTTP to HTTPS redirection.

Coolify

The platform supports configuring wildcard domains, e.g. *.example.com. This allows you to use generated domain names for each application under that domain, e.g. my-app.example.com.

If you don't configure a wildcard domain, the domain auto-generation process will generate domain names under the service/domain sslip.io for quick access & testing. But of course you can also set a custom domain www.somethingelse.com

Coolify also supports automatic redirection of e.g. example.com to www.example.com and allows enforcing HTTPs

Dokploy

Dokploy provides two ways to add domains, free domains and domains you bought. Free domains are provided by traefik.me but are limited to HTTP only.

The UI also supports uploading your own x509 certificates, this can also be used to enable HTTPs for traefik.me . It further supports automatic SSL certificate provisioning via Let's Encrypt (ACME). You can also use other custom certificate providers

User, Teams & Permission Management

CapRover

It offers a Pro plan that includes two-factor authentication, but otherwise only supports single-user mode.

Coolify

Coolify supports creating multiple teams (though this feature did not work in my test with version v4.0.0-beta.431), inviting new users, and assigning them one of three roles: admin, owner, or member. Additionally, it offers two-factor authentication for logins.

Dokploy

It allows organizing resources by organizations and supports two-factor authentication. New members can be invited, though in the self-hosted version this requires manually sharing the invitation link, while in the cloud version it works directly. See this issue for further details.

Invitations are scoped to organizations, meaning users only gain access to resources within the org they are invited to, unless they receive separate invitations for additional orgs.

Currently, only one admin role is allowed per instance, but multiple permission levels are available to manage users effectively.

Git Push Deployment with GitHub

CapCover

It supports push-based deployments by configuring a webhook in your repository that triggers on a branch push. Once CapRover receives the webhook call, it pulls the code, builds it, and deploys it automatically, though the UI for this feature appears limited.

Coolify

Supports automatic deployments on commits and pull requests, working with both public and private repositories. For private repos, integration is possible via either a GitHub App or deploy keys. When using the GitHub App, enabling auto-deploy is as simple as ticking the Auto Deploy box in the resource configuration.

Dokploy

Automatic deployment can be set up using either webhooks or the Dokploy API, though this is limited to Applications and Docker Compose.

For GitHub, autodeploy works out of the box with no additional configuration, and once Git integration is complete, you only need to set the trigger type on the application provider.

Logging, Monitoring & Notifications

CapRover

It provides a monitoring dashboard and a server-side Nginx log analyzer, with metrics powered by Netdata (though only available on the leader node).

Logs can be viewed per app, but the functionality is quite limited. Notifications are handled through the Netdata module, supporting delivery via email, Slack, Telegram, and Pushbullet.

Coolify

Notifications can be triggered for deployments, backups, scheduled tasks, and server events such as cleanups or disk usage.

They can be sent via email, Discord, Telegram, Slack, or Pushover, and logs can also be drained to third-party applications like Axiom and New Relic.

Dokploy

It provides basic graphs for CPU, memory, and disk usage, along with notifications for events such as app deployments, build errors, database backups, Docker cleanup tasks, and Dokploy restarts.

Notifications can be sent through Slack, Telegram, Discord, email, Gotify, or ntfy, but external log drains are not supported.

Configuration Backup & Volume Backup

CapRover

Backup and restore functionality is still experimental. While it works for most resources, images require using a Docker registry and volumes need a custom solution—both approaches having their own pros and cons..

Coolify

Coolify offers two ways to back up the instance itself: automatic backups to S3 storage or manual backups triggered on demand. However, it does not provide an automatic or UI-integrated solution for backing up and restoring volumes, though the documentation offers clear guidance on how to handle this manually.

Dokploy

Dokploy supports configuration backups to an S3 destination, covering the entire file system and database, with the option to schedule them regularly.

It also provides integrated database backups and supports volume backups for applications and Docker Compose. Named Docker volumes can be selected from a list and backed up to S3, with the option to temporarily stop containers during the process to avoid file locks or corruption, making it a very convenient solution.

Multi environment support

CapRover

Doesn't support environments

Coolify

It supports three types of shared variables: team-based, project-based, and environment-based (such as prod, staging, or dev). However, environment-level variables were unclear in usage, and team-based variables did not work in testing.

Dokploy

You can configure environment variables at different scopes: project-wide variables that apply to all applications, environment-level variables limited to a specific environment, and application-scoped variables that affect only a single application.

Preview Deployments

CapRover

Doesn't support preview deployments

Coolify

It offers a great way to test applications before merging into the main branch by creating preview deployments that act like a staging environment.

Preview URLs can be templated with identifiers such as the pull request ID (e.g., PR 123 becomes 123.example.com). Automated preview deployments can also be enabled, making each new pull request instantly available at its own preview URL by simply ticking Preview Deployments under Configuration → Advanced.

Dokploy

Preview deployments only work with applications sourced from GitHub and linked via a GitHub App, and they should be used exclusively with private repositories to prevent external users from triggering builds and deployments.

You can limit the number of preview deployments per application, use either auto-generated or custom domains, and apply label filters to control which pull requests trigger a preview deployment.

Verdict

A key capability missing across all three platforms is robust observability integration.

CapRover

I found that CapRover lacks in documentation. I also found its UI quite basic and sometimes buggy. E.g. when I removed a worker node from the Swarm, the UI was not updating. Also the featureset is quite limited compared to the Coolify & Dokploy.

Besides that it still a respected and widely used solution according to high download metric of 100M+ on Docker Hub.

Coolify

I found the UX sometimes a bit difficult to understand intuitively, e.g. when looking for the HTTPs settings. Also when create new applications which are sourced via a private Git repositoriy, it would be nice to have search function since the list of repos can get long. Also having backup support for volumes would be aweseome to have.

Other than that, the platform is quite mature with a lot of features at hand. I liked the multi-server setup to horizontally scale the installation. Also the documentation is in a very good shape!

Dokploy

Although being the youngest of these three projects, Dokploy is my personal favorite.

I find it a feature-rich PaaS platform with a pretty good documentation. Installation is dead simple and it provides full Docker Compose support. The UX is well designed and I liked the support for multiple organizations allowing for clear separation between stacks. A highlight is the nicely integrated backup feature allowing for scheduled backups of volumes, databases and configuration.

Further reading

Welcome to Dokploy | Dokploy

Dokploy is a open source alternative to Heroku, Vercel, and Netlify.

Dokploy Docs

Coolify Docs

Self hosting with superpowers: An open-source & self-hostable Heroku / Netlify / Vercel alternative.

Get Started

Getting Started · CapRover

## Simple Setup

Introduction

In my previous article, I demonstrated how we can build a codeless REST API with Data API Builder and how the endpoints can be write-protected by introducing roles with the help of Azure AD.

Creating and securing a codeless REST API on Azure using Data API Builder

This article describes how we can build a codeless REST API using Data API Builder and host it securely on Azure Container Instances

Matthias' BlogMatthias Güntert

Unfortunately, the described architecture doesn't provide HTTPS out of the box, which makes its use insecure. This is especially true for any operations requiring an access token.

So in this article, I'll describe an architecture that will protect our codeless REST API with a reverse proxy providing automatic HTTPS to further reduce maintenance!

For this article, the REST API will build on the famous AdventureWorksLT data set, that we host on a slim Azure SQL database.

Then, we will use Caddy as a sidecar to the Data API Builder runtime, and host the container group on Azure Container Instances.

💡 Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go. Some benchmarks promise 4 times higher performance than Nginx.

Features & Components

• Caddy 2, acting as a reverse proxy and providing automatic HTTPS
• Data API Builder
• An Azure Container Instance Group
• Azure SQL Server & Database
• Azure Storage Account hosting configuration files
• Let's Encrypt and the ACME protocol

Without further ado, let's get started 🧪

Step by step...

🔎 In the sections below, readers of my previous article on the Data API Builder might realize some repetiting parts. I want my articles to be as easy as possible to follow, that's why I decided to list the required sub-steps again...

Azure SQL Database

🪛 First, we'll need an Azure SQL server to host our demo database

az group create `
  --location westeurope `
  --name rg-demo

az sql server create `
  --name sql-azureblue `
  --resource-group rg-demo `
  --admin-password "your-password" `
  --admin-user "sqladmin"

🪛 Next, let's set up the database and use the Adventure Works LT sample data.

 az sql db create `
   --name sqldb-adventureworks `
   --resource-group rg-demo `
   --server sql-azureblue `
   --backup-storage-redundancy Local `
   --edition Basic `
   --capacity 5 `
   --max-size 2GB `
   --sample-name AdventureWorksLT

🪛 Finally, we need to make sure, that all Azure Services are able to access our database.

az sql server firewall-rule create `
  --server sql-azureblue `
  --resource-group rg-demo `
  --name AllowAzureServices `
  --start-ip-address 0.0.0.0 ` 
  --end-ip-address 0.0.0.0

Don't worry because of the IP range. The command won't open up the server to the entire Internet. Instead, it ticks the checkbox saying Allow Azure service and resources to access this server.

Azure Storage Account & File Shares

For the purpose of this article, we'll need four file shares, which are

• dab-config
• proxy-caddyfile
• proxy-config
• proxy-data

The dab-config file share will host the dab-config.json file providing input to the Data API Builder runtime. The proxy-caddyfile file share will host the Caddyfile, which configures our reverse proxy, proxy-config will host the Caddy configuration directory and last but not least, proxy-data will persist the Caddy data directory.

🔎 From the Caddy docs: [...] The Caddy data directory stores TLS certificates, private keys, OCSP staples, and other necessary information to the data directory. It should not be purged without an understanding of the implications.

🪛 Okay, let's create the storage account named stdabtlsdemo and the beforementioned file shares.

# Create storage account 
az storage account create `
  --name stdabtlsdemo `
  --resource-group rg-demo `
  --location westeurope

# Store connection string 
$env:AZURE_STORAGE_CONNECTION_STRING = $(az storage account show-connection-string --name stdabtlsdemo --resource-group rg-demo --output tsv)

# Create file shares
az storage share create `
  --name dab-config `
  --account-name stdabtlsdemo
  
az storage share create `
  --name proxy-caddyfile `
  --account-name stdabtlsdemo

az storage share create `
  --name proxy-config `
  --account-name stdabtlsdemo
  
  az storage share create `
  --name proxy-data `
  --account-name stdabtlsdemo

Before we move on and create the Azure Container Instance Group, let's have a closer look at the configuration files.

💡 You can find all configuration files in my GitHub repository.

Data API Builder Runtime Configuration

Here is a basic runtime configuration, that exposes a single endpoint $baseUrl/api/product for public read access. The endpoint gets fed by data coming from the table SalesLT.Product.

Further, the connection string is injected by an environment variable called DATABASE_CONNECTION_STRING.

{
    "$schema": "https://dataapibuilder.azureedge.net/schemas/v0.5.35/dab.draft.schema.json",
    "data-source": {
        "database-type": "mssql",
        "options": {
            "set-session-context": false
        },
        "connection-string": "@env('DATABASE_CONNECTION_STRING')"
    },
    "runtime": {
        "rest": {
            "enabled": true,
            "path": "/api"
        },
        "graphql": {
            "allow-introspection": true,
            "enabled": true,
            "path": "/graphql"
        },
        "host": {
            "mode": "development",
            "cors": {
                "origins": [],
                "allow-credentials": false
            },
            "authentication": {
                "provider": "StaticWebApps"
            }
        }
    },
    "entities": {
        "product": {
            "source": "SalesLT.Product",
            "permissions": [
                {
                    "role": "anonymous",
                    "actions": [
                        "read"
                    ]
                }
            ]
        }
    }
}

🪛 Now is a good time to copy the file dab-config.json to the share called dab-config.

The Caddy runtime configuration

At a first glance, configuring Caddy as a reverse proxy seems straightforward. However, there are some implications worth mentioning.

dab-tls-demo-api.westeurope.azurecontainer.io {
	reverse_proxy http://localhost:5000
}

As we'll run Caddy as a sidecar to the DAB runtime, both containers need to communicate with each other. By default, DAB runs on 5000/TCP and doesn't provide SSL. This is why the reverse_proxy directive is prefixed with http.

Also, containers within an ACI group can only communicate via localhost with each other. This contrasts with the configuration you might be familiar with when creating docker-compose.yaml files. There, you can reference the containers by name, which is not possible with ACI groups.

Further, the Caddy runtime (read ACME client) needs to be reachable by the defined domain name (dab-tls-demo-api.westeurope.azurecontainer.io in my example), otherwise, Let's Encrypt won't be able to issue certificates.

🪛 Now copy the file Caddyfile to the share called proxy-cadyfile.

The ACI Group YAML configuration

The configuration defines two containers called reverse-proxy and data-api-builder, from which only Caddy gets exposed to the Internet by 80/TCP and 443/TCP. The instance binds to the hostname dab-tls-demo-api, which later will be reachable via dab-tls-demo-api.westeurope.azurecontainer.io.

Then, we mount the beforementioned Azure File Shares to the container and inject the database connection string as a secret environment variable into the data-api-builder container.

name: ci-adventureworks-tls-api
apiVersion: "2021-10-01"
location: westeurope
properties:
  containers:
    - name: reverse-proxy
      properties:
        image: caddy:2.6
        ports:
          - protocol: TCP
            port: 80
          - protocol: TCP
            port: 443
        resources:
          requests:
            memoryInGB: 1
            cpu: 1
          limits:
            memoryInGB: 1
            cpu: 1
        volumeMounts:
          - name: proxy-caddyfile
            mountPath: /etc/caddy
          - name: proxy-data
            mountPath: /data
          - name: proxy-config
            mountPath: /config

    - name: data-api-builder
      properties:
        image: mcr.microsoft.com/azure-databases/data-api-builder:0.5.35
        resources:
          requests:
            memoryInGB: 1
            cpu: 1
          limits:
            memoryInGB: 1
            cpu: 1
        volumeMounts:
          - name: dab-config
            mountPath: /dab-config
        environmentVariables:
          - name: DATABASE_CONNECTION_STRING
            secureValue: "<your-connection-string>"
          - name: ASPNETCORE_LOGGING__CONSOLE__DISABLECOLORS
            value: true
        command:
          - dotnet
          - Azure.DataApiBuilder.Service.dll
          - --ConfigFileName
          - /dab-config/dab-config.json

  ipAddress:
    ports:
      - protocol: TCP
        port: 80
      - protocol: TCP
        port: 443
    type: Public        
    dnsNameLabel: dab-tls-demo-api

  osType: Linux

  volumes:
    - name: proxy-caddyfile
      azureFile: 
        shareName: proxy-caddyfile
        storageAccountName: stdabtlsdemo 
        storageAccountKey: "<your-key>"
    - name: proxy-data
      azureFile: 
        shareName: proxy-data
        storageAccountName: stdabtlsdemo 
        storageAccountKey: "<your-key>"
    - name: proxy-config
      azureFile: 
        shareName: proxy-config
        storageAccountName: stdabtlsdemo 
        storageAccountKey: "<your-key>"
    - name: dab-config
      azureFile: 
        shareName: dab-config
        storageAccountName: stdabtlsdemo 
        storageAccountKey: "<your-key>"

🪛 Obviously, you need to replace the placeholders with your values. You can retrieve the storage account keys as follows...

az storage account keys list `
  --resource-group rg-demo `
  --account-name stdabtlsdemo `
  --query [0].value `
  --output tsv

... and the connection string.

az sql db show-connection-string `
  --client ado.net `
  --name sqldb-adventureworks `
  --server sql-azureblue `
  --output tsv 

🔎 Unfortunately, for now, we can't mount subfolders of a single Azure File Share into containers, and therefore need a dedicated file share for every configuration file 😞

✅ Checkpoint

By now, your File Shares should like as follows.

.
└── File shares/
    ├── dab-config/
    │   └── dab-config.json
    ├── proxy-caddyfile/
    │   └── Caddyfile
    ├── proxy-config/
    │   └── (empty)
    └── proxy-data /
        └── (empty)

Azure Container Instance Group

Now that you have replaced the values we can finally fire up the container group.

az container create `
  --resource-group rg-demo `
  --file ci-adventureworks-tls-api.yaml

Testing it out

Give the containers some time to start and then verify if TLS 1.3 is properly set up for our API. The easiest way is to use a browser, so go ahead and copy the URL https://dab-tls-demo-api.westeurope.azurecontainer.io/api/product/ProductID/680 to your favorite browser...

Success!!! 🚀🚀🚀As depicted in the screenshot, the certificate got issued by Let's Encrypt, and our data in transit is secured from prying eyes. 💪🏼

Considerations

• For production environments, you'd usually create a CNAME record, e.g. api.mydomain.io, which points to dab-tls-demo-api.westeurope.azurecontainer.io. If you do so, remember to use this CNAME record in your Caddyfile, and not the DNS label managed by ACI!
• Also, you'd usually build your own docker image, instead of mounting the Caddyfile from an Azure Storage Account.

Conclusion

With the help of the Data API Builder and Caddy, we have provisioned a TLS-secured, codeless REST API ready to serve our requirements!

Both key components, DAB and Caddy, have helped us to keep development time and maintenance low. 🚀

Again, that was fun! 😎 Stay tuned for more articles!

Further reading

Docker

Welcome - Caddy Documentation

Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go

Caddy Web Server

YAML reference for container group - Azure Container Instances

Reference for the YAML file supported by Azure Container Instances to configure a container group

Microsoft Learntomvcassidy

data-api-builder-article/dab-with-caddy-and-tls at main · matthiasguentert/data-api-builder-article

Contribute to matthiasguentert/data-api-builder-article development by creating an account on GitHub.

GitHubmatthiasguentert

Introduction

In production & enterprise-grade setups an AKS cluster gets usually configured to authenticate users against Azure Entra ID and perform authorziation decisions based on the Kubernetes RBAC model.

This makes sense since Entra ID is usually the central identity provider with on-premises Active Directory, while the Kubernetes API still manages authorization decisions.

But have you ever wondered how the Azure Entra ID integration works and why the additional helper binary called kubelogin is required?

In this post, we'll look at Kubernetes and its authentication mechanism and see how it's connected and integrated into Entra ID. Let's get started!

How Kubernetes authentication works without Entra ID

As you might know, Kubernetes has no objects representing normal user accounts, unlike it has for service accounts.

Instead, any HTTP request hitting the Kubernetes API presenting a valid certificate signed by the cluster's CA is considered authenticated.

In this case, the user's identity is derived from the common name field in the subject of the certificate.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1234
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = ca
        Validity
            Not Before: Jan 10 19:41:52 2024 GMT
            Not After : Jan 10 19:51:52 2026 GMT
        Subject: O = system:masters, CN = masterclient
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus: ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:...            
    Signature Algorithm: sha256WithRSAEncryption
         ...

After authentication, the Kubernetes RBAC sub-system takes care of authorization decisions and passes the request on to the admission controllers.

Running kubectl auth whoami, will return the following

ATTRIBUTE   VALUE
Username    masterclient
Groups      [system:masters system:authenticated]

The depicted user certificate from above has been taken and decoded from the kubeconfig file. It gets pulled and installed by Azure CLI, amongst a private key and the CA certificate, when you execute az aks get-credentials.

apiVersion: v1
clusters:
    - cluster:
        certificate-authority-data: <base64-encoded-ca-certificate>
        server: https://your-aks-cluster-dns-yd6y5kxt.hcp.switzerlandnorth.azmk8s.io:443
      name: your-aks-cluster
contexts:
    - context:
        cluster: your-aks-cluster
        user: clusterUser_your-resource-group_your-aks-cluster
      name: your-aks-cluster
current-context: your-aks-cluster
kind: Config
preferences: {}
users:
    - name: clusterUser_your-resource-group_your-aks-cluster
      user:
        client-certificate-data: <base64-encoded-user-certificate>
        client-key-data: <base64-encoded-user-key>
        token: <token>

The certificate-authority-data key holds the cluster certificate, while the client-certificate-data and client-key-data keys hold the user's certificate and corresponding private key.

Okay, this is all nice and swell 🧐, but how does Entra ID and kubelogin fit in here?

The thing with kubectl and kubelogin

Regarding authentication and authorization, OAuth 2.0 and OpenID Connect are the go-to protocols and defacto standards on the Internet.

However, these protocols are not natively understood by kubectl, which can only work with certificates and bearer tokens.

To be more precise, kubectl can attach a bearer token to its requests towards the Kubernetes API, but it can't fetch or refresh any bearer tokens.

⚠️ Please be aware that there are three projects on GitHub with the name kubelogin 🤯 This article refers to Microsoft implementation found here.

So this is why the kubelogin binary is required to execute any OAuth flows and pass the retrieved tokens on to kubectl. The following diagram gives a high-level overview of the process.

After executing az aks get-credentials on an Entra ID integrated cluster, your kubeconfig will carry a key called exec in a user object. This entry defines which client-go credential plugin to call and which arguments should be used.

apiVersion: v1
clusters: ...
contexts: ...
current-context: your-aks-cluster
kind: Config
preferences: {}
users:
    - name: clusterUser_your-resource-group_your-aks-cluster
      user:
        exec:
            apiVersion: client.authentication.k8s.io/v1beta1
            args:
                - get-token
                - --environment
                - AzurePublicCloud
                - --server-id
                - 6dae42f8-4368-4678-94ff-3960e28e3630
                - --client-id
                - 80faf920-1908-4b52-b5ef-a8e7bedfc67a
                - --tenant-id
                - <your-tenant-id>
                - --login
                - interactive
            command: kubelogin
            env: null
            installHint: ...
            provideClusterInfo: false

From the example above, the entire kubelogin , including its arguments, looks as follows.

kubelogin get-token \
  --environment AzurePublicCloud \
  --server-id 6dae42f8-4368-4678-94ff-3960e28e3630 \
  --client-id 80faf920-1908-4b52-b5ef-a8e7bedfc67a \
  --tenant-id <your-tenant-id> \
  --login interactive

These two Entra ID app registrations, client and server app, are managed by the AKS resource provider for you and configured when you execute:

az aks create -g <group> -n <cluster> --enable-aad --aad-admin-group-object-ids <id> [--aad-tenant-id <id>]

Okay, so kubelogin covers the client-side functionality of the authentication process. But how do things work on the Kubernetes API side, and what does the big picture look like?

Kubelogin, Entra ID & OpenID Connect

Before we get to the big picture, let's have a look the following diagram, that depicts the relation between the components and how they relate to each other in OAuth 2.0 terminology.

The user (or resource owner) delegates the rights to access it's identifying data to the client, kubectl, and kubelogin.

Further, the client must be registered with Entra ID (the authorization server), which authorizes the token requests.

Last but not least, AKS, the resource server, has a trust relation to Azure Entra ID. This is required so that AKS can validate the received bearer tokens against Entra ID.

The entire picture

Now that we better understand the components, we can merge the diagrams and add more details.

Let's walk through each of the steps.

Steps 1, 2

The user executes kubectl which in turn invokes kubelogin with the server-id, client-id and tenant-id.

Steps 3 - 7

The browser opens, and the user is asked to authenticate. The credentials are verified, and an authorization code is requested by communicating with the Entra IDs authorization endpoint, which is then passed back to kubelogin.

Steps 8, 9, 10

Kubelogin exchanges the received authorization code for a bearer token by communicating with the token endpoint. Then, the token returns to kubectl, which attaches it to the users' HTTP request to the Kubernetes API.

GET /api/v1/namespaces/test/pods
Authorization: Bearer eyJ0eXAiOiJKV7QiLCJhbGciOiJSUzI1N...

Step 11

At this point, the Kubernetes API must check that the token signature is valid.

Therefore, it discovers the JWK URI endpoint by using a well-known URL and fetches the public key published at the jwks_uri key.

https://sts.windows.net/<tenant-id>/.well-known/openid-configuration

Then, it decodes the token and validates if...

• ... the intended recipient match (aud audience claim)
• ... the token time window is valid (exp expiration time, nbf not before)
• ... the client ID matches (appid)

Step 12

After all authentication conditions are met, the user is considered authenticated, and the request is passed on to the kube-api's authorization decision...

Conclusion

Let's wrap up 📑

• Kubernetes has no user objects, and they can't be created by the API.
• The authentication mechanism uses OpenID Connect.
• kubectl does not implement any OAuth flows.
• A client-go exec plugin called kubelogin is required, which implements OAuth.
• There are three different kubelogin projects on GitHub; don't get confused.
• The client and server IDs are static and reused across multiple (all?) AKS deployments that are Entra ID integrated.
• The client and server app registrations in Entra ID are managed for you by Microsoft.

That's it for today. I hope you enjoyed reading this article. Always happy to receive feedback! Happy hacking 👨🏽‍💻🤓

Further reading

Enable managed identity authentication on Azure Kubernetes Service - Azure Kubernetes Service

Learn how to enable Microsoft Entra ID on Azure Kubernetes Service with kubelogin and authenticate Azure users with credentials or managed roles.

Microsoft LearnMGoedtel

GitHub - kubernetes/client-go: Go client for Kubernetes.

Go client for Kubernetes. Contribute to kubernetes/client-go development by creating an account on GitHub.

GitHubkubernetes

Introduction - Azure Kubelogin

A Kubernetes credential (exec) plugin implementing azure authentication

Azure Kubelogin

Authenticating

This page provides an overview of authentication. Users in Kubernetes All Kubernetes clusters have two categories of users: service accounts managed by Kubernetes, and normal users. It is assumed that a cluster-independent service manages normal users in the following ways: an administrator distribu…

Kubernetes

OpenID Connect (OIDC) on the Microsoft identity platform - Microsoft identity platform

Sign in Microsoft Entra users by using the Microsoft identity platform’s implementation of the OpenID Connect extension to OAuth 2.0.

Microsoft LearnOwenRichards1

Introduction

In this post, I will demonstrate how test containers can be leveraged for proper DAL integration testing of ASP.NET Core, EF Core, and Postgres.

I will outline why you will want to use it over other common integration testing scenarios and demonstrate how it can be used together with the WebApplicationFactory to fully run your ASP.NET Core application in memory and create a reusable fixture for your testbed.

Test Containers & DAL testing scenarios

If you are a Spring Boot/Java developer, you might have heard of a library called testcontainers. The Java library provides "... lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container."

For this post, I will use its C# counterpart, called .NET Testcontainers. This NuGet package follows the idea of its Java predecessor and provides throwaway Docker instances for testing purposes.

It is built on top of the .NET Docker remote API and comes with a couple of pre-configured configurations, e.g., Postgres, Microsoft SQL Server, MySQL, Redis, RabbitMQ, and a couple more.

GitHub - testcontainers/testcontainers-dotnet: A library to support tests with throwaway instances of Docker containers for all compatible .NET Standard versions.

A library to support tests with throwaway instances of Docker containers for all compatible .NET Standard versions. - GitHub - testcontainers/testcontainers-dotnet: A library to support tests with ...

GitHubtestcontainers

EF Core Testing Strategies & Test Containers

You can follow two paths when choosing an EF Core testing strategy. You either use a test double or run your test against a production database.

There are different kinds of test doubles you can choose from, which are:

• SQLite (in-memory mode)
• EF Core in-memory provider
• Mock/stub the DBContext and DBSet
• Introduce a repository layer between EF Core and your application code and mock or stub that layer.

These strategies have pros and cons, which I will not fully elaborate on here.

However, they all share one important drawback: The test doubles do not behave exactly like your production database. Let me name a few important points:

• The same LINQ query may return different results on different providers due to differences in case sensitivity
• Provider-specific methods cannot be tested
• Limited testing of referential integrity
• Limited raw SQL support

So, depending on the complexity of your application, these difficulties will sooner or later result in false-positive test results (functionality is broken, but the test passes) or will leave some functionality untested.

This inevitably leads to the point where you want to test against a production database. However, involving a production database also has its hurdles.

First, you must set up an RDBMS on your developer machine and a build server (and maintain it).

Second, since the database is a shared dependency on the testing code, special effort is required to manage test database instances and their states.

A shared dependency is a dependency that is shared between tests and provides means for those tests to affect each other’s outcome. - (Khorikov, 2020, p.28)

Enter test containers

Using ephemeral containers relieves you of both of the aforementioned burdens.

First, there is no need for a complex RDBMS setup, and second, your tests will always start with a known state since each test can use a fresh container.

Using test containers instead of a fully-fledged RDBMS installation makes the database an out-of-process dependency since tests no longer work with the same instance.

An out-of-process dependency is a dependency that runs outside the application’s execution process; it’s a proxy to data that is not yet in the memory. - (Khorikov, 2020, p.28)

Last, your integration tests benefit from the full feature set of the involved RDBMS.

This is what a basic test setup looks like. It uses xUnits IAsyncLifetime interface to ensure the container is ready to serve requests before the test runs.

public sealed class PostgreSqlTest : IAsyncLifetime
{
    private readonly PostgreSqlContainer _postgreSqlContainer = new PostgreSqlBuilder()
        .WithImage("postgres:14.7")
        .WithDatabase("db")
        .WithUsername("postgres")
        .WithPassword("postgres")
        .WithCleanUp(true)
        .Build(); 
    
    [Fact]
    public void ExecuteCommand()
    {
        using var connection = new NpgsqlConnection(_postgreSqlContainer.GetConnectionString());
        using var command = new NpgsqlCommand();
        connection.Open();
        command.Connection = connection;
        command.CommandText = "SELECT 1";
        command.ExecuteReader();
    }

    public Task InitializeAsync()
    {
        return _postgreSqlContainer.StartAsync();
    }

    public Task DisposeAsync()
    {
        return _postgreSqlContainer.DisposeAsync().AsTask();
    }
}

You'll need to add the testcontainers and the module Nuget packages to your xUnit project.

dotnet add package Testcontainers
dotnet add package Testcontainers.PostgreSql

Now that the stage is set, let's move on and introduce ASP.NET Cores WebApplicationFactory before we put everything to work in the last chapter.

ASP.NET Core & WebApplicationFactory

The WebApplicationFactory is a class that allows running an in-memory version of your real application by using a test web host and a test web server.

The NuGet package provides the typeMicrosoft.AspNetCore.Mvc.Testing and is using your application's real configuration, DI service registration, and middleware pipeline.

Here is a basic integration test making use of the WebApplicationFactory together with xUnits IClassFixture interface, which is a marker interface. It tells xUnit to build an instance of T before building the test class and inject the instance into the test class' constructor.

public class IntegrationTest : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;

    public IntegrationTest(WebApplicationFactory<Program> factory)
    {
        _factory = factory;
    }

    [Fact]
    public async Task Should_return_weather_forecast_on_http_get()
    {
        var client = _factory.CreateClient();

        var response = await client.GetAsync("/WeatherForecast");

        response.EnsureSuccessStatusCode();
    }
}

To make this test work, you'll have to add a reference from your test project to your ASP.NET Core project and add public partial class Program {} to it.

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();

app.Run();

public partial class Program { }

By running the real application in memory, you can keep as much distance as possible between your tests and your application's inner workings, which eases the testing of the observable behavior. This approach reduces test fragility by focusing on the whats instead of the hows.

Custom WebApplicationFactory & dependencies

Now let's see how we can create a custom WebApplicationFactory and how to replace dependencies.

As mentioned at the beginning of this section, the factory allows running the application in memory just as it would in production. This implies that EF Core will also connect to your productive database if you don't replace this shared dependency (DbContext) with one pointing to your test container.

Following the simple example above, we would have to replace this dependency for each and every integration test. Instead, we will create a custom factory. This is as simple as inheriting from WebApplicationFactory.

Next, we will remove the database context from the DI container, register a new one pointing to the test container and make sure the database schema gets properly initialized by calling context.Database.EnsureCreated().

public class CustomFactory : WebApplicationFactory<Program>
{
    // Gives a fixture an opportunity to configure the application before it gets built.
    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            // Remove AppDbContext
            var descriptor = services.SingleOrDefault(d => d.ServiceType == typeof(DbContextOptions<AppDbContext>));
            if (descriptor != null) services.Remove(descriptor);
            
            // Add DB context pointing to test container
            services.AddDbContext<AppDbContext>(options => { options.UseNpgsql("the new connection string"); });
            
            // Ensure schema gets created
            var serviceProvider = services.BuildServiceProvider();

            using var scope = serviceProvider.CreateScope();
            var scopedServices = scope.ServiceProvider;
            var context = scopedServices.GetRequiredService<AppDbContext>();
            context.Database.EnsureCreated();
        });
    }
}

This is not the most beautiful code in the world... let's move the removal- and schema creation part to an extension method.

public static class ServiceCollectionExtensions
{
    public static void RemoveDbContext<T>(this IServiceCollection services) where T : DbContext
    {
        var descriptor = services.SingleOrDefault(d => d.ServiceType == typeof(DbContextOptions<T>));
        if (descriptor != null) services.Remove(descriptor);
    }

    public static void EnsureDbCreated<T>(this IServiceCollection services) where T : DbContext
    {
        var serviceProvider = services.BuildServiceProvider();

        using var scope = serviceProvider.CreateScope();
        var scopedServices = scope.ServiceProvider;
        var context = scopedServices.GetRequiredService<T>();
        context.Database.EnsureCreated();
    }
}

This results in a much cleaner factory class.

public class CustomFactory : WebApplicationFactory<Program>
{
    // Gives a fixture an opportunity to configure the application before it gets built.
    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            // Remove AppDbContext
            services.RemoveDbContext<AppDbContext>();
            
            // Add DB context pointing to test container
            services.AddDbContext<AppDbContext>(options => { options.UseNpgsql("the new connection string"); });
            
            // Ensure schema gets created
            services.EnsureDbCreated<AppDbContext>();
        });
    }
}

Pay close attention to call builder.ConfigureTestServices() and not builder.ConfigureServices() when testing an ASP.NET Core application prior to version 6.

The last method will be executed before the WebApplicationFactory calls Startup.ConfigureServices(), which means your production DI registration code, will override your changes, and you might test against your production database!

☝🏻Order of execution 💣

1. builder.ConfigureServices() inside your WebApplicationFactory
2. Startup.ConfigureServices() from your application code
3. builder.ConfigureTestServices() inside WebApplicationFactory

Putting everything to work

The only thing left is to merge everything together. I have introduced generics to make it reusable across different projects in a solution.

public class IntegrationTestFactory<TProgram, TDbContext> : WebApplicationFactory<TProgram>, IAsyncLifetime
    where TProgram : class where TDbContext : DbContext
{
    private readonly TestcontainerDatabase _container;

    public IntegrationTestFactory()
    {
        _container = new TestcontainersBuilder<PostgreSqlTestcontainer>()
            .WithDatabase(new PostgreSqlTestcontainerConfiguration
            {
                Database = "test_db",
                Username = "postgres",
                Password = "postgres",
            })
            .WithImage("postgres:11")
            .WithCleanUp(true)
            .Build();
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.ConfigureTestServices(services =>
        {
            services.RemoveProdAppDbContext<TDbContext>();
            services.AddDbContext<TDbContext>(options => { options.UseNpgsql(_container.ConnectionString); });
            services.EnsureDbCreated<TDbContext>();
        });
    }

    public async Task InitializeAsync() => await _container.StartAsync();

    public new async Task DisposeAsync() => await _container.DisposeAsync();
}

And here is a basic test making use of the custom factory.

public class Tests : IClassFixture<IntegrationTestFactory<Program, AppDbContext>>
{
    private readonly IntegrationTestFactory<Program, AppDbContext> _factory;

    public Tests(IntegrationTestFactory<Program, AppDbContext> factory) => _factory = factory;

    [Fact]
    public async Task Foo()
    {
        var client = _factory.CreateClient();

        var response = await client.GetAsync("/weatherforecast");
        
        response.EnsureSuccessStatusCode();
    }
}

Final thoughts

This solution is nice because it balances resistance to refactoring, protection against regressions, and fast feedback (see Khorikov, 2020, p. 88).

Last but not least, the tests are runnable on GitHub without further modifications to the virtual environments 🤓

Further reading

Integration tests in ASP.NET Core

Learn how integration tests ensure that an app’s components function correctly at the infrastructure level, including the database, file system, and network.

Microsoft DocsRick-Anderson

Converting integration tests to .NET Core 3.0: Upgrading to ASP.NET Core 3.0 - Part 5

In this post I discuss the changes required to upgrade integration tests that use WebApplicationFactory or TestServer to ASP.NET Core 3.0.

Andrew Lock | .NET EscapadesAndrew Lock

virtual-environments/Ubuntu2004-Readme.md at main · actions/virtual-environments

GitHub Actions virtual environments. Contribute to actions/virtual-environments development by creating an account on GitHub.

GitHubactions

Testcontainers for .NET

logo

Introduction

Currently, I am evaluating different Identity and Access Management solutions as an alternative to ASP.NET Core Identity. Besides FusionAuth I wanted to test-drive Auth0 together with ASP.NET Core Blazor.

Although Auth0 provides rich documentation, no easy-to-follow example was available for ASP.NET Core 9.

This article fills this gap and runs you through the required steps to get started with Auth0.

TL;DR?

Here is the direct link to my demo repository

GitHub - matthiasguentert/auth0-blazor-server-net9

Contribute to matthiasguentert/auth0-blazor-server-net9 development by creating an account on GitHub.

GitHubmatthiasguentert

Configure Auth0

First, we need to create a new application. For a Blazor Server, we need to select Regular Web Applications.

After successful creation, make a note of your Domain and Client ID found under Settings.

Configure Callback URLs

Next, we need to adjust the list of Allowed Callback URLs and add https://localhost:7063/callback for local testing.

Configure Logout URLs

For the logout to work, we need to add https://localhost:7063

Install and configure the SDK

Register middleware

Add the required NuGet package Auth0.AspNetCore.Authentication to your project and register the Auth0 middleware with the DI container.

builder.Services.AddAuth0WebAppAuthentication(options =>
{
    options.Domain = builder.Configuration["Auth0:Domain"];
    options.ClientId = builder.Configuration["Auth0:ClientId"];
    options.Scope = "openid profile email";
});

Add configuration

As you can see from above, the setup expects the following keys to exist in your appsettings.json. Paste the information accordingly.

  "Auth0": {
    "Domain": "<your-domain>",
    "ClientId": "<your-client-id>"
  }

Add the login endpoint

Now it's time to create a minimal API endpoint to provide the login functionality.

This is where we call ChallengeAsync and pass the authentication properties created by the LoginAuthenticationPropertiesBuilder and the Auth0 authentication schema. The latter also defines the default callback path /callback.

From the official documentation

After successfully calling HttpContext.ChallengeAsync(), the user will be redirected to Auth0 and signed in to both the OIDC middleware and the cookie middleware upon being redirected back to your application. This will allow the users to be authenticated on subsequent requests.

app.MapGet("/Login", async Task (HttpContext httpContext, string returnUrl = "/") =>
{
    var authenticationProperties = new LoginAuthenticationPropertiesBuilder()
        .WithRedirectUri(returnUrl)
        .Build();

    await httpContext.ChallengeAsync(Auth0Constants.AuthenticationScheme, authenticationProperties);
});

Add the logout endpoint

Logging out happens by calling SignOutAsync on the HttpContext. As you can see, this call was made twice.

The first call will log the user out of Auth0, and the second will log the user out of your application. This will also log a user out of other applications that rely on SSO (single sign-on)

app.MapGet("/Logout", async (HttpContext httpContext) =>
{
    var authenticationProperties = new LogoutAuthenticationPropertiesBuilder()
        .WithRedirectUri("/")
        .Build();

    await httpContext.SignOutAsync(Auth0Constants.AuthenticationScheme, authenticationProperties);
    await httpContext.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme);
});

Adjust routing

Next, we'll modify the routing configuration defined at Components/Routes.razor. Make sure to add the required using statements to _Imports.razor.

...
@using Microsoft.AspNetCore.Authorization
@using Microsoft.AspNetCore.Components.Authorization

Add <CascadingAuthenticationState>, which provides the authentication state (e.g., whether the user is logged in) to all components below it in the hierarchy. It wraps the entire router, so any page can access the current authentication state using the AuthenticationStateProvider.

<CascadingAuthenticationState>
  <Router AppAssembly="typeof(Program).Assembly">
    <Found Context="routeData">
      <AuthorizeRouteView RouteData="routeData" DefaultLayout="@typeof(Layout.MainLayout)">
        <Authorizing>
          <p>Authorization is in progress, please wait!</p>
        </Authorizing>
        <NotAuthorized>
          <p>You're not authorized, please log in!</p>
        </NotAuthorized>
      </AuthorizeRouteView>
      <FocusOnNavigate RouteData="routeData" Selector="h1"/>
    </Found>
  </Router>
</CascadingAuthenticationState>

The router uses the AuthorizedRouteView element, which is like the regular RouteView but with built-in support for authentication and authorization.

The Authorizing element, defines content that will be rendered while asynchronous authorization is in progress, and NotAuthorized defines the content that will be displayed if the user is not authorized.

FocusOnNavigate will focus on the first <h1> element, which is helpful for accessibility and screen readers.

Conclusion

That's all that is required for the basic setup. For demo purposes, I have added a Profile page and adjusted the NavMenu.

To summarize

• Create an Auth0 application
• Add the required NuGet package
• Add configuration settings
• Add middleware and endpoints
• Adjusts routing

Further reading

GitHub - matthiasguentert/auth0-blazor-server-net9

Contribute to matthiasguentert/auth0-blazor-server-net9 development by creating an account on GitHub.

GitHubmatthiasguentert

Router Class (Microsoft.AspNetCore.Components.Routing)

A component that supplies route data corresponding to the current navigation state.

Microsoft Learndotnet-bot

GitHub - auth0/auth0-aspnetcore-authentication: SDK for integrating Auth0 in ASPNET Core

SDK for integrating Auth0 in ASPNET Core. Contribute to auth0/auth0-aspnetcore-authentication development by creating an account on GitHub.

GitHubauth0

Auth0 Authentication API

Introduction

With this blog post, I'll demonstrate how we can automatically register Ingress resources running on an AKS cluster with a public Azure DNS zone so they can be easily reached outside your cluster.

Further, I'll demonstrate how certificates can be automatically obtained from Let's Encrypt and then be assigned to those Ingress resources.

For that purpose, I'll use two Kubernetes controllers: ExternalDNS and Cert-Manager.

ExternalDNS for Kubernetes is a controller that automates the management of DNS records for Kubernetes resources by syncing them with various DNS providers, like Azure DNS zones.

On the other hand, Cert-Manager automates the management and issuance of SSL/TLS certificates for applications running within a Kubernetes cluster.

For enhanced security, I'll use a Managed Identity with federated credentials, which requires using the Workload Identity feature on my AKS cluster.

This setup comes with a couple of compelling benefits... 🏆

Why you want to automate DNS and certificate management

1. End-to-end Automation: Combining ExternalDNS and Cert-Manager provides end-to-end automation for exposing services securely over the internet. Cert-Manager handles the issuance and renewal of SSL/TLS certificates, while ExternalDNS manages DNS records, ensuring that the services are secure and reachable.
2. Reducing the risk of certificate expiration: Cert-Manager supports the great ACME protocol. Combined with a Certificate Authority like Let's Encrypt, it streamlines obtaining and renewing SSL/TLS certificates. This eliminates the risk of expired certificates!
3. Self-Service Capabilities: Developers can use annotations on Ingress resources to define DNS and certificate requirements for their services. This empowers development teams to manage their service configurations within established policies, reducing the burden on central IT or operations teams.
4. Logging and Auditing: Both tools provide logging and auditing capabilities, making monitoring and tracking changes to DNS records and certificates easier. This is valuable for compliance, troubleshooting, and security purposes.

By combining Cert-Manager and ExternalDNS, we create a robust and automated solution that ensures our Ingress resources are secured with up-to-date certificates and that DNS records are always synchronized with the underlying services.

This approach greatly simplifies the operation of an Azure Kubernetes Cluster and helps accelerate the development cycle of developers using the Azure Kubernetes Service. 🚀

High-Level Steps ✍🏼

These are the high-level steps. I assume you already have an AKS cluster and Azure DNS Zone deployed. Also, I assume you have a running NGINX Ingress Controller setup.

1. Enable Azure AD Workload Identities on AKS Cluster.
2. Create and configure a Managed Identity and assign the required privileges.
3. Install and/or configure NGINX Ingress Controller
4. Install and configure External-DNS.
5. Install and configure Cert-Manager.

Now that the context is set let's get to work👷🏽‍♂️🚧

Steps

Enable Azure AD Workload Identities

First, we must update our existing AKS cluster to support workload identities. Make sure your Azure CLI is least on version 2.47.0 or later.

az aks update `
  --name <cluster-name> `
  --resource-group <cluster-rg> `
  --subscription "<subscription>" `
  --enable-workload-identity `
  --enable-oidc-issuer

💡 What is Azure AD Workload Identity?

Azure AD Workload Identity with AKS is a feature that bridges Kubernetes-native service accounts to Azure AD identities. This mapping allows a service account to access Azure AD-protected resources.

Afterward, we need to retrieve and store the issuer URL. The URL comes in the form of https://westeurope.oic.prod-aks.azure.com/<some-guid>/<another-guid>/.

$issuerUrl = az aks show `
  --name aks-azureblue `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --query oidcIssuerProfile.issuerUrl `
  --output tsv

Creating and configuring a Managed Identity

Let's start by creating and storing the Managed ID in the same resource group as the AKS cluster (YMMV).

Alternatively, you can reuse the existing one named <cluster-name>-agentpool. This gets deployed into the infrastructure resource group at cluster-creation time. For this article, I'll create a dedicated one.

az identity create `
  --name id-aks-azureblue-workload-identity `
  --resource-group rg-kubernetes `
  --subscription "<subscription>"

Configure Federated Credentials

Now that the managed identity is in place, we must configure the federated credentials.

Later, we will install ExternalDNS and cert-manager on the Kubernetes cluster. Each of the helm charts will create a separate service account, which means we have to create two federated credentials to map both service accounts to the managed identity.

Let's start with the federated credential for the ExternalDNS service account, which will live in a namespace with the same name. The subject parameter follows the following format: system:serviceaccount:<namespace>:<serviceaccountname>

az identity federated-credential create `
  --identity-name id-aks-azureblue-workload-identity `
  --name external-dns-credentials `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --issuer $issuerUrl `
  --subject "system:serviceaccount:external-dns:external-dns"

Okay, only one left for the cert-manager service account.

az identity federated-credential create `
  --identity-name id-aks-azureblue-workload-identity `
  --name cert-manager-credentials `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --issuer $issuerUrl `
  --subject "system:serviceaccount:cert-manager:cert-manager"

Role Assignments

So far, so good. Now, it is time to equip the managed identity with some privileges.

Cert-Manager requires the DNS Zone Contributor role on the DNS Zone to create TXT records (DNS-01 challenge). This is needed to prove the ownership of the DNS domain to Let's Encrypt. You can read more about it here.

Challenge Types

When you get a certificate from Let’s Encrypt, our servers validate that you control the domain names in that certificate using “challenges,” as defined by the ACME standard. Most of the time, this validation is handled automatically by your ACME client, but if you need to make some more complex con…

Let's Encrypt

Then, ExternalDNS also requires extended privileges on the DNS zone to create and optionally remove A records at run-time. In my case, the DNS Zone is called dev.azureblue.io. In addition, ExternalDNS requires the Reader role to the resource group holding the DNS zone.

# Retrieve and store object id of managed identity
$assigneeObjectId = az identity show `
  --name id-aks-azureblue-workload-identity `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --query principalId `
  --output tsv
  
# Retrieve and store id of dns zone
$dnsZoneId = az network dns zone show `
  --name dev.azureblue.io `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --query id `
  --output tsv

# Retrieve and store id of resource group
$resourceGroupId = az group show `
  --name rg-kubernetes `
  --subscription "<subscription>" `
  --query id `
  --output tsv

# Assign the DNS Zone Contributor role on the zone
az role assignment create `
  --role "DNS Zone Contributor" `
  --assignee-object-id $assigneeObjectId `
  --assignee-principal-type ServicePrincipal `
  --scope $dnsZoneId
  
# Assign the Reader role on the resource group
az role assignment create `
  --role "Reader" `
  --assignee-object-id $assigneeObjectId `
  --assignee-principal-type ServicePrincipal `
  --scope $resourceGroupId

Checkpoint ✅

Up until this point, we should have:

• ✅ Enabled workload identity on the existing AKS cluster
• ✅ Created and configured a managed identity
• ✅ Assigned the required roles

Until now, everything is configured on the Azure side, and we can move on to install the required Kubernetes components.

NGINX Ingress Controller Configuration

Ensure that your nginx-ingress deployment has the following argument added to it. This is required for ExternalDNS.

- --publish-service=<ingress-nginx-namespace>/<nginx-controller-service-name>

I used the following steps to deploy ingress-nginx...

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  --install \
  --namespace ingress-nginx \
  --create-namespace \
  --values values.yaml

... with this value file.

controller:
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"
    extraArgs:
      # Required for ExternalDNS
      publish-service: nginx-system/ingress-nginx-controller

Installing & Configuring ExternalDNS

Now that the Nginx Ingress Controller is properly configured let's install ExternalDNS.

This is the Helm Chart Values file I've used for configuration. Of course, replace the values accordingly.

fullnameOverride: external-dns
policy: sync
serviceAccount:
  annotations:
    azure.workload.identity/client-id: <your-client-id>
podLabels:
  azure.workload.identity/use: "true"
provider: azure
secretConfiguration:
  enabled: true
  mountPath: "/etc/kubernetes/"
  data:
    azure.json: |
      {
        "subscriptionId": "<subscription-id>",
        "resourceGroup": "rg-kubernetes",
        "useWorkloadIdentityExtension": true
      }

The configuration does a couple of important things.

1. serviceAccount.annotations: azure.workload.identity/client-id: ... & podLabels: azure.workload.identity/use: "true" enables workload identity for the service account used by ExternalDNS.
2. policy: sync ensures that the deletion of Ingress resources also results in the deletion of the corresponding A record.

The client ID can be retrieved as follows.

az identity show `
  --name id-aks-azureblue-workload-identity `
  --resource-group rg-kubernetes `
  --subscription "<subscription>" `
  --query clientId `
  --output tsv

Now, let's install the corresponding Helm Chart and ensure we use the correct namespace, as our previously created federated credential needs to match it.

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update
helm upgrade external-dns external-dns/external-dns `
  --install `
  --namespace external-dns `
  --create-namespace `
  --values values.yaml

Checkpoint ✅

Before moving on, testing that the ExternalDNS setup works as expected is advisable. So go ahead and deploy the following manifest to a namespace called demo. Of course, you need to adjust the Ingress resource according to your environment.

kubectl create namesapce demo
kubectl apply -f deployment.yaml 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: demo
  name: echo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: echo-pod
        image: hashicorp/http-echo
        args:
          - "-text=Hello World!"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  namespace: demo
  name: demo-service
spec:
  type: ClusterIP
  selector:
    app: demo
  ports:
    - port: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: demo
  name: demo-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: demo1.dev.azureblue.io
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: demo-service
                port:
                  number: 5678  

As can be seen from the manifest, we are creating a Deployment and a Service. Then, we expose it to the Internet through an Ingress using the hostname demo1.dev.azureblue.io.

As we haven't set up any certificates yet, disabling SSL-Redirection on the Ingress is important so it can be tested using the HTTP protocol.

After a few seconds, you should see an A and a TXT record in your public DNS zone. Navigating to http://demo1.dev.azureblue.io should display Hello World!.

As depicted in the screenshot above, the default TTL is 300 seconds. Optionally, this can be adjusted to your requirements by annotating the Ingress resource with external-dns.alpha.kubernetes.io/ttl: "valueInSeconds".

💡 The default TTL value of 300 seconds can be adjusted by annotating the Ingress with external-dns.alpha.kuberntes.io/ttl: "valueInSeconds"

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: demo
  name: demo-ingress
  annotations:
    ... 
    external-dns.alpha.kubernetes.io/ttl: "60"
spec:
...

When configuring ExternalDNS, we used the parameter policy: sync, remember? This parameter keeps your DNS Zone in sync with your Ingress annotations.

If this isn't set, the default behavior is to upsert DNS record changes without deleting them if an Ingress gets deleted, leaving stale DNS records over time.

💡 To keep your Azure DNS zone in sync with your Ingress controller use the "policy: sync" setting.

So, let's also test that functionality and delete the previously created deployment (or only the Ingress resource).

kubectl delete namespace demo

After a few seconds, the TXT and A record should be removed from the DNS zone.

Cert-Manager Installation & Configuration

Now that we have successfully automated the DNS registration process, we can get to automate the certificate management.

In a previous step, we prepared the Azure Managed Identity with a federated credential for Cert-Manager. Now, we need to enable it for the cert-manager deployment itself. This is done by the label azure.workload.identity/use: "true" that must be applied to the pod.

💡 The azure-workload-identity mutating admission webhook will only modify pods with this label to inject the service account token volume projection and Azure-specific environment variables (Microsoft, 2023).

Also, the following Custom Resource Definitions need to be installed.

• ClusterIssuer, Issuer
• CertificateRequests, Certificates
• Challenges
• Orders

So here is a basic configuration I've used with the official Helm Chart.

podLabels:
  azure.workload.identity/use: "true"
serviceAccount:
  labels:
    azure.workload.identity/use: "true"
installCRDs: "true"   

When installing, make sure to use the namespace cert-manager, since it needs to match the configured federated credentials in earlier steps.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade cert-manager jetstack/cert-manager \
  --install \
  --namespace cert-manager \
  --create-namespace \
  --values values.yaml

Create Cluster Issuer

Now, we need to define an Issuer resource. These types of resources represent certificate authorities (CAs), which can sign certificates in response to certificate signing requests. We can create either one of the type ClusterIssuer or one of the type Issuer , which then will be scoped to a namespace.

Cert-Manager supports various types of issuers, such as HashiCorp Vault, Venafi, and others. Since the goal is to automate certificate management fully, I'll use one called acme.

Obviously, this issuer represents Certificate Authority servers implementing the Automated Certificate Management Environment (ACME) protocol, such as Let's Encrypt.

ACME requires a challenge to be solved to verify that we own the domain; details can be found in the link below. For this demo, I'll stick with the DNS01 challenge.

Challenge Types

When you get a certificate from Let’s Encrypt, our servers validate that you control the domain names in that certificate using “challenges,” as defined by the ACME standard. Most of the time, this validation is handled automatically by your ACME client, but if you need to make some more complex con…

Let's Encrypt

The following manifest will create two ClusterIssuers, one for the staging and another for the prod environment of Let's Encrypt.

You'll have to replace and adjust email, hostedZoneName, resourceGroupName, subscriptionId and clientId to your environment.

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <your-mail-address>
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - dns01:
        azureDNS:
          hostedZoneName: dev.azureblue.io
          resourceGroupName: rg-kubernetes
          subscriptionID: <subscriptionId>
          environment: AzurePublicCloud
          managedIdentity:
            clientID: <clientId>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your-mail-address>
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        azureDNS:
          hostedZoneName: dev.azureblue.io
          resourceGroupName: rg-kubernetes
          subscriptionID: <subscriptionId>
          environment: AzurePublicCloud
          managedIdentity:
            clientID: <clientId>

After applying, make sure both are in a ready state.

kubectl get clusterissuer -o wide

NAME                  READY   STATUS                                                 AGE
letsencrypt-prod      True    The ACME account was registered with the ACME server   1h
letsencrypt-staging   True    The ACME account was registered with the ACME server   1h

Testing the setup with a demo deployment

Finally, we reached the point where we can test the entire setup. 🏆Go ahead and adjust the demo deployment to your environment, especially the hostnames from the Ingress resource.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: demo
  name: echo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: echo-pod
        image: hashicorp/http-echo
        args:
          - "-text=Hello World!"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  namespace: demo
  name: demo-service
spec:
  type: ClusterIP
  selector:
    app: demo
  ports:
    - port: 5678 # The port that will be exposed by this service
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: demo
  name: demo-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    external-dns.alpha.kubernetes.io/ttl: "60"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - demo1.dev.azureblue.io
      secretName: demo-tls
  rules:
    - host: demo1.dev.azureblue.io
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: demo-service
                port:
                  number: 5678  

After applying, give Cert-Manager some time to solve the DNS01 challenge. It will create a secret called demo-tls for you, holding the certificate and private key for the Ingress.

kubectl describe secret demo-tls -n demo

Name:         demo-tls
Namespace:    demo
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: demo1.dev.azureblue.io
              cert-manager.io/certificate-name: demo-tls
              cert-manager.io/common-name: demo1.dev.azureblue.io
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
tls.crt:  5534 bytes
tls.key:  1675 bytes

Eh voilà!! 🎉🎊🥂

Recap & conclusion

• With ExternalDNS, we can customize the TTL value by using an annotation external-dns.alpha.kuberntes.io/ttl: "valueInSeconds"
• ExternalDNS can be configured to keep the Azure DNS zone tidy. The relevant parameter is called policy: sync.
• There are two major ACME challenge types to prove ownership of a domain. These are HTTP01 and DNS01. Both come with pros and cons. Here, we have been using the DNS-01 challenge type.
• We created a Managed Identity and configured federated credential, so that the two Kubernetes service accounts (cert-manager and external-dns) can be authorized and authenticated against Azure services (Azure DNS zone in our case)
• Automating DNS & certificate management reduces the burden of manual tasks carried out by your platform team 😎

Thanks for reading. As always, feedback is welcome!

Happy hacking! Matthias 🤓

Further reading

Use an Azure AD workload identity on Azure Kubernetes Service (AKS) - Azure Kubernetes Service

Learn about Azure Active Directory workload identity for Azure Kubernetes Service (AKS) and how to migrate your application to authenticate using this identity.

Microsoft LearnMGoedtel

GitHub - kubernetes-sigs/external-dns: Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services - GitHub - kubernetes-sigs/external-dns: Configure external DNS servers (AWS Route53, ...

GitHubkubernetes-sigs

cert-manager

Cloud native X.509 certificate management for Kubernetes and OpenShift

cert-manager

Introduction

When dealing with enterprise-grade Azure deployments, tracking a specific resource down to the Terraform code it stems from can be difficult...

Also, finding the person responsible who can be asked questions regarding the configuration used is not always obvious.

This is especially true in setups where multiple teams maintain Azure deployments with Terraform.

So, when looking at an Azure resource within the Azure portal, I find it helpful to get a quick answer to the following questions:

• Who can I contact if you have questions regarding the resource configuration?
• Where can I find the Terraform code that describes the resource?
• At which specific commit hash do I need to look?
• When was the last time a change was made to that resource?

You might be aware of a tool called yor an extensible auto-tagger for IaC files written in Go. Unfortunately, this tool didn't work for me and produced many git blame warnings I couldn't solve.

Instead, we will leverage a Terraform provider called metio/git to tag Azure resources for the purpose described and let an Azure DevOps pipeline run terraform apply for us.

GitHub - metio/terraform-provider-git: Terraform provider for local Git operations

Terraform provider for local Git operations. Contribute to metio/terraform-provider-git development by creating an account on GitHub.

GitHubmetio

Let's get to it! 😎

Discussion

These are the tags we will add to our Azure resources

• git_repo, git_branch and git_commit_hash holding the shortened 8-digit version of the latest commit. So we know where to find the code defining the resources.
• git_commit_message and git_commit_timestamp so we get a hint about when the code got committed and what has changed
• Tags called git_author_name and git_author_email so we know who wrote the deployed code.

We can pull all of this information by using three data sources, which are git_commit, git_repository, and git_remote.

data "git_commit" "head" {
  directory = var.git_directory
  revision  = "@"
}

data "git_repository" "repo" {
  directory = var.git_directory
}

data "git_remote" "remote" {
  directory = var.git_directory
  name      = "origin"
}

It's worth noting that I am using the head shortcut @ , which refers to the latest commit in the branch and represents the current state of the working directory.

Further, I feed the repository folder by a variable called var.git_directory. This will be required later when we execute terraform apply by an Azure DevOps pipeline.

For ease of use, we can create a map like this and put it into a locals block. See the documentation for more details regarding the data source schema.

locals {
  git_tags = {
    git_commit_hash      = substr(data.git_commit.head.sha1, 0, 8)
    git_commit_message   = data.git_commit.head.message
    git_commit_timestamp = data.git_commit.head.author.timestamp
    git_author_name      = data.git_commit.head.author.name
    git_author_email     = data.git_commit.head.author.email
    git_repo             = data.git_remote.remote.urls[0]
    git_branch           = data.git_repository.repo.branch
  }
}

Later, we can add the gathered information to a resource like this:

resource "azurerm_resource_group" "foobar" {
  name     = "rg-tagging-demo-1"
  location = "switzerlandnorth"

  tags = local.git_tags
}

So far, so good. Let's move on to the Azure DevOps pipeline. I use the Azure Pipelines Terraform Tasks extension, which can be installed via Marketplace.

Azure Pipelines Terraform Tasks - Visual Studio Marketplace

Extension for Azure DevOps - Tasks to install and execute terraform with Azure Pipelines for Azure and AWS.

Visual Studio Marketplace

Here is the pipeline.

trigger: none

pool:
  vmImage: ubuntu-latest

steps:
  - checkout: self

  - task: TerraformInstaller@2
    displayName: "Terraform install"
    inputs:
      terraformVersion: "latest"

  - task: TerraformCLI@2
    displayName: "Terraform init"
    inputs:
      command: 'init'
      backendType: 'azurerm'
      backendServiceArm: 'service-connection-1'
      backendAzureRmResourceGroupName: 'rg-automation'
      backendAzureRmStorageAccountName: '<hidden>'
      backendAzureRmContainerName: 'tfstate'
      backendAzureRmKey: 'terraform.tfstate'
      allowTelemetryCollection: false

  - task: TerraformCLI@2
    displayName: "Terraform apply"
    inputs:
      command: 'apply'
      environmentServiceName: 'service-connection-1'
      commandOptions: '-var="git_directory=$(Build.SourcesDirectory)"'
      allowTelemetryCollection: false

In the first step, I check out the repository with checkout: self. It's worth noting that Azure DevOps does a checkout on a commit, not a branch, which results in a detached head. Next, I install the latest version of the terraform binary and initialize the backend.

The last step applies the terraform code. Please note that I passed the ADO variable Build.SourcesDirectory on to the terraform configuration. This way, the terraform git provider can pick up the proper directory from the build agent.

After a successful pipeline run, the resources carry the expected Azure tags, as depicted below. Cool, isn't it 😎

💡Please note that commit hashes can get lost when executing terraform apply from a feature branch that later gets merged back to main

Therefore, in the described situation, you should always apply your terraform code from the main branch (or another stable branch). Otherwise, you might be surprised not to find a matching commit in your repository🤯

Conclusion

• Azure DevOps does a checkout on a commit, not a branch, resulting in a detached HEAD
• Commit hashes can get lost when applying from feature branches
• We need to pass the git repository of the build agents to the terraform configuration by using a variable git_directory=$(Build.SourcesDirectory)

A proper Azure tagging strategy is crucial when navigating & managing your Azure deployment. But especially in large enterprise setups where resources are being deployed utilizing IaC, adding git metadata makes everyone's life easier.

I hope you enjoyed reading my article and would gladly receive feedback! What does your Azure tagging strategy look like?

Happy hacking! 😎

Further reading

Terraform Registry

Azure Pipelines Terraform Tasks - Visual Studio Marketplace

Extension for Azure DevOps - Tasks to install and execute terraform with Azure Pipelines for Azure and AWS.

Visual Studio Marketplace

Introduction

If you come from a traditional software development background, you're likely familiar with using feature flags or toggles.

Put simply, a feature toggle acts as a switch to turn a specific feature on or off. It allows code to be released into production without activating it immediately—or only under certain conditions.

This approach enables gradual rollouts, making it possible to introduce changes incrementally and minimize risk. It also supports quick rollbacks, allowing you to disable problematic features without redeploying.

Additionally, feature toggles are commonly used for A/B testing scenarios, where new functionality is enabled for a subset of users to compare their responses with those of a control group.

However, the context is quite different when working with Terraform, an IaC language. So, why should you consider using feature flags with Terraform?

The reasons are flexibility & backwards compatibility. Let's see how we can put them to work.

Toggling with count meta-argument

A fundamental feature toggle in HCL looks as follows.

variable "enable_feature" {
  type    = bool 
  default = false
}

resource "azurerm_resource_group" "rg" {
  count = var.enable_feature ? 1 : 0

  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

Here, the meta-argument count is used to define how many instances of a resource the provider should create. In our scenario, we combine it with a conditional expression condition ? true_value : false_value. So, when the variable enable_feature becomes true then a single instance (1) of the resource gets created.

This pattern is peculiar when we want to reference resources to each other. For example, the following code won't work!

variable "enable_feature" {
  type    = bool 
  default = false
}

resource "azurerm_resource_group" "rg" {
  count = var.enable_feature ? 1 : 0

  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

resource "azurerm_network_security_group" "nsg" {
  count = var.enable_nsg ? 1 : 0

  name                = "nsg-feature-toggle-demo"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
}

Trying to apply this configuration will result in the following error.

│ Error: Missing resource instance key
│ 
│   on main.tf line 39, in resource "azurerm_network_security_group" "nsg":
│   39:   resource_group_name = azurerm_resource_group.rg.name
│ 
│ Because azurerm_resource_group.rg has "count" set, its attributes must be accessed on specific instances.
│ 
│ For example, to correlate with indices of a referring resource, use:
│     azurerm_resource_group.rg[count.index]

Since the count meta-argument creates instances, we need to reference the specific resource by its key. So this will fix it.

variable "enable_feature" {
  type    = bool
  default = false
}

resource "azurerm_resource_group" "rg" {
  count = var.enable_feature ? 1 : 0

  name     = "rg-feature-toggle-demo"
  location = var.location
}

resource "azurerm_network_security_group" "nsg" {
  count = var.enable_feature ? 1 : 0

  name                = "nsg-foobar"
  resource_group_name = azurerm_resource_group.rg[0].name
  location            = azurerm_resource_group.rg[0].location
}

Using for_each with a map

An alternative approach to count is using the for_each argument. Here is an example.

variable "enable_feature" {
  type    = bool
  default = false
}

resource "azurerm_resource_group" "rg" {
  for_each = var.enable_feature ? { "enabled" = "enabled" } : {}

  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

resource "azurerm_network_security_group" "nsg" {
  for_each = var.enable_feature ? { "enabled" = "enabled" } : {}

  name                = "nsg-foobar"
  resource_group_name = azurerm_resource_group.rg["enabled"].name
  location            = azurerm_resource_group.rg["enabled"].location
}

Again, we are using a conditional expression var.enable_feature ? { "enabled" = "enabled" } : {}. If the expression becomes true a map with a single element is returned (the map has a key named enabled with a value of enabled), that for_each can iterate. If the expression becomes false an empty map is returned.

The version above can be slightly optimized for better readability.

resource "azurerm_resource_group" "rg" {
  for_each = var.enable_feature ? { "enabled" = "enabled" } : {}

  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

resource "azurerm_network_security_group" "nsg" {
  for_each = var.enable_feature ? { "enabled" = azurerm_resource_group.rg["enabled"] } : {}

  name                = "nsg-foobar"
  resource_group_name = each.value.name
  location            = each.value.location
}

This time, we directly assign the referenced value to a key named enabled and access its attributes by the each object.

Toggling specific arguments

This time, we don't want to toggle the entire resource; we only wish to switch a specific attribute on and off.

In the example below, again, we are using a conditional expression that returns the desired map of Azure tags we'd like to assign in case var.enable_tags is true, otherwise, we assign null.

variable "enable_tags" {
  type    = bool
  default = false
}

resource "azurerm_resource_group" "rg" {
  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

resource "azurerm_network_security_group" "nsg" {
  name                = "nsg-foobar"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  tags = var.enable_tags ? { environment = "dev" } : null
}

Toggling features within modules

Consider a fictional scenario where you'd like to create a vnet including subnet and dynamically toggle the creation of a network security group.

terraform {
... 
}

provider "azurerm" {
...
}

module "vnet" {
  source = "./vnet_module"

  enable_nsg = false
  
  ... Potentially more useful attributes here
}

Nothing stops us from using the same count construct within a child module.

variable "enable_nsg" {
  type    = bool
  default = false
}

resource "azurerm_resource_group" "vnet" {
  name     = "rg-feature-toggle-demo"
  location = "switzerlandnorth"
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-foobar"
  resource_group_name = azurerm_resource_group.vnet.name
  location            = azurerm_resource_group.vnet.location

  address_space = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "snet1" {
  name                 = "snet1"
  resource_group_name  = azurerm_resource_group.vnet.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_network_security_group" "nsg" {
  count = var.enable_nsg ? 1 : 0

  name                = "nsg-foobar"
  resource_group_name = azurerm_resource_group.vnet.name
  location            = "switzerlandnorth"
}

resource "azurerm_subnet_network_security_group_association" "example" {
  count = var.enable_nsg ? 1 : 0

  subnet_id                 = azurerm_subnet.snet1.id
  network_security_group_id = azurerm_network_security_group.nsg[0].id
}

Again, we use the count meta-argument to dynamically create the network security group and its subnet association.

Another benefit that comes with feature toggles

Besides the flexibility that comes with this toggle, there is another benefit, which might not be so obvious - backwards compatibility.

Consider the case, where multiple root modules are using your vnet child module. That's what we write modules for, right? You might not even know, how many root modules in the enterprise are relaying on your shiny vnet module.

But still, you need to carry on and further enhance the module with a new feature, let's say you decide new vnets should use an edge zone. When looking at the documentation, you read ...

edge_zone - (Optional) Specifies the Edge Zone within the Azure Region where this Virtual Network should exist. Changing this forces a new Virtual Network to be created.

If we simply add the attribute to our child, the next terraform apply will re-create the vnet, which is not always what we want.

Terraform will perform the following actions:

  # module.vnet.azurerm_virtual_network.vnet must be replaced
-/+ resource "azurerm_virtual_network" "vnet" {
      ~ dns_servers             = [] -> (known after apply)
      + edge_zone               = "switzerlandnorth" # forces replacement
    ...
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Instead we can toggle the desired attribute, provide a default value of false to the toggle variable, and don't have to worry that other users of the root module will have to recreate their resources.

variable "enable_edge_zone" {
  type    = bool
  default = false
}

...

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-foobar"
  resource_group_name = azurerm_resource_group.vnet.name
  location            = azurerm_resource_group.vnet.location

  edge_zone     = var.enable_edge_zone ? "switzerlandnorth" : null
  address_space = ["10.0.0.0/16"]
}

Summary

• Feature toggles with Terraform provide flexibility but also provide backwards compatibility for child modules
• We can use both count and for_each constructs to realize feature toggles
• I prefer the count version since it enhances readability
• The count meta-argument creates instances of resources, and therefor, when referenced by other resources, needs to be access by its index

That's it for today, thanks for reading. 😎

Further reading

Feature Toggles, Blue-Green Deployments & Canary Tests with Terraform

In this post, we demonstrate some approaches to feature toggling, blue-green deployment, and canary testing of Terraform resources to mitigate impact to production infrastructure.

HashiCorpRosemary Wang

The count Meta-Argument - Configuration Language | Terraform | HashiCorp Developer

Count helps you efficiently manage nearly identical infrastructure resources without writing a separate block for each one.

The count Meta-Argument - Configuration Language | Terraform | HashiCorp Developer

Conditional Expressions - Configuration Language | Terraform | HashiCorp Developer

Conditional expressions select one of two values. You can use them to define defaults to replace invalid values.

Conditional Expressions - Configuration Language | Terraform | HashiCorp Developer

The for_each Meta-Argument - Configuration Language | Terraform | HashiCorp Developer

The for_each meta-argument allows you to manage similar infrastructure resources without writing a separate block for each one.

The for_each Meta-Argument - Configuration Language | Terraform | HashiCorp Developer