This article is also available in Italian here.

The pace of today’s modern software development and release is getting faster and faster.

Just a couple of years ago, before implementing a feature, there was a lot of upfront analysis. That was often based on assumptions made by the product and business teams.

That process was slow and sometimes when the feature finally reached the customer, it was still a little bit bugged. In addition, the first feedback was not always as satisfying as expected.

Finally, we are seeing many teams that are bringing the customer and the final users into the development process, involving them in the Agile process.

Bigger features are broken down into smaller individual deliverable pieces that can still bring value as well as prepare the way for future steps. And, in the meantime, collecting feedback.

Still, it can happen that, before making a feature generally available, we want to first show it to a limited set of customers and see their first reactions.

Demo sessions and dedicated environments where the users can play with the new functionalities are great tools for that.

However, these are still something that diverges from what is already in production, where the real things happen, where the load is heavy and performances are always a pain.

Wouldn't it be nice to just release everything in production and make the new feature available only to the beta testers? And if their feedback is good, extend the feature to a larger set of users, in a way to also measures how it impacts the overall performance of the system?

Feature Flags are one of the tools that allow that.

With this article, other than providing some basics about Feature Flags, I want to suggest some ideas on how to implement them with a very powerful AWS service: AWS AppConfig.

This article comes with a companion demo app that is serverless and the frontend is built in C# with Blazor WASM.
It is a very basic (minimal) web app that contains a bunch of cake recipes and, clicking on each of them, it shows the list of the baking steps.
The source code is in this GitHub repository

Feature Flag

Feature Flags are also called Feature Toggles.
In this section, they are addressed as toggles, as they are referred to on Martin Fowler’s website.

Let’s now try to categorize the different types of flags so that, when it is time to use them, we can better understand their purpose and their lifespan.

Release Toggle

When developing a big or complex feature, there is the need to check the code into the main branch, to not have a hard-to-maintain, long-lasting branch, until the feature is ready.

The feature can therefore be released in production but stay silent behind the toggle.

This technique is actually what enables trunk-based development.

Experiment Toggle

Used for A/B testing. The example described in the introduction is a case for an experiment toggle. There is a need to better understand how people may react to the feature and plan the next moves based on their feedback.

This is also useful to flush out bugs that were not caught during development and testing before the official go-live.

Ops Toggle

Used to check on the performances of a new feature, understand how it weighs on the system and eventually adjusts the knobs to properly configure it.

Another option for this kind of toggle is when gradually retiring an existing feature.

Permission Toggle

Used when making a feature available to a limited/premium tier of users.

Similar to Experiment Toggle, however, Permission Toggles have a longer life span: they could stay for the entire life of the product.

Implementation

The easiest way to implement a flag is using an if statement.

if( featureIsEnabled("use-new-feature") ){
   return newListWithPagination();
   } else {
   return oldList();
   }
}

The featureIsEnabled function checks the value of the flag, probably reading it from a configuration file or a database table or, even better, a cloud provider’s service.

For DotNet applications, Microsoft provides the FeatureManagement library. It’s an abstraction built on top of IConfiguration and it has services that can be injected and also a tag helper to switch on and off sections of views. So there is no need for additional code to wrap the configuration and bring it where the flag has to be checked.

The demo app for this article does not use this library, it would have added too much complexity for such a basic website. However, in a real-world app, it could probably add more value.

AWS AppConfig Feature Flags

In the AWS cloud, the service that deals with environments is AWS System Manager and AWS AppConfig is its feature specific for managing configurations.

In May 2022, AWS announced the general availability of the new Feature Flags section in AWS AppConfig.

To configure the feature flags with this service is required to go through the creation of three pieces.

• First, create the application: that is the application that the flag’s status will affect or even a general namespace to group many flags.

• Second, create the environments: these are logical deployable groups. They can map to the traditional meaning of environment such as Dev, Test, Prod, Beta and even also, be component/platform-specific, like Web, Mobile and Backend.

• Third, define the configuration profiles: this is where the configuration is detailed and, of course, where the flags are defined.

Deployment Strategy

To deploy a set of flags, a deployment strategy must be created. This contains the breakdown of the steps that concur with the deployment.

Applications can be deployed in different ways regarding impact and nature.

It is possible to deploy the configuration linearly in 5 steps, gradually releasing it to a subset of the population, 20% each step. This allows us to monitor how the system is reacting to the changes throughout the entire process.

After the last step, there is the bake time: AWS AppConfig checks for alarms in AWS CloudWatch for a couple of minutes. If no alarm is reported in this window, the deployment is considered successful, otherwise, a rollback is performed.

It comes with some ready-to-use strategies:

• AppConfig.AllAtOnce to quickly deploy the configuration in one single shot [Quick]

• AppConfig.Linear50PercentEvery30Seconds every 30 seconds, the configuration is released to 50% of the invocations. In the end, the bake time is 1 minute [Test/Dev]

• AppConfig.Canary10Percent20Minutes the entire deployment happens in 20 minutes however, it is an exponential strategy: the growth factor is 10% at each step over 20 minutes and the bake time is 10 minutes [Prod]

This service is quite flexible and also allows for custom deployment strategies, to tune each step based on the application’s specific needs.

Demo

Let’s now create a configuration for our app in AWS AppConfig

In the AWS web console, we search for AWS AppConfig

We start by clicking the Create Application button.

Create a Feature Flag

To create Feature Flags, leave the “Feature Flag” radio button active.
Freeform Configuration is for the more traditional kind of configuration.

Let’s create our first flag configuration: RecipeFlag which will contain the flags used by the app.

The first flag is the one to switch from the full-page visualization of the steps of the recipe to the wizard-style visualization.

This is going to be initially an experimental flag, to see how the customers react to the pagination feature.

Let's click on the Add new flag button

Fill in the name and description fields and also check the Short-term flag checkbox. This is to highlight that ideally, the wizard visualization will soon be turned on, right after adjustments from the users' feedback.

After the flag is created, for now, leave the switch off.

On with the flag that pilots the pagination of the recipe list. This will:

• switch ON and OFF the pagination

• provide the number of recipes to show on a single page

This is going to be initially an experimental flag, to see how the customers react to the pagination feature. However, later it can become an OPS flag, to tweak the pagination when there are many items to return and not stress too much the system.

While creating the flag, click on the Add new attribute button and add the data of the pageSize attribute.

The pageSize attribute is a numeric attribute and its value can span between a minimum of 5, to a maximum of 10. Right now its value is 5.

The maximum and minimum values are constraints that are applied when changing the value of the attribute.

At this point, there are two disabled and not published flags.

However, to store this configuration, we have to save it in a new version (Save new version button). Every time that a configuration changes, a new version has then to be created. In this way, it is easy to switch from one version to another as needed.

Let’s deploy this first version, by clicking on the Start Deployment button.
To initiate a deployment, we have to specify in which environment the release has to happen. If there is no environment available, it is possible to click on the Create Environment button to initiate the creation of a new one. In this demo, the environment name is Beta.
Then select the version to deploy and the strategy. Here we pick the preconfigured strategy AppConfig.Linear50PercentEvery30Seconds (Test/Demo). Then click on Start Deployment to trigger the deployment and look at its progress.

After about 30 seconds

After about one minute

After one minute of baking time

Now that the flag is deployed, we can move to the backend that will supply the flags to the web application.

Backend

All we need to do for this article is to have an HTTP endpoint to invoke to get the features’ configuration.

• HTTP endpoint on AWS API Gateway to return the configuration of the features

• AWS Lambda gets the feature flags from AWS AppConfig and maps them in a way that better matches the “language” of our application rather than using AWS terminology.

For this part of the demo app, we can use AWS SAM.

Here is the template for the provisioning of the resources

  BlazorAppConfigFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      Handler: "BlazorAppConfig.Function::BlazorAppConfig.
                Function.Function::FunctionHandler"
      Runtime: dotnet6
      CodeUri: ""
      Description: Retrieves Recipe Flag configuration
      MemorySize: 256
      Timeout: 600
      Layers:
        - !Sub 
           arn:aws:lambda:${AWS::Region}:
           434848589818:layer:AWS-AppConfig-Extension:82
      Policies:        
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - appconfig:GetLatestConfiguration
                - appconfig:StartConfigurationSession
              Resource: !Sub 'arn:aws:appconfig:${AWS::Region}:${AWS::AccountId}:application/${AppConfigAppId}/environment/${AppConfigAppEnvironmentId}/configuration/${AppConfigtId}'
      Environment:
        Variables:
          RECIPE_CONFIG_PATH: !Ref RecipeConfigPath
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Path: /config
            Method: get
            ApiId: !Ref HttpApi

As you can see, there is an AWS Lambda function (BlazorAppConfigFunction) that is triggered by a GET invocation of an HttpApi endpoint (/config).

Here are some things to point out.

Permissions

To successfully interact with AWS AppConfig, our function needs adequate permissions (check the Policies section in the YAML snippet).

In particular, the two actions that should be enabled to read the configuration are:

• StartConfigurationSession to initialize a session with AppConfig

• GetLatestConfiguration to retrieve the latest available configuration

The Lambda must have access to the exact resource that it is dealing with: the app configuration, for the environment that we previously created (in this demo I called it Beta).

To achieve this we have to compose the resource’s ARN address.

• AppConfigAppId: is the Application ID and can be found on the main page of the configuration

• AppConfigEnvironmentId: is the Environment ID and is in the environment’s tab inside the configuration of the application

• AppConfigId: is the Configuration profile ID and it is also in the main tab inside the configuration

The ARN for my configuration in my account is

arn:aws:appconfig:eu-west-1:${AWS::AccountId}:application/oygjinns/environment/uptt1ta/configuration/flhu164

AWS-AppConfig-Extension

AWS-AppConfig-Extension is a Lambda Layer provided by AWS with the intent of simplifying the interaction with AWS AppConfig from AWS Lambda.

It adds a companion process that runs in parallel with the function and takes care of getting the configuration from the service (running the StartConfigurationSession and GetLatestConfiguration on its own), adding a caching layer and auto-updating the values.

The communication between our function and this process is done with simple HTTP invocations on port 2772, passing the address of the configuration

/applications/{application configuration name}/environments/{environment}/configurations/{configuration-name}

In our demo app it is:

/applications/appconfig-blazor/environments/Beta/configurations/RecipeFlag

In my implementation, I am passing this parameter as an environment variable, so that I can inject it through the SAM template.

Here is the C# code for the interaction with the extension

private async Task<FeatureResponse?> GetConfiguration()
{
  using(var client = new HttpClient())
  {
      client.BaseAddress = new Uri("http://localhost:2772");

      var response = await client.GetAsync(BlazorConfigPath);

      if(response.IsSuccessStatusCode)
      {
          return await response.Content.ReadFromJsonAsync<FeatureResponse>();
      }
  }
  return null;
}

This is not exactly a properly coded solution, but it is enough to have an easy-to-follow code sample. In the GitHub repository, there is a better-architected implementation.

Done with the coding, we can prepare our function for the deployment. First, we have to build it.

sam build

and then we can run the deploy

sam deploy -–guided

The –-guided flag helps when running the first deployment so that you can provide each parameter, one by one.

Hopefully, everything worked fine and invoking the HTTP that has been created, the API will return:

{
   "StepsListWizard": {
     "Enabled": false
   },
   "PagedRecipesList": {
     "PageSize": 0,
     "Enabled": false
   }
}

Note that PageSize is 0. It is not a value included in the range previously specified. This is because the flag that it belongs to is not enabled, so it gets the default numeric value.

The AWS-AppConfig-Extension is very handy and it is possible to pilot its behavior by changing its specific environment variables. Some of the customizations are: enabling prefetching of the configuration or tweaking the refresh frequency.

Applying changes

Whenever a change in a flag is applied it has then to be published to make it available to the client apps.
Let’s try switching ON the pagination flag

And also changing the value of the pageSize attribute to 8.

Confirm the changes, Save the new configuration and then Publish the newly created version.

When the publishing is finished, the HTTP API endpoint will return this message

{
   "StepsListWizard": {
     "Enabled": false
   },
   "PagedRecipesList": {
     "PageSize": 8,
     "Enabled": true
   }
}

As you can see, now the pagination flag (PagedRecipesList) is ON and the page size has the value that we just set.

Demo WebApp

Here we are focusing on a possible way to switch ON and OFF the features of the UI based on the features configuration that is supplied by the HTTP API that we just created.

Blazor’s component model plays quite nicely in this scenario: it eases the rendering of a certain component when the feature is ON, and another when it is OFF.

Practically speaking, for showing the list of recipes on the home page, there are two options:

• pagination feature OFF: show the entire list

• pagination feature ON: show the paginated list of recipes, with the provided page size

Here is the code in the Razor Component

@if (StorageConfig.FeaturesConfiguration.IsPagedRecipesEnabled)
{
   <PagedRecipesListComponent 
           RecipesList="Recipes"
           PageSize="StorageConfig.FeaturesConfiguration.RecipesListPageSize"         
    />
}
else
{
   <RecipesListComponent RecipesList="Recipes" />
}

Looking at the code of the component that implements the pagination feature (PagedRecipesListComponent), you can see that it is also in itself using RecipesListComponent, which is just a component that brings a list of recipes and renders it.

Composition is a very powerful concept that we should never forget.

Sometimes, there is the need to inject the child component that deals with some rendering logic. To do that it is possible to use templated components.

Pagination Feature OFF

Pagination Feature ON

Wizard Recipe Steps Feature OFF

Wizard Recipe Steps Feature ON

As you can see, because this last feature is still experimental and we would like to have feedback from the customer, we can show a banner with the link to the survey page.

Conclusions

The Feature Flag functionality provided by AWS AppConfig is very handy.

With just a couple of clicks on the web console, you can create, update and monitor our flags. The deployment strategy can also be customized based on the application’s needs.

Other than that, it is already a system, separated from the main application, that deals with flags. The AWS AppConfig UI could become the company-wide system to manage feature flags. Even more important, the user that can get access to this section of the AWS web console can be part of the OPS team or people that have a specific set of permissions. All the operations done can be monitored.
The entire service has its own CLI commands and integration with CloudFormation and indeed with the CDK. This means that changes can be dealt with using good Infrastructure as Code practices.

It may not be as powerful and feature-rich as a third-party product that does just that; however, what it can offer is already enough for many use cases.

Alternatives

When working with Blazor Server or traditional ASP.NET applications, using the FeatureManagement library is probably the natural choice to start with.

This library is built on top of IConfiguration, so it can read the flags configuration from the appsettings.json file or even directly from AWS AppConfig, using the NuGet AWSSDK.AppConfig.

Cold Start

One of the critical points when using DotNet with AWS Lambda is the Cold Start: the time that is required to initialize the container instance that runs the code of our function.

The Lambda, the backend of the demo app, has a total execution time of 1003 milliseconds, cold start included. Further consequent execution, which probably finds the container ready, will take a few milliseconds less, also, they will get the values of the flags from the cache.

A web application can preload the flags during startup, while it is already loading some other backend data. Then it can periodically pool for flag updates in the background.

At the end of the day, it all depends on what you are using the flags for and how you are architecting your application.

Pricing

The costs for AWS AppConfig are included in those for AWS System Manager. So you are paying for each configuration item and for each operation (GET, Describe, Update) done through the API.

AWS Lambda is billed on execution time. The cold start is not included in the billing. Also, this service has a great initial free tier with 1 million free requests per month.

AWS API Gateway is also billed on a per-request base.

Generally speaking, when dealing with cloud services, it is always necessary to have some initial understanding of the app and how it will be used. Then constantly monitor how it behaves to tweak the configuration and check and minimize the spending.

Further Readings

The first thing to read, to get good basics on Feature Flags (or Feature Toggles) is the article published on Martin Fowler’s website. For developers, it is crucial to have a good understanding of what is suggested in the section where the implementation techniques are described.

This article is inspired by this AWS Workshop.

Code for this article

The code for this article can be found in this GitHub repository.

In the iac folder, there are two additional alternatives to the AWS SAM template that you saw here.

There is a SAM template that has the entire configuration and feature flags included.

It is possible to use it to have the entire backend configured and running.

Similarly, there is the corresponding CDK project with the code to provision the entire infrastructure already configured.

Demo App Disclaimer

The web application contains in its code much of the logic regarding the features.

Ideally, each feature may have a different target, based on the architecture of the app.

Indeed, the page size is something that can be dealt with in the backend, so that the web frontend just renders the items that come from the HTTP API.