Apache Kafka on Heroku Add-on Migration

Last updated January 13, 2026

Scaling up or down between plan levels of Apache Kafka on Heroku is normally seamless and performed in-place. However, there are a few circumstances when actual data migration is required. This document provides an overview of those conditions and the applicable processes.

When to Migrate Between Kafka Add-ons?

There are 3 cases when migrating between Kafka add-ons is necessary:

  • You have a multi-tenant Kafka (Kafka Basic) add-on and you want to start using a dedicated Kafka add-on.
  • You have a dedicated Kafka add-on and you want to start using a multi-tenant Kafka (Kafka Basic) add-on.
  • You have a multi-tenant Kafka (Kafka Basic) add-on and you want to manually upgrade it to use a different Kafka version.

How Do I Handle the Migration?

In many scenarios, your application can enter a maintenance window and migrate to a new add-on without modifying your application’s code. In general, we recommend entering a maintenance window if you can, because it drastically reduces the complexity of the migration and doesn’t require significant changes to your app.

If your application can’t enter a maintenance window, you must migrate to a new add-on by double-writing to both sets of topics, and cutting over from the old add-on to the new one after the new add-on has received writes for a time period longer than your retention time.

How Do I Migrate Between Add-ons While in a Maintenance Window?

The high-level steps for migrating during a maintenance window are:

  1. Provision the new add-on with all relevant topics, as well as the consumer groups if applicable (only if the target is a Kafka Basic add-on).
  2. Enter your maintenance window.
  3. Stop your Kafka producers.
  4. Ensure your Kafka consumers are fully caught up.
  5. Switch over to the new add-on.
  6. Start your Kafka producers and consumers.
  7. Exit your maintenance window.
$ heroku addons:create heroku-kafka:basic-0 --as NEW_KAFKA -a example-app
$ heroku kafka:topics:create my-topic-name NEW_KAFKA -a example-app
$ heroku kafka:consumer-groups:create my-group-name KAFKA -a example-app

$ heroku ps:scale producer=0 -a example-app
# check consumers
heroku ps:scale consumer=0 -a example-app
heroku maintenance:on -a example-app
# kafka-parallel-2019
heroku addons:attach kafka-symmetrical-26061 --as OLD_KAFKA -a example-app

$ heroku addons:attach kafka-parallel-2019 --as KAFKA -a example-app
$ heroku ps:scale producer=1 consumer=1 -a example-app

$ heroku addons:destroy kafka-symmetrical-26061 -a example-app

How Do I Migrate Between Add-ons Without Entering a Maintenance Window?

The high-level steps for migrating without entering a maintenance window are:

  1. Prepare your app for double-write.
  2. Provision the new Kafka add-on with all relevant topics and consumer groups.
  3. Double-write to both the old and the new add-ons.
  4. Wait for the new add-on to contain the same historical data as the old add-on.
  5. Stop producing to the old add-on.
  6. Destroy the old add-on.

These steps are described in greater detail in this section.

Step 1: Prepare Your App for Double-Write

Your app must support two sets of Kafka config vars (one for each add-on).

This example uses KAFKA_URL, KAFKA_CLIENT_CERT, KAFKA_CLIENT_CERT_KEY, and KAFKA_TRUSTED_CERT for the old Kafka add-on before double-writing begins, and it uses them for the new Kafka add-on after double-writing begins.

This example uses OLD_KAFKA_URL, OLD_KAFKA_CLIENT_CERT, OLD_KAFKA_CLIENT_CERT_KEY and OLD_KAFKA_TRUSTED_CERT for the old Kafka add-on after double-writing begins. This set of config vars exists only while double-writing is taking place.

Two additional config vars are required, which tell producers and consumers where to write to and read from:

  • PRODUCER_ADDON_NAMES is used by producers to discover which add-ons to write to.
  • CONSUMER_ADDON_NAME is used by consumers to discover which add-on to read from.

You must add support to your app for:

  • Producing to all add-ons specified in PRODUCER_ADDON_NAMES
  • Consuming from the add-on specified in CONSUMER_ADDON_NAME

Consumers must handle duplicate messages idempotently. For more information, see the article on robust usage of Apache Kafka on Heroku.

Step 2: Provision the New Add-on

Before provisioning the new add-on, attach your existing Kafka add-on with a new name in preparation:

$ heroku addons:attach kafka-symmetrical-26061 --as OLD_KAFKA -a example-app
$ heroku addons:create heroku-kafka:basic-0 --as KAFKA -a example-app

Step 3: Create Topics and Consumer Groups on the New Add-on

Get a list of topics and consumer groups from your old add-on:

$ heroku kafka:topics OLD_KAFKA -a example-app
$ heroku kafka:consumer-groups OLD_KAFKA -a example-app

Now, you can create those topics as well as the consumer groups (if applicable, only for multi-tenant Kafka) on your new add-on:

$ heroku kafka:topics:create my-topic-name KAFKA -a example-app
$ heroku kafka:consumer-groups:create my-group-name KAFKA -a example-app

Step 4: Double-Write to the Old and New Add-ons

Your app must produce to both sets of topics and consume from the old add-on’s topics while the new add-on’s topics fill with data:

$ heroku config:set PRODUCER_ADDON_NAMES=OLD_KAFKA,KAFKA -a example-app
$ heroku config:set CONSUMER_ADDON_NAME=OLD_KAFKA -a example-app

Step 5: Wait for the New Add-on to Contain Enough Historical Data

After the new add-on has been receiving writes for longer than your retention time, both add-ons represent the same data. This means you can switch your consumers from the old add-on to the new add-on:

$ heroku config:set CONSUMER_ADDON_NAME=KAFKA -a example-app

Step 6: Stop Producing to the Old Add-on

When you’re comfortable consuming from the new add-on, you can stop producing to the old add-on:

$ heroku config:set PRODUCER_ADDON_NAMES=KAFKA -a example-app

Step 7: Destroy the Old Add-on

Because your app is no longer consuming from the old add-on, it’s safe to destroy it:

$ heroku addons:destroy OLD_KAFKA_URL -a example-app