{"id":36853,"title":"Connecting to an Apache Kafka on Heroku Cluster","updated_at":"2026-06-30T18:09:34.915Z","slug":"connecting-to-kafka-on-heroku","content":"[Apache Kafka on Heroku](kafka-on-heroku) is accessible from many different programming languages and frameworks.\r\n\r\n## Apache Kafka on Heroku \u0026 SSL\r\n\r\n\u003enote\r\n\u003eConnecting to Apache Kafka on Heroku requires setting the client’s SSL configuration to not verify the server hostname. See [Language Support](#language-support) for more information on how to set the SSL configuration depending on the language.\r\n\r\nAll connections to Kafka support SSL encryption and authentication. If you provision the cluster in a Private Space, you can connect via plaintext. Shield Spaces don’t support plaintext connections.\r\n\r\nConnecting over SSL means all traffic is encrypted and authenticated via an SSL client certificate.\r\n\r\nTo connect over SSL, use the following environment variables:\r\n\r\n\u003enote\r\n\u003eAs with all Heroku add-ons, important [environment and configuration variables can change](add-ons#config-var-values-can-change). It’s important to design your app to handle updates to these values, especially if systems outside of Heroku access these resources.\r\n\r\n- `KAFKA_URL`: A comma-separated list of SSL URLs to the Kafka brokers making up the cluster.\r\n- `KAFKA_TRUSTED_CERT`: The brokers' SSL certificate (in PEM format), to check that you're connecting to the right servers.\r\n- `KAFKA_CLIENT_CERT`:  The required client certificate (in PEM format) to authenticate clients against the broker.\r\n- `KAFKA_CLIENT_CERT_KEY`: The required client certificate key (in PEM format) to authenticate clients against the broker.\r\n\r\nKafka clusters require authenticating using the provided client certificate. Kafka denies any requests not using the client certificate.\r\n\r\nConnecting to an Apache Kafka on Heroku cluster requires disabling server hostname validation. Adjust your client SSL configuration to set `ssl.endpoint.identification.algorithm` to an empty string. [Some Kafka clients](#language-support) provide their own specific option for this setting.\r\n\r\n\u003e note\r\n\u003e Only `basic` plans provide `KAFKA_PREFIX`. If you're using a `basic` plan, see [Multi-Tenant Apache Kafka on Heroku](https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku#differences-to-dedicated-kafka-plans).\r\n\r\n## Rotating Credentials\r\n\r\nIt's a good security practice to rotate the credentials for important services regularly. To rotate your cluster’s credentials, use the `heroku kafka:credentials --reset` command.\r\n\r\n```term\r\n$ heroku kafka:credentials HEROKU_KAFKA_GRAY_URL --reset -a example-app\r\nResetting credentials for kafka-animated-39618\r\n```\r\n\r\nThis command executes these steps: \r\n\r\n1. Generate a new credential for your cluster.\r\n1. Existing topics and consumer groups receive new ACLs to allow the new client certificate credential.\r\n1. When the new credential is ready, we update the related config vars on your Heroku app.\r\n1. For 5 minutes, both the old client certificate and new client certificate remain valid. This period gives time for the new client certificate to cycle into your app.\r\n1. After 5 minutes, the old client certificate credential expires and is no longer valid.\r\n\r\n## Language Support\r\n\r\nThere are many client libraries for Kafka across many languages. The following listed are libraries that we either helped to improve, or have up-to-date support for Kafka features, including critical capabilities like support for SSL.\r\n\r\n\u003e callout\r\n\u003e Basic multi-tenant Kafka plans require a prefix on topics and consumer groups. See the [differences between multi-tenant and dedicated](https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku#differences-to-dedicated-kafka-plans). When integrating Kafka consumers, make sure to prefix topics and consumer groups with the value of the `KAFKA_PREFIX` environment variable. Otherwise, Kafka doesn’t receive the messages, and errors like `Broker: Topic authorization failed` or `Broker: Group authorization failed` can appear in Kafka debug events.\r\n\r\n#### Using Kafka in Ruby Applications\r\n\r\nWe recommend using [`rdkafka-ruby`](https://github.com/karafka/rdkafka-ruby) when connecting to Kafka from Ruby. \r\n\r\n[This example](https://github.com/heroku/heroku-kafka-demo-ruby) shows how to write and consume messages in Ruby.\r\n\r\nWhen using `rdkafka-ruby`, you must specify `\"enable.ssl.certificate.verification\" =\u003e false` in your client configuration to connect to Apache Kafka on Heroku.\r\n\r\n#### Using Kafka in Java Applications\r\n\r\nWe recommend using [`kafka-clients`](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients) when connecting to Kafka from Java. See the [Kafka project documentation](https://kafka.apache.org/documentation.html#api) for more information.\r\n\r\n[This example](https://github.com/heroku/heroku-kafka-demo-java) shows how to write and consume messages in Java. [This section of the demo app](https://github.com/heroku/heroku-kafka-demo-java/blob/main/src/main/java/com/heroku/kafka/demo/KafkaConfig.java#L46-L62) provides a good example of working with the TrustStore and KeyStore for Java Virtual Machine (JVM) apps.\r\n\r\nOn recent versions of the Java Kafka clients, you must set `ssl.endpoint.identification.algorithm` to an empty string to disable server hostname validation. For example, `ssl.endpoint.identification.algorithm=`.\r\n\r\n#### Using Kafka in Go Applications\r\n\r\nWe recommend using [`sarama`](https://github.com/IBM/sarama) when connecting to Kafka from Go.\r\n\r\n[This example](https://github.com/heroku/heroku-kafka-demo-go) shows how to write and consume messages in Go. \r\n\r\nDue to the strictness of the `tls` package in Go, when using the `sarama` client, you must set `InsecureSkipVerify` to `true` to disable server hostname validation.\r\n\r\n#### Using Kafka in Python Applications\r\n\r\nWe recommend using [`confluent-kafka-python`](https://github.com/confluentinc/confluent-kafka-python) when connecting to Kafka in Python. This client library wraps the [C/C++ Kafka library](https://github.com/confluentinc/librdkafka).\r\n\r\nWe also recommend the [`kafka-python`](https://github.com/dpkp/kafka-python) library, especially for scenarios where wrapping the C/C++ libraries is less than ideal. Heroku has created the [`kafka-helper`](https://github.com/heroku/kafka-helper) library to make [`kafka-python`](https://github.com/dpkp/kafka-python) easier to use. To disable server hostname validation with `kafka-python`, set [`ssl_check_hostname`](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaClient.html) to `False`.\r\n\r\nThe [`aiokafka` client is based on the `kafka-python` library](https://aiokafka.readthedocs.io/en/stable/). To disable server hostname when using this client, set the [SSL context `check_hostname` option](https://aiokafka.readthedocs.io/en/stable/api.html#helpers) to `False`.\r\n\r\n#### Using Kafka in Node.js Applications\r\n\r\nWe recommend using [`kafkajs`](https://github.com/tulios/kafkajs) when connecting to Kafka from Node.js apps.\r\n\r\nTo disable server hostname verification with `kafkajs`, set [`ssl.rejectUnauthorized` to `false`](https://kafka.js.org/docs/configuration#ssl) in your client’s SSL configuration.\r\n\r\n#### Using Kafka in PHP Applications\r\n\r\nThe [`rdkafka` extension](https://github.com/arnaud-lb/php-rdkafka) for PHP [is available on Heroku](php-support#extensions). It provides bindings for the [`librdkafka`](https://github.com/confluentinc/librdkafka) library and supports SSL.\r\n\r\nSet `ssl.endpoint.identification.algorithm` to an empty string to disable server hostname validation, following the [`librdkafka` configuration properties](https://github.com/confluentinc/librdkafka/blob/616fb6e8074e82e0302f1c7d2896d7e577f8eb9b/CONFIGURATION.md).\r\n\r\n#### Using Kafka in Other Languages or Frameworks\r\n\r\nThe [Confluence Client wiki](https://cwiki.apache.org/confluence/display/KAFKA/Clients) lists clients and code examples for other languages and frameworks.","html_content":"\u003cp\u003e\u003ca href=\"https://devcenter.heroku.com/articles/kafka-on-heroku\"\u003eApache Kafka on Heroku\u003c/a\u003e is accessible from many different programming languages and frameworks.\u003c/p\u003e\n\u003ch2 id=\"apache-kafka-on-heroku-ssl\"\u003eApache Kafka on Heroku \u0026amp; SSL\u003c/h2\u003e\n\u003cdiv class=\"note\"\u003e\n\u003cp\u003eConnecting to Apache Kafka on Heroku requires setting the client’s SSL configuration to not verify the server hostname. See \u003ca href=\"#language-support\"\u003eLanguage Support\u003c/a\u003e for more information on how to set the SSL configuration depending on the language.\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003eAll connections to Kafka support SSL encryption and authentication. If you provision the cluster in a Private Space, you can connect via plaintext. Shield Spaces don’t support plaintext connections.\u003c/p\u003e\n\u003cp\u003eConnecting over SSL means all traffic is encrypted and authenticated via an SSL client certificate.\u003c/p\u003e\n\u003cp\u003eTo connect over SSL, use the following environment variables:\u003c/p\u003e\n\u003cdiv class=\"note\"\u003e\n\u003cp\u003eAs with all Heroku add-ons, important \u003ca href=\"https://devcenter.heroku.com/articles/add-ons#config-var-values-can-change\"\u003eenvironment and configuration variables can change\u003c/a\u003e. It’s important to design your app to handle updates to these values, especially if systems outside of Heroku access these resources.\u003c/p\u003e\n\u003c/div\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eKAFKA_URL\u003c/code\u003e: A comma-separated list of SSL URLs to the Kafka brokers making up the cluster.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eKAFKA_TRUSTED_CERT\u003c/code\u003e: The brokers’ SSL certificate (in PEM format), to check that you’re connecting to the right servers.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eKAFKA_CLIENT_CERT\u003c/code\u003e:  The required client certificate (in PEM format) to authenticate clients against the broker.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eKAFKA_CLIENT_CERT_KEY\u003c/code\u003e: The required client certificate key (in PEM format) to authenticate clients against the broker.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eKafka clusters require authenticating using the provided client certificate. Kafka denies any requests not using the client certificate.\u003c/p\u003e\n\u003cp\u003eConnecting to an Apache Kafka on Heroku cluster requires disabling server hostname validation. Adjust your client SSL configuration to set \u003ccode\u003essl.endpoint.identification.algorithm\u003c/code\u003e to an empty string. \u003ca href=\"#language-support\"\u003eSome Kafka clients\u003c/a\u003e provide their own specific option for this setting.\u003c/p\u003e\n\u003cdiv class=\"note\"\u003e\n\u003cp\u003eOnly \u003ccode\u003ebasic\u003c/code\u003e plans provide \u003ccode\u003eKAFKA_PREFIX\u003c/code\u003e. If you’re using a \u003ccode\u003ebasic\u003c/code\u003e plan, see \u003ca href=\"https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku#differences-to-dedicated-kafka-plans\"\u003eMulti-Tenant Apache Kafka on Heroku\u003c/a\u003e.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch2 id=\"rotating-credentials\"\u003eRotating Credentials\u003c/h2\u003e\n\u003cp\u003eIt’s a good security practice to rotate the credentials for important services regularly. To rotate your cluster’s credentials, use the \u003ccode\u003eheroku kafka:credentials --reset\u003c/code\u003e command.\u003c/p\u003e\n\u003cpre class=\"language-term\"\u003e\u003ccode class=\"language-term\"\u003e$ heroku kafka:credentials HEROKU_KAFKA_GRAY_URL --reset -a example-app\nResetting credentials for kafka-animated-39618\n\u003c/code\u003e\u003c/pre\u003e\n\u003cp\u003eThis command executes these steps:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eGenerate a new credential for your cluster.\u003c/li\u003e\n\u003cli\u003eExisting topics and consumer groups receive new ACLs to allow the new client certificate credential.\u003c/li\u003e\n\u003cli\u003eWhen the new credential is ready, we update the related config vars on your Heroku app.\u003c/li\u003e\n\u003cli\u003eFor 5 minutes, both the old client certificate and new client certificate remain valid. This period gives time for the new client certificate to cycle into your app.\u003c/li\u003e\n\u003cli\u003eAfter 5 minutes, the old client certificate credential expires and is no longer valid.\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"language-support\"\u003eLanguage Support\u003c/h2\u003e\n\u003cp\u003eThere are many client libraries for Kafka across many languages. The following listed are libraries that we either helped to improve, or have up-to-date support for Kafka features, including critical capabilities like support for SSL.\u003c/p\u003e\n\u003cdiv class=\"callout\"\u003e\n\u003cp\u003eBasic multi-tenant Kafka plans require a prefix on topics and consumer groups. See the \u003ca href=\"https://devcenter.heroku.com/articles/multi-tenant-kafka-on-heroku#differences-to-dedicated-kafka-plans\"\u003edifferences between multi-tenant and dedicated\u003c/a\u003e. When integrating Kafka consumers, make sure to prefix topics and consumer groups with the value of the \u003ccode\u003eKAFKA_PREFIX\u003c/code\u003e environment variable. Otherwise, Kafka doesn’t receive the messages, and errors like \u003ccode\u003eBroker: Topic authorization failed\u003c/code\u003e or \u003ccode\u003eBroker: Group authorization failed\u003c/code\u003e can appear in Kafka debug events.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch4 id=\"using-kafka-in-ruby-applications\"\u003eUsing Kafka in Ruby Applications\u003c/h4\u003e\n\u003cp\u003eWe recommend using \u003ca href=\"https://github.com/karafka/rdkafka-ruby\"\u003e\u003ccode\u003erdkafka-ruby\u003c/code\u003e\u003c/a\u003e when connecting to Kafka from Ruby.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/heroku/heroku-kafka-demo-ruby\"\u003eThis example\u003c/a\u003e shows how to write and consume messages in Ruby.\u003c/p\u003e\n\u003cp\u003eWhen using \u003ccode\u003erdkafka-ruby\u003c/code\u003e, you must specify \u003ccode\u003e\"enable.ssl.certificate.verification\" =\u0026gt; false\u003c/code\u003e in your client configuration to connect to Apache Kafka on Heroku.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-java-applications\"\u003eUsing Kafka in Java Applications\u003c/h4\u003e\n\u003cp\u003eWe recommend using \u003ca href=\"https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients\"\u003e\u003ccode\u003ekafka-clients\u003c/code\u003e\u003c/a\u003e when connecting to Kafka from Java. See the \u003ca href=\"https://kafka.apache.org/documentation.html#api\"\u003eKafka project documentation\u003c/a\u003e for more information.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/heroku/heroku-kafka-demo-java\"\u003eThis example\u003c/a\u003e shows how to write and consume messages in Java. \u003ca href=\"https://github.com/heroku/heroku-kafka-demo-java/blob/main/src/main/java/com/heroku/kafka/demo/KafkaConfig.java#L46-L62\"\u003eThis section of the demo app\u003c/a\u003e provides a good example of working with the TrustStore and KeyStore for Java Virtual Machine (JVM) apps.\u003c/p\u003e\n\u003cp\u003eOn recent versions of the Java Kafka clients, you must set \u003ccode\u003essl.endpoint.identification.algorithm\u003c/code\u003e to an empty string to disable server hostname validation. For example, \u003ccode\u003essl.endpoint.identification.algorithm=\u003c/code\u003e.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-go-applications\"\u003eUsing Kafka in Go Applications\u003c/h4\u003e\n\u003cp\u003eWe recommend using \u003ca href=\"https://github.com/IBM/sarama\"\u003e\u003ccode\u003esarama\u003c/code\u003e\u003c/a\u003e when connecting to Kafka from Go.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/heroku/heroku-kafka-demo-go\"\u003eThis example\u003c/a\u003e shows how to write and consume messages in Go.\u003c/p\u003e\n\u003cp\u003eDue to the strictness of the \u003ccode\u003etls\u003c/code\u003e package in Go, when using the \u003ccode\u003esarama\u003c/code\u003e client, you must set \u003ccode\u003eInsecureSkipVerify\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e to disable server hostname validation.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-python-applications\"\u003eUsing Kafka in Python Applications\u003c/h4\u003e\n\u003cp\u003eWe recommend using \u003ca href=\"https://github.com/confluentinc/confluent-kafka-python\"\u003e\u003ccode\u003econfluent-kafka-python\u003c/code\u003e\u003c/a\u003e when connecting to Kafka in Python. This client library wraps the \u003ca href=\"https://github.com/confluentinc/librdkafka\"\u003eC/C++ Kafka library\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eWe also recommend the \u003ca href=\"https://github.com/dpkp/kafka-python\"\u003e\u003ccode\u003ekafka-python\u003c/code\u003e\u003c/a\u003e library, especially for scenarios where wrapping the C/C++ libraries is less than ideal. Heroku has created the \u003ca href=\"https://github.com/heroku/kafka-helper\"\u003e\u003ccode\u003ekafka-helper\u003c/code\u003e\u003c/a\u003e library to make \u003ca href=\"https://github.com/dpkp/kafka-python\"\u003e\u003ccode\u003ekafka-python\u003c/code\u003e\u003c/a\u003e easier to use. To disable server hostname validation with \u003ccode\u003ekafka-python\u003c/code\u003e, set \u003ca href=\"https://kafka-python.readthedocs.io/en/master/apidoc/KafkaClient.html\"\u003e\u003ccode\u003essl_check_hostname\u003c/code\u003e\u003c/a\u003e to \u003ccode\u003eFalse\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eThe \u003ca href=\"https://aiokafka.readthedocs.io/en/stable/\"\u003e\u003ccode\u003eaiokafka\u003c/code\u003e client is based on the \u003ccode\u003ekafka-python\u003c/code\u003e library\u003c/a\u003e. To disable server hostname when using this client, set the \u003ca href=\"https://aiokafka.readthedocs.io/en/stable/api.html#helpers\"\u003eSSL context \u003ccode\u003echeck_hostname\u003c/code\u003e option\u003c/a\u003e to \u003ccode\u003eFalse\u003c/code\u003e.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-node-js-applications\"\u003eUsing Kafka in Node.js Applications\u003c/h4\u003e\n\u003cp\u003eWe recommend using \u003ca href=\"https://github.com/tulios/kafkajs\"\u003e\u003ccode\u003ekafkajs\u003c/code\u003e\u003c/a\u003e when connecting to Kafka from Node.js apps.\u003c/p\u003e\n\u003cp\u003eTo disable server hostname verification with \u003ccode\u003ekafkajs\u003c/code\u003e, set \u003ca href=\"https://kafka.js.org/docs/configuration#ssl\"\u003e\u003ccode\u003essl.rejectUnauthorized\u003c/code\u003e to \u003ccode\u003efalse\u003c/code\u003e\u003c/a\u003e in your client’s SSL configuration.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-php-applications\"\u003eUsing Kafka in PHP Applications\u003c/h4\u003e\n\u003cp\u003eThe \u003ca href=\"https://github.com/arnaud-lb/php-rdkafka\"\u003e\u003ccode\u003erdkafka\u003c/code\u003e extension\u003c/a\u003e for PHP \u003ca href=\"https://devcenter.heroku.com/articles/php-support#extensions\"\u003eis available on Heroku\u003c/a\u003e. It provides bindings for the \u003ca href=\"https://github.com/confluentinc/librdkafka\"\u003e\u003ccode\u003elibrdkafka\u003c/code\u003e\u003c/a\u003e library and supports SSL.\u003c/p\u003e\n\u003cp\u003eSet \u003ccode\u003essl.endpoint.identification.algorithm\u003c/code\u003e to an empty string to disable server hostname validation, following the \u003ca href=\"https://github.com/confluentinc/librdkafka/blob/616fb6e8074e82e0302f1c7d2896d7e577f8eb9b/CONFIGURATION.md\"\u003e\u003ccode\u003elibrdkafka\u003c/code\u003e configuration properties\u003c/a\u003e.\u003c/p\u003e\n\u003ch4 id=\"using-kafka-in-other-languages-or-frameworks\"\u003eUsing Kafka in Other Languages or Frameworks\u003c/h4\u003e\n\u003cp\u003eThe \u003ca href=\"https://cwiki.apache.org/confluence/display/KAFKA/Clients\"\u003eConfluence Client wiki\u003c/a\u003e lists clients and code examples for other languages and frameworks.\u003c/p\u003e","published_at":"2026-06-30T18:09:34.895Z","meta_description":"Connect to Apache Kafka on Heroku from different programming languages and frameworks.","api_url":"https://devcenter.heroku.com/api/v1/articles/36853","view_url":"https://devcenter.heroku.com/articles/connecting-to-kafka-on-heroku","approved":false,"submitted_for_approval_at":null}