New connections from the ip address are dropped if the limit is reached. . If FIPS mode is enabled, broker listener security protocols, TLS versions and cipher suites will be validated based on FIPS compliance requirement. Override picking an S3 endpoint. Currently applies only to OAUTHBEARER. You can find code samples for the consumer in different languages in these guides. The default is TLSv1.3 when running with Java 11 or newer, TLSv1.2 otherwise. The default is TLSv1.2,TLSv1.3 when running with Java 11 or newer, TLSv1.2 otherwise. In the event that the JWT includes a kid header value that isnt in the JWKS file, the broker will reject the JWT and authentication will fail. This configuration is ignored if log.message.timestamp.type=LogAppendTime.The maximum timestamp difference allowed should be no greater than log.retention.ms to avoid unnecessarily frequent log rolling. The number of samples maintained to compute metrics. Set client to use TLS when connecting to ZooKeeper. The old secret that was used for encoding dynamically configured passwords. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting. Broker Configs Topic Configs Producer Configs Consumer Configs Kafka Streams Configs AdminClient Configs Kafka Connect Configs The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. The create topic policy class that should be used for validation. The controller would trigger a leader balance if it goes above this value per broker. Frequency at which tiered objects cleanup is run for deleted topics. To learn more about topics in Kafka, see the Topics module - Apache Kafka 101 and Kafka Internals free courses. The metrics polling interval (in seconds) which can be used in kafka.metrics.reporters implementations. This thread pool is also used to garbage collect data in tiered storage that has been deleted. openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 The maximum allowed session timeout for registered consumers. Setting this configuration to true allows the SASL authentication to attempt to perform authentication asynchronously. -1 means that broker failures will not trigger balancing actions, Controls what causes the Confluent DataBalancer to start rebalance operations. This config specifies the upper capacity limit for producer incoming bytes per second per broker. Examples: RACK1, us-east-1d. In Linux, you may also need to configure somaxconn and tcp_max_syn_backlog kernel parameters accordingly to make the configuration takes effect. The URL for the OAuth/OIDC identity provider. Valid policies are: delete and compact, The maximum eligible segments that can be deleted during every check. A long value representing the upper bound (bytes/sec) on throughput for cluster link replication. The SO_SNDBUF buffer of the socket server sockets. This will be used in rack aware replication assignment for fault tolerance. . Rack of the broker. If the listener name is not a security protocol, listener.security.protocol.map must also be set. An explicit value overrides any true or false value set via the zookeeper.ssl.hostnameVerification system property (note the different name and values; true implies https and false implies blank). If this property is not specified, the S3 client will use the DefaultAWSCredentialsProviderChain to locate the credentials. Enables auto leader balancing. If the key is not set or set to empty string, brokers will disable the delegation token support. In the latest message format version, records are always grouped into batches for efficiency. Specifies the enabled cipher suites to be used in ZooKeeper TLS negotiation (csv). Note that this configuration is ignored if an extension of KafkaPrincipalBuilder is provided by the principal.builder.class configuration. By default, principal names of the form {username}/{hostname}@{REALM} are mapped to {username}. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. This configuration accepts the standard compression codecs (gzip, snappy, lz4, zstd). For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler, The fully qualified name of a class that implements the Login interface. Configuration names can optionally be prefixed with listener prefix and SASL mechanism name in lower-case. The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name. This must be enabled before tiering could be enabled by using confluent.tier.enable property. The iteration count used for encoding dynamically configured passwords. The amount of time to sleep when there are no logs to clean, The total memory used for log deduplication across all cleaner threads. For example, listener.name.sasl_ssl.plain.sasl.server.callback.handler.class=com.example.CustomPlainCallbackHandler. This config determines the amount of time to wait before retrying. Currently applies only to OAUTHBEARER. The number of samples to retain in memory for alter log dirs replication quotas, The time span of each sample for alter log dirs replication quotas. The maximum number of pending connections on the socket. If log.message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. Specifies whether to enable Online Certificate Status Protocol in the ZooKeeper TLS protocols. For example, read_committed consumers rely on reading transaction markers in order to detect the boundaries of each transaction. The backoff increases exponentially for each consecutive failure up to confluent.replica.fetch.backoff.max.ms. The password of the private key in the key store file or the PEM key specified in ssl.keystore.key. This can be useful in some cases where external load balancers are used. If the log.cleaner.max.compaction.lag.ms or the log.cleaner.min.compaction.lag.ms configurations are also specified, then the log compactor considers the log eligible for compaction as soon as either: (i) the dirty ratio threshold has been met and the log has had dirty (uncompacted) records for at least the log.cleaner.min.compaction.lag.ms duration, or (ii) if the log has had dirty (uncompacted) records for at most the log.cleaner.max.compaction.lag.ms period. If no principal builder is defined, the default behavior depends on the security protocol in use. The default value is 3600000. The Kafka node will generate a snapshot when either the maximum time interval is reached or the maximum bytes limit is reached. Default value 1 day. If it is not set, the metadata log is placed in the first log directory from log.dirs. This class should implement the org.apache.kafka.server.policy.CreateClusterLinkPolicy interface. Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example, listener.name.internal.max.connections. The amount of time the client will wait for the socket connection to be established. The frequency with which the partition rebalance check is triggered by the controller. The number of threads to use for various background processing tasks. Specify hostname as 0.0.0.0 to bind to all interfaces. The JWT will be inspected for the standard OAuth iss claim and if this value is set, the broker will match it exactly against what is in the JWTs iss claim. The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. If the leader imbalance exceeds leader.imbalance.per.broker.percentage, leader rebalance to the preferred leader for partitions is triggered. Name of listener used for communication between brokers. Allowed values in recent JVMs are TLSv1.2 and TLSv1.3. A list of classes to use as metrics reporters. A value of zero disables time based snapshot generation. cleanup.policy The number of threads that the server uses for processing requests, which may include disk I/O, The number of threads that the server uses for receiving requests from the network and sending responses to the network, The number of threads per data directory to be used for log recovery at startup and flushing at shutdown, The number of threads that can move replicas between log directories, which may include disk I/O. Instead, everything could be configured via environment variables, and we will store Kafka's. The fully qualified name of a class that implements the ClientQuotaCallback interface, which is used to determine quota limits applied to client requests. The SecureRandom PRNG implementation to use for SSL cryptography operations. The Confluent DataBalancer will attempt to keep incoming data throughput below this limit. This avoids repeatedly sending requests in a tight loop under some failure scenarios. personal data will be processed in accordance with our Privacy Policy. If the config for the listener name is not set, the config will fallback to the generic config (i.e. Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation, Be the first to get updates and new content, Confluent Platform Configuration Reference, metadata.log.max.record.bytes.between.snapshots, hostname1:port1,hostname2:port2,hostname3:port3, hostname1:port1,hostname2:port2,hostname3:port3/chroot/path, listener.name.internal.max.connection.creation.rate, org.apache.zookeeper.ClientCnxnSocketNetty, zookeeper.ssl.endpoint.identification.algorithm, org.apache.kafka.server.policy.AlterConfigPolicy, org.apache.kafka.server.authorizer.Authorizer, org.apache.kafka.common.metrics.JmxReporter, confluent.tier.fenced.segment.delete.delay.ms, org.apache.kafka.server.policy.CreateClusterLinkPolicy, org.apache.kafka.server.policy.CreateTopicPolicy, listener.name.internal.ssl.keystore.location, org.apache.kafka.common.metrics.MetricsReporter, org.apache.kafka.common.security.auth.SecurityProviderCreator, Deploy Hybrid Confluent Platform and Cloud Environment, Tutorial: Introduction to Streaming Application Development, Clickstream Data Analysis Pipeline Using ksqlDB, Replicator Schema Translation Example for Confluent Platform, DevOps for Kafka with Kubernetes and GitOps, Case Study: Kafka Connect management with GitOps, Configure Automatic Startup and Monitoring, Migrate Confluent Cloud ksqlDB applications, Connect ksqlDB to Confluent Control Center, Connect Confluent Platform Components to Confluent Cloud, Pipelining with Kafka Connect and Kafka Streams, Tutorial: Moving Data In and Out of Kafka, Single Message Transforms for Confluent Platform, Configuring Kafka Client Authentication with LDAP, Authorization using Role-Based Access Control, Tutorial: Group-Based Authorization Using LDAP, Configure Audit Logs using the Confluent CLI, Configure MDS to Manage Centralized Audit Logs, Configure Audit Logs using the Properties File, Log in to Control Center when RBAC enabled, Create Hybrid Cloud and Bridge-to-Cloud Deployments, Transition Standard Active-Passive Data Centers to a Multi-Region Stretched Cluster, Replicator for Multi-Datacenter Replication, Tutorial: Replicating Data Across Clusters, Installing and Configuring Control Center, Check Control Center Version and Enable Auto-Update, Connecting Control Center to Confluent Cloud, Confluent Monitoring Interceptors in Control Center, Docker Configuration Parameters for Confluent Platform, Configure a Multi-Node Environment with Docker, Confluent Platform Metadata Service (MDS), Configure the Confluent Platform Metadata Service (MDS), Configure Confluent Platform Components to Communicate with MDS over TLS/SSL, Configure mTLS Authentication and RBAC for Kafka Brokers, Configure Kerberos Authentication for Brokers Running MDS, Configure LDAP Group-Based Authorization for MDS, kafka.common.TopicPlacement$TopicPlacementValidator@72bc6553, kafka.tier.backupobjectlifecycle.RetentionToBackupConfigValidator$@66982506, [uncompressed, zstd, lz4, snappy, gzip, producer], [0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0, 3.2-IV0, 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0], org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder, [PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL], PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL.