Expose additional Kafka target configuration options
To accommodate high frequency messages to Apache Kafka brokers, additional tuning of Kafka producers can be necessary to prevent high CPU and Disk I/O load. The PI Integrator should expose these parameters to allow users the ability to adjust their system as necessary. The following are common parameters that would allow for this:
- Batch Size (batch.size) - Linger time (linger.msg) - Buffer memory (buffer.memory) - LZ4 compression
Additionally, many projects may require that acknowledgements be sent back to the producer to confirm that they were properly received. The PI Integrator should provide the option as for whether or not it will wait for an acknowledgement to be sent back and retry if one was not received. This could be done with the "acks" producer configuration option.
Ting-Ying Chung commented