Oracle GoldenGate Adapter/Handler for KafkaConnect (opensource) is released on 07/Jul/2016.
Summary
Kafka Connect is a tool for streaming data between Apache Kafka and other
data systems in a scalable and reliable way. Kafka Connect can ingest entire databases or collect metrics from all your
application servers into Kafka topics, making the data available for stream processing with low
latency. An export connector can deliver data from Kafka topics into secondary indexes like
Elasticsearch or into batch systems such as Hadoop for offline analysis. (Source: Confluent Website)
This Kafka Connect handler lets you to integrate using the Kafka Connect API which can be managed using Confluent Platform. The Kafka Connect Handler takes change data capture operations from a source trail file and generates data structs (org.apache.kafka.connect.data.Struct) as well as the associated schemas (org.apache.kafka.connect.data.Schema). The data structs are serialized via configured converters then enqueued onto Kafka topics. The topic name used corresponds to the fully qualified source table name as obtained from the GoldenGate trail file. Individual operations consist of inserts, updates, and delete operations executed on the source RDBMS. Insert and update operation data include the after change data. Delete operations include the before change data. A primary key update is a special case for an update where one or more of the primary key(s) is/are changed. The primary key update represents a special case in that without the before image data it is not possible to determine what row is actually changing when only in possession of the after change data. The default behavior of a primary key update is to ABEND in the Kafka Connect formatter. However, the formatter can be configured to simply treat these operations as regular updates or to treat them as deletes and then an insert which is the closest big data modeling to the substance of the transaction.
Difference between official GoldenGate Kafka Handler and KafkaConnect Handler
The Kafka Handler officially released in Oracle GoldenGate for Big Data 12.2.0.1.x is slightly different in functionality than the Kafka Connect Handler/Formatter included in this opensource component.
- The officially released Kafka Handler interfaces with pluggable formatters to output the data to Kafka in XML, JSON, Avro, or delimited text format.
- The Kafka Connect Handler/Formatter builds up Kafka Connect Schemas and Structs. It relies on the Kafka Connect framework to perform the serialization using the Kafka Connect converters before delivering the data to topic.
Compatibility Matrix
- Oracle GoldenGate for Big Data 12.2.0.1.1
- Confluent.io Kafka/Kafka Connect 0.9.0.1-cp
Download Location
It is available for download at java.net