WorkerSourceTask are being killed due to closed FM Server connections
This issue occurs frequently and is shown here at the example of the jdbc-FM_ZKM_Archive-Zeitpunkte-source
connector:
In kafka-ui the task is reported as Failed with sparse information on the cause:
The connector configuration:
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"validate.non.null": "false",
"connection.pool": "false",
"tasks.max": "1",
"transforms": "KEY",
"transforms.KEY.fields": "Zeitpunkt_ID",
"errors.deadletterqueue.context.headers.enable": "true",
"transforms.KEY.type": "org.apache.kafka.connect.transforms.ValueToKey",
"table.whitelist": "Zeitpunkte",
"mode": "bulk",
"topic.prefix": "private.cdc.zkm-archive.01.",
"errors.deadletterqueue.topic.name": "dlq.cdc.zkm-archive.01",
"poll.interval.ms": "300000",
"name": "jdbc-FM_ZKM_Archive-Zeitpunkte-source",
"errors.tolerance": "all",
"connection.url": "jdbc:filemaker://fms-wissen.zkm.de:2399/ZKM-Archive?user=****&password=****&SocketTimeout=10000",
"errors.log.enable": "true"
}
The according entry in the log:
kafka-connect | [2022-06-24 13:01:23,635] ERROR SQL exception while running query for table: BulkTableQuerier{table='"Zeitpunkte"', query='null', topicPrefix='private.cdc.zkm-archive.01.'} (io.confluent.connect.jdbc.source.JdbcSourceTask)
kafka-connect | com.filemaker.jdbc.FMSQLException: [FileMaker][FileMaker JDBC] Connection is closed. Fetch operation is not permitted.
kafka-connect | at com.filemaker.jdbc.FM_API.next(Unknown Source)
kafka-connect | at com.filemaker.jdbc.FM_API.next(Unknown Source)
kafka-connect | at com.filemaker.jdbc1.CommonJ1ResultSet.next(Unknown Source)
kafka-connect | at io.confluent.connect.jdbc.source.TableQuerier.next(TableQuerier.java:107)
kafka-connect | at io.confluent.connect.jdbc.source.BulkTableQuerier.next(BulkTableQuerier.java:39)
kafka-connect | at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:388)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:304)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
kafka-connect | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
kafka-connect | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
kafka-connect | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
kafka-connect | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
kafka-connect | at java.base/java.lang.Thread.run(Thread.java:829)
kafka-connect | [2022-06-24 13:01:23,637] INFO WorkerSourceTask{id=jdbc-FM_ZKM_Archive-Zeitpunkte-source-0} Either no records were produced by the task since the last offset commit, or every record has been filtered out by a transformation or dropped due to transformation or conversion errors. (org.apache.kafka.connect.runtime.WorkerSourceTask)
kafka-connect | [2022-06-24 13:01:23,639] WARN WorkerSourceTask{id=jdbc-FM_ZKM_Archive-Zeitpunkte-source-0} After being scheduled for shutdown, the orphan task threw an uncaught exception. A newer instance of this task might be already running (org.apache.kafka.connect.runtime.WorkerTask)
kafka-connect | java.lang.NullPointerException
kafka-connect | at com.filemaker.jdbc.MessageBlock.send(Unknown Source)
kafka-connect | at com.filemaker.jdbc.ProtocolMessage.send(Unknown Source)
kafka-connect | at com.filemaker.jdbc1.CommonJ1Connection.commit(Unknown Source)
kafka-connect | at io.confluent.connect.jdbc.source.TableQuerier.releaseLocksQuietly(TableQuerier.java:125)
kafka-connect | at io.confluent.connect.jdbc.source.TableQuerier.reset(TableQuerier.java:115)
kafka-connect | at io.confluent.connect.jdbc.source.BulkTableQuerier.reset(BulkTableQuerier.java:39)
kafka-connect | at io.confluent.connect.jdbc.source.JdbcSourceTask.resetAndRequeueHead(JdbcSourceTask.java:452)
kafka-connect | at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:425)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:304)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:248)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
kafka-connect | at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)
kafka-connect | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
kafka-connect | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
kafka-connect | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
kafka-connect | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
kafka-connect | at java.base/java.lang.Thread.run(Thread.java:829)
kafka-connect | [2022-06-24 13:01:23,639] INFO Stopping JDBC source task (io.confluent.connect.jdbc.source.JdbcSourceTask)
kafka-connect | [2022-06-24 13:01:23,639] INFO Closing resources for JDBC source task (io.confluent.connect.jdbc.source.JdbcSourceTask)
kafka-connect | [2022-06-24 13:01:23,640] INFO Closing connection #2 to FilemakerDialect (io.confluent.connect.jdbc.util.CachedConnectionProvider)
kafka-connect | [2022-06-24 13:01:23,640] INFO [Producer clientId=connector-producer-jdbc-FM_ZKM_Archive-Zeitpunkte-source-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
kafka-connect | [2022-06-24 13:01:23,640] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
kafka-connect | [2022-06-24 13:01:23,640] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
kafka-connect | [2022-06-24 13:01:23,641] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
kafka-connect | [2022-06-24 13:01:23,641] INFO App info kafka.producer for connector-producer-jdbc-FM_ZKM_Archive-Zeitpunkte-source-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
and right afterwards more WorkerSourceTask are being killed due to broken connections.