Streaming Platform Engineer Kafka

Ideally resides within 2 hour drive of Richmond VA primarily remote

Technical skills and proficiency requirements

• Kafka internals - Consumer group protocol ISR controller election log compaction - production experience required not tutorial knowledge - Expert
• Confluent Cloud - CKU model cluster types RBAC metrics API managed connectors Confluent CLI - Expert
• Schema Registry - Avro schema design backward/forward compatibility breaking change detection Schema Registry ACLs - Expert
• Kafka Connect - Debezium CDC configuration BQ Sink connector GCS Sink connector SMT (Single Message Transforms) - Advanced
• AWS networking - VPC security groups IAM roles for Confluent AWS Secrets Manager Direct Connect/VPN concepts - Intermediate
• GCP networking - Private Service Connect Private Google Access VPC peering VPC-SC access levels for cross-cloud Ips - Intermediate
• Python - confluent-kafka-python library custom producer/consumer patterns offset management Schema Registry client - Advanced
• Terraform - Confluent Terraform provider GCP provider for network resources AWS provider for SA key storage - Intermediate
• Dual-cloud KMS - Confluent BYOK via AWS KMS GCP Cloud KMS - separate key estates rotation governance - Intermediate

Key responsibilities

• Kafka cluster ownership - Confluent Cloud provisioning CKU sizing network peering to GCP VPC billing monitoring
• Topic design - Naming conventions partition count retention policy compaction strategy per topic
• Schema Registry - Compatibility rule governance schema evolution reviews Avro/Protobuf standards enforcement
• Integration path ownership - Kafka Connect BQ Sink (Option 2) GCS Sink (Option 3) Dataflow KafkaIO auth (Option 4)
• SA key governance - GCP service account keys stored in Confluent - rotation schedule exposure monitoring
• Consumer group monitoring - Lag alerting per topic rebalancing tuning offset management
• Cross-cloud ops - Joint ownership of AWS-GCP boundary with Platform Infra Engineer - runbook incident response
• Egress monitoring - AWS CloudWatch bytes-out alerting Confluent metrics API daily egress cost review

Please provide years of experience:

• Kafka streaming
• Python
• Confluent
• Streaming
• AWS
• Platform Engineering
• Data Governance
• Cloud integration