Data Integration Engineer

Mandatory Skills

Databricks & Python

Job Description

Core Technical Skills

Python - Advanced level (pandas requests boto3 libraries)

API Integration - REST APIs authentication (OAuth API keys) error handling

Cloud Platforms - AWS (S3 EC2 IAM) proficiency

Databricks/Spark - PySpark Delta Lake SQL optimization

Data Pipeline Tools - Airflow Databricks Workflows or equivalent orchestration

SQL - Complex queries window functions performance tuning

Security & Enterprise Knowledge

Understanding of security telemetry (logs alerts events metrics)

Familiarity with at least 2-3 security platforms (JAMF Intune Qualys CrowdStrike

Okta)

Experience with AWS CloudTrail or similar cloud audit logs

Knowledge of data security and compliance (encryption access controls)

Additional Competencies

Version control (Git)

CI/CD pipeline familiarity

Data quality and validation frameworks

Performance tuning and optimization mindset

Education Qualificaiton

BE BTech MCA

Roles & Responsibilities

Data Integration & Connectivity

Design and build connectors between enterprise security platforms (JAMF Intune Qualys

CrowdStrike Okta AWS CloudTrail) and Databricks

Establish secure API connections and authentication mechanisms with each source system

Implement data extraction logic using REST APIs SDKs and native connectors

Handle varying data formats and transformation requirements across heterogeneous sources

Data Pipeline Development

Develop automated Python scripts for continuous data ingestion

Build orchestration workflows using Apache Airflow Databricks Workflows or similar tools

Implement error handling retry mechanisms and monitoring for pipeline reliability

Ensure near real-time or scheduled data synchronization based on requirements

AWS S3 & Data Staging

Configure S3 bucket architecture and data partitioning strategy

Land raw security telemetry data in S3 with proper folder structures

Implement lifecycle policies and data retention strategies

Ensure proper encryption and access controls

Databricks Platform Configuration

Create and optimize Databricks tables