Must Have Matillion ETL Snowflake Python
Job Details:
Data Pipeline Development: Design construct test and maintain highly scalable data management systems. Develop and implement architectures that support the extraction transformation and loading (ETL) of data from various sources.
Data Integration: Integrate structured and unstructured data from multiple data sources into a unified data system ensuring data quality and consistency.
Data Warehousing: Build and maintain data warehouses and data lakes to store and retrieve vast amounts of data efficiently. Optimize the performance of databases and queries to meet business needs.
Data Processing: Implement data processing frameworks (e.g. Hadoop Spark) to process large datasets in real-time or batch processing.
Automation and Monitoring: Automate manual processes optimize data delivery and develop data monitoring systems to ensure data integrity and accuracy.
Collaboration: Work closely with data scientists analysts and other stakeholders to understand data needs and provide technical solutions that meet business requirements.
Data Governance: Ensure data governance policies are followed including data security data privacy and compliance with regulations.
Performance Tuning: Optimize the performance of ETL processes databases and data pipelines to handle large volumes of data and reduce processing times.