Middle Data Engineer - Snoonu

Main responsibilities will include creating pipelines with complex ETL logic making performance optimizations investigating cases in collaboration with BI and Data Science team Our workflow emphasizes adaptability collaboration and focus on delivering value to customers and teammates We look at the entire development process and optimize it as a whole involving a culture of continuous platform and personal improvement Skills Required Python programming skills with a focus on code reuse parallelization and ensuring reliable pipeline functionality Skills in workflow management frameworks (Airflow) with the ability to define and run complex DAGs Extensive SQL skills including DML and DDL (ALTER TABLE GENERATEARRAY LEAD ) and query performance optimization Py Spark knowledge for adhoc Spark jobs window functions and Spark SQL for aggregate and join operations Understanding of Data Warehousing and Data Lake concepts dimensional approach ability to follow appropriate indexing partitioning and other optimizations to support efficient data retrieval 2 years of experience with cloud providers or proficiency in using Databases/Cloud Storage APIs Responsibilities Create new batch/streaming pipelines for newly appearing databases Participating in integrated storage layer creation investigating linkage between services Improve/expand pipelines with new tables or fields according to business requests Participate in technical debt management redefine tables partitioning create summarised tables and proceed migration create metadata Use SQL and business context understanding skills to help engineers BI analysts and product managers get insights Our Stack GCP platform Big Query Databricks AWS streaming services Tableau Redash Benefits 100% remote work with ability to work from office in the countries where we have offices Qatar Kazakhstan Tajikistan Free courses and and personal growth activities The opportunity for growth in a missiondriven wellfunded and already profitable startup Cultural enrichment interacting with Teammates from more than 50 countries