Senior Data Lake Engineer - Databricks
Senior Data Lake Engineer - Databricks
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Not Disclosed
Salary Not Disclosed
1 Vacancy
Job Description
Overview/ Job Responsibilities
We are seeking a highly experienced and skilled Senior Data Lake Engineer to join our team. As the Senior Data Lake Engineer you will play a critical role in establishing and configuring an enterprise-level Databricks solution to support our federal customer organizations data lake initiatives. This position offers a unique opportunity to work with cutting-edge technologies and shape the future of our federal customers data infrastructure.
If you are a highly skilled and experienced Senior Data Lake Engineer with expertise in Databricks and passion for building scalable and secure data lake solutions we would like to hear from you.
Responsibilities:
- Lead the design implementation and configuration of an enterprise Data Lake solution utilizing Databricks ensuring scalability reliability and optimal performance.
- Collaborate with cross-functional teams to gather requirements understand data integration needs and define data lake architecture and governance policies.
- Establish and configure Databricks workspaces clusters and storage components optimizing the solution for efficient data processing query performance and data governance.
- Design and implement data ingestion pipelines to efficiently extract transform and load data from various sources into the data lake using Databricks tools and services.
- Develop and maintain data lake security frameworks including access controls encryption solutions and data masking techniques to protect sensitive data.
- Collaborate with data engineers and data scientists to optimize data pipelines develop data transformations and ensure data quality and integrity.
- Monitor and tune Databricks clusters and workloads to ensure performance reliability and cost optimization utilizing automated scaling and resource management techniques.
- Implement best practices for data governance data cataloging metadata management and data lineage within Databricks adhering to regulatory and compliance requirements.
- Collaborate with infrastructure teams to ensure data lake infrastructure meets scalability and availability requirements leveraging Databricks cluster management and AWS/Azure services.
- Develop and maintain documentation and guidelines related to the Databricks solution including architecture diagrams standards and processes.
- Stay up to date with the latest advancements in Databricks big data technologies and cloud platforms continuously evaluating and implementing new features and capabilities.
- Provide technical guidance and mentorship to junior data engineers promoting best practices and fostering a culture of continuous learning and growth.
- Collaborate with stakeholders to understand their data analytics and reporting needs and develop scalable data models and data transformation processes to support these requirements.
- Support data lake-related incident resolutions troubleshooting data quality issues performance bottlenecks and other data-related challenges.
- Collaborate with data governance and compliance teams to ensure data privacy security and compliance guidelines are adhered to within the data lake solution.
- Participate in the evaluation and selection of new tools technologies and services to enhance the data lake infrastructure.
Minimum Qualifications
- Bachelors degree in computer science information technology or a related field. Equivalent experience will also be considered.
- Proven experience in building and configuring enterprise-level data lake solutions using Databricks in an AWS or Azure environment.
- In-depth knowledge of Databricks architecture including workspaces clusters storage notebook development and automation capabilities.
- Strong expertise in designing and implementing data ingestion pipelines data transformations and data quality processes using Databricks.
- Experience with big data technologies such as Apache Spark Apache Hive Delta Lake and Hadoop.
- Solid understanding of data governance principles data modeling data cataloging and metadata management.
- Hands-on experience with cloud platforms like AWS or Azure including relevant services like S3 EMR Glue Data Factory etc.
- Proficiency in SQL and one or more programming languages (Python Scala or Java) for data manipulation and transformation.
- Knowledge of data security and privacy best practices including data access controls encryption and data masking techniques.
- Strong problem-solving and analytical skills with the ability to identify and resolve complex data-related issues.
- Excellent interpersonal and communication skills with the ability to collaborate effectively with technical and non-technical stakeholders.
- Experience in a senior or lead role providing technical guidance and mentorship to junior team members.
- Relevant certifications such as Databricks Certified Developer or Databricks Certified Professional are highly desirable.
- Eligibility/Clearance Requirements: Must be able to provide proof of U.S. Citizenship.
Desired Qualifications
Clearance Preference:
- Active DHS/CISA suitability - 1st priority
- Any DHS badge DoD Top Secret - 2nd choice
- DoD Top Secret willingness to obtain DHS/CISA suitability - 3rd choice (it can take 10-60 days to obtain suitability work can only begin once suitability is fully adjudicated).
About Sev1Tech LLC
Founded in 2010 Sev1Tech provides IT engineering and program management solutions 1Tech focuses on providing program and IT support services to critical missions across Federal and Commercial Clients. Our Mission is to Build better companies. Enable better government. Protect our nation. Build better humans across the country.
- Join the Sev1Tech family where you can achieve great accomplishments while fostering a satisfying and rewarding career progression. Please apply directly through the website at: any additional questions or to submit any referrals please contact:
- Sev1Tech is anEqual Opportunity and Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race color religion sex pregnancy sexual orientation gender identity national origin age protected veteran status or disability status.
Required Experience:
Senior IC
Employment Type
Unclear
