[Remote] 100% Remote-Data Engineer
Note: The job is a remote job and is open to candidates in USA. Beacon Hill is a premier National Information Technology Staffing Group, and they are seeking a Senior Data Engineer to build and support production-grade data pipelines. The role involves migrating existing data assets to Databricks Unity Catalog, optimizing pipelines, and ensuring data quality and reliability.
Responsibilities
- Migrate existing data assets and pipelines to Databricks Unity Catalog
- Refactor and optimize existing pipelines using modern engineering standards and best practices
- Build and maintain scalable PySpark-based data pipelines
- Develop new source-of-truth datasets and modernize legacy data workflows
- Design and implement workflow orchestration using Databricks Jobs
- Deploy and manage pipelines using Databricks Asset Bundles
- Ensure data pipelines are reliable, observable, and production-ready
- Implement data quality controls, monitoring, and operational alerting
- Support platform migrations, testing, validation, and production cutovers
- Partner with engineering teams to improve platform stability, scalability, and data reliability
Skills
- Strong hands-on experience building and supporting production-grade data pipelines using PySpark and Apache Spark
- Extensive experience working within Databricks environments
- Experience with Databricks Unity Catalog migrations and modernization initiatives
- Experience with Databricks Unity Catalog governance and security models
- Strong understanding of ETL/ELT design, data modeling, and incremental data processing
- Experience modernizing legacy data platforms and data migration projects
- Experience working with large-scale datasets in cloud-based data platforms
- Experience with workflow orchestration, scheduling, and pipeline automation
- Knowledge of cloud data platforms such as Snowflake and/or Google Cloud
- Experience implementing data quality checks, monitoring, logging, and alerting
- Strong troubleshooting, collaboration, and problem-solving skills
- Familiarity with advertising, analytics, measurement, or large-scale data ecosystems
- Experience with CI/CD and DataOps best practices for data engineering environments
Benefits
- Medical, dental, vision, and federal and state leave programs as required by applicable agency regulations to those that meet eligibility
- Upon successfully being hired, details will be provided related to our benefit offerings
Company Overview