Roles and Responsibilities:
- Design, develop, and deploy data pipelines and ETL processes using Azure Data Factory.
- Implement data integration solutions, ensuring data flows efficiently and reliably between various data sources and destinations.
- Collaborate with data architects and analysts to understand data requirements and translate them into technical specifications.
- Build and maintain scalable and optimized data storage solutions using GCP and Teradata.
- Develop and manage data transformation and cleansing processes to ensure data quality and accuracy.
- Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner.
- Optimize data pipelines for performance, cost, and scalability
Critical Skills to Possess:
- Proven experience as a Data Engineer, with a strong focus on Azure cloud technologies.
- Proficiency in GCP services.
- Solid understanding of data modeling, database design principles, and data warehousing concepts.
- Experience with data integration, ETL processes, and data transformation using tools like Azure Data Factory, Azure Logic Apps, or similar technologies.
- Strong programming skills in languages such as SQL, Python, or Spark.
- Familiarity with version control systems (e.g., Git) and CI/CD pipelines for code deployment.
- Knowledge of data security and compliance practices within GCP
Preferred Qualifications:
- BS degree in Computer Science or Engineering or equivalent experience