Job Description

Role: Data Engineer

Locations: Melbourne   

Duration: Contract – 6 months (with possible extensions)

Talent Street is looking for a Data Engineer for one of our large Victorian State government clients.

About the role:

The highly skilled Data Engineer is to join the Data Migration project team. The role is critical in designing, implementing, and optimising data pipelines for a large-scale data migration initiative. The ideal candidates must have strong expertise in Azure Data Factory (ADF), ETL processes, SQL Server, and Azure Storage (Blob, ADLS, Gen2).

This role will be responsible for planning and executing full and incremental data loads, ensuring data transformation, cleansing, and validation, handling exception scenarios, and optimizing performance. Additionally, you will work closely with cross-functional teams, including data reporting, developers, and business stakeholders, to ensure smooth data migration and high data quality.

Key Responsibilities:

1. Data Pipeline Development & Migration

• Design, develop, and manage Azure Data Factory (ADF) pipelines to facilitate end-to-end data migration.
• Implement full load and incremental load strategies for migrating data from various sources to the target system using dynamic pipeline.
• Handle pipeline scheduling, monitoring, and execution ensuring reliability and minimal downtime.
• Optimize data pipelines for performance and scalability.

2. Data Transformation & Cleansing

• Ensure data integrity, accuracy, and consistency through transformation and cleansing techniques.
• Develop ETL logic to map source-to-target transformations efficiently.
• Implement data validation checks to identify and handle discrepancies during migration.

3. Exception Handling & Troubleshooting

• Implement robust error-handling mechanisms in Azure Data Factory pipelines.
• Identify, diagnose, and resolve issues related to data processing, transformation failures, or connectivity problems.
• Maintain detailed logs and proactively monitor data flow issues.

4. SQL Development & Database Management on Azure

• Develop complex SQL queries, stored procedures, functions, and scripts for data extraction, transformation, and loading (ETL).
• Work with Azure SQL Server databases to design and optimize data models for efficient migration.
• Perform data profiling, indexing, and partitioning for enhanced query performance.
• Support data warehouse development and maintenance.

5. Azure Storage & Cloud Integration

• Work extensively with Azure Storage solutions (Blob & ADLS Gen2) to manage and process large datasets.
• Ensure secure and optimized data transfer between environments.
• Manage access permissions and security compliance for Azure data storage.

6. Testing & Validation

• Conduct pre-migration and post-migration data validation to ensure completeness and accuracy.
• Create test cases and scripts to verify data integrity across multiple environments.
• Work with QA teams and business users to validate data consistency after migration.

7. Collaboration & Documentation

• Work closely with business analysts, solution architects, and other stakeholders to understand data migration requirements.
• Document ETL processes, data mappings, pipeline configurations, and troubleshooting steps for knowledge sharing.
• Provide technical support and guidance to the team regarding data engineering best practices.

Required Qualifications & Skills:

1. Experience & Background - Essential

• 5-6 years of hands-on experience in data engineering, data migration, and ETL development.
• Strong background in Azure Data Factory (ADF) with expertise in:
  • Pipeline planning, designing, execution, and monitoring
  • Handling full and incremental loads
  • Exception handling and performance tuning
  • Data transformation and cleansing
• Experience with SQL Server, including database design, writing complex stored procedures, and performance optimization.

2. Technical Skills

• Strong expertise in Azure Data Factory (ADF) and its integration with other Azure services.
• Proficiency in SQL Server and data warehousing concepts.
• Hands-on experience in Azure Storage solutions (Blob, ADLS Gen2) for handling structured and unstructured data - Essential.
• Good understanding of ETL frameworks and data transformation techniques - Essential.
• Experience with pipeline scheduling, exception handling, and monitoring.
• Knowledge of Power BI or other data visualization tools is an added advantage.
• Familiarity with Microsoft Fabric is a plus.

3. Problem-Solving & Analytical Thinking

• Strong problem-solving skills to troubleshoot migration challenges.
• Ability to analyze large datasets and identify inconsistencies or performance bottlenecks.
• Experience in optimizing SQL queries and ADF pipelines for better performance.

4. Collaboration & Communication

• Ability to communicate effectively with technical and non-technical stakeholders.
• Experience working in cross-functional teams to align data migration activities with business goals.
• Strong documentation skills to maintain clear and comprehensive project records.

Preferred Qualifications (Good to Have):

• Experience with Microsoft Fabric for data integration and processing.
• Proficiency in Power BI or similar BI tools for reporting and visualization.
• Knowledge of data governance and compliance standards in cloud environments.
• Familiarity with other Azure cloud services related to data engineering.

Please apply now or contact Cheska Mitra, Delivery Consultant @Talent Street on  cmitra@talentstreet.com.au for further information.