Ofertas › GCS
GCS · London, United Kingdom · £? - ? per year
Salary: £? - ? per year Requirements: Strong commercial experience as a Data Engineer or Senior Data Engineer Expert-level Python development skills Extensive hands-on experience with Databricks in enterprise environments Strong Apache Spark and PySpark expertise Advanced SQL development and optimisation skills Experience designing and building large-scale ETL and data integration solutions Strong understanding of distributed computing concepts and data processing frameworks Experience with modern cloud platforms, including Azure, AWS or GCP Knowledge of software engineering principles, testing and code quality practices Experience working in Agile development environments Excellent problem-solving and troubleshooting capabilities Desirable: experience in investment banking, financial services or other highly regulated environments Desirable: exposure to risk, trading, treasury or regulatory data domains Desirable: experience with Kafka, streaming technologies or real-time data processing Desirable: Databricks certifications Desirable: experience with Airflow, Databricks Workflows or similar orchestration tools Desirable: CI/CD pipeline development and DevOps practices Desirable: experience with Docker and Kubernetes Desirable: experience supporting data science or machine learning workloads Responsibilities: Design, develop and maintain scalable data pipelines using Python, Databricks and PySpark Build and optimise ETL/ELT solutions that process large-scale datasets Develop high-quality, reusable code and engineering frameworks Deliver robust, performant and maintainable data solutions Troubleshoot and resolve complex production and performance issues Enhance and modernise enterprise data platforms Develop and support Databricks-based data solutions and Lakehouse architectures Use Delta Lake and Spark to deliver scalable data processing capabilities Optimise distributed data processing workloads for performance and reliability Implement data quality controls and monitoring solutions Contribute to platform stability, scalability and operational excellence Work with business and technology stakeholders to understand requirements and deliver effective solutions Participate in Agile delivery activities, including sprint planning, refinement and estimation Support strategic data initiatives across multiple business areas Collaborate with architects, engineers and product teams to deliver high-quality outcomes Identify and resolve technical risks and implement