Job Description
What wearelookingfor:
Experienceandeducation:
- Bachelor's degree in Computer Science, InformationTechnology, Electronics or a related engineering discipline.Amaster's (MCA, MSc, MTech) is an advantage.
- Onetothreeyearsofprofessionalsoftwareengineering,withcodeyouhavewritten runninginproduction.
- Production pipelinesyouhavebuiltand run, withexperiencethatgoesbeyond notebooksandreports.
- The fullchangelifecycle,notthebuildphasealone, andcodereviewgivenandreceived ina professionalcodebase.
- Exposuretostreamingorchangedatacapture,ortodatagovernanceandprivacywork,isanadvantage.
Coretechnical skills:
- SQL: joins,aggregation,windowfunctions,incrementalandmergepatterns, andtheabilitytoreadaqueryplan.
- Python:clean,testablecodethatlivesoutsideanotebook.
- Databricks:theworkspace,jobsandSQLwarehouses,withDeltaLaketables,partitioningandcompaction.
- PySparkandSparkSQL:thefundamentals,withawarenessofpartitioningandshufflebehaviour.
- dbtandAirflow:transformationandtestingindbt,andrunninganddebuggingjobsinAirfloworDatabricksWorkflows.
- AWS: S3,Parquet andDeltafiles, Glue,Lambda andIAMbasics.
- MySQL and DynamoDB:the operational sources you ingest from, including change data capture against MySQLand export patterns from DynamoDB.
- Unity Catalog:governance, lineage and access control, including how a grant or a row-level filter changes what your pipeline is allowed to read.
- Power BIorDatabricksSQL:dashboardsontopofwhatyoudeliver,sothenumbers getconsumedandnotjust landed.
Nice to have:
- Kafka, Kinesis or change data capture tooling such as Debezium against MySQL sources.
- MLflow,or feature engineeringfor machine learningworkloads.
- Terraformfordatainfrastructure.
- Databricks Data EngineerAssociate,AWS Data Engineer Associate, or a generative AI credential.
AI-assisteddevelopmentcapability:
- Daily, deliberate use of an approved assistant such as Claude, Copilot, Cursor or DatabricksAssistant, with the habit of checking its output rather than trusting it.
- Retrieval basics:preparingcontent forretrieval,building anembedding pipeline,andloadingavectorstoresuchas pgvectororOpenSearch.
How you work:
- Deliversreliably,andflagsslippage earlyratherthanlate.
- Curious and coachable: asks rather than guesses, and carries review feedback into the next change.
- Quality-minded, and thinks about the person who reads the code next.
- Team-first. Helps peers, joins incident reviews, shares what they learn.
- Checks the number. Reconciles against source by instinct.
Whatyouwillworkon:
Pipelineandplatformengineering:
- Ingestion pipelines:start with batch and move into streaming over time, handling schema change, late and duplicate data, and reconciliation against source.
- Transformationlogic:writteninPySpark,SparkSQLanddbtacrossthebronze,silverandgoldLakehouse layers.
- Opentableformats:whatDeltaLakeorIcebergactually givesyou,including transactions,schemaevolution, timetravelandcompaction.
- Data qualitytests:completeness,freshness,uniquenessandreferentialchecks,withalertsthatreachanamedowner.
- Orchestration:DatabricksWorkflows orAirflowjobs,includingdependencyhandling,failurerecovery andbackfill.
- GovernanceinUnityCatalog:classification,ownership andaccesscontrolontheassetsyoutouch.
Privacyandcorrectness:
- Handlepersonaldataproperly: classificationandmasking,andwhyresidencyconstraintsacrossthemarketsservedshapepipelinedesign.
- Reconcilebeforeyoudeclare: checkoutputagainstsourcebeforeyoucallapipeline correct.
- Startfromsyntheticormaskeddata:avoidproductionpersonaldatawhereverasubstitutewilldo.
Craft,qualityandproductionsupport:
- Tests ship withthe change:same branch,same review.Afollow-upticket is not atest plan.
- Instrumentanddocumentasyoubuild:logs,metricsanderrormessagesthatmakethepipelinesupportableforthenextengineer.
- Support your consumers:investigate data questions from analysts and other engineering teams, and close the loop with a fix or an explanation.
- Productiondefectsendtoend:reproduce,diagnose, fixandtest,thenconfirmthefixinproduction.
About this job listing
This job opportunity is provided through our
external job listing network. MyJobAlerts helps
you discover job opportunities and redirects you
to the original listing to apply.