Job Opportunity Posted yesterday Updated 19 Nov 2026

Software Engineer - Data Engineering

Job Description

What wearelookingfor:

Experienceandeducation:

  • Bachelor's degree in Computer Science, InformationTechnology, Electronics or a related engineering discipline.Amaster's (MCA, MSc, MTech) is an advantage.
  • Onetothreeyearsofprofessionalsoftwareengineering,withcodeyouhavewritten runninginproduction.
  • Production pipelinesyouhavebuiltand run, withexperiencethatgoesbeyond notebooksandreports.
  • The fullchangelifecycle,notthebuildphasealone, andcodereviewgivenandreceived ina professionalcodebase.
  • Exposuretostreamingorchangedatacapture,ortodatagovernanceandprivacywork,isanadvantage.

Coretechnical skills:

  • SQL: joins,aggregation,windowfunctions,incrementalandmergepatterns, andtheabilitytoreadaqueryplan.
  • Python:clean,testablecodethatlivesoutsideanotebook.
  • Databricks:theworkspace,jobsandSQLwarehouses,withDeltaLaketables,partitioningandcompaction.
  • PySparkandSparkSQL:thefundamentals,withawarenessofpartitioningandshufflebehaviour.
  • dbtandAirflow:transformationandtestingindbt,andrunninganddebuggingjobsinAirfloworDatabricksWorkflows.
  • AWS: S3,Parquet andDeltafiles, Glue,Lambda andIAMbasics.
  • MySQL and DynamoDB:the operational sources you ingest from, including change data capture against MySQLand export patterns from DynamoDB.
  • Unity Catalog:governance, lineage and access control, including how a grant or a row-level filter changes what your pipeline is allowed to read.
  • Power BIorDatabricksSQL:dashboardsontopofwhatyoudeliver,sothenumbers getconsumedandnotjust landed.

Nice to have:

  • Kafka, Kinesis or change data capture tooling such as Debezium against MySQL sources.
  • MLflow,or feature engineeringfor machine learningworkloads.
  • Terraformfordatainfrastructure.
  • Databricks Data EngineerAssociate,AWS Data Engineer Associate, or a generative AI credential.

AI-assisteddevelopmentcapability:

  • Daily, deliberate use of an approved assistant such as Claude, Copilot, Cursor or DatabricksAssistant, with the habit of checking its output rather than trusting it.
  • Retrieval basics:preparingcontent forretrieval,building anembedding pipeline,andloadingavectorstoresuchas pgvectororOpenSearch.

How you work:

  • Deliversreliably,andflagsslippage earlyratherthanlate.
  • Curious and coachable: asks rather than guesses, and carries review feedback into the next change.
  • Quality-minded, and thinks about the person who reads the code next.
  • Team-first. Helps peers, joins incident reviews, shares what they learn.
  • Checks the number. Reconciles against source by instinct.

Whatyouwillworkon:

Pipelineandplatformengineering:

  • Ingestion pipelines:start with batch and move into streaming over time, handling schema change, late and duplicate data, and reconciliation against source.
  • Transformationlogic:writteninPySpark,SparkSQLanddbtacrossthebronze,silverandgoldLakehouse layers.
  • Opentableformats:whatDeltaLakeorIcebergactually givesyou,including transactions,schemaevolution, timetravelandcompaction.
  • Data qualitytests:completeness,freshness,uniquenessandreferentialchecks,withalertsthatreachanamedowner.
  • Orchestration:DatabricksWorkflows orAirflowjobs,includingdependencyhandling,failurerecovery andbackfill.
  • GovernanceinUnityCatalog:classification,ownership andaccesscontrolontheassetsyoutouch.

Privacyandcorrectness:

  • Handlepersonaldataproperly: classificationandmasking,andwhyresidencyconstraintsacrossthemarketsservedshapepipelinedesign.
  • Reconcilebeforeyoudeclare: checkoutputagainstsourcebeforeyoucallapipeline correct.
  • Startfromsyntheticormaskeddata:avoidproductionpersonaldatawhereverasubstitutewilldo.

Craft,qualityandproductionsupport:

  • Tests ship withthe change:same branch,same review.Afollow-upticket is not atest plan.
  • Instrumentanddocumentasyoubuild:logs,metricsanderrormessagesthatmakethepipelinesupportableforthenextengineer.
  • Support your consumers:investigate data questions from analysts and other engineering teams, and close the loop with a fix or an explanation.
  • Productiondefectsendtoend:reproduce,diagnose, fixandtest,thenconfirmthefixinproduction.
About this job listing
This job opportunity is provided through our external job listing network. MyJobAlerts helps you discover job opportunities and redirects you to the original listing to apply.