Senior Data Engineer - Teradata/Databricks & AI/ML

Description:

Experience ( Relevant) : 10+ Years

Shift Timings : 2pm to 11pm IST

Hybrid : 3 days/ week 

Start date : ASAP

Top 3 skills which is mandatory :

Please focus on the Databricks AI/ML skills and we can be flexible on the cost for the right resource.

" Target individuals with Databricks Machine Learning or Databricks Generative AI Associate certification "

Key Responsibilities

Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.

Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.

Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.

Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.

Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.

Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.

Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.

Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.

Required Skills & Experience

Teradata DBA Skills

Skill / Technology

Min. Experience

Relevance to Work Order

Teradata DBA / Administration 7+ years

Primary platform under analysis; deep knowledge of system internals required

Teradata DBQL (Database Query Logging) 5+ years

Core data source — SQL text, object usage, timestamps, user/app IDs, row counts, steps

Teradata System Views & Space Metadata 5+ years

Object inventory, space analysis, last read/write tracking

Teradata SQL & Performance Tuning 7+ years

Writing complex analytical queries across DBQL and system catalog tables

Teradata ResUsage / Workload Statistics 3+ years

Quantifying CPU/IO cost impact for dataset removal/archival recommendations Data Classification (hot/warm/cold) 3+ years

Categorizing datasets for tiered storage management recommendations

ETL Pipeline Assessment (DataStage, Autosys) 4+ years

Identifying and recommending decommission of stale ingestion pipelines

Teradata BTEQ / FastExport / TPT 3+ years

Data extraction, log export, and metadata collection utilities

Databricks Engineering Skills

Databricks Platform (Workspaces, Clusters, Jobs) 4+ years

Core analysis and solution delivery platform for the engagement

Apache Spark (PySpark / Spark SQL) 5+ years

Large-scale log ingestion, transformation, and analytical processing

Delta Lake 3+ years

Building reliable, ACID-compliant data pipelines for log-driven analysis

Databricks Notebooks & Workflows 3+ years

Developing repeatable, shareable analytical notebooks for customer handoff

Python (pandas, numpy, matplotlib) 5+ years

Data wrangling, statistical analysis, and visualization of usage heatmaps

SQL (Spark SQL / ANSI SQL) 7+ years

Analytical querying across large DBQL datasets in Databricks

Unity Catalog / Data Governance in Databricks 2+ years

Organizing deliverable assets and maintaining data lineage within the solution

Dashboarding (Databricks SQL / Power BI) 3+ years

Usage heatmaps, partition access patterns, and recommendation visualizations

AI / ML Skills

Machine Learning (scikit-learn, MLflow) 3+ years

Anomaly detection on access patterns, predictive cold-data classification

LLM / Generative AI Integration 2+ years

LLM-assisted log interpretation, automated recommendation narrative generation

Feature Engineering on Time-Series Log Data 3+ years

Extracting access frequency, recency, and seasonality signals from DBQL logs

Clustering & Classification Algorithms 3+ years

Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers

MLflow / Experiment Tracking 2+ years

Tracking model runs for reproducible analysis and customer handoff artifacts

Natural Language Processing (NLP) 2+ years

Parsing and classifying SQL text from DBQL for workload pattern analysis

Data Visualization (Plotly, Matplotlib, Seaborn) 3+ years

Generating usage heatmaps and partition access charts for stakeholder readouts

Preferred Qualifications

Familiarity with DataStage orchestration log parsing

Prior work on data platform cost optimization or cloud migration readiness assessments

Experience working in healthcare payer data environments (HIPAA awareness)

• Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs)