Aware Corpolation (Aug 2025 - current)

Data Engineer

• Manage Data catalog

  I specialized in managing the enterprise Data Catalog using DataHub. My responsibilities included orchestrating automated metadata ingestion from diverse environments like Azure SQL and MSSQL. Additionally, I designed and implemented access control policies to manage user permissions effectively, ensuring both data security and seamless data discovery for the team.
  • Implemented a centralized Data Catalog: using DataHub hosted on Azure Kubernetes Service (AKS) to enhance data governance and discoverability.
  • Orchestrated an end-to-end CI/CD pipeline: by creating custom Dockerfiles, managing container images via Azure Container Registry (ACR), and automating deployments using Jenkins.
  • Engineered metadata ingestion pipelines: from hybrid sources (Azure SQL and MSSQL) to capture technical schemas, descriptions, ownership, and end-to-end Data Lineage.
  • Integrated DataHub’s GraphQL API: With a custom Frontend application, enabling seamless access to metadata and governance insights for end-users.

Key Skill: Datahub (GraphQL) , Azure (AKS , ACR) , Kubernetes , Dockers , Jenkins , Azure SQL , Data Governance

Bluebik Vulcan (Jan 2025 - Aug 2025)

Data Engineer

• Cost Optimization

  I have been commissioned to optimize costs in Google Cloud Storage (GCS), to reduce storage costs by analyzing data usage patterns and implementing storage policies that align with data access frequency.
  • Storage Usage Patterns: Reviewed object access logs to identify inactive (cold) data, monitored storage growth and usage trends using Cloud Monitoring and Logging.
  • Data by Access Frequency: Categorized data into Hot (frequently accessed), Cold (rarely accessed), and Archived (long-term retention).
  • Lifecycle Policies: Configured GCS lifecycle rules to automatically move or delete files based on access time.
  • Redundant or Unused Files: Wrote Python scripts to detect and delete duplicate or obsolete files, coordinated with data owners to validate file necessity.

• Vector Search

  Built a scalable and intelligent search system that combines full-text search and vector search (Hybrid Search) using Google Cloud Spanner as the backend, with data pipeline orchestration via Dagster.
  • Built a Dagster Data Pipeline: Developed a pipeline to extract data from CMS-stored tables in BigQuery, transformed and loaded the data into Google Cloud Spanner.
  • Spanner Schema for Hybrid Search: Defined schema to store both text fields and vector embeddings (index, partitions), ensured data consistency and optimized schema for hybrid querying.
  • Full-text and Vector Search: Enabled full-text indexing within Spanner, integrated vector similarity search using approximate nearest neighbor (ANN) techniques.
  • Hybrid Search Logic in App Layer: Combined full-text and vector search results into a unified ranking system, built an API layer to expose the search functionality to client applications.

Key Skill: GCP, GCS, GKE, Dagster, Datahub, BigQuery, Spanner, VectorSearch, Full-TextSearch

A Lot tech (Aug 2023-Jan 2025)

Data Engineer

My responsibilities encompass extracting data from PDF files through web scraping, designing and maintaining efficient ETL pipelines, automating daily data workflows, and delivering impactful reports and visualizations. These efforts empower the marketing team with strategic insights to drive informed decision-making.

Key Skill: Bs4, request, GCP API, JupyterLab, Colab, SQL, PowerBI