Dan Siegel
I build high-performance data systems and write efficient, custom catalog extensions. From managing complex pipelines to executing Redshift-to-Snowflake migrations that slash compute bills by 40%, I bridge engineering depth with business delivery.
Services
Cloud & Infrastructure Architecture
Production-grade cloud setups and automation. Designing highly secure, self-healing platforms with built-in cost optimization.
- GitOps delivery (Flux, Helm, Kustomize)
- Production Kubernetes (k8s) & cloud security
- Snowflake warehouse resizing & cost audits
Resilient & High-Performance Pipelines
Developing low-latency event-driven streams, real-time message routing, and reliable anomaly triage. Building hardened, self-healing data delivery paths for mission-critical operations.
- Low-latency Go WebSocket snipers & NATS JetStream
- Fault-tolerant Apache Flink & Apache Kafka systems
- Hardened, self-healing CDC (Debezium, Kafka)
Analytics Engineering & DS/MLOps
Bridging the gap from raw database schemas to production ML models. Transforming chaotic data pools into optimized Kimball star schemas, while engineering low-latency feature stores and model registries (MLflow) to deploy model predictions at scale.
- Dimensional modeling & lakehouse design (Snowflake, Databricks)
- MLOps feature engineering & model lifecycles (MLflow, Spark)
- Metric stores, semantic layers, and automated BI enablement
Agile Program Delivery & Architecture Audits
Aligning data engineering strategy with business execution. Auditing cloud architectures, rescuing high-risk projects, and running end-to-end software procurement, vendor selection, and SOW scoping pipelines.
- High-risk project rescue & migration recovery
- End-to-end technology procurement & SOW/RFP pipelines
- Enterprise architecture audits & cost-benefit reviews
- Agile/Scrum team governance & roadmap velocity alignment
Projects & Writings
Case studies, C++ extensions, and real-time streaming architectures. Read my detailed engineering posts on Substack.
DuckSync (DuckDB Snowflake Caching)
Why do dashboards query static Snowflake datasets hundreds of times a day and burn unnecessary compute? I built DuckSync, a C++ DuckDB community extension, to intercept SQL queries, rewrite AST paths, analyze Snowflake table metadata (last_altered), and cache rows locally as Parquet files managed by a Postgres catalog. This keeps the Snowflake data warehouse asleep, drastically slashing compute costs.
WhaleDoxer (Real-time Prediction Market Tracker)
A real-time paper-trading and suspicion engine designed to monitor information asymmetry and anomalies in Polymarket and Kalshi order books. It runs a sub-millisecond hot path using a lightweight Go sniper node (8µs p95 latency), streams raw events through NATS JetStream, and routes tripwires to Python forensics tasks executing wallet identity tracking and composite Brier scoring.
Salesforce Reverse ETL Framework
A cost-efficient, open-source alternative to expensive commercial Reverse ETL tools. Built by extending Apache Airflow operators and embedding dbt SQL validation models to sync clean warehouse records directly to Salesforce accounts and leads, avoiding high SaaS licensing fees and platform lock-in.
Experience
- Architected and optimized data models that reduced processing costs by 40% and increased query performance, migration from Redshift to Snowflake enabled BI teams to deliver insights 30% faster.
- Led data infrastructure modernization using Terraform and Kubernetes, cutting deployment time from days to hours and improving time-to-market by 50%.
- Designed and implemented robust data pipelines that automated 95% of manual processes, ensuring 99.9% data reliability while reducing operational overhead by 60%.
- Partnered with teams to build reverse ETL paths synchronizing analytics and operational systems, increasing customer acquisition metrics by 25%.
- Managed a team of 3 data engineers, delivering 100% of roadmap deliverables on-time and raising team velocity by 35%.
- Automated 80% of manual pipelines and designed infrastructure for 3x scalability, generating $200K in annual cloud infrastructure cost savings.
- Constructed data solutions leveraging AWS Athena, Glue, Talend, Python, Databricks, Spark, and dbt to process 15+ data sources, cutting ingestion latency by 60%.
- Built robust ETL/ELT flows across 8+ data platforms (Netezza, Oracle, SQL Server, Snowflake, APIs), raising data validity to 99.5% and cutting integration times by 50%.
- Optimized hospital analytical processes, automating 70% of manual tasks and generating $150K in annual operational savings.
- Developed statement of work parameters, business requirements, and integrated automated BI dashboards utilizing SQL warehouse backends.
- Served as Agile coach and Scrum product owner for development sprints.
- Provided oversight and support of the implementation of SaaS HCM software for new clients or existing client upsells.
- Controlled project requirements, scope, and change management to ensure on-time achievement of project milestones and deliverables.
- Ranked #1 against over 500 peers within market segmentation at 177% to Plan YTD at time of departure.
- Created and managed contingent workforce process including submission processes, onboarding / offboarding, vendor management policies, and played a key role in MSP implementation.
- Leveraged Agile and Kanban methodologies and developed recruitment analytics for risk evaluation, resource forecasting, and velocity optimization.
- Controlled document registry and project timelines by implementing collaboration platforms including Confluence and Sharepoint.
🕰️ Early Career & Internships (2008 – 2012) Click to expand
- Human Resources Scheduler (Contract) • Ask.com, Oakland CA (5/2012 – 10/2012)
- Recruiting Coordinator (Contract) • P.C.A.O.B., Washington DC (5/2011 – 8/2011)
- Analyst (Contract) • Thomson Reuters, Washington DC (1/2011 – 5/2011)
- Finance Assistant • Congressman Bill Foster, Batavia IL (5/2010 – 11/2010)
- Policy Assistant (Contract) • GLSEN, Washington DC (1/2010 – 4/2010)
- Intern • Steve Shannon for Attorney General, Fairfax VA (8/2009 – 11/2009)
- Intern • The White House, CEQ, Washington DC (2/2009 – 5/2009)
- Intern • Congresswoman Bordallo, Washington DC (9/2008 – 12/2008)