NYC Data Job Market Pulse

Overview

After finishing the NHL project, I started actively job searching, and kept seeing the same thing in postings: mentions of the "Modern Data Stack" — dbt, Snowflake, and tools I hadn't touched yet. I figured my next project should put those to use directly. At the same time, I noticed there were several different roles within the data space — Data Analyst, Data Engineer, Analytics Engineer, Data Scientist — each a real option for me to pursue as a candidate, but with blurry, overlapping lines between them. I got curious about what actually separates them: their responsibilities, requirements, and salaries. Rather than guess, I figured the best way to find out was to build something that could answer the question directly — and to point it at the market I'm actually living in and applying to right now, New York City.

The project ingests job postings from three sources — two APIs and one scraped site — across those four roles. It runs on an ELT architecture: raw payloads land in Snowflake VARIANT objects untouched, get enriched by an LLM that extracts structured metadata from each posting's freeform description, and are transformed through a three-stage dbt process (staging, intermediate, mart) before reaching a Streamlit dashboard.

The Investigation

The central question isn't "which role pays more." It's something harder:

Are these four roles actually distinct? And if so, where are the real boundaries — and are they moving?

To get at that, I needed more than job titles and salary ranges. I needed to know what the postings were actually asking for underneath the title. So the pipeline is designed around two layers of truth: what the market says a role is (the listed title), and what it actually is (extracted by an LLM from the full description). That gap — between what a job is called and what it actually requires — is where the interesting stuff lives.

The dashboard is organized around three questions:

• The Landscape: What does the market look like right now? How many postings are there per role type, how are they distributed across sources, and how is volume changing over time as the pipeline accumulates data?
• Under the Hood: What are these roles actually asking for? Where do the tech stacks and paradigms overlap across role types? How often does the LLM's classification of a job diverge from its listed title — and what does that tell us about how accurately the market is labeling its own roles? Which roles acknowledge AI, and which don't?
• Job Explorer: Given all of this, what does a specific posting actually require? The explorer surfaces the full LLM-extracted metadata alongside the raw description so you can see both layers at once — what the job is called, and what it's actually asking for.

Early Signals

The dataset spans all four role types under investigation — Data Analyst, Data Engineer, Analytics Engineer, and Data Scientist — with 448 postings from three sources (Built In NYC, TheirStack, JSearch), spanning late May through late June 2026. These are early observations from an accumulating dataset, not final conclusions.

Architecture

The NYC Data Job Market Tracker operates on a modular ELT (Extract, Load, Transform) architecture across four layers. Original source payloads are preserved unmodified at every stage — nothing is overwritten, and every transformation is independently rerunnable without re-hitting the APIs.

• Ingestion Layer: A GitHub Actions cron job fires every Monday and Thursday at 9am EDT, executing a Python ingestion client that pulls from three sources simultaneously: the JSearch API (RapidAPI), the TheirStack API, and a custom Built In NYC scraper built with BeautifulSoup using a two-pass approach — search page crawl followed by individual listing parsing. A pre-ingestion deduplication check runs before any metered API call is made, dropping duplicate postings before they consume credits.
• Raw Storage Layer: All three sources write structured JSON payloads into a dedicated Snowflake RAW database, partitioned by source schema (RAW.JSEARCH, RAW.THEIRSTACK, RAW.BUILTIN). A pipeline tracking schema (RAW.PIPELINE) records every run with timestamps, row counts, and per-source API credit consumption.
• LLM Enrichment & Transformation Layer: Rather than relying on fragile regex to extract entities from unstructured job descriptions, the pipeline routes each posting through GPT-4o-mini with a structured extraction prompt. The enrichment layer outputs a standardized JSON schema per posting — role archetype, tech stack arrays, paradigms, inferred seniority, years of experience required, degree requirement, and whether the posting acknowledges AI. Salary is a clear example of where this earns its keep: many postings state a range only in prose rather than in a structured field, and parsing that out raises usable salary coverage from 35.7% to 61.6% of postings. From there, dbt handles all schema normalization and cross-source unification across three tiers: staging views, ephemeral intermediates, and a production mart table (FCT_JOB_POSTINGS).
• Analytics & Visualization Layer: The mart table is surfaced into Streamlit across four pages. Data loads with a one-hour cache and all charts pull dynamically from the live mart — as new postings are ingested and transformed, the dashboard updates automatically.

Engineering Decisions

The interesting parts of this project aren't the tools — they're the tradeoffs. Here are the decisions that shaped the architecture and why.

Problems I Found In My Own Data

Conclusion

The biggest shift in this project wasn't a tool or a technique — it was moving from analyzing a fixed, static dataset to building something that's alive. The NHL project worked with a closed set of drafts that would never change. This one doesn't sit still: new postings arrive twice a week, and the data itself evolves underneath the analysis. That changes what "fixing a problem" means. It's not enough to patch the code going forward — when the explicitly_encourages_applicants prompt was over-triggering, or ingestion_query turned out to be the wrong signal for role classification, the fix had to be applied twice: once so future postings come in correctly, and once to re-run against everything already sitting in the warehouse. A live pipeline doesn't let you just move on from a mistake; it makes you go back and clean up after it too.

There's a pattern underneath those fixes worth naming directly: this project's whole premise is that what a job posting is called and what it actually is often don't match — and that same gap kept showing up inside my own pipeline. ingestion_query claimed to identify a posting's role; it didn't. TheirStack claimed to deduplicate results; it didn't, reliably. A boolean flag claimed to detect encouragement toward underqualified applicants; it was matching boilerplate instead. The project's thesis turned out to be true recursively — not just about the job market, but about the tools and labels I was trusting to describe my own data. Catching that consistently, rather than after the fact, is most of what the notebook-first habit described above is actually for.

Beyond the findings themselves, this project gave me hands-on experience with a fuller slice of the data stack than the NHL project did: architecting ingestion against real API constraints and cost limits, designing an LLM enrichment layer for a task LLMs are genuinely well-suited to — pulling structured signal out of messy human-written text — standardizing that signal across sources that don't agree with each other, and building a pipeline that has to keep working as the world underneath it keeps changing. That combination — comfortable across ingestion, transformation, and analysis, not just one layer of it — is the kind of range I'm hoping to bring to a data role in this role market.