pg_durable: Microsoft open sources in-database durable execution

Microsoft has open-sourced pg_durable, a PostgreSQL extension that enables in-database, fault-tolerant execution of complex workflows, removing the need for external orchestrators. This development allows teams to define durable, resumable processes directly within PostgreSQL, improving reliability and simplifying architecture.

pg_durable is a PostgreSQL extension designed to support fault-tolerant, long-running SQL workflows. It allows users to define a graph of SQL steps that are checkpointed and can resume after crashes, restarts, or failures. The extension is integrated into Microsoft’s Azure HorizonDB, a cloud service built on PostgreSQL, emphasizing performance and reliability. Users can define workflows using SQL operators, start them with df.start(), and query their status and results directly from PostgreSQL. The extension addresses common pain points such as rerunning partial work after failures, managing long transactions, and coordinating parallel tasks without external tools like Airflow or Temporal. Microsoft states that pg_durable runs entirely within PostgreSQL, requiring no additional infrastructure, and supports use cases like data ingestion pipelines, AI workflows, scheduled maintenance, and API integrations. The extension is available as a Debian package for PostgreSQL 17 and 18, with source code also provided for custom builds. It is designed to work with existing SQL workflows, wrapping complex logic in SQL functions or exposing HTTP endpoints when needed. The project is actively maintained and integrated into Microsoft’s cloud offerings, with plans for ongoing updates and broader compatibility.

Why It Matters

This release marks a significant shift in how complex, reliable data workflows are managed within PostgreSQL environments. By embedding durable execution capabilities directly into the database, it reduces reliance on external orchestrators, simplifies architecture, and enhances operational resilience. For organizations building data pipelines, AI models, or managing large-scale batch jobs, this approach can reduce failure points, improve auditability, and streamline workflows. It also exemplifies Microsoft’s strategy to bring compute closer to data, leveraging PostgreSQL’s extensibility to deliver robust, self-contained solutions.

Background

Traditionally, complex workflows in data engineering and AI pipelines rely on external orchestrators like Airflow, Temporal, or custom app-tier schedulers, often involving multiple systems, state management tables, and retry logic. This complexity increases operational overhead and introduces failure points. Microsoft’s open-source initiative with pg_durable aims to embed this logic within PostgreSQL itself, aligning with industry trends toward database-native workflow management. The concept of durable, checkpointed SQL functions has been explored in various forms, but this release formalizes and simplifies it for broader adoption, especially within Azure’s cloud ecosystem. The project builds on existing PostgreSQL extension capabilities and aims to make reliable, long-running processes more accessible to SQL-centric teams.

“pg_durable brings fault-tolerant, in-database workflow execution directly into PostgreSQL, eliminating the need for external orchestration tools.”

— Microsoft spokesperson

“By integrating pg_durable, HorizonDB provides a native solution for reliable, resumable workflows, enhancing performance and operational resilience.”

— Azure HorizonDB team

What Remains Unclear

It is still unclear how extensively pg_durable will be adopted outside of Microsoft’s Azure HorizonDB environment or how it will perform in large-scale, production workloads. Details about compatibility with other PostgreSQL distributions, long-term maintenance, and community support are still emerging. Additionally, the extent to which complex, branching workflows can be implemented within the SQL-based model remains to be tested in real-world scenarios.

What’s Next

Microsoft plans to continue developing pg_durable, with upcoming features likely to include broader support for complex control flow, integration with other orchestration tools, and expanded documentation. The extension is expected to be made available to the wider PostgreSQL community beyond Azure HorizonDB, potentially through official PostgreSQL extension repositories. Users and developers can expect ongoing updates, bug fixes, and performance improvements as adoption grows.

Key Questions

What is pg_durable and how does it work?

pg_durable is a PostgreSQL extension that enables defining fault-tolerant, resumable workflows using SQL. It checkpoints each step of a workflow, allowing it to resume after crashes or failures, all within the database.

Who can benefit from using pg_durable?

Backend and data engineers, DBAs, SREs, and teams building data pipelines or AI workflows that require durability, auditability, and resilience can benefit from pg_durable.

Does using pg_durable require external tools or infrastructure?

No. It runs as a PostgreSQL extension within the database, with no need for external orchestrators or additional infrastructure.

Are there any limitations to using pg_durable?

Yes. It is designed for SQL-based workflows; complex logic requiring arbitrary code or external systems may need wrapping or external orchestration. Also, it requires PostgreSQL 17 or 18 and cannot be used if the workflow involves many heterogeneous systems outside PostgreSQL.

Source: Hacker News