Community-supported integrations.
View all tagsThe Anthropic integration allows you to easily interact with the Anthropic REST API using the Anthropic Python API to build AI steps into your Dagster pipelines. You can also log Anthropic API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.
With the Census integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful.
The Chroma library allows you to easily interact with Chroma's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets.
With the Cube integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API.
The community-supported DingTalk package provides an integration with DingTalk.
The Evidence library offers a component to easily generate dashboards from your Evidence project.
The community-supported dagster-contrib-gcp package provides integrations with Google Cloud Platform (GCP) services.
The Gemini library allows you to easily interact with the Gemini REST API using the Gemini Python API to build AI steps into your Dagster pipelines. You can also log Gemini API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption.
The community-supported Nomad package provides an integration with HashiCorp Nomad.
A package for integrating HashiCorp Vault into Dagster so that you can securely manage tokens and passwords.
The community-supported Hex package provides an integration with Hex.
With this integration you can trigger Hightouch syncs and monitor them from within Dagster. Fine-tune when Hightouch syncs kick-off, visualize their dependencies, and monitor the steps in your data activation workflow.
This library provides I/O managers for reading and writing Apache Iceberg tables. It also provides a Dagster resource for accessing Iceberg tables.
The Java Pipes client provides a Java implementation of the Dagster Pipes protocol that can be used to orchestrate data processing pipelines written in Java from Dagster, while receiving logs and metadata from the Java application.
By integrating with lakeFS, a big data scale version control system, you can leverage the versioning capabilities of lakeFS to track changes to your data. This integration allows you to have a complete lineage of your data, from the initial raw data to the transformed and processed data, making it easier to understand and reproduce data transformations.
The Meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer specification.
An integration with Microsoft Teams to post messages to MS Teams from any Dagster op or asset.
The community-supported Modal package provides an integration with Modal.
The community-supported MSSQL BCP package is a custom Dagster I/O manager for loading data into SQL Server using the BCP utility.
Leverage the Not Diamond resource to easily determine which LLM provider is most appropriate for your use case.
The community-supported obstore package provides an integration with obstore, providing three lean integrations with object stores, ADLS, GCS & S3.
With this integration you can create a Open Metadata service to ingest metadata produced by the Dagster application. View the Ingestion Pipeline running from the Open Metadata Service Page.
Patito is a data validation framework for Polars, based on Pydantic.
The Perian integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment.
The Polars integration allows using Polars eager or lazy DataFrames as inputs and outputs with Dagster’s assets and ops. Type annotations are used to control whether to load an eager or lazy DataFrame. Lazy DataFrames can be sinked as output. Multiple serialization formats (Parquet, Delta Lake, BigQuery) and filesystems (local, S3, GCS, …) are supported.
The Qdrant library lets you integrate Qdrant's vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster.
The community-supported Ray package allows orchestrating distributed Ray compute from Dagster pipelines.
The Rust Pipes client allows full observability into your Rust workloads when orchestrating through Dagster.
Connect Dagster to Secoda and see metadata related to your Dagster assets, asset groups and jobs right in Secoda. Simplify your team's access, and remove the need to switch between tools.
The community-supported Teradata package provides an integration with Teradata Vantage.
The Weaviate library allows you to easily interact with Weaviate's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets.
Use Dagster and Weights & Biases (W&B) to orchestrate your MLOps pipelines and maintain ML assets.