Sign in

Repository Index

47,133 repositories indexed

Tags: Data pipelines and messaging
Tags

A repository shows when it has any selected Tag.

Clear Tags
Filters

Filters

Clear all filters

1–24 of 99 · Most stars

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

  • python
  • luigi
  • orchestration-framework
  • scheduling

The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

  • metadata
  • datadiscovery
  • dataquality
  • data-profiling

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

  • workflow-schedule
  • azkaban
  • airflow
  • task-scheduler

Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

  • chat
  • chat-application
  • chatapp
  • go

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

  • data-integration
  • high-performance
  • offline
  • real-time

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Generic command line non-JVM Apache Kafka producer and consumer

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

  • bigquery
  • copy-database
  • data-ingestion
  • data-integration

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

  • java
  • bitset
  • roaring-bitmaps
  • roaringbitmap