What is TacoQ?

TacoQ is a task queue system that allows you to schedule tasks to be executed asynchronously in workers outside your application. (e.g. in a different container).

Core Concepts: Rapid-fire Overview

To properly use TacoQ, it is recommended that you have a basic understanding of the core concepts of task queues. This section will provide a rapid-fire overview of the most important topics so you can hit the ground running.

Tasks

Tasks are a unit of work that can be scheduled to be executed asynchronously. Here are some core properties of tasks that you should know about:

• Tasks have a worker_kind and a task_kind, which are used to identify which set of workers should execute each task (respectively). If you are familiar with message brokers, you can think of them as routing keys.
• Tasks have a priority value, which is used to sort tasks and determine which ones to assign to workers first. Once a task reaches a worker, its priority will no longer matter and it will be executed no matter what its priority is.
• When generated, tasks get an id, which is a unique identifier for the task that can be later used to retrieve the task's status and results.

Message Broker

The message broker carries task assignments, status updates, and results across the entire TacoQ system. The broker is also responsible for routing the tasks to the correct set of workers based on its properties.

TacoQ uses RabbitMQ as its message broker.

Workers

Workers receive task assignments from the message broker, execute them, and then send the result back through the message broker so they can be stored in the database.

TacoQ uses Postgres as its database.

Publisher Client

To publish a task, you can use a publisher client that abstracts away the message broker and allows you to send a task to the workers and database.

The publisher does not need to be a dedicated application. Instead, any service can embed the publisher in its code to publish tasks, even workers.

Relay

The relay is a unique concept to TacoQ. It is a service that acts as TacoQ's core engine, and has the following characteristics:

• It abstracts away the data store where tasks are registered behind a REST API.
• It consumes messages from the broker to continuously update the data store.

The relay is horizontally scalable and can be easily replicated.

What makes TacoQ different?

TacoQ aims to differentiate itself from other task queues by providing the following features:

• Multi-language interoperability (e.g. schedule a CPU-intensive task to be executed by a Rust worker from a Python application, generate a cute PDF in a Javascript worker from a Rust API).
• Modern SDKs for popular languages like Python, supporting async functions, pydantic, having a type safe API, built-in hot reloading, and more.
• Integration via REST API, allowing languages without a dedicated SDK to easily schedule and retrieve task results.
• Great developer experience with a focus on programmer ergonomics, good documentation, and explainability (for example, with built-in observability).

You can learn more about TacoQ's architecture and design in the System Architecture section.