PostsIntroducing Cleo 2.0: The Next Generation of Distributed Task Processing

Introducing Cleo 2.0: The Next Generation of Distributed Task Processing

3 min read
by Fofsinx

Discover Cleo, a powerful Redis-based distributed task queue system with advanced group processing capabilities, real-time monitoring, and sophisticated task orchestration.

🚀 Introducing Cleo 2.0

We're thrilled to announce the release of Cleo 2.0, a major update to our distributed task processing framework. This release brings significant improvements in task management, group processing, monitoring capabilities, and developer experience.

🎯 Key Features

Enhanced Task Decorators

The @task decorator now provides more granular control over task execution:

@task({
  id: "process-video",
  priority: TaskPriority.HIGH,
  queue: "media",
  group: "video-processing",
  backoff: {
    type: "exponential",
    delay: 1000
  },
  weight: 10,
  removeOnComplete: true
})
  • Smart Backoff Strategies: Choose between exponential and fixed backoff for retries
  • Task Weights: Prioritize tasks within groups
  • Auto-cleanup: Automatically remove completed tasks
  • Event Filtering: Precise event handling with taskId filtering

Intelligent Group Processing

Improved group processing with multiple strategies:

  • Round Robin: Fair distribution with last-processed time tracking
  • FIFO: Strict order processing with complete task history
  • Priority-based: Dynamic priority adjustment with group statistics

Comprehensive Monitoring

Enhanced observability with detailed metrics:

  • Worker Metrics: Track processing time, success rates, and resource usage
  • Group Statistics: Monitor group performance and task distribution
  • Task History: Complete audit trail of task lifecycle events
  • Real-time Events: Comprehensive event system for monitoring

Production-Ready Features

New capabilities for production deployments:

  • Multi-tenant Support: Redis key prefixing for tenant isolation
  • Enhanced Security: TLS support and secure Redis connections
  • Rate Limiting: Control processing rates at queue and group levels
  • Graceful Shutdown: Clean worker shutdown with task completion

💡 Developer Experience

We've focused on making Cleo more developer-friendly:

Simplified Configuration

cleo.configure({
  redis: {
    host: "redis.example.com",
    tls: true,
    keyPrefix: "tenant1:"
  },
  worker: {
    concurrency: 10,
    maxMemoryUsage: 1024 * 1024 * 512
  }
});

Intuitive Queue Classes

@QueueClass({
  defaultOptions: {
    queue: "notifications",
    maxRetries: 3
  },
  includeInherited: true
})
class NotificationService {
  // Methods automatically become tasks
}

Type-Safe Event Handling

queueManager.onTaskEvent(ObserverEvent.TASK_COMPLETED, 
  (taskId, status, data) => {
    console.log(`Task ${taskId} completed with result:`, data.result);
});

📈 Performance Improvements

  • Optimized Redis Operations: Reduced Redis operations per task
  • Smart Caching: Improved caching of task and group metadata
  • Efficient Event Handling: Filtered event propagation
  • Memory Management: Better memory usage in long-running workers

🔄 Migration Guide

Upgrading from Cleo 1.x is straightforward:

  1. Update dependencies to 2.0.0
  2. Replace deprecated retry options with new backoff configuration
  3. Update event handlers to use taskId filtering
  4. Review and update group processing strategies

🎯 What's Next?

We're already working on exciting features for future releases:

  • GraphQL API for task management
  • Kubernetes integration
  • Machine learning powered scheduling
  • Enhanced dashboard features
  • Additional processing strategies

🙏 Thank You

A huge thank you to our community for your feedback and contributions. Cleo 2.0 is a significant step forward in distributed task processing, and we're excited to see what you'll build with it.

Ready to get started? Check out our updated documentation:

Fofsinx

Fofsinx

Software Engineer | Writer | Designer