Introducing Cleo 2.0: The Next Generation of Distributed Task Processing
Discover Cleo, a powerful Redis-based distributed task queue system with advanced group processing capabilities, real-time monitoring, and sophisticated task orchestration.
🚀 Introducing Cleo 2.0
We're thrilled to announce the release of Cleo 2.0, a major update to our distributed task processing framework. This release brings significant improvements in task management, group processing, monitoring capabilities, and developer experience.
🎯 Key Features
Enhanced Task Decorators
The @task
decorator now provides more granular control over task execution:
@task({
id: "process-video",
priority: TaskPriority.HIGH,
queue: "media",
group: "video-processing",
backoff: {
type: "exponential",
delay: 1000
},
weight: 10,
removeOnComplete: true
})
- Smart Backoff Strategies: Choose between exponential and fixed backoff for retries
- Task Weights: Prioritize tasks within groups
- Auto-cleanup: Automatically remove completed tasks
- Event Filtering: Precise event handling with taskId filtering
Intelligent Group Processing
Improved group processing with multiple strategies:
- Round Robin: Fair distribution with last-processed time tracking
- FIFO: Strict order processing with complete task history
- Priority-based: Dynamic priority adjustment with group statistics
Comprehensive Monitoring
Enhanced observability with detailed metrics:
- Worker Metrics: Track processing time, success rates, and resource usage
- Group Statistics: Monitor group performance and task distribution
- Task History: Complete audit trail of task lifecycle events
- Real-time Events: Comprehensive event system for monitoring
Production-Ready Features
New capabilities for production deployments:
- Multi-tenant Support: Redis key prefixing for tenant isolation
- Enhanced Security: TLS support and secure Redis connections
- Rate Limiting: Control processing rates at queue and group levels
- Graceful Shutdown: Clean worker shutdown with task completion
💡 Developer Experience
We've focused on making Cleo more developer-friendly:
Simplified Configuration
cleo.configure({
redis: {
host: "redis.example.com",
tls: true,
keyPrefix: "tenant1:"
},
worker: {
concurrency: 10,
maxMemoryUsage: 1024 * 1024 * 512
}
});
Intuitive Queue Classes
@QueueClass({
defaultOptions: {
queue: "notifications",
maxRetries: 3
},
includeInherited: true
})
class NotificationService {
// Methods automatically become tasks
}
Type-Safe Event Handling
queueManager.onTaskEvent(ObserverEvent.TASK_COMPLETED,
(taskId, status, data) => {
console.log(`Task ${taskId} completed with result:`, data.result);
});
📈 Performance Improvements
- Optimized Redis Operations: Reduced Redis operations per task
- Smart Caching: Improved caching of task and group metadata
- Efficient Event Handling: Filtered event propagation
- Memory Management: Better memory usage in long-running workers
🔄 Migration Guide
Upgrading from Cleo 1.x is straightforward:
- Update dependencies to 2.0.0
- Replace deprecated retry options with new backoff configuration
- Update event handlers to use taskId filtering
- Review and update group processing strategies
🎯 What's Next?
We're already working on exciting features for future releases:
- GraphQL API for task management
- Kubernetes integration
- Machine learning powered scheduling
- Enhanced dashboard features
- Additional processing strategies
🙏 Thank You
A huge thank you to our community for your feedback and contributions. Cleo 2.0 is a significant step forward in distributed task processing, and we're excited to see what you'll build with it.
Ready to get started? Check out our updated documentation: