This blog discusses how enterprise job schedulers can fall short in cloud environments and how they should be built from the ground up, with the cloud in mind.
Often referred to as workload automation, enterprise job scheduling manages task execution on remote servers; resubmitting jobs when there are failures and launching jobs based on any dependencies in the sequence. Traditional enterprise job schedulers offer a graphical user interface (GUI) for this management, and have been used for a number of years for on-premise infrastructure.
Schedulers work in the background, managing multiple threads or pipelines simultaneously and require little thought or intervention. This modus operandi is possible as the number of available nodes and locations are known, and generally owned and stable. Therefore, they can merely be left to wait until work is allocated to them – organisations don’t have to fire them up, provision them or configure them, they can just use them as and when they have a job that needs to be completed.
This static ‘sit and wait’ approach, means the traditional schedulers are built in a similarly fixed way, requiring the simple configuration of how many nodes are available and where. While this is a perfectly suitable approach when the nodes are guaranteed, it does not necessarily translate to the cloud. In the cloud you have a wide range of different sources you can use, with fluctuating availabilities; therefore, a static scheduler can fall short in this dynamic environment.
For a scheduler to work in the cloud, it needs to be cloud native, fault tolerant and scalable.
While some traditional schedulers can work in the cloud, there will likely always be a ceiling to their performance. Instead of repurposing an existing tool, it is more advantageous to build a scheduler from the ground up, with the cloud in mind. With a truly cloud native tool, its potential will not be limited.
As instances are often pre-emptible in the cloud, a scheduler needs to be able respond to workers being lost or reclaimed by the provider. It needs to be able to replace workers in the middle of a job, without allowing the entire job to fail and restart. It also needs to be able to intelligently allocate workers based on given criteria, such as a particular region or cloud provider, so the organisation can maintain control over where the work is executed.
In order to meet the requirements of compute-intensive workloads at peak times, a scheduler needs to be scalable. It needs to be able to handle thousands of tasks simultaneously, across multiple regions, machine shapes and instance types, for organisations to make the most of the resources available to them.
So what’s the true value of cloud native scheduler?
With a scheduler that can operate in the cloud, customers can execute more workloads, at a greater scale than ever before. They can avoid the need to queue and the workload complexity, size and range does not need to be constrained. This means that more processes can be performed, insights can be uncovered faster, and previously unthinkable scale can be unlocked.
The YellowDog Scheduler is designed to work in the cloud. It is equipped to elegantly respond when compute conditions change and has been built to scale.
It also sits on top of, and works with, our Compute Service. Therefore, organisations can dynamically identify where the best workers are – using our intelligent provisioning templates and our Best Source of Compute algorithms – and use our Scheduler Service to automatically place the jobs in that preferential compute.
With an easy-to-use GUI, organisations don’t need to have touch points with multiple providers to execute their workloads. They can view and manage everything from one Platform.
You are seeing this because you are using a browser that is not supported. The YellowDog website is built using modern technology and standards. We recommend upgrading your browser with one of the following to properly view our website:Windows
Please note that this is not an exhaustive list of browsers. We also do not intend to recommend a particular manufacturer's browser over another's; only to suggest upgrading to a browser version that is compliant with current standards to give you the best and most secure browsing experience.