Home >News >An Introduction to Meta Scheduling

An Introduction to Meta Scheduling


In our latest interview, Niall introduces us to the concept of Meta Scheduling: what it is, how it works and what the benefits are to our users. Watch to find out more.

What is Meta Scheduling?

So, we define Meta Scheduling – in the YellowDog context – as the orchestration and management of third-party schedulers by the YellowDog Scheduler.

Effectively, this means the YellowDog Scheduler becomes a coarse-grained scheduler, taking very, very big jobs, and the third-party tool handles the fine-grained scheduling. All of this takes place on a cluster that YellowDog provisions.

How Does Meta Scheduling Work?

The YellowDog Scheduler is able to apply standard actions to nodes at specific points or ‘events’, throughout the lifecycle of a cluster. The nodes can either be in the cloud or on-premise and actions can include ‘read a file’, ‘run a command’ and so on. Ultimately, the nodes can be set to have very specific content and scripting.

The application of these actions and events can be done either via our Java or Python SDK, or third-party applications, such as Ansible.

Essentially, the Scheduler becomes responsible for the orchestration and management of the cluster you want to build. Node actions can also be used for things such as the creation of file servers or DNS servers.

Throughout the lifecycle of the cluster you have, you can scale the nodes up and down as needed. Once your job is complete, having been managed by your third-party scheduler, the cluster is seamlessly discarded.

What’s the Benefit of Integrating with Third-Party Tools?

It really opens up the use of the YellowDog Scheduler to the tools that customers are used to using.

So, for example, if you use Slurm for your workloads, and you’re happy with the way it works, you can still use YellowDog to provision the cluster on which your workload runs.

This enables you to build clusters using the provisioning strategies we provide, which means you can create repeatable infrastructures, regardless of the provider(s) or regions you use.

This benefit of repeatability, using the tools you’re familiar with, is a really significant advantage for our customers.

What Are the Other Benefits of Meta Scheduling?

In addition to repeatable infrastructures, Meta Scheduling enables you to manage everything from a single place. There is no requirement to look elsewhere to view clusters running on different cloud providers or regions.

Meta Scheduling allows people to see everything in a consistent format and build up a ‘library’ of infrastructures.

It also enables them to build these infrastructures cost-effectively, as everything is viewed from a holistic perspective, rather than in siloes. With YellowDog, you’re able to have a standardised view on what you’re using and the costs in real time.

Share this: Latest News Articles
Mark Noctor joins YellowDog as Chief Commercial Officer 09.05.2024
YellowDog enables Nextflow users to go Hybrid and Multi-Cloud 08.11.2023
Developing virtual twin hearts with Hybrid Cloud 24.07.2023
YellowDog Please upgrade your browser

You are seeing this because you are using a browser that is not supported. The YellowDog website is built using modern technology and standards. We recommend upgrading your browser with one of the following to properly view our website:

Windows Mac

Please note that this is not an exhaustive list of browsers. We also do not intend to recommend a particular manufacturer's browser over another's; only to suggest upgrading to a browser version that is compliant with current standards to give you the best and most secure browsing experience.