An Introduction to Meta Scheduling

In our latest interview, Niall introduces us to the concept of Meta Scheduling: what it is, how it works and what the benefits are to our users. Watch to find out more.

What is Meta Scheduling?

So, we define Meta Scheduling – in the YellowDog context – as the orchestration and management of third-party schedulers by the YellowDog Scheduler.

Effectively, this means the YellowDog Scheduler becomes a coarse-grained scheduler, taking very, very big jobs, and the third-party tool handles the fine-grained scheduling. All of this takes place on a cluster that YellowDog provisions.

How Does Meta Scheduling Work?

The YellowDog Scheduler is able to apply standard actions to nodes at specific points or ‘events’, throughout the lifecycle of a cluster. The nodes can either be in the cloud or on-premise and actions can include ‘read a file’, ‘run a command’ and so on. Ultimately, the nodes can be set to have very specific content and scripting.

The application of these actions and events can be done either via our Java or Python SDK, or third-party applications, such as Ansible.

Essentially, the Scheduler becomes responsible for the orchestration and management of the cluster you want to build. Node actions can also be used for things such as the creation of file servers or DNS servers.

Throughout the lifecycle of the cluster you have, you can scale the nodes up and down as needed. Once your job is complete, having been managed by your third-party scheduler, the cluster is seamlessly discarded.

What’s the Benefit of Integrating with Third-Party Tools?

It really opens up the use of the YellowDog Scheduler to the tools that customers are used to using.

So, for example, if you use Slurm for your workloads, and you’re happy with the way it works, you can still use YellowDog to provision the cluster on which your workload runs.

This enables you to build clusters using the provisioning strategies we provide, which means you can create repeatable infrastructures, regardless of the provider(s) or regions you use.

This benefit of repeatability, using the tools you’re familiar with, is a really significant advantage for our customers.

What Are the Other Benefits of Meta Scheduling?

In addition to repeatable infrastructures, Meta Scheduling enables you to manage everything from a single place. There is no requirement to look elsewhere to view clusters running on different cloud providers or regions.

Meta Scheduling allows people to see everything in a consistent format and build up a ‘library’ of infrastructures.

It also enables them to build these infrastructures cost-effectively, as everything is viewed from a holistic perspective, rather than in siloes. With YellowDog, you’re able to have a standardised view on what you’re using and the costs in real time.