Home
Getting started
Bring your software with Docker
Core concepts
How-to guides
Monitor and debugging
Developer tools
Glossary
A pool is composed of one or several instances that can run a set of tasks. It is a way to reserve a number of computing nodes on Qarnot that will be running and ready to accept new tasks. When a task is started inside a pool, it is dispatched to an available slot which will not need to be rebooted. For this reason, the task will start quicker.
By default, each instance will run on its own compute node, and every instance will use the same profile.
Do not confuse a pool instance with a task instance. By default, pool instances are the computing nodes that will run the tasks in that pool.
Without Multi-slotting 1 pool instance = 1 computing node. With Multi-slotting 1 pool instance = 1 slot (you can refer to the Multi-slots section below for more information).
Here is a simple example where we create a pool of 4 instances, and run a quick Hello World from each one of those nodes.
import qarnot
# Connect to Qarnot
conn = qarnot.connection.Connection(client_token="<<<MY_SECRET_TOKEN>>>")
# Create a pool with 4 instances
pool = conn.create_pool("Hello world - pool", "docker-batch", 4)
# Submit the pool and wait 3 minutes for the nodes to start
pool.submit()
# Create a Hello World task of instances associated
# with the previously submitted pool
task = conn.create_task('helloworld', pool, 4)
# Print Hello World from each instance
task.constants['DOCKER_CMD'] = 'echo Hello World from node ${INSTANCE_ID}!'
# Run the task
task.run()
# Close the pool
pool.close()
We created a pool of 4 nodes. Then we created a task of 4 instances on that pool. This means that each of those 4 instances will run on one of the 4 nodes in the pool.
Let's say, for example, we created a task of 16 instances associated to the same pool. Then each one of the 4 nodes will run 4 task instances.
This gives the possibility to create an open compute session on Qarnot's computing platform and have control over how many task instances will run on however many nodes.
If your task is launched in a pool, the working directory won't be/job, it will be/job/<JOB_UUID>.For buckets associated with the task, the default resource path (where your input buckets' files and output bucket files are mounted) won't be/job, it will be/job/<JOB_UUID>.For resource buckets associated with the pool, the default resource path will be/job. Finally, it is not possible to associate a result bucket with a pool, but it is always possible to associate a result bucket with a task running on a pool.
Pools come with an interesting optional feature called Multi-slots. A multi-slot pool can run multiple tasks on a given node. As an example, it is possible to define a pool with 4 slots per node which means any node in the pool can host up to 4 different slots .i.e 4 tasks that run on the same node.
This feature is particularly interesting for software running on a single CPU core. Multi-slotting allows you to launch multiple instances of such software on a single computing node, optimizing the number of cores used.
There are a few important things to keep in mind about multi-slots:
An elastic pool is a pool capable of automatically managing the number of its slots to allocate them according to the needs of its tasks. It is well suited for multiple task computing, when you need a variable amount of task running in the same environment with periodic productivity peaks.
Below is a simple Hello World example to showcase elastic pools using the Python SDK.
import qarnot
# Connect to Qarnot
conn = qarnot.connection.Connection(client_token="<<<MY_SECRET_TOKEN>>>")
# Create a pool with 1 instance
elastic_pool = conn.create_pool("Hello world - elastic pool", "docker-batch", 1)
"""
Setup the elasticity of the pool
Ranging from 0 to 5 instances
Updated every 90 seconds
"""
elastic_pool.setup_elastic(
minimum_total_slots=0,
maximum_total_slots=5,
resize_period=90
)
# Submit the pool
elastic_pool.submit()
"""
Create a Hello World task associated
with the previously submitted pool
This task will be running only on 1 instance
"""
task_1 = conn.create_task('helloworld - 1', elastic_pool, 1)
# Print Hello World
task_1.constants['DOCKER_CMD'] = 'echo Hello World from node ${INSTANCE_ID}!'
# Run the task
task_1.run()
"""
Create a second Hello World task
with 5 instances this time
"""
task_2 = conn.create_task('helloworld - 5', elastic_pool, 5)
# Print Hello World from each instance
task_2.constants['DOCKER_CMD'] = 'echo Hello World from node ${INSTANCE_ID}!'
# Run the task
task_2.run()
# Close the pool
elastic_pool.close()
In the above example, we did the following:
This elasticity is useful for pools with a long lifetime, with tasks added during the cycle of their life. It prevents the inactivity of a part of the pool instances during a period of low activity, and also increases the productivity of the pool when there is a peak in tasks.
A pool is described by the following properties:
During the execution, the pool will have other properties:
Submitted: The pool has been submitted.PartiallyDispatched: Some instances have been dispatched to computing nodes, but none is running yet.FullyDispatched: All your instances have been dispatched, but none is running yet.PartiallyExecuting: Some instances of the pool are running, but some are still in queue.FullyExecuting: All the remaining instances of the pool are running (some may have completed already).Closing: The pool is currently in the closing process.Closed: The pool has been closed by the user.PendingDelete: The pool is currently in the deletion process.For more information on pools please consult the following articles: