Chaining together tasks with jobs

‍

The following simple example showcases how a job can be used to chain together multiple Qarnot tasks.

Job explanation

In this use case we will link together two tasks. The workflow will go as follows:

• We will create a job named Hello World - job which will be associated with both tasks
• A first task, Hello World - Write, will write a simple Hello World! in a text file which will be stored inside an output bucket named write-output,
• A second task, Hello World - Read, will take as input the previous task's output bucket, read the content of the text file and print it to Tasq.

Below is the code to run this example on your end.

Python

import qarnot

# Connect to Qarnot
conn = qarnot.connection.Connection(client_token="<<<MY_SECRET_TOKEN>>>")

# Create and submit a job
job = conn.create_job('Hello World - job', useDependencies=True)
job.submit()

# Create a write task associated to the job
# This task will write Hello World in a .txt file
write_task = conn.create_task('Hello World - Write', 'docker-batch', 1, job=job)
write_task.constants['DOCKER_CMD'] = 'sh -c "echo Hello World ! > output.txt"'

# Create an output bucket for the write task
write_task_output = conn.create_bucket('write-output')
write_task.results = write_task_output

# Create a read task associated to the job
# This task will read the contents of output.txt and print it
read_task = conn.create_task('Hello World - Read', 'docker-batch', 1, job=job)
read_task.constants['DOCKER_CMD'] = 'cat output.txt'

# Use the write task's output bucket as an input
read_task.resources.append(write_task_output)

# Submit the write task 
write_task.submit()

# Submit the read task as being dependant of the write task
read_task.set_task_dependencies_from_tasks([write_task])
read_task.submit()

‍

#! Bash

#!/bin/bash

# Your infos
export QARNOT_CLIENT_TOKEN="<<<MY_SECRET_TOKEN>>>"

# =============== Job creation =============== #
# Create and submit a job
job_uuid=$(qarnot job create \
  --name "Hello World - job" \
  --is-dependant "true" \
  --format JSON | tee /dev/tty | jq -r '.Job')

# =============== Writer creation =============== #
# Create an output bucket for the write task
qarnot bucket create \
  --name write-output1 \
  --format JSON

# Create a write task associated to the job
# This task will write Hello World in a .txt file
write_uuid=$(qarnot task create \
  --profile docker-batch \
  --name "Hello World - Write" \
  --result "write-output1" \
  --job "${job_uuid}" \
  --instance 1 \
  --constants "DOCKER_CMD=sh -c 'echo Hello World ! > output.txt'"\
  --format JSON | tee /dev/tty | jq -r '.Task')

# =============== Reader creation =============== #
# Create a read task associated to the job
# This task will read the contents of output.txt and print it
qarnot task create \
  --name "Hello World - Read" \
  --resources "write-output1" \
  --job "${job_uuid}" \
  --dependents "${write_uuid}" \
  --profile docker-batch \
  --instance 1 \
  --constants "DOCKER_CMD=cat output.txt"\
  --format JSON

‍

Once you have launched the above script, make sure to go to Tasq to monitor the progress of the job.

Note that the second task will start running only after the first one has finished. This is guaranteed by using a job and setting the write task as a dependency of the read task (line 31).

Going further

A job can be described by the following properties:

• an UUID, to uniquely identify the job,
• a name, so you can easily identify your jobs when you use the API, one of our SDKs, or Tasq,
• a shortname, if you want to specify your own unique job identifier,
• a flag use dependencies, to specify that tasks inside the job can be dependent,
• a max wall time, to close the job after a certain amount of time,
• a boolean indicates if the job is in a pool. The pool open session where the job's tasks will run.

A job is only a container and needs tasqs to be attached to itself in order to be useful. If a job has no pool, the task depending on the job must have a profile.

During the execution, the job will have other properties:

• a state, among the following values. The order in which they are represented here also reflects the life cycle of the job.
  
  • Active: the job has been activated.
  • Terminating: the job is currently terminating.
  • ‍Completed: the job has been completed.
  • ‍Deleting: the job is currently in the deletion process.

When all the tasks of a job are done, or when the max wall time is reached, the job terminates.