Launching a computation with a SDK

‍

Qarnot provides SDKs to communicate with the cloud platform in several programming languages: Python, C# and Node.js.

Here is an introduction to the usage of the different SDKs to launch computations on the platform. You can refer to the Advanced use cases section of this guide to find more elaborate examples or to the individual documentation of each SDK (Python, C#, Node.js) for a deeper understanding of their specificities.

Prerequisites

Before starting a computation on Qarnot's cloud platform, you should have previously created an account. Create your account and retrieve your unique authentication token. You will need it to establish a secured connection to the platform.

Your authentication token is personal and should be kept safely as it allows connecting to your account and launch computations on your behalf. Be careful not to leak your token on public scripts or source codes. If needed, your token can be renewed in your account interface.

Install a SDK

Use the appropriate package manager in your project folder or virtual environment depending on the language you chose: pip3 install qarnot, dotnet add package QarnotSDK or npm install @qarnot/sdk.

When using Python, if you plan on sending large files to the platform, consider installing also the optional request-toolbelt dependency. Check the Qarnot Python SDK installation.

Connect to the cloud platform

Replace <<<MY_SECRET_TOKEN>>> with your personal authentication token.‍

Python

conn = qarnot.connection.Connection(client_token = '<<<MY_SECRET_TOKEN>>>')

‍

C#

var myQarnotToken = "<<<MY_SECRET_TOKEN>>>";
var connection = new QarnotSDK.Connection(myQarnotToken);

‍

Node.js

const Qarnot = new QarnotSDK({
    auth:'<<<MY_SECRET_TOKEN>>>'
});

Create a task

A task  describes a payload to execute on Qarnot cloud platform. The most basic parameters to describe a task are its name, the profile to use and the number of instances to start in parallel.

In these examples, all tasks are "Hello World" on 4 instances using the most basic profile: docker-batch.

Python

task = conn.create_task('Hello World in Python', 'docker-batch', 4)

‍

C#

var task = connection.CreateTask("Hello World in C#", "docker-batch", 4);

‍

Node.js

Qarnot.tasks.run({
    name: 'Hello World in Node.js',
    profile: 'docker-batch',
    instanceCount: 4

‍

To learn more about other profiles and how to use them, please visit the Choosing the right profile page.

Add constants

Through constants  you can parametrize the profile of your task. In these examples, we will use constants to express the Docker image to use and the Docker command to run.

‍

Python

task.constants['DOCKER_REPO'] = 'ubuntu'
task.constants['DOCKER_TAG'] = 'latest'
task.constants['DOCKER_CMD'] = 'echo Hello World from ${INSTANCE_ID}!'

‍

C#

task.Constants.Add("DOCKER_REPO", "ubuntu");
task.Constants.Add("DOCKER_TAG", "latest");
task.Constants.Add("DOCKER_CMD", "echo Hello World from node ${INSTANCE_ID}!");

‍

Node.js

    constants: [
      {
        key: 'DOCKER_REPO',
        value: 'ubuntu'
      }, 
      {
        key: 'DOCKER_TAG',
        value: 'latest'
      }, 
      {
        key: 'DOCKER_CMD',
        value: 'echo hello world from node ${INSTANCE_ID}!'
      } 
    ]
  });

Here we used the DOCKER_REPO and DOCKER_TAG constants for sake of comprehension but it was not necessary since they are filled in with the default values of the profile docker-batch defining the task. You can view all the constants and default values of your profiles in the Profiles page on Tasq.

For more examples on constants and Qarnot-specific environment variables like INSTANCE_ID and how to use them, refer to the Using Environment Variable article.

Launch the task

Finally, launch the task on Qarnot and print its results.

‍

Python

task.run()
print(task.stdout())

‍

C#

await task.RunAsync();
Console.WriteLine(await task.StdoutAsync());
Console.WriteLine("End of the run!");

‍

Node.js

// You are all set already!
// See full code below

The run method returns only when the task is completed. For long-running tasks, you might want to check the submit method.

Then run python3 hello_world.py, dotnet run or node hello-world.js depending on the language you chose and how you named your script. You will find below the entire scripts that will launch "Hello World" tasks.

‍

Python

import qarnot

conn = qarnot.connection.Connection(client_token = '<<<MY_SECRET_TOKEN>>>')

task = conn.create_task('Hello World in Python', 'docker-batch', 4)

task.constants['DOCKER_REPO'] = 'ubuntu'
task.constants['DOCKER_TAG'] = 'latest'
task.constants['DOCKER_CMD'] = 'echo Hello World from ${INSTANCE_ID}!'

task.run()
print(task.stdout())

‍

C#

using QarnotSDK;

namespace HelloWorldQarnot
{
    class Program
    {

        static async Task Main(string[] args)
        {
            var myQarnotToken = "<<<MY_SECRET_TOKEN>>>";
            var connection = new QarnotSDK.Connection(myQarnotToken);
            
            var task = connection.CreateTask("Hello World in C#", "docker-batch", 4);
            
            task.Constants.Add("DOCKER_REPO", "ubuntu");
            task.Constants.Add("DOCKER_TAG", "latest");
            task.Constants.Add("DOCKER_CMD", "echo hello world from node ${INSTANCE_ID}!"); // The docker command to be launch
            
            await task.RunAsync();
            Console.WriteLine(await task.StdoutAsync());

            Console.WriteLine("End of the run!");
        }
    }
}

‍

Node.js

const QarnotSDK = require('@qarnot/sdk');

const Qarnot = new QarnotSDK({
    auth:'<<<MY_SECRET_TOKEN>>>'
});

Qarnot.tasks.run({
    
    name: 'Hello World in Node.js',
    profile: 'docker-batch',
    instanceCount: 4,
    
    constants: [
      {
        key: 'DOCKER_REPO',
        value: 'ubuntu'
      }, 
      {
        key: 'DOCKER_TAG',
        value: 'latest'
      }, 
      {
        key: 'DOCKER_CMD',
        value: 'echo hello world from node ${INSTANCE_ID}!'
      } 
    ]
  });

Add inputs and retrieve outputs

On Qarnot cloud platform, data is managed with buckets. Find below a simple Hello World using data. For more examples on how to manage your data on Qarnot, please visit the Managing your data section.

‍

Python

import qarnot

conn = qarnot.connection.Connection(client_token = '<<<MY_SECRET_TOKEN>>>')

input_bucket = conn.create_bucket('my-input-bucket')
input_bucket.add_string("hello world !", "input.txt")

output_bucket = conn.create_bucket('my-output-bucket')

task = conn.create_task('Hello World with data in Python', 'docker-batch', 1)

task.resources.append(input_bucket)
task.results = output_bucket

task.constants['DOCKER_CMD'] = 'sh -c "cat input.txt | rev > output.txt"'

task.run('my-output')

with open('my-output/output.txt', 'r') as result_file:
    print(result_file.read())

‍

C#

using QarnotSDK;

namespace HelloWorldQarnot
{
    class Program
    {

        static async Task Main(string[] args)
        {
            var myQarnotToken = "<<<MY_SECRET_TOKEN>>>";
            var connection = new QarnotSDK.Connection(myQarnotToken);
            
            var input_bucket = await connection.CreateBucketAsync("my-input-bucket");
            await input_bucket.UploadStringAsync("hello world !", "input.txt");

            var output_bucket = await connection.CreateBucketAsync("my-output-bucket");
            
            var task = connection.CreateTask("Hello World with data in C#", "docker-batch", 1);

            task.Resources = new List<QAbstractStorage> {input_bucket};
            task.Results = output_bucket;
            task.Constants.Add("DOCKER_CMD","sh -c 'cat input.txt | rev > output.txt'");
            
            await task.RunAsync();
            Console.WriteLine(await output_bucket.DownloadStringAsync("output.txt"));
        }
    }
}

‍

Node.js

const QarnotSDK = require('@qarnot/sdk');

const Qarnot = new QarnotSDK({
    auth:'<<<MY_SECRET_TOKEN>>>'
});

Qarnot.tasks.run({
    
    name: 'Hello World in Node.js',
    profile: 'docker-batch',
    instanceCount: 4,
    
    constants: [
      {
        key: 'DOCKER_REPO',
        value: 'ubuntu'
      }, 
      {
        key: 'DOCKER_TAG',
        value: 'latest'
      }, 
      {
        key: 'DOCKER_CMD',
        value: 'echo hello world from node ${INSTANCE_ID}!'
      } 
    ]
  });

Related Article

For more information on Launching tasks please consult the following articles

• Core concept Tasks: which describes how tasks behave and their status
• Core concept Profiles: which describes how profiles works
• Choosing the right Profile
• Monitoring Resources
• Troubleshooting