Fetching logs

‍

Accessing the logs of your application is often necessary, whether it is to debug a script, or simply to follow a computation. In this page, we go through the different ways to access your logs with Qarnot.

Through our REST API

You can access your application standard output and standard error through our REST API. Please note that they are reported with a best effort logic. If the log rate is too high, some parts will be truncated and replaced by [...]. If you need precise logs, it is better to save them in a log file, as explained in the second section.

On Tasq

By default, the standard output and standard error of your application is available in Tasq, you can access them in the task page:

‍

You can try it with the following code:

#!/usr/bin/env python3

import qarnot

conn = qarnot.Connection(client_token="<<< PUT YOUR SECRET TOKEN >>>")
task = conn.create_task("test logs Tasq", "docker-batch", 1)

task.constants["DOCKER_CMD"] = 'sh -c "echo This work for both stdout...; \
                                       sleep 5; \
                                       echooo ... and stderr!"'

task.submit()

‍

This task will end up with an error state, which is normal because the last command echooo doesn’t exist. It is used here to show that, as expected, the standard error is also available.

We can notice that the error is also printed in the “STDOUT“ panel. As a general rule, every outputs of the docker container will be displayed in this panel, and Qarnot related errors will be displayed in the "STDERR" panel.

On the terminal

If necessary, you can also print the standard output and the standard error on your terminal with one of our SDKs:

#!/usr/bin/env python3

import qarnot
import sys

conn = qarnot.Connection(client_token="<<< PUT YOUR SECRET TOKEN >>>")
task = conn.create_task("test logs terminal", "docker-batch", 1)

task.constants["DOCKER_CMD"] = 'sh -c "echo This work for both stdout...; \
                                       sleep 5; \
                                       echooo ... and stderr!"'

task.submit()

done = False
while not done:
    # Wait for 5 seconds and returns True if the task is finished 
    done = task.wait(5)

    # Display fresh stdout / stderr
    sys.stdout.write(task.fresh_stdout())
    sys.stderr.write(task.fresh_stderr())

‍

In this case, the logs will be both displayed in Tasq and in the terminal, where the following lines will be printed:

0> This work for both stdout...
0> sh: 1: echooo: not found

‍

Please note that once again this task will end up with an error state because of the non-existence of the echooo command.

In a log file

The recommended way to access your logs is to write them in a log file stored in your output bucket and on your laptop, and that you can check during the task or at the end of it:

#!/usr/bin/env python3

import qarnot

conn = qarnot.Connection(client_token="<<< PUT YOUR SECRET TOKEN >>>")
task = conn.create_task("test logs file", "docker-batch", 1)

output_bucket = conn.create_bucket("output")
task.results = output_bucket

task.constants["DOCKER_CMD"] = 'sh -c "echo This work for both stdout... >> test.log 2>&1; \
                                       sleep 5; \
                                       echooo ... and stderr! >> test.log 2>&1"'

# Upload the content of /job to your output bucket every 5 seconds
task.snapshot(5)

task.submit()

done = False
while not done:
    # Download the content of the output bucket locally
    task.download_results('my_output_folder')
    # Wait for 5 seconds and returns True if the task is finished 
    done = task.wait(5)

‍

As expected, the task will once again end up with an error state because the echooo command doesn't exist.

You can notice that, in this case, the standard output and the standard error are not available anymore in Tasq. This is because they are both redirected to the file. If you want to have both Tasq display and the log file, you can use the following code (make sure your docker image contains the tee command before):

#!/usr/bin/env python3

import qarnot

conn = qarnot.Connection(client_token="<<< PUT YOUR SECRET TOKEN >>>")
task = conn.create_task("test logs file and Tasq", "docker-batch", 1)

output_bucket = conn.create_bucket("output")
task.results = output_bucket

task.constants["DOCKER_CMD"] = 'sh -c "echo This work for both stdout... 2>&1 | tee -a test.log; \
                                       sleep 5; \
                                       echooo ... and stderr! 2>&1 | tee -a test.log"'

# Upload the content of /job to your output bucket every 5 seconds
task.snapshot(5)

task.submit()

done = False
while not done:
    # Download the content of the output bucket locally
    task.download_results('my_output_folder')
    # Wait for 5 seconds and returns True if the task is finished 
    done = task.wait(5)

‍

This time, the task will succeed as the succesful tee command is the one determining the exit status of this little script.

Related article

For more information on Monitoring and debugging please consult the following articles

• Core concept Tasks: which describes how tasks behave and their status
• Monitoring Resources
• Monitor your computations with the Web UI
• Monitor your computations with a SDK
• Troubleshooting
• Error Codes