Tutorial: Using Blacklist and Whitelist with Regular Expressions

Using Qarnot’s HPC platform, the whitelist / blacklist mechanism appears twice in the simulation set up. In this tutorial, we'll explore how to use it with regular expressions (regex) to allow file management during your simulations.

‍

Understanding Snapshots and File Tailing

Snapshots allow you to periodically save files from your running simulation to your results folder. This feature is crucial for ensuring data safety.
File tailing allows you to choose which file to visualize during a simulation. This is particularly useful to monitor progress in real-time.

Both of these features use the whitelist / blacklist mechanism, with regular expressions (regex) being the input parameters.

‍

Setting Up Regex Filters

To control which files are included in your snapshots and file tailing, you can use regex patterns in two ways:

Whitelist: Include only files matching the pattern
Blacklist: Exclude files matching the pattern

‍

Whitelist / blacklist - Step-by-Step Guide

Step 1: Access the Advanced Settings

Navigate to the "Advanced Settings" section of your simulation setup.
Locate the "Snapshot" subsection to set up snapshots
1. Enter the desired frequency for snapshots in seconds.
2. For example, enter "3600" for hourly snapshots.
Locate the “Monitoring - File tailing” subsection to set up your file monitoring.

‍

Step 2: Choose Whitelist or Blacklist

Decide whether you want to include (whitelist) or exclude (blacklist) specific files.

‍

Step 3: Write Your Regex Pattern

For whitelist:

^(file1|file2|.*\.txt)$

This pattern will include:

Exact matches: "file1" and "file2"
Any file with a .txt extension

For blacklist:

^(temp.*|.*\.log)$

This pattern will exclude:

Any file starting with "temp"
Any file with a .log extension

‍

Step 4: Test Your Regex

Before finalizing, test your regex pattern to ensure it matches the intended files. On this site, you can test your regexes.

‍

Common Regex Patterns

Here are some useful regex patterns for snapshot control:

Pattern

‍

Tips for Optimal Use

Be specific with your patterns to avoid unnecessary file transfers.
Remember that frequent snapshots of large files can impact cluster performance.
Use the provided preset regex buttons for common patterns if you're unsure.

By mastering the use of regex with blacklist and whitelist, you can efficiently manage your simulation files, ensuring you capture the essential data while optimizing performance.

‍