Tutorial: Using Blacklist and Whitelist with Regular Expressions Using Qarnot’s HPC platform, the whitelist / blacklist mechanism appears twice in the simulation set up. In this tutorial, we'll explore how to use it with regular expressions (regex) to allow file management during your simulations.
Understanding Snapshots and File Tailing Snapshots allow you to periodically save files from your running simulation to your results folder. This feature is crucial for ensuring data safety. File tailing allows you to choose which file to visualize during a simulation. This is particularly useful to monitor progress in real-time. Both of these features use the whitelist / blacklist mechanism, with regular expressions (regex) being the input parameters.
Setting Up Regex Filters To control which files are included in your snapshots and file tailing, you can use regex patterns in two ways:
Whitelist: Include only files matching the pattern Blacklist: Exclude files matching the pattern Whitelist / blacklist - Step-by-Step Guide Step 1: Access the Advanced Settings Navigate to the "Advanced Settings" section of your simulation setup. Locate the "Snapshot" subsection to set up snapshots Enter the desired frequency for snapshots in seconds. For example, enter "3600" for hourly snapshots. Locate the “Monitoring - File tailing” subsection to set up your file monitoring. Step 2: Choose Whitelist or Blacklist Decide whether you want to include (whitelist) or exclude (blacklist) specific files.
Step 3: Write Your Regex Pattern For whitelist:
^(file1|file2|.*\.txt)$ This pattern will include:
Exact matches: "file1" and "file2" Any file with a .txt extension For blacklist:
^(temp.*|.*\.log)$ This pattern will exclude:
Any file starting with "temp" Any file with a .log extension Step 4: Test Your Regex Before finalizing, test your regex pattern to ensure it matches the intended files. On this site , you can test your regexes.
Common Regex Patterns Here are some useful regex patterns for snapshot control:
Pattern
Tips for Optimal Use Be specific with your patterns to avoid unnecessary file transfers. Remember that frequent snapshots of large files can impact cluster performance. Use the provided preset regex buttons for common patterns if you're unsure. By mastering the use of regex with blacklist and whitelist, you can efficiently manage your simulation files, ensuring you capture the essential data while optimizing performance.
