MyQueue

MyQueue is a frontend for SLURM/PBS/LSF that makes handling of tasks easy. It has a command-line interface called mq with a number of Sub-commands and a Python interface for managing Workflows. Simple to set up: no system administrator or database required.

Features

• Easy task submission:
  • from the command line: mq submit <task> -R <cores>:<time>
  • from Python: myqueue.submit()
• Automatic restarting of timed-out/out-of-memory tasks with more time/cores
• Remembers your finished and failed tasks
• Powerful list command for monitoring
• Can be used together with Python venv’s (see Using a Python virtual environment)
• Folder-based Workflows

Contents:

• Installation
• Configure your queue
  • Guessing your configuration
  • Name of scheduler
  • Description of node types
  • MPI-run command
  • Parallel Python interpreter
  • Extra arguments for submit command
  • Maximum disk space
  • Notifications
• A quick start
• Documentation
  • Tasks
    • Python module
    • Function in a Python module
    • Python script
    • Shell command
    • Shell-script
  • Using a Python virtual environment
  • Resources
  • States
  • Examples
• How it works
  • Your queue
  • The daemon background process
  • More than one configuration file
• How to …
  • Delete lost tasks
  • Remove many tasks
  • Start from scratch
• Command-line interface
  • Sub-commands
    • Help: Show how to use this tool
    • List (ls): List tasks in queue
    • Submit: Submit task(s) to queue
    • Resubmit: Resubmit failed or timed-out tasks
    • Remove (rm): Remove or cancel task(s)
    • Info: Show detailed information about MyQueue or a task
    • Workflow: Submit tasks from Python script or several scripts matching pattern
    • Run: Run task(s) on local computer
    • Kick: Restart T and M tasks (timed-out and out-of-memory)
    • Modify: Modify task(s)
    • Init: Initialize new queue
    • Sync: Make sure SLURM/LSF/PBS and MyQueue are in sync
    • Completion: Set up tab-completion for Bash
    • Config: Create config.py file
    • Daemon: Interact with the background process
• Workflows
  • Simple example
    • Handling many tasks
  • Workflow script
    • Resources
    • Functions
    • Dependencies
  • Workflows with if-statements
  • Old workflow script
    • Examples
• Python API
  • Simple examples
  • Advanced example
  • API
    • Tasks
    • Commands
    • States
    • Resources
    • Queue
    • Schedulers
• Release notes
  • Next release
  • Version 21.8.0
  • Version 21.7.0
  • Version 21.4.2
  • Version 21.4.1
  • Version 21.4.0
  • Version 21.2.0
  • Version 21.1.0
  • Version 20.11.3
  • Version 20.11.2
  • Version 20.11.1
  • Version 20.11.0
  • Version 20.9.1
  • Version 20.9.0
  • Version 20.5.0
  • Version 20.1.2
  • Version 20.1.1
  • Version 20.1.0
  • Version 19.11.1
  • Version 19.11.0
  • Version 19.10.1
  • Version 19.10.0
  • Version 19.9.0
  • Version 19.8.0
  • Version 19.6.0
  • Version 19.5.0
  • Version 19.2.0
  • Version 19.1.0
  • Version 18.12.0
  • Version 0.1.0
• Development
  • Git repository
  • Getting help
  • Testing
  • Documentation
  • New release

Indices and tables

• Index
• Module Index
• Search Page