MyQueue

MyQueue is a frontend for SLURM/PBS/LSF that makes handling of tasks easy. It has a command-line interface called mq with a number of Sub-commands and a Python interface for managing Workflows. Simple to set up: no system administrator or database required.

Features

• Easy task submission:

  • from the command line: mq submit <task> -R <cores>:<time>

  • from Python: myqueue.submit()

• Automatic restarting of timed-out/out-of-memory tasks with more time/cores

• Remembers your finished and failed tasks

• Powerful list command for monitoring

• Can be used together with Python venv‘s (see Using a Python virtual environment)

• Folder-based Workflows

Contents:

• Installation
• Configure your queue
  • Guessing your configuration
  • Name of scheduler
  • Description of node types
  • MPI-run command
  • Parallel Python interpreter
  • Extra arguments for submit command
  • Task weight
  • Notifications
• A quick start
• Documentation
  • Tasks
    • Python module
    • Function in a Python module
    • Python script
    • Shell command
    • Shell-script
  • Using a Python virtual environment
  • Resources
  • States
  • Examples
• Release notes
  • Next release
  • Version 23.4.0
  • Version 23.1.0
  • Version 22.12.0
  • Version 22.11.3
  • Version 22.11.2
  • Version 22.11.1
  • Version 22.11.0
  • Version 22.9.0
  • Version 22.7.1
  • Version 22.7.0
  • Version 22.6.0
  • Version 22.3.0
  • Version 22.1.0
  • Version 21.8.0
  • Version 21.7.0
  • Version 21.4.2
  • Version 21.4.1
  • Version 21.4.0
  • Version 21.2.0
  • Version 21.1.0
  • Version 20.11.3
  • Version 20.11.2
  • Version 20.11.1
  • Version 20.11.0
  • Version 20.9.1
  • Version 20.9.0
  • Version 20.5.0
  • Version 20.1.2
  • Version 20.1.1
  • Version 20.1.0
  • Version 19.11.1
  • Version 19.11.0
  • Version 19.10.1
  • Version 19.10.0
  • Version 19.9.0
  • Version 19.8.0
  • Version 19.6.0
  • Version 19.5.0
  • Version 19.2.0
  • Version 19.1.0
  • Version 18.12.0
  • Version 0.1.0
• How it works
  • Your queue
  • The daemon background process
  • More than one configuration file
• How to …
  • Delete lost tasks
  • Remove many tasks
  • Start from scratch
  • See what’s in your .myqueue/queue.sqlite3 file
• Command-line interface
  • Sub-commands
    • Help: Show how to use this tool
    • List (ls): List tasks in queue
    • Submit: Submit task(s) to queue
    • Resubmit: Resubmit done, failed or timed-out tasks
    • Remove (rm): Remove or cancel task(s)
    • Info: Show detailed information about MyQueue or a task
    • Workflow: Submit tasks from Python script or several scripts matching pattern
    • Kick: Restart T and M tasks (timed-out and out-of-memory)
    • Modify: Modify task(s)
    • Init: Initialize new queue
    • Sync: Make sure SLURM/LSF/PBS and MyQueue are in sync
    • Completion: Set up tab-completion for Bash
    • Config: Create config.py file
    • Daemon: Interact with the background process
• Workflows
  • Simple example
    • Handling many tasks
  • Workflow script
    • workflow()
    • run()
    • RunHandle
    • Resources
    • Functions
    • Dependencies
  • Workflows with if-statements
  • Old workflow script
    • create_tasks()
    • Examples
• Python API
  • Simple examples
  • Advanced example
  • API
    • submit()
    • Tasks
    • Commands
    • States
    • Resources
    • Queue
    • Schedulers
• Development
  • Git repository
  • Getting help
  • Testing
  • Documentation
  • New release

Indices and tables

• Index

• Module Index

• Search Page