Release notes

(version numbers can be interpreted as year.month.bug-fix-release-number)

Next release

See https://gitlab.com/myqueue/myqueue/-/merge_requests

Version 26.3.0

  • Calling a Python function from a workflow (run(function=...)) will no longer write the return value to a file called <task-name>.result in the JSON format. Use run(function=..., serialize=True) to get the old behavior

Version 25.11.0

  • TAB-completions for node names: mq submit ... -R 40:<TAB>.

  • Added a tool for comparing jobs in two similar folders: python -m myqueue.compare folder1/ folder2/.

Version 25.4.0

  • IMPORTANT: The meaning of a resource specification like 24:1:1h has changed! With version 24.10.0 and earlier, 24 cores and processes would be allocated, but mpiexec would not be called. The equivalent specification with new MyQueue is s:24:1h. From now on, 24:1:1h will allocate 24 cores and one process. See Resources and Using MPI or not?.

  • Resource specifications can now specify if mpiexec should be called or not. See Resources and Using MPI or not?.

  • Renamed default branch from master to main.

  • mq sync will now only remove your own tasks (in case you are sharing folders with other users).

Version 24.10.0

  • Introduced an environment variable MYQUEUE_PREAMBLE. Its value will be inserted at the beginning of the script that is submitted. See Preamble.

  • Number of GPUs per node can now be part of a resource specification. See Resources and myqueue.workflow.run().

  • Automatically adds BASH-completion line to $VIRTUAL_ENV/bin/activate script.

Version 24.9.0

  • We now check that the task-folder is writable.

  • Output from mq list is now shortened to fit the text-terminal width: verylongword -> ver…ord.

  • Local scheduler can now run more than one task at a time.

  • New nodename configuration variable: Description of node types.

  • Setting nodename to the empty string will skip --partition=nodename on SLURM.

Version 24.5.1

Version 24.5.0

  • SLURM jobs will now be submitted with --cpus-per-task set to the correct value.

  • The MYQUEUE_TASK_ID environment variable will now be set to the task ID so that running tasks can inspect it.

  • New configuration variable for Serial Python interpreter.

Version 24.1.0

  • Drop support for Python 3.7.

  • Move from setup.py to pyproject.toml.

  • The resubmit command will now remove the old task. Use --keep to get the old behavior.

  • Restarted (OOM’ed or timed out) tasks will now be cleared from the queue.

  • Improved parsing or .err files.

Version 23.4.0

  • Fixed broken tab-completion for names and ids: mq ls -i <tab> (MR: !132).

  • Failed dependencies would block everthing in a workflow. Should be fixed in MR: !135.

Version 23.1.0

  • Fixed a problem with dependencies inside subfolders (issue: #51).

Version 22.12.0

Version 22.11.3

Version 22.11.2

Version 22.11.1

Version 22.11.0

Important

No more <task-name>.state files. MyQueue will only know the state af a task if it is listed in your queue. There are two exceptions to this rule:

  1. If a task is set to create some files like here:

    def workflow():
    run(..., creates=['file1.abc', 'file2.xyz'], ...)

    then MyQueue will consider the task done if those files exist. See myqueue.workflow.run().

  2. If a task is a Python function like here:

    def workflow():
    run(function=func, args=[...], name='abc', ...)

    then MyQueue will consider the task done if the result file exists (in this case abc.result). See myqueue.caching.json_cached_function.

  • Your queue is no longer stored in a .myqueue/queue.json file. Instead, it is now in a sqlite3 file in .myqueue/queue.sqlite3. Your old JSON file will automatically be migrated to the new format.

  • Removed the mq run command (it may return later: issue: #44).

  • Calling a Python function from a workflow (run(function=...)) will now write the return value to a file called <task-name>.result in the JSON format. Previously the return value was written to the .state file.

  • Removing tasks part of a workflow now needs a --force (as MyQueue will no longer know the states of such tasks).

  • Most commands have been sped up by delaying import of rich and networkx.

  • The resubmit command will no longer remove the old task. Use --remove to get the old behavior.

  • The Resources of a task now includes a task-weight. This can be used to limit the number of running tasks. See more here: Task weight.

Version 22.9.0

  • Hitting CTRL-C in the middle of submitting jobs is now safe.

Version 22.7.1

Version 22.7.0

  • Tasks will no longer activate a virtual environment if a venv/ folder is found in one of the parent folders.

  • Tasks submitted from an activated virtual environment will now activate that environment when the job starts running.

  • Better error message when sbatch/qsub/bsub fails.

  • Improved parsing of stderr from failed jobs.

  • Depth first submit ordering. A workflow with an A task and a B task where B depends on A would previously run all the A tasks and then all the B tasks. The order will now be A, B in the first folder, then A, B in the next folder and so on.

Version 22.6.0

  • Fixed bug related to several users having write access to the same .myqueue/ folder.

Version 22.3.0

  • There is now one background daemon per user. This will allow several users to share a .myqueue/ folder.

Version 22.1.0

  • The List (ls): List tasks in queue command can now list several folders instead of, as previously, only one. They must all belong to the same .myqueue/ folder though.

Version 21.8.0

  • The simple “local” scheduler is now feature complete. See Name of scheduler.

  • The mpi_implementations configuration option is no longer needed and has been deprecated.

  • MyQueue no longer tries to keep track of all your .myqueue/ folders. Consequently, the --all option has been removed from the list, kick and sync commands.

  • There is a new mq info --all [folder] command that will searsch for your .myqueue/ folders and print a status line for each.

  • There is now one background daemon per .myqueue/ folder. See The daemon background process.

Version 21.7.0

  • Email notifications: mq modify ... -N dA. See Modify: Modify task(s) and Notifications.

  • You can now use mq info to get information about your MyQueue installation:

    • version

    • location of the source code

    • location of .myqueue/ folder

    • configuration

Version 21.4.2

  • Make things work with Python 3.7.

Version 21.4.1

  • Backwards compatibility fix.

Version 21.4.0

  • For workflow tasks, name.done and name.FAILED files have now been replaced by a name.state file. MyQueue will still read the old files, but no longer write them.

Version 21.2.0

Version 21.1.0

Version 20.11.3

  • Bugfix: LSF-backend fixes.

Version 20.11.2

  • Bugfix: Don’t remove FAILED-files in dry-run mode.

Version 20.11.1

  • Fix “workflow target” bug and MQ: comments bug.

Version 20.11.0

Version 20.9.1

  • Fix workflow+openmpi issue.

Version 20.9.0

  • Red error messages.

  • Progress-bar.

Version 20.5.0

  • Using pytest for testing.

  • Simple local queue for use without a real scheduler.

  • New extra_args configuration parameter (Extra arguments for submit command). Replaces, now deprecated, features and reservation parameters.

  • Use python3 -m myqueue.config to auto-configure your system.

  • Memory usage is now logged.

Version 20.1.2

  • Bug-fix release with fix for single-process tasks (see Resources).

Version 20.1.1

  • This is the version submitted to JOSS.

Version 20.1.0

  • New shortcuts introduced for specifying States: a is qhrd and A is FCMT.

Version 19.11.1

Version 19.11.0

  • Small bugfixes and improvements.

Version 19.10.1

  • Added support for LSF scheduler.

  • Added --max-tasks option for submit and workflow commands.

Version 19.10.0

  • Shell-style wildcard matching of task names and error messages is now possible:

    $ mq ls -n "*abc-??.py"
$ mq resubmit -s F -e "*ZeroDivision*"

  • Three new Command-line interface options: mq -V/--version, mq ls --not-recursive and mq submit/workflow -f/--force.

  • All task-events (queued, running, stopped) are now logged to ~/.myqueue/log.csv. List tasks from log-file with:

    $ mq ls -L ...

Version 19.9.0

  • New -C option for the mq ls command for showing only the count of tasks in the queue:

    $ mq ls -C
running: 12, queued: 3, FAILED: 1, total: 16

  • A background process will now automatically kick your queues every ten minutes.

  • Project moved to a new myqueue group: https://gitlab.com/myqueue/myqueue/

Version 19.8.0

  • The module:function syntax has been changed to module@function.

  • Arguments to tasks are now specified like this:

    $ mq submit [options] "<task> arg1 arg2 ..." [folder1 [folder2 ...]]

  • New run command:

    $ mq run [options] "<task> arg1 arg2 ..." [folder1 [folder2 ...]]

Version 19.6.0

  • Tasks will now activate a virtual environment if a venv/ folder is found in one of the parent folders. The activation script will be venv/activate or venv/bin/activate if venv/activate does not exist.

Version 19.5.0

Version 19.2.0

  • Fix test-suite.

Version 19.1.0

  • Recognizes mpiexex variant automatically.

  • New “detailed information” subcommand.

Version 18.12.0

  • The restart parameter is now an integer (number of restarts) that counts down to zero. Avoids infinite loop.

Version 0.1.0

Initial release.