@freyablekman may I ask a practice question? How is the meta analysis with more data performed when the primary analysis has been conducted on various distributed compute environments in the first place?

I am asking, because we (some people from the #Snakemake community) recently had an online workshop with representatives of CERN. But I do not recall the meta analysis challenge being brought up. (But I might be mistaken.) And the Snakemake Hackathon is upcoming, where we want to invest some time into CERN matters.

Beep, Beep - I am your friendly #Snakemake release announcement bot.

There is a new release of the Snakemake executor for #SLURM on #HPC systems. Its version now is 2.3.1!

Give us some time, and you will automatically find the plugin on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainers here on Mastodon!
@rupdecat and @johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake-executor-plugin-slurm/issues.

See https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.3.1 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* Add direct dependency on PyYAML: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/414

Beep, Beep - I am your friendly #Snakemake release announcement bot.

There is a new release of the Snakemake executor for #SLURM on #HPC systems. Its version now is 2.3.0!

Give us some time, and you will automatically find the plugin on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainers here on Mastodon!
@rupdecat and @johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake-executor-plugin-slurm/issues.

See https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.3.0 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

* attempt for detailed error reporting for job-steps: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/327
* automated partition writeout: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/410
* excluding broken nodes from further job submission: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/411

Beep, Beep - I am your friendly #Snakemake release announcement bot.

There is a new release of the Snakemake executor for #SLURM on #HPC systems. Its version now is 2.3.0!

Give us some time, and you will automatically find the plugin on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainers here on Mastodon!
@rupdecat and @johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake-executor-plugin-slurm/issues.

See https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.3.0 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

* attempt for detailed error reporting for job-steps: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/327
* automated partition writeout: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/410
* excluding broken nodes from further job submission: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/411

I want to reach out: I have this pending release for the SLURM executor (https://github.com/snakemake/snakemake-executor-plugin-slurm/pull/412 ). It implements better error feedback (in case of hardware failures and otherwise). It would need some thorough checking, and I cannot provoke too many hardware failures. Is anyone working on an older cluster? 😉

Feedback (as issues) would be appreciated. Also nice, if you tell me it is working, here.

#Snakemake #HPC #SLURM #ReproducibleComputing

On a similar note: there is another (draft) PR. The #SLURM executor plugin for #Snakemake is capable of respecting partition definitions since v. 2.

I had the notion, that this is rather difficult to set this up manually and wrote a little command line helper. It queries the SLURM config and writes out a preliminary partition configuration template. This still requires manual adaptation, I'm afraid.

A small step forward as it requires both an understanding of Snakemake and your local SLURM setup. The world is as is it is, the phantasy of admin teams is unlimited and a one-fits-all solution is not on the horizon.

Still, if you want to try it out and provide feedback, this would be very much appreciated! All suggestions are welcome!

I want to reach out: I have this pending release for the SLURM executor (https://github.com/snakemake/snakemake-executor-plugin-slurm/pull/412 ). It implements better error feedback (in case of hardware failures and otherwise). It would need some thorough checking, and I cannot provoke too many hardware failures. Is anyone working on an older cluster? 😉

Feedback (as issues) would be appreciated. Also nice, if you tell me it is working, here.

#Snakemake #HPC #SLURM #ReproducibleComputing

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.15.0!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.15.0 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

* expose logger plugin handlers via get_log_handlers: https://github.com/snakemake/snakemake/issues/3897

𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* catch OSError raised during benchmarking: https://github.com/snakemake/snakemake/issues/3931
* ensure that fetching of remote git repo updates properly handles commit ids: https://github.com/snakemake/snakemake/issues/3933

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.15.0!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.15.0 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬

* expose logger plugin handlers via get_log_handlers: https://github.com/snakemake/snakemake/issues/3897

𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* catch OSError raised during benchmarking: https://github.com/snakemake/snakemake/issues/3931
* ensure that fetching of remote git repo updates properly handles commit ids: https://github.com/snakemake/snakemake/issues/3933

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.14.6!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.14.6 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* create local clone of git repos for source files from hosting providers: https://github.com/snakemake/snakemake/issues/3643
* create potentially missing .snakemake folder in case of very long command lines for spawned jobs: https://github.com/snakemake/snakemake/issues/3894
* make ilp solver enumeration lazy: https://github.com/snakemake/snakemake/issues/3900
* Prevent broken report_href links by using deterministic report IDs with fixed prefix length: https://github.com/snakemake/snakemake/issues/3889
* refactor LoggerManager setup and scope: https://github.com/snakemake/snakemake/issues/3851
* yield proper error message in case a local git source file is not retrievable: https://github.com/snakemake/snakemake/issues/3892

𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧

* explain how to pass nested config via CLI: https://github.com/snakemake/snakemake/issues/3885

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.14.6!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.14.6 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* create local clone of git repos for source files from hosting providers: https://github.com/snakemake/snakemake/issues/3643
* create potentially missing .snakemake folder in case of very long command lines for spawned jobs: https://github.com/snakemake/snakemake/issues/3894
* make ilp solver enumeration lazy: https://github.com/snakemake/snakemake/issues/3900
* Prevent broken report_href links by using deterministic report IDs with fixed prefix length: https://github.com/snakemake/snakemake/issues/3889
* refactor LoggerManager setup and scope: https://github.com/snakemake/snakemake/issues/3851
* yield proper error message in case a local git source file is not retrievable: https://github.com/snakemake/snakemake/issues/3892

𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧

* explain how to pass nested config via CLI: https://github.com/snakemake/snakemake/issues/3885

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.14.3!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.14.3 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* ensure that queue input jobs wait if the queue is still empty upon job creation: https://github.com/snakemake/snakemake/issues/3866
* fix issues with cyclic dependencies when using the update and before_update flag: https://github.com/snakemake/snakemake/issues/3857

The last blog post I wrote was about Life Science Support on HPC clusters. Honestly? It was more of a rant. Not a good blog post.

So, someone suggested I delete it, which I did. It took me a long time to recover. Now, I have re-written this blog post. I think it is better. I weighted every phrase. It still has some rant flavour.

Here it is: https://blogs.fediscience.org/rupture-de-catenaire/2025/12/15/empowering-life-scientists-what-hpc-support-should-look-like/

The upcoming articles will have more of an example character. But I still will not be able to update on a regular basis.

#Bioinformatics #HPC #ReproducibleComputing #Snakemake #Nextflow

@tkuhn perhaps, I should do a few #nanopub onboarding sessions embedded with my planned #Snakemake Hackathon and the Snakemake implementation for nanopubs in a workshop (for which I still need to do some work) next year.

I'm not sure, I followed you with your remark - is this better? https://w3id.org/np/RAADhF-xHWOeATa1pKZelZ2Jwbpjxrxn8Md-yKVRxdHho

Beep, Beep - I am your friendly #Snakemake release announcement bot.

There is a new release of the Snakemake executor for #SLURM on #HPC systems. Its version now is 2.0.2!

Give us some time, and you will automatically find the plugin on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainers here on Mastodon!
@rupdecat and @johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake-executor-plugin-slurm/issues.

See https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.0.2 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* partition cluster selection: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/385

@Dutch_Reproducibility_Network

In fact, I am a #Snakemake co-maintainer and teacher. I was not aware of WorkflowHub - and that was an omission on my part. We actually support and favour this kind of registration: https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#uploading-workflows-to-workflowhub

In my original post, I also neglected to mention the integration of WorkflowHub with #RO-Crate and in turn, the integration of RO-Crates with nanopubs. I am actively working on a better support for #nanopub and RO-Crates with @fbartusch. The question, how I teach that stands: The #HPC world (at least my bubble) is not really supportive for #ReproducibleComputing . All #OpenScience shenanigans are frowned upon. And PIs in my vicinity are still on this level: https://phdcomics.com/comics/archive.php?comicid=1531 - so, how do we educate the educators?

Beep, Beep - I am your friendly #Snakemake release announcement bot.

There is a new release of the Snakemake executor for #SLURM on #HPC systems. Its version now is 2.0.2!

Give us some time, and you will automatically find the plugin on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainers here on Mastodon!
@rupdecat and @johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake-executor-plugin-slurm/issues.

See https://github.com/snakemake/snakemake-executor-plugin-slurm/releases/tag/v2.0.2 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* partition cluster selection: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/385

BEEP, BEEP - I am your friendly #Snakemake release announcement bot.

There is a new release of Snakemake. Its version now is 9.14.3!

Give us some time, and you will automatically find it on #Bioconda and #Pypi.

If you want to discuss the release, you will find the maintainer here on Mastodon!
@johanneskoester

If you discover any issues, please report them on https://github.com/snakemake/snakemake/issues.

See https://github.com/snakemake/snakemake/releases/tag/v9.14.3 for details. Here is the header of the changelog:
𝑅𝑒𝑙𝑒𝑎𝑠𝑒 𝑁𝑜𝑡𝑒𝑠 (𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑦 𝑎𝑏𝑏𝑟𝑖𝑔𝑒𝑑):
𝐁𝐮𝐠 𝐅𝐢𝐱𝐞𝐬

* ensure that queue input jobs wait if the queue is still empty upon job creation: https://github.com/snakemake/snakemake/issues/3866
* fix issues with cyclic dependencies when using the update and before_update flag: https://github.com/snakemake/snakemake/issues/3857