How do I submit to COSMA8?
You must be part of a DiRAC project that has an allocation on COSMA8, and be part of the cosma8 group (use the id command to see this). If you are not part of the cosma8 group, but should be, please contact cosma-support. To submit a job script, you need to use -p cosma8.
UCX ERROR: ivb_reg_mr
Try setting the following option in your SLURM batch script (note, this may impact performance):
export UCX_IB_REG_METHODS=direct
This is potentially due to a bug in Lustre 2.12.6. A possible solution is to reduce memory usage. We cannot yet upgrade to newer versions of Lustre, as these have been tested and found to be unstable on COSMA (including 2.12.8 and 2.12.9).