FAQs

This page answers some commonly asked questions when using the Kay National HPC System. If you are a new user, you may find our Getting Started Guide helpful.

If your question is not here or if you still have problems, please do not hesitate to contact the ICHEC Helpdesk (email support@ichec.ie).

1. Getting started
1. 1.1 Where should a new user begin?
2. Your Central ICHEC Account
3. Accessing the National HPC Service
4. Accessing the Kay National HPC System
5. Compilers and software applications
6. Submitting batch jobs
7. Other common questions
8. Security
9. Getting Help - Using the ICHEC Helpdesk

1. Getting started

1.1 Where should a new user begin?

New ICHEC users should refer to the Kay Getting Started Guide as a starting point.
Documentation and specific topics can be found here.
New users should also refer to the Education & Training section for training courses and upcoming workshop schedules.

2. Your Central ICHEC Account

You can sign up for an ICHEC account here. Once you have confirmed your password, your account will be sent for approval which may take up-to 24 hours. Once approved, you will receive information on how to login to the Project and User Management Application.

2.2 What is my ICHEC username?

Your ICHEC username is the same for all ICHEC systems, such as the Project and User Management Application and the Helpdesk portal. It may be quoted in previous correspondence from ICHEC. It is not your email address.

If you have forgotten it, please email support@ichec.ie and we will advise what your username is.

If you are trying to login to the Project and User Management Application, make sure you are using your ICHEC username and not your email address.

If your account has been inactive for more than 12 months or if we have lost contact with you, your ICHEC account may be been deactivated. In this case, please contact the ICHEC Helpdesk (email support@ichec.ie).

2.4 I have forgotten my password - can you help?

Please send an email to support@ichec.ie and we will send you a password reset link to your registered email address. For security reasons, this link will only be valid for 48 hours.

2.5 How do I change my password?

2.6 I wish to update my details or I am moving institution - what should I do?

Please notify the ICHEC Helpdesk (email support@ichec.ie) with your updated details as soon as possible from your registered email address. It is ICHEC policy that usernames cannot be reused later and details can only be updated from a verified email address.

Note: If your email address bounces or if we are unable to contact you, your account will be marked inactive and you will not be able to login to any ICHEC system or submit jobs. After approximately 12 months, your account entry will be anonymised. It is urgent that users with existing data or who wish to keep their account must contact the ICHEC Helpdesk ASAP. For security reasons, you must email us using your registered ICHEC email address or through the online Helpdesk portal.

2.7 What mailing lists can I be subscribed to?

The primary mailing lists are:

ichec-announce@lists.ichec.ie: This is a low-traffic list for announcements regarding the Centre (optional). Users are asked when they register for an ICHEC account if they wish to receive these emails.

If you wish to receive updates about the Centre, please email us at support@ichec.ie, asking to be subscribed to the ichec-announce mailing list. Likewise, please email us if you do not wish to receive these emails.

ichec-kay-users@lists.ichec.ie: This mailing list is intended to provide essential notifications and information relevant to the operation and services of the ICHEC National HPC infrastructure. All users on the Kay National HPC system have been included in order that they receive essential notifications.

ICHEC Staff will never ask you for your username or password.
Never disclose your credentials regardless of whether the person contacting you purports to be from ICHEC or otherwise as it may be an attempt to steal your login credentials.
Do not click any links
Do not open any attachments
Do not enter any personal details on the fraudulent email or website
Report the ’phishing’ email to the ICHEC Helpdesk (email support@ichec.ie)
Delete the email.
Check our IT Security page for more information.

2.10 How do I close my account?

Please notify the ICHEC Helpdesk (email support@ichec.ie). Note that if you are a Kay user, your home/work directories will be permanently deleted.

3. Accessing the National HPC Service

3.1 What is the overall National Service Project application process?

An overview of the application procedure from start to submitting jobs:

If the PI does not already have an ICHEC account, the PI must first register for an ICHEC account. An ICHEC account can take 24 hours to be set up. The PI will receive an email when their account has been activated.
A PI submits an application for a National HPC Service Project via the Project and User Management Application.
The PI will receive an email letting them know if the project has been accepted or rejected. The email will let them know the further steps required.
If the project is accepted, other users who wish to take part in the project (e.g. members of the research group) can login to the Project and User Management Application with their ICHEC username and password, and apply to join the project.
New users to ICHEC who wish to take part in the National Service Project (apart from the PI) who do not have an ICHEC account should register for an ICHEC account. An ICHEC account can take 24 hours to be set up. The user will receive an email when their account has been activated.
The PI approves users who wish to join the project.
All new projects have a Start Date associated with them, a member of the ICHEC Systems team will normally activate the project on this date and advise the PI when this has been completed.
If the user is new to Kay, a member of the ICHEC Systems team will email the user letting them know that their is has been set up and will send further instructions.
PI and users log in and run jobs.

If the PI or users of the project have any questions about this process, please do not hesitate to contact the ICHEC Helpdesk.

3.2 The project I'm looking for is not listed in the Project and User Management Application?

Only active projects are listed in the Project and User Management Application. If the project is not listed, check with your PI that the project has not met its expiry date.

3.3 Will I receive a reminder about my project expiration date?

Yes. You can check when your project is due to expire on the Project and User Management Application. You may be eligible to request a project expiration, if you wish to request one you should contact the ICHEC Helpdesk.

4 weeks before the project expires, all members of your project will receive an email reminding them of the upcoming project expiration date.
On the date the project expires, job submission will be automatically disabled. The project members will receive an email reminding them to remove their data from the Kay National HPC System.
12 weeks after the project expires, access to the work directory will denied. If the project member does not have any other active projects or condo memberships on Kay, they will be unable to login to Kay. In order to comply with Data Protection requirements, an automated process will start and all project data will be erased - note that this can't be reverted. As per our Acceptable Usage Policy, project work directories are NOT backed up; make sure to implement a regular backup schedule for your valuable data. Users are strongly advised to backup their data before their project expires.

3.4 Can I extend a National Service Project?

In some cases it may be possible to extend a National Service Project end-date. Users are advised to contact the ICHEC Helpdesk if they need more time.

The Condominium service ceased operation in Nov 2023 with the end of life of the Kay system.

4. Accessing the Kay National HPC System

4.1 How do I connect to Kay?

In order to gain access to Kay, you must:

Have an active central ICHEC account
Be a member of an active National Service Project. You can request your own National Service Project or join an existing one, from the Project and User Management Application.

Remember, in order to connect to Kay via SSH:

You will need to login using your central ICHEC account username. You will be requested to authenticate using both your central ICHEC account password and SSH key-based authentication.
All users must initially configure an SSH key on each device they use to connect to Kay. Please note that the latter requires that you generate SSH public-private key pair on each local workstation (and not on Kay) for security reasons (e.g. ssh-keygen for Linux/Mac, MobaXterm for Windows; DO NOT store your private key on Kay). Check out our tutorial - Setting up SSH Keys - for more details.

Remember: Your ICHEC username and password is the same for accessing any ICHEC service. This is not your email address. If you have forgotten your username or password, please contact the ICHEC Helpdesk.

To connect to Kay:

ssh username@kay.ichec.ie

After entering this command, your key will be authenticated and will be prompted for a password. If you are using Windows, you can use an SSH client such as MobaXterm.

You will need SSH access outbound (port 22/tcp) to login to these machines. We have noticed in some institutions, you will have to login to a network server with this SSH access first (e.g. DIAS). If you get a "Connection timed out" error, please contact your local network administrator first to ensure you have the appropriate ports open. Failing this, please submit an issue to the ICHEC Helpdesk.

4.3 What are Kay's SSH host key fingerprints?

For information, you will see one of the following displayed when you first connect to Kay via SSH from your device:

RSA: 2048 SHA256:i7+fqr+IZ/SchKAebTqXvxAoC69vOF2fmyDGRdR8/xQ

ECDSA: 256 SHA256:RY/VxEXdMhW32oU9QmGh/tVt03/ln22z0VgRBL6kIHA

ED25519: 256 SHA256:+hjJtwC6fX0H0/gaSLVbpDxKtEOJtyOUW7x7DzHn6Is

If they do not match, take a screenshot, click 'No'/'Reject' and then contact the ICHEC Helpdesk (email support@ichec.ie).

4.4 Can I connect from home or when traveling?

Yes. You connect using the same way as you would from your local institution's network.

If you are connecting using the same device, the process is the same.

If you are connecting using another device, please generate an SSH keypair on the intended machine and email your additional public key to the ICHEC Helpdesk so we can whitelist it for you. Then, the login process will be the same.

4.5 Can I access Kay using a VPN?

You do not need a VPN to access Kay.

Some Third-Level Institutions's provide a restricted VPN service which if turned on, may prevent access to Kay. You will need to ask your local ICT Helpdesk to whitelist 87.44.64.0/24 over tcp/22 (SSH). In general, we recommend disconnecting from any restricted VPN when connecting to Kay. Unfortunately, third-party VPNs are outside of our control and is beyond the scope of our support.

4.6 What is SSH? How does it work?

Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information. The public key is stored in ~/.ssh/id_ed25519.pub and the private key is stored in ~/.ssh/id_ed25519. Only share your public key. Never share your private key! To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the filesystem. This will prevent people from using it even if they gain access to your files.

4.7 How do I SFTP files between my desktop and ICHEC if I have to use a local SSH proxy?

If your desktop is on a private network and/or you need to ssh to ICHEC machines via a local ssh proxy then the best way to transfer files directly between your desktop and ICHEC is to use sftp over an ssh tunnel. To create the tunnel use something like:

ssh -f -N -L 2222:kay.ichec.ie:22 username@sshproxy.localdomain

Then all connections to port 2222 on your desktop will be forwarded via SSH proxy to kay.ichec.ie. So to ftp between ICHEC and your desktop you can simply use:

sftp -oPort=2222 ichec_username@localhost

4.8 I cannot use SFTP but SSH works?

This problem can occur if you have commands in your ~/.bashrc file which produce output, perhaps an error message which can cause the problem to appear to be intermittent. Such commands should be placed in your ~/.bash_profile file instead where they will not interfere with the SFTP connection process.

4.9 When I try to connect via SSH, I get a "Permission denied (publickey)" error?

You may get this error for a variety of reasons, including:

Your username may be incorrect. You should use the same username as the one you use for all ICHEC services, including the Project and User Management Application.
You may not be an active Kay user: You must be a member of an active National Service Project. Check if you are a member of an active project using the Project and User Management Application.
You have not configured an SSH keypair on your device: You will need to initially configure an SSH key on each device that you use to access Kay.
You are not connecting using the correct key.

If you are still having trouble, please contact the ICHEC Helpdesk (email support@ichec.ie) and we'll be happy to assist you. So that we can help you as efficiently as possible, please include as much detail including:

Screenshots
Error outputs
Steps that you have tried from start to finish
A copy of your SSH public key
Output from the following command:

ssh username@kay.ichec.ie -vvvv

4.10 When I try to connect via SSH, I get a "Permission denied (password)" error?

This error indicates that you have successfully connected with a key but the password you are using is incorrect. This password is your central ICHEC account password (i.e. the one used for Project and User Management Application).

If you have forgotten your password, please contact the ICHEC Helpdesk (email support@ichec.ie) and we will send you a password reset link.

4.11 What's the difference between an SSH key "passphrase" and an account "password"?

SSH keys may be encrypted during initial setup (recommended but this is optional). During the login process, when prompted for a "passphrase", this refers to the password you used to encrypt/decrypt your SSH key.
After SSH key authentication, you will be prompted for your central ICHEC account password. This is the same as the one you use for all ICHEC services (e.g. Project and User Management Application).

If your SSH key is encrypted with a password, you can configure an SSH agent to prevent being asked for your key passphrase as shown in an example below:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

However, you will still be prompted for your central ICHEC account password and this cannot be saved.

We understand this can be frustrating. The same policy applies to all ICHEC Users and Staff. These measures are due to increased security risks associated with HPC systems across the EU. We have taken these security measures to ensure that our systems, users and their research are protected. If you would like to change your central ICHEC account password to something easier to remember, you can do so using the Project and User Management Application.

4.13 When I connect via SSH, I'm getting a "Potential security breach", "SSH host key verification failed" or "key mismatch" warning?

When you see this error, you should treat it very carefully and read this answer in full. It may be possible you are seeing a man-in-the-middle attack.

In June 2020, our login key host keys changed. If you connected to Kay prior to June 2020, then you will see a warning and so you should clear out the old host keys cached locally using the following commands (valid for Linux, macOS and MobaXterm) on your local laptop/workstation:

ssh-keygen -R kay.ichec.ie
ssh-keygen -R login.kay.ichec.ie
ssh-keygen -R login1.kay.ichec.ie
ssh-keygen -R login2.kay.ichec.ie
ssh-keygen -R login3.kay.ichec.ie
ssh-keygen -R 87.44.64.200
ssh-keygen -R 87.44.64.201
ssh-keygen -R 87.44.64.202

When you connect for the first time, host key fingerprints should be one of the following as follows:

RSA: 2048 SHA256:i7+fqr+IZ/SchKAebTqXvxAoC69vOF2fmyDGRdR8/xQ

ECDSA: 256 SHA256:RY/VxEXdMhW32oU9QmGh/tVt03/ln22z0VgRBL6kIHA

ED25519: 256 SHA256:+hjJtwC6fX0H0/gaSLVbpDxKtEOJtyOUW7x7DzHn6Is

If they do not match or you are in doubt, take a screenshot, click 'No'/'Reject' and then contact the ICHEC Helpdesk (email support@ichec.ie).

If you are getting a "Algorithm negotiation failed" error, or similar, you should update your client to the latest version. Kay is configured with a hardened SSH configuration to ensure that clients connect using secure methods.

ICHEC recommends using the latest version of MobaXterm if you are a Windows user, or the native terminal if you are a Linux or macOS user.

Some users have reported that they cannot login to Kay using SSH Secure Shell Client (including version 3.2.9). This is now available as SSH Tectia Secure Shell Client and users should upgrade if they use this specific client.

4.15 Can I opt-out of maintenance emails for Kay?

Emails sent to Kay users are mandatory and every user accepted this when they create the account. We only send important emails to inform users with maintenance updates or critical issues.

If you don't want to receive those emails, we can deactivate your account and you'll not receive them anymore. If you wish to deactivate or close your account, email support@ichec.ie.

5. Compilers and software applications

5.1 What languages are available?

Fortran, C, and C++ are available on the ICHEC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Documentation section. A number of scripting and interpreted languages are also available.

5.2 What software applications are available?

Please refer to the Software section for an up-to-date overview. If there is a particular package you would like to see installed, please contact the ICHEC Helpdesk with your request.

5.3 Is help with Makefiles available?

For some examples and guidelines regarding Makefiles see our tutorial on Building and Porting. If you have specific questions, feel free to contact us via the ICHEC Helpdesk.

5.4 How do I run Gaussian calculations on Kay?

We have included instructions on the Gaussian page of our software section.

6. Submitting batch jobs

6.1 What is a batch request?

On the Kay cluster, batch processing is managed by the Slurm Workload Manager. Slurm batch requests (jobs) are shell scripts that provide timing, memory, and processor information. For example scripts, see the Kay support page. Slurm uses squeue -a to check the status, and qdel to delete a batch request. For more information also see Slurm Workload Manager, PBS to Slurm, and Slurm Commands pages.

6.2 How do I submit a job to Kay?

All jobs should be submitted to Kay via the Slurm Manager. Slurm uses sbatch my_script_name to submit a job. A sample script can be found on the Kay support page.

6.3 How can I execute a number of serial runs (task farming)?

A taskfarm utility is in place on Kay you can find details here.

6.4 How can I find out how much resources (CPU core hours) are available to my project?

Use the mybalance command as follows:

username@kay:$ mybalance 
=======================================================================
Core Hours Allocation Information for account : myproject
=======================================================================
Allocated Core Hours           :         833333.33
Project Consumed  Core Hours   :          32153.70
Percentage of Project Consumed :              3.86
=======================================================================

This command will return the number of core hours available to all your projects (in the above example myproject). So for instance, if you wish to run a 48 CPU core job for 24 hours, you will need to ensure that you have a minimum of 24*48=1152 core hours on your project's account.

6.5 How is the order that jobs run in on Kay decided?

The order that jobs run in is decided by the Slurm Workload Manager based on a priority it calculates for each job. The calculation of queue priority is relatively complex in order to deal with long queues, large numbers of jobs and various classes of project. Below is a list of the factors that affect queue priority:

Jobs gain priority based on the time they wait as an idle job in the queue.
Fair share targets are applied that balance resource usage over time across both projects and project types.
An expansion factor is used to modify job priority based on requested walltime. Shorter jobs are given extra priority.
Only idle jobs gain queue priority. Jobs blocked due to soft or hard limits (see explanation below) can't run and don't gain queue priority over time.

In addition to priority, a backfill mechanism can allow jobs to skip the queue fill space that would otherwise have been unused. This allows smaller jobs with shorter runtimes to maximise our utilisation while larger jobs wait for processors to free up.

Kay has hard and soft limits on the number of jobs and processors a single user can use. The soft limits apply when there is a queue and the considerably higher hard limits apply when there are no jobs waiting to run. When a user exceeds whatever limit is in force all their queued jobs are moved from the queue to a blocked state and are only unblocked when a job completes or the limit is no longer active. Blocked jobs are only checked periodically and not on the regular job scheduling interval.

In addition there is a limit on the number of jobs a user may have queued even when they have not exceeded the hard or soft limits. Additional jobs will be moved to a blocked state until a queued job runs or is cancelled. For further information see the scheduling policies page.

7. Other common questions

7.1 How do I run remote X applications?

If you are connecting from a MS Windows machine you will need to have Xming, Hummingbird Exceed or similar installed and running on your workstation. You also need to ensure that X11 forwarding is enabled.

On a typical Unix system X11 support will already be in place and you need connect using:

ssh -X username@kay.ichec.ie

Now you can execute any of your X applications and it should forward the X application via SSH to your local screen. For example:

$ xclock &

You should now see the Xclock is running on your local screen.

In addition to individual user's home directories e.g. /ichec/home/users/username each project has an associated project directory /ichec/work/project_code which is accessible by all members of that group. By default a user's home directory and its contents cannot be read by other users though you can change the directory permissions to allow this if you wish.

chmod -R 750 directory_name

This command will make the contents of the named directory readable by the other members of you project.

7.3 Can I download files on the compute nodes?

No. The compute nodes do not have access to internet. You can download the files on the login nodes before running your simulations.

7.4 How do I remove ^M characters at the end of input lines?

UNIX treats the end of lines differently than other operating systems. Sometimes when editing files in both Windows and UNIX environments, a CTRL-M character gets added to the end of each line as ^M in vi. To remove the ^M characters use the following command:

dos2unix myfilename

8. Security

Please refer to our Security FAQs page.

9. Getting Help - Using the ICHEC Helpdesk

9.1 What is the ICHEC Helpdesk?

The ICHEC Helpdesk is the primary mechanism to access the user support in ICHEC. We welcome questions, issues and comments which will be addressed by one of our Computational Scientists and Systems Administrators. The wide ranging expertise of our computational scientists can address all issues you may encounter using our facilities. Learn more about our support here.

9.2 How do I contact the ICHEC Helpdesk?

There are two main entry points to ICHEC's support teams:

You can log a ticket online using the ICHEC Helpdesk portal
Email support@ichec.ie

9.3 What are the opening hours for the ICHEC Helpdesk?

Our User Support Opening Hours are from 09:00 - 17:00 Monday - Friday (Excluding Bank & Public Holidays).

9.4 How should I send large files to the ICHEC Helpdesk?

We recommend using HEAnet Filesender to send large files.

9.5 How do I acknowledge ICHEC in my work?

We would appreciate a formal acknowledgement of ICHEC by inclusion in any resulting publications of the following sentence or some variation thereof: "The authors wish to acknowledge the Irish Centre for High-End Computing (ICHEC) for the provision of computational facilities and support".

Please notify us of the publication at support@ichec.ie.

Sometimes it can be useful to view your current and previous Helpdesk requests. If you don't have access, please send an email to support@ichec.ie and we will give your access. Remember that you can always log a ticket by email, you do not need to use the Helpdesk portal.