Important Update on Cori Status
June 30, 2016 by Rebecca Hartman-Baker
We are grateful for your patience as we upgrade Cori to the latest operating system, a necessary step in enabling support for the upcoming Knights Landing Phase 2 of Cori. At 2:00 pm Pacific time, we will release the machine back to users. Before you log in and begin working on Cori again, here are a few important things you should know.
Access to Cori
Cori's host keys have changed. You will see a warning and may be prohibited from logging on due to your ssh security settings. Under normal circumstances, we would encourage users to report this security warning to us. However, due to the change in host keys, this warning is expected and does not need to be reported.
Because the host key has changed, when you try to log in to cori.nersc.gov, which gets resolved to a particular IP address, your SSH client detects that the host key is not the same as the host key for Cori in the known_hosts file. It may abort with output warning you of the problem, leaving it up to the user to determine if this really is Cori, with a new identifier, or if you're being hacked.
On a Mac or Linux machine, the warning says "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" along with a reference to a man-in-the-middle attack and some other language designed to scare you.
In this case, because NERSC has informed you that the host keys for Cori have changed, you can safely go into your local computer's ~/.ssh/known_hosts file and delete any lines referencing cori, coriXY (where X and Y are integers), or cori.nersc.gov (the warning message will tell you the offending line number). When you next log in to Cori, it will be as if you're logging in for the first time, and you'll be asked to confirm the host key (by typing "yes").
On a Windows machine using PuTTY, a window with "WARNING - POTENTIAL SECURITY BREACH!" will appear, and inform you of the differing keys. You can choose to carry on connecting and update the key by selecting "Yes", carry on connecting without updating by selecting "No", or abandon the connection by selecting "Cancel". While under normal circumstances we would recommend selecting "Cancel" and informing NERSC, since NERSC has informed you of the new Cori host keys, you can safely select "Yes".
For more information, please see: https://www.nersc.gov/users/accounts/user-accounts/computer-security/ssh-key-fingerprints/
Cori has been upgraded to a completely new version of the OS. This is akin to going from Mac OS 9 to OS X. The implications of this change are as follows:
- You must recompile and relink all user codes. Old binaries will fail at startup with an initialization error.
- Not all NERSC-provided libraries/software are available yet. We must recompile all libraries, and there has not been sufficient time for us to complete this task. We thought it was better to return the system to users now rather than wait for all software to be available. At the top of our to-do list are the binaries we supply to users, such as VASP.
- If there is a module available for the software in question, then we have completed its reinstallation, but may not have fully tested it. If you encounter an error in the available software or there is something that you require that is not available yet, please send in a ticket.
- R has been upgraded to version 3.3.1, however some packages requested by users for version 3.2.2 remain to be installed. These will be handled in the next few days.
- VTune is not yet available.
- Shifter is not yet available.
- User cron jobs are unavailable and we will update when we have more information on when this capability will be restored.
Python on Cori
We decided to take this opportunity to revamp the way that we deliver Python to users on Cori.
NERSC has retired its builds of Python in favor of Anaconda Python with threaded MKL support for numpy, scipy, scikit-learn, and numexpr. Some Python packages previously provided as modules are now available through the root Anaconda environment. Others remain as standalone modules or will return in the coming days. Users should note that conda environments replace virtualenv; we advise users to convert virtualenv to conda environments.
Most previous standalone python modules for machine learning (eg Theano, Keras, Lasagne, etc. ) are now available in a custom anaconda environment that may be sourced after module loading the default Anaconda Python. Also, TensorFlow is a new addition to this environment. Neon will be added shortly.
For more information, please see: http://www.nersc.gov/users/data-analytics/data-analytics/python/ for a general overview, and http://www.nersc.gov/users/data-analytics/data-analytics/python/anaconda-python/#toc-anchor-2 for instructions on how to set up conda environments.
We are still working to recreate the queue settings. This is a top priority so we expect to have them finished soon.
- CCM mode is currently unavailable.
- salloc is also unavailable.
- The realtime queue is not available on Cori, but remains available on Edison.
- The completed jobs and MyNERSC Queue displays of user job status are not currently functional.
- We plan to reboot the system every Wednesday from 8 am to noon Pacific time for the next few weeks as we make minor tweaks and adjustments to the system setup.
- There may be times when parts of the system are reserved for testing or otherwise unavailable to users.
No Charge for Jobs through July 4
As we are going into a three-day weekend with minimal support available, all jobs on Cori will not be charged to user repos through 11:59 pm July 4, 2016.
For the latest updated information on the status of the system, please see http://www.nersc.gov/users/computational-systems/cori/updates-and-status/
Please submit a ticket to the NERSC consultants via one of the following:
- MyNERSC: my.nersc.gov (go to "My Tickets" and click on "Create New Ticket")
- Emailing firstname.lastname@example.org
For urgent requests, NERSC operations staff are available 24/7 at 1-800-66-NERSC, Option 1. Given that this is a holiday weekend, please be patient if some issues cannot be addressed until after July 4.