NERSCPowering Scientific Discovery Since 1974

2014 NERSC User Survey Results

Executive Summary

Satisfaction with NERSC remained extremely high in all major categories in 2014, according to 673 users who responded to the 2014 NERSC user survey. The survey satisfaction scores mirrored those achieved in 2013, a year that saw record or near-record scores across the board. The 673 responses was a new high, surpassing the previous mark of 613 in 2013. Those users accounted for 72 percent of all computing hours used at NERSC in 2014.

The overall satisfaction rating of 6.50 on a 7-point scale equaled the best ever recorded and the same as in 2013. The average of all satisfaction scores was 6.33, also the same as the previous year. Only three of the 105 statisfaction questions showed a decrease compared to the previous year.

Survey Format

NERSC conducts its yearly survey of users to gather feedback on the quality of its services and computational resources. The survey helps both DOE and NERSC staff judge how well NERSC is meeting the needs of users and points to areas where NERSC can improve.

The survey is conducted on the web, and in 2014 consists of 105 satisfaction questions that are scored numerically. In addition, we solicit free-form feedback from the users. In December 2014, 5,000 authorized users (those with registered accounts who have signed the computer policy use form) were invited by email to take the 2014 user survey. The survey was open through January 19, 2015.

7-Point Survey Satisfaction Scale

The survey uses a seven-point rating scale, where “1” is “very dissatisfied” and “7” indicates “very satisfied.” For each question the average score and standard deviation are computed. 

Text ValueNumerical Value
Very Satisfied 7
Mostly Satisfied 6
Somewhat Satisfied 5
Neutral 4
Somewhat Dissatisfied 3
Mostly Dissatisfied 2
Very Dissatisfied 1

3-Point Usefulness and Importance Scale

Questions that asked if a system or service was useful or important used the following scale.

Text ValueNumerical Value
Very Useful (or Important) 3
Somewhat Useful (or Important) 2
Not Useful (or Important) 5

Changes from one year to the next were considered significant if they passed the t-test criteria at the 90% confidence level.

Overall Satisfaction

The average response to the item "Please rate your overall satisfaction with NERSC" was 6.50 on the seven-point satisfaction scale. This was the highest rating ever (to with statistical error) since the survey was created in its current form in 2003.

The following figure shows the overall satisfaction rating from 2003-2014. The red line labeled "Target" is the minimum acceptable DOE target for NERSC.

Overall Satisfaction Questions

Satisfaction was the same as in 2013 in all fives areas surveyed in the first section of the survey: "Overall Satisfaction."  As with the "Overall Satisfaction with NERSC" score, none of the five area have ever received a statistically significant higher rating. (Results shown here include responses from all NERSC users – including those from JGI and PDSF users – for 2012, 2013, and 2014.

Questions Asked on the "Overall Satisfaction" Survey Page

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
NERSC Overall 666 6.50 0.80 -
Services 640 6.58 0.78 -
Computing Resources 660 6.37 0.87 -
Data Resources 563 6.31 1.02 -
HPC Software 554 6.26 1.02 -
Total for 5 Questions   6.41 0.89 -

The most common rating in all five categories was "Very Satisfied (7)". The distributions for the Overall Satisfaction categories are show below.

Other Overall Satisfaction Questions

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Security 358 6.71 0.69 -
Consulting 479 6.64 0.80 -
Account Support 521 6.71 0.74 +0.09
PDSF 51 6.43 1.14 -
Project Global File System 278 6.61 0.71 -
HPSS 295 6.45 0.89 -
Web Site 520 6.53 0.72 -
Global Scratch File System 367  6.53  0.84  -
Hopper 416 6.27 0.89 -0.19
Carver 265  6.36  0.87 -
Projectb File System 108 6.42 0.86 -
Genepool 59 6.17 0.81 -
Edison* 390 6.31 0.88 -
NX 135 6.10 1.15 -

Highest Rated Items

 The 10 highest-rated items (of all 105 satisfaction questions) on the survey involved data storage systems (6 items), consulting and account support (2 items), and NERSC networking and cybersecurity (2). This tells us that users think NERSC takes good care of their data and makes it readily available, and provides excellent consulting, account support, networking, and cybersecurity.

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Project File System Data Integrity 265 6.77 0.59 +0.12
Project File System Availability 265  6.75  0.55 -
Account Support 521 6.71 0.74 +0.09
NERSC Cybersecurity 443 6.71 0.65 -
HPSS Data Integrity 263 6.71 0.64 -
HPSS Availability 269 6.70 0.65 -
Global Scratch Availability 350 6.66 0.73 -
Consulting Overall 479  6.64  0.80 -
Internal NERSC Network 371  6.63  0.73  -
Global Project File System Overall 278 6.61 0.71 -


Lowest Rated Items

The 10 lowest-rated items (of all 105 satisfaction questions) on the survey involved the batch queues and the associated wait times (5), and data policies and analysis software (5). Batch wait times on Hopper and Edison were below the NERSC minimum target of 5.25, reflecting the strong demand for access to NERSC systems.

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Data Analysis Software 258 5.98 1.26 -
Hopper Batch Queue Structure 397  5.97  1.11 -
Long Team Data Retention 277 5.96 1.27 -
Workflow Software 200 5.91 1.32 -
Visualization Software 200  5.88  1.23 -
Edison Batch Queue Structure 377  5.76  1.24 -
Carver Batch Wait Time 160 5.63 1.28 -
Hopper Batch Wait Time 404 5.17 1.52 -
Edison Batch Wait Time 383 4.87 1.56 -0.50

Demographic Responses

Large, Medium, and Small MPP Users

For purpose of survey analysis, users were divided into those who used more than 3 million MPP hours (Large), between 500,000 and 3 million MPP hours (Medium), and less than 500,000 MPP hours (Small). Overall, there was not a great difference among the groups, in contrast to the previous year when larger users expressed greater satisfaction with NERSC.

Satisfaction by NERSC Experience

Users that had been computing at NERSC reported more satisfaction in most areas, the exception being in data resources where new users were the happiest.

Scientific Domains

While users in all scientific domains rated NERSC highly, there were some differences among the groups. The plots below show differences from the average of all user responses. Researchers in accelerators, applied math, chemistry, geosciences, lattice guage theory, and fusion research ranked NERSC higher than average, while scores were lower users in astrophysics and biosciences. An explanation for these variations are not immediately clear.

Satisfaction by Allocation Type

Projects receive allocations via a number of methods. "Production" accounts are allocated by DOE program mangers through the ERCAP allocations process, "ALCC" (ASCR Leadership Computing Challenge) accounts are allocated by DOE's Office of Advanced Scientific Computing, the NERSC Director has a reserve of time to allocate, NERSC awards small "Startup" accounts, and there was a "Data Pilot" program in 2013.

Production accounts make up the vast majority of accounts (and thus survey responses), so their responses define the average and that group shows little variation from it. ALCC and Startup users rated NERSC higher than average in all categories, while Director's Reserve and Data Pilot users had mixed responses.


Hopper has been a stable, productive system over the last three years and its satisfaction scores remained high, but down slightly from 2013 . The addition of the production Edison system in 2014 helped relieve some of the demand on Hopper, but the rating for queue wait times still fell to 5.17, but was still above the pre-Edison value of 4.90.

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Hopper Overall 416 6.27 0.89 -0.19
Uptime 405 6.47 -0.19 -
Disk Configuration and I/O 385 6.18 1.07 -
Ability to run interactively 299 6.11 1.13 -
Batch queue structure 397 5.97 1.11 -
Batch queue wait times 404 5.17 1.52 -
Total for 6 Questions   6.02 1.08  

Representative User Comments

" 1. Hopper and Edison were clearly oversubscribed during at least the third quarter of this last calendar year. Turnaround became unworkable for a number of projects. Only more hardware or smaller allocations can really fix this, although the situation has improved following queue adjustments. Given the planned machine move and periods with lower machine availability, some careful consideration should be given to queue structures (etc.) to minimize the crush in the second half of 2015. 2. Some (more) careful expectations management may be needed for Cori/KL. For many applications it might be a lot of work to get performance not significantly better than a similar era Xeon. (Energy usage should be much better, but the allocations are in hours, not joules.) "

" The queue time last year for hopper is quite long, compared to the years before 2013, as far as I remember. This seems odd as more computational resources such as Edison are also available. Maybe some of the queue policies can be further improved. "

" Small to medium jobs are sometimes on the queue for a few consecutive days - and this is extremely inconvenient! "

" Please provide more capabilities for high-throughput jobs. It is very difficult with many users competing for time in a single high-throughput queue (thruput on hopper), especially when small projects need to compete with larger collaborations, which already have their own dedicated resources. I would hope that if NERSC provides dedicated resources to these large collaborations, then the other computing environments would be accessible to smaller projects. Currently, this is not the case for the thruput queue, the only high throughput queue available at NERSC "

" Something needs to be done about the queue wait times on Hopper and Edison. I often have to wait 1 to 2 weeks for a job to start requesting 1000 - 5000 processors. "

" Decrease wait times for short to medium jobs on hopper. "

" Reduce the batch queue wait times on Hopper and Edison, perhaps by altering the priorities or lowering the limits on the run times of jobs (e.g., 12 hrs for jobs with more than 512 cores, etc). Every code should be able to checkpoint. Also, use a 'fair share' system so that users lose priority as they run more jobs. "

" The queue time is very important for me. Now it will take me several days to finish one job on Hopper, compared with the very short waiting time (less than 1 day) in early 2014. "

" Currently, the batch wait times on both Hopper and Edison are terrible. I wish this could be improved going forward in future. "

" I have had an extremely positive experience with NERSC. Edison and hopper have been indispensable for my research over the past 2 years. NERSC's consultants have been extremely helpful and responsive. "

" Excellent programming environment, excellent hardware, good queueing policies on hopper; the higi-priority debug queue is extremely useful in particular (not all supercomputing centers have them, regrettably) "

" I have been very impressed with the stability of the computing systems I have used (Hopper and Edison). Any issues I had were resolved very quickly and thoroughly. Planned outages of systems were communicated well in advance of when they would occur, allowing me to schedule my tasks appropriately. "

" It is very easy to install and run the simulation I need to. Considering I do not have no experience using super computer, it took just few hours to run my case in Hopper. "


Edison entered preproduction in 2014 and had improved stability and uptime compared to pre-production time in 2013. However, long queue waits caused some user dissatisfaction.

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Edison Overall 390 6.31 0.88 -
Uptime 383 6.40 0.84 +1.04
Disk Configuration and I/O 352 6.25 1.07 +0.23
Ability to run interactively 273 5.98 1.26 -
Batch queue structure 377 5.76 1.24 -
Batch queue wait times 383 4.87 1.56 -0.50
Total for 6 Questions   5.92 1.14  

Representative User Comments

" I believe that edison is over subscribed. The queue wait times for say 5000 cores for the full 48 hours can approach two weeks. There also seem to be users requesting large numbers of cores (50,0000 or so) for ridiculously short periods of time (like 2 hours or less). These jobs may limit the ability of the batch system to backfill effectively and so should probably not be allowed (or given much lower priority). "

" Edison is awesome. "

" I am particularly interested in the performance stability of my applications. Recently, the performance of my codes on edison becomes unpredictable. Different runs reported widely different timings. I am wondering how NERSC will address these issues ? "

" Edison is one of the best machines in the world. "

" Job waiting time on Edison is too long, particularly for regular_small queue, for which the priority should be increased. I know it is a policy to encourage users to use a large number of cores. However, the regular_small queue is a very important way to use the limited Edison resource most efficiently and productively. Saying that, for many applications with available NERSC resource (allocation), running high-core_number jobs can quickly burn out yearly allocations, and produce less scientific results than running low-core-number jobs. "

" Something needs to be done about the queue wait times on Hopper and Edison. I often have to wait 1 to 2 weeks for a job to start requesting 1000 - 5000 processors. Our codes would also benefit from having some machines with more memory per processor. 8GB/processor would be ideal. "


Satisfaction scores for Carver remained high as it continued to be a reliable resource for users that like a standard LINUX computing environment. As with other system, the most dissatisfaction was with the queue wait time.

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2012 to 2013
Carver Overall 171 6.36 0.87 -
Uptime 184 6.54 0.88 -
Ability to run interactively 141 6.16 1.21 -
Batch queue structure 157 6.12 1.26 -
Batch queue wait times 160 5.63 1.28 -
Total for 5 Questions   6.17 1.06  

Representative user comments:

Wait times on small jobs can be very long (especially on Carver). Given that most of what I do is in the form of small jobs, rectifying this would help a lot.

We often need to run a short job on thousands of nodes and many people use the debug queue to do that. In fact some people submit several debug queue jobs. Many times I need to run I/O intensive jobs, which don't scale well.


TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2012 to 2013
Genepool Overall 59 6.17 0.81 -
Uptime 52 6.36 1.08 -
Ability to run interactively 52 6.27 1.01 -
Batch queue structure 49 5.98 1.11 -
Batch queue wait times 51 5.86 1.11 -
Filesystem configuration and I/O performance 52 5.81 1.37 -
Data storage, archiving, and retrieval 54 5.72 1.46 -
Total for 7 Questions   6.03 1.13  


TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2012 to 2013
PDSF Overall 51 6.43 1.14 -
Uptime 49 6.65 0.80 -
Ability to run interactively 48 6.13 1.63 -
Batch queue structure 44 6.25 1.10 -
Filesystem configuration and I/O performance 47 6.26 1.21 -
Connection to external data repositories 39 6.28 1.02 +0.28
Total for 6 Questions   6.46 0.82  


TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
HPSS Overall 281 6.45 0.69 -
Reliability (Data Integrity) 263 6.71 0.64 -
Availability (Uptime) 269 6.70 0.65 -
Data Transfer Rates 265 6.36 0.96 -
Data Access Time 260 6.35 0.95 -
User Interface 266 6.02 1.32 -
Total for 6 Questions   6.43 0.90  

Representative Comments

" My main complaint is that HPSS is confusing. There is information on how to use it on the web site, but for users (like me) who are not necessarily Linux experts it is not very clear. The Globus GUI is nice, but I usually need to use the command-line htar since I have lots of files, and when using this command it's not very clear when I have successfully backed up my data and how I can check what I've stored on HPSS and what I haven't. "

" Transferring large files to HPSS is not easy - very time consuming and hence inconvenient. It would be great if we could simply use sftp to store/retrieve archived data. "

" Proactive backup from HPSS. I hate it when I forget stuff on scratch and its purged. "

" I realize that for good reasons we are not able to log-in directly to HPSS. But some ways to display the stored files there --- without relying on the memory of the user to remember all the files the user stored --- can be very helpful. "

Project File System

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2013
Project Overall 278 6.61 0.71 -
Reliability (Data Integrity) 265 6.77 0.59 +0.12
Availability (Uptime) 265/td> 6.75 0.55 -
File and Directory Operations 249 6.50 0.86 -
I/O Bandwidth 262 6.40 0.97 +0.22
Total for 5 Questions   6.61 0.73  


TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Software Overall 554 6.26 1.02 -
Programming Libraries) 511 6.38 0.91 -
Programming Environment 509 6.47 0.82 -
Applications Software 499 6.32 0.96 -
Performance and Debugging Tools 393 6.13 1.06 -
Data Analysis Software 258 5.98 1.23 -
Visualization Software 255 5.88 1.23 -
Total for 6 Questions   6.22 1.02  


TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2012 to 2013
Consulting Overall 479 6.64 0.80 -
Response Time 474 6.59 0.88 -
Quality of Technical Advice 470 6.55 0.86 -
Response to Special Requests 320 6.49 0.97 -
Time to Solution 465 6.43 0.97 -
On-line Help Desk 300 6.54 0.95 -
Data Analysis and Visualization Assistance 104 6.00 1.25 -
Total for 7 Questions   6.52 0.91  

Representative User Comments

You are doing excellent in 1. providing consult services to help our daily works; 2. providing a stable computing environment that we can depend on to complete our work; 3. very friendly and helpful online informations to help us to make our work done. Anyway, many thanks.

Helpful and knowledgeable consultants.

I've been quite happy with the computing and consulting services.

Consulting is excellent. An overall spirit of helpfulness is evident in all interactions from HPSS to disks to computing.

Communication is also very good. Between the "live status", MOTD, and emails with long-range plans, I feel that NERSC lets the users know what to expect and what is happening.

Consulting team is awesome! Thanks for everything!

[NERSC should] 1) Better communication of outages (planned and unplanned): Generally these are good, but they are occasionally incomplete or work gets added in at the last minute without informing the users. Generally the consultants have a great handle of what work is planned, but there has been occasions where work has been added in without informing the consultants or stake-holders. Also during the outages, there should be somewhat regular status updates on the progress of work (e.g., what is complete, what is behind, etc). This will help us plan better for restoration of services. This will also be critical for the planned move for NERSC from OSF to the hill. 2) Stakeholder notifications: Occasionally individual systems are rebooted or serviced. There seems to be no mechanism for informing users of those systems that an event has occurred that may disrupt their services or processes. Notifications should extend to all NERSC hardware (such as tape and DTNs) and not just the big hardware. 3) Business continuity: Provide a means of archiving critical data off-site. There is archival data at NERSC that should be protected should a catastrophic event occur (local and/or regional). While it is desirable to have co-located archiving, there should be a means for externally storing a copy of mission and science critical data outside of the bay area.:w

Accounts and Allocations

TopicNumber of ResponsesAverage RatingStandard DeviationStatistically Significant Changes 2013 to 2014
Account Support and Passwords 521 6.71 0.74 +0.09
NIM Web Accounting Interface 504 6.53 0.86 -
Allocations Process 469 6.43 0.93 -


The NERSC home page is great. But I cannot figure out how to get from that to my personal home page"NIM home". This page is great for monitoring account usage and other administrative tasks related to my account, which is the primary reason I vist the NERSC web site. As a result, I have my browser configured to open up to my "NIM home" page when I login. The needed links for these tasks - My Stuff, Reports, Actions, Account Usage, and so on are right there. But if I wish to find another page, say the hardware of Cory, I just cannot manage to get to that from my home page. None of the links on the "NIM home" page can get me to the NERSC home page. In particular, the Search link only offers links for administrative functions. So I always have to open a new tab on the browser and open up "NERSC home" to find this other type of info. It is a pain...


TopicNumber of ResponsesAverage Usefulness Rating (1-3)Standard Deviation
Email 502 2.73 0.47
Center Status on Web 488 2.74 0.52
MOTD 462 2.57 0.61

Are you adequately informed about NERSC changes?

Yes: 388 (98.2%), No: 7 (1.8%)


TopicNumber of Responses Usefulness Rating (1-3) Satisfaction Rating (1-7)Standard DeviationStatistically Significant Change 2013-2014
Web Tutorials 252 2.63 6.38 0.93 -
New Users Guide 315 2.79 6.58 0.77 -
Training Presentations on Web 183 2.44 6.27 0.95 -
Classes 150 2.30 6.17 1.20 -
Video Tutorials 123 2.30 6.23 1.15 -


TopicNumber of Responses Satisfaction Rating (1-7)Standard DeviationStatistically Significant Change 2013-2014
Web Site Overall ( 504 6.53 0.72 -
System Status Info 444 6.60 0.74 -
Accuracy of Information 473 6.56 0.80 -
MyNERSC ( 439 6.51 0.89 -
Timeliness of Information 467 6.50 0.82 +0.11
Ease of Finding Information 485 6.31 0.95 -
Searching 372 6.04 1.19 -
Ease of Use From Mobile Devices 111 5.99 1.32 -
Mobile Web Site 103 6.09 1.33 -

Data Analytics and Visualization

TopicNumber of Responses Importance Rating (1-3) Satisfaction Rating (1-7)Standard DeviationStatistically Significant Change 2012-2013
Data Analytics and Visualization Assistance Data Analysis and Visualization Assistance 104 6.00 1.25 -
Ability of perform data analysis 173 2.69 6.22 1.05 -
NERSC Databases 101 2.50 6.15 1.17 -
NERSC Science Gateways 187 2.45 6.22 1.10 -

Where to you perform analysis and visualization of data produced at NERSC?

All at NERSC 67 13.1%
Most at NERSC 95 18.6%
Half at NERSC 104 20.4%
Most elsewhere 132 25.9%
All elsewhere 91 17.8%
I don't need data analysis or visualization 21 4.1%


How important are each of the following to you?

TopicNumber of ResponsesAverage Importance Rating (1-3)Standard Deviation
Scratch Directory Quotas 411 2.80 0.41
Project Directory Quotas 313 2.72 0.48
Archival storage space 267 2.70 0.50
Scratch File System Purge Policies 282 2.69 0.51
I/O bandwidth to local disk 301 2.69 0.48
Ability to checkpoint jobs 230 2.67 0.56
Data Transfer Nodes 258 2.67 0.50
Analytics and visualization assistance 113 2.62 0.63
Long-term data retention 236 2.59 0.57
Data management tools 166 2.55 0.57
Access to Databases at NERSC 162 2.51 0.63
Science gateways (see 172 2.45 0.65
Interactive Jobs 373 2.27 0.76
Serial Jobs 381 2.15 0.82

Programming Models

We asked users to let us know if they had experience with the programming models shown in the chart below. By far, users are most familiar with OpenMP. This will be important for programming on next-generation systems, where mixed MPI and OpenMP is expected to be the main programming model, with other languages and paradigs vying for acceptance.

Application Readiness

We asked users if about how ready their codes were for using the Intel Xeon Phi processors that would be in Cori.

ItemNumber of ResponsesI Do Not KnowNot at All ReadySomewhat ReadyVery Ready
Overall Readiness 396 155 (39.1%) 93 (23.5%) 104 (26.3%) 44 (11.1%)
On package memory (MCDRAM) Readiness 378 219 (57.9%) 67 (17.7%) 59 (15.6%) 33 ( 8.7%)
Vectorization Readiness 390 143 (36.7%) 67 (17.2%) 100 (25.6%) 80 (20.5%)
OpenMP (threading) Readiness 408 95 (23.3%) 85 (20.8%) 108 (26.5%) 120 (29.4%)