http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/ http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/resolved-projectb-filesystem-outage-july-9-2012/ <p> </p> <div> <div>The projectb filesystem had a hardware failure that potentially generated I/O errors.  The filesystem logs indicate that the earliest abnormal event on the filesystem occurred at 9:19AM and the filesystem was taken down for maintenance at 10:42AM.  The filesystem returned to service at 11:20AM.  Jobs running on the cluster would not have been able to read from or write to the projectb filesystem between 10:42AM and 11:20AM.</div> </div> <div> <div> </div> </div> <div> <div>Between 9:19AM and 10:42AM one out of the 20 GPFS controllers on projectb was down, and didn't failover (as it should have).</div> </div> <div> <div> </div> </div> <div> <div><strong>This means:</strong></div> </div> <div> <div>1/20 file I/O operations could have failed between 9:19AM and 10:42AM</div> </div> <div> <div> </div> </div> <div> <div>If your job was performing a large number of short reads and writes, then there is a better chance you were affected.</div> </div> <div> <div> </div> </div> <div> <div>Any data that was successfully written (to a complete file) is good.</div> </div> <div> <div> </div> </div> <div> <div>Any data that was written in random I/O (e.g. fseek/fwrite) could be suspect, and should be looked at with care.</div> </div> <div> <div> </div> </div> <div> <div><em>Please check your data that was written between <strong>9:19AM and 10:42AM</strong>.</em></div> </div> <div> <div><em>Please check your jobs that were operating between these time periods if they were performing I/O on projectb.</em></div> </div> <div> <div><em>Please file a ticket with NERSC if you need help.  (<a href="http://help.nersc.gov/" target="_blank">http://help.nersc.gov</a>)</em></div> </div> <div> <div><em><br/></em></div> </div> <div> <div>Sincerely,</div> </div> <div> <div>Doug Jacobsen</div> </div> <div> <div>NERSC Bioinformatics Computing Consultant</div> </div> <p> </p> Mon, 09 Jul 2012 11:56:08 -0700 http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/resolved-projectb-filesystem-outage-july-9-2012/ http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/important-notice-about-using-house/ <h2>Description</h2> <div> <div>There have been a lot of issues recently with NFS hangs on the gpint machines.</div> </div> <div> <div>The origin of the gpint hanging has been determined to be a defect in the Isilon filesystem software, and happens when a file being written is simultaneously opened for reading on the same host.</div> </div> <div> <div> </div> </div> <div> <div>This most frequently happens when people tail files being written by the same machine.</div> </div> <div> <div> </div> </div> <div> <div>E.g.:  DO NOT DO THIS:</div> </div> <div> <div>gpint17 $ ./somewritingProcess &gt; outfile &amp;</div> </div> <div> <div>gpint17 $ tail -f outfile</div> </div> <div> <div> </div> </div> <div> <div>This very common (and desirable) operation has been determined to hang the filesystem on the host reading/writing the file.  We are working with the vendor to try to correct this situation, but in the meantime a work-around is to read the file from a different machine than the writer:</div> </div> <div> <div> </div> </div> <div> <div>E.g.: THIS IS OK</div> </div> <div> <div>gpint17 $ ./somewritingProcess &gt; outfile &amp;</div> </div> <div> <div>gpint17 $ ssh gpintXX</div> </div> <div> <div>gpintXX $ tail -f outfile</div> </div> <div> <div> </div> </div> <div> <div>Please note that these problems are not limited to the gpint machines - but any machine connected to /house.  Please write me back with any questions, and file tickets at <a href="http://help.nersc.gov/" target="_blank">http://help.nersc.gov</a> if you run into any trouble.</div> </div> <p> </p> Fri, 06 Jul 2012 12:31:02 -0700 http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/important-notice-about-using-house/ http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/can-t-see-tickets-submitted-by-other-users/ <p> </p> <h2>Description</h2> <p>Currently only a user who submits a ticket can view and modify it.  This is because the ticket system does not have any concept of groups.</p> <h2>Solution</h2> <p>We are in the process of importing group information from our accounting database which will enable users to see tickets submitted by other users.</p> <h2>Status</h2> <p>In progress.  Testing in development test system. </p> Thu, 07 Jun 2012 23:03:33 -0700 http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/can-t-see-tickets-submitted-by-other-users/ http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/make-long-queue-work-fairly-across-all-groups/ <h2>Description</h2> <p>Currently a single user or group of users can take up all available slots for the long queue, blocking other users.</p> <h2>Solution</h2> <p>We plan to limit the number of long job slots a group can use.</p> <h2>Status</h2> <p>Long queue slots have been limited to 320 per user</p> Wed, 04 Apr 2012 08:00:00 -0700 http://www.nersc.gov/users/computational-systems/genepool/updates-and-status/open-issues/make-long-queue-work-fairly-across-all-groups/