Running Jobs
Full Document
Managing Memory Usage
Each Bassi node has 32 GB of total memory. Some memory is used by the operating system (AIX, HPS switch, etc); the remainder - about 26.4 GB - is available for user applications. If your code tries to use more than the available physical memory it will start writing memory pages to disk. This will certainly cause poor code performance and excessive paging will lead to unpredictable behavior: code crashes, hangs, and possibly node crashes. Any job that tries to use more than 27032 MB of memory on a node will be killed by the system software. Please plan your runs to fit into the available physical memory on each node.
Bassi's POWER 5 nodes support a "large page" memory configuration. High Performance scientific applications generally benefit from using "large page." By default, parallel applications built on Bassi are enabled to use large pages. It is possible to build binaries that do not use large pages, however. See Memory Considerations for more details.
The Bassi compute nodes are configured to use 20 GB of large-page memory. The size of the large-page pool is set at boot time and cannot be otherwise changed. If a large-page-enabled application requests more memory than is available in large pages, additional memory will be allocated and used from the small memory pool (perhaps at lower performance). However, a code that runs only in small pages cannot access the large-page memory pool.
Please use the following table to help make effective use of the memory on the Bassi compute nodes. For running interactive serical utilities on the login nodes, see Running Interactive Jobs.
| Bassi Compute Nodes (8 CPUs per node) | |
|---|---|
| Total Physical Memory: | 32 GB |
| Large-Page Memory Pool Size: | 20 GB |
| Large-Page Memory Available to Large-Page Enabled User Applications: | 18.922 GB |
| Memory in Small Pages: | 12 GB |
| Small-Page Memory Available to User Applications: | ~7-8 GB |
| Total Memory Available to Large-Page Enabled User Applications: | 26.4 GB |
| Total Memory Available to Non-Large-Page Enabled User Applications: | ~8 GB max |
Job Memory Limits
The maximum memory your code can use on a single node is 26.4 GB (27032 MB). If you want to be able to access the maximum amount of memory, consult the following table to see what you need to specify in your LoadLeveler batch script:| Tasks per Node | LL directive |
|---|---|
| 8 | No specification needed. |
| 7 | #@resources = ConsumableMemory(3861 mb) |
| 6 | #@resources = ConsumableMemory(4505 mb) |
| 5 | #@resources = ConsumableMemory(5406 mb) |
| 4 | #@resources = ConsumableMemory(6758 mb) |
| 3 | #@resources = ConsumableMemory(9010 mb) |
| 2 | #@resources = ConsumableMemory(13516 mb) |
| 1 | #@resources = ConsumableMemory(27032 mb) |
Memory limits are being enforced via LoadLeveler. Jobs that request more than 27032 MB (26.4 GB) of memory per node will not start. Default values are in place so that job scripts that request 8 tasks per node can access the full 26.4 GB (27032 MB) of memory. Jobs will be killed if they try to access more than that amount of memory per node.
Jobs that run with fewer than 8 tasks per node need to correctly specify their memory usage in batch scripts. The maximum memory a job can use on a single node is calculated by:
(tasks per node requested) * (ConsumableMemory requested per task)
The default value of ConsumableMemory per task is set at 3379 mb. If you run fewer than 8 tasks per node and want to access the full amount of memory permitted (27032 mb), you need add a line to your LoadLeveler batch script:
#@resources = ConsumableMemory(N mb)
where N is the amount of memory per task that your job will use. NOTE: Each task is not limited to N mb; rather, all tasks on a given node are limited to using an aggregate amount of memory equal to: (number of tasks on that node)*(N mb) with a maximum limit of 27032 mb. N must be an integer. Here "tasks" refers to the number of independent instances of your executable that are started (initiated) on a node.
For example, to run 2 tasks per node and access the maximum amount of memory, set this in your batch script:#@tasks_per_node = 2 #@resources = ConsumableMemory(13516 mb)
