TY - GEN
T1 - Efficient data restructuring and aggregation for I/O acceleration in PIDX
AU - Kumar, Sidharth
AU - Vishwanath, Venkatram
AU - Carns, Philip
AU - Levine, Joshua A.
AU - Latham, Robert
AU - Scorzelli, Giorgio
AU - Kolla, Hemanth
AU - Grout, Ray
AU - Ross, Robert
AU - Papka, Michael E.
AU - Chen, Jacqueline
AU - Pascucci, Valerio
PY - 2012
Y1 - 2012
N2 - Hierarchical, multiresolution data representations enable interactive analysis and visualization of large-scale simulations. One promising application of these techniques is to store high performance computing simulation output in a hierarchical Z (HZ) ordering that translates data from a Cartesian coordinate scheme to a one-dimensional array ordered by locality at different resolution levels. However, when the dimensions of the simulation data are not an even power of 2, parallel HZ ordering produces sparse memory and network access patterns that inhibit I/O performance. This work presents a new technique for parallel HZ ordering of simulation datasets that restructures simulation data into large (power of 2) blocks to facilitate efficient I/O aggregation. We perform both weak and strong scaling experiments using the S3D combustion application on both Cray-XE6 (65,536 cores) and IBM Blue Gene/P (131,072 cores) platforms. We demonstrate that data can be written in hierarchical, multiresolution format with performance competitive to that of native data-ordering methods.
AB - Hierarchical, multiresolution data representations enable interactive analysis and visualization of large-scale simulations. One promising application of these techniques is to store high performance computing simulation output in a hierarchical Z (HZ) ordering that translates data from a Cartesian coordinate scheme to a one-dimensional array ordered by locality at different resolution levels. However, when the dimensions of the simulation data are not an even power of 2, parallel HZ ordering produces sparse memory and network access patterns that inhibit I/O performance. This work presents a new technique for parallel HZ ordering of simulation datasets that restructures simulation data into large (power of 2) blocks to facilitate efficient I/O aggregation. We perform both weak and strong scaling experiments using the S3D combustion application on both Cray-XE6 (65,536 cores) and IBM Blue Gene/P (131,072 cores) platforms. We demonstrate that data can be written in hierarchical, multiresolution format with performance competitive to that of native data-ordering methods.
UR - http://www.scopus.com/inward/record.url?scp=84877712237&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84877712237&partnerID=8YFLogxK
U2 - 10.1109/SC.2012.54
DO - 10.1109/SC.2012.54
M3 - Conference contribution
AN - SCOPUS:84877712237
SN - 9781467308069
T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC
BT - 2012 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2012
T2 - 2012 24th International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2012
Y2 - 10 November 2012 through 16 November 2012
ER -