NERSCPowering Scientific Discovery Since 1974

Analytics & Visualization

Overview and Strategies

Data analysis and data mining tools are used to compare datasets or find features within a dataset. Visualization of data is one of the primary tools for data exploration, and may precede or inspire more formal data analyses. The technologies described above may be used individually or together to explore data. Data exploration discusses how analytics technologies can be integrated to provide a framework for discovery. In general, scientific data management and workflow management are enabling technologies. Scientific data management provides tools for efficient access to large amounts of data, as well as supporting data organization and security. Workflow management describes a systematic approach to data processing pipelines or the pre-processing and post-processing steps involved in running simulations. Workflow management tools can be used to automate repetitive processing tasks and make processing pipelines more robust.

discusses how analytics technologies can be integrated to provide a framework for discovery.

How to choose the right system at NERSC:

  • Small datasets
  • Large datasets

How to choose the right tools and software to use

Analytics and Visualization Software at NERSC

Data analysis and visualization are two steps in data understanding, often interleaved and symbiotic, so many of the available tools characterized as one category, end-up having some functionalities of the other. Bellow find a list of tools organized under Analytics or Visualization, but have in mind that they may have a yet large intersection in terms of their functions.

Visualization Analytics
Visit Matlab
Paraview Mathematica
  Python
AVS R
  Python tools - Numpy, Scipy, iPython, matplotlib
   
  TCL/TK
  SQL

Link to Visit Software page

Link to Paraview Software page

R

SQL

Matlab

Mathematica

Python tools - Numpy, Scipy, iPython, matplotlib

 

(This may be a restful query, a dynamic list of Visit pages.)

Case Studies

(Dani to decide what needs to be brought over)

Visualization

Workflow Management?

Data Analysis and Mining


Data Exploration

  • Case Study: Astrophysics

Quick Links: NERSC Tools for Data Exploration

 

jet-sdavid.jpg

Visualization

Visualization facilitates data exploration. It often supports simulation since it allows inspection of the output varying in time or with changes in parameter values, or for the locations of interesting regions in large data sets. Read More »

Hadoop

| Tags: Data

Hadoop is an open-source implementation of the popular MapReduce Programming Model. In addition to MapReduce and a dsitributed File System (HDFS), Hadoop has a rich ecosystem of high-level languages (i.e. Pig) and data storage models (HBase, Hive, etc). Read More »

energy16gunther.jpg

Data Exploration

Among the modalities of data exploration, visual exploration is often necessary to guarantee that the algorithms are performing what they are supposed using some dataset. This page illustrates some of the scientific problems explored in terms of visualization tools. Read More »

microCTdani.jpg

Data Analysis and Mining

Data analysis techniques include post-processing (e.g., data statistics) of experimental datasets and/or simulation output, as well as the use of mathematical methods (e.g., filtering data) and statistical tests. Data mining usually refers to the application of more advanced mathematical techniques such as classification, clustering, pattern recognition, etc. Read More »

volrend-swes.png

Strategies for Choosing Analytics and Visualization Software and Hardware at NERSC

Data analysis (or analytics) and visualization are two steps in data understanding, often interleaved and symbiotic, so that many of the available tools characterized as one category, end-up having some functionalities of the other. Bellow find links to software tools grouped under Analytics or Visualization, but have in mind that their functions may be interchangeable. VisualizationAnalyticsVisit Matlab Python tools: Numpy, Scipy, iPython, matplotlib Paraview Mathematica Perl IDL Python TCL/TK… Read More »

collage.jpg

Workflow Management

Workflow management refers to the process of connecting various software tools based on specific input and output parameters. The goal of workflow management is to automate a specific sets of tasks. Read More »