High-Performance Computing, or HPC as it is generally known, has been a mainstay of infrastructure and hardware engineering groups for decades. Recent advancements in cloud computing have enabled a democratization of this once-specialized space that was restricted to hardware experts. Using proprietary parallel computing frameworks, RxDS has developed unique cluster-computing capabilities that can scale to more than 20,000+ AWS cores automatically, provision EC2 instances based on cost-based or performance-based instance selection, provide middleware connectivity, and much more.

Our flagship products, qPar and qIO, offer a wide range of performance and I/O functionalities for HPC Workloads. 

hpc01

qPar Common Features

  • Simple API for burst jobs, remote execution
  • Database storage for tasks, inputs, outpost, and trace
  • IPC-based bidirectional communication for orchestration and scheduling
  • Task scheduling managed inside the programming environment such as Matlab avoids startup delays
  • Simple orchestration controlled by one database
  • Use any instance type for any type of job. Only one AMI to maintain for all sleeves.
  • Choose your own strategy approach: Reduce Costs (shortest idle time, launch only when tasks ready), Improve Performance (warm up nodes ahead, long idle time in case new tasks come up)

 

 
hpc02

qPar and I/O

Further, qPar includes a comprehensive orchestration and scheduling layer that can automate scalable parallel workloads. The platform can run up to 20,000 tasks in parallel from a single node to reliably manage orchestration across thousands of independent tasks. Utilizing AWS spot instances delivers between 50-70% cost savings over running on-demand.

Simplicity of administration is achieved via a pre-built image. The administrative interface can capture task failures including stack trace from remote nodes. A user-friendly API for the Python, Matlab, and Julia programmer allows developers to utilize the capabilities from a familiar programming environment.

Recognizing that I/O is one of the common bottlenecks in data-heavy workloads, we have also developed our own I/O library. Known as qIO, it can deliver a peak speed of 15 Gigabytes per second of read on a single AWS instance.

qIO is built on strong technical foundations:

  • Unique C-based library to communicate with the database
  • No reliance of ODBC/JDBC drivers
  • Complex datatypes handle via low-level APIs - direct memory access to both Julia database objects
  • Configurable write-handlers and stream-handlers to efficiently support thousands of database clients

Schedule a Meeting