• Operating System: CentOS/Alma has been selected as the foundational operating system, providing a stable and robust platform for HPC operations.
• Drivers: The suite includes critical drivers such as CUDA for GPU acceleration, alongside network and storage drivers, to facilitate efficient data management and connectivity.
• File Systems: Support for both local file systems and Lustre, a high-performance distributed file system, is integrated to manage extensive data volumes effectively.
• Provisioning: The xCAT (Extreme Cluster Administration Toolkit) is employed for provisioning, automating the deployment and configuration of cluster resources.
• Resource Management: SLURM (Simple Linux Utility for Resource Management) is utilized for job scheduling and resource management, optimizing the allocation and use of resources.
• Cluster Management: It is achieved through tools such as Ganglia and Nagios, supplemented by C-DAC tools and Osticket for efficient issue tracking and support.
• Communication Libraries: Essential communication libraries, including MPI (Message Passing Interface), Intel MPI, MVAPICH, and PGAS (Partitioned Global Address Space), are integrated to enable effective parallel communication and data exchange.
• Development Tools: The suite features GNU Compiler & Intel oneAPI Compilers to support a broad spectrum of programming languages and optimizing computational performance.
• Application Libraries: Key libraries such as NetCDF for network data, mathematical libraries for numerical computations, GNU libraries, and Deep Learning/Machine Learning (DL/ML) libraries are included to support a wide range of application needs.
• Visualization Tools: Scientific visualization and data analysis are facilitated through tools like GNU Plot, ParaView, and VMD (Visual Molecular Dynamics), enabling detailed examination and interpretation of simulation results.
• Performance Monitoring: The software stack incorporates performance monitoring tools and benchmarks to continuously evaluate and enhance system performance.
• Software Ecosystem: The creation of software products like C-CHAKSHU, CHReME, HPC Tutor  PARADE, CAPC, User Creation Portal and Ticketing Support platforms reflects our emphasis on nurturing a robust software ecosystem. These solutions cater to diverse needs ranging from computational analysis and data visualization to project management, thereby augmenting research efficiency and productivity across scientific disciplines.
  • C-CHAKSHU: C-Chakshu is an HPC multi-cluster monitoring and Management platform that provides a unified dashboard over the web for all NSM sites with different geographic locations across India. Its web-enabled interface manages clusters of varied configurations and facilitates researchers and scientists of varied domains to run their applications with minimal efforts.
  • CHReME: To access Linux-based HPC environments, scientists and researchers require expertise in Linux and HPC, which is something many do not have. CHReME empowers users with an intuitive GUI to exploit HPC resources and provides a layer of abstraction to shield them from the complexity of accessing HPC resources. This enables them to concentrate on their core research/scientific work. It’s Web Interface makes clusters of different magnitudes easy to manage and monitor, which makes things easier for the HPC systems administrator while enabling researchers and scientists of varied domains to carry out their scientific simulations with minimal effort.
  • HPC Tutor: HPC Tutor is a revolutionary web based blended learning platform, designed to empower novice users with hands-on experience in the realm of High-Performance Computing (HPC). It seamlessly integrates theoretical knowledge with practical application, offering an immersive and educational journey in HPC Domain. This platform presents a unified web page, merging comprehensive presentation materials seamlessly with direct shell access to HPC clusters. This unique feature ensures a user-friendly and cohesive learning experience.
  • User Creation Portal : The User Creation Portal streamlines the account creation process by automating data collection and eliminating manual paperwork. It also enhances transparency by enabling users to monitor the progress of their requests and receive timely updates via email. This centralized platform ensures data accuracy and integrity, as users are required to provide official information verified through secure authentication mechanisms. Moreover, the portal offers a robust access control system, ensuring that sensitive user data is accessible only to authorized administrative personnel. This not only maintains data privacy but also facilitates efficient user management and oversight. By consolidating user information in a single location, the portal simplifies account auditing, reporting, and compliance processes, thereby improving overall operational efficiency and security.
  • PARADE: ParaDE is an environment for creating parallel applications on HPC platforms. Being web-based, it helps to access HPC irrespective of geographical location and carry out complete software development from compiling, debugging, automatic job submission, and project management, using multiple parallel paradigms such as OpenMP, MPI, etc. It enables users to just log in to the environment and use all software on the cluster without the hassles of installation or configuration on the client machine
  • CAPC: CAPC is an innovative solution that automatically parallelizes sequential code to different types of target parallel hardware without any inputs/contributions from the user. It liberates the application developer from the complexities of learning new parallel paradigms to write parallel programs for multicore and GPU parallel architectures
  • OS Ticket:  It is a ticketing system that streamlines HPC user support by routing inquiries, allowing for customization of data collection, and preventing duplicate responses from agents. It empowers agents with fast responses, SLA management, and collaboration tools. Users benefit from a quick response portal. This Desk provides valuable reporting and scales with the NSM vision, making it a powerful asset for managing user support. This HPC software stack ensures optimal performance and compatibility with NSM systems. It also aligns with India’s strategic goals of fostering technological self-reliance. By leveraging open-source technologies, the stack provides a flexible and cost-effective solution, driving innovation and bolstering capabilities within the national research and academic communities.

This HPC software stack ensures optimal performance and compatibility with NSM systems. It also aligns with India’s strategic goals of fostering technological self-reliance. By leveraging open-source technologies, the stack provides a flexible and cost-effective solution, driving innovation and bolstering capabilities within the national research and academic communities.

The deployment of this software suite has significantly enhanced the computational capabilities of NSM systems, empowering researchers and scientists to conduct high-impact research and achieve substantial advancements across various scientific and engineering disciplines.

This suit was deployed across all NSM HPC Facilities

• Trinetra-POC: This was proof of concept phase, with focus being getting know-how about switchless networks, 3D Torus topology, and onload/hybrid architecture of Network Controller Chip. The hardware used was based 40Gbps physical link layer for IO and PCIe Gen-3.0 2x interface for host interface. The PoC hardware was used for extensive validation of key architectural concepts and hardware/software co-verification.
• Trinetra-A is based on100Gbps physical link layer, PCIe Gen-3.0 x8 host interface, and uses 3D Torus as network topology. All hardware, software and firmware components of the network have been developed successfully. Trinetra-A is currently deployed in PARAM Rudra 1PF system at CDAC Pune.
• Trinetra-B is the latest hardware developed with performance and topology enhancements, leveraging experience gained during PoC and TrA development phases. TrB is based around PCIe Gen-3.0 x16 host interface, and 200Gbps physical link layer. Ten links @ 200Gbps allow for Dragonfly/ Supercluster topology. TrB planned to be deployed in PARAM 20PF system.

Trinetra development effort is expected to continue under forthcoming NSM2.0 with primary aim of indigenous development of Exascale systems.

• Use Cases
  • HPC and Enterprise applications
• Salient Features
  • High Performance: High bandwidth and low latency, supported by innovative hardware capabilities and a lightweight software stack.
  • Scalability and Power Efficiency: Designed to ensure real performance, scalability, and power efficiency, while facilitating experimentation and application optimization.
  • Topology Flexibility: Supports 3D Torus (Trinetra-A) and Dragonfly/Supercluster (Trinetra-B) topologies for large-scale scalability eliminating need for dedicated switches.
  • Multi-Processor Support: Compatible with multiple processor and platform architectures.
  • Industry Standards Compliance: Supports standard HPC programming interfaces like MPI and legacy socket interfaces.

• Fifth/Fourth Generation Xeon Emerald Rapids/ Sapphire Rapids Server Processor with 56 cores
• Dual Sockets with TDP support up to 300W per CPU.
• Memory: DDR5, RDIMM 4800 MT/s 1-SPC, 8 Channels per socket supporting up to 2 TB/socket.
• Other Interfaces supported:
  • One OCP NIC 3.0 cards
  • Four PCIe Gen5.0 (32GT/s) x16 ports,
  • Three UPI 2.0 links support per CPU (12.8 GT/s, 14.4 GT/s, 16 GT/s)
  • Two (stacked) M.2 NVMe
  • One x8 PCIe 3.0 Ethernet card

This server is compliant with EMI/EMC and safety certifications such as CE, FCC, BIS and UL.

Its development and manufacturing has been instrumental in capability building, boost to manufacturing industry, Partner development in India.

Versatile design of this server is allowing possibility of many variants

• Sixth Generation Xeon Granite Rapids Server Processor with up to 120 cores
• Dual Sockets with TDP support up to 500W per CPU.
• Memory support: DDR5; 12 Channels of DDR5 per socket supporting up to 3TB/socket. Memory Speed DDR5 RDIMM 6400 MT/s 1-SPC; 8800 MCR DIMM
• DC-SCM 2.0 Specification compatible
• Other Interfaces supported:
  • Two OCP NIC 3.0 cards
  • Up to 8 PCIe Gen5.0 (32GT/s) x16 ports,
  • Up to 4 x16 CXL 2.0 Ports
  • Up to 6 UPI 2.0 links support per CPU (2.5 GT/s, 16 GT/s, 20 GT/s, and 24 GT/s).

• Suitable for diverse applications such as, High performance computing, cloud and edge computing and communication
• Dual socket, 2nd Gen Intel Xeon Scalable Processors (upto TDP 165W)
• 20 DDR4 DIMM slots LRDIMM, RDIMM (total upto 1.28TB)
• Support for Intel Optane Persistent Memory (NVDIMM)
• Two PCIe 3.0 x16 slots with GPU/Accelerator support
• Onboard 2 x 10Gbps ethernet Interfaces
• One additional high speed network slot (Upto 100Gbps, OCP NIC 3.0 compatible)
• One 1GbE BMC management port
• Support for IPMI2.0
• Support for KVM
• Support for two U.2 NVMe/SATA

This server is compliant with EMI/EMC and safety certifications such as CE, FCC, BIS and UL.

Its development and manufacturing has been instrumental in capability building, boost to manufacturing industry, Partner development in India.

Versatile design of this server is allowing possibility of many variants