Table 1 Specifications of the hybrid cloud system at the Center for Genomic Medicine, Kyoto University, Japan.
From: Design and implementation of a hybrid cloud system for large-scale human genomic research
Subsystem | Name | Specification | Size |
|---|---|---|---|
System A | File System | GPFS ESS JBOD (5U84) | 2.1 PB |
File System | NAS | 3.6 PB | |
Compute Nodes | Intel Xeon Haswell E7-8890 v3 (Haswell, 18 cores 2.5 GHz × 4)/512 GiB | Three nodes | |
Compute Nodes | Intel Xeon Ivy Bridge E7-4880 v2 (Ivy Bridge, 15 cores 2.5 GHz × 4)/512 GiB | One node | |
Compute Nodes | Intel Xeon E5-2698 v4 (Broadwell, 20 cores 2.2 GHz × 2)/8× Tesla P100 GPU/512GiB | One node | |
Job Schedular | Slurm | ||
Container | Singularity v3 | ||
Location | Center for Genomic Medicine, Kyoto University, Japan | ||
System B | System name | Laurel 2 | |
File System | Lustre ExaScaler (SFA14K) | 0.71(24) PB | |
Compute Nodes | Intel Xeon E5-2694 v4 (Broadwell, 18 cores 2.1 GHz × 2)/128 GiB | 17 (850) nodes | |
Network | Between network gateways and SINET (100 Gbps) Between file systems and compute nodes (100 Gbps) | ||
Job Schedular | PBS | ||
Container | Singularity v3 | ||
Location | Academic Center for Computing and Media Studies, Kyoto University, Japan | ||
System C | System name | Oakbridge-CX (OBCX) | |
File System | Lustre ExaScaler (ES18KE × 2) | 0.7 (12.4) PB | |
Compute Nodes | Intel Xeon Platinum 8280 (CascadeLake, 28 cores 2.7 GHz × 2)/192 GiB | 256 (1368) nodes | |
Network | Between the private network gateway and SINET (10 Gbps) Between public network gateways to SINET (40 Gbps × 2) Between file systems and compute nodes (100 Gbps) | ||
Job Schedular | Fujitsu Technical Computing Suite- | ||
Container | Singularity v3 | ||
Location | Information Technology Center, the University of Tokyo, Japan | ||
System D | System name | mdx | |
File System | Lustre File Sytem (NVMe) | 0.15 (1) PB | |
File System | Lustre File Sytem (HDD) | 0.3 (16.3) PB | |
Compute Nodes | Virtualization environment with 608 cores and 1024 GiB in total. 76 cores (152 vCPUs) and 128 GiB are assigned to each compute node. The physical CPU is Intel Xeon Platinum 8368 (IceLake, 38 cores 2.4 GHz × 2). | 8 (368) nodes | |
Network | Between compute nodes and SINET (25 Gbps) | ||
Job Schedular | Slurm | ||
Container | Singularity v3 | ||
Location | Information Technology Center, the University of Tokyo, Japan | ||
System E/P | File System | FSx for Lustre | On-demand |
File System | EBS | On-demand | |
File System | S3 | On-demand | |
Network | Between System A to AWS (AWS Direct Connect via SINET (10 Gbps)) | ||
Compute Nodes | AWS assigns requested instance if compute nodes are physically available. For our GATK joint-genotyping in System E, spot instance of r5.large (2 × vCPU (Intel Xeon Platinum 8000 3.1 GHz)/16 GiB)) was used (max 320 nodes). Available max resources as spot instance of vCPU and memory in AWS region in Japan are hpc6a.48xlarge (AMD EPYC Milan) with 192 vCPUs and x2iedn.32xlarge (Intel Xeon IceLake) 4096 GiB at 1/Jul/2022, respectively. | Spot | |
Location | AWS region in Japan (Systems E and P are in different virtual private cloud) | ||
