Accessing Storage Configuration Settings
You’ll see the storage configuration page immediately after creating any storage resource.
You can also access configuration settings from the Storage tab. Click the name of any existing storage to configure it.
About the Storage Configuration Page
When you navigate to the storage configuration page, there are five tabs for customization.
Sessions
By default, you’ll see the Sessions tab when you navigate to storage settings. This tab shows your previous sessions for using the storage as well as provision and deletion logs.
For persistent storage resources, you can see the creation and deletion logs in this tab. All logs for current and previous sessions will be displayed. If a persistent storage resource is active, you’ll see all the clusters that the storage is attached to.
Ephemeral storage resources do not have a Sessions page. Instead, you can see their creation logs by navigating to Resources > ResourceName > Sessions > Logs > Storages.
For more information about persistent and ephemeral storage, please see Storage Types.
Definition
Here, you can adjust the configuration parameters of your storage. For more information, see General Settings below.
JSON
This tab shows the code version of your storage configuration settings. Here, you can manually adjust the parameters seen in the Definition tab.
Properties
This tab shows the name that you entered when the storage was created. Here, you can change the name and upload a new thumbnail for the cluster.
Sharing
This tab lets you control storage sharing options for users in your organization who are assigned to the same group.
For more information, please see Sharing Storage.
General Settings
Storage resources have these settings in the Definition tab of the configuration page.
Cloud Infrastructure
Use this dropdown menu to choose your infrastructure, which determines which region your storage will be created in.
Group
Use this dropdown menu to select the group name that your organization uses to allocate costs. This menu is especially important if your organization uses multiple groups.
If you’re not sure which group to select, you can contact us or your organization’s PW platform administrator.
Settings for Cloud Filesystems
AWS FSx for Lustre
The configuration parameters below are exclusive to AWS FSx for Lustre.
Availability Zone
Use this dropdown menu to select the Availability Zone that your infrastructure will be deployed in. An Availability Zone refers to an isolated location inside a region.
We recommend that your storage Availability Zone matches the Availability Zone of the cluster you’re pairing the storage with.
Storage Capacity (GB)
Use this field to enter the total capacity of your storage. This field is set to 1200 by default. 1200 is the minimum amount of storage and must be increased in increments of 2400.
Please note that altering the storage size will affect your estimated hourly cost.
Filesystem Deployment
Use this dropdown menu to select which type of filesystem your storage will be deployed as. Currently, the options are scratch and persistent.
Scratch filesystems are meant for shorter-term workloads. They provide a higher throughput per TiB of storage capacity. If a scratch filesystem fails, your data is not replicated.
Persistent filesystems are meant for long-term workloads. If this filesystem fails, your data will automatically replicate in the same Availability Zone.
Filesystem Compression
Use this dropdown menu to select whether your filesystem will be compressed. Currently, the options are LZ4 and NONE.
LZ4 compresses data when writing to Lustre. When reading from Lustre, LZ4 decompresses data while minimally impacting performance. Using LZ4 can improve read and write performance as well as reduce storage capacity.
S3 Import Path (Optional)
Use this field to enter an S3 bucket path to import data from. The format for this field is s3://BucketName.
S3 Export Path (Optional)
Use this field to enter an S3 bucket path to export data to. The format for this field is s3://BucketName.
Lustre on Azure
The configuration parameters below are exclusive to Lustre on Azure.
Region
This field identifies which region the storage will be deployed in. Please note that this field is locked because the region is determined by which Cloud Infrastructure is selected.
Lustre Image
Use this dropdown menu to select the Lustre image that your storage will use. The image has prepackaged software that our engineers have installed to maximize performance.
We recommend selecting Latest for the best performance.
MDS Nodes
Use this field to enter the number of nodes on your storage’s MDS. This field is set to 1 by default and will be sufficient for most workloads.
MDS stands for Metadata Service. In Lustre filesystems, the MDS manages the namespace hierarchy, records the layout of files, and handles object allocation on object storage targets (OSTs).
MDS fields largely affect data transfer speeds.
MDS Machine Type
Use this dropdown menu to select the machine type for your MDS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.
Please note that altering the MDS machine type may affect your estimated hourly cost.
MDS Boot Disk Type
Use this dropdown menu to select the type of disk your MDS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the boot disk type will slightly affect your estimated hourly cost.
MDS Boot Disk Size
Use this field to enter the total capacity of your MDS boot disk storage. This field is set to 20 by default.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
OSS Nodes
Use this field to enter the number of nodes on your storage’s OSS. This field is set to 2 by default.
Changing the number of OSS disk nodes changes your storage’s capacity and overall throughput; if you’re interested in tuning your Lustre filesystem, this parameter as well as OSS disk type will most strongly affect your storage’s performance.
OSS stands for Object Storage Servers. In Lustre filesystems, OSS provide the bulk data storage for all file content.
OSS Machine Type
Use this dropdown menu to select the machine type for your OSS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.
Please note that altering the OSS machine type may affect your estimated hourly cost.
OSS Boot Disk Type
Use this dropdown menu to select the type of disk your OSS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the boot disk type will slightly affect your estimated hourly cost.
OSS Boot Disk Size
Use this field to enter the total capacity of your OSS boot disk storage. This field is set to 20 by default.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
Accelerated Networking
Use this toggle button to enable accelerated networking, which improves networking performance for large workloads on multiple cloud clusters.
For more information, see the Azure documentation on accelerated networking.
Lustre on Google
The configuration parameters below are exclusive to Lustre on Google.
Region*
Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.
*This setting is only available on persistent storage options. For ephemeral storage options, the Region is determined by the attached cluster.
Zone*
Use this dropdown menu to select the zone to use your storage. A zone refers to an isolated location inside a region.
We recommend that your storage zone matches the zone of the cluster you’re pairing the storage with.
*This setting is only available on persistent storage options. For ephemeral storage options, the Zone is determined by the attached cluster.
MDS Nodes
Use this field to enter the number of nodes on your storage’s MDS. This field is set to 1 by default and will be sufficient for most workloads.
MDS stands for Metadata Service. In Lustre filesystems, the MDS manages the namespace hierarchy, records the layout of files, and handles object allocation on object storage targets (OSTs).
MDS fields largely affect data transfer speeds.
MDS Machine Type
Use this dropdown menu to select the machine type for your MDS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.
Please note that altering the MDS machine type may affect your estimated hourly cost.
MDS Boot Disk Type
Use this dropdown menu to select the type of disk your MDS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the boot disk type will slightly affect your estimated hourly cost.
MDS Boot Disk Size (GB)
Use this field to enter the total capacity of your MDS boot disk storage. This field is set to 20 by default.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
MDT Disk Type
Use this dropdown menu to select the type of disk your MDT nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the boot disk type will slightly affect your estimated hourly cost.
MDT stands for Metadata Target. Metadata content is stored on MDTs.
MDT fields largely affect data transfer speeds.
MDT Disk Size (GB) (Optional)
Use this field to enter the total capacity of your MDT boot disk storage.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
OSS Nodes
Use this field to enter the number of nodes on your storage’s OSS. This field is set to 2 by default.
Changing the number of OSS disk nodes changes your storage’s capacity and overall throughput; if you’re interested in tuning your Lustre filesystem, this parameter as well as OSS disk type will most strongly affect your storage’s performance.
OSS stands for Object Storage Servers. In Lustre filesystems, OSS provide the bulk data storage for all file content.
OSS Machine Type
Use this dropdown menu to select the machine type for your OSS nodes. The machine type determines the CPUs and amount of memory available. Certain machine types may also have specialty hardware, such as GPUs or low-latency networking options.
Please note that altering the OSS machine type may affect your estimated hourly cost.
OSS Boot Disk Type
Use this dropdown menu to select the type of disk your OSS nodes will use. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the boot disk type will slightly affect your estimated hourly cost.
OSS Boot Disk Size (GB)
Use this field to enter the total capacity of your OSS boot disk storage. This field is set to 20 by default.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
OST Disk Type
Use this dropdown menu to select the type of disk for your OST nodes. Currently, the options are local-ssd, pd-balanced, pd-standard, and pd-ssd.
Generally speaking, the ssd options will be faster than balanced options, and balanced options will be faster than standard options.
Please note that altering the OST disk type will affect your estimated hourly cost.
OST stands for Object Storage Target. An OST is a set of storage volumes that OSS provides access to.
OST Disk Size (GB)
Use this field to enter the total capacity of your OST boot disk storage. This field is set to 1500 by default.
Please note that altering the boot disk size will slightly affect your estimated hourly cost.
Lustre Version
Use this field to enter the version of Lustre software that your storage will use.
This field is set to latest-release by default, and we recommend using this setting for the best performance.
Lustre Image
Use this dropdown menu to select the Lustre image that your storage will use. The image has prepackaged software that our engineers have installed to maximize performance.
We recommend selecting Latest for the best performance.
GVNIC
Use this toggle button to enable Google Virtual Network Interface Card (gVNIC), which supports higher network bandwidths from 50–100 Gbps. This parameter affects both MDS and OSS nodes.
This storage feature can be toggled independently of the gVNIC option on your cluster; your cluster’s configuration will not be affected.
For more information, see the Google documentation on gVNIC.
Tier 1
Use this toggle button to enable Tier_1, which increases maximum egress bandwidth (upload speed) to 50–100 Gps. If Tier_1 is off, the egress bandwidth will range from 10–32 Gbps. This parameter affects both MDS and OSS nodes.
Please note that Tier-1 is only supported if gVNIC is also active. If you try to start Tier-1 by itself, the PW platform will display the error message Tier_1 is only supported if gVNIC is on.
For more information, see the Google documentation on Tier_1.
Google Filestore
The configuration parameters below are exclusive to Google Filestore.
Please note that Google Filestore measures storage in tebibytes (TiB); a tebibyte is equivalent to 1,099,511,627,776 bytes, or 1.1 teraybtes (TB).
Filestore Tier
Use this dropdown menu to select which service tier to use.
The tier BASIC_HDD supports 1 TiB to 10 TiB.
The tier BASIC_SSD supports 2.5 TiB to 10 TiB.
For more information, see the Google documentation on Filestore tiers.
Size TiB
Use this field and slider to choose the size of your storage in TiB.
Zone
Use this dropdown menu to select the zone to use your storage. A zone refers to an isolated location inside a region.
We recommend that your storage zone matches the zone of the cluster you’re pairing the storage with.
AWS Elastic File System
The configuration parameters below are exclusive to AWS Elastic File System (EFS).
Throughput Mode
Use this dropdown menu to select how much throughput will be available to your file system. AWS offers three throughput modes: bursting, elastic, and provisioned.
Bursting Throughput scales with the amount of storage in your file system.
Elastic Throughput (AWS recommended) is useful for spiky/unpredictable workloads, if your performance requirements are difficult to forecast, or when your application drives throughput at an average-to-peak ratio of 5% or less.
Provisioned Throughput is useful when you know your workload's performance requirements, or when your application drives throughput at an average-to-peak ratio of 5% or more.
For more information, please see the AWS documentation on throughput modes.
Settings for Cloud Object Stores
AWS S3 Bucket
The configuration parameters for AWS S3 Buckets are explained below.
Region
Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.
Google Cloud Bucket
The configuration parameters for Google Cloud Buckets are explained below.
Region
Use this dropdown menu to select the region that your storage will be deployed into. A region represents a geographic area.