Monitoring Costs
The PW platform features a dashboard that tracks costs across all cloud service providers (CSPs). The cost dashboard shows metrics down to an individual user’s cluster session. Cost data is pulled directly from CSPs to show charges by usage type, such as compute instances and storage.
The dashboard’s real-time cost tracking feature is designed to provide an accurate approximation of cloud costs between billing updates, capturing resource usage since the most recent official billing update. Official billing updates can be spaced by many hours; a large cluster can incur substantial expenses in that time window, so near real-time billing is an important feature to help users monitor their costs.
Please note that you won’t see the cost dashboard if your organization has not enabled it.
The Cost Dashboard
To reach the cost dashboard, navigate to the Cost page.
By default, you’ll see the following data in the cost dashboard. You can change the view at any time; for more information, please see Filters and Layout below.
Please note that all monetary values are listed in U.S. dollars (USD).
Graphs
The cost dashboard features several graphs to help users visualize data.
Unless otherwise noted, all filters can change a graph’s displayed data. For more details, please see Filters below.
Daily Cost
This graph shows the accumulated daily cost for users in a group within the selected Time filter.
You can mouse over any bar to see detailed costs for that day. For more details, please see About Cost Types below.
Total Spend Over Time
This graph shows a group’s total expenditures to date.
Because this graph shows an accumulation of cost data for a group, only the Group filter changes its displayed data.
Cost by User
This graph shows a a breakdown of costs by group users within the selected Time filter.
You can mouse over unlabeled pieces of the graph to see usernames.
Cost by Type
This graph shows a breakdown of costs by type within the selected Time filter.
You can mouse over unlabeled pieces of the graph to see cost types.
For more details about these costs, please see About Cost Types below.
Metrics
The cost dashboard features several data modules that display important usage metrics.
Unless otherwise noted, all filters can change a metric’s displayed data. For more details, please see Filters below.
Filtered Cost
This metric shows the costs for the filters you’ve applied.
For example, the cost dashboard defaults to data for the month to date and includes real-time tracking. The Filtered Cost module displays data only for those users within those parameters until you change the filters.
Total Allocations
This metric shows the total allocations that have been allotted for a group.
Allocations are set by PW platform administrators; for more information, please see Enforcing Allocations (coming soon).
Remaining Allocations
This metric shows how much a group can spend before reaching its set cost allocation.
Active Users
This metric shows the active users for the filters you’ve applied.
For example, the cost dashboard defaults to data for the month to date and includes real-time tracking. There could be one hundred total users in your selected group, but with these filters, Active Users would show how many users used clusters and/or storage within that time period.
Total Spend
This metric shows the amount all users in your group have spent.
Because this graph shows an accumulation of cost data for a group, only the Group filter changes its displayed data.
If real-time tracking is enabled, this metric updates every three minutes.
If real-time tracking is disabled, update time is CSP dependent.
- AWS updates two to three times each day. They don't have a set billing time.
- Azure updates once each day. Their billing time depends on when your billing report was created.
- Google updates five to eight times each day. They don't have a set billing time.
Percent Used Allocations
This metric shows the percentage of allocations used by all users in a group.
Because this graph shows an accumulation of cost data for a group, only the Group filter changes its displayed data.
If real-time tracking is enabled, this metric updates every three minutes. If real-time tracking is disabled, update time is CSP dependent.
Tables
The cost dashboard features several tables that display important usage data.
All filters can change a table’s displayed data. For more details, please see Filters below.
Please note that all tables list data in descending order of costs.
User Cost
This table shows a breakdown of costs by user within the selected time frame.
Cost by Session
This table shows a breakdown of costs by users’ sessions within the selected time frame.
Cost by Instance Type
This table shows a breakdown of costs by instance type within the selected time frame.
Cost by Type
This table shows a breakdown of costs by type within the selected time frame.
For more details about these costs, please see About Cost Types below.
Customizing the Cost Dashboard
You can customize the cost dashboard at any time by using different filters or changing the layout of the page.
Filters
There are a variety of options for filtering the cost dashboard’s data, including:
- Group
- Time
- Realtime
- User
- Type
- Pool
- Session
Three filters must have options selected: Group, Realtime, and Time. These options are pinned to the top of the Cost page. By default, these options are set to the first group you were added to, included real-time billing, and costs for the month to date. You can click any of these options to change them.
To add additional filters, click Filter Options and select any filter from the list. Next, use the dropdown menu to select the filter parameter. All filter dropdown menus include a search bar for quickly finding parameters.
Please note that some filters are conditional. For example, you must select a User before you can select a Pool. You must select a Pool before you can select a Session.
Layout
You can change the layout of the cost dashboard at any time. Your changes will not affect other users in your organization.
Click Options, then Unlock Layout.
When the cost dashboard is in editing mode, a bracket will appear in the bottom-right corner for each data module.
Drag and drop modules to change their positions on the page.
To resize a module, click the bracket in the bottom-right corner and drag vertically or horizontally.
Click the delete icon to remove a module from the page.
When you’re done making changes, click Options > Save Layout, then Lock Layout. Your cost dashboard’s layout will remain in this state until you make further changes.
Click Options > Reset Layout, then Save Layout to revert the page to its default state.
Printing Data
You can print the cost dashboard data at any time.
Click Options, then Print.
A Print window will appear. Select the option for Save as PDF. Click Save.
The cost dashboard page will be downloaded to your device as a PDF.
About Cost Types
Project costs can be divided into three broad categories: compute, storage, and network charges.
Compute costs include the controller, compute nodes, and custom images. Because compute nodes are capable of starting and stopping elastically based on scheduler load, charges can be minimized when a cluster is left on with a period of idle time between jobs.
Storage costs include disks attached to compute nodes, as well as optional storage deployed with a cluster. Other storage charges include persistent storage maintained outside of cluster configuration. For more information, please see Storage Types.
Network charges may accrue if data is transferred from the cloud or across regions.
Compute Charges
Compute
This cost type includes instance-related charges. Primarily, these costs include the instances themselves. Some CSPs also include attached items, such as external network interfaces.
Compute Nodes
This cost type is similar to Compute costs, but only includes data for compute nodes.
Storage Charges
Storage
This cost type is associated with pwbilling and ca-cloudmgmt, which applies only to AWS instances.
Storage-Disk
This cost type applies to disk storage on Azure instances.
Storage-EBS
This cost type applies to AWS instances. Elastic Block Storage (EBS) is a subset of the AWS service Elastic Cloud Compute (EC2). Storage-EBS includes instance local disks as well as apps and image snapshots.
Storage-EFS
This cost type applies to AWS instances. Elastic Filesystem (EFS) is an AWS feature.
Storage-File
This cost type applies to Azure instances with Premium LRS All Other Operations enabled.
Storage-GCS
This cost type applies to Google instances with Google Cloud Storage (GCS) enabled.
Storage-Image
This cost type includes snapshots on all CSPs.
Storage-Lustre
This cost type applies to Lustre storage resources.
Storage-PD
This cost type includes local boot disks for Google instances. PD stands for persistent disk.
Storage-S3
This cost type applies to AWS instances. Simple Storage Service (S3) is a bucket storage resource from AWS.
Network Charges
Egress
This cost type applies to data transfers out of the cloud.
Ingress
This cost type applies to data transfers into the cloud.
To better understand what counts as ingress and egress on a particular cloud, please see their respective documentation.
Other Charges
Infrastructure
This cost type includes cloud infrastructure charges, such as network transit gateways.
Other
This cost type includes miscellaneous charges that are not essential for tracking purposes. Charges in this category include items like API requests, security operations, and CSP-specific support fees.
Unknown
This cost type includes charges that haven't been labeled on the PW platform yet.
Our cost types are based on the way CSPs name their charges. Because each CSP stores many different types of cost data, our engineering team must sort, consolidate, and map that data to corresponding PW cost types. Unknown costs are re-assessed quarterly.