Scalyr has a default log volume dashboard that shows log volume breakdown against hosts and K8s cluster (if applicable).
It provides convenience to the end-users to monitor their log volume. However, as the log volumes increase on the account, you may need more insights on what the major contributors are and require a breakdown against terms other than the host.
This can be easily accomplished by leveraging the
join statement in PowerQuery. All you need to do is to join the two tables by serverHost/k8s-cluster, and group by the custom field again to get the complete picture of the log volume breakdown by any custom field in your logs.
For example, I want to break down log volume using the field
serverIp in my logs. I can simply use the following query to get the answer.
vol = ($tag == "logVolume" metric='logBytes'|group MB = sum(value)/1024/1024 by host = host? host : k8s-deployment|sort -MB),
ServerIp = (serverIP == *|group serverIP = any(serverIP ) by host = serverHost)
| group mb = sum(MB) by serverIP
| sort -mb
Note: keep in mind that the selected custom field
serverIP needs to have a 1:1 relationship with the host to get an accurate result. Otherwise, selecting any(serverIp) will grab a random IP from the set and attach it to the host. You can use the query
serverIp = * | group estimate_distinct(serverIp) by host = serverHost to verify the 1 to 1 mapping relationship.