Data integrity problem after the vmstorage breaks down

[SRE-Rick]

Is your question request related to a specific component?

victoriametrics vmstorage

Describe the question in detail

Hello, I have a question for you. I have three vmstorage copies and two vminsert copies insert-replicationFactor =2. Normally, the same data will be randomly written into two vmstorage copies. Assuming that vmstorage01 and vmstorage02 are broken down, data will be written to the vmstorage03 E03 device (in this case, the data is only stored in the vmstorage03 device and is not backed up in the E03 device), then when my vmstorage01 02 recovers, Will data in vmstorage03 be automatically synchronized to vmstorage01 02?
Best regards

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[jiekun]

Will data in vmstorage03 be automatically synchronized to vmstorage01 02

I believevmstorage is not aware of each other. And no data migration between vmstorage nodes will happen.

[SRE-Rick]

vmstorage03中的数据会自动同步到vmstorage01 02吗

相信vmstorage彼此都不认识。并且vmstorage节点之间不会发生数据迁移。

In this case, some data is stored in the vmstorage03 source at one point

[jiekun]

vmstorage03中的数据会自动同步到vmstorage01 02吗

相信vmstorage彼此都不认识。并且vmstorage节点之间不会发生数据迁移。

In this case, some data is stored in the vmstorage03 source at one point

I believe so. Typically vmselect send request to all vmstorage nodes, get their response and merge them into one tss.

[SRE-Rick]

I believe so. Typically vmselect send request to all vmstorage nodes, get their response and merge them into one tss.

Thank you very much! I have one more question for you. I have three storage nodes. When the disk usage of three UDsns reaches 80%, add two UDsns to relieve the disk pressure. How should I adjust the configuration so that new data is written to two new nodes? I want disk usage on all storage nodes to be as even as possible

[jiekun]

1. If you are deploying everything on Kubernetes, it would be beneficial to use VMOperator. You can find replicaCount config in VMCluster CRD, and once you increase this value, the operator will start new vmstorage nodes , while vmselect and vminsert will also affected and send/read data to/from different vmstorage nodes.
2. If you are maintaining all components manually, you can find docs here: https://docs.victoriametrics.com/cluster-victoriametrics/#cluster-resizing-and-scalability

And in order to make disk usage to be as even as possible, you can refer those instruction on docs:

In order to handle uneven disk space usage distribution after adding new vmstorage node it is possible to update vminsert configuration to route newly ingested metrics only to new storage nodes. Once disk usage will be similar configuration can be updated to include all nodes again. Note that vmselect nodes need to reference all storage nodes for querying.