Index snapshot and deletion
Index Management Policy
Below is an example of an OpenDistro based index management policy for a 90-day retention.
{
"policy": {
"description": "Takes an index snapshot.",
"schema_version": 1,
"error_notification": null,
"default_state": "current",
"states": [
{
"name": "old",
"actions": [],
"transitions": [
{
"state_name": "old",
"conditions": {
"min_index_age": "1d"
}
}
]
},
{
"name": "current",
"actions": [
{
"snapshot": {
"repository": "siemonster-<instance_name>",
"snapshot": "siemonster-<instance_name>-snapshot"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "90d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
]
}
}
Below is an example of an OpenSearch based index management policy for a 90-day retention.
{
"policy": {
"description": "Takes an index snapshot.",
"schema_version": 13,
"error_notification": null,
"default_state": "current",
"states": [
{
"name": "old",
"actions": [],
"transitions": [
{
"state_name": "old",
"conditions": {
"min_index_age": "1d"
}
}
]
},
{
"name": "current",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"snapshot": {
"repository": "siemonster-<instance_name>",
"snapshot": "siemonster-<instance_name>-snapshot"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "90d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"wazuh-alerts-4.x-*",
"wazuh-archives-4.x-*",
"wazuh-alerts-*"
],
"priority": 0
}
]
}
}Creating the Policy in Index Management
The following is a step by step to create the policy in Index Management.
1. Log into your SIEMonster Environment
2. Click the Dashboard tile
3. Once the tile opens and the screen loads, click the 3-line/burger menu dropdown in the top left corner
4. Click on the “Index management” shortcut
5. If you are not automatically taken to the index management policy screen, Click Index Policies in the top left.
6. For a new policy Click the “Create Policy” button in the top right.
7. For OpenDistro:
a. Specify the name in the Policy ID block NOTE: This may not contain uppercase letters or symbols. The only symbols allowed are underscores or hyphen.
b. Paste the text from the sample provided earlier in the document
c. Set the retention period as required by changing the number of days
d. Click Create
e. Click on “Index Management” at the top left
f. Click “Indices” in the left column
g. Type “wazuh-alerts” in the search box
h. Click the Select All button to select all indices on the page
i. Do this for each index that has no in the “Managed by Policy” column, there might be multiple pages, go through all of them and apply the policy where necessary.
8. For OpenSearch:
a. Click JSON editor and Click Continue
b. Specify the name in the Policy ID block NOTE: This may not contain uppercase letters or symbols. The only symbols allowed is underscore or hyphen.
c. Paste the text from the sample provided earlier in the document
d. Set the retention period as required by changing the number of days
e. Specify additional index patterns if they should be required.
Leave the system for 30 to 45 minutes. If the policy has not been applied to all or only some indices that match the patterns specified in 8.e above, please perform the step in 4.1.7 e to i
f. Click Create
The above steps will then proceed to initialize the snapshots for the indicated indices. Depending on the number of indices and their size this can take quite some time. The progress of this can be checked by going to Index management and then selecting “Managed Indices” from the left column. This will indicate the current state of the index management, if a retention has been applied and the snapshot has successfully completed it will be in a transition state until it reaches the age specified in the policy.
Adding the policy to the index template (OpenDistro Only)
For OpenDistro it is required to add some configuration items to the index template to automatically apply the policy to future indices. The following steps need to be followed exactly to prevent problems with data and Elastic ingestion.
1. At the top of your instance landing place, Click on Modules, then click on Comrade
2. Click on the instance block names SIEMonster
3. Click the three bar/burger menu at the top left and Click Index Templates
4. Find the index template that applies to your Wazuh-alerts-* index pattern and Click the edit pencil on the right hand side NOTE: take care not to delete the template as this will have significant impact on your data and ingestion
5. The default template will look like this:
6. On line 15 in the example, go to the end of the line and Press [ENTER], then paste the following
"opendistro": {
"index_state_management": {
"policy_id": "daily_snapshot"
}
},7. Ensure that the line starting with policy_id reflects the name you have assigned to your index management policy. It should look like the following once complete:
8. Click Update Template
Your index management policy should now apply to all index patterns specified.
Troubleshooting
There may be situations where snapshots are not running or not completed successfully. This can be caused by a variety of reasons. The following items will assist you with troubleshooting the root cause and applying a fix.
Indices are no longer automatically receiving policy assignment or are not listed in Index Management under Managed Indices
• Check if the index template still contains the modification applied in section 4.2.7, if it is missing re-apply it, go to Index management and apply the policy manually to each index that isn’t currently associated with the index policy
• An error message appears in Index management with a message relating to an inability to access the S3 repository or NFS data store. Ensure the NFS share is still accessible and has sufficient storage. If it is an S3 bucket re-apply the S3 credentials. The S3 credentials are stored in a credential store that is not persistent for various logistical reasons and if the POD/Container was restarted the system will need this re-applied on all Master and data nodes in OpenDistro/OpenSearch.
1000 shards are reached regardless of policy applied
This will occur under the following circumstances:
• The index management policy is not being applied and removing data older than the retention. Check that the configuration from the steps provided above are still in place.
• There are additional index patterns that create a daily index that is not covered by the Index management policy. Either add them to the policy or create a new index template and policy specific to them using the steps above.
Elasticsearch out of storage
Adjust the policy to have a shorter retention period or add additional disks. Note: Policy changes won’t be retrospectively applied. It will only apply to new indexes added after the changes have been made. In cases where the disk needs to be recovered one can delete the indices covered by the shorter retention period to free up some disk, then remove the existing policy from the indices and re-apply the updated template.
Failed to create Snapshot - Invalid snapshot name
When applying the policy to many indices at the same time many may be triggered at the same time thereby creating the same snapshot name, to repair these simply select the index in index management and select retry policy, leave it on retry policy from current action Click retry.