Index Management and Snapshot Management
Index Management
The Index Mangement component of the system administers various aspect of the the backend system relating to indexes. To access this component, Click “Event Search” then Expand the three bar menu, Expand Management and Click “Index Management”
Once the page loads you will see all options available under Index Management, which are:
State management indexes
Indexes
Data streams
Templates
Aliases
Rollup Jobs
Transform Jobs
Notification Settings
There is also a Snapshot Management section on the same page containing:
Snapshot policies
Snapshots
Repositories
State management policies
The state management policies section relates to policies applied to indices that will manage their states between hot, warm and cold. There are various options available in such policies that can be utilized. The most basic is controlling the hot and cold periods, where indices are migrated to cold storage for longer term preservation. This is most often required for compliance purposes.
This section will list any policies currently applied and allow you to see their configuration by clicking on the blue url with the name of the policy. In the above screenshot the policy and url is “SIEMonster Data Lifecycle”
When you click on the policy it will show you the current configuration in the webui. This configuration screen consists of three sections, Policy settings, ISM templates and States.
Policy Settings
The Policy Settings section shows the following information:
Policy Name
Policy Description
Error notifications (if an error state exists it will be flagged here)
Sequence Number (this is an internal reference number to OpenSearch for basic function and event tracking)
Priority term (this refers to the priority of the policy in case multiple policies exist)
Last update (the last time this policy was updated)
ISM templates
The ISM templates section contains the following information
Index Patterns (these are the index patterns this policy applies to)
Priority (this refers to the priority of the policy in case multiple templates exist)
States
The States section shows the transition path of all states within the policy. The screenshot below indicates how the policy applies to the states of the data stored in OpenSearch in the Index Patterns indicated in the section https://siemonster.atlassian.net/wiki/spaces/current/pages/edit-v2/284590097#ISM-templates
Viewing and Editing the policy
Viewing
To view the policy Click the “View JSON” button in the top right corner of the page.
This will bring up the JSON data contained within the policy.
{
"id": "SIEMonster Data Lifecycle",
"seqNo": 29795,
"primaryTerm": 7,
"policy": {
"policy_id": "SIEMonster Data Lifecycle",
"description": "Manages the lifecycle of SIEM data",
"last_updated_time": 1746407547193,
"schema_version": 19,
"error_notification": null,
"default_state": "hot_state",
"states": [
{
"name": "hot_state",
"actions": [],
"transitions": [
{
"state_name": "warm_state",
"conditions": {
"min_index_age": "2d"
}
}
]
},
{
"name": "warm_state",
"actions": [
{
"timeout": "2h",
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"force_merge": {
"max_num_segments": 1
}
}
],
"transitions": [
{
"state_name": "delete_state",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "delete_state",
"actions": [
{
"timeout": "2h",
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"wazuh-alerts-*",
"wazuh-states-*"
],
"priority": 1,
"last_updated_time": 1746407477823
}
]
}
}In the above policy example you will see that the warm to hot transition occurs after 2 days and when indices reaches 30 days they are deleted. As this is s a demo environment that is more than sufficient, in a production environment these will need to be adjusted based on compliance requirements. In the policy below you can see a retention of 90 days and it only applies to the indices that relates to wazuh-alerts-*
{
"policy": {
"policy_id": "SIEMonster Data Lifecycle",
"description": "Manages the lifecycle of SIEM data",
"error_notification": null,
"default_state": "hot_state",
"states": [
{
"name": "hot_state",
"actions": [],
"transitions": [
{
"state_name": "warm_state",
"conditions": {
"min_index_age": "2d"
}
}
]
},
{
"name": "warm_state",
"actions": [
{
"timeout": "2h",
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"force_merge": {
"max_num_segments": 1
}
}
],
"transitions": [
{
"state_name": "delete_state",
"conditions": {
"min_index_age": "90d"
}
}
]
},
{
"name": "delete_state",
"actions": [
{
"timeout": "2h",
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"wazuh-alerts-*"
],
"priority": 1
}
]
}
}
Editing
If you wish to edit the policy, you can click the Edit button in the top right corner.
This will prompt you to either use the “Visual editor” or the “JSON Editor”
The visual editor will provide you with the following display
Within this display it is easy to add or remove additional indices to be included in the policy and to add remove states.
Choosing the option to use the “JSON Editor” will provide the following display
In this view you can directly edit the JSON that contains all the policy specifics including the time periods for each state. A quick change can be applied here to the “min_index_age” field that specifies how old an index should be before deletion. If you have a requirement for PCIDSS for example, you would set this to 90 days.
🔖 NOTE: It is recommended to use the SIEMonster supplied policy and only adjusting the time period before deletion. Should you require a custom policy based on other requirements it is suggested to raise a support ticket so that a policy may be generated and supplied to you for your specific needs. A misconfiguration of the policies can lead to unexpected data loss!
Further information on policies are available on the official OpenSearch documentation site at the following URL:
https://docs.opensearch.org/docs/latest/im-plugin/ism/policies/
Indexes
The indexes page shows all indices currently within the system in a table.
The table displays all relevant information on the indices including if they are assigned a policy to process them.
You can note in the above screenshot that t he indices that relate to wazuh-monitoring-* are not part of the policy and they have a “Managed by policy” status of no. If you want them managed by a policy you can either add their indices to an existing policy or create a policy for them based on the information supplied above.
Data streams
Data streams are not currently utilized by the default SIEMonster implementation. For more information about this function please see the official OpenSearch documentation at the following link
https://docs.opensearch.org/docs/latest/im-plugin/data-streams/
Templates
The templates section contains index templates that govern various aspects of the data stored within indices. These includes default fields and their types and also the sharding configuration covered in the preceding section locate here Index Management | Shards-and-Replicas-Background
Please take care NOT to make modification to the index templates supplied by SIEMonster unless instructed to do so by the technical team. This can have far reaching consequences on not only ingestion and search performance but could also affect data validity and retention.
Aliases
The aliases section relates to aliases created to point to multiple indices. This only contains the default aliases created by OpenSearch and is not customized by SIEMonster.
Rollup jobs
This is not currently utilized by SIEMonster. Further information can be found at the following URL
https://docs.opensearch.org/docs/latest/im-plugin/index-rollups/index/
Transform jobs
Transform jobs are used to create summarized views of specific data based on the configuration used. This is not currently in use by SIEMonster. Further information can be found at the following URL.
https://docs.opensearch.org/docs/latest/im-plugin/index-transforms/index/
Notification settings
Notification settings can be enable to alert end users of actions being automatically taken by the system based on policies, templates and other automation built into this section.
To enable any of these items you are required to configure the channel(s) for the event first. Please Click on “Manage channels” at the top right to be redirected to the channels configuration page.
The next step is to Click on “Create Channel” at the top right
The channel will require a name and you can also add an optional description
In the configuration section you need to select the type of channel you wish to use and provide it’s configuration parameters
The channels supported for alerts are:
Slack
Amazon Chime
Custom webhook
E-mail
Amazon SNS
Microsoft Teams
Each type of channel has it’s own configuration options that need to be completed.
Snapshot Management
The snapshot management section deals with the snapshot (backups) configuration of the indices. Snapshot policies govern what is backed up and how often it is backed up. On a default installation you will notice that there are already snapshots present that contain a uuid as identifier. These are snapshots created by AWS as the OpenSearch used is a managed product from AWS.
Tampering with the automated snapshots created by AWS management has the potential to leave your OpenSearch unrecoverable. Please do NOT tamper, modify or remove any of these snapshots.
Snapshot policies
Snapshot policies is where you apply the wanted configuration for taking snapshots. On all deployments after SIEMonster V10 the snapshot policy named daily-policy should be deployed. This will ensure you have daily snapshots for restoring data.
Clicking on the daily-polilcy will provide you with the following information
Editing the policy is not recommended. If you have specific requirements for the policy that is unmet by the default such as more frequent snapshots, a longer snapshot retention period etc, please contact SIEMonster Support to facilitate amending your policy.
Snapshots
This section displays all the current snapshots available in the system, when they were run and if they were successful or not. Do not be concerned if the table takes a while to load if you have a higher number of snapshots as the metadata for the snapshots are read from S3 which has a slight amount of latency due to the type of storage.
Repositories
The repositories section relates to all the snapshot repository/storage configurations. Please do not modify any of the settings from the default. If you have additional needs please contact SIEMonster Support to assist with making the correct modifications.