LogID count alert aggregator

README.md

@@ -1,106 +1,24 @@
-# DetectMateLibrary
-
-Main library to run the different components in DetectMate.
-
-## Main structure
-
-The library contains the next components:
-
-* **Parsers**: parse the logs received from the reader.
-* **Detectors**: return alerts if anomalies are detected.
-* **Alert Aggregation**: aggregate the alerts produced by the detectors.
-* **Schemas**: standard data classes used in DetectMate.
-```
- +--------+     +-----------+    +-------------------+
- | Parser | --> |  Detector | -> | Alert Aggregation |
- +--------+     +-----------+    +-------------------+
-```
-## Developer setup:
-
-### Step 1: Install python dependencies
-Set up the dev environment and install pre-commit hooks:
-
-```bash
-uv sync --dev
-uv run prek install
-```
-
-### Step 2: Install Protobuf dependencies
-
-To install in Linux do:
-
-```bash
-sudo apt install -y protobuf-compiler
-protoc --version
-```
-
-This dependency is only needed if a proto file is modified. To compile the proto file do:
-```bash
-protoc --proto_path=src/detectmatelibrary/schemas/ --python_out=src/detectmatelibrary/schemas/ src/detectmatelibrary/schemas/schemas.proto
-```
 
-### Step 3: Run unit tests
-Run the tests:
-
-```bash
-uv run --dev pytest -q
-```
-
-Run the tests with coverage (add --cov-report=html to generate an HTML report):
-
-```bash
-uv run --dev pytest --cov=. --cov-report=term-missing
-```
-
-## Workspace generator (`mate create`)
-
-DetectMateLibrary includes a small CLI helper to bootstrap standalone workspaces
-for custom parsers and detectors. This is useful if you want to develop and test
-components in isolation while still using the same library and schemas.
-
-### Usage
-
-The CLI entry point is `mate` with a `create` command:
-
-```bash
-mate create --type <parser|detector> --name <workspace_name> --dir <target_dir>
-```
+# DetectMateLibrary
 
-| Option   | Description                                                                                                                                                                   |
-|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `--type` | Component type to generate:<br>- `parser`: CoreParser-based template<br>- `detector`: CoreDetector-based template                                                             |
-| `--name` | Name of the component and package:<br>- Creates package dir: `<target_dir>/<name>/`<br>- Creates main file: `<name>.py`<br>- Derives class names: `<Name>` and `<Name>Config` |
-| `--dir`  | Directory where the workspace will be created                                                                                                                                 |
+![UML](docs/img/logo.png)
 
+DetectMate is a flexible, component-based library for parsing and log anomaly detection.
 
-### What gets generated
+## Installation
 
-For example:
+Download us from `Pip`!!
 
 ```bash
-mate create --type parser --name custom_parser --dir ./workspaces/custom_parser
-```
-
-will create:
-
-```text
-workspaces/custom_parser/          # workspace root
-├── custom_parser/                 # Python package
-│   ├── __init__.py
-│   └── custom_parser.py           # CoreParser-based template
-├── tests/
-│   └── test_custom_parser.py      # generated from template to test custom_parser
-├── data.json                      # example data to run the code
-├── LICENSE.md                     # copied from main project
-├── .gitignore                     # copied from main project
-├── .pre-commit-config.yaml        # copied from main project
-├── pyproject.toml                 # minimal project + dev extras
-└── README.md                      # setup instructions
+pip install detectmatelibrary
 ```
 
 ## Documentation
 
-- [Project Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/)
+Main documentations:
+
+- [Library Documentation](https://ait-detectmate.github.io/DetectMateLibrary/latest/) (this repo)
+- [Service Documentation](https://github.com/ait-detectmate/DetectMateService/tree/main)
 
 ## Contribution
 

docs/alert_aggregator.md

@@ -19,6 +19,7 @@ This document explains the expected API and how to implement an alert aggregator
   With the default `buffer_size` of 1, it is called for every alert with a list of one.
 - Use a typed `Config` class (subclass of `CoreAlertAggregatorConfig`) to hold runtime parameters.
 
+
 ## CoreAlertAggregator: minimal API
 
 ```python
@@ -62,6 +63,35 @@ fields left for `aggregate_alerts()` to fill are `description` and `alertDetails
     unless your `aggregate_alerts()` keeps it, including their `detectedTimestamp` and
     `receivedTimestamp`.
 
-## Available alert aggregators
+## Alert aggregator methods
+
+The alert aggregator are numbered from simplest to most complex, and the sidebar lists them in the same order.
+
+| # | Detector | What it detects |
+|---|---|---|
+| 00 | [Basic Concat Aggregator](alert_aggregators/basic_concatenation.md) | Concat a set of alerts. |
+| 01 | [LogID Count Aggregator](alert_aggregators/logid_count.md) | Count the amount of times a logID is in alerts. |
+
+
+### Common parameters (all alert aggregators)
+
+There are some parameters, that **every** alert aggregator inherits from `CoreAlertAggregatorConfig`/`CoreConfigCoreAlertAggregator`/`BasicConfig`, regardless of what it does. The other parameters, that are **specific** for the respective detector, are explained right at the detectors documentation page, later on.
+
+<!-- Start common_arguments -->
+???+ note "Top level"
+
+    | Field | Type | Default | Description |
+    |---|---|---|---|
+    | `auto_config` | boolean | False | Runs the configuration step before the training process. |
+
+???+ note "params"
 
-- [Basic Concat](alert_aggregators/basic_concatenation.md)
+    | Field | Type | Default | Description |
+    |---|---|---|---|
+    | `start_id` | integer | 10 | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | Data used for training, if None, training is not done. |
+    | `data_use_configure` | integer, null | None | Data used for configuration, if None, configuration is not done. |
+    | `use_config_data_as_training` | boolean | True | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End common_arguments -->

docs/alert_aggregators/basic_concatenation.md

@@ -2,25 +2,59 @@
 
 The basic concat alert aggregation approach aggregates x number of alerts into one output schema.
 
+## In/out
+
 |            | Schema                     | Description        |
 |------------|----------------------------|--------------------|
 | **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
 | **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
 
+## At a glance
 
-# Configuration example
+| Learns from training data | Auto-configuration | Needs `events` | Federation |
+|---|---|---|---|
+| ❌ | ❌ | ❌ | ❌ |
 
-```yaml
-alert_aggregators:
-    BasicConcatAggregator:
-        method_type: "basic_concat_aggregator"
-        buffer_size: 3
-        deduplicate_values: False
-        auto_config: False
-```
+## Description
+
+A basic approach that simply concatenates the alerts without verifying anything specific.
 
-## Example usage
+
+## Example
 
 ```python
 --8<-- "docs/examples/alert_aggregators/basic_concat.py:example_1"
 ```
+
+## Configuration file
+
+The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)).
+
+```yaml
+--8<-- "docs/examples/alert_aggregators/basic_concat.yaml"
+```
+
+## Configuration arguments
+
+All parameters this detector accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this detector or `shared` with other detectors (see the [Alert Aggregation overview](../alert_aggregator.md#common-parameters-all-alert-aggregators)). Where this detector changes a shared default, the shared value is shown in brackets.
+
+<!-- Start arguments -->
+??? note "Top level"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. |
+    | `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
+
+???+ note "params"
+
+    | Field | Type | Default | Scope | Description |
+    |---|---|---|---|---|
+    | `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. |
+    | `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
+    | `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
+    | `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
+    | `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
+    | `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+    | `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End arguments -->

docs/alert_aggregators/logid_count.md

@@ -0,0 +1,60 @@
+# LogID Count Alert Aggregation
+
+The logID count alert aggregation tracks how many times each log ID appears in the alerts, and if it exceeds a specified threshold, it triggers the aggregation.
+
+## In/out
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [DetectorSchema](../schemas.md)    | Alerts from detectors  |
+| **Output** | [AggregateSchema](../schemas.md) | Aggregated alerts    |
+
+## At a glance
+
+| Learns from training data | Auto-configuration | Needs `events` | Federation |
+|---|---|---|---|
+| ❌ | ❌ | ❌ | ❌ |
+
+## Description
+
+A log ID counter stores, in a fixed-size table, the counts for different log IDs. If the count for any log ID from the Detector Schemas exceeds a defined threshold, it returns an alert aggregation.
+
+The fixed-size table operates as a FIFO: once it is full, it removes the oldest entry and inserts the new log ID.
+## Example
+
+```python
+--8<-- "docs/examples/alert_aggregators/logid_count.py:example_1"
+```
+
+## Configuration file
+
+The configuration used by the example above. It sets only what this use case needs; every other parameter keeps its default (see [Configuration arguments](#configuration-arguments)).
+
+```yaml
+--8<-- "docs/examples/alert_aggregators/logid_count.yaml"
+```
+
+## Configuration arguments
+
+All parameters this detector accepts, grouped by the YAML block they go in. **Scope** tells whether a parameter is `specific` to this detector or `shared` with other detectors (see the [Alert Aggregation overview](../alert_aggregator.md#common-parameters-all-alert-aggregators)). Where this detector changes a shared default, the shared value is shown in brackets.
+
+<!-- Start arguments -->
+??? note "Top level"
+
+| Field | Type | Default | Scope | Description |
+|---|---|---|---|---|
+| `method_type` | string | basic_concat_aggregator | shared | Indicates what type of method it is. |
+| `auto_config` | boolean | False | shared | Runs the configuration step before the training process. |
+
+???+ note "params"
+
+| Field | Type | Default | Scope | Description |
+|---|---|---|---|---|
+| `buffer_size` | integer | 3 | specific | Buffer size use in the aggregation. |
+| `start_id` | integer | 10 | shared | Number used to start the unique ID generator. |
+| `data_use_training` | integer, null | None | shared | Data used for training, if None, training is not done. |
+| `data_use_configure` | integer, null | None | shared | Data used for configuration, if None, configuration is not done. |
+| `use_config_data_as_training` | boolean | True | shared | Combine the configured data in the training process if True. |
+| `train_buffer_max_records` | integer | 100000 | shared | Configure records kept in memory for training (use_config_data_as_training) before the buffer spills to Parquet files on disk, in parts of this many records. |
+| `train_buffer_dir` | string, null | None | shared | Local directory for the spilled training buffer. None uses the system temp directory (TMPDIR). Each spill goes to a private detectmate-train-* directory, removed after training reads it; a killed process leaves it behind. |
+<!-- End arguments -->

docs/examples/alert_aggregators/basic_concat.py

@@ -4,18 +4,11 @@
 
 import detectmatelibrary.schemas as schemas
 
+import yaml
 
-aggregations_config = {
-    "alert_aggregators": {
-        "BasicConcatAggregator": {
-            "method_type": "basic_concat_aggregator",
-            "buffer_size": 3,
-            "deduplicate_values": False,
-            "auto_config": False,
-        }
-    }
-}
 
+with open("docs/examples/alert_aggregators/basic_concat.yaml") as f:
+    config = yaml.safe_load(f)
 
 input_ = schemas.DetectorSchema({
     "detectorID": "1",
@@ -30,9 +23,7 @@
     "alertDetails": {"99 problems": "but logs aint one"}
 })
 
-alert_aggregator = BasicConcatAggregation(
-    "BasicConcatAggregator", aggregations_config
-)
+alert_aggregator = BasicConcatAggregation("BasicConcatAggregator", config)
 alert_aggregator.process(input_)
 
 # --8<-- [end:example_1]

docs/examples/alert_aggregators/basic_concat.yaml

@@ -0,0 +1,5 @@
+alert_aggregators:
+    BasicConcatAggregator:
+        method_type: "basic_concat_aggregator"
+        buffer_size: 3
+        auto_config: False

docs/examples/alert_aggregators/logid_count.py

@@ -0,0 +1,35 @@
+# --8<-- [start:example_1]
+from detectmatelibrary.alert_aggregation.logid_count import LogIDCountAggegation
+
+import detectmatelibrary.schemas as schemas
+
+import yaml
+
+
+with open("docs/examples/alert_aggregators/logid_count.yaml") as f:
+    config = yaml.safe_load(f)
+
+
+def generate_schema(log_id: str | list[str]) -> schemas.DetectorSchema:
+    log_id = log_id if isinstance(log_id, list) else [log_id]
+    return schemas.DetectorSchema({
+        "detectorID": "1",
+        "detectorType": "dummy",
+        "alertID": "2",
+        "detectedTimestamp": 0,
+        "logIDs": log_id,
+        "score": 0.2,
+        "logTimestamps": [-1],
+        "description": "hello there",
+        "receivedTimestamp": 1,
+        "alertDetails": {"99 problems": "but logs aint one"}
+    })
+
+
+alert_aggregator = LogIDCountAggegation("LogIDCountAggregator", config=config)
+
+print(alert_aggregator.process(generate_schema("a")) is None)  # True
+print(alert_aggregator.process(generate_schema("b")) is None)  # True
+print(alert_aggregator.process(generate_schema("a")) is None)  # False
+
+# --8<-- [end:example_1]

docs/examples/alert_aggregators/logid_count.yaml

@@ -0,0 +1,5 @@
+alert_aggregators:
+    LogIDCountAggregator:
+        method_type: "logid_count_aggregator"
+        max_count: 2
+        max_memory: 5000