From 936bca47e23bc4bef49d9d8ba626eb7e4cfe3949 Mon Sep 17 00:00:00 2001 From: yoon nguyen Date: Wed, 9 Jul 2025 15:34:50 -0400 Subject: [PATCH 1/5] add agent pre-ingestion tag list --- .../en/agent/logs/advanced_log_collection.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/content/en/agent/logs/advanced_log_collection.md b/content/en/agent/logs/advanced_log_collection.md index bb853850f4419..1b887b68270fc 100644 --- a/content/en/agent/logs/advanced_log_collection.md +++ b/content/en/agent/logs/advanced_log_collection.md @@ -636,6 +636,59 @@ All the logs collected by the Datadog Agent are impacted by the global processin **Note**: The Datadog Agent does not start the log collector if there is a format issue in the global processing rules. Run the Agent's [status subcommand][6] to troubleshoot any issues. +## Agent-Added Tags + +The Datadog Agent automatically adds certain tags to all logs it collects before sending them to Datadog. These tags are added **pre-ingestion**, and are included in the payload that gets delivered to Datadog. + +### Automatically Added Tags + +The following tags are automatically added to logs by the Datadog Agent: + +| Tag | Description | +|-----|-------------| +| `source` | The source of the log (file path, integration name, etc.) +| `service` | The service name if configured in the log collection +| `env` | The environment tag if configured globally +| `version` | The Agent version +| `filename` | Base name of the tailed file +| `dirname` | Directory containing the tailed file +| `source_host` | IP address of the socket source host +| `event_type` | Type of the Windows event +| `event_source` | Source of the Windows event +| `event_id` | Windows Event ID if configured +| `sid` | Windows Security identifier if configured +| `truncated` | Source of truncation if configured +| `multiline` | Source of multiline aggregation if configured +| `aggregated_json` | Indicates that the log was aggregated from multiple JSON log entries + +**Note**: Some tags are only added for specific log sources and some tags can be optionally enabled/disabled via configurations. + +### Pre-ingestion Processing + +Since these tags are added pre-ingestion, they: + +* Are included in all log data delivered to Datadog +* Increase the overall size of your log data +* Are available for filtering, searching, and aggregation in the Log Explorer +* Can be used in log-based metrics and queries + +### Best Practices + +1. **Monitor tag cardinality**: High cardinality tags (tags with many unique values) can impact data volume +2. **Use tag filtering**: Exclude unnecessary tags to optimize data collection +3. **Standardize tag values**: Use consistent tag values across your infrastructure +4. **Review tag usage**: Regularly review which tags are being added and their impact on data volume + +### Further Configuration + +For more advanced tag management, see: +- [Tagging Best Practices][9] +- [Agent Configuration Files][5] +- [Container Tagging][10] + +[9]: /getting_started/tagging/ +[10]: /agent/docker/tag/ + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} From f0957034725c4276494f0966992e53bd6e9d54e6 Mon Sep 17 00:00:00 2001 From: yoon nguyen Date: Mon, 21 Jul 2025 14:12:19 -0400 Subject: [PATCH 2/5] add conditions for tags --- .../en/agent/logs/advanced_log_collection.md | 41 ++++++++----------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/content/en/agent/logs/advanced_log_collection.md b/content/en/agent/logs/advanced_log_collection.md index 1b887b68270fc..50a12584e6f25 100644 --- a/content/en/agent/logs/advanced_log_collection.md +++ b/content/en/agent/logs/advanced_log_collection.md @@ -644,24 +644,22 @@ The Datadog Agent automatically adds certain tags to all logs it collects before The following tags are automatically added to logs by the Datadog Agent: -| Tag | Description | -|-----|-------------| -| `source` | The source of the log (file path, integration name, etc.) -| `service` | The service name if configured in the log collection -| `env` | The environment tag if configured globally -| `version` | The Agent version -| `filename` | Base name of the tailed file -| `dirname` | Directory containing the tailed file -| `source_host` | IP address of the socket source host -| `event_type` | Type of the Windows event -| `event_source` | Source of the Windows event -| `event_id` | Windows Event ID if configured -| `sid` | Windows Security identifier if configured -| `truncated` | Source of truncation if configured -| `multiline` | Source of multiline aggregation if configured -| `aggregated_json` | Indicates that the log was aggregated from multiple JSON log entries - -**Note**: Some tags are only added for specific log sources and some tags can be optionally enabled/disabled via configurations. +| Tag | Description | Conditions | +|-----|-------------|-------------| +| `source` | The source of the log (file path, integration name, etc.) | Always when available | +| `service` | The service name if configured in the log collection | Always when available | +| `env` | The environment tag if configured globally | Always when available | +| `version` | The Agent version | Always when available | +| `filename` | Base name of the tailed file | File-based sources only | +| `dirname` | Directory containing the tailed file | File-based sources only | +| `source_host` | IP address of the socket source host | Socket sources (TCP/UDP) only | +| `event_type` | Type of the Windows event | Windows events only | +| `event_source` | Source of the Windows event | Windows events only | +| `event_id` | Windows Event ID | Windows events only, if `tag_event_id: true` | +| `sid` | Windows Security identifier | Windows events only, if `tag_sid: true` | +| `truncated` | Source of truncation | If `logs_config.tag_truncated_logs: true` | +| `multiline` | Source of multiline aggregation | If `logs_config.tag_multi_line_logs: true` | +| `aggregated_json` | Indicates that the log was aggregated from multiple JSON log entries | If `logs_config.auto_multi_line.tag_aggregated_json: true` | ### Pre-ingestion Processing @@ -672,13 +670,6 @@ Since these tags are added pre-ingestion, they: * Are available for filtering, searching, and aggregation in the Log Explorer * Can be used in log-based metrics and queries -### Best Practices - -1. **Monitor tag cardinality**: High cardinality tags (tags with many unique values) can impact data volume -2. **Use tag filtering**: Exclude unnecessary tags to optimize data collection -3. **Standardize tag values**: Use consistent tag values across your infrastructure -4. **Review tag usage**: Regularly review which tags are being added and their impact on data volume - ### Further Configuration For more advanced tag management, see: From b9f027c5bea7c958edf770fa2124ed42d95adbb3 Mon Sep 17 00:00:00 2001 From: Esther Kim Date: Mon, 28 Jul 2025 14:48:39 -0400 Subject: [PATCH 3/5] Add a new page for Agent Tags (#30747) * Add a new page for Agent Tags * Apply suggestions from code review * Add to Agent logs further reading --- config/_default/menus/main.en.yaml | 13 +++-- content/en/agent/logs/_index.md | 3 ++ .../en/agent/logs/advanced_log_collection.md | 44 ---------------- content/en/agent/logs/agent_tags.md | 52 +++++++++++++++++++ 4 files changed, 64 insertions(+), 48 deletions(-) create mode 100644 content/en/agent/logs/agent_tags.md diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 5dcd8936b9ce6..4d9eabdbb67cc 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -424,26 +424,31 @@ menu: identifier: agent_logs parent: agent weight: 5 + - name: Agent tags + url: agent/logs/agent_tags/ + parent: agent + identifier: agent_tags + weight: 501 - name: Advanced Configurations identifier: agent_logs_advanced_log_collection url: agent/logs/advanced_log_collection parent: agent_logs - weight: 501 + weight: 502 - name: Proxy identifier: agent_logs_proxy url: agent/logs/proxy parent: agent_logs - weight: 502 + weight: 503 - name: Transport identifier: agent_logs_transport url: agent/logs/log_transport parent: agent_logs - weight: 503 + weight: 504 - name: Multi-Line Detection identifier: multi_line_detection url: agent/logs/auto_multiline_detection parent: agent_logs - weight: 505 + weight: 506 - name: Configuration url: agent/configuration parent: agent diff --git a/content/en/agent/logs/_index.md b/content/en/agent/logs/_index.md index 4be6aa3e2d9ca..aec8edf389f55 100644 --- a/content/en/agent/logs/_index.md +++ b/content/en/agent/logs/_index.md @@ -2,6 +2,9 @@ title: Host Agent Log collection description: Use the Datadog Agent to collect your logs and send them to Datadog further_reading: +- link: "agent/logs/agent_tags/" + tag: "Documentation" + text: "Agent tags automatically added to logs" - link: "agent/logs/advanced_log_collection/#filter-logs" tag: "Documentation" text: "Filter logs sent to Datadog" diff --git a/content/en/agent/logs/advanced_log_collection.md b/content/en/agent/logs/advanced_log_collection.md index 57565276259f8..6a6828b5ade3a 100644 --- a/content/en/agent/logs/advanced_log_collection.md +++ b/content/en/agent/logs/advanced_log_collection.md @@ -651,50 +651,6 @@ All JSON-formatted log lines are treated as a single line of logs, and only the **3. What happens when there are both global rules and integration-specific rules?** Integration-specific rules completely override global rules for the particular integration. -## Agent-Added Tags - -The Datadog Agent automatically adds certain tags to all logs it collects before sending them to Datadog. These tags are added **pre-ingestion**, and are included in the payload that gets delivered to Datadog. - -### Automatically Added Tags - -The following tags are automatically added to logs by the Datadog Agent: - -| Tag | Description | Conditions | -|-----|-------------|-------------| -| `source` | The source of the log (file path, integration name, etc.) | Always when available | -| `service` | The service name if configured in the log collection | Always when available | -| `env` | The environment tag if configured globally | Always when available | -| `version` | The Agent version | Always when available | -| `filename` | Base name of the tailed file | File-based sources only | -| `dirname` | Directory containing the tailed file | File-based sources only | -| `source_host` | IP address of the socket source host | Socket sources (TCP/UDP) only | -| `event_type` | Type of the Windows event | Windows events only | -| `event_source` | Source of the Windows event | Windows events only | -| `event_id` | Windows Event ID | Windows events only, if `tag_event_id: true` | -| `sid` | Windows Security identifier | Windows events only, if `tag_sid: true` | -| `truncated` | Source of truncation | If `logs_config.tag_truncated_logs: true` | -| `multiline` | Source of multiline aggregation | If `logs_config.tag_multi_line_logs: true` | -| `aggregated_json` | Indicates that the log was aggregated from multiple JSON log entries | If `logs_config.auto_multi_line.tag_aggregated_json: true` | - -### Pre-ingestion Processing - -Since these tags are added pre-ingestion, they: - -* Are included in all log data delivered to Datadog -* Increase the overall size of your log data -* Are available for filtering, searching, and aggregation in the Log Explorer -* Can be used in log-based metrics and queries - -### Further Configuration - -For more advanced tag management, see: -- [Tagging Best Practices][9] -- [Agent Configuration Files][5] -- [Container Tagging][10] - -[9]: /getting_started/tagging/ -[10]: /agent/docker/tag/ - ## Further Reading {{< partial name="whats-next/whats-next.html" >}} diff --git a/content/en/agent/logs/agent_tags.md b/content/en/agent/logs/agent_tags.md new file mode 100644 index 0000000000000..cf0cee41abcec --- /dev/null +++ b/content/en/agent/logs/agent_tags.md @@ -0,0 +1,52 @@ +--- +title: Agent tags +further_reading: +- link: "/getting_started/tagging/" + tag: "Documentation" + text: "Tagging Best Practices" +- link: "/agent/configuration/agent-configuration-files/" + tag: "Documentation" + text: "Agent Configuration Files" +- link: "/agent/docker/tag/" + tag: "Documentation" + text: "Container Tagging" +--- + +## Overview + +The Datadog Agent automatically adds certain tags to all logs it collects before sending them to Datadog. These tags are added **pre-ingestion**, and are included in the payload that gets delivered to Datadog. + +## Pre-ingestion processing + +Since these tags are added pre-ingestion, they: + +* Are included in all log data delivered to Datadog +* Increase the overall size of your log data +* Are available for filtering, searching, and aggregation in the Log Explorer +* Can be used in log-based metrics and queries + +## Agent tags automatically added to logs + +The following tags are automatically added to logs by the Datadog Agent: + +| Tag | Description | Conditions | +|-----|-------------|-------------| +| `source` | The source of the log (file path, integration name, etc.) | Always when available | +| `service` | The service name if configured in the log collection | Always when available | +| `env` | The environment tag if configured globally | Always when available | +| `version` | The Agent version | Always when available | +| `filename` | Base name of the tailed file | File-based sources only | +| `dirname` | Directory containing the tailed file | File-based sources only | +| `source_host` | IP address of the socket source host | Socket sources (TCP/UDP) only | +| `event_type` | Type of the Windows event | Windows events only | +| `event_source` | Source of the Windows event | Windows events only | +| `event_id` | Windows Event ID | Windows events only, if `tag_event_id: true` | +| `sid` | Windows Security identifier | Windows events only, if `tag_sid: true` | +| `truncated` | Source of truncation | If `logs_config.tag_truncated_logs: true` | +| `multiline` | Source of multi-line aggregation | If `logs_config.tag_multi_line_logs: true` | +| `aggregated_json` | Indicates that the log was aggregated from multiple JSON log entries | If `logs_config.auto_multi_line.tag_aggregated_json: true` | + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + From a6032bcc5917dbbda114251e922a9042f7f3e4e0 Mon Sep 17 00:00:00 2001 From: Esther Kim Date: Mon, 28 Jul 2025 15:23:40 -0400 Subject: [PATCH 4/5] Update config/_default/menus/main.en.yaml --- config/_default/menus/main.en.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 4d9eabdbb67cc..a5cc537ec350f 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -424,7 +424,7 @@ menu: identifier: agent_logs parent: agent weight: 5 - - name: Agent tags + - name: Log Agent tags url: agent/logs/agent_tags/ parent: agent identifier: agent_tags From 948f9c7a8fef2688e1f31f27ebf89008ed9dcfc2 Mon Sep 17 00:00:00 2001 From: Esther Kim Date: Mon, 28 Jul 2025 15:24:36 -0400 Subject: [PATCH 5/5] Update content/en/agent/logs/agent_tags.md --- content/en/agent/logs/agent_tags.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/agent/logs/agent_tags.md b/content/en/agent/logs/agent_tags.md index cf0cee41abcec..aebe215ee830d 100644 --- a/content/en/agent/logs/agent_tags.md +++ b/content/en/agent/logs/agent_tags.md @@ -1,5 +1,5 @@ --- -title: Agent tags +title: Log Agent tags further_reading: - link: "/getting_started/tagging/" tag: "Documentation"