Compare commits

...
Author SHA1 Message Date
shown aae421b51c Merge branch 'master' into fix/005-greptime-compose-port-binding 2026-07-29 22:40:57 +08:00
shown 59c003aaef [fix] prevent masked notice secret replay to attacker URLs (#4262)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-29 20:54:26 +08:00
liuhyandClaude b97472cb83 [fix] bind greptime compose data-store ports to localhost
The hertzbeat-postgresql-greptimedb docker-compose exposes the Postgres
and GreptimeDB data-store ports on all interfaces (0.0.0.0). Both
services run with default credentials shipped in the compose file
(POSTGRES_PASSWORD=123456, the greptime static user provider greptime=greptime),
so binding them to 0.0.0.0 lets any reachable host authenticate to the
underlying data stores with those defaults.

These ports exist only for the operator to inspect the data stores from
the host; the hertzbeat container reaches them over the internal compose
network and does not need the host mappings. Bind the host-side mappings
to 127.0.0.1 so the default-credential services are not remotely
accessible.

The hertzbeat web/collector ports (1157/1158) are intentionally left
unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 07:00:40 -07:00
Duansg b1b242f9bf [chore] upgrade hertzbeat version (#4255) 2026-07-28 09:40:56 +08:00
shown cdcb5f74b7 fix(ai): improve SOP schedule error handling (#4254)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-27 22:38:00 +08:00
Duansg 0f6b995be2 [fix] skip SQL Server integration test on arm64 (#4253) 2026-07-27 05:20:30 -07:00
shown a4b486d992 fix(ai): improve alert rule validation and parsing (#4251)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-26 23:07:48 +08:00
eb7ca9728d improve(dashboard): hide empty category cards, add empty state (#4242)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
2026-07-26 18:32:23 +08:00
NekoPunch 0ae41860e3 test: fix two flaky tests (#4250) 2026-07-26 18:21:10 +08:00
NekoPunchandTomsun28 4a8f78b6f9 fix(alerter): keep webhook query params intact (#4249)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-26 17:36:47 +08:00
NekoPunchandaias00 70ddfda1de [docs] refactor blog list UI to card grid (#4219)
Co-authored-by: aias00 <liuhongyu@apache.org>
2026-07-26 15:51:32 +08:00
79c48a4bfc fix(ai): remove SOP schedules when deleting conversations (#4247)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
Signed-off-by: shown <yuluo08290126@gmail.com>
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-26 14:38:41 +08:00
Tomsun28andDevin 9ec371a5c5 ci(doc-deploy): fix pnpm cache setup and upgrade Node to 22 (#4248)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-26 14:22:28 +08:00
Tomsun28 c805d633b6 docs: add new contributors and update notice year (#4245) 2026-07-25 20:22:02 +08:00
NekoPunchandTomsun28 c12372d82d fix(web-app): replace removed stylelint --syntax flag (#4241)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-25 14:28:14 +08:00
NekoPunchandTomsun28 63cc017bba fix(web-app): point threshold user guide link to correct docs (#4239)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-25 14:19:44 +08:00
NekoPunchandTomsun28 9531a2fcd3 fix(web-app): add missing Korean observability menu translations (#4240)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-25 14:13:10 +08:00
Yang Chen 717521369c fix: update GreptimeDB Docker Compose to v1.1.3 (#4244) 2026-07-25 14:08:51 +08:00
Yang Chen c8f2da472e fix: add SSE authentication and correct GreptimeDB SQL (#4243) 2026-07-25 13:09:13 +08:00
Logic 6152c07320 refactor: evolve common-core payload compatibility (#4237) 2026-07-24 15:07:35 +08:00
Duansg b832d7ace4 [fix] fix horizontal scroll not working in periodic threshold preview table (#4238) 2026-07-24 10:39:36 +08:00
DuansgandTomsun28 3ab68b5814 [improve] degrade observability consoles gracefully when storage lacks log/trace/metric support (#4233)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-24 00:54:05 +08:00
shown 1d9edb244c feat: use spring ai MethodToolCallbackProvider replace custom reflect… (#4234)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-23 20:41:28 +08:00
Duansg 1429115282 [improve] pre-create hertzbeat_logs table during greptime signal initialization (#4231) 2026-07-23 11:34:35 +08:00
4b217e7097 feat(status-page): add configurable time range for component history (#4222)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-23 00:23:37 +08:00
shown 032a4852f7 fix(ai): propagate security context to tool callbacks (#4230)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-22 23:46:33 +08:00
Duansg 6b78b784a2 [improve] remove unused log management translations (#4229) 2026-07-22 14:33:17 +08:00
Duansg 11cd5bb0c7 [improve] reject masked receiver edit when the stored receiver does not exist (#4225) 2026-07-22 01:32:24 +08:00
hengyussandTomsun28 a9eb231989 fix:prometheus timeout monitor key mismatch (#4215)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-21 23:59:29 +08:00
shownandTomsun28 9b31baec8a fix(ai): improve SOP execution and provider config handling (#4224)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-21 23:34:06 +08:00
DuansgandTomsun28 0a38a73756 [improve] form-urlencode SQL request body in GreptimeSqlQueryExecutor (#4223)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-21 23:19:43 +08:00
Duansg c9e3338c9f [improve] mask secret fields of notice receiver in rest api response (#4220) 2026-07-21 23:11:30 +08:00
shown 107f7d02ff fix(log): send initial SSE event and periodic heartbeat (#4221)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
2026-07-20 22:42:25 +08:00
shown e3acbbd6db chore(ci): Add workflow_dispatch trigger to nightly build (#4213) 2026-07-18 01:45:13 +08:00
31bc7d8a51 [feature] Add Korean i18n (#4197)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2026-07-17 21:51:27 +08:00
Duansg 79427099a2 [improve] bump grpc-java from 1.56.1 to 1.76.3 (#4211) 2026-07-17 10:40:17 +08:00
Duansg 0edc363bcc fix: remove duplicated ASF license headers left by license-eye auto-fix (#4210) 2026-07-16 22:58:42 +08:00
47538dbc1a feat(observability): add Angular three-signal transition (#4195)
Signed-off-by: Duansg <siguoduan@gmail.com>
Signed-off-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
2026-07-16 20:39:17 +08:00
356 changed files with 15793 additions and 3138 deletions
+204
View File
@@ -2713,6 +2713,210 @@
"contributions": [
"code"
]
},
{
"login": "orangeCatDeveloper",
"name": "NekoPunch",
"avatar_url": "https://avatars.githubusercontent.com/u/95899648?v=4",
"profile": "https://github.com/orangeCatDeveloper",
"contributions": [
"code",
"doc"
]
},
{
"login": "wy471x",
"name": "wy471x",
"avatar_url": "https://avatars.githubusercontent.com/u/52033069?v=4",
"profile": "https://wy471x.github.io/",
"contributions": [
"code"
]
},
{
"login": "hengyuss",
"name": "hengyuss",
"avatar_url": "https://avatars.githubusercontent.com/u/81064732?v=4",
"profile": "https://github.com/hengyuss",
"contributions": [
"code"
]
},
{
"login": "moduvoice",
"name": "moduvoice",
"avatar_url": "https://avatars.githubusercontent.com/u/291867022?v=4",
"profile": "https://moduvoice.com/",
"contributions": [
"doc"
]
},
{
"login": "hutiefang76",
"name": "hutiefang76",
"avatar_url": "https://avatars.githubusercontent.com/u/137664623?v=4",
"profile": "https://github.com/hutiefang76",
"contributions": [
"code"
]
},
{
"login": "ZhouYinLong-lab",
"name": "柳含知 Liu Hanzhi",
"avatar_url": "https://avatars.githubusercontent.com/u/250777154?v=4",
"profile": "https://zylatent.com/",
"contributions": [
"doc"
]
},
{
"login": "wilmerdooley",
"name": "wilmerdooley",
"avatar_url": "https://avatars.githubusercontent.com/u/259930736?v=4",
"profile": "https://github.com/wilmerdooley",
"contributions": [
"code"
]
},
{
"login": "Zmjjeff7",
"name": "Zhenhong Guo",
"avatar_url": "https://avatars.githubusercontent.com/u/175370943?v=4",
"profile": "https://github.com/Zmjjeff7",
"contributions": [
"code"
]
},
{
"login": "abhyudayareddy",
"name": "abhyudayareddy",
"avatar_url": "https://avatars.githubusercontent.com/u/54602866?v=4",
"profile": "https://github.com/abhyudayareddy",
"contributions": [
"code"
]
},
{
"login": "neon-hippo",
"name": "neon-hippo",
"avatar_url": "https://avatars.githubusercontent.com/u/165560498?v=4",
"profile": "https://github.com/neon-hippo",
"contributions": [
"code"
]
},
{
"login": "P-Peaceful",
"name": "P_Peaceful",
"avatar_url": "https://avatars.githubusercontent.com/u/52856161?v=4",
"profile": "https://github.com/P-Peaceful",
"contributions": [
"code",
"doc"
]
},
{
"login": "zhusaidong",
"name": "zhusaidong",
"avatar_url": "https://avatars.githubusercontent.com/u/3039961?v=4",
"profile": "https://github.com/zhusaidong",
"contributions": [
"code"
]
},
{
"login": "zhehenlu",
"name": "zhlu",
"avatar_url": "https://avatars.githubusercontent.com/u/31504542?v=4",
"profile": "https://github.com/zhehenlu",
"contributions": [
"code",
"doc"
]
},
{
"login": "brettgervasoni",
"name": "brettgervasoni",
"avatar_url": "https://avatars.githubusercontent.com/u/34056000?v=4",
"profile": "https://github.com/brettgervasoni",
"contributions": [
"code"
]
},
{
"login": "Darshan-paul",
"name": "Darshan-paul",
"avatar_url": "https://avatars.githubusercontent.com/u/211450705?v=4",
"profile": "https://github.com/Darshan-paul",
"contributions": [
"code"
]
},
{
"login": "04cb",
"name": "layla",
"avatar_url": "https://avatars.githubusercontent.com/u/111667698?v=4",
"profile": "https://github.com/04cb",
"contributions": [
"doc"
]
},
{
"login": "miantalha45",
"name": "Talha Amjad",
"avatar_url": "https://avatars.githubusercontent.com/u/155809113?v=4",
"profile": "https://github.com/miantalha45",
"contributions": [
"code"
]
},
{
"login": "turanalmammadov",
"name": "Turan Almammadov",
"avatar_url": "https://avatars.githubusercontent.com/u/16321061?v=4",
"profile": "https://turanalmammadov.com/",
"contributions": [
"code",
"doc"
]
},
{
"login": "zhaoyangplus",
"name": "zhaoyangplus",
"avatar_url": "https://avatars.githubusercontent.com/u/245090302?v=4",
"profile": "https://github.com/zhaoyangplus",
"contributions": [
"code"
]
},
{
"login": "yexuanyang",
"name": "Yang Yexuan",
"avatar_url": "https://avatars.githubusercontent.com/u/73885401?v=4",
"profile": "https://github.com/yexuanyang",
"contributions": [
"code"
]
},
{
"login": "markguo123",
"name": "markguo123",
"avatar_url": "https://avatars.githubusercontent.com/u/155072651?v=4",
"profile": "https://github.com/markguo123",
"contributions": [
"doc"
]
},
{
"login": "leo-934",
"name": "leo",
"avatar_url": "https://avatars.githubusercontent.com/u/55838224?v=4",
"profile": "https://github.com/leo-934",
"contributions": [
"code",
"doc",
"blog"
]
}
],
"contributorsPerLine": 7,
Binary file not shown.

After

Width:  |  Height:  |  Size: 194 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 315 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 282 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 262 KiB

+1 -1
View File
@@ -43,7 +43,7 @@ jobs:
version: 10
- uses: actions/setup-node@v4
with:
node-version: '20'
node-version: '22'
cache: pnpm
cache-dependency-path: home/pnpm-lock.yaml
- uses: actions/setup-python@v4
+7 -7
View File
@@ -47,20 +47,20 @@ jobs:
with:
fetch-depth: 0
# Setup pnpm (must run before setup-node so the pnpm cache can be configured)
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 10
# Setup Node.js environment
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
node-version: '22'
cache: pnpm
cache-dependency-path: home/pnpm-lock.yaml
# Setup pnpm
- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: latest
# Install dependencies in home directory
- name: Install Dependencies
working-directory: home
+1
View File
@@ -23,6 +23,7 @@ on:
- cron: '0 0 * * *'
push:
branches: [ action* ]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
+4
View File
@@ -56,3 +56,7 @@ application-mysql.yml
application-pg.yml
/package.json
/yarn.lock
# Local Codex workspace files
.codex/
AGENTS.md
+1 -1
View File
@@ -1,5 +1,5 @@
Apache HertzBeat™
Copyright 2024-2025 The Apache Software Foundation
Copyright 2024-2026 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
+28 -46
View File
@@ -49,52 +49,6 @@
![HertzBeat](home/static/img/docs/hertzbeat-arch.png)
## ⛄ Supported
> We define all metrics collection types such as `mysql`, `jvm`, and `k8s` as `YML` templates, allowing users to import them to support corresponding types of monitoring.
> Welcome everyone to contribute your customized general metrics type YML template during use.
- [Website](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-website.yml), [Port Telnet](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-port.yml),
[Http Api](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api.yml), [Ping Connect](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ping.yml),
[Jvm](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jvm.yml), [SiteMap](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-fullsite.yml),
[Ssl Certificate](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ssl_cert.yml), [SpringBoot2](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot2.yml),
[FTP Server](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ftp.yml), [SpringBoot3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot3.yml),
[Udp Port](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-udp_port.yml), [Dns](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dns.yml),
[Pop3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-pop3.yml), [Ntp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ntp.yml),
[Api Code](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api_code.yml), [Smtp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-smtp.yml),
[Nginx](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nginx.yml)
- [Mysql](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mysql.yml), [PostgreSQL](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-postgresql.yml),
[MariaDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mariadb.yml), [Redis](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis.yml),
[ElasticSearch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-elasticsearch.yml), [SqlServer](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-sqlserver.yml),
[Oracle](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-oracle.yml), [MongoDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mongodb.yml),
[DM](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dm.yml), [OpenGauss](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opengauss.yml),
[ClickHouse](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-clickhouse.yml), [IoTDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-iotdb.yml),
[Redis Cluster](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_cluster.yml), [Redis Sentinel](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_sentinel.yml)
[Doris BE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_be.yml), [Doris FE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_fe.yml),
[Memcached](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-memcached.yml), [NebulaGraph](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-nebula_graph.yml)
- [Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-linux.yml), [Ubuntu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ubuntu.yml),
[CentOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-centos.yml), [Windows](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-windows.yml),
[EulerOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-euleros.yml), [Fedora CoreOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-coreos.yml),
[OpenSUSE](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opensuse.yml), [Rocky Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rockylinux.yml),
[Red Hat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redhat.yml), [FreeBSD](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-freebsd.yml),
[AlmaLinux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-almalinux.yml), [Debian Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-debian.yml)
- [Tomcat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tomcat.yml), [Nacos](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nacos.yml),
[Zookeeper](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-zookeeper.yml), [RabbitMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rabbitmq.yml),
[Flink](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-flink.yml), [Kafka](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kafka.yml),
[ShenYu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-shenyu.yml), [DynamicTp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dynamic_tp.yml),
[Jetty](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jetty.yml), [ActiveMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-activemq.yml),
[Spring Gateway](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spring_gateway.yml), [EMQX MQTT](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-emqx.yml),
[AirFlow](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-airflow.yml), [Hive](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hive.yml),
[Spark](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spark.yml), [Hadoop](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hadoop.yml)
- [Kubernetes](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kubernetes.yml), [Docker](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-docker.yml)
- [CiscoSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-cisco_switch.yml), [HpeSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hpe_switch.yml),
[HuaweiSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-huawei_switch.yml), [TpLinkSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tplink_switch.yml),
[H3cSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-h3c_switch.yml)
- And More Your Custom Template.
- Notified Support `Discord` `Slack` `Telegram` `Email` `DingTalk` `WeChat` `FeiShu` `Webhook` `SMS` `ServerChan`.
## 🐕 Quick Start
- If you wish to deploy HertzBeat locally, please refer to the following Deployment Documentation for instructions.
@@ -551,6 +505,34 @@ Thanks to these wonderful people, welcome to join us:
<td align="center" valign="top" width="14.28%"><a href="https://shadwal.space/"><img src="https://avatars.githubusercontent.com/u/119167601?v=4?s=100" width="100px;" alt="Sahil Shadwal"/><br /><sub><b>Sahil Shadwal</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Sahil-Shadwal" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/BhanuNidumolu"><img src="https://avatars.githubusercontent.com/u/180380413?v=4?s=100" width="100px;" alt="N.Bhanu Prasad"/><br /><sub><b>N.Bhanu Prasad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=BhanuNidumolu" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://prakashh-portfolio.vercel.app/"><img src="https://avatars.githubusercontent.com/u/183058331?v=4?s=100" width="100px;" alt="Prakash Kumar"/><br /><sub><b>Prakash Kumar</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Prakash1185" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/orangeCatDeveloper"><img src="https://avatars.githubusercontent.com/u/95899648?v=4?s=100" width="100px;" alt="NekoPunch"/><br /><sub><b>NekoPunch</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://wy471x.github.io/"><img src="https://avatars.githubusercontent.com/u/52033069?v=4?s=100" width="100px;" alt="wy471x"/><br /><sub><b>wy471x</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wy471x" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hengyuss"><img src="https://avatars.githubusercontent.com/u/81064732?v=4?s=100" width="100px;" alt="hengyuss"/><br /><sub><b>hengyuss</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hengyuss" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://moduvoice.com/"><img src="https://avatars.githubusercontent.com/u/291867022?v=4?s=100" width="100px;" alt="moduvoice"/><br /><sub><b>moduvoice</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=moduvoice" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hutiefang76"><img src="https://avatars.githubusercontent.com/u/137664623?v=4?s=100" width="100px;" alt="hutiefang76"/><br /><sub><b>hutiefang76</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hutiefang76" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://zylatent.com/"><img src="https://avatars.githubusercontent.com/u/250777154?v=4?s=100" width="100px;" alt="柳含知 Liu Hanzhi"/><br /><sub><b>柳含知 Liu Hanzhi</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=ZhouYinLong-lab" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/wilmerdooley"><img src="https://avatars.githubusercontent.com/u/259930736?v=4?s=100" width="100px;" alt="wilmerdooley"/><br /><sub><b>wilmerdooley</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wilmerdooley" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Zmjjeff7"><img src="https://avatars.githubusercontent.com/u/175370943?v=4?s=100" width="100px;" alt="Zhenhong Guo"/><br /><sub><b>Zhenhong Guo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Zmjjeff7" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/abhyudayareddy"><img src="https://avatars.githubusercontent.com/u/54602866?v=4?s=100" width="100px;" alt="abhyudayareddy"/><br /><sub><b>abhyudayareddy</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=abhyudayareddy" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/neon-hippo"><img src="https://avatars.githubusercontent.com/u/165560498?v=4?s=100" width="100px;" alt="neon-hippo"/><br /><sub><b>neon-hippo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=neon-hippo" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/P-Peaceful"><img src="https://avatars.githubusercontent.com/u/52856161?v=4?s=100" width="100px;" alt="P_Peaceful"/><br /><sub><b>P_Peaceful</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhusaidong"><img src="https://avatars.githubusercontent.com/u/3039961?v=4?s=100" width="100px;" alt="zhusaidong"/><br /><sub><b>zhusaidong</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhusaidong" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhehenlu"><img src="https://avatars.githubusercontent.com/u/31504542?v=4?s=100" width="100px;" alt="zhlu"/><br /><sub><b>zhlu</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/brettgervasoni"><img src="https://avatars.githubusercontent.com/u/34056000?v=4?s=100" width="100px;" alt="brettgervasoni"/><br /><sub><b>brettgervasoni</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=brettgervasoni" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Darshan-paul"><img src="https://avatars.githubusercontent.com/u/211450705?v=4?s=100" width="100px;" alt="Darshan-paul"/><br /><sub><b>Darshan-paul</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Darshan-paul" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/04cb"><img src="https://avatars.githubusercontent.com/u/111667698?v=4?s=100" width="100px;" alt="layla"/><br /><sub><b>layla</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=04cb" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/miantalha45"><img src="https://avatars.githubusercontent.com/u/155809113?v=4?s=100" width="100px;" alt="Talha Amjad"/><br /><sub><b>Talha Amjad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=miantalha45" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://turanalmammadov.com/"><img src="https://avatars.githubusercontent.com/u/16321061?v=4?s=100" width="100px;" alt="Turan Almammadov"/><br /><sub><b>Turan Almammadov</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhaoyangplus"><img src="https://avatars.githubusercontent.com/u/245090302?v=4?s=100" width="100px;" alt="zhaoyangplus"/><br /><sub><b>zhaoyangplus</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhaoyangplus" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/yexuanyang"><img src="https://avatars.githubusercontent.com/u/73885401?v=4?s=100" width="100px;" alt="Yang Yexuan"/><br /><sub><b>Yang Yexuan</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=yexuanyang" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/markguo123"><img src="https://avatars.githubusercontent.com/u/155072651?v=4?s=100" width="100px;" alt="markguo123"/><br /><sub><b>markguo123</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=markguo123" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/leo-934"><img src="https://avatars.githubusercontent.com/u/55838224?v=4?s=100" width="100px;" alt="leo"/><br /><sub><b>leo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Documentation">📖</a> <a href="#blog-leo-934" title="Blogposts">📝</a></td>
</tr>
</tbody>
</table>
+42 -58
View File
@@ -19,13 +19,13 @@
[![Contribute with Gitpod](https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod&color=green)](https://gitpod.io/#https://github.com/apache/hertzbeat)
**官网: [hertzbeat.apache.org](https://hertzbeat.apache.org)**
**官网: [hertzbeat.apache.org](https://hertzbeat.apache.org)**
**邮件: <a href="mailto:dev-subscribe@hertzbeat.apache.org">发送至 ```dev-subscribe@hertzbeat.apache.org```</a>** 订阅邮件列表
## 🎡 <font color="green">介绍</font>
[Apache HertzBeat™](https://github.com/apache/hertzbeat) 是 AI 驱动的下一代开源实时观测系统。指标日志统一收集,告警一站分发,智能管控分析。无需 Agent,高性能集群,提供强大的自定义监控和状态页构建能力。
[Apache HertzBeat™](https://github.com/apache/hertzbeat) 是 AI 驱动的下一代开源实时观测系统。指标日志统一收集,告警一站分发,智能管控分析。无需 Agent,高性能集群,提供强大的自定义监控和状态页构建能力。
### 特点
@@ -39,70 +39,26 @@
- 提供强大的状态页构建能力,轻松向用户传达您产品服务的实时状态。
> `HertzBeat`的统一平台,AI智能,强大自定义,多类型支持,高性能,易扩展,希望能帮助用户快速方便实现观测需求。
> `HertzBeat`的统一平台,AI智能,强大自定义,多类型支持,高性能,易扩展,希望能帮助用户快速方便实现观测需求。
----
----
## 🥐 模块
## 🥐 模块
![hertzBeat](home/static/img/docs/hertzbeat-arch.png)
![hertzBeat](home/static/img/docs/hertzbeat-arch.png)
## ⛄ 已支持
## 🐕 快速开始
> 我们将监控采集类型(mysql,jvm,k8s)都定义为 `yml` 模板,用户可以导入这些模板来支持对应类型的监控!
> 欢迎大家一起贡献你使用过程中自定义的通用指标模板。
- [Website](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-website.yml), [Port Telnet](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-port.yml),
[Http Api](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api.yml), [Ping Connect](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ping.yml),
[Jvm](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jvm.yml), [SiteMap](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-fullsite.yml),
[Ssl Certificate](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ssl_cert.yml), [SpringBoot2](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot2.yml),
[FTP Server](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ftp.yml), [SpringBoot3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot3.yml),
[Udp Port](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-udp_port.yml), [Dns](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dns.yml),
[Pop3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-pop3.yml), [Ntp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ntp.yml),
[Api Code](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api_code.yml), [Smtp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-smtp.yml),
[Nginx](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nginx.yml)
- [Mysql](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mysql.yml), [PostgreSQL](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-postgresql.yml),
[MariaDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mariadb.yml), [Redis](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis.yml),
[ElasticSearch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-elasticsearch.yml), [SqlServer](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-sqlserver.yml),
[Oracle](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-oracle.yml), [MongoDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mongodb.yml),
[DM](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dm.yml), [OpenGauss](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opengauss.yml),
[ClickHouse](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-clickhouse.yml), [IoTDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-iotdb.yml),
[Redis Cluster](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_cluster.yml), [Redis Sentinel](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_sentinel.yml)
[Doris BE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_be.yml), [Doris FE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_fe.yml),
[Memcached](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-memcached.yml), [NebulaGraph](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-nebula_graph.yml)
- [Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-linux.yml), [Ubuntu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ubuntu.yml),
[CentOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-centos.yml), [Windows](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-windows.yml),
[EulerOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-euleros.yml), [Fedora CoreOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-coreos.yml),
[OpenSUSE](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opensuse.yml), [Rocky Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rockylinux.yml),
[Red Hat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redhat.yml), [FreeBSD](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-freebsd.yml),
[AlmaLinux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-almalinux.yml), [Debian Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-debian.yml)
- [Tomcat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tomcat.yml), [Nacos](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nacos.yml),
[Zookeeper](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-zookeeper.yml), [RabbitMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rabbitmq.yml),
[Flink](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-flink.yml), [Kafka](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kafka.yml),
[ShenYu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-shenyu.yml), [DynamicTp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dynamic_tp.yml),
[Jetty](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jetty.yml), [ActiveMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-activemq.yml),
[Spring Gateway](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spring_gateway.yml), [EMQX MQTT](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-emqx.yml),
[AirFlow](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-airflow.yml), [Hive](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hive.yml),
[Spark](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spark.yml), [Hadoop](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hadoop.yml)
- [Kubernetes](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kubernetes.yml), [Docker](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-docker.yml)
- [CiscoSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-cisco_switch.yml), [HpeSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hpe_switch.yml),
[HuaweiSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-huawei_switch.yml), [TpLinkSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tplink_switch.yml),
[H3cSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-h3c_switch.yml)
- 和更多自定义指标模板。
- 通知支持 `Discord` `Slack` `Telegram` `邮件` `钉钉` `微信` `飞书` `短信` `Webhook` `Server酱`
## 🐕 快速开始
- 如果您是想将 HertzBeat 部署到内网环境搭建监控系统,请参考下面的部署文档进行操作。
- 如果您是想将 HertzBeat 部署到内网环境搭建监控系统,请参考下面的部署文档进行操作。
### 🍞 HertzBeat 安装
> HertzBeat 支持通过源码安装启动,Docker 容器运行和安装包方式安装部署,CPU 架构支持 x86/arm64。
##### 方式一:Docker 方式快速安装
##### 方式一:Docker 方式快速安装
1. `docker` 环境仅需一条命令即可开始
1. `docker` 环境仅需一条命令即可开始
```shell
docker run -d -p 1157:1157 -p 1158:1158 --name hertzbeat apache/hertzbeat
@@ -164,7 +120,7 @@
通过 [Docker-Compose 部署脚本](script/docker-compose) 一次性把 postgresql/mysql 数据库, victoria-metrics/iotdb/tdengine 时序数据库和 hertzbeat 安装部署。
详细步骤参考 [通过 Docker-Compose 安装 HertzBeat](script/docker-compose/README.md)
详细步骤参考 [通过 Docker-Compose 安装 HertzBeat](script/docker-compose/README.md)
##### 方式五:Kubernetes Helm Charts 部署 hertzbeat+collector+postgresql+tsdb
@@ -176,8 +132,8 @@
## ✨ Contributors
Thanks these wonderful people, welcome to join us:
[贡献者指南](CONTRIBUTING.md)
Thanks these wonderful people, welcome to join us:
[贡献者指南](CONTRIBUTING.md)
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
@@ -548,6 +504,34 @@ Thanks these wonderful people, welcome to join us:
<td align="center" valign="top" width="14.28%"><a href="https://shadwal.space/"><img src="https://avatars.githubusercontent.com/u/119167601?v=4?s=100" width="100px;" alt="Sahil Shadwal"/><br /><sub><b>Sahil Shadwal</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Sahil-Shadwal" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/BhanuNidumolu"><img src="https://avatars.githubusercontent.com/u/180380413?v=4?s=100" width="100px;" alt="N.Bhanu Prasad"/><br /><sub><b>N.Bhanu Prasad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=BhanuNidumolu" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://prakashh-portfolio.vercel.app/"><img src="https://avatars.githubusercontent.com/u/183058331?v=4?s=100" width="100px;" alt="Prakash Kumar"/><br /><sub><b>Prakash Kumar</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Prakash1185" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/orangeCatDeveloper"><img src="https://avatars.githubusercontent.com/u/95899648?v=4?s=100" width="100px;" alt="NekoPunch"/><br /><sub><b>NekoPunch</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://wy471x.github.io/"><img src="https://avatars.githubusercontent.com/u/52033069?v=4?s=100" width="100px;" alt="wy471x"/><br /><sub><b>wy471x</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wy471x" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hengyuss"><img src="https://avatars.githubusercontent.com/u/81064732?v=4?s=100" width="100px;" alt="hengyuss"/><br /><sub><b>hengyuss</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hengyuss" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://moduvoice.com/"><img src="https://avatars.githubusercontent.com/u/291867022?v=4?s=100" width="100px;" alt="moduvoice"/><br /><sub><b>moduvoice</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=moduvoice" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hutiefang76"><img src="https://avatars.githubusercontent.com/u/137664623?v=4?s=100" width="100px;" alt="hutiefang76"/><br /><sub><b>hutiefang76</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hutiefang76" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://zylatent.com/"><img src="https://avatars.githubusercontent.com/u/250777154?v=4?s=100" width="100px;" alt="柳含知 Liu Hanzhi"/><br /><sub><b>柳含知 Liu Hanzhi</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=ZhouYinLong-lab" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/wilmerdooley"><img src="https://avatars.githubusercontent.com/u/259930736?v=4?s=100" width="100px;" alt="wilmerdooley"/><br /><sub><b>wilmerdooley</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wilmerdooley" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Zmjjeff7"><img src="https://avatars.githubusercontent.com/u/175370943?v=4?s=100" width="100px;" alt="Zhenhong Guo"/><br /><sub><b>Zhenhong Guo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Zmjjeff7" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/abhyudayareddy"><img src="https://avatars.githubusercontent.com/u/54602866?v=4?s=100" width="100px;" alt="abhyudayareddy"/><br /><sub><b>abhyudayareddy</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=abhyudayareddy" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/neon-hippo"><img src="https://avatars.githubusercontent.com/u/165560498?v=4?s=100" width="100px;" alt="neon-hippo"/><br /><sub><b>neon-hippo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=neon-hippo" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/P-Peaceful"><img src="https://avatars.githubusercontent.com/u/52856161?v=4?s=100" width="100px;" alt="P_Peaceful"/><br /><sub><b>P_Peaceful</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhusaidong"><img src="https://avatars.githubusercontent.com/u/3039961?v=4?s=100" width="100px;" alt="zhusaidong"/><br /><sub><b>zhusaidong</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhusaidong" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhehenlu"><img src="https://avatars.githubusercontent.com/u/31504542?v=4?s=100" width="100px;" alt="zhlu"/><br /><sub><b>zhlu</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/brettgervasoni"><img src="https://avatars.githubusercontent.com/u/34056000?v=4?s=100" width="100px;" alt="brettgervasoni"/><br /><sub><b>brettgervasoni</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=brettgervasoni" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Darshan-paul"><img src="https://avatars.githubusercontent.com/u/211450705?v=4?s=100" width="100px;" alt="Darshan-paul"/><br /><sub><b>Darshan-paul</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Darshan-paul" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/04cb"><img src="https://avatars.githubusercontent.com/u/111667698?v=4?s=100" width="100px;" alt="layla"/><br /><sub><b>layla</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=04cb" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/miantalha45"><img src="https://avatars.githubusercontent.com/u/155809113?v=4?s=100" width="100px;" alt="Talha Amjad"/><br /><sub><b>Talha Amjad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=miantalha45" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://turanalmammadov.com/"><img src="https://avatars.githubusercontent.com/u/16321061?v=4?s=100" width="100px;" alt="Turan Almammadov"/><br /><sub><b>Turan Almammadov</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhaoyangplus"><img src="https://avatars.githubusercontent.com/u/245090302?v=4?s=100" width="100px;" alt="zhaoyangplus"/><br /><sub><b>zhaoyangplus</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhaoyangplus" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/yexuanyang"><img src="https://avatars.githubusercontent.com/u/73885401?v=4?s=100" width="100px;" alt="Yang Yexuan"/><br /><sub><b>Yang Yexuan</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=yexuanyang" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/markguo123"><img src="https://avatars.githubusercontent.com/u/155072651?v=4?s=100" width="100px;" alt="markguo123"/><br /><sub><b>markguo123</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=markguo123" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/leo-934"><img src="https://avatars.githubusercontent.com/u/55838224?v=4?s=100" width="100px;" alt="leo"/><br /><sub><b>leo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Documentation">📖</a> <a href="#blog-leo-934" title="Blogposts">📝</a></td>
</tr>
</tbody>
</table>
@@ -555,11 +539,11 @@ Thanks these wonderful people, welcome to join us:
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
## 💬 社区交流
##### Channel
##### Channel
[订阅邮件列表](https://lists.apache.org/list.html?dev@hertzbeat.apache.org) : 发送邮件至 `dev-subscribe@hertzbeat.apache.org` 来订阅邮件列表.
+28 -44
View File
@@ -48,50 +48,6 @@
![hertzBeat](home/static/img/docs/hertzbeat-arch.png)
## ⛄ 利用可能
> `mysql`、`Java仮想マシン`、`k8s` などのすべての監視コレクション タイプを `YML` 監視テンプレートとして定義し、ユーザーがそれらをインポートして対応するタイプの監視をサポートできるようにしています。
> カスタマイズされた一般的なモニターのテンプレートを提供することを歓迎します。
- [Website](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-website.yml), [Port Telnet](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-port.yml),
[Http Api](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api.yml), [Ping Connect](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ping.yml),
[Jvm](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jvm.yml), [SiteMap](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-fullsite.yml),
[Ssl Certificate](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ssl_cert.yml), [SpringBoot2](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot2.yml),
[FTP Server](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ftp.yml), [SpringBoot3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-springboot3.yml),
[Udp Port](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-udp_port.yml), [Dns](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dns.yml),
[Pop3](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-pop3.yml), [Ntp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ntp.yml),
[Api Code](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-api_code.yml), [Smtp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-smtp.yml),
[Nginx](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nginx.yml)
- [Mysql](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mysql.yml), [PostgreSQL](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-postgresql.yml),
[MariaDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mariadb.yml), [Redis](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis.yml),
[ElasticSearch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-elasticsearch.yml), [SqlServer](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-sqlserver.yml),
[Oracle](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-oracle.yml), [MongoDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-mongodb.yml),
[DM](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dm.yml), [OpenGauss](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opengauss.yml),
[ClickHouse](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-clickhouse.yml), [IoTDB](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-iotdb.yml),
[Redis Cluster](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_cluster.yml), [Redis Sentinel](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redis_sentinel.yml)
[Doris BE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_be.yml), [Doris FE](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-doris_fe.yml),
[Memcached](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-memcached.yml), [NebulaGraph](https://github.com/apache/hertzbeat/blob/master/hertzbeat-manager/src/main/resources/define/app-nebula_graph.yml)
- [Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-linux.yml), [Ubuntu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-ubuntu.yml),
[CentOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-centos.yml), [Windows](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-windows.yml),
[EulerOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-euleros.yml), [Fedora CoreOS](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-coreos.yml),
[OpenSUSE](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-opensuse.yml), [Rocky Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rockylinux.yml),
[Red Hat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-redhat.yml), [FreeBSD](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-freebsd.yml),
[AlmaLinux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-almalinux.yml), [Debian Linux](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-debian.yml)
- [Tomcat](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tomcat.yml), [Nacos](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-nacos.yml),
[Zookeeper](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-zookeeper.yml), [RabbitMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-rabbitmq.yml),
[Flink](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-flink.yml), [Kafka](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kafka.yml),
[ShenYu](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-shenyu.yml), [DynamicTp](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-dynamic_tp.yml),
[Jetty](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-jetty.yml), [ActiveMQ](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-activemq.yml),
[Spring Gateway](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spring_gateway.yml), [EMQX MQTT](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-emqx.yml),
[AirFlow](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-airflow.yml), [Hive](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hive.yml),
[Spark](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-spark.yml), [Hadoop](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hadoop.yml)
- [Kubernetes](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-kubernetes.yml), [Docker](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-docker.yml)
- [CiscoSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-cisco_switch.yml), [HpeSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-hpe_switch.yml),
[HuaweiSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-huawei_switch.yml), [TpLinkSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-tplink_switch.yml),
[H3cSwitch](https://raw.githubusercontent.com/apache/hertzbeat/master/hertzbeat-manager/src/main/resources/define/app-h3c_switch.yml)
- その他、カスタマイズされたモニターテンプレート。
- 通知利用可能 `Discord` `Slack` `Telegram` `Email` `DingTalk` `WeChat` `FeiShu` `Webhook` `SMS` `ServerChan`
## 🐕 クイックスタート
- HertzBeat をローカルに展開する場合は、以下のデプロイメントドキュメントを参照してください。
@@ -551,6 +507,34 @@ Thanks these wonderful people, welcome to join us:
<td align="center" valign="top" width="14.28%"><a href="https://shadwal.space/"><img src="https://avatars.githubusercontent.com/u/119167601?v=4?s=100" width="100px;" alt="Sahil Shadwal"/><br /><sub><b>Sahil Shadwal</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Sahil-Shadwal" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/BhanuNidumolu"><img src="https://avatars.githubusercontent.com/u/180380413?v=4?s=100" width="100px;" alt="N.Bhanu Prasad"/><br /><sub><b>N.Bhanu Prasad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=BhanuNidumolu" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://prakashh-portfolio.vercel.app/"><img src="https://avatars.githubusercontent.com/u/183058331?v=4?s=100" width="100px;" alt="Prakash Kumar"/><br /><sub><b>Prakash Kumar</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Prakash1185" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/orangeCatDeveloper"><img src="https://avatars.githubusercontent.com/u/95899648?v=4?s=100" width="100px;" alt="NekoPunch"/><br /><sub><b>NekoPunch</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=orangeCatDeveloper" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://wy471x.github.io/"><img src="https://avatars.githubusercontent.com/u/52033069?v=4?s=100" width="100px;" alt="wy471x"/><br /><sub><b>wy471x</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wy471x" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hengyuss"><img src="https://avatars.githubusercontent.com/u/81064732?v=4?s=100" width="100px;" alt="hengyuss"/><br /><sub><b>hengyuss</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hengyuss" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://moduvoice.com/"><img src="https://avatars.githubusercontent.com/u/291867022?v=4?s=100" width="100px;" alt="moduvoice"/><br /><sub><b>moduvoice</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=moduvoice" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hutiefang76"><img src="https://avatars.githubusercontent.com/u/137664623?v=4?s=100" width="100px;" alt="hutiefang76"/><br /><sub><b>hutiefang76</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hutiefang76" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://zylatent.com/"><img src="https://avatars.githubusercontent.com/u/250777154?v=4?s=100" width="100px;" alt="柳含知 Liu Hanzhi"/><br /><sub><b>柳含知 Liu Hanzhi</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=ZhouYinLong-lab" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/wilmerdooley"><img src="https://avatars.githubusercontent.com/u/259930736?v=4?s=100" width="100px;" alt="wilmerdooley"/><br /><sub><b>wilmerdooley</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=wilmerdooley" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Zmjjeff7"><img src="https://avatars.githubusercontent.com/u/175370943?v=4?s=100" width="100px;" alt="Zhenhong Guo"/><br /><sub><b>Zhenhong Guo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Zmjjeff7" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/abhyudayareddy"><img src="https://avatars.githubusercontent.com/u/54602866?v=4?s=100" width="100px;" alt="abhyudayareddy"/><br /><sub><b>abhyudayareddy</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=abhyudayareddy" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/neon-hippo"><img src="https://avatars.githubusercontent.com/u/165560498?v=4?s=100" width="100px;" alt="neon-hippo"/><br /><sub><b>neon-hippo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=neon-hippo" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/P-Peaceful"><img src="https://avatars.githubusercontent.com/u/52856161?v=4?s=100" width="100px;" alt="P_Peaceful"/><br /><sub><b>P_Peaceful</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=P-Peaceful" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhusaidong"><img src="https://avatars.githubusercontent.com/u/3039961?v=4?s=100" width="100px;" alt="zhusaidong"/><br /><sub><b>zhusaidong</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhusaidong" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhehenlu"><img src="https://avatars.githubusercontent.com/u/31504542?v=4?s=100" width="100px;" alt="zhlu"/><br /><sub><b>zhlu</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=zhehenlu" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/brettgervasoni"><img src="https://avatars.githubusercontent.com/u/34056000?v=4?s=100" width="100px;" alt="brettgervasoni"/><br /><sub><b>brettgervasoni</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=brettgervasoni" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Darshan-paul"><img src="https://avatars.githubusercontent.com/u/211450705?v=4?s=100" width="100px;" alt="Darshan-paul"/><br /><sub><b>Darshan-paul</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Darshan-paul" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/04cb"><img src="https://avatars.githubusercontent.com/u/111667698?v=4?s=100" width="100px;" alt="layla"/><br /><sub><b>layla</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=04cb" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/miantalha45"><img src="https://avatars.githubusercontent.com/u/155809113?v=4?s=100" width="100px;" alt="Talha Amjad"/><br /><sub><b>Talha Amjad</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=miantalha45" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://turanalmammadov.com/"><img src="https://avatars.githubusercontent.com/u/16321061?v=4?s=100" width="100px;" alt="Turan Almammadov"/><br /><sub><b>Turan Almammadov</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=turanalmammadov" title="Documentation">📖</a></td>
</tr>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/zhaoyangplus"><img src="https://avatars.githubusercontent.com/u/245090302?v=4?s=100" width="100px;" alt="zhaoyangplus"/><br /><sub><b>zhaoyangplus</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=zhaoyangplus" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/yexuanyang"><img src="https://avatars.githubusercontent.com/u/73885401?v=4?s=100" width="100px;" alt="Yang Yexuan"/><br /><sub><b>Yang Yexuan</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=yexuanyang" title="Code">💻</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/markguo123"><img src="https://avatars.githubusercontent.com/u/155072651?v=4?s=100" width="100px;" alt="markguo123"/><br /><sub><b>markguo123</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=markguo123" title="Documentation">📖</a></td>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/leo-934"><img src="https://avatars.githubusercontent.com/u/55838224?v=4?s=100" width="100px;" alt="leo"/><br /><sub><b>leo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Code">💻</a> <a href="https://github.com/apache/hertzbeat/commits?author=leo-934" title="Documentation">📖</a> <a href="#blog-leo-934" title="Blogposts">📝</a></td>
</tr>
</tbody>
</table>
@@ -18,14 +18,21 @@
package org.apache.hertzbeat.ai.config;
import com.usthe.sureness.subject.SubjectSum;
import org.springframework.core.NamedInheritableThreadLocal;
import com.usthe.sureness.util.SurenessContextHolder;
import java.util.Map;
import java.util.function.Supplier;
import org.springframework.ai.chat.model.ToolContext;
import org.springframework.core.NamedThreadLocal;
/**
* Context holder for AI agent security context.
*/
public final class McpContextHolder {
static final String SUBJECT_CONTEXT_KEY = McpContextHolder.class.getName() + ".subject";
private static final ThreadLocal<SubjectSum> subjectHolder =
new NamedInheritableThreadLocal<>("MCP Security and User Identification Context");
new NamedThreadLocal<>("MCP Security and User Identification Context");
private McpContextHolder() {}
@@ -44,10 +51,73 @@ public final class McpContextHolder {
return subjectHolder.get();
}
/**
* Creates a security context that is propagated only through the Spring AI tool invocation chain.
*
* @param subject current authenticated subject, may be {@code null}
* @return tool context without null values
*/
public static Map<String, Object> createToolContext(SubjectSum subject) {
return subject == null ? Map.of() : Map.of(SUBJECT_CONTEXT_KEY, subject);
}
/**
* Retrieves and validates the authenticated subject from the Spring AI tool context.
*
* @param toolContext Spring AI tool context
* @return authenticated subject, or {@code null} when absent
*/
public static SubjectSum getSubject(ToolContext toolContext) {
if (toolContext == null) {
return null;
}
Object subject = toolContext.getContext().get(SUBJECT_CONTEXT_KEY);
return subject instanceof SubjectSum subjectSum ? subjectSum : null;
}
/**
* Executes a synchronous tool invocation within the given subject scope and restores the original thread
* contexts afterward.
*
* <p>Spring AI tool callbacks may run on pooled threads, so they cannot rely on a {@link ThreadLocal} left on
* the Servlet request thread. This method binds both MCP and Sureness contexts so the tool and downstream
* services observe the same user identity.</p>
*
* @param subject current request subject, may be {@code null}
* @param operation synchronous tool invocation
* @param <T> tool result type
* @return tool invocation result
*/
public static <T> T callWithSubject(SubjectSum subject, Supplier<T> operation) {
SubjectSum previousMcpSubject = getSubject();
SubjectSum previousSurenessSubject = SurenessContextHolder.getBindSubject();
replaceSubjects(subject);
try {
return operation.get();
} finally {
replaceSubjects(previousMcpSubject, previousSurenessSubject);
}
}
/**
* Clears the context from the thread to prevent memory leaks.
*/
public static void clear() {
subjectHolder.remove();
}
private static void replaceSubjects(SubjectSum subject) {
replaceSubjects(subject, subject);
}
private static void replaceSubjects(SubjectSum mcpSubject, SubjectSum surenessSubject) {
clear();
SurenessContextHolder.unbindSubject();
if (mcpSubject != null) {
setSubject(mcpSubject);
}
if (surenessSubject != null) {
SurenessContextHolder.bindSubject(surenessSubject);
}
}
}
@@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.config;
import org.springframework.ai.chat.model.ToolContext;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.ai.tool.metadata.ToolMetadata;
/**
* Establishes a request-scoped security context for Spring AI tool callbacks.
*
* <p>The wrapper keeps the tool definition and metadata unchanged. It binds the identity only for the delegated
* call and restores the worker thread's original state after both successful and failed invocations.</p>
*/
public final class SecurityContextToolCallback implements ToolCallback {
private final ToolCallback delegate;
public SecurityContextToolCallback(ToolCallback delegate) {
this.delegate = delegate;
}
@Override
public ToolDefinition getToolDefinition() {
return delegate.getToolDefinition();
}
@Override
public ToolMetadata getToolMetadata() {
return delegate.getToolMetadata();
}
@Override
public String call(String toolInput) {
return call(toolInput, null);
}
@Override
public String call(String toolInput, ToolContext toolContext) {
return McpContextHolder.callWithSubject(
McpContextHolder.getSubject(toolContext),
() -> delegate.call(toolInput, toolContext));
}
}
@@ -17,13 +17,10 @@
package org.apache.hertzbeat.ai.controller;
import com.usthe.sureness.subject.SubjectSum;
import com.usthe.sureness.util.SurenessContextHolder;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.ai.config.McpContextHolder;
import org.apache.hertzbeat.ai.pojo.dto.ChatRequestContext;
import org.apache.hertzbeat.ai.pojo.dto.ChatResponseChunk;
import org.apache.hertzbeat.ai.pojo.dto.SecurityData;
@@ -75,8 +72,6 @@ public class ChatController {
public Flux<ServerSentEvent<ChatResponseChunk>> streamChat(@Valid @RequestBody ChatRequestContext context) {
try {
// Validate message is not empty
SubjectSum subject = SurenessContextHolder.getBindSubject();
McpContextHolder.setSubject(subject);
if (context.getMessage() == null || context.getMessage().trim().isEmpty()) {
ChatResponseChunk errorResponse = ChatResponseChunk.builder()
.conversationId(context.getConversationId())
@@ -18,12 +18,13 @@
package org.apache.hertzbeat.ai.pojo.dto;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.usthe.sureness.subject.SubjectSum;
import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
import org.apache.hertzbeat.common.entity.ai.ChatMessage;
/**
@@ -47,4 +48,11 @@ public class ChatRequestContext {
* Conversation history messages for context
*/
private List<ChatMessage> conversationHistory;
/**
* Authenticated subject captured by the server. It cannot be supplied through client JSON and is never included
* in model messages.
*/
@JsonIgnore
private SubjectSum subject;
}
@@ -36,6 +36,7 @@ import org.springframework.context.annotation.Lazy;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import tools.jackson.core.type.TypeReference;
/**
* Scheduled executor that checks for due SOP schedules and executes them.
@@ -82,7 +83,11 @@ public class SopScheduleExecutor {
log.info("Found {} due schedules to execute", dueSchedules.size());
for (SopSchedule schedule : dueSchedules) {
executeSchedule(schedule);
try {
executeSchedule(schedule);
} catch (Exception e) {
log.error("Unexpected error processing scheduled SOP {}", schedule.getId(), e);
}
}
} catch (Exception e) {
log.error("Error checking due schedules", e);
@@ -108,11 +113,12 @@ public class SopScheduleExecutor {
// Parse parameters
Map<String, Object> params = new HashMap<>();
if (schedule.getSopParams() != null && !schedule.getSopParams().isEmpty()) {
try {
params = JsonUtil.fromJson(schedule.getSopParams(), Map.class);
} catch (Exception e) {
log.warn("Failed to parse SOP params: {}", schedule.getSopParams());
Map<String, Object> parsedParams = JsonUtil.fromJson(
schedule.getSopParams(), new TypeReference<>() {});
if (parsedParams == null) {
throw new IllegalArgumentException("SOP schedule parameters must be a valid JSON object");
}
params = parsedParams;
}
// Execute SOP
@@ -19,10 +19,13 @@
package org.apache.hertzbeat.ai.service.impl;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.ai.config.McpContextHolder;
import org.apache.hertzbeat.ai.config.SecurityContextToolCallback;
import org.apache.hertzbeat.ai.sop.model.SopDefinition;
import org.apache.hertzbeat.ai.sop.model.SopParameter;
import org.apache.hertzbeat.ai.sop.registry.SkillRegistry;
@@ -45,9 +48,11 @@ import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.ApplicationContext;
import org.springframework.util.StringUtils;
import reactor.core.publisher.Flux;
import java.io.IOException;
@@ -69,7 +74,7 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
private final GeneralConfigDao generalConfigDao;
private ModelProviderConfig modelProviderConfig;
private volatile ModelProviderConfig modelProviderConfig;
private final SkillRegistry skillRegistry;
@@ -78,7 +83,9 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
@Qualifier("hertzbeatTools")
private ToolCallbackProvider toolCallbackProvider;
private boolean isConfigured = false;
private volatile boolean configured;
private volatile boolean configurationLoaded;
@Value("classpath:/prompt/system-message.st")
private Resource systemResource;
@@ -122,11 +129,15 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
// Build system prompt with dynamic skills list and conversation ID
// The conversationId is injected into the prompt so AI can pass it to schedule tools
String systemPrompt = buildSystemPrompt(context.getConversationId());
ToolCallback[] toolCallbacks = Arrays.stream(toolCallbackProvider.getToolCallbacks())
.map(SecurityContextToolCallback::new)
.toArray(ToolCallback[]::new);
return chatClient.prompt()
.messages(messages)
.system(systemPrompt)
.tools(toolCallbackProvider)
.tools((Object[]) toolCallbacks)
.toolContext(McpContextHolder.createToolContext(context.getSubject()))
.stream()
.content()
.doOnComplete(() -> log.info("Streaming completed for conversation: {}", context.getConversationId()))
@@ -150,7 +161,8 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
.replace(CONVERSATION_ID_PLACEHOLDER, String.valueOf(conversationId));
// add extra prompt for protected model to guide it to use protected tools
if (Objects.equals(modelProviderConfig.getParticipationModel(), "PROTECTED")) {
ModelProviderConfig currentConfig = modelProviderConfig;
if (currentConfig != null && Objects.equals(currentConfig.getParticipationModel(), "PROTECTED")) {
Map<String, Object> metadata = new HashMap<>();
metadata.put("conversationId", conversationId);
return template + SystemPromptTemplate.builder().resource(extraResourceProtected).build()
@@ -202,19 +214,32 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
@EventListener(AiProviderConfigChangeEvent.class)
public void onAiProviderConfigChange(AiProviderConfigChangeEvent event) {
GeneralConfig providerConfig = generalConfigDao.findByType("provider");
this.modelProviderConfig = JsonUtil.fromJson(providerConfig.getContent(), ModelProviderConfig.class);
refreshProviderConfiguration();
}
@Override
public boolean isConfigured() {
if (!isConfigured) {
GeneralConfig providerConfig = generalConfigDao.findByType("provider");
ModelProviderConfig modelProviderConfig = JsonUtil.fromJson(providerConfig.getContent(),
ModelProviderConfig.class);
isConfigured = modelProviderConfig != null && modelProviderConfig.getApiKey() != null;
this.modelProviderConfig = modelProviderConfig;
if (!configurationLoaded) {
synchronized (this) {
if (!configurationLoaded) {
refreshProviderConfiguration();
}
}
}
return isConfigured;
return configured;
}
/**
* Atomically refreshes the configuration snapshot after enabling, disabling, or switching providers.
*/
private synchronized void refreshProviderConfiguration() {
GeneralConfig providerConfig = generalConfigDao.findByType("provider");
ModelProviderConfig refreshedConfig = null;
if (providerConfig != null && StringUtils.hasText(providerConfig.getContent())) {
refreshedConfig = JsonUtil.fromJson(providerConfig.getContent(), ModelProviderConfig.class);
}
modelProviderConfig = refreshedConfig;
configured = refreshedConfig != null && StringUtils.hasText(refreshedConfig.getApiKey());
configurationLoaded = true;
}
}
@@ -17,10 +17,13 @@
package org.apache.hertzbeat.ai.service.impl;
import com.usthe.sureness.subject.SubjectSum;
import com.usthe.sureness.util.SurenessContextHolder;
import java.util.Optional;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.ai.dao.ChatConversationDao;
import org.apache.hertzbeat.ai.dao.ChatMessageDao;
import org.apache.hertzbeat.ai.dao.SopScheduleDao;
import org.apache.hertzbeat.ai.pojo.dto.ChatRequestContext;
import org.apache.hertzbeat.ai.pojo.dto.ChatResponseChunk;
import org.apache.hertzbeat.ai.pojo.dto.SecurityData;
@@ -55,6 +58,9 @@ public class ConversationServiceImpl implements ConversationService {
@Autowired
private ChatMessageDao messageDao;
@Autowired
private SopScheduleDao sopScheduleDao;
@Autowired
private ChatClientProviderService chatClientProviderService;
@@ -104,6 +110,8 @@ public class ConversationServiceImpl implements ConversationService {
// Stream response from AI service
StringBuilder fullResponse = new StringBuilder();
ChatMessage finalChatMessage = chatMessage;
SubjectSum subject = SurenessContextHolder.getBindSubject();
context.setSubject(subject);
return chatClientProviderService.streamChat(context)
.map(chunk -> {
fullResponse.append(chunk);
@@ -193,6 +201,8 @@ public class ConversationServiceImpl implements ConversationService {
@Override
@Transactional(rollbackFor = Exception.class)
public void deleteConversation(Long conversationId) {
// Delete associated schedules first to prevent tasks from writing orphaned messages.
sopScheduleDao.deleteByConversationId(conversationId);
List<ChatMessage> messages = messageDao.findByConversationIdOrderByGmtCreateAsc(conversationId);
if (!messages.isEmpty()) {
messageDao.deleteAll(messages);
@@ -41,9 +41,6 @@ import reactor.core.publisher.Flux;
@Service
public class SopEngineImpl implements SopEngine {
// Thread-local context for each execution
private static final ThreadLocal<Map<String, Object>> CONTEXT_BUS = ThreadLocal.withInitial(HashMap::new);
private final List<SopExecutor> executors;
@Autowired
@@ -55,22 +52,12 @@ public class SopEngineImpl implements SopEngine {
public Flux<String> execute(SopDefinition definition, Map<String, Object> inputParams) {
return Flux.create(sink -> {
try {
OutputConfig outputConfig = resolveOutputConfig(definition);
Map<String, Object> context = prepareContext(definition, inputParams, outputConfig);
log.info("Starting execution of SOP: {}", definition.getName());
sink.next("Starting SOP: " + definition.getName() + " (v" + definition.getVersion() + ")");
// Initialize context with input parameters
Map<String, Object> context = CONTEXT_BUS.get();
context.clear();
context.putAll(inputParams);
// Add language configuration to context
OutputConfig outputConfig = definition.getOutput();
if (outputConfig != null) {
context.put("_language", outputConfig.getLanguageCode());
} else {
context.put("_language", "zh");
}
for (SopStep step : definition.getSteps()) {
sink.next("Executing step [" + step.getId() + "]: " + step.getType());
@@ -106,10 +93,9 @@ public class SopEngineImpl implements SopEngine {
sink.next("SOP " + definition.getName() + " completed successfully.");
sink.complete();
} catch (Exception e) {
log.error("Error executing SOP {}: {}", definition.getName(), e.getMessage(), e);
String sopName = definition == null ? "unknown" : definition.getName();
log.error("Error executing SOP {}: {}", sopName, e.getMessage(), e);
sink.error(e);
} finally {
CONTEXT_BUS.remove();
}
});
}
@@ -118,40 +104,20 @@ public class SopEngineImpl implements SopEngine {
public SopResult executeSync(SopDefinition definition, Map<String, Object> inputParams) {
long startTime = System.currentTimeMillis();
List<StepResult> stepResults = new ArrayList<>();
Map<String, Object> context = new HashMap<>(inputParams);
// Apply default values for parameters that are not provided
if (definition.getParameters() != null) {
for (SopParameter param : definition.getParameters()) {
if (!context.containsKey(param.getName()) && param.getDefaultValue() != null) {
context.put(param.getName(), param.getDefaultValue());
}
}
}
// Get output configuration first
OutputConfig outputConfig = definition.getOutput();
if (outputConfig == null) {
outputConfig = OutputConfig.builder()
.type("simple")
.format("text")
.language("zh")
.build();
}
// Add language configuration to context
context.put("_language", outputConfig.getLanguageCode());
SopResult.SopResultBuilder resultBuilder = SopResult.builder()
.sopName(definition.getName())
.sopVersion(definition.getVersion())
.sopName(definition == null ? null : definition.getName())
.sopVersion(definition == null ? null : definition.getVersion())
.startTime(startTime);
resultBuilder.outputType(outputConfig.getOutputType());
resultBuilder.outputFormat(outputConfig.getFormat() != null ? outputConfig.getFormat() : "text");
resultBuilder.language(outputConfig.getLanguageCode());
try {
OutputConfig outputConfig = resolveOutputConfig(definition);
Map<String, Object> context = prepareContext(definition, inputParams, outputConfig);
resultBuilder.outputType(outputConfig.getOutputType());
resultBuilder.outputFormat(outputConfig.getFormat() != null ? outputConfig.getFormat() : "text");
resultBuilder.language(outputConfig.getLanguageCode());
log.info("Starting sync execution of SOP: {}", definition.getName());
for (SopStep step : definition.getSteps()) {
@@ -213,10 +179,55 @@ public class SopEngineImpl implements SopEngine {
return resultBuilder.build();
} catch (Exception e) {
log.error("Error executing SOP {}: {}", definition.getName(), e.getMessage(), e);
String sopName = definition == null ? "unknown" : definition.getName();
log.error("Error executing SOP {}: {}", sopName, e.getMessage(), e);
return buildFailedResult(resultBuilder, stepResults, e.getMessage(), startTime);
}
}
/**
* Applies defaults and validates required parameters consistently for every execution entry point.
*/
private Map<String, Object> prepareContext(SopDefinition definition, Map<String, Object> inputParams,
OutputConfig outputConfig) {
if (definition.getSteps() == null || definition.getSteps().isEmpty()) {
throw new IllegalArgumentException("SOP must contain at least one step");
}
Map<String, Object> context = inputParams == null ? new HashMap<>() : new HashMap<>(inputParams);
if (definition.getParameters() != null) {
for (SopParameter parameter : definition.getParameters()) {
Object value = context.get(parameter.getName());
if (isMissing(value) && parameter.getDefaultValue() != null) {
context.put(parameter.getName(), parameter.getDefaultValue());
value = parameter.getDefaultValue();
}
if (parameter.isRequired() && isMissing(value)) {
throw new IllegalArgumentException("Required SOP parameter is missing: " + parameter.getName());
}
}
}
context.put("_language", outputConfig.getLanguageCode());
return context;
}
private boolean isMissing(Object value) {
return value == null || value instanceof String text && text.isBlank();
}
private OutputConfig resolveOutputConfig(SopDefinition definition) {
if (definition == null) {
throw new IllegalArgumentException("SOP definition must not be null");
}
if (definition.getOutput() != null) {
return definition.getOutput();
}
return OutputConfig.builder()
.type("simple")
.format("text")
.language("zh")
.build();
}
private SopResult buildFailedResult(SopResult.SopResultBuilder builder,
List<StepResult> stepResults, String error, long startTime) {
@@ -17,8 +17,16 @@
package org.apache.hertzbeat.ai.sop.registry;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.apache.hertzbeat.ai.sop.engine.SopEngine;
import org.apache.hertzbeat.ai.sop.model.SopDefinition;
import org.apache.hertzbeat.ai.sop.model.SopParameter;
import org.apache.hertzbeat.ai.sop.model.SopResult;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
@@ -49,33 +57,78 @@ public class SopToolCallback implements ToolCallback {
@Override
public String call(String arguments) {
// TODO: Parse arguments and execute SOP
return "SOP " + definition.getName() + " execution started.";
Map<String, Object> inputParams = parseArguments(arguments);
SopResult result = sopEngine.executeSync(definition, inputParams);
return result.toAiResponse();
}
private String buildInputSchema() {
// Build JSON Schema for SOP parameters
StringBuilder schema = new StringBuilder();
schema.append("{\"type\":\"object\",\"properties\":{");
if (definition.getParameters() != null && !definition.getParameters().isEmpty()) {
boolean first = true;
for (var param : definition.getParameters()) {
if (!first) {
schema.append(",");
Map<String, Object> schema = new LinkedHashMap<>();
Map<String, Object> properties = new LinkedHashMap<>();
List<String> required = new ArrayList<>();
if (definition.getParameters() != null) {
for (SopParameter parameter : definition.getParameters()) {
Map<String, Object> property = new LinkedHashMap<>();
property.put("type", mapType(parameter.getType()));
if (parameter.getDescription() != null) {
property.put("description", parameter.getDescription());
}
schema.append("\"").append(param.getName()).append("\":");
schema.append("{\"type\":\"").append(mapType(param.getType())).append("\"");
if (param.getDescription() != null) {
schema.append(",\"description\":\"").append(param.getDescription()).append("\"");
if (parameter.getDefaultValue() != null) {
property.put("default", convertDefaultValue(parameter));
}
properties.put(parameter.getName(), property);
if (parameter.isRequired()) {
required.add(parameter.getName());
}
schema.append("}");
first = false;
}
}
schema.append("}}");
return schema.toString();
schema.put("type", "object");
schema.put("properties", properties);
if (!required.isEmpty()) {
schema.put("required", required);
}
schema.put("additionalProperties", false);
return JsonUtil.toJson(schema);
}
@SuppressWarnings("unchecked")
private Map<String, Object> parseArguments(String arguments) {
if (arguments == null || arguments.isBlank()) {
return new HashMap<>();
}
Map<String, Object> inputParams = JsonUtil.fromJson(arguments, Map.class);
if (inputParams == null) {
throw new IllegalArgumentException("SOP tool arguments must be a valid JSON object");
}
return inputParams;
}
private Object convertDefaultValue(SopParameter parameter) {
String defaultValue = parameter.getDefaultValue();
try {
return switch (mapType(parameter.getType())) {
case "boolean" -> parseBooleanDefault(parameter, defaultValue);
case "integer" -> Long.valueOf(defaultValue);
case "number" -> Double.valueOf(defaultValue);
default -> defaultValue;
};
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Invalid default value for SOP parameter: " + parameter.getName(), e);
}
}
private boolean parseBooleanDefault(SopParameter parameter, String defaultValue) {
String normalizedValue = defaultValue.trim();
if ("true".equalsIgnoreCase(normalizedValue)) {
return true;
}
if ("false".equalsIgnoreCase(normalizedValue)) {
return false;
}
throw new IllegalArgumentException(
"Invalid boolean default value for SOP parameter: " + parameter.getName());
}
private String mapType(String type) {
@@ -17,31 +17,38 @@
package org.apache.hertzbeat.ai.sop.registry;
import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Arrays;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.core.annotation.AnnotationUtils;
import org.springframework.stereotype.Component;
import org.springframework.util.ClassUtils;
import org.springframework.util.ReflectionUtils;
/**
* Registry for all @Tool annotated methods.
* Automatically discovers and registers tool methods on first use.
* Registry for Spring AI tools.
*
* <p>Scans beans containing {@link Tool} methods on first use and delegates argument deserialization,
* proxy method resolution, and result conversion to Spring AI's {@link MethodToolCallbackProvider}.
*/
@Slf4j
@Component
public class ToolRegistry {
private final ApplicationContext applicationContext;
private final Map<String, ToolMethod> tools = new HashMap<>();
private volatile Map<String, RegisteredTool> tools = Map.of();
private volatile boolean initialized = false;
@Autowired
@@ -49,16 +56,11 @@ public class ToolRegistry {
this.applicationContext = applicationContext;
}
/**
* Initialize the registry by scanning all beans for @Tool methods.
* Uses double-checked locking for thread safety.
*/
private void ensureInitialized() {
if (!initialized) {
synchronized (this) {
if (!initialized) {
log.info("Scanning for @Tool annotated methods...");
scanAllBeans();
tools = scanAllBeans();
initialized = true;
log.info("Registered {} tool methods: {}", tools.size(), tools.keySet());
}
@@ -66,58 +68,79 @@ public class ToolRegistry {
}
}
private void scanAllBeans() {
String[] beanNames = applicationContext.getBeanDefinitionNames();
for (String beanName : beanNames) {
try {
Object bean = applicationContext.getBean(beanName);
scanBeanForTools(bean);
} catch (Exception e) {
// Skip beans that cannot be instantiated
log.trace("Skipping bean {}: {}", beanName, e.getMessage());
}
private Map<String, RegisteredTool> scanAllBeans() {
log.info("Scanning for @Tool annotated methods...");
List<Object> toolBeans = Arrays.stream(applicationContext.getBeanDefinitionNames())
.filter(this::containsToolMethod)
.map(applicationContext::getBean)
.toList();
if (toolBeans.isEmpty()) {
return Map.of();
}
ToolCallback[] callbacks = MethodToolCallbackProvider.builder().toolObjects(toolBeans.toArray())
.build().getToolCallbacks();
Map<String, RegisteredTool> discoveredTools = new LinkedHashMap<>();
for (ToolCallback callback : callbacks) {
String toolName = callback.getToolDefinition().name();
discoveredTools.put(toolName, new RegisteredTool(callback, getRequiredParameters(callback)));
}
return Collections.unmodifiableMap(discoveredTools);
}
private void scanBeanForTools(Object bean) {
Class<?> clazz = bean.getClass();
// Handle Spring proxies
if (clazz.getName().contains("$$")) {
clazz = clazz.getSuperclass();
private boolean containsToolMethod(String beanName) {
Class<?> beanType = applicationContext.getType(beanName, false);
if (beanType == null) {
return false;
}
for (Method method : clazz.getMethods()) {
Tool toolAnnotation = method.getAnnotation(Tool.class);
if (toolAnnotation != null) {
String toolName = toolAnnotation.name();
if (toolName.isEmpty()) {
toolName = method.getName();
}
ToolMethod toolMethod = new ToolMethod(bean, method, toolAnnotation);
tools.put(toolName, toolMethod);
log.debug("Registered tool: {} -> {}.{}",
toolName, clazz.getSimpleName(), method.getName());
}
return Arrays.stream(ReflectionUtils.getDeclaredMethods(ClassUtils.getUserClass(beanType)))
.anyMatch(this::isToolMethod);
}
private boolean isToolMethod(Method method) {
return AnnotationUtils.findAnnotation(method, Tool.class) != null;
}
@SuppressWarnings("unchecked")
private Set<String> getRequiredParameters(ToolCallback callback) {
Map<String, Object> schema = JsonUtil.fromJson(callback.getToolDefinition().inputSchema(), Map.class);
if (schema == null || !(schema.get("required") instanceof List<?> required)) {
return Set.of();
}
return required.stream()
.map(String::valueOf)
.collect(Collectors.toUnmodifiableSet());
}
/**
* Invoke a tool by name with the given arguments.
* Invokes a tool with the provided arguments.
*/
public String invoke(String toolName, Map<String, Object> args) {
ensureInitialized();
ToolMethod toolMethod = tools.get(toolName);
if (toolMethod == null) {
throw new IllegalArgumentException("Unknown tool: " + toolName
RegisteredTool tool = tools.get(toolName);
if (tool == null) {
throw new IllegalArgumentException("Unknown tool: " + toolName
+ ". Available tools: " + tools.keySet());
}
return toolMethod.invoke(args);
Map<String, Object> safeArgs = args == null ? Map.of() : args;
validateRequiredParameters(tool.requiredParameters(), safeArgs);
String result = tool.callback().call(JsonUtil.toJson(safeArgs));
Object convertedResult = JsonUtil.fromJson(result, Object.class);
return convertedResult instanceof String text ? text : result;
}
private void validateRequiredParameters(Set<String> requiredParameters, Map<String, Object> args) {
for (String parameter : requiredParameters) {
Object value = args.get(parameter);
if (value == null || value instanceof String text && text.isBlank()) {
throw new IllegalArgumentException("Required tool parameter is missing: " + parameter);
}
}
}
/**
* Check if a tool exists.
* Checks whether a tool exists.
*/
public boolean hasMethod(String toolName) {
ensureInitialized();
@@ -125,158 +148,13 @@ public class ToolRegistry {
}
/**
* Get all registered tool names.
* Returns an immutable snapshot of all registered tool names.
*/
public Set<String> getToolNames() {
ensureInitialized();
return tools.keySet();
return Set.copyOf(tools.keySet());
}
/**
* Get tool method info.
*/
public ToolMethod getToolMethod(String toolName) {
ensureInitialized();
return tools.get(toolName);
}
/**
* Represents a registered tool method.
*/
public static class ToolMethod {
private final Object bean;
private final Method method;
private final Tool annotation;
private final List<ParamInfo> paramInfos;
public ToolMethod(Object bean, Method method, Tool annotation) {
this.bean = bean;
this.method = method;
this.annotation = annotation;
this.paramInfos = extractParamInfos(method);
}
private List<ParamInfo> extractParamInfos(Method method) {
List<ParamInfo> infos = new ArrayList<>();
Parameter[] parameters = method.getParameters();
for (Parameter param : parameters) {
ToolParam toolParam = param.getAnnotation(ToolParam.class);
String name = (toolParam != null && !toolParam.description().isEmpty())
? param.getName() : param.getName();
boolean required = toolParam != null && toolParam.required();
infos.add(new ParamInfo(name, param.getType(), required));
}
return infos;
}
/**
* Invoke this tool method with the given arguments.
*/
public String invoke(Map<String, Object> args) {
try {
Object[] methodArgs = new Object[paramInfos.size()];
for (int i = 0; i < paramInfos.size(); i++) {
ParamInfo paramInfo = paramInfos.get(i);
Object value = args.get(paramInfo.name);
methodArgs[i] = convertValue(value, paramInfo.type);
}
Object result = method.invoke(bean, methodArgs);
return result != null ? result.toString() : "";
} catch (Exception e) {
log.error("Failed to invoke tool {}: {}", annotation.name(), e.getMessage(), e);
throw new RuntimeException("Tool invocation failed: " + annotation.name(), e);
}
}
private Object convertValue(Object value, Class<?> targetType) {
if (value == null) {
return null;
}
if (targetType.isAssignableFrom(value.getClass())) {
return value;
}
String strValue = String.valueOf(value);
if (targetType == String.class) {
return strValue;
} else if (targetType == Integer.class || targetType == int.class) {
return strValue.isEmpty() ? null : Integer.valueOf(strValue);
} else if (targetType == Long.class || targetType == long.class) {
return strValue.isEmpty() ? null : Long.valueOf(strValue);
} else if (targetType == Boolean.class || targetType == boolean.class) {
return Boolean.valueOf(strValue);
} else if (targetType == Byte.class || targetType == byte.class) {
return strValue.isEmpty() ? null : Byte.valueOf(strValue);
} else if (targetType == Double.class || targetType == double.class) {
return strValue.isEmpty() ? null : Double.valueOf(strValue);
} else if (targetType == Float.class || targetType == float.class) {
return strValue.isEmpty() ? null : Float.valueOf(strValue);
} else if (targetType == List.class) {
return parseList(strValue);
}
return value;
}
@SuppressWarnings("unchecked")
private List<Long> parseList(String value) {
if (value == null || value.isEmpty()) {
return new ArrayList<>();
}
List<Long> result = new ArrayList<>();
for (String s : value.split(",")) {
if (!s.trim().isEmpty()) {
result.add(Long.valueOf(s.trim()));
}
}
return result;
}
public String getName() {
return annotation.name();
}
public String getDescription() {
return annotation.description();
}
public List<ParamInfo> getParamInfos() {
return paramInfos;
}
}
/**
* Parameter information for a tool method.
*/
public static class ParamInfo {
private final String name;
private final Class<?> type;
private final boolean required;
public ParamInfo(String name, Class<?> type, boolean required) {
this.name = name;
this.type = type;
this.required = required;
}
public String getName() {
return name;
}
public Class<?> getType() {
return type;
}
public boolean isRequired() {
return required;
}
private record RegisteredTool(ToolCallback callback, Set<String> requiredParameters) {
}
}
@@ -27,8 +27,12 @@ import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hertzbeat.manager.pojo.dto.Hierarchy;
import tools.jackson.databind.ObjectMapper;
@@ -42,6 +46,14 @@ import tools.jackson.databind.ObjectMapper;
@lombok.experimental.UtilityClass
public class UtilityClass {
private static final Pattern SINGLE_EQUALS_PATTERN = Pattern.compile("(?<![<>=!])=(?!=)");
private static final Pattern UPPERCASE_LOGICAL_PATTERN = Pattern.compile("\\b(AND|OR)\\b");
private static final Pattern FUNCTION_FIELD_PATTERN = Pattern.compile(
"!?\\b(?:equals|contains|matches|exists)\\s*\\(\\s*([a-zA-Z_][a-zA-Z0-9_]*)");
private static final Pattern COMPARISON_FIELD_PATTERN = Pattern.compile(
"(?:^|[\\s(])([a-zA-Z_][a-zA-Z0-9_]*)\\s*(?:>=|<=|==|!=|>|<)");
private static final Pattern QUOTED_VALUE_PATTERN = Pattern.compile("\"[^\"]*\"|'[^']*'");
/**
* Validates the syntax of field conditions expression
* @param fieldConditions Field conditions string to validate
@@ -111,31 +123,26 @@ public class UtilityClass {
* Validates that only supported operators are used
*/
public String validateOperators(String fieldConditions) {
// Define supported operators for different field types
String[] numericOperators = {">", "<", ">=", "<=", "==", "!=", "exists()", "!exists()"};
String[] stringOperators = {"equals(", "contains(", "matches(", "exists()", "!equals(", "!contains(", "!matches(", "!exists()"};
String[] logicalOperators = {" and ", " or "};
// Remove quotes and function calls temporarily for operator checking
String tempExpression = fieldConditions
.replaceAll("\"[^\"]*\"", "VALUE") // Remove quoted strings
.replaceAll("'[^']*'", "VALUE") // Remove single quoted strings
.replaceAll("\\w+\\([^)]*\\)", "FUNCTION"); // Remove function calls
// Check for invalid operators (common mistakes)
String[] invalidOperators = {"&&", "||", "AND", "OR", "=", "!="};
for (String invalidOp : invalidOperators) {
if (tempExpression.contains(invalidOp)) {
if (invalidOp.equals("&&") || invalidOp.equals("||")) {
return String.format("Error: Use 'and'/'or' instead of '%s' for logical operations", invalidOp);
}
if (invalidOp.equals("AND") || invalidOp.equals("OR")) {
return String.format("Error: Use lowercase '%s' for logical operations", invalidOp.toLowerCase());
}
if (invalidOp.equals("=")) {
return "Error: Use '==' for equality comparison, not '='";
}
}
if (tempExpression.contains("&&") || tempExpression.contains("||")) {
String invalidOp = tempExpression.contains("&&") ? "&&" : "||";
return String.format("Error: Use 'and'/'or' instead of '%s' for logical operations", invalidOp);
}
Matcher uppercaseLogicalMatcher = UPPERCASE_LOGICAL_PATTERN.matcher(tempExpression);
if (uppercaseLogicalMatcher.find()) {
return String.format("Error: Use lowercase '%s' for logical operations",
uppercaseLogicalMatcher.group(1).toLowerCase());
}
// Reject only a standalone equals sign without rejecting >=, <=, ==, or !=.
if (SINGLE_EQUALS_PATTERN.matcher(tempExpression).find()) {
return "Error: Use '==' for equality comparison, not '='";
}
// Check for unsupported special characters that might indicate syntax errors
@@ -150,11 +157,6 @@ public class UtilityClass {
* Validates logical connectors syntax
*/
public String validateLogicalConnectors(String fieldConditions) {
// Check for proper spacing around logical operators
if (fieldConditions.matches(".*(\\S(and|or)\\S).*")) {
return "Error: Logical operators 'and'/'or' must be surrounded by spaces";
}
// Check for consecutive logical operators
if (fieldConditions.matches(".*(and\\s+and|or\\s+or|and\\s+or\\s+and|or\\s+and\\s+or).*")) {
return "Error: Consecutive logical operators found. Use parentheses to group conditions properly.";
@@ -237,7 +239,7 @@ public class UtilityClass {
String[] pairs = input.split(",");
for (String pair : pairs) {
String[] keyValue = pair.split(":");
String[] keyValue = pair.split(":", 2);
if (keyValue.length == 2) {
result.put(keyValue[0].trim(), keyValue[1].trim());
}
@@ -299,34 +301,22 @@ public class UtilityClass {
* Handles simple cases like "field > 80", "equals(field, 'value')", complex expressions
*/
public List<String> extractFieldNamesFromConditions(String fieldConditions) {
List<String> fieldNames = new ArrayList<>();
Set<String> fieldNames = new LinkedHashSet<>();
// Split by logical operators (and, or) and parentheses, but preserve the field names
// This is a simple implementation - could be enhanced with a proper parser
String[] parts = fieldConditions.split("\\s+(and|or|&&|\\|\\|)\\s+|[()]+");
for (String part : parts) {
part = part.trim();
if (part.isEmpty()) {
continue;
}
// Handle equals() function: equals(fieldName, "value")
if (part.contains("equals(")) {
String fieldName = extractFieldFromEquals(part);
if (fieldName != null && !fieldNames.contains(fieldName)) {
fieldNames.add(fieldName);
}
} else {
// Handle simple comparisons: fieldName > value, fieldName <= value
String fieldName = extractFieldFromComparison(part);
if (fieldName != null && !fieldNames.contains(fieldName)) {
fieldNames.add(fieldName);
}
}
// Extract the first function argument without splitting parentheses and bypassing field validation.
Matcher functionMatcher = FUNCTION_FIELD_PATTERN.matcher(fieldConditions);
while (functionMatcher.find()) {
fieldNames.add(functionMatcher.group(1));
}
return fieldNames;
// Remove quoted values before scanning comparisons to avoid treating text such as "value > 1" as a field.
Matcher comparisonMatcher = COMPARISON_FIELD_PATTERN.matcher(
QUOTED_VALUE_PATTERN.matcher(fieldConditions).replaceAll(""));
while (comparisonMatcher.find()) {
fieldNames.add(comparisonMatcher.group(1));
}
return new ArrayList<>(fieldNames);
}
/**
@@ -391,7 +381,8 @@ public class UtilityClass {
} else {
// Category, app, or metric node
// Determine node type based on children
boolean hasLeafChildren = hierarchy.getChildren().stream()
List<Hierarchy> children = hierarchy.getChildren();
boolean hasLeafChildren = children != null && children.stream()
.anyMatch(child -> child.getIsLeaf() != null && child.getIsLeaf());
if (hasLeafChildren) {
@@ -402,9 +393,9 @@ public class UtilityClass {
node.put("description", "Application with available metrics");
}
if (hierarchy.getChildren() != null && !hierarchy.getChildren().isEmpty()) {
if (children != null && !children.isEmpty()) {
ArrayNode childrenArray = mapper.createArrayNode();
for (Hierarchy child : hierarchy.getChildren()) {
for (Hierarchy child : children) {
childrenArray.add(formatHierarchyAsJson(mapper, child));
}
node.set("children", childrenArray);
@@ -13,23 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# SOP Result Messages - Default (Chinese)
sop.result.title=SOP执行结果
sop.result.name=SOP名称
@@ -13,23 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# SOP Result Messages - English
sop.result.title=SOP Execution Result
sop.result.name=SOP Name
@@ -13,23 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# SOP Result Messages - Chinese
sop.result.title=SOP执行结果
sop.result.name=SOP名称
@@ -65,12 +65,16 @@ POST /api/ai/sop/execute/{skillName}/ai
```
YAML 定义 → SkillRegistry → SopEngine → Executors → SopResult
ToolExecutor / LlmExecutor
SopToolCallback ToolExecutor / LlmExecutor
ToolRegistry(自动发现 @Tool 方法)
```
所有执行入口都由 `SopEngine` 统一补齐默认参数并校验必填参数。技能注册为 Spring AI
工具时,`SopToolCallback` 会根据参数定义生成 JSON Schema、解析模型传入的 JSON 对象,
同步执行 SOP 并返回 `SopResult`;调用方不应在控制器或工具外层重复实现参数校验。
## 添加新工具
只需添加 `@Tool` 注解,无需修改其他代码:
@@ -13,23 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
name: daily_inspection
description: "Execute daily health inspection on all monitors and generate a comprehensive report"
version: "1.0"
@@ -13,23 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
name: mysql_slow_query_diagnosis
description: "Diagnose MySQL slow queries and provide optimization recommendations"
version: "1.0"
@@ -0,0 +1,146 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.config;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.Mockito.mock;
import com.usthe.sureness.subject.SubjectSum;
import com.usthe.sureness.util.SurenessContextHolder;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.model.ToolContext;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
/**
* Verifies the security context scope used when model tools execute across threads.
*/
class McpContextHolderTest {
@AfterEach
void clearContext() {
McpContextHolder.clear();
SurenessContextHolder.clear();
}
@Test
void callWithSubjectShouldBindAndClearBothContexts() {
SubjectSum subject = mock(SubjectSum.class);
String result = McpContextHolder.callWithSubject(subject, () -> {
assertSame(subject, McpContextHolder.getSubject());
assertSame(subject, SurenessContextHolder.getBindSubject());
return "result";
});
assertEquals("result", result);
assertNull(McpContextHolder.getSubject());
assertNull(SurenessContextHolder.getBindSubject());
}
@Test
void callWithSubjectShouldRestoreIndependentPreviousContexts() {
SubjectSum previousMcpSubject = mock(SubjectSum.class);
SubjectSum previousSurenessSubject = mock(SubjectSum.class);
SubjectSum currentSubject = mock(SubjectSum.class);
McpContextHolder.setSubject(previousMcpSubject);
SurenessContextHolder.bindSubject(previousSurenessSubject);
McpContextHolder.callWithSubject(currentSubject, () -> {
assertSame(currentSubject, McpContextHolder.getSubject());
assertSame(currentSubject, SurenessContextHolder.getBindSubject());
return null;
});
assertSame(previousMcpSubject, McpContextHolder.getSubject());
assertSame(previousSurenessSubject, SurenessContextHolder.getBindSubject());
}
@Test
void callWithSubjectShouldRestoreContextsAfterFailure() {
SubjectSum previousSubject = mock(SubjectSum.class);
SubjectSum currentSubject = mock(SubjectSum.class);
McpContextHolder.setSubject(previousSubject);
SurenessContextHolder.bindSubject(previousSubject);
assertThrows(IllegalStateException.class, () ->
McpContextHolder.callWithSubject(currentSubject, () -> {
throw new IllegalStateException("tool failed");
}));
assertSame(previousSubject, McpContextHolder.getSubject());
assertSame(previousSubject, SurenessContextHolder.getBindSubject());
}
@Test
void toolContextShouldCarryOnlyValidSubject() {
SubjectSum subject = mock(SubjectSum.class);
ToolContext toolContext = new ToolContext(McpContextHolder.createToolContext(subject));
assertSame(subject, McpContextHolder.getSubject(toolContext));
assertNull(McpContextHolder.getSubject(null));
assertNull(McpContextHolder.getSubject(new ToolContext(
java.util.Map.of(McpContextHolder.SUBJECT_CONTEXT_KEY, "invalid"))));
assertEquals(java.util.Map.of(), McpContextHolder.createToolContext(null));
}
@Test
void callbackShouldExposeSubjectOnlyDuringDelegateCall() {
SubjectSum subject = mock(SubjectSum.class);
ToolContext toolContext = new ToolContext(McpContextHolder.createToolContext(subject));
ToolCallback delegate = new ToolCallback() {
@Override
public ToolDefinition getToolDefinition() {
return ToolDefinition.builder()
.name("test")
.description("test")
.inputSchema("{}")
.build();
}
@Override
public String call(String input) {
return input;
}
@Override
public String call(String input, ToolContext context) {
if (context == null) {
return input;
}
assertSame(subject, McpContextHolder.getSubject());
assertSame(subject, SurenessContextHolder.getBindSubject());
return "ok";
}
};
SecurityContextToolCallback callback = new SecurityContextToolCallback(delegate);
assertEquals("test", callback.getToolDefinition().name());
assertEquals(delegate.getToolMetadata().returnDirect(), callback.getToolMetadata().returnDirect());
assertEquals("plain", callback.call("plain"));
String result = callback.call("{}", toolContext);
assertEquals("ok", result);
assertNull(McpContextHolder.getSubject());
assertNull(SurenessContextHolder.getBindSubject());
}
}
@@ -0,0 +1,111 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.schedule;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyMap;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
import static org.mockito.Mockito.when;
import java.util.List;
import org.apache.hertzbeat.ai.dao.ChatMessageDao;
import org.apache.hertzbeat.ai.service.SopScheduleService;
import org.apache.hertzbeat.ai.sop.engine.SopEngine;
import org.apache.hertzbeat.ai.sop.model.SopDefinition;
import org.apache.hertzbeat.ai.sop.model.SopResult;
import org.apache.hertzbeat.ai.sop.registry.SkillRegistry;
import org.apache.hertzbeat.common.entity.ai.ChatMessage;
import org.apache.hertzbeat.common.entity.ai.SopSchedule;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
/**
* Verifies that due SOP schedules are isolated from each other and reject invalid parameters.
*/
@ExtendWith(MockitoExtension.class)
class SopScheduleExecutorTest {
@Mock
private SopScheduleService scheduleService;
@Mock
private SopEngine sopEngine;
@Mock
private SkillRegistry skillRegistry;
@Mock
private ChatMessageDao chatMessageDao;
private SopScheduleExecutor executor;
@BeforeEach
void setUp() {
executor = new SopScheduleExecutor(scheduleService, sopEngine, skillRegistry, chatMessageDao);
}
@Test
void checkShouldContinueAfterOneScheduleFailsToUpdate() {
SopSchedule first = schedule(1L, null);
SopSchedule second = schedule(2L, null);
SopDefinition definition = SopDefinition.builder().name("daily_inspection").build();
SopResult result = SopResult.builder()
.status("SUCCESS")
.content("ok")
.build();
when(scheduleService.getDueSchedules()).thenReturn(List.of(first, second));
when(skillRegistry.getSkill("daily_inspection")).thenReturn(definition);
when(sopEngine.executeSync(any(SopDefinition.class), anyMap())).thenReturn(result);
doThrow(new IllegalStateException("database unavailable"))
.when(scheduleService).updateAfterExecution(1L);
executor.checkAndExecuteDueSchedules();
verify(sopEngine, times(2)).executeSync(any(SopDefinition.class), anyMap());
verify(scheduleService).updateAfterExecution(2L);
}
@Test
void checkShouldRejectInvalidScheduleParameters() {
SopSchedule schedule = schedule(1L, "not-json");
when(scheduleService.getDueSchedules()).thenReturn(List.of(schedule));
when(skillRegistry.getSkill("daily_inspection"))
.thenReturn(SopDefinition.builder().name("daily_inspection").build());
executor.checkAndExecuteDueSchedules();
verifyNoInteractions(sopEngine);
verify(chatMessageDao).save(any(ChatMessage.class));
verify(scheduleService).updateAfterExecution(1L);
}
private SopSchedule schedule(Long id, String params) {
return SopSchedule.builder()
.id(id)
.conversationId(10L)
.sopName("daily_inspection")
.sopParams(params)
.build();
}
}
@@ -0,0 +1,141 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.service.impl;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.anyMap;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import com.usthe.sureness.subject.SubjectSum;
import java.lang.reflect.Proxy;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.hertzbeat.ai.config.McpContextHolder;
import org.apache.hertzbeat.ai.config.SecurityContextToolCallback;
import org.apache.hertzbeat.ai.pojo.dto.ChatRequestContext;
import org.apache.hertzbeat.ai.sop.registry.SkillRegistry;
import org.apache.hertzbeat.base.dao.GeneralConfigDao;
import org.apache.hertzbeat.common.entity.dto.ModelProviderConfig;
import org.apache.hertzbeat.common.entity.manager.GeneralConfig;
import org.apache.hertzbeat.common.support.event.AiProviderConfigChangeEvent;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.StaticApplicationContext;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.test.util.ReflectionTestUtils;
import reactor.core.publisher.Flux;
/**
* Verifies that the provider configuration cache reacts to enable and disable events.
*/
class ChatClientProviderServiceImplTest {
@Test
void configurationChangeShouldRefreshConfiguredState() {
AtomicReference<GeneralConfig> currentConfig = new AtomicReference<>();
GeneralConfigDao configDao = configDao(currentConfig);
ChatClientProviderServiceImpl service = new ChatClientProviderServiceImpl(null, configDao, null);
assertFalse(service.isConfigured());
currentConfig.set(providerConfig("sk-test"));
service.onAiProviderConfigChange(changeEvent());
assertTrue(service.isConfigured());
currentConfig.set(providerConfig(" "));
service.onAiProviderConfigChange(changeEvent());
assertFalse(service.isConfigured());
}
@Test
@SuppressWarnings("unchecked")
void streamChatShouldAttachSubjectAndWrapEveryToolCallback() {
ApplicationContext applicationContext = mock(ApplicationContext.class);
SkillRegistry skillRegistry = mock(SkillRegistry.class);
ChatClient chatClient = mock(ChatClient.class);
ChatClient.ChatClientRequestSpec requestSpec = mock(ChatClient.ChatClientRequestSpec.class);
ChatClient.StreamResponseSpec streamSpec = mock(ChatClient.StreamResponseSpec.class);
ToolCallback delegate = mock(ToolCallback.class);
SubjectSum subject = mock(SubjectSum.class);
ChatClientProviderServiceImpl service = new ChatClientProviderServiceImpl(
applicationContext, configDao(new AtomicReference<>()), skillRegistry);
when(applicationContext.getBean("openAiChatClient", ChatClient.class)).thenReturn(chatClient);
when(chatClient.prompt()).thenReturn(requestSpec);
when(requestSpec.messages(anyList())).thenReturn(requestSpec);
when(requestSpec.system(anyString())).thenReturn(requestSpec);
when(requestSpec.tools(any(Object[].class))).thenReturn(requestSpec);
when(requestSpec.toolContext(anyMap())).thenReturn(requestSpec);
when(requestSpec.stream()).thenReturn(streamSpec);
when(streamSpec.content()).thenReturn(Flux.just("answer"));
when(skillRegistry.getAllSkills()).thenReturn(java.util.List.of());
ReflectionTestUtils.setField(service, "systemResource", new ByteArrayResource(
"skills={dynamically_injected_skills_list}; conversation={current_conversation_id}"
.getBytes(StandardCharsets.UTF_8)));
ReflectionTestUtils.setField(service, "toolCallbackProvider", ToolCallbackProvider.from(delegate));
ChatRequestContext context = ChatRequestContext.builder()
.message("question")
.conversationId(42L)
.subject(subject)
.build();
service.streamChat(context).collectList().block();
ArgumentCaptor<Object[]> callbacksCaptor = ArgumentCaptor.forClass(Object[].class);
ArgumentCaptor<Map<String, Object>> contextCaptor = ArgumentCaptor.forClass(Map.class);
verify(requestSpec).tools(callbacksCaptor.capture());
verify(requestSpec).toolContext(contextCaptor.capture());
assertInstanceOf(SecurityContextToolCallback.class, callbacksCaptor.getValue()[0]);
assertSame(subject, McpContextHolder.getSubject(new org.springframework.ai.chat.model.ToolContext(
contextCaptor.getValue())));
}
private GeneralConfigDao configDao(AtomicReference<GeneralConfig> currentConfig) {
return (GeneralConfigDao) Proxy.newProxyInstance(
GeneralConfigDao.class.getClassLoader(),
new Class<?>[]{GeneralConfigDao.class},
(proxy, method, args) -> "findByType".equals(method.getName()) ? currentConfig.get() : null);
}
private GeneralConfig providerConfig(String apiKey) {
ModelProviderConfig modelConfig = new ModelProviderConfig();
modelConfig.setApiKey(apiKey);
return GeneralConfig.builder()
.type("provider")
.content(JsonUtil.toJson(modelConfig))
.build();
}
private AiProviderConfigChangeEvent changeEvent() {
return new AiProviderConfigChangeEvent(new StaticApplicationContext());
}
}
@@ -20,23 +20,29 @@ package org.apache.hertzbeat.ai.service.impl;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import com.usthe.sureness.subject.SubjectSum;
import com.usthe.sureness.util.SurenessContextHolder;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.hertzbeat.ai.dao.ChatConversationDao;
import org.apache.hertzbeat.ai.dao.ChatMessageDao;
import org.apache.hertzbeat.ai.dao.SopScheduleDao;
import org.apache.hertzbeat.ai.pojo.dto.ChatRequestContext;
import org.apache.hertzbeat.ai.pojo.dto.ChatResponseChunk;
import org.apache.hertzbeat.ai.service.ChatClientProviderService;
import org.apache.hertzbeat.common.entity.ai.ChatConversation;
import org.apache.hertzbeat.common.entity.ai.ChatMessage;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.InjectMocks;
import org.mockito.InOrder;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.codec.ServerSentEvent;
@@ -56,14 +62,24 @@ class ConversationServiceImplTest {
@Mock
private ChatMessageDao messageDao;
@Mock
private SopScheduleDao sopScheduleDao;
@Mock
private ChatClientProviderService chatClientProviderService;
@InjectMocks
private ConversationServiceImpl conversationService;
@AfterEach
void clearSecurityContext() {
SurenessContextHolder.clear();
}
@Test
void streamChatShouldKeepCompleteConversationHistory() {
SubjectSum subject = org.mockito.Mockito.mock(SubjectSum.class);
SurenessContextHolder.bindSubject(subject);
ChatConversation conversation = ChatConversation.builder()
.id(CONVERSATION_ID)
.title("已命名会话")
@@ -104,5 +120,28 @@ class ConversationServiceImplTest {
ArgumentCaptor<ChatRequestContext> contextCaptor = ArgumentCaptor.forClass(ChatRequestContext.class);
verify(chatClientProviderService).streamChat(contextCaptor.capture());
assertEquals(history, contextCaptor.getValue().getConversationHistory());
assertEquals(subject, contextCaptor.getValue().getSubject());
}
/**
* Deleting a conversation must remove its schedules before they can push more messages.
*/
@Test
void deleteConversationShouldRemoveSchedulesMessagesAndConversationInOrder() {
ChatMessage message = ChatMessage.builder()
.id(11L)
.conversationId(CONVERSATION_ID)
.role("user")
.content("message to delete")
.build();
when(messageDao.findByConversationIdOrderByGmtCreateAsc(CONVERSATION_ID))
.thenReturn(List.of(message));
conversationService.deleteConversation(CONVERSATION_ID);
InOrder deletionOrder = inOrder(sopScheduleDao, messageDao, conversationDao);
deletionOrder.verify(sopScheduleDao).deleteByConversationId(CONVERSATION_ID);
deletionOrder.verify(messageDao).deleteAll(List.of(message));
deletionOrder.verify(conversationDao).deleteById(CONVERSATION_ID);
}
}
@@ -0,0 +1,125 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.sop.engine;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.hertzbeat.ai.sop.executor.SopExecutor;
import org.apache.hertzbeat.ai.sop.model.SopDefinition;
import org.apache.hertzbeat.ai.sop.model.SopParameter;
import org.apache.hertzbeat.ai.sop.model.SopResult;
import org.apache.hertzbeat.ai.sop.model.SopStep;
import org.junit.jupiter.api.Test;
/**
* Verifies that all SOP execution entry points follow the same parameter contract.
*/
class SopEngineImplTest {
@Test
void executeSyncShouldReturnFailureForNullDefinition() {
SopEngineImpl engine = new SopEngineImpl(List.of());
SopResult result = engine.executeSync(null, Map.of());
assertEquals("FAILED", result.getStatus());
assertTrue(result.getError().contains("must not be null"));
}
@Test
void executeSyncShouldRejectMissingRequiredParameter() {
SopEngineImpl engine = new SopEngineImpl(List.of(new RecordingExecutor()));
SopResult result = engine.executeSync(definition(requiredParameter()), Map.of());
assertEquals("FAILED", result.getStatus());
assertTrue(result.getError().contains("monitorId"));
assertTrue(result.getSteps().isEmpty());
}
@Test
void executeSyncShouldApplyDefaultParameter() {
AtomicReference<Map<String, Object>> capturedContext = new AtomicReference<>();
SopEngineImpl engine = new SopEngineImpl(List.of(new RecordingExecutor(capturedContext)));
SopResult result = engine.executeSync(definition(defaultParameter()), Map.of());
assertEquals("SUCCESS", result.getStatus());
assertEquals("10", capturedContext.get().get("limit"));
assertEquals("zh", capturedContext.get().get("_language"));
assertEquals("10", result.getData().get("limit"));
}
@Test
void executeStreamShouldApplyTheSameParameterRules() {
AtomicReference<Map<String, Object>> capturedContext = new AtomicReference<>();
SopEngineImpl engine = new SopEngineImpl(List.of(new RecordingExecutor(capturedContext)));
List<String> events = engine.execute(definition(defaultParameter()), null).collectList().block();
assertNotNull(events);
assertTrue(events.getLast().contains("completed successfully"));
assertEquals("10", capturedContext.get().get("limit"));
}
private SopDefinition definition(SopParameter parameter) {
return SopDefinition.builder()
.name("test-sop")
.version("1.0")
.parameters(List.of(parameter))
.steps(List.of(SopStep.builder().id("query").type("tool").build()))
.build();
}
private SopParameter requiredParameter() {
return SopParameter.builder().name("monitorId").type("long").required(true).build();
}
private SopParameter defaultParameter() {
return SopParameter.builder().name("limit").type("integer").defaultValue("10").build();
}
private static final class RecordingExecutor implements SopExecutor {
private final AtomicReference<Map<String, Object>> capturedContext;
private RecordingExecutor() {
this(new AtomicReference<>());
}
private RecordingExecutor(AtomicReference<Map<String, Object>> capturedContext) {
this.capturedContext = capturedContext;
}
@Override
public boolean support(String type) {
return "tool".equals(type);
}
@Override
public Object execute(SopStep step, Map<String, Object> context) {
capturedContext.set(Map.copyOf(context));
return "ok";
}
}
}
@@ -0,0 +1,136 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.sop.registry;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.hertzbeat.ai.sop.engine.SopEngine;
import org.apache.hertzbeat.ai.sop.model.OutputType;
import org.apache.hertzbeat.ai.sop.model.SopDefinition;
import org.apache.hertzbeat.ai.sop.model.SopParameter;
import org.apache.hertzbeat.ai.sop.model.SopResult;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.junit.jupiter.api.Test;
import reactor.core.publisher.Flux;
/**
* Verifies the input contract and execution behavior of a SOP exposed as a Spring AI tool.
*/
class SopToolCallbackTest {
@Test
@SuppressWarnings("unchecked")
void schemaShouldDeclareRequiredParametersAndEscapeDescriptions() {
SopDefinition definition = definition();
SopToolCallback callback = new SopToolCallback(definition, new RecordingEngine());
Map<String, Object> schema = JsonUtil.fromJson(callback.getToolDefinition().inputSchema(), Map.class);
Map<String, Object> properties = (Map<String, Object>) schema.get("properties");
Map<String, Object> monitorId = (Map<String, Object>) properties.get("monitorId");
Map<String, Object> limit = (Map<String, Object>) properties.get("limit");
assertEquals(List.of("monitorId"), schema.get("required"));
assertEquals(false, schema.get("additionalProperties"));
assertEquals("integer", monitorId.get("type"));
assertEquals("包含 \"引号\" 的说明", monitorId.get("description"));
assertEquals(10, limit.get("default"));
}
@Test
void callShouldParseArgumentsAndExecuteSop() {
RecordingEngine engine = new RecordingEngine();
SopToolCallback callback = new SopToolCallback(definition(), engine);
String response = callback.call("{\"monitorId\":42}");
assertEquals(42, engine.inputParams.get().get("monitorId"));
assertTrue(response.contains("SUCCESS"));
assertFalse(response.contains("execution started"));
}
@Test
void callShouldRejectInvalidJson() {
SopToolCallback callback = new SopToolCallback(definition(), new RecordingEngine());
assertThrows(IllegalArgumentException.class, () -> callback.call("not-json"));
}
@Test
void schemaShouldRejectInvalidBooleanDefault() {
SopParameter enabled = SopParameter.builder()
.name("enabled")
.type("boolean")
.defaultValue("yes")
.build();
SopDefinition definition = SopDefinition.builder()
.name("invalid-default")
.description("包含非法布尔默认值")
.parameters(List.of(enabled))
.build();
assertThrows(IllegalArgumentException.class,
() -> new SopToolCallback(definition, new RecordingEngine()));
}
private SopDefinition definition() {
SopParameter monitorId = SopParameter.builder()
.name("monitorId")
.type("long")
.description("包含 \"引号\" 的说明")
.required(true)
.build();
SopParameter limit = SopParameter.builder()
.name("limit")
.type("integer")
.defaultValue("10")
.build();
return SopDefinition.builder()
.name("diagnose")
.description("诊断工具")
.version("1.0")
.parameters(List.of(monitorId, limit))
.build();
}
private static final class RecordingEngine implements SopEngine {
private final AtomicReference<Map<String, Object>> inputParams = new AtomicReference<>();
@Override
public Flux<String> execute(SopDefinition definition, Map<String, Object> inputParams) {
return Flux.empty();
}
@Override
public SopResult executeSync(SopDefinition definition, Map<String, Object> inputParams) {
this.inputParams.set(inputParams);
return SopResult.builder()
.sopName(definition.getName())
.sopVersion(definition.getVersion())
.status("SUCCESS")
.outputType(OutputType.SIMPLE)
.build();
}
}
}
@@ -0,0 +1,136 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.sop.registry;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.junit.jupiter.api.Test;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.ai.tool.execution.ToolExecutionException;
import org.springframework.context.support.GenericApplicationContext;
/**
* Verifies tool discovery, argument validation, and type conversion by Spring AI callbacks.
*/
class ToolRegistryTest {
@Test
void invokeShouldConvertStructuredArguments() {
try (GenericApplicationContext context = contextWith(ToolFixture.class)) {
ToolRegistry registry = new ToolRegistry(context);
String result = registry.invoke("inspect", Map.of(
"monitorId", 42,
"ids", List.of(1, 2),
"enabled", true));
assertEquals("Long|Long|Boolean", result);
assertTrue(registry.hasMethod("inspect"));
assertEquals(Set.of("inspect", "optional"), registry.getToolNames());
}
}
@Test
void invokeShouldRejectMissingRequiredParameter() {
try (GenericApplicationContext context = contextWith(ToolFixture.class)) {
ToolRegistry registry = new ToolRegistry(context);
IllegalArgumentException error = assertThrows(IllegalArgumentException.class,
() -> registry.invoke("inspect", Map.of("ids", List.of(1), "enabled", true)));
assertEquals("Required tool parameter is missing: monitorId", error.getMessage());
}
}
@Test
void invokeShouldRejectInvalidBooleanInsteadOfSilentlyUsingFalse() {
try (GenericApplicationContext context = contextWith(ToolFixture.class)) {
ToolRegistry registry = new ToolRegistry(context);
assertThrows(ToolExecutionException.class, () -> registry.invoke("inspect", Map.of(
"monitorId", 42,
"ids", List.of(1),
"enabled", "not-a-boolean")));
}
}
@Test
void initializationShouldRejectDuplicateToolNames() {
try (GenericApplicationContext context = contextWith(ToolFixture.class, DuplicateToolFixture.class)) {
ToolRegistry registry = new ToolRegistry(context);
IllegalArgumentException error = assertThrows(
IllegalArgumentException.class, registry::getToolNames);
assertTrue(error.getMessage().contains("inspect"));
}
}
@Test
void emptyRegistryShouldReportUnknownTool() {
try (GenericApplicationContext context = contextWith()) {
ToolRegistry registry = new ToolRegistry(context);
assertTrue(registry.getToolNames().isEmpty());
assertThrows(IllegalArgumentException.class, () -> registry.invoke("missing", null));
}
}
private GenericApplicationContext contextWith(Class<?>... beanTypes) {
GenericApplicationContext context = new GenericApplicationContext();
for (Class<?> beanType : beanTypes) {
context.registerBean(beanType);
}
context.refresh();
return context;
}
public static final class ToolFixture {
@Tool(name = "inspect", description = "Inspect argument types")
public String inspect(
@ToolParam(description = "Monitor ID", required = true) Long monitorId,
@ToolParam(description = "Monitor ID list", required = false) List<Long> ids,
@ToolParam(description = "Whether enabled", required = true) Boolean enabled) {
return "%s|%s|%s".formatted(
monitorId.getClass().getSimpleName(),
ids.getFirst().getClass().getSimpleName(),
enabled.getClass().getSimpleName());
}
@Tool(name = "optional", description = "Tool without arguments")
public String optional() {
return "optional";
}
}
public static final class DuplicateToolFixture {
@Tool(name = "inspect", description = "Duplicate-name test tool")
public String inspect() {
return "duplicate";
}
}
}
@@ -0,0 +1,99 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.utils;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.List;
import java.util.Map;
import org.apache.hertzbeat.manager.pojo.dto.Hierarchy;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import tools.jackson.databind.ObjectMapper;
import tools.jackson.databind.node.ObjectNode;
/**
* Verifies input parsing and hierarchy-data tolerance in the AI alert rule utilities.
*/
class UtilityClassTest {
@ParameterizedTest
@ValueSource(strings = {
"cpu_usage >= 80",
"cpu_usage <= 80",
"cpu_usage == 80",
"cpu_usage != 80"
})
void validateExpressionSyntaxShouldAcceptSupportedComparisonOperators(String expression) {
assertEquals("VALID", UtilityClass.validateExpressionSyntax(expression));
}
@Test
void validateExpressionSyntaxShouldNotTreatLogicalTextInsideFieldNameAsOperator() {
assertEquals("VALID", UtilityClass.validateExpressionSyntax("processor_count > 1"));
}
@Test
void validateExpressionSyntaxShouldStillRejectSingleEquals() {
assertEquals("Error: Use '==' for equality comparison, not '='",
UtilityClass.validateExpressionSyntax("cpu_usage = 80"));
}
@Test
void parseKeyValuePairsShouldPreserveColonsInValue() {
Map<String, String> result = UtilityClass.parseKeyValuePairs(
"runbook:https://example.org:8443/alerts, severity:critical");
assertEquals("https://example.org:8443/alerts", result.get("runbook"));
assertEquals("critical", result.get("severity"));
}
@Test
void extractFieldNamesShouldHandleFunctionsAndComparisons() {
List<String> fields = UtilityClass.extractFieldNamesFromConditions(
"equals(VmName, \"prod\") and contains(host, \"db\") "
+ "and (cpu_usage >= 80 or cpu_usage <= 20)");
assertEquals(List.of("VmName", "host", "cpu_usage"), fields);
}
@Test
void extractFieldNamesShouldIgnoreComparisonTextInsideQuotedValue() {
List<String> fields = UtilityClass.extractFieldNamesFromConditions(
"equals(message, \"fake_field > 1\")");
assertEquals(List.of("message"), fields);
}
@Test
void formatHierarchyAsJsonShouldAcceptNonLeafWithoutChildren() {
Hierarchy hierarchy = new Hierarchy();
hierarchy.setValue("linux");
hierarchy.setLabel("Linux");
hierarchy.setIsLeaf(false);
ObjectNode result = UtilityClass.formatHierarchyAsJson(new ObjectMapper(), hierarchy);
assertEquals("app", result.get("type").asText());
assertFalse(result.has("children"));
assertTrue(result.has("value"));
}
}
@@ -30,6 +30,7 @@ import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.apache.hertzbeat.common.entity.alerter.NoticeRule;
import org.apache.hertzbeat.common.entity.alerter.NoticeTemplate;
import org.apache.hertzbeat.alert.service.NoticeConfigService;
import org.apache.hertzbeat.alert.util.NoticeReceiverMaskUtil;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.http.ResponseEntity;
@@ -87,14 +88,16 @@ public class NoticeConfigController {
@Parameter(description = "en: Recipient name,support fuzzy query", example = "tom") @RequestParam(required = false) final String name,
@Parameter(description = "en: List current page", example = "0") @RequestParam(defaultValue = "0") final int pageIndex,
@Parameter(description = "en: Number of list pages", example = "8") @RequestParam(defaultValue = "8") final int pageSize) {
return ResponseEntity.ok(Message.success(noticeConfigService.getNoticeReceivers(name, pageIndex, pageSize)));
return ResponseEntity.ok(Message.success(noticeConfigService.getNoticeReceivers(name, pageIndex, pageSize)
.map(NoticeReceiverMaskUtil::mask)));
}
@GetMapping(path = "/receivers/all")
@Operation(summary = "Get a list of all message notification recipients",
description = "Get a list of all message notification recipients")
public ResponseEntity<Message<List<NoticeReceiver>>> getAllReceivers() {
return ResponseEntity.ok(Message.success(noticeConfigService.getAllNoticeReceivers()));
return ResponseEntity.ok(Message.success(noticeConfigService.getAllNoticeReceivers().stream()
.map(NoticeReceiverMaskUtil::mask).toList()));
}
@GetMapping(path = "/receiver/{id}")
@@ -106,7 +109,7 @@ public class NoticeConfigController {
if (noticeReceiver == null) {
return ResponseEntity.ok(Message.fail(FAIL_CODE, "The relevant information of the recipient could not be found, please check whether the parameters are correct or refresh the page"));
}
return ResponseEntity.ok(Message.success(noticeReceiver));
return ResponseEntity.ok(Message.success(NoticeReceiverMaskUtil.mask(noticeReceiver)));
}
@PostMapping(path = "/rule")
@@ -17,6 +17,7 @@
package org.apache.hertzbeat.alert.notice.impl;
import java.net.URI;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.alert.notice.AlertNoticeException;
@@ -47,6 +48,15 @@ final class WebHookAlertNotifyHandlerImpl extends AbstractAlertNotifyHandlerImpl
throw new AlertNoticeException("Webhook URL is null or empty");
}
// Send the URL verbatim via the URI overload: the String overload treats it
// as a URI template and re-encodes it, corrupting pre-encoded query params
URI hookUri;
try {
hookUri = URI.create(hookUrl);
} catch (IllegalArgumentException e) {
throw new AlertNoticeException("Invalid webhook URL: " + e.getMessage());
}
HttpHeaders headers = new HttpHeaders();
if ("Basic".equalsIgnoreCase(receiver.getHookAuthType())) {
headers.setBasicAuth(receiver.getHookAuthToken());
@@ -59,7 +69,7 @@ final class WebHookAlertNotifyHandlerImpl extends AbstractAlertNotifyHandlerImpl
webhookJson = webhookJson.replace(",\n }", "\n }");
HttpEntity<String> alertHttpEntity = new HttpEntity<>(webhookJson, headers);
ResponseEntity<String> entity = restTemplate.postForEntity(hookUrl, alertHttpEntity, String.class);
ResponseEntity<String> entity = restTemplate.postForEntity(hookUri, alertHttpEntity, String.class);
if (entity.getStatusCode().value() < HttpStatus.BAD_REQUEST.value()) {
log.debug("Send WebHook: {} Success", hookUrl);
} else {
@@ -31,6 +31,7 @@ import org.apache.hertzbeat.alert.dao.NoticeReceiverDao;
import org.apache.hertzbeat.alert.dao.NoticeRuleDao;
import org.apache.hertzbeat.alert.dao.NoticeTemplateDao;
import org.apache.hertzbeat.alert.service.NoticeConfigService;
import org.apache.hertzbeat.alert.util.NoticeReceiverMaskUtil;
import org.apache.hertzbeat.common.entity.alerter.SingleAlert;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
@@ -176,9 +177,43 @@ public class NoticeConfigServiceImpl implements NoticeConfigService, CommandLine
@Override
public void editReceiver(NoticeReceiver noticeReceiver) {
resolveMaskedSecrets(noticeReceiver);
noticeReceiverDao.save(noticeReceiver);
}
/**
* The rest api returns receivers with masked secret fields, so a receiver submitted
* from the ui may carry the mask placeholder instead of the real secret.
* Restore such fields from the stored entity before using the receiver.
* @throws IllegalArgumentException if the receiver carries an id but no stored receiver
* exists for it: without the stored entity the mask cannot
* be resolved, and saving would persist the placeholder
*/
private void resolveMaskedSecrets(NoticeReceiver noticeReceiver) {
resolveMaskedSecrets(noticeReceiver, false);
}
/**
* Bind stored secrets to their original notification type and destination for test messages.
*/
private void resolveMaskedSecretsForTest(NoticeReceiver noticeReceiver) {
resolveMaskedSecrets(noticeReceiver, true);
}
private void resolveMaskedSecrets(NoticeReceiver noticeReceiver, boolean testMessage) {
if (noticeReceiver == null || noticeReceiver.getId() == null) {
return;
}
NoticeReceiver existing = noticeReceiverDao.findById(noticeReceiver.getId())
.orElseThrow(() -> new IllegalArgumentException(
"The receiver with id " + noticeReceiver.getId() + " does not exist."));
if (testMessage) {
NoticeReceiverMaskUtil.resolveMaskForTest(noticeReceiver, existing);
} else {
NoticeReceiverMaskUtil.resolveMask(noticeReceiver, existing);
}
}
@Override
public void deleteReceiver(Long receiverId) {
noticeReceiverDao.deleteById(receiverId);
@@ -319,6 +354,7 @@ public class NoticeConfigServiceImpl implements NoticeConfigService, CommandLine
@Override
public boolean sendTestMsg(NoticeReceiver noticeReceiver) {
resolveMaskedSecretsForTest(noticeReceiver);
Map<String, String> labels = new HashMap<>(8);
labels.put(CommonConstants.LABEL_INSTANCE, "127.0.0.1");
labels.put(CommonConstants.LABEL_ALERT_NAME, "CPU Usage Alert");
@@ -0,0 +1,184 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.alert.util;
import java.util.List;
import java.util.Objects;
import java.util.function.BiConsumer;
import java.util.function.Function;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
/**
* Masks the secret fields of {@link NoticeReceiver} before it is exposed through the rest api,
* and resolves masked values back to the stored secrets when an edited receiver is submitted.
* A secret long enough keeps its last characters visible so different tokens stay
* distinguishable in the ui, while an unchanged secret round-trips through the ui
* without ever leaving the server.
*/
public final class NoticeReceiverMaskUtil {
/**
* Fixed-length placeholder replacing the hidden part of a secret. Its length is
* constant on purpose so the mask never reveals how long the real secret is.
*/
public static final String SECRET_MASK = "******";
/**
* A secret shorter than this is masked entirely: revealing a suffix of a short
* secret would give away too large a fraction of it.
*/
private static final int MIN_LENGTH_TO_SHOW_SUFFIX = 12;
/**
* Number of trailing characters kept visible for a long secret,
* enough to tell configured tokens apart.
*/
private static final int VISIBLE_SUFFIX_LENGTH = 4;
private record SecretField(Function<NoticeReceiver, String> getter, BiConsumer<NoticeReceiver, String> setter) {
}
private static final List<SecretField> SECRET_FIELDS = List.of(
new SecretField(NoticeReceiver::getHookAuthToken, NoticeReceiver::setHookAuthToken),
new SecretField(NoticeReceiver::getAccessToken, NoticeReceiver::setAccessToken),
new SecretField(NoticeReceiver::getTgBotToken, NoticeReceiver::setTgBotToken),
new SecretField(NoticeReceiver::getSlackWebHookUrl, NoticeReceiver::setSlackWebHookUrl),
new SecretField(NoticeReceiver::getAppSecret, NoticeReceiver::setAppSecret),
new SecretField(NoticeReceiver::getDiscordBotToken, NoticeReceiver::setDiscordBotToken),
new SecretField(NoticeReceiver::getSmnAk, NoticeReceiver::setSmnAk),
new SecretField(NoticeReceiver::getSmnSk, NoticeReceiver::setSmnSk),
new SecretField(NoticeReceiver::getServerChanToken, NoticeReceiver::setServerChanToken),
new SecretField(NoticeReceiver::getGotifyToken, NoticeReceiver::setGotifyToken),
new SecretField(NoticeReceiver::getNtfyToken, NoticeReceiver::setNtfyToken));
private NoticeReceiverMaskUtil() {
}
/**
* Return a copy of the receiver with all secret fields masked.
* The given entity is not modified as it may still be attached to the persistence context.
* @param receiver receiver to mask, may be null
* @return masked copy, or null if receiver is null
*/
public static NoticeReceiver mask(NoticeReceiver receiver) {
if (receiver == null) {
return null;
}
NoticeReceiver masked = receiver.toBuilder().build();
for (SecretField field : SECRET_FIELDS) {
field.setter().accept(masked, maskValue(field.getter().apply(masked)));
}
return masked;
}
/**
* Replace every secret field of the incoming receiver that still holds the masked form
* of the stored secret with the stored value, so an edit that did not touch a secret keeps it.
* A re-entered secret or a cleared field is left untouched.
* @param incoming receiver submitted by the ui, modified in place
* @param existing receiver currently stored in the database
* @throws IllegalArgumentException if a submitted mask does not match the stored secret
*/
public static void resolveMask(NoticeReceiver incoming, NoticeReceiver existing) {
if (incoming == null || existing == null) {
return;
}
for (SecretField field : SECRET_FIELDS) {
String submitted = field.getter().apply(incoming);
String stored = field.getter().apply(existing);
if (isMaskOf(submitted, stored)) {
field.setter().accept(incoming, stored);
} else if (isMaskValue(submitted)) {
throw new IllegalArgumentException(
"The submitted secret mask does not match the stored secret.");
}
}
}
/**
* Resolve masked secrets for a test message while binding them to their persisted destination.
* A caller that changes the notification type or a URL receiving authentication data
* must submit the new secret explicitly instead of reusing a stored secret mask.
* @param incoming receiver submitted for a test message, modified in place
* @param existing receiver currently stored in the database
* @throws IllegalArgumentException if a masked secret is combined with a changed destination
*/
public static void resolveMaskForTest(NoticeReceiver incoming, NoticeReceiver existing) {
if (incoming == null || existing == null) {
return;
}
boolean containsStoredSecretMask = SECRET_FIELDS.stream()
.anyMatch(field -> isMaskOf(field.getter().apply(incoming), field.getter().apply(existing)));
if (containsStoredSecretMask && !Objects.equals(incoming.getType(), existing.getType())) {
throw new IllegalArgumentException(
"The notification type cannot be changed when reusing a masked secret.");
}
rejectChangedSecretDestination(
incoming.getHookAuthToken(),
existing.getHookAuthToken(),
incoming.getHookUrl(),
existing.getHookUrl(),
"webhook URL");
rejectChangedSecretDestination(
incoming.getNtfyToken(),
existing.getNtfyToken(),
incoming.getNtfyServerUrl(),
existing.getNtfyServerUrl(),
"ntfy server URL");
resolveMask(incoming, existing);
}
private static void rejectChangedSecretDestination(
String submittedSecret,
String storedSecret,
String submittedDestination,
String storedDestination,
String destinationName) {
if (isMaskOf(submittedSecret, storedSecret)
&& !Objects.equals(submittedDestination, storedDestination)) {
throw new IllegalArgumentException(
"The " + destinationName + " cannot be changed when reusing a masked secret.");
}
}
private static boolean isMaskOf(String submitted, String stored) {
if (submitted == null || StringUtils.isBlank(stored)) {
return false;
}
return submitted.equals(maskValue(stored));
}
private static boolean isMaskValue(String value) {
return value != null
&& value.startsWith(SECRET_MASK)
&& (value.length() == SECRET_MASK.length()
|| value.length() == SECRET_MASK.length() + VISIBLE_SUFFIX_LENGTH);
}
private static String maskValue(String value) {
if (StringUtils.isBlank(value)) {
return value;
}
if (value.length() < MIN_LENGTH_TO_SHOW_SUFFIX) {
return SECRET_MASK;
}
return SECRET_MASK + value.substring(value.length() - VISIBLE_SUFFIX_LENGTH);
}
}
@@ -31,6 +31,7 @@ import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import org.apache.hertzbeat.alert.service.impl.NoticeConfigServiceImpl;
import org.apache.hertzbeat.alert.util.NoticeReceiverMaskUtil;
import org.apache.hertzbeat.common.constants.CommonConstants;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.apache.hertzbeat.common.entity.alerter.NoticeRule;
@@ -174,6 +175,7 @@ class NoticeConfigControllerTest {
NoticeReceiver receiver1 = new NoticeReceiver();
receiver1.setId(1L);
receiver1.setName("Receiver1");
receiver1.setTgBotToken("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM");
NoticeReceiver receiver2 = new NoticeReceiver();
receiver2.setId(2L);
@@ -197,6 +199,7 @@ class NoticeConfigControllerTest {
.andExpect(status().isOk())
.andExpect(jsonPath("$.data.content[0].id").value(1))
.andExpect(jsonPath("$.data.content[0].name").value("Receiver1"))
.andExpect(jsonPath("$.data.content[0].tgBotToken").value(NoticeReceiverMaskUtil.SECRET_MASK + "voJM"))
.andExpect(jsonPath("$.data.content[1].id").value(2))
.andExpect(jsonPath("$.data.content[1].name").value("Receiver2"))
.andExpect(jsonPath("$.data.totalElements").value(2))
@@ -222,6 +225,8 @@ class NoticeConfigControllerTest {
this.mockMvc.perform(MockMvcRequestBuilders.get("/api/notice/receiver/{id}", 7565463543L))
.andExpect(status().isOk())
.andExpect(jsonPath("$.code").value((int) CommonConstants.SUCCESS_CODE))
.andExpect(jsonPath("$.data.accessToken").value(NoticeReceiverMaskUtil.SECRET_MASK + "739d"))
.andExpect(jsonPath("$.data.email").value("2762242004@qq.com"))
.andReturn();
}
@@ -37,11 +37,17 @@ import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.test.util.ReflectionTestUtils;
import org.springframework.test.web.client.MockRestServiceServer;
import org.springframework.web.client.RestTemplate;
import static org.springframework.test.web.client.match.MockRestRequestMatchers.requestTo;
import static org.springframework.test.web.client.response.MockRestResponseCreators.withSuccess;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.net.URI;
import java.util.ResourceBundle;
/**
@@ -93,7 +99,7 @@ class WebHookAlertNotifyHandlerImplTest {
ResponseEntity<String> responseEntity =
new ResponseEntity<>("null", HttpStatus.OK);
when(restTemplate.postForEntity(any(String.class), any(), eq(String.class))).thenReturn(responseEntity);
when(restTemplate.postForEntity(any(URI.class), any(), eq(String.class))).thenReturn(responseEntity);
webHookAlertNotifyHandler.send(receiver, template, groupAlert);
}
@@ -103,7 +109,7 @@ class WebHookAlertNotifyHandlerImplTest {
ResponseEntity<String> responseEntity =
new ResponseEntity<>("null", HttpStatus.INTERNAL_SERVER_ERROR);
when(restTemplate.postForEntity(any(String.class), any(), eq(String.class))).thenReturn(responseEntity);
when(restTemplate.postForEntity(any(URI.class), any(), eq(String.class))).thenReturn(responseEntity);
assertThrows(AlertNoticeException.class,
@@ -117,11 +123,27 @@ class WebHookAlertNotifyHandlerImplTest {
ResponseEntity<String> responseEntity =
new ResponseEntity<>("null", HttpStatus.OK);
when(restTemplate.postForEntity(eq(receiver.getHookUrl()), any(), eq(String.class))).thenReturn(responseEntity);
when(restTemplate.postForEntity(eq(URI.create(receiver.getHookUrl())), any(), eq(String.class))).thenReturn(responseEntity);
webHookAlertNotifyHandler.send(receiver, template, groupAlert);
}
@Test
public void testHookUrlWithPercentEncodedQuerySentVerbatim() {
String hookUrl = "https://example.environment.api.powerplatform.com/workflows/wf1/triggers/manual/paths/invoke"
+ "?api-version=1&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=UejZsrJyaZwwAm_jArn7Ze0PIf";
receiver.setHookUrl(hookUrl);
RestTemplate realRestTemplate = new RestTemplate();
MockRestServiceServer mockServer = MockRestServiceServer.createServer(realRestTemplate);
mockServer.expect(requestTo(hookUrl)).andRespond(withSuccess());
ReflectionTestUtils.setField(webHookAlertNotifyHandler, "restTemplate", realRestTemplate);
webHookAlertNotifyHandler.send(receiver, template, groupAlert);
mockServer.verify();
}
@Test
public void testNotifyAlertWithNullOrEmptyUrl() {
// Test null URL
@@ -145,7 +167,7 @@ class WebHookAlertNotifyHandlerImplTest {
ResponseEntity<String> responseEntity =
new ResponseEntity<>("null", HttpStatus.OK);
when(restTemplate.postForEntity(any(String.class), any(), eq(String.class))).thenReturn(responseEntity);
when(restTemplate.postForEntity(any(URI.class), any(), eq(String.class))).thenReturn(responseEntity);
// Test various valid URLs that should work
receiver.setHookUrl("https://hooks.slack.com/services/T123/B456/complete-token");
@@ -170,7 +192,7 @@ class WebHookAlertNotifyHandlerImplTest {
ResponseEntity<String> responseEntity =
new ResponseEntity<>("null", HttpStatus.OK);
when(restTemplate.postForEntity(eq(receiver.getHookUrl()), any(), eq(String.class))).thenReturn(responseEntity);
when(restTemplate.postForEntity(eq(URI.create(receiver.getHookUrl())), any(), eq(String.class))).thenReturn(responseEntity);
webHookAlertNotifyHandler.send(receiver, template, groupAlert);
@@ -17,25 +17,6 @@
* under the License.
*/
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.hertzbeat.alert.reduce;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -17,25 +17,6 @@
* under the License.
*/
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.hertzbeat.alert.reduce;
import static org.junit.jupiter.api.Assertions.assertEquals;
@@ -22,6 +22,7 @@ import org.apache.hertzbeat.alert.dao.NoticeRuleDao;
import org.apache.hertzbeat.alert.dao.NoticeTemplateDao;
import org.apache.hertzbeat.alert.notice.AlertNoticeDispatch;
import org.apache.hertzbeat.alert.service.impl.NoticeConfigServiceImpl;
import org.apache.hertzbeat.alert.util.NoticeReceiverMaskUtil;
import org.apache.hertzbeat.common.cache.CacheFactory;
import org.apache.hertzbeat.common.entity.alerter.GroupAlert;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
@@ -48,12 +49,15 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -211,11 +215,105 @@ class NoticeConfigServiceTest {
@Test
void editReceiver() {
final NoticeReceiver noticeReceiver = mock(NoticeReceiver.class);
final NoticeReceiver noticeReceiver = new NoticeReceiver();
noticeReceiver.setId(5L);
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.of(noticeReceiver));
noticeConfigService.editReceiver(noticeReceiver);
verify(noticeReceiverDao, times(1)).save(noticeReceiver);
}
@Test
void editReceiverKeepsStoredSecretWhenMasked() {
final NoticeReceiver stored = new NoticeReceiver();
stored.setId(5L);
stored.setTgBotToken("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM");
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.of(stored));
final NoticeReceiver incoming = new NoticeReceiver();
incoming.setId(5L);
incoming.setTgBotToken(NoticeReceiverMaskUtil.SECRET_MASK + "voJM");
noticeConfigService.editReceiver(incoming);
assertEquals("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM", incoming.getTgBotToken());
verify(noticeReceiverDao, times(1)).save(incoming);
}
@Test
void editReceiverRejectsUnknownId() {
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.empty());
final NoticeReceiver incoming = new NoticeReceiver();
incoming.setId(5L);
incoming.setTgBotToken(NoticeReceiverMaskUtil.SECRET_MASK + "voJM");
assertThrows(IllegalArgumentException.class, () -> noticeConfigService.editReceiver(incoming));
verify(noticeReceiverDao, never()).save(any());
}
@Test
void sendTestMsgResolvesMaskedSecret() {
final NoticeReceiver stored = new NoticeReceiver();
stored.setId(5L);
stored.setTgBotToken("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM");
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.of(stored));
final NoticeReceiver incoming = new NoticeReceiver();
incoming.setId(5L);
incoming.setTgBotToken(NoticeReceiverMaskUtil.SECRET_MASK + "voJM");
noticeConfigService.sendTestMsg(incoming);
assertEquals("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM", incoming.getTgBotToken());
}
@Test
void sendTestMsgRejectsUnknownId() {
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.empty());
final NoticeReceiver incoming = new NoticeReceiver();
incoming.setId(5L);
incoming.setTgBotToken(NoticeReceiverMaskUtil.SECRET_MASK + "voJM");
assertThrows(IllegalArgumentException.class, () -> noticeConfigService.sendTestMsg(incoming));
verify(dispatcherAlarm, never()).sendNoticeMsg(any(), any(), any());
}
@Test
void sendTestMsgRejectsMaskedSecretReplayToChangedWebhookUrl() {
final NoticeReceiver stored = new NoticeReceiver();
stored.setId(5L);
stored.setType((byte) 2);
stored.setHookUrl("https://trusted.example/hook");
stored.setHookAuthToken("hook-auth-token-abcd");
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.of(stored));
final NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(stored);
incoming.setHookUrl("https://attacker.example/collect");
assertThrows(IllegalArgumentException.class, () -> noticeConfigService.sendTestMsg(incoming));
verify(dispatcherAlarm, never()).sendNoticeMsg(any(), any(), any());
}
@Test
void sendTestMsgRejectsBareMaskReplayToChangedWebhookUrl() {
final NoticeReceiver stored = new NoticeReceiver();
stored.setId(5L);
stored.setType((byte) 2);
stored.setHookUrl("https://trusted.example/hook");
stored.setHookAuthToken("hook-auth-token-abcd");
when(noticeReceiverDao.findById(5L)).thenReturn(Optional.of(stored));
final NoticeReceiver incoming = new NoticeReceiver();
incoming.setId(5L);
incoming.setType((byte) 2);
incoming.setHookUrl("https://attacker.example/collect");
incoming.setHookAuthToken(NoticeReceiverMaskUtil.SECRET_MASK);
assertThrows(IllegalArgumentException.class, () -> noticeConfigService.sendTestMsg(incoming));
verify(dispatcherAlarm, never()).sendNoticeMsg(any(), any(), any());
}
@Test
void deleteReceiver() {
final Long receiverId = 23342525L;
@@ -288,7 +386,7 @@ class NoticeConfigServiceTest {
@Test
void sendTestMsg() {
final NoticeReceiver noticeReceiver = mock(NoticeReceiver.class);
final NoticeReceiver noticeReceiver = new NoticeReceiver();
final NoticeTemplate noticeTemplate = null;
noticeConfigService.sendTestMsg(noticeReceiver);
verify(dispatcherAlarm, times(1)).sendNoticeMsg(eq(noticeReceiver), eq(noticeTemplate), any(GroupAlert.class));
@@ -0,0 +1,194 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.alert.util;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.junit.jupiter.api.Test;
import static org.apache.hertzbeat.alert.util.NoticeReceiverMaskUtil.SECRET_MASK;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
/**
* Test case for {@link NoticeReceiverMaskUtil}
*/
class NoticeReceiverMaskUtilTest {
private NoticeReceiver buildReceiverWithSecrets() {
NoticeReceiver receiver = new NoticeReceiver();
receiver.setId(1L);
receiver.setName("tom");
receiver.setEmail("tom@usthe.com");
receiver.setHookUrl("https://example.com/hook");
receiver.setHookAuthToken("hook-auth-token-abcd");
receiver.setAccessToken("c03a568a306f8fd84dab51ff03cf6af6ba676a3be940c904e1df2de34853739d");
receiver.setTgBotToken("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM");
receiver.setSlackWebHookUrl("https://hooks.slack.com/services/X/Y/Zt0k3n");
receiver.setAppSecret("oUydwn92ey0lnuY02MixNa57eNK-20dJn5NEOG-u2uE");
receiver.setDiscordBotToken("MTA2NTMwMzU0ODY4Mzg4MjUzNw.discord.t0kn");
receiver.setSmnAk("NCVBODJOEYHSW3VNSMAK");
receiver.setSmnSk("nmSNhUJN9MlpPl8lfCsgdA0KvHCL9JSMSK");
receiver.setServerChanToken("SCT193569TSNm6xIabdjqeZPtOGOWcvU1e");
receiver.setGotifyToken("A845h__ZMqDxZlO");
receiver.setNtfyToken("tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2");
return receiver;
}
@Test
void maskKeepsSuffixOfLongSecrets() {
NoticeReceiver receiver = buildReceiverWithSecrets();
NoticeReceiver masked = NoticeReceiverMaskUtil.mask(receiver);
assertEquals(SECRET_MASK + "abcd", masked.getHookAuthToken());
assertEquals(SECRET_MASK + "739d", masked.getAccessToken());
assertEquals(SECRET_MASK + "voJM", masked.getTgBotToken());
assertEquals(SECRET_MASK + "0k3n", masked.getSlackWebHookUrl());
assertEquals(SECRET_MASK + "u2uE", masked.getAppSecret());
assertEquals(SECRET_MASK + "t0kn", masked.getDiscordBotToken());
assertEquals(SECRET_MASK + "SMAK", masked.getSmnAk());
assertEquals(SECRET_MASK + "SMSK", masked.getSmnSk());
assertEquals(SECRET_MASK + "vU1e", masked.getServerChanToken());
assertEquals(SECRET_MASK + "xZlO", masked.getGotifyToken());
assertEquals(SECRET_MASK + "NIz2", masked.getNtfyToken());
assertEquals(receiver.getId(), masked.getId());
assertEquals(receiver.getName(), masked.getName());
assertEquals(receiver.getEmail(), masked.getEmail());
assertEquals(receiver.getHookUrl(), masked.getHookUrl());
}
@Test
void maskHidesShortSecretsEntirely() {
NoticeReceiver receiver = new NoticeReceiver();
receiver.setAccessToken("short-token");
receiver.setGotifyToken("tiny");
NoticeReceiver masked = NoticeReceiverMaskUtil.mask(receiver);
assertEquals(SECRET_MASK, masked.getAccessToken());
assertEquals(SECRET_MASK, masked.getGotifyToken());
}
@Test
void maskDoesNotModifyOriginalAndKeepsEmptySecrets() {
NoticeReceiver receiver = buildReceiverWithSecrets();
NoticeReceiverMaskUtil.mask(receiver);
assertEquals("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM", receiver.getTgBotToken());
assertEquals("nmSNhUJN9MlpPl8lfCsgdA0KvHCL9JSMSK", receiver.getSmnSk());
NoticeReceiver empty = new NoticeReceiver();
NoticeReceiver maskedEmpty = NoticeReceiverMaskUtil.mask(empty);
assertNull(maskedEmpty.getAccessToken());
assertNull(maskedEmpty.getNtfyToken());
assertNull(NoticeReceiverMaskUtil.mask(null));
}
@Test
void resolveMaskRestoresOnlyMaskedFields() {
NoticeReceiver existing = buildReceiverWithSecrets();
NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(existing);
incoming.setAccessToken("new-access-token-1234");
incoming.setGotifyToken(null);
NoticeReceiverMaskUtil.resolveMask(incoming, existing);
assertEquals("new-access-token-1234", incoming.getAccessToken());
assertNull(incoming.getGotifyToken());
assertEquals("tk_AgQdq7mVBoFD37zQVN29RhuMzNIz2", incoming.getNtfyToken());
assertEquals("hook-auth-token-abcd", incoming.getHookAuthToken());
assertEquals("1499012345:AAEOB_wEYS-DZyPM3h5NzI8voJM", incoming.getTgBotToken());
assertEquals("https://hooks.slack.com/services/X/Y/Zt0k3n", incoming.getSlackWebHookUrl());
assertEquals("oUydwn92ey0lnuY02MixNa57eNK-20dJn5NEOG-u2uE", incoming.getAppSecret());
assertEquals("MTA2NTMwMzU0ODY4Mzg4MjUzNw.discord.t0kn", incoming.getDiscordBotToken());
assertEquals("NCVBODJOEYHSW3VNSMAK", incoming.getSmnAk());
assertEquals("nmSNhUJN9MlpPl8lfCsgdA0KvHCL9JSMSK", incoming.getSmnSk());
assertEquals("SCT193569TSNm6xIabdjqeZPtOGOWcvU1e", incoming.getServerChanToken());
}
@Test
void resolveMaskRejectsMaskWhenNothingIsStored() {
NoticeReceiver existing = new NoticeReceiver();
NoticeReceiver incoming = new NoticeReceiver();
incoming.setAccessToken(SECRET_MASK);
assertThrows(
IllegalArgumentException.class,
() -> NoticeReceiverMaskUtil.resolveMask(incoming, existing));
}
@Test
void resolveMaskRejectsBareMaskAsWildcard() {
NoticeReceiver existing = buildReceiverWithSecrets();
NoticeReceiver incoming = new NoticeReceiver();
incoming.setAccessToken(SECRET_MASK);
assertThrows(
IllegalArgumentException.class,
() -> NoticeReceiverMaskUtil.resolveMask(incoming, existing));
}
@Test
void resolveMaskRestoresBareMaskForShortStoredSecret() {
NoticeReceiver existing = new NoticeReceiver();
existing.setAccessToken("short-token");
NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(existing);
NoticeReceiverMaskUtil.resolveMask(incoming, existing);
assertEquals("short-token", incoming.getAccessToken());
}
@Test
void resolveMaskForTestRejectsMaskedWebhookSecretForChangedUrl() {
NoticeReceiver existing = buildReceiverWithSecrets();
existing.setType((byte) 2);
NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(existing);
incoming.setHookUrl("https://attacker.example/collect");
assertThrows(
IllegalArgumentException.class,
() -> NoticeReceiverMaskUtil.resolveMaskForTest(incoming, existing));
assertEquals(SECRET_MASK + "abcd", incoming.getHookAuthToken());
}
@Test
void resolveMaskForTestRejectsMaskedNtfySecretForChangedServer() {
NoticeReceiver existing = buildReceiverWithSecrets();
existing.setType((byte) 15);
existing.setNtfyServerUrl("https://ntfy.example");
NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(existing);
incoming.setNtfyServerUrl("https://attacker.example");
assertThrows(
IllegalArgumentException.class,
() -> NoticeReceiverMaskUtil.resolveMaskForTest(incoming, existing));
assertEquals(SECRET_MASK + "NIz2", incoming.getNtfyToken());
}
@Test
void resolveMaskForTestRestoresSecretForUnchangedDestination() {
NoticeReceiver existing = buildReceiverWithSecrets();
existing.setType((byte) 2);
NoticeReceiver incoming = NoticeReceiverMaskUtil.mask(existing);
NoticeReceiverMaskUtil.resolveMaskForTest(incoming, existing);
assertEquals("hook-auth-token-abcd", incoming.getHookAuthToken());
}
}
@@ -222,6 +222,8 @@ public class CommonDispatcher implements MetricsTaskDispatch, CollectDataDispatc
String monitorKey;
if (metrics.isHasSubTask()) {
monitorKey = job.getId() + "-" + metrics.getName() + "-sub-" + metrics.getSubTaskId();
} else if (metrics.getPrometheus() != null) {
monitorKey = String.valueOf(job.getId());
} else {
monitorKey = job.getId() + "-" + metrics.getName();
}
@@ -52,7 +52,7 @@ spring:
collector:
info:
version: ${COLLECTOR_VERSION:1.8.0}
version: ${COLLECTOR_VERSION:1.9.0}
ip: ${COLLECTOR_IP:}
dispatch:
entrance:
@@ -85,6 +85,15 @@ class SqlServerJdbcTemplateIntegrationTest {
@BeforeAll
void setUp() throws Exception {
// Checked before the Docker probe so the skip costs nothing: Microsoft ships
// no arm64 image for SQL Server on Linux -- 2017, 2019 and 2022 are all
// amd64-only. Under emulation on Apple Silicon sqlservr crashes during
// startup (core dump, never logs "ready for client connections"), and
// Rosetta does not help. This is not a timeout to be tuned: the full
// version matrix runs on the amd64 CI runners instead.
Assumptions.assumeFalse(
"aarch64".equals(System.getProperty("os.arch")),
"SQL Server has no arm64 image; matrix runs on amd64 CI");
Assumptions.assumeTrue(
DockerClientFactory.instance().isDockerAvailable(),
"Docker is required for integration tests");
@@ -104,17 +104,23 @@ class CommonHttpClientVirtualThreadTest {
@Test
void dispatchConnectionPoolCleanupClosesExpiredAndIdleConnections() throws Exception {
CountDownLatch latch = new CountDownLatch(1);
CountDownLatch expiredConnectionsClosed = new CountDownLatch(1);
CountDownLatch idleConnectionsClosed = new CountDownLatch(1);
PoolingHttpClientConnectionManager manager = mock(PoolingHttpClientConnectionManager.class);
doAnswer(invocation -> {
latch.countDown();
expiredConnectionsClosed.countDown();
return null;
}).when(manager).closeExpiredConnections();
doAnswer(invocation -> {
idleConnectionsClosed.countDown();
return null;
}).when(manager).closeIdleConnections(40, TimeUnit.SECONDS);
CommonHttpClient.setConnectionManagerForTest(manager);
CommonHttpClient.dispatchConnectionPoolCleanup();
assertTrue(latch.await(5, TimeUnit.SECONDS));
assertTrue(expiredConnectionsClosed.await(5, TimeUnit.SECONDS));
assertTrue(idleConnectionsClosed.await(5, TimeUnit.SECONDS));
verify(manager, times(1)).closeExpiredConnections();
verify(manager, times(1)).closeIdleConnections(40, TimeUnit.SECONDS);
}
@@ -17,6 +17,8 @@
package org.apache.hertzbeat.common.constants;
import java.time.Duration;
/**
* Http Constants.
*/
@@ -56,11 +58,17 @@ public interface NetworkConstants {
*/
interface HttpClientConstants {
int READ_TIME_OUT = 6 * 1000;
int WRITE_TIME_OUT = 6 * 1000;
int CONNECT_TIME_OUT = 6 * 1000;
Duration READ_TIMEOUT = Duration.ofSeconds(6);
Duration WRITE_TIMEOUT = Duration.ofSeconds(6);
Duration CONNECT_TIMEOUT = Duration.ofSeconds(6);
Duration GREPTIME_QUERY_READ_TIMEOUT = Duration.ofSeconds(5);
Duration GREPTIME_QUERY_CONNECT_TIMEOUT = Duration.ofSeconds(2);
Duration GREPTIME_WRITE_READ_TIMEOUT = Duration.ofSeconds(3);
Duration GREPTIME_WRITE_CONNECT_TIMEOUT = Duration.ofSeconds(2);
Duration GREPTIME_INIT_READ_TIMEOUT = Duration.ofSeconds(10);
Duration GREPTIME_INIT_CONNECT_TIMEOUT = Duration.ofSeconds(3);
int MAX_IDLE_CONNECTIONS = 20;
int KEEP_ALIVE_TIMEOUT = 30 * 1000;
Duration KEEP_ALIVE_TIMEOUT = Duration.ofSeconds(30);
}
}
@@ -0,0 +1,24 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
/** Entity-free OpenTelemetry log query context. */
public record LogQueryFilter(Long start, Long end, String traceId, String spanId, Integer severityNumber,
String severityText, String search, String serviceName, String serviceNamespace,
String environment, String resourceFilter) {
}
@@ -0,0 +1,29 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
/**
* One OTLP metric sample.
*
* @param timestamp epoch milliseconds
* @param value numeric sample value
*/
public record MetricPoint(long timestamp, double value) {
}
@@ -0,0 +1,30 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
import java.util.Map;
/**
* OTLP metric time series.
*
* @param labels series labels
* @param points ordered samples
*/
public record MetricSeries(Map<String, String> labels, List<MetricPoint> points) {
}
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
/**
* Which observability query consoles the current storage backend supports.
*
* @param log log query console supported
* @param trace trace query console supported
* @param metric OTLP metric query console supported
*/
public record ObservabilityCapability(boolean log, boolean trace, boolean metric) {
}
@@ -0,0 +1,32 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
/**
* OTLP metrics query result.
*
* @param query effective PromQL expression
* @param start query start in epoch milliseconds
* @param end query end in epoch milliseconds
* @param stepSeconds query resolution
* @param series returned time series
*/
public record OtlpMetricsConsole(String query, long start, long end, int stepSeconds, List<MetricSeries> series) {
}
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
/**
* Discoverable OTLP metric names.
*
* @param metricNames sorted metric names
*/
public record OtlpMetricsInventory(List<String> metricNames) {
}
@@ -0,0 +1,32 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
/**
* Stable pagination payload shared by the three signal workbenches.
*
* @param content current page content
* @param pageIndex zero-based page index
* @param pageSize requested page size
* @param totalElements total matching records
* @param <T> row type
*/
public record SignalPage<T>(List<T> content, int pageIndex, int pageSize, long totalElements) {
}
@@ -0,0 +1,24 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
/** Complete trace detail. */
public record TraceDetail(TraceListItem summary, List<TraceSpanNode> spans) {
}
@@ -0,0 +1,26 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.Map;
/** Trace summary row. */
public record TraceListItem(String traceId, String rootSpanId, String serviceName, String serviceNamespace,
String rootSpanName, long durationNanos, String status, long startTime,
int errorSpanCount, Map<String, String> resourceAttributes) {
}
@@ -0,0 +1,23 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
/** Trace aggregate statistics. */
public record TraceOverview(long totalCount, long errorCount, double errorRate, double averageDurationMillis,
double p95DurationMillis) {
}
@@ -0,0 +1,24 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.Map;
/** Event emitted while a trace span is active. */
public record TraceSpanEvent(String name, String time, Map<String, Object> attributes) {
}
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.entity.dto.observability;
import java.util.List;
import java.util.Map;
/** Trace span used by the Angular waterfall. */
public record TraceSpanNode(String traceId, String spanId, String parentSpanId, String spanName,
String serviceName, String status, String spanKind, String statusMessage,
long durationNanos, long startTime, Map<String, String> resourceAttributes,
Map<String, String> spanAttributes, List<TraceSpanEvent> spanEvents) {
}
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.security;
/** Validates an API token for OTLP ingestion without coupling the signal module to manager. */
public interface OtlpAccessTokenValidator {
/**
* @param token raw bearer token
* @return null when accepted, otherwise a safe rejection reason
*/
String validate(String token);
}
@@ -0,0 +1,26 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.support.exception;
/** Signals that a required backing store cannot currently serve a request. */
public class StorageUnavailableException extends RuntimeException {
public StorageUnavailableException(String message, Throwable cause) {
super(message, cause);
}
}
@@ -17,8 +17,11 @@
package org.apache.hertzbeat.common.util;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.Arrays;
import java.util.Base64;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
@@ -46,6 +49,12 @@ public final class AesUtil {
private static final String AES = "AES";
private static final byte[] PAYLOAD_HEADER = {'H', 'B', 'A', '2'};
private static final int IV_LENGTH = 16;
private static final SecureRandom SECURE_RANDOM = new SecureRandom();
/**
* Encryption key The AES encryption key is 16 bits.
* If the AES encryption key is larger than 16 bits, an error message is displayed
@@ -87,14 +96,21 @@ public final class AesUtil {
SecretKeySpec keySpec = new SecretKeySpec(encryptKey.getBytes(StandardCharsets.UTF_8), AES);
// cipher based on the algorithm AES
Cipher cipher = Cipher.getInstance(ALGORITHM_STR);
byte[] initializationVector = new byte[IV_LENGTH];
SECURE_RANDOM.nextBytes(initializationVector);
// init cipher Encrypt_mode or Decrypt_mode operation, the second parameter is the KEY used
cipher.init(Cipher.ENCRYPT_MODE, keySpec, new IvParameterSpec(encryptKey.getBytes(StandardCharsets.UTF_8)));
cipher.init(Cipher.ENCRYPT_MODE, keySpec, new IvParameterSpec(initializationVector));
// get content bytes, must utf-8
byte[] byteEncode = content.getBytes(StandardCharsets.UTF_8);
// encode content to byte array
byte[] byteAes = cipher.doFinal(byteEncode);
byte[] payload = ByteBuffer.allocate(PAYLOAD_HEADER.length + initializationVector.length + byteAes.length)
.put(PAYLOAD_HEADER)
.put(initializationVector)
.put(byteAes)
.array();
// base64 encode content
return Base64.getEncoder().encodeToString(byteAes);
return Base64.getEncoder().encodeToString(payload);
} catch (Exception e) {
log.error("aes encode content error: {}", e.getMessage(), e);
return content;
@@ -135,13 +151,36 @@ public final class AesUtil {
SecretKeySpec keySpec = new SecretKeySpec(decryptKey.getBytes(StandardCharsets.UTF_8), AES);
// cipher based on the algorithm AES
Cipher cipher = Cipher.getInstance(ALGORITHM_STR);
// init cipher Encrypt_mode or Decrypt_mode operation, the second parameter is the KEY used
cipher.init(Cipher.DECRYPT_MODE, keySpec, new IvParameterSpec(decryptKey.getBytes(StandardCharsets.UTF_8)));
// base64 decode content
byte[] bytesContent = Base64.getDecoder().decode(content);
byte[] initializationVector;
if (hasPayloadHeader(bytesContent)) {
if (bytesContent.length <= PAYLOAD_HEADER.length + IV_LENGTH) {
throw new IllegalArgumentException("Invalid encrypted payload");
}
initializationVector = Arrays.copyOfRange(
bytesContent, PAYLOAD_HEADER.length, PAYLOAD_HEADER.length + IV_LENGTH);
bytesContent = Arrays.copyOfRange(
bytesContent, PAYLOAD_HEADER.length + IV_LENGTH, bytesContent.length);
} else {
initializationVector = decryptKey.getBytes(StandardCharsets.UTF_8);
}
// init cipher Encrypt_mode or Decrypt_mode operation, the second parameter is the KEY used
cipher.init(Cipher.DECRYPT_MODE, keySpec, new IvParameterSpec(initializationVector));
// decode content to byte array
return cipher.doFinal(bytesContent);
}
private static boolean hasPayloadHeader(byte[] payload) {
if (payload.length < PAYLOAD_HEADER.length) {
return false;
}
for (int index = 0; index < PAYLOAD_HEADER.length; index++) {
if (payload[index] != PAYLOAD_HEADER[index]) {
return false;
}
}
return true;
}
/**
* Determine whether it is encrypted
@@ -20,10 +20,14 @@ package org.apache.hertzbeat.common.util;
import static org.apache.hertzbeat.common.util.AesUtil.aesDecode;
import static org.apache.hertzbeat.common.util.AesUtil.aesEncode;
import static org.apache.hertzbeat.common.util.AesUtil.isCiphertext;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Base64;
import org.junit.jupiter.api.Test;
/**
@@ -43,12 +47,31 @@ class AesUtilTest {
void testAesEncode() {
String originalText = "This is a secret message";
String encryptedText = aesEncode(originalText, VALID_KEY);
String secondEncryptedText = aesEncode(originalText, VALID_KEY);
assertNotEquals(originalText, encryptedText);
assertNotEquals(encryptedText, secondEncryptedText);
byte[] encryptedPayload = Base64.getDecoder().decode(encryptedText);
byte[] secondEncryptedPayload = Base64.getDecoder().decode(secondEncryptedText);
assertArrayEquals(new byte[] {'H', 'B', 'A', '2'}, Arrays.copyOfRange(encryptedPayload, 0, 4));
assertFalse(Arrays.equals(
VALID_KEY.getBytes(StandardCharsets.UTF_8), Arrays.copyOfRange(encryptedPayload, 4, 20)));
assertFalse(Arrays.equals(
Arrays.copyOfRange(encryptedPayload, 4, 20),
Arrays.copyOfRange(secondEncryptedPayload, 4, 20)));
String decryptedText = aesDecode(encryptedText, VALID_KEY);
assertEquals(originalText, decryptedText);
}
@Test
void testLegacyCiphertextCanBeDecoded() {
String legacyCiphertext = "muJNZwhxg173v6EZAXb5TK8L5XlmDE5xmc9XTryH6Qk=";
assertEquals("This is a secret message", aesDecode(legacyCiphertext, VALID_KEY));
assertTrue(isCiphertext(legacyCiphertext, VALID_KEY));
}
@Test
void testAesDecode() {
// Test with invalid key
@@ -83,11 +106,8 @@ class AesUtilTest {
String invalidBase64Text = "InvalidBase64";
assertFalse(isCiphertext(invalidBase64Text, VALID_KEY));
// Test with invalid key
originalText = "This is a secret message";
encryptedText = aesEncode(originalText, VALID_KEY);
String invalidKey = "6543210987654321";
assertFalse(isCiphertext(encryptedText, invalidKey));
// Valid base64 that cannot be decrypted: 3 bytes is not a block multiple
assertFalse(isCiphertext("AAAA", VALID_KEY));
}
@Test
@@ -17,25 +17,6 @@
* under the License.
*/
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.hertzbeat.common.entity.alerter;
import io.swagger.v3.oas.annotations.media.Schema;
@@ -48,7 +48,7 @@ import org.springframework.data.jpa.domain.support.AuditingEntityListener;
@Entity
@Table(name = "hzb_notice_receiver")
@Data
@Builder
@Builder(toBuilder = true)
@AllArgsConstructor
@NoArgsConstructor
@Schema(description = "Message notification recipient entity")
@@ -63,13 +63,14 @@ import static org.mockito.Mockito.doAnswer;
@SpringBootTest(classes = org.apache.hertzbeat.startup.HertzBeatApplication.class, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestPropertySource(properties = {
"warehouse.store.duckdb.enabled=false",
"warehouse.store.greptime.enabled=true"
"warehouse.store.greptime.enabled=true",
"hertzbeat.otlp.grpc.enabled=false"
})
@Slf4j
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class LogPeriodicAlertE2eTest {
private static final String VECTOR_IMAGE = "timberio/vector:latest-alpine";
private static final String VECTOR_IMAGE = "timberio/vector:0.56.0-alpine";
private static final int VECTOR_PORT = 8686;
private static final String VECTOR_CONFIG_PATH = "/etc/vector/vector.yml";
private static final String ENV_HERTZBEAT_PORT = "HERTZBEAT_PORT";
@@ -60,7 +60,7 @@ import static org.mockito.Mockito.doAnswer;
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class LogRealTimeAlertE2eTest {
private static final String VECTOR_IMAGE = "timberio/vector:latest-alpine";
private static final String VECTOR_IMAGE = "timberio/vector:0.56.0-alpine";
private static final int VECTOR_PORT = 8686;
private static final String VECTOR_CONFIG_PATH = "/etc/vector/vector.yml";
private static final String ENV_HERTZBEAT_PORT = "HERTZBEAT_PORT";
@@ -49,7 +49,7 @@ import static org.junit.jupiter.api.Assertions.assertNotNull;
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class LogIngestionE2eTest {
private static final String VECTOR_IMAGE = "timberio/vector:latest-alpine";
private static final String VECTOR_IMAGE = "timberio/vector:0.56.0-alpine";
private static final int VECTOR_PORT = 8686;
private static final String VECTOR_CONFIG_PATH = "/etc/vector/vector.yml";
private static final String ENV_HERTZBEAT_PORT = "HERTZBEAT_PORT";
@@ -52,13 +52,14 @@ import static org.junit.jupiter.api.Assertions.assertNotNull;
@SpringBootTest(classes = org.apache.hertzbeat.startup.HertzBeatApplication.class, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestPropertySource(properties = {
"warehouse.store.duckdb.enabled=false",
"warehouse.store.greptime.enabled=true"
"warehouse.store.greptime.enabled=true",
"hertzbeat.otlp.grpc.enabled=false"
})
@Slf4j
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class GreptimeLogStorageE2eTest {
private static final String VECTOR_IMAGE = "timberio/vector:latest-alpine";
private static final String VECTOR_IMAGE = "timberio/vector:0.56.0-alpine";
private static final int VECTOR_PORT = 8686;
private static final String VECTOR_CONFIG_PATH = "/etc/vector/vector.yml";
private static final String ENV_HERTZBEAT_PORT = "HERTZBEAT_PORT";
@@ -70,17 +70,22 @@ resourceRole:
- /api/sse/**===post===[admin,user]
- /api/chat/**===get===[admin,user]
- /api/chat/**===post===[admin,user]
- /api/logs/sse/**===get===[admin,user,guest]
- /api/logs/ingest/**===post===[admin,user]
- /api/otlp/**===post===[admin,user]
- /api/ingestion/otlp/**===get===[admin,user,guest]
- /api/logs/**===get===[admin,user,guest]
- /api/traces/**===get===[admin,user,guest]
# config the resource restful api that need bypass auth protection
# rule: api===method
# eg: /api/v1/source3===get means /api/v1/source3===get can be access by anyone, no need auth.
excludedResource:
- /api/alert/sse/**===*
- /api/logs/sse/**===*
- /api/account/auth/**===*
- /api/i18n/**===get
- /api/apps/hierarchy===get
- /api/observability/capability===get
- /api/push/**===*
- /api/status/page/public/**===*
- /api/manager/sse/**===*
+14 -1
View File
@@ -71,6 +71,19 @@
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
</dependency>
<!-- OTLP gRPC server, version managed by grpc-bom in the root pom -->
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
</dependency>
<!-- awaitility for async testing -->
<dependency>
<groupId>org.awaitility</groupId>
@@ -80,4 +93,4 @@
</dependency>
</dependencies>
</project>
</project>
@@ -0,0 +1,106 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.config;
import java.nio.charset.StandardCharsets;
import org.apache.hertzbeat.warehouse.constants.WarehouseConstants;
import org.apache.hertzbeat.warehouse.db.GreptimeSqlQueryExecutor;
import org.apache.hertzbeat.warehouse.store.history.tsdb.greptime.GreptimeProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.core.io.ClassPathResource;
import org.springframework.http.ContentDisposition;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Component;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.util.StringUtils;
import org.springframework.web.client.RestTemplate;
/** Fails startup clearly when the required Greptime signal schema cannot be prepared. */
@Component
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
public class GreptimeSignalInitializer {
private static final String TRACE_SCHEMA = "greptime/tables/hzb_traces.sql";
private static final String LOG_SCHEMA = "greptime/tables/hertzbeat_logs.sql";
private static final String LOG_PIPELINE = "greptime/pipelines/hertzbeat_otlp_log_v1.yaml";
private final GreptimeProperties greptimeProperties;
private final GreptimeSqlQueryExecutor sqlQueryExecutor;
private final RestTemplate restTemplate;
public GreptimeSignalInitializer(GreptimeProperties greptimeProperties,
GreptimeSqlQueryExecutor sqlQueryExecutor,
@Qualifier(WarehouseConstants.GREPTIME_INIT_REST_TEMPLATE)
RestTemplate restTemplate) {
this.greptimeProperties = greptimeProperties;
this.sqlQueryExecutor = sqlQueryExecutor;
this.restTemplate = restTemplate;
}
@EventListener(ApplicationReadyEvent.class)
public void initialize() {
try {
String traceSchema = new ClassPathResource(TRACE_SCHEMA)
.getContentAsString(StandardCharsets.UTF_8).strip();
sqlQueryExecutor.execute(StringUtils.trimTrailingCharacter(traceSchema, ';'));
sqlQueryExecutor.execute("ALTER TABLE hzb_traces ADD COLUMN IF NOT EXISTS "
+ "\"resource_attributes.service.namespace\" STRING NULL");
sqlQueryExecutor.execute("ALTER TABLE hzb_traces ADD COLUMN IF NOT EXISTS "
+ "\"resource_attributes.deployment.environment.name\" STRING NULL");
String logSchema = new ClassPathResource(LOG_SCHEMA)
.getContentAsString(StandardCharsets.UTF_8).strip();
sqlQueryExecutor.execute(StringUtils.trimTrailingCharacter(logSchema, ';'));
uploadLogPipeline(new ClassPathResource(LOG_PIPELINE)
.getContentAsString(StandardCharsets.UTF_8));
if (sqlQueryExecutor.execute("SELECT 1 AS ready").isEmpty()) {
throw new IllegalStateException("GreptimeDB readiness query returned no data");
}
} catch (Exception exception) {
throw new IllegalStateException("GreptimeDB is required for the three-signal release but initialization failed",
exception);
}
}
private void uploadLogPipeline(String pipeline) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
if (StringUtils.hasText(greptimeProperties.username()) && StringUtils.hasText(greptimeProperties.password())) {
headers.setBasicAuth(greptimeProperties.username(), greptimeProperties.password(), StandardCharsets.UTF_8);
}
HttpHeaders partHeaders = new HttpHeaders();
partHeaders.setContentType(MediaType.parseMediaType("application/x-yaml"));
partHeaders.setContentDisposition(ContentDisposition.formData().name("file")
.filename("hertzbeat_otlp_log_v1.yaml").build());
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new HttpEntity<>(pipeline, partHeaders));
String endpoint = StringUtils.trimTrailingCharacter(greptimeProperties.httpEndpoint(), '/')
+ "/v1/pipelines/hertzbeat_otlp_log_v1";
ResponseEntity<String> response = restTemplate.exchange(endpoint, HttpMethod.POST,
new HttpEntity<>(body, headers), String.class);
if (!response.getStatusCode().is2xxSuccessful()) {
throw new IllegalStateException("GreptimeDB log pipeline upload failed: " + response.getStatusCode());
}
}
}
@@ -0,0 +1,201 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.config;
import io.grpc.ForwardingServerCallListener;
import io.grpc.Metadata;
import io.grpc.Server;
import io.grpc.ServerCall;
import io.grpc.ServerCallHandler;
import io.grpc.ServerInterceptor;
import io.grpc.ServerInterceptors;
import io.grpc.Status;
import io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder;
import io.grpc.stub.StreamObserver;
import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest;
import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceResponse;
import io.opentelemetry.proto.collector.logs.v1.LogsServiceGrpc;
import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest;
import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceResponse;
import io.opentelemetry.proto.collector.metrics.v1.MetricsServiceGrpc;
import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest;
import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceResponse;
import io.opentelemetry.proto.collector.trace.v1.TraceServiceGrpc;
import java.io.IOException;
import java.net.InetSocketAddress;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.security.OtlpAccessTokenValidator;
import org.apache.hertzbeat.log.service.OtlpSignalForwarder;
import org.apache.hertzbeat.log.service.SignalQueryRejectedException;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard.Workload;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.util.StringUtils;
/** OTLP/gRPC listener for metrics, logs, and traces. */
@Configuration
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
public class OtlpGrpcServerConfig {
@Bean(initMethod = "start", destroyMethod = "stop")
@ConditionalOnProperty(prefix = "hertzbeat.otlp.grpc", name = "enabled", havingValue = "true",
matchIfMissing = true)
public OtlpGrpcServerRunner otlpGrpcServerRunner(
@Value("${hertzbeat.otlp.grpc.host:0.0.0.0}") String host,
@Value("${hertzbeat.otlp.grpc.port:4317}") int port,
OtlpSignalForwarder signalForwarder,
SignalWorkloadGuard workloadGuard,
ObjectProvider<OtlpAccessTokenValidator> tokenValidatorProvider) {
OtlpAccessTokenValidator validator = tokenValidatorProvider.getIfAvailable(
() -> token -> "OTLP token validation is unavailable");
ServerInterceptor interceptor = new BearerTokenInterceptor(validator);
return new OtlpGrpcServerRunner(host, port, signalForwarder, workloadGuard, interceptor);
}
@Slf4j
@RequiredArgsConstructor
static final class OtlpGrpcServerRunner {
private final String host;
private final int port;
private final OtlpSignalForwarder signalForwarder;
private final SignalWorkloadGuard workloadGuard;
private final ServerInterceptor authInterceptor;
private Server server;
public void start() throws IOException {
server = NettyServerBuilder.forAddress(new InetSocketAddress(host, port))
.addService(ServerInterceptors.intercept(new MetricsService(signalForwarder, workloadGuard),
authInterceptor))
.addService(ServerInterceptors.intercept(new LogsService(signalForwarder, workloadGuard),
authInterceptor))
.addService(ServerInterceptors.intercept(new TracesService(signalForwarder, workloadGuard),
authInterceptor))
.build().start();
log.info("OTLP gRPC listener started on {}:{}", host, port);
}
public void stop() {
if (server != null) {
server.shutdownNow();
}
}
}
@RequiredArgsConstructor
static final class MetricsService extends MetricsServiceGrpc.MetricsServiceImplBase {
private final OtlpSignalForwarder signalForwarder;
private final SignalWorkloadGuard workloadGuard;
@Override
public void export(ExportMetricsServiceRequest request,
StreamObserver<ExportMetricsServiceResponse> observer) {
try {
byte[] response = workloadGuard.execute(Workload.OTLP_WRITE,
() -> signalForwarder.forwardProtobuf("metrics", request.toByteArray()));
observer.onNext(response.length == 0 ? ExportMetricsServiceResponse.getDefaultInstance()
: ExportMetricsServiceResponse.parseFrom(response));
observer.onCompleted();
} catch (Exception exception) {
onError(observer, exception);
}
}
}
@RequiredArgsConstructor
static final class LogsService extends LogsServiceGrpc.LogsServiceImplBase {
private final OtlpSignalForwarder signalForwarder;
private final SignalWorkloadGuard workloadGuard;
@Override
public void export(ExportLogsServiceRequest request, StreamObserver<ExportLogsServiceResponse> observer) {
try {
byte[] response = workloadGuard.execute(Workload.OTLP_WRITE,
() -> signalForwarder.forwardProtobuf("logs", request.toByteArray()));
observer.onNext(response.length == 0 ? ExportLogsServiceResponse.getDefaultInstance()
: ExportLogsServiceResponse.parseFrom(response));
observer.onCompleted();
} catch (Exception exception) {
onError(observer, exception);
}
}
}
@RequiredArgsConstructor
static final class TracesService extends TraceServiceGrpc.TraceServiceImplBase {
private final OtlpSignalForwarder signalForwarder;
private final SignalWorkloadGuard workloadGuard;
@Override
public void export(ExportTraceServiceRequest request, StreamObserver<ExportTraceServiceResponse> observer) {
try {
byte[] response = workloadGuard.execute(Workload.OTLP_WRITE,
() -> signalForwarder.forwardProtobuf("traces", request.toByteArray()));
observer.onNext(response.length == 0 ? ExportTraceServiceResponse.getDefaultInstance()
: ExportTraceServiceResponse.parseFrom(response));
observer.onCompleted();
} catch (Exception exception) {
onError(observer, exception);
}
}
}
private static void onError(StreamObserver<?> observer, Exception exception) {
observer.onError(toGrpcStatus(exception).withDescription(exception.getMessage()).withCause(exception)
.asRuntimeException());
}
static Status toGrpcStatus(Exception exception) {
if (exception instanceof IllegalArgumentException) {
return Status.INVALID_ARGUMENT;
}
if (exception instanceof SignalQueryRejectedException) {
return Status.RESOURCE_EXHAUSTED;
}
return Status.UNAVAILABLE;
}
@RequiredArgsConstructor
static final class BearerTokenInterceptor implements ServerInterceptor {
private static final Metadata.Key<String> AUTHORIZATION =
Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER);
private final OtlpAccessTokenValidator tokenValidator;
@Override
public <RequestT, ResponseT> ServerCall.Listener<RequestT> interceptCall(
ServerCall<RequestT, ResponseT> call, Metadata headers,
ServerCallHandler<RequestT, ResponseT> next) {
String authorization = headers.get(AUTHORIZATION);
if (!StringUtils.hasText(authorization) || !authorization.regionMatches(true, 0, "Bearer ", 0, 7)) {
call.close(Status.UNAUTHENTICATED.withDescription("Missing OTLP access token"), new Metadata());
return new ServerCall.Listener<>() { };
}
String rejectReason = tokenValidator.validate(authorization.substring(7).trim());
if (rejectReason != null) {
call.close(Status.UNAUTHENTICATED.withDescription(rejectReason), new Metadata());
return new ServerCall.Listener<>() { };
}
ServerCall.Listener<RequestT> listener = next.startCall(call, headers);
return new ForwardingServerCallListener.SimpleForwardingServerCallListener<>(listener) { };
}
}
}
@@ -21,14 +21,10 @@ import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.function.Supplier;
import org.apache.hertzbeat.warehouse.store.history.tsdb.HistoryDataReader;
import org.springframework.data.domain.Page;
@@ -37,8 +33,12 @@ import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Sort;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.constants.CommonConstants;
import org.apache.hertzbeat.common.entity.dto.Message;
import org.apache.hertzbeat.common.entity.dto.observability.LogQueryFilter;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard.Workload;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
@@ -54,10 +54,17 @@ import org.springframework.web.bind.annotation.RestController;
@Slf4j
public class LogQueryController {
private final HistoryDataReader historyDataReader;
private static final int MAX_PAGE_SIZE = 200;
public LogQueryController(HistoryDataReader historyDataReader) {
private static final String LOG_STORAGE_UNAVAILABLE_MSG =
"Log query is not supported by the current storage, please enable a log capable storage such as GreptimeDB";
private final HistoryDataReader historyDataReader;
private final SignalWorkloadGuard workloadGuard;
public LogQueryController(HistoryDataReader historyDataReader, SignalWorkloadGuard workloadGuard) {
this.historyDataReader = historyDataReader;
this.workloadGuard = workloadGuard;
}
@GetMapping("/list")
@@ -78,12 +85,20 @@ public class LogQueryController {
@RequestParam(value = "severityText", required = false) String severityText,
@Parameter(description = "Log content search keyword", example = "error")
@RequestParam(value = "search", required = false) String search,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "resource", required = false) String resource,
@Parameter(description = "Page index starting from 0", example = "0")
@RequestParam(value = "pageIndex", required = false, defaultValue = "0") Integer pageIndex,
@Parameter(description = "Number of items per page", example = "20")
@RequestParam(value = "pageSize", required = false, defaultValue = "20") Integer pageSize) {
Page<LogEntry> result = getPagedLogs(start, end, traceId, spanId, severityNumber, severityText, search, pageIndex, pageSize);
return ResponseEntity.ok(Message.success(result));
return executeIfLogSupported(Workload.LOG_LIST, () -> {
LogQueryFilter filter = filter(start, end, traceId, spanId, severityNumber, severityText, search,
serviceName, serviceNamespace, environment, resource);
Page<LogEntry> result = getPagedLogs(filter, pageIndex, pageSize);
return ResponseEntity.ok(Message.success(result));
});
}
@GetMapping("/stats/overview")
@@ -103,29 +118,18 @@ public class LogQueryController {
@Parameter(description = "Log severity text (TRACE, DEBUG, INFO, WARN, ERROR, FATAL)", example = "INFO")
@RequestParam(value = "severityText", required = false) String severityText,
@Parameter(description = "Log content search keyword", example = "error")
@RequestParam(value = "search", required = false) String search) {
List<LogEntry> logs = getFilteredLogs(start, end, traceId, spanId, severityNumber, severityText, search);
@RequestParam(value = "search", required = false) String search,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "resource", required = false) String resource) {
return executeIfLogSupported(Workload.LOG_AGGREGATE,
() -> doOverviewStats(filter(start, end, traceId, spanId, severityNumber, severityText, search,
serviceName, serviceNamespace, environment, resource)));
}
Map<String, Object> overview = new HashMap<>();
overview.put("totalCount", logs.size());
// Count by severity levels according to OpenTelemetry standard
// TRACE: 1-4, DEBUG: 5-8, INFO: 9-12, WARN: 13-16, ERROR: 17-20, FATAL: 21-24
long fatalCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 21 && log.getSeverityNumber() <= 24).count();
long errorCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 17 && log.getSeverityNumber() <= 20).count();
long warnCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 13 && log.getSeverityNumber() <= 16).count();
long infoCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 9 && log.getSeverityNumber() <= 12).count();
long debugCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 5 && log.getSeverityNumber() <= 8).count();
long traceCount = logs.stream().filter(log -> log.getSeverityNumber() != null && log.getSeverityNumber() >= 1 && log.getSeverityNumber() <= 4).count();
overview.put("fatalCount", fatalCount);
overview.put("errorCount", errorCount);
overview.put("warnCount", warnCount);
overview.put("infoCount", infoCount);
overview.put("debugCount", debugCount);
overview.put("traceCount", traceCount);
return ResponseEntity.ok(Message.success(overview));
private ResponseEntity<Message<Map<String, Object>>> doOverviewStats(LogQueryFilter filter) {
return ResponseEntity.ok(Message.success(historyDataReader.queryLogOverviewAggregate(filter)));
}
@GetMapping("/stats/trace-coverage")
@@ -145,26 +149,19 @@ public class LogQueryController {
@Parameter(description = "Log severity text (TRACE, DEBUG, INFO, WARN, ERROR, FATAL)", example = "INFO")
@RequestParam(value = "severityText", required = false) String severityText,
@Parameter(description = "Log content search keyword", example = "error")
@RequestParam(value = "search", required = false) String search) {
List<LogEntry> logs = getFilteredLogs(start, end, traceId, spanId, severityNumber, severityText, search);
@RequestParam(value = "search", required = false) String search,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "resource", required = false) String resource) {
return executeIfLogSupported(Workload.LOG_AGGREGATE,
() -> doTraceCoverageStats(filter(start, end, traceId, spanId, severityNumber, severityText, search,
serviceName, serviceNamespace, environment, resource)));
}
private ResponseEntity<Message<Map<String, Object>>> doTraceCoverageStats(LogQueryFilter filter) {
Map<String, Object> result = new HashMap<>();
// Trace coverage statistics
long withTraceId = logs.stream().filter(log -> log.getTraceId() != null && !log.getTraceId().isEmpty()).count();
long withSpanId = logs.stream().filter(log -> log.getSpanId() != null && !log.getSpanId().isEmpty()).count();
long withBothTraceAndSpan = logs.stream().filter(log ->
log.getTraceId() != null && !log.getTraceId().isEmpty()
&& log.getSpanId() != null && !log.getSpanId().isEmpty()).count();
long withoutTrace = logs.size() - withTraceId;
Map<String, Long> traceCoverage = new HashMap<>();
traceCoverage.put("withTrace", withTraceId);
traceCoverage.put("withoutTrace", withoutTrace);
traceCoverage.put("withSpan", withSpanId);
traceCoverage.put("withBothTraceAndSpan", withBothTraceAndSpan);
result.put("traceCoverage", traceCoverage);
result.put("traceCoverage", historyDataReader.queryLogOverviewAggregate(filter).get("traceCoverage"));
return ResponseEntity.ok(Message.success(result));
}
@@ -185,49 +182,48 @@ public class LogQueryController {
@Parameter(description = "Log severity text (TRACE, DEBUG, INFO, WARN, ERROR, FATAL)", example = "INFO")
@RequestParam(value = "severityText", required = false) String severityText,
@Parameter(description = "Log content search keyword", example = "error")
@RequestParam(value = "search", required = false) String search) {
List<LogEntry> logs = getFilteredLogs(start, end, traceId, spanId, severityNumber, severityText, search);
// Group by hour
Map<String, Long> hourlyStats = logs.stream()
.filter(log -> log.getTimeUnixNano() != null)
.collect(Collectors.groupingBy(
log -> {
long timestampMs = log.getTimeUnixNano() / 1_000_000L;
LocalDateTime dateTime = LocalDateTime.ofInstant(
Instant.ofEpochMilli(timestampMs),
ZoneId.systemDefault());
return dateTime.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:00"));
},
Collectors.counting()));
@RequestParam(value = "search", required = false) String search,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "resource", required = false) String resource) {
return executeIfLogSupported(Workload.LOG_AGGREGATE,
() -> doTrendStats(filter(start, end, traceId, spanId, severityNumber, severityText, search,
serviceName, serviceNamespace, environment, resource)));
}
private ResponseEntity<Message<Map<String, Object>>> doTrendStats(LogQueryFilter filter) {
Map<String, Object> result = new HashMap<>();
result.put("hourlyStats", hourlyStats);
result.put("hourlyStats", historyDataReader.queryLogTrendAggregate(filter));
return ResponseEntity.ok(Message.success(result));
}
private List<LogEntry> getFilteredLogs(Long start, Long end, String traceId, String spanId,
Integer severityNumber, String severityText, String search) {
// Use the new multi-condition query method
return historyDataReader.queryLogsByMultipleConditions(start, end, traceId, spanId, severityNumber, severityText, search);
private <T> ResponseEntity<Message<T>> executeIfLogSupported(Workload workload,
Supplier<ResponseEntity<Message<T>>> operation) {
if (!historyDataReader.supportsLogQuery()) {
return ResponseEntity.ok(Message.fail(CommonConstants.FAIL_CODE, LOG_STORAGE_UNAVAILABLE_MSG));
}
return workloadGuard.execute(workload, operation);
}
private Page<LogEntry> getPagedLogs(Long start, Long end, String traceId, String spanId,
Integer severityNumber, String severityText, String search,
Integer pageIndex, Integer pageSize) {
// Calculate pagination parameters
int offset = pageIndex * pageSize;
private LogQueryFilter filter(Long start, Long end, String traceId, String spanId, Integer severityNumber,
String severityText, String search, String serviceName, String serviceNamespace,
String environment, String resource) {
return new LogQueryFilter(start, end, traceId, spanId, severityNumber, severityText, search, serviceName,
serviceNamespace, environment, resource);
}
// Get total count and paginated data
long totalElements = historyDataReader.countLogsByMultipleConditions(start, end, traceId, spanId, severityNumber, severityText, search);
List<LogEntry> pagedLogs = historyDataReader.queryLogsByMultipleConditionsWithPagination(
start, end, traceId, spanId, severityNumber, severityText, search, offset, pageSize);
private Page<LogEntry> getPagedLogs(LogQueryFilter filter, Integer pageIndex, Integer pageSize) {
int effectivePage = Math.max(pageIndex == null ? 0 : pageIndex, 0);
int effectiveSize = Math.max(1, Math.min(pageSize == null ? 20 : pageSize, MAX_PAGE_SIZE));
int offset = effectivePage * effectiveSize;
long totalElements = historyDataReader.countObservabilityLogs(filter);
List<LogEntry> pagedLogs = historyDataReader.queryObservabilityLogs(filter, offset, effectiveSize);
// Create PageRequest (sorted by timestamp descending)
Sort sort = Sort.by(Sort.Direction.DESC, "timeUnixNano");
PageRequest pageRequest = PageRequest.of(pageIndex, pageSize, sort);
PageRequest pageRequest = PageRequest.of(effectivePage, effectiveSize, sort);
// Return Spring Data Page object
return new PageImpl<>(pagedLogs, pageRequest, totalElements);
}
}
}
@@ -0,0 +1,65 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.controller;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.hertzbeat.common.entity.dto.Message;
import org.apache.hertzbeat.common.entity.dto.observability.ObservabilityCapability;
import org.apache.hertzbeat.log.service.ThreeSignalQueryService;
import org.apache.hertzbeat.warehouse.store.history.tsdb.HistoryDataReader;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
/**
* Report which observability query features the current storage backend supports,
* so the UI can render guidance instead of firing queries that cannot be served.
*
* <p>The three capabilities are probed through two different mechanisms on purpose: log query
* is dispatched per storage via {@link HistoryDataReader#supportsLogQuery()} (multiple storages
* implement log read/write), while trace/metric consoles are served only by the conditionally
* registered {@link ThreeSignalQueryService} (GreptimeDB only for now).
* TODO: once trace/metric query grows storage-dispatched implementations like logs (or log query
* is absorbed into the three-signal contract), unify the probes on a single abstraction.
*/
@RestController
@RequestMapping(path = "/api/observability", produces = "application/json")
@Tag(name = "Observability Capability Controller")
public class ObservabilityCapabilityController {
private final HistoryDataReader historyDataReader;
private final ObjectProvider<ThreeSignalQueryService> threeSignalQueryService;
public ObservabilityCapabilityController(HistoryDataReader historyDataReader,
ObjectProvider<ThreeSignalQueryService> threeSignalQueryService) {
this.historyDataReader = historyDataReader;
this.threeSignalQueryService = threeSignalQueryService;
}
@GetMapping("/capability")
@Operation(summary = "Query observability capability",
description = "Return whether log, trace and OTLP metric query consoles are supported by the current storage backend.")
public ResponseEntity<Message<ObservabilityCapability>> capability() {
boolean traceAndMetricSupported = threeSignalQueryService.getIfAvailable() != null;
return ResponseEntity.ok(Message.success(new ObservabilityCapability(
historyDataReader.supportsLogQuery(), traceAndMetricSupported, traceAndMetricSupported)));
}
}
@@ -0,0 +1,80 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.controller;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.nio.charset.StandardCharsets;
import org.apache.hertzbeat.log.service.OtlpSignalForwarder;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard.Workload;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
/** Standard OTLP/HTTP gateway for all three signals. */
@RestController
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
@RequestMapping("/api/otlp/v1")
@Tag(name = "OTLP Signal Controller")
public class OtlpSignalController {
private final OtlpSignalForwarder signalForwarder;
private final SignalWorkloadGuard workloadGuard;
public OtlpSignalController(OtlpSignalForwarder signalForwarder, SignalWorkloadGuard workloadGuard) {
this.signalForwarder = signalForwarder;
this.workloadGuard = workloadGuard;
}
@PostMapping("/metrics")
@Operation(summary = "Ingest OTLP metrics")
public ResponseEntity<byte[]> metrics(@RequestBody byte[] content, @RequestHeader HttpHeaders headers) {
return forward("metrics", content, headers);
}
@PostMapping("/logs")
@Operation(summary = "Ingest OTLP logs")
public ResponseEntity<byte[]> logs(@RequestBody byte[] content, @RequestHeader HttpHeaders headers) {
return forward("logs", content, headers);
}
@PostMapping("/traces")
@Operation(summary = "Ingest OTLP traces")
public ResponseEntity<byte[]> traces(@RequestBody byte[] content, @RequestHeader HttpHeaders headers) {
return forward("traces", content, headers);
}
private ResponseEntity<byte[]> forward(String signal, byte[] content, HttpHeaders headers) {
return workloadGuard.execute(Workload.OTLP_WRITE,
() -> signalForwarder.forwardHttp(signal, content, headers));
}
@ExceptionHandler(IllegalArgumentException.class)
public ResponseEntity<byte[]> invalidPayload(IllegalArgumentException exception) {
return ResponseEntity.status(HttpStatus.BAD_REQUEST)
.body(exception.getMessage().getBytes(StandardCharsets.UTF_8));
}
}
@@ -0,0 +1,64 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.controller;
import static org.apache.hertzbeat.common.constants.CommonConstants.FAIL_CODE;
import org.apache.hertzbeat.common.entity.dto.Message;
import org.apache.hertzbeat.common.support.exception.StorageUnavailableException;
import org.apache.hertzbeat.log.service.SignalQueryRejectedException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.client.RestClientException;
/** Maps bounded signal workload rejection to a retryable HTTP response. */
@RestControllerAdvice(basePackages = "org.apache.hertzbeat.log.controller")
@Order(Ordered.HIGHEST_PRECEDENCE)
public class SignalWorkloadExceptionHandler {
@ExceptionHandler(IllegalArgumentException.class)
public ResponseEntity<Message<Void>> handleInvalidQuery(IllegalArgumentException exception) {
return ResponseEntity.badRequest().body(Message.fail(FAIL_CODE, exception.getMessage()));
}
@ExceptionHandler(SignalQueryRejectedException.class)
public ResponseEntity<Message<Void>> handleRejected(SignalQueryRejectedException exception) {
return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS)
.header(HttpHeaders.RETRY_AFTER, "1")
.body(Message.fail(FAIL_CODE, exception.getMessage()));
}
@ExceptionHandler(StorageUnavailableException.class)
public ResponseEntity<Message<Void>> handleStorageUnavailable(StorageUnavailableException exception) {
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.header(HttpHeaders.RETRY_AFTER, "1")
.body(Message.fail(FAIL_CODE, exception.getMessage()));
}
@ExceptionHandler(RestClientException.class)
public ResponseEntity<Message<Void>> handleStorageClientFailure(RestClientException exception) {
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.header(HttpHeaders.RETRY_AFTER, "1")
.body(Message.fail(FAIL_CODE, "GreptimeDB storage is unavailable"));
}
}
@@ -0,0 +1,136 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.controller;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.hertzbeat.common.entity.dto.Message;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsConsole;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsInventory;
import org.apache.hertzbeat.common.entity.dto.observability.SignalPage;
import org.apache.hertzbeat.common.entity.dto.observability.TraceDetail;
import org.apache.hertzbeat.common.entity.dto.observability.TraceListItem;
import org.apache.hertzbeat.common.entity.dto.observability.TraceOverview;
import org.apache.hertzbeat.log.service.ThreeSignalQueryService;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard.Workload;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
/** Public entity-free query APIs for OTLP metrics and traces. */
@RestController
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
@Tag(name = "Three Signal Query Controller")
public class ThreeSignalQueryController {
private final ThreeSignalQueryService queryService;
private final SignalWorkloadGuard workloadGuard;
public ThreeSignalQueryController(ThreeSignalQueryService queryService, SignalWorkloadGuard workloadGuard) {
this.queryService = queryService;
this.workloadGuard = workloadGuard;
}
@GetMapping(path = "/api/ingestion/otlp/metrics/console", produces = "application/json")
@Operation(summary = "Query OTLP metrics")
public ResponseEntity<Message<OtlpMetricsConsole>> metrics(
@RequestParam(value = "query", required = false) String query,
@RequestParam(value = "start", required = false) Long start,
@RequestParam(value = "end", required = false) Long end,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "filter", required = false) String filter,
@RequestParam(value = "groupBy", required = false) String groupBy,
@RequestParam(value = "aggregation", required = false) String aggregation,
@RequestParam(value = "step", required = false) Integer step,
@RequestParam(value = "operationName", required = false) String operationName) {
return workloadGuard.execute(Workload.METRICS, () -> ResponseEntity.ok(Message.success(
queryService.queryMetrics(query, start, end, serviceName, serviceNamespace, environment,
filter, groupBy, aggregation, step, operationName))));
}
@GetMapping(path = "/api/ingestion/otlp/metrics/inventory", produces = "application/json")
@Operation(summary = "List OTLP metric names")
public ResponseEntity<Message<OtlpMetricsInventory>> metricInventory(
@RequestParam(value = "start", required = false) Long start,
@RequestParam(value = "end", required = false) Long end,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "limit", defaultValue = "100") Integer limit) {
return workloadGuard.execute(Workload.METRICS, () -> ResponseEntity.ok(Message.success(
queryService.metricInventory(start, end, serviceName, serviceNamespace, environment, limit))));
}
@GetMapping(path = "/api/traces/list", produces = "application/json")
@Operation(summary = "Query traces")
public ResponseEntity<Message<SignalPage<TraceListItem>>> traces(
@RequestParam(value = "start", required = false) Long start,
@RequestParam(value = "end", required = false) Long end,
@RequestParam(value = "traceId", required = false) String traceId,
@RequestParam(value = "errorOnly", required = false) Boolean errorOnly,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "operationName", required = false) String operationName,
@RequestParam(value = "minDurationMs", required = false) Long minDurationMs,
@RequestParam(value = "maxDurationMs", required = false) Long maxDurationMs,
@RequestParam(value = "pageIndex", defaultValue = "0") Integer pageIndex,
@RequestParam(value = "pageSize", defaultValue = "20") Integer pageSize) {
return workloadGuard.execute(Workload.TRACES, () -> ResponseEntity.ok(Message.success(
queryService.queryTraces(start, end, traceId, errorOnly, serviceName, serviceNamespace,
environment, operationName, minDurationMs, maxDurationMs, pageIndex, pageSize))));
}
@GetMapping(path = "/api/traces/stats/overview", produces = "application/json")
@Operation(summary = "Get trace overview")
public ResponseEntity<Message<TraceOverview>> traceOverview(
@RequestParam(value = "start", required = false) Long start,
@RequestParam(value = "end", required = false) Long end,
@RequestParam(value = "traceId", required = false) String traceId,
@RequestParam(value = "errorOnly", required = false) Boolean errorOnly,
@RequestParam(value = "serviceName", required = false) String serviceName,
@RequestParam(value = "serviceNamespace", required = false) String serviceNamespace,
@RequestParam(value = "environment", required = false) String environment,
@RequestParam(value = "operationName", required = false) String operationName,
@RequestParam(value = "minDurationMs", required = false) Long minDurationMs,
@RequestParam(value = "maxDurationMs", required = false) Long maxDurationMs) {
return workloadGuard.execute(Workload.TRACES, () -> ResponseEntity.ok(Message.success(
queryService.traceOverview(start, end, traceId, errorOnly, serviceName, serviceNamespace,
environment, operationName, minDurationMs, maxDurationMs))));
}
@GetMapping(path = "/api/traces/{traceId}", produces = "application/json")
@Operation(summary = "Get trace detail")
public ResponseEntity<Message<TraceDetail>> traceDetail(@PathVariable("traceId") String traceId) {
return workloadGuard.execute(Workload.TRACES,
() -> ResponseEntity.ok(Message.success(queryService.traceDetail(traceId))));
}
@GetMapping(path = "/api/traces/{traceId}/spans", produces = "application/json")
@Operation(summary = "Get trace spans")
public ResponseEntity<Message<?>> traceSpans(@PathVariable("traceId") String traceId) {
return workloadGuard.execute(Workload.TRACES,
() -> ResponseEntity.ok(Message.success(queryService.traceDetail(traceId).spans())));
}
}
@@ -38,6 +38,7 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
@@ -51,6 +52,8 @@ import java.util.concurrent.atomic.AtomicLong;
public class LogSseManager {
private static final long BATCH_INTERVAL_MS = 200;
private static final long HEARTBEAT_INTERVAL_SECONDS = 15;
private static final long RECONNECT_TIME_MS = 5_000;
private static final int MAX_BATCH_SIZE = 1000;
private static final int MAX_QUEUE_SIZE = 10000;
@@ -70,6 +73,8 @@ public class LogSseManager {
public LogSseManager() {
scheduler.scheduleAtFixedRate(this::flushBatch, BATCH_INTERVAL_MS, BATCH_INTERVAL_MS, TimeUnit.MILLISECONDS);
scheduler.scheduleAtFixedRate(this::sendHeartbeat, HEARTBEAT_INTERVAL_SECONDS,
HEARTBEAT_INTERVAL_SECONDS, TimeUnit.SECONDS);
}
@PreDestroy
@@ -96,6 +101,13 @@ public class LogSseManager {
emitter.onError((ex) -> removeEmitter(clientId));
emitters.put(clientId, new SseSubscriber(emitter, filters));
try {
emitter.send(SseEmitter.event()
.comment("connected")
.reconnectTime(RECONNECT_TIME_MS));
} catch (IOException | IllegalStateException e) {
safeComplete(clientId, emitter);
}
return emitter;
}
@@ -167,6 +179,31 @@ public class LogSseManager {
}
}
void sendHeartbeat() {
for (Map.Entry<Long, SseSubscriber> entry : emitters.entrySet()) {
Long clientId = entry.getKey();
SseEmitter emitter = entry.getValue().emitter;
try {
senderPool.submit(() -> sendHeartbeat(clientId, emitter));
} catch (RejectedExecutionException e) {
if (!senderPool.isShutdown()) {
log.warn("Failed to schedule SSE heartbeat for client {}", clientId, e);
}
}
}
}
private void sendHeartbeat(Long clientId, SseEmitter emitter) {
try {
emitter.send(SseEmitter.event().comment("heartbeat"));
} catch (IOException | IllegalStateException e) {
safeComplete(clientId, emitter);
} catch (Exception e) {
log.error("Failed to send SSE heartbeat to client {}", clientId, e);
safeComplete(clientId, emitter);
}
}
private void safeComplete(Long clientId, SseEmitter emitter) {
try {
emitter.complete();
@@ -0,0 +1,29 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service;
import org.springframework.http.HttpHeaders;
import org.springframework.http.ResponseEntity;
/** Forwards validated OTLP payloads to the configured GreptimeDB instance. */
public interface OtlpSignalForwarder {
ResponseEntity<byte[]> forwardHttp(String signal, byte[] content, HttpHeaders requestHeaders);
byte[] forwardProtobuf(String signal, byte[] content);
}
@@ -0,0 +1,26 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service;
/** Raised when a signal workload exceeds its isolated concurrency budget. */
public class SignalQueryRejectedException extends RuntimeException {
public SignalQueryRejectedException(String workload) {
super("Signal workload is overloaded: " + workload);
}
}
@@ -0,0 +1,90 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service;
import java.time.Duration;
import java.util.EnumMap;
import java.util.Map;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import org.springframework.stereotype.Component;
/** Isolates signal reads and writes with bounded concurrency and a short bounded queue. */
@Component
public class SignalWorkloadGuard {
private static final Duration QUEUE_WAIT = Duration.ofMillis(50);
private final Map<Workload, Slot> slots = new EnumMap<>(Workload.class);
public SignalWorkloadGuard() {
this(Map.of(
Workload.METRICS, new Limits(8, 16),
Workload.LOG_LIST, new Limits(8, 16),
Workload.LOG_AGGREGATE, new Limits(4, 8),
Workload.TRACES, new Limits(6, 12),
Workload.OTLP_WRITE, new Limits(16, 32)));
}
SignalWorkloadGuard(Map<Workload, Limits> limits) {
limits.forEach((workload, value) -> slots.put(workload, new Slot(value)));
}
public <T> T execute(Workload workload, Supplier<T> operation) {
Slot slot = slots.get(workload);
if (slot == null || !slot.queue().tryAcquire()) {
throw new SignalQueryRejectedException(workload.name());
}
boolean acquired = false;
try {
acquired = slot.active().tryAcquire(QUEUE_WAIT.toMillis(), TimeUnit.MILLISECONDS);
if (!acquired) {
throw new SignalQueryRejectedException(workload.name());
}
return operation.get();
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new SignalQueryRejectedException(workload.name());
} finally {
slot.queue().release();
if (acquired) {
slot.active().release();
}
}
}
/** Independently limited signal workload families. */
public enum Workload {
METRICS,
LOG_LIST,
LOG_AGGREGATE,
TRACES,
OTLP_WRITE
}
/** Active and waiting request limits for a workload family. */
record Limits(int concurrency, int queueSize) {
}
private record Slot(Semaphore active, Semaphore queue) {
private Slot(Limits limits) {
this(new Semaphore(limits.concurrency(), true), new Semaphore(limits.queueSize(), true));
}
}
}
@@ -0,0 +1,47 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsConsole;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsInventory;
import org.apache.hertzbeat.common.entity.dto.observability.SignalPage;
import org.apache.hertzbeat.common.entity.dto.observability.TraceDetail;
import org.apache.hertzbeat.common.entity.dto.observability.TraceListItem;
import org.apache.hertzbeat.common.entity.dto.observability.TraceOverview;
/** Entity-free query boundary for OTLP metrics and traces. */
public interface ThreeSignalQueryService {
OtlpMetricsConsole queryMetrics(String query, Long start, Long end, String serviceName,
String serviceNamespace, String environment, String filter, String groupBy,
String aggregation, Integer step, String operationName);
OtlpMetricsInventory metricInventory(Long start, Long end, String serviceName, String serviceNamespace,
String environment, Integer limit);
SignalPage<TraceListItem> queryTraces(Long start, Long end, String traceId, Boolean errorOnly,
String serviceName, String serviceNamespace, String environment,
String operationName, Long minDurationMs, Long maxDurationMs,
Integer pageIndex, Integer pageSize);
TraceOverview traceOverview(Long start, Long end, String traceId, Boolean errorOnly, String serviceName,
String serviceNamespace, String environment, String operationName,
Long minDurationMs, Long maxDurationMs);
TraceDetail traceDetail(String traceId);
}
@@ -0,0 +1,219 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service.impl;
import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;
import io.opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest;
import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest;
import io.opentelemetry.proto.collector.trace.v1.ExportTraceServiceRequest;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.HexFormat;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.zip.GZIPInputStream;
import org.apache.hertzbeat.log.service.OtlpSignalForwarder;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.apache.hertzbeat.warehouse.constants.WarehouseConstants;
import org.apache.hertzbeat.warehouse.store.history.tsdb.greptime.GreptimeProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.client.RestTemplate;
/** Greptime native OTLP forwarder for metrics, logs, and traces. */
@Service
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
public class GreptimeOtlpSignalForwarder implements OtlpSignalForwarder {
private static final String PROTOBUF = "application/x-protobuf";
private static final String GREPTIME_DATABASE_HEADER = "X-Greptime-DB-Name";
private static final String GREPTIME_TRACE_TABLE_HEADER = "X-Greptime-Trace-Table-Name";
private static final String GREPTIME_PIPELINE_HEADER = "X-Greptime-Pipeline-Name";
private static final String GREPTIME_LOG_TABLE_HEADER = "X-Greptime-Log-Table-Name";
private static final String GREPTIME_LOG_PIPELINE_HEADER = "X-Greptime-Log-Pipeline-Name";
private static final String GREPTIME_PROMOTE_RESOURCE_HEADER =
"X-Greptime-OTLP-Metric-Promote-Resource-Attrs";
private static final String PROMOTED_RESOURCE_ATTRIBUTES = String.join(";", List.of(
"service.name", "service.namespace", "service.version", "deployment.environment.name",
"host.name", "k8s.namespace.name", "k8s.pod.name"));
private static final Set<String> SIGNALS = Set.of("metrics", "logs", "traces");
private final GreptimeProperties greptimeProperties;
private final RestTemplate restTemplate;
public GreptimeOtlpSignalForwarder(GreptimeProperties greptimeProperties,
@Qualifier(WarehouseConstants.GREPTIME_WRITE_REST_TEMPLATE)
RestTemplate restTemplate) {
this.greptimeProperties = greptimeProperties;
this.restTemplate = restTemplate;
}
@Override
public ResponseEntity<byte[]> forwardHttp(String signal, byte[] content, HttpHeaders requestHeaders) {
HttpHeaders safeHeaders = requestHeaders == null ? new HttpHeaders() : requestHeaders;
byte[] normalized = maybeDecompress(content, safeHeaders);
MediaType contentType = safeHeaders.getContentType();
byte[] protobuf = contentType != null && MediaType.APPLICATION_JSON.includes(contentType)
? jsonToProtobuf(signal, normalized) : validateProtobuf(signal, normalized);
byte[] response = forwardProtobuf(signal, protobuf);
if (contentType != null && MediaType.APPLICATION_JSON.includes(contentType)) {
return ResponseEntity.ok().contentType(MediaType.APPLICATION_JSON)
.body("{}".getBytes(StandardCharsets.UTF_8));
}
return ResponseEntity.ok().contentType(MediaType.parseMediaType(PROTOBUF)).body(response);
}
@Override
public byte[] forwardProtobuf(String signal, byte[] content) {
String normalizedSignal = normalizeSignal(signal);
byte[] validContent = validateProtobuf(normalizedSignal, content);
HttpHeaders headers = greptimeHeaders(normalizedSignal);
ResponseEntity<byte[]> response = restTemplate.exchange(
endpoint(greptimeProperties.httpEndpoint(), "/v1/otlp/v1/" + normalizedSignal),
HttpMethod.POST,
new HttpEntity<>(validContent, headers),
byte[].class);
if (!response.getStatusCode().is2xxSuccessful()) {
throw new IllegalStateException("GreptimeDB rejected OTLP " + normalizedSignal);
}
return response.getBody() == null ? new byte[0] : response.getBody();
}
private HttpHeaders greptimeHeaders(String signal) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.parseMediaType(PROTOBUF));
headers.setAccept(List.of(MediaType.parseMediaType(PROTOBUF)));
headers.set(GREPTIME_DATABASE_HEADER, StringUtils.hasText(greptimeProperties.database())
? greptimeProperties.database() : "public");
if ("metrics".equals(signal)) {
headers.set(GREPTIME_PROMOTE_RESOURCE_HEADER, PROMOTED_RESOURCE_ATTRIBUTES);
} else if ("traces".equals(signal)) {
headers.set(GREPTIME_TRACE_TABLE_HEADER, "hzb_traces");
headers.set(GREPTIME_PIPELINE_HEADER, "greptime_trace_v1");
} else {
headers.set(GREPTIME_LOG_TABLE_HEADER, WarehouseConstants.LOG_TABLE_NAME);
headers.set(GREPTIME_LOG_PIPELINE_HEADER, "hertzbeat_otlp_log_v1");
}
if (StringUtils.hasText(greptimeProperties.username()) && StringUtils.hasText(greptimeProperties.password())) {
String credentials = greptimeProperties.username() + ":" + greptimeProperties.password();
headers.setBasicAuth(Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8)));
}
return headers;
}
private byte[] jsonToProtobuf(String signal, byte[] content) {
Message.Builder builder = switch (normalizeSignal(signal)) {
case "metrics" -> ExportMetricsServiceRequest.newBuilder();
case "logs" -> ExportLogsServiceRequest.newBuilder();
case "traces" -> ExportTraceServiceRequest.newBuilder();
default -> throw new IllegalArgumentException("Unsupported OTLP signal");
};
try {
JsonFormat.parser().ignoringUnknownFields().merge(normalizeOtlpJson(content), builder);
return builder.build().toByteArray();
} catch (InvalidProtocolBufferException exception) {
throw new IllegalArgumentException("Malformed OTLP " + signal + " JSON payload", exception);
}
}
private String normalizeOtlpJson(byte[] content) {
String jsonText = new String(content, StandardCharsets.UTF_8);
if (!JsonUtil.isJsonStr(jsonText)) {
throw new IllegalArgumentException("Malformed OTLP JSON payload");
}
Object json = JsonUtil.fromJson(jsonText, Object.class);
if (json == null) {
throw new IllegalArgumentException("Malformed OTLP JSON payload");
}
normalizeIdBytes(json);
return JsonUtil.toJson(json);
}
@SuppressWarnings("unchecked")
private void normalizeIdBytes(Object value) {
if (value instanceof Map<?, ?> rawMap) {
Map<String, Object> map = (Map<String, Object>) rawMap;
for (Map.Entry<String, Object> entry : map.entrySet()) {
Object child = entry.getValue();
if (("traceId".equals(entry.getKey()) || "spanId".equals(entry.getKey())
|| "parentSpanId".equals(entry.getKey()))
&& child instanceof String id && id.matches("[0-9a-fA-F]+") && id.length() % 2 == 0) {
entry.setValue(Base64.getEncoder().encodeToString(HexFormat.of().parseHex(id)));
} else {
normalizeIdBytes(child);
}
}
} else if (value instanceof List<?> list) {
list.forEach(this::normalizeIdBytes);
}
}
private byte[] validateProtobuf(String signal, byte[] content) {
byte[] safeContent = content == null ? new byte[0] : content;
try {
switch (normalizeSignal(signal)) {
case "metrics" -> ExportMetricsServiceRequest.parseFrom(safeContent);
case "logs" -> ExportLogsServiceRequest.parseFrom(safeContent);
case "traces" -> ExportTraceServiceRequest.parseFrom(safeContent);
default -> throw new IllegalArgumentException("Unsupported OTLP signal");
}
return safeContent;
} catch (InvalidProtocolBufferException exception) {
throw new IllegalArgumentException("Malformed OTLP " + signal + " protobuf payload", exception);
}
}
private byte[] maybeDecompress(byte[] content, HttpHeaders headers) {
byte[] safeContent = content == null ? new byte[0] : content;
if (!"gzip".equalsIgnoreCase(headers.getFirst(HttpHeaders.CONTENT_ENCODING))) {
return safeContent;
}
try (GZIPInputStream input = new GZIPInputStream(new ByteArrayInputStream(safeContent));
ByteArrayOutputStream output = new ByteArrayOutputStream()) {
input.transferTo(output);
return output.toByteArray();
} catch (IOException exception) {
throw new IllegalArgumentException("Malformed gzip OTLP payload", exception);
}
}
private String normalizeSignal(String signal) {
String normalized = StringUtils.hasText(signal) ? signal.toLowerCase(Locale.ROOT) : "";
if (!SIGNALS.contains(normalized)) {
throw new IllegalArgumentException("Unsupported OTLP signal");
}
return normalized;
}
private String endpoint(String base, String path) {
return StringUtils.trimTrailingCharacter(base, '/') + path;
}
}
@@ -0,0 +1,523 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.service.impl;
import java.net.URI;
import java.time.Instant;
import java.time.format.DateTimeParseException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
import org.apache.hertzbeat.common.entity.dto.observability.MetricPoint;
import org.apache.hertzbeat.common.entity.dto.observability.MetricSeries;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsConsole;
import org.apache.hertzbeat.common.entity.dto.observability.OtlpMetricsInventory;
import org.apache.hertzbeat.common.entity.dto.observability.SignalPage;
import org.apache.hertzbeat.common.entity.dto.observability.TraceDetail;
import org.apache.hertzbeat.common.entity.dto.observability.TraceListItem;
import org.apache.hertzbeat.common.entity.dto.observability.TraceOverview;
import org.apache.hertzbeat.common.entity.dto.observability.TraceSpanEvent;
import org.apache.hertzbeat.common.entity.dto.observability.TraceSpanNode;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.apache.hertzbeat.common.support.exception.StorageUnavailableException;
import org.apache.hertzbeat.log.service.ThreeSignalQueryService;
import org.apache.hertzbeat.warehouse.db.GreptimeSqlQueryExecutor;
import org.apache.hertzbeat.warehouse.constants.WarehouseConstants;
import org.apache.hertzbeat.warehouse.store.history.tsdb.greptime.GreptimeProperties;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.util.StringUtils;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.RestClientException;
import org.springframework.web.util.UriComponentsBuilder;
import tools.jackson.core.type.TypeReference;
/** GreptimeDB implementation of the entity-free three-signal query contract. */
@Service
@ConditionalOnProperty(prefix = "warehouse.store.greptime", name = "enabled", havingValue = "true")
public class GreptimeThreeSignalQueryService implements ThreeSignalQueryService {
private static final String TRACE_TABLE = "hzb_traces";
private static final Pattern SAFE_IDENTIFIER = Pattern.compile("[A-Za-z_:][A-Za-z0-9_:.-]*");
private static final Pattern SAFE_LABEL = Pattern.compile("[A-Za-z_][A-Za-z0-9_]*");
private static final Set<String> AGGREGATIONS = Set.of("sum", "avg", "min", "max", "count");
private static final long DEFAULT_WINDOW_MILLIS = 30 * 60 * 1000L;
private static final int DEFAULT_STEP_SECONDS = 30;
private static final int MAX_PAGE_SIZE = 200;
private final GreptimeProperties greptimeProperties;
private final GreptimeSqlQueryExecutor sqlQueryExecutor;
private final RestTemplate restTemplate;
public GreptimeThreeSignalQueryService(GreptimeProperties greptimeProperties,
GreptimeSqlQueryExecutor sqlQueryExecutor,
@Qualifier(WarehouseConstants.GREPTIME_QUERY_REST_TEMPLATE)
RestTemplate restTemplate) {
this.greptimeProperties = greptimeProperties;
this.sqlQueryExecutor = sqlQueryExecutor;
this.restTemplate = restTemplate;
}
@Override
public OtlpMetricsConsole queryMetrics(String query, Long start, Long end, String serviceName,
String serviceNamespace, String environment, String filter,
String groupBy, String aggregation, Integer step, String operationName) {
validateTimeRange(start, end);
long effectiveEnd = end == null ? Instant.now().toEpochMilli() : end;
long effectiveStart = start == null ? effectiveEnd - DEFAULT_WINDOW_MILLIS : start;
int effectiveStep = step == null ? DEFAULT_STEP_SECONDS : Math.max(1, Math.min(step, 3600));
String effectiveQuery = buildPromql(query, serviceName, serviceNamespace, environment, filter,
groupBy, aggregation, operationName);
URI uri = UriComponentsBuilder.fromUriString(greptimeProperties.httpEndpoint())
.path("/v1/prometheus/api/v1/query_range")
.queryParam("db", greptimeProperties.database())
.queryParam("query", effectiveQuery)
.queryParam("start", effectiveStart / 1000.0)
.queryParam("end", effectiveEnd / 1000.0)
.queryParam("step", effectiveStep)
.build().encode().toUri();
Map<?, ?> response = getPrometheus(uri);
return new OtlpMetricsConsole(effectiveQuery, effectiveStart, effectiveEnd, effectiveStep,
readMetricSeries(response));
}
@Override
public OtlpMetricsInventory metricInventory(Long start, Long end, String serviceName, String serviceNamespace,
String environment, Integer limit) {
int effectiveLimit = Math.max(1, Math.min(limit == null ? 100 : limit, 500));
URI uri = UriComponentsBuilder.fromUriString(greptimeProperties.httpEndpoint())
.path("/v1/prometheus/api/v1/label/__name__/values")
.queryParam("db", greptimeProperties.database())
.build().encode().toUri();
Map<?, ?> response = getPrometheus(uri);
Object rawData = response == null ? null : response.get("data");
if (!(rawData instanceof List<?> values)) {
return new OtlpMetricsInventory(List.of());
}
List<String> names = values.stream().map(String::valueOf).filter(StringUtils::hasText)
.sorted().limit(effectiveLimit).toList();
return new OtlpMetricsInventory(names);
}
private Map<?, ?> getPrometheus(URI uri) {
HttpHeaders headers = new HttpHeaders();
if (StringUtils.hasText(greptimeProperties.username())
&& StringUtils.hasText(greptimeProperties.password())) {
headers.setBasicAuth(greptimeProperties.username(), greptimeProperties.password());
}
try {
ResponseEntity<Map> response = restTemplate.exchange(uri, HttpMethod.GET,
new HttpEntity<>(headers), Map.class);
Map<?, ?> body = response.getBody() == null ? Map.of() : response.getBody();
if ("error".equals(body.get("status"))) {
throw new IllegalArgumentException("Invalid PromQL query");
}
return body;
} catch (IllegalArgumentException exception) {
throw exception;
} catch (HttpClientErrorException.BadRequest exception) {
throw new IllegalArgumentException("Invalid PromQL query", exception);
} catch (RestClientException exception) {
throw new StorageUnavailableException("GreptimeDB metrics storage is unavailable", exception);
}
}
@Override
public SignalPage<TraceListItem> queryTraces(Long start, Long end, String traceId, Boolean errorOnly,
String serviceName, String serviceNamespace, String environment,
String operationName, Long minDurationMs, Long maxDurationMs,
Integer pageIndex, Integer pageSize) {
validateTraceRange(start, end, minDurationMs, maxDurationMs);
int effectivePage = Math.max(pageIndex == null ? 0 : pageIndex, 0);
int effectiveSize = Math.max(1, Math.min(pageSize == null ? 20 : pageSize, MAX_PAGE_SIZE));
String filters = traceFilters(start, end, traceId, serviceName, serviceNamespace, environment,
operationName, minDurationMs, maxDurationMs);
String errorExpression = "SUM(CASE WHEN span_status_code IN ('STATUS_CODE_ERROR', 'ERROR') "
+ "THEN 1 ELSE 0 END)";
String rootSpan = "parent_span_id IS NULL OR parent_span_id = ''";
StringBuilder grouped = new StringBuilder("SELECT trace_id, ")
.append(rootValue("span_id", "root_span_id", rootSpan))
.append(", ").append(rootValue("service_name", "service_name", rootSpan))
.append(", ").append(rootValue("\"resource_attributes.service.namespace\"", "service_namespace", rootSpan))
.append(", ").append(rootValue("\"resource_attributes.deployment.environment.name\"", "environment", rootSpan))
.append(", ").append(rootValue("span_name", "root_span_name", rootSpan))
.append(", ").append(rootValue("duration_nano", "duration_nano", rootSpan)).append(", ")
.append("MIN(timestamp) AS start_time, ").append(errorExpression).append(" AS error_span_count, ")
.append("MAX(span_id) AS sampled_span_id FROM ").append(TRACE_TABLE);
if (StringUtils.hasText(filters)) {
grouped.append(" WHERE ").append(filters);
}
grouped.append(" GROUP BY trace_id");
if (Boolean.TRUE.equals(errorOnly)) {
grouped.append(" HAVING ").append(errorExpression).append(" > 0");
}
String sql = "SELECT *, COUNT(*) OVER () AS total_count FROM (" + grouped
+ ") trace_page ORDER BY start_time DESC LIMIT " + effectiveSize
+ " OFFSET " + effectivePage * effectiveSize;
List<Map<String, Object>> rows = sqlQueryExecutor.execute(sql);
long total = rows.isEmpty() ? 0 : asLong(rows.getFirst().get("total_count"));
return new SignalPage<>(rows.stream().map(this::toTraceListItem).toList(), effectivePage, effectiveSize, total);
}
@Override
public TraceOverview traceOverview(Long start, Long end, String traceId, Boolean errorOnly, String serviceName,
String serviceNamespace, String environment, String operationName,
Long minDurationMs, Long maxDurationMs) {
validateTraceRange(start, end, minDurationMs, maxDurationMs);
String filters = traceFilters(start, end, traceId, serviceName, serviceNamespace, environment,
operationName, minDurationMs, maxDurationMs);
String errorExpression = "SUM(CASE WHEN span_status_code IN ('STATUS_CODE_ERROR', 'ERROR') "
+ "THEN 1 ELSE 0 END)";
StringBuilder grouped = new StringBuilder("SELECT trace_id, MAX(duration_nano) AS duration_nano, ")
.append(errorExpression).append(" AS error_span_count FROM ").append(TRACE_TABLE);
if (StringUtils.hasText(filters)) {
grouped.append(" WHERE ").append(filters);
}
grouped.append(" GROUP BY trace_id");
if (Boolean.TRUE.equals(errorOnly)) {
grouped.append(" HAVING ").append(errorExpression).append(" > 0");
}
String sql = "SELECT COUNT(*) AS total_count, "
+ "SUM(CASE WHEN error_span_count > 0 THEN 1 ELSE 0 END) AS error_count, "
+ "AVG(duration_nano) AS average_duration, approx_percentile_cont(duration_nano, 0.95) AS p95_duration "
+ "FROM (" + grouped + ") trace_overview";
List<Map<String, Object>> rows = sqlQueryExecutor.execute(sql);
if (rows.isEmpty()) {
return new TraceOverview(0, 0, 0, 0, 0);
}
Map<String, Object> row = rows.getFirst();
long total = asLong(row.get("total_count"));
long errors = asLong(row.get("error_count"));
return new TraceOverview(total, errors, total == 0 ? 0 : (double) errors / total,
asDouble(row.get("average_duration")) / 1_000_000.0,
asDouble(row.get("p95_duration")) / 1_000_000.0);
}
@Override
public TraceDetail traceDetail(String traceId) {
if (!StringUtils.hasText(traceId)) {
throw new IllegalArgumentException("traceId is required");
}
String sql = "SELECT * FROM " + TRACE_TABLE + " WHERE trace_id = '" + escapeSql(traceId)
+ "' ORDER BY timestamp ASC LIMIT 5000";
List<TraceSpanNode> spans = sqlQueryExecutor.execute(sql).stream().map(this::toSpan).toList();
if (spans.isEmpty()) {
return new TraceDetail(null, List.of());
}
TraceSpanNode root = spans.stream().filter(span -> !StringUtils.hasText(span.parentSpanId()))
.findFirst().orElse(spans.getFirst());
long start = spans.stream().mapToLong(TraceSpanNode::startTime).min().orElse(root.startTime());
long end = spans.stream().mapToLong(span -> span.startTime() + span.durationNanos() / 1_000_000L)
.max().orElse(start + root.durationNanos() / 1_000_000L);
long duration = Math.max(root.durationNanos(), Math.max(0, end - start) * 1_000_000L);
int errors = (int) spans.stream().filter(span -> isError(span.status())).count();
String namespace = root.resourceAttributes().getOrDefault("service.namespace", "");
TraceListItem summary = new TraceListItem(traceId, root.spanId(), root.serviceName(), namespace,
root.spanName(), duration, errors > 0 ? "ERROR" : "OK", start, errors,
root.resourceAttributes());
return new TraceDetail(summary, spans);
}
private String buildPromql(String query, String serviceName, String serviceNamespace, String environment,
String filter, String groupBy, String aggregation, String operationName) {
String metric = StringUtils.hasText(query) ? query.trim() : "up";
if (!SAFE_IDENTIFIER.matcher(metric).matches()) {
return metric;
}
Map<String, String> labels = new LinkedHashMap<>();
putIfText(labels, "service_name", serviceName);
putIfText(labels, "service_namespace", serviceNamespace);
putIfText(labels, "deployment_environment_name", environment);
putIfText(labels, "http_route", operationName);
parseFriendlyFilter(filter, labels);
String selector = metric + labels.entrySet().stream()
.map(entry -> entry.getKey() + "=\"" + escapePromql(entry.getValue()) + "\"")
.collect(java.util.stream.Collectors.joining(",", "{", "}"));
if (labels.isEmpty()) {
selector = metric;
}
String normalizedAggregation = StringUtils.hasText(aggregation)
? aggregation.toLowerCase(Locale.ROOT) : null;
if (normalizedAggregation == null || !AGGREGATIONS.contains(normalizedAggregation)) {
return selector;
}
if (StringUtils.hasText(groupBy) && SAFE_LABEL.matcher(groupBy).matches()) {
return normalizedAggregation + " by (" + groupBy + ") (" + selector + ")";
}
return normalizedAggregation + "(" + selector + ")";
}
private String rootValue(String column, String alias, String rootSpan) {
return "COALESCE(MAX(CASE WHEN " + rootSpan + " THEN " + column + " END), MAX(" + column + ")) AS " + alias;
}
private void parseFriendlyFilter(String filter, Map<String, String> labels) {
if (!StringUtils.hasText(filter)) {
return;
}
for (String clause : filter.split(",")) {
String[] pair = clause.split("=", 2);
if (pair.length != 2) {
continue;
}
String key = pair[0].trim().replace('.', '_');
String value = pair[1].trim().replaceAll("^\"|\"$", "");
if (SAFE_LABEL.matcher(key).matches() && StringUtils.hasText(value)) {
labels.put(key, value);
}
}
}
private List<MetricSeries> readMetricSeries(Map<?, ?> response) {
if (response == null || !(response.get("data") instanceof Map<?, ?> data)
|| !(data.get("result") instanceof List<?> results)) {
return List.of();
}
List<MetricSeries> series = new ArrayList<>();
for (Object result : results) {
if (!(result instanceof Map<?, ?> item)) {
continue;
}
Map<String, String> labels = new LinkedHashMap<>();
if (item.get("metric") instanceof Map<?, ?> metricLabels) {
metricLabels.forEach((key, value) -> labels.put(String.valueOf(key), String.valueOf(value)));
}
List<MetricPoint> points = new ArrayList<>();
if (item.get("values") instanceof List<?> values) {
for (Object rawPoint : values) {
if (rawPoint instanceof List<?> point && point.size() >= 2) {
points.add(new MetricPoint((long) (asDouble(point.get(0)) * 1000), asDouble(point.get(1))));
}
}
}
series.add(new MetricSeries(labels, points));
}
return series;
}
private String traceFilters(Long start, Long end, String traceId, String serviceName, String serviceNamespace,
String environment, String operationName, Long minDurationMs, Long maxDurationMs) {
List<String> filters = new ArrayList<>();
if (start != null) {
filters.add("timestamp >= to_timestamp_millis(" + start + ")");
}
if (end != null) {
filters.add("timestamp <= to_timestamp_millis(" + end + ")");
}
addTextFilter(filters, "trace_id", traceId);
addTextFilter(filters, "service_name", serviceName);
addTextFilter(filters, "span_name", operationName);
addFlattenedResourceFilter(filters, "resource_attributes.service.namespace", serviceNamespace);
addFlattenedResourceFilter(filters, "resource_attributes.deployment.environment.name", environment);
if (minDurationMs != null) {
filters.add("duration_nano >= " + Math.max(0, minDurationMs) * 1_000_000L);
}
if (maxDurationMs != null) {
filters.add("duration_nano <= " + Math.max(0, maxDurationMs) * 1_000_000L);
}
return String.join(" AND ", filters);
}
private void addTextFilter(List<String> filters, String column, String value) {
if (StringUtils.hasText(value)) {
filters.add(column + " = '" + escapeSql(value) + "'");
}
}
private void addFlattenedResourceFilter(List<String> filters, String column, String value) {
if (StringUtils.hasText(value) && !"all".equalsIgnoreCase(value)) {
filters.add("\"" + column + "\" = '" + escapeSql(value) + "'");
}
}
private TraceListItem toTraceListItem(Map<String, Object> row) {
int errors = (int) asLong(row.get("error_span_count"));
return new TraceListItem(asString(row.get("trace_id")), asString(row.get("root_span_id")),
asString(row.get("service_name")), asString(row.get("service_namespace")),
asString(row.get("root_span_name")), asLong(row.get("duration_nano")),
errors > 0 ? "ERROR" : "OK", asEpochMillis(row.get("start_time")), errors,
traceResourceAttributes(row));
}
private TraceSpanNode toSpan(Map<String, Object> row) {
return new TraceSpanNode(asString(row.get("trace_id")), asString(row.get("span_id")),
asString(row.get("parent_span_id")), asString(row.get("span_name")),
asString(row.get("service_name")), asString(row.get("span_status_code")),
asString(row.get("span_kind")), asString(row.get("span_status_message")),
asLong(row.get("duration_nano")), asEpochMillis(row.get("timestamp")),
traceResourceAttributes(row), traceSpanAttributes(row),
asSpanEvents(row.get("span_events")));
}
private Map<String, String> traceResourceAttributes(Map<String, Object> row) {
Map<String, String> attributes = new LinkedHashMap<>(asStringMap(row.get("resource_attributes")));
row.forEach((key, value) -> putFlattenedAttribute(attributes, key, value, "resource_attributes."));
putIfText(attributes, "service.namespace", asString(row.get("service_namespace")));
putIfText(attributes, "deployment.environment.name", asString(row.get("environment")));
return attributes;
}
private Map<String, String> traceSpanAttributes(Map<String, Object> row) {
Map<String, String> attributes = new LinkedHashMap<>(asStringMap(row.get("span_attributes")));
row.forEach((key, value) -> putFlattenedAttribute(attributes, key, value, "span_attributes."));
return attributes;
}
private void putFlattenedAttribute(Map<String, String> target, String key, Object value, String prefix) {
if (key.startsWith(prefix) && value != null) {
putIfText(target, key.substring(prefix.length()), asString(value));
}
}
private List<TraceSpanEvent> asSpanEvents(Object value) {
if (!StringUtils.hasText(asString(value)) && !(value instanceof List<?>)) {
return List.of();
}
List<Map<String, Object>> events;
if (value instanceof List<?> list) {
events = list.stream()
.filter(Map.class::isInstance)
.map(Map.class::cast)
.map(this::toStringKeyMap)
.toList();
} else {
events = JsonUtil.fromJson(asString(value), new TypeReference<>() { });
}
if (events == null) {
return List.of();
}
return events.stream().map(event -> new TraceSpanEvent(
asString(event.get("name")),
asString(event.getOrDefault("time", event.get("timeUnixNano"))),
asObjectMap(event.get("attributes")))).toList();
}
private Map<String, Object> toStringKeyMap(Map<?, ?> source) {
Map<String, Object> result = new LinkedHashMap<>();
source.forEach((key, item) -> result.put(String.valueOf(key), item));
return result;
}
private Map<String, Object> asObjectMap(Object value) {
if (value instanceof Map<?, ?> map) {
return toStringKeyMap(map);
}
return Collections.emptyMap();
}
private Map<String, String> asStringMap(Object value) {
if (value instanceof Map<?, ?> map) {
Map<String, String> result = new LinkedHashMap<>();
map.forEach((key, item) -> result.put(String.valueOf(key), String.valueOf(item)));
return result;
}
if (!StringUtils.hasText(asString(value))) {
return Collections.emptyMap();
}
Map<String, Object> parsed = JsonUtil.fromJson(asString(value), new TypeReference<>() { });
if (parsed == null) {
return Collections.emptyMap();
}
Map<String, String> result = new LinkedHashMap<>();
parsed.forEach((key, item) -> result.put(key, String.valueOf(item)));
return result;
}
private long asEpochMillis(Object value) {
if (value instanceof Number number) {
long timestamp = number.longValue();
return timestamp > 10_000_000_000_000L ? timestamp / 1_000_000L : timestamp;
}
String text = asString(value);
try {
return Instant.parse(text).toEpochMilli();
} catch (DateTimeParseException ignored) {
return asLong(value);
}
}
private long asLong(Object value) {
if (value instanceof Number number) {
return number.longValue();
}
try {
return (long) Double.parseDouble(asString(value));
} catch (NumberFormatException ignored) {
return 0;
}
}
private double asDouble(Object value) {
if (value instanceof Number number) {
return number.doubleValue();
}
try {
return Double.parseDouble(asString(value));
} catch (NumberFormatException ignored) {
return 0;
}
}
private String asString(Object value) {
return value == null ? "" : String.valueOf(value);
}
private boolean isError(String status) {
return "ERROR".equalsIgnoreCase(status) || "STATUS_CODE_ERROR".equalsIgnoreCase(status);
}
private void putIfText(Map<String, String> target, String key, String value) {
if (StringUtils.hasText(value) && !"all".equalsIgnoreCase(value)) {
target.put(key, value.trim());
}
}
private String escapeSql(String value) {
return value.replace("'", "''");
}
private String escapePromql(String value) {
return value.replace("\\", "\\\\").replace("\"", "\\\"");
}
private void validateTraceRange(Long start, Long end, Long minDurationMs, Long maxDurationMs) {
validateTimeRange(start, end);
if ((minDurationMs != null && minDurationMs < 0) || (maxDurationMs != null && maxDurationMs < 0)) {
throw new IllegalArgumentException("Trace duration must not be negative");
}
if (minDurationMs != null && maxDurationMs != null && minDurationMs > maxDurationMs) {
throw new IllegalArgumentException("Minimum trace duration must not exceed maximum duration");
}
}
private void validateTimeRange(Long start, Long end) {
if (start != null && end != null && start > end) {
throw new IllegalArgumentException("Query start time must not exceed end time");
}
}
}
@@ -0,0 +1,71 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: 2
processors:
- vrl:
source: |
.time_unix_nano = .Timestamp
.observed_time_unix_nano = .ObservedTimestamp
.trace_id = .TraceId
.span_id = .SpanId
.trace_flags = .TraceFlags
.severity_text = .SeverityText
.severity_number = .SeverityNumber
.body = .Body
.attributes = .LogAttributes
.resource = .ResourceAttributes
.instrumentation_scope = .InstrumentationScope
.dropped_attributes_count = .DroppedAttributesCount
.
- select:
fields:
- time_unix_nano
- observed_time_unix_nano
- trace_id
- span_id
- trace_flags
- severity_text
- severity_number
- body
- attributes
- resource
- instrumentation_scope
- dropped_attributes_count
transform:
- field: time_unix_nano
type: epoch, ns
index: timestamp
- field: observed_time_unix_nano
type: epoch, ns
- fields:
- trace_id
- span_id
type: string
index: skipping
- fields:
- attributes
- resource
- instrumentation_scope
type: json
- field: body
type: string
- field: severity_text
type: string
- fields:
- severity_number
- trace_flags
- dropped_attributes_count
type: int32
@@ -0,0 +1,32 @@
-- Licensed to the Apache Software Foundation (ASF) under one or more
-- contributor license agreements. See the NOTICE file distributed with
-- this work for additional information regarding copyright ownership.
-- The ASF licenses this file to You under the Apache License, Version 2.0
-- (the "License"); you may not use this file except in compliance with
-- the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- Schema must stay in sync with the transform section of
-- greptime/pipelines/hertzbeat_otlp_log_v1.yaml, which GreptimeDB would
-- otherwise use to auto-create this table on first ingestion.
CREATE TABLE IF NOT EXISTS hertzbeat_logs (
"time_unix_nano" TIMESTAMP(9) TIME INDEX,
"observed_time_unix_nano" TIMESTAMP(9) NULL,
"trace_id" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"span_id" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"trace_flags" INT NULL,
"severity_text" STRING NULL,
"severity_number" INT NULL,
"body" STRING NULL,
"attributes" JSON NULL,
"resource" JSON NULL,
"instrumentation_scope" JSON NULL,
"dropped_attributes_count" INT NULL
) WITH ('append_mode' = 'true');
@@ -0,0 +1,38 @@
-- Licensed to the Apache Software Foundation (ASF) under one or more
-- contributor license agreements. See the NOTICE file distributed with
-- this work for additional information regarding copyright ownership.
-- The ASF licenses this file to You under the Apache License, Version 2.0
-- (the "License"); you may not use this file except in compliance with
-- the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
CREATE TABLE IF NOT EXISTS hzb_traces (
"timestamp" TIMESTAMP(9) TIME INDEX,
"timestamp_end" TIMESTAMP(9) NULL,
"duration_nano" BIGINT UNSIGNED NULL,
"trace_id" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"span_id" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"parent_span_id" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"span_kind" STRING NULL,
"span_name" STRING NULL,
"span_status_code" STRING NULL,
"span_status_message" STRING NULL,
"trace_state" STRING NULL,
"scope_name" STRING NULL,
"scope_version" STRING NULL,
"service_name" STRING NULL SKIPPING INDEX WITH(granularity = '10240', type = 'BLOOM'),
"resource_attributes.service.namespace" STRING NULL,
"resource_attributes.deployment.environment.name" STRING NULL,
"resource_attributes" JSON NULL,
"span_attributes" JSON NULL,
"span_events" JSON NULL,
"span_links" JSON NULL,
PRIMARY KEY("service_name")
) WITH ('append_mode' = 'true', 'table_data_model' = 'greptime_trace_v1');
@@ -0,0 +1,95 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.config;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.util.List;
import java.util.Map;
import org.mockito.InOrder;
import org.apache.hertzbeat.warehouse.db.GreptimeSqlQueryExecutor;
import org.apache.hertzbeat.warehouse.store.history.tsdb.greptime.GreptimeProperties;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;
@ExtendWith(MockitoExtension.class)
class GreptimeSignalInitializerTest {
@Mock
private GreptimeSqlQueryExecutor sqlQueryExecutor;
@Mock
private RestTemplate restTemplate;
private GreptimeSignalInitializer initializer;
@BeforeEach
void setUp() {
initializer = new GreptimeSignalInitializer(
new GreptimeProperties(true, "127.0.0.1:4001", "http://127.0.0.1:4000",
"public", "greptime", "secret"),
sqlQueryExecutor,
restTemplate);
}
@Test
void shouldPrepareTraceSchemaPipelineAndReadiness() {
when(sqlQueryExecutor.execute(anyString())).thenAnswer(invocation ->
invocation.getArgument(0, String.class).startsWith("SELECT 1")
? List.of(Map.of("ready", 1)) : List.of());
when(restTemplate.exchange(anyString(), eq(HttpMethod.POST), any(HttpEntity.class), eq(String.class)))
.thenReturn(ResponseEntity.ok("ok"));
initializer.initialize();
InOrder sqlOrder = org.mockito.Mockito.inOrder(sqlQueryExecutor);
sqlOrder.verify(sqlQueryExecutor).execute(org.mockito.ArgumentMatchers.contains(
"CREATE TABLE IF NOT EXISTS hzb_traces"));
sqlOrder.verify(sqlQueryExecutor).execute(org.mockito.ArgumentMatchers.startsWith(
"ALTER TABLE hzb_traces ADD COLUMN IF NOT EXISTS \"resource_attributes.service.namespace\""));
sqlOrder.verify(sqlQueryExecutor).execute(org.mockito.ArgumentMatchers.startsWith(
"ALTER TABLE hzb_traces ADD COLUMN IF NOT EXISTS \"resource_attributes.deployment.environment.name\""));
sqlOrder.verify(sqlQueryExecutor).execute(org.mockito.ArgumentMatchers.contains(
"CREATE TABLE IF NOT EXISTS hertzbeat_logs"));
sqlOrder.verify(sqlQueryExecutor).execute("SELECT 1 AS ready");
verify(restTemplate).exchange(eq("http://127.0.0.1:4000/v1/pipelines/hertzbeat_otlp_log_v1"),
eq(HttpMethod.POST), any(HttpEntity.class), eq(String.class));
}
@Test
void shouldFailStartupWhenGreptimeIsUnavailable() {
when(sqlQueryExecutor.execute(anyString())).thenThrow(new IllegalStateException("offline"));
assertThatThrownBy(initializer::initialize)
.isInstanceOf(IllegalStateException.class)
.hasMessage("GreptimeDB is required for the three-signal release but initialization failed")
.hasCauseInstanceOf(IllegalStateException.class);
}
}
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.config;
import static org.assertj.core.api.Assertions.assertThat;
import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.ClassPathResource;
class GreptimeSignalPipelineTest {
@Test
void shouldStoreOtlpBodyAsStringAndAttributesAsJson() throws Exception {
String pipeline = new ClassPathResource("greptime/pipelines/hertzbeat_otlp_log_v1.yaml")
.getContentAsString(StandardCharsets.UTF_8);
assertThat(pipeline).contains("- field: body\n type: string");
assertThat(pipeline).doesNotContain("- body\n - attributes");
assertThat(pipeline).contains("- attributes\n - resource\n - instrumentation_scope\n type: json");
}
}
@@ -0,0 +1,107 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.config;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import io.grpc.ManagedChannel;
import io.grpc.Metadata;
import io.grpc.Server;
import io.grpc.ServerInterceptors;
import io.grpc.Status;
import io.grpc.StatusRuntimeException;
import io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder;
import io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder;
import io.grpc.stub.MetadataUtils;
import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest;
import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceResponse;
import io.opentelemetry.proto.collector.metrics.v1.MetricsServiceGrpc;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import org.apache.hertzbeat.common.security.OtlpAccessTokenValidator;
import org.apache.hertzbeat.log.service.OtlpSignalForwarder;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.log.service.SignalQueryRejectedException;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
class OtlpGrpcServerConfigTest {
private Server server;
private ManagedChannel channel;
@BeforeEach
void setUp() throws Exception {
OtlpSignalForwarder forwarder = mock(OtlpSignalForwarder.class);
SignalWorkloadGuard guard = mock(SignalWorkloadGuard.class);
OtlpAccessTokenValidator validator = token -> "probe-token".equals(token) ? null : "Invalid token";
when(guard.execute(eq(SignalWorkloadGuard.Workload.OTLP_WRITE), any())).thenAnswer(invocation -> {
Supplier<?> action = invocation.getArgument(1);
return action.get();
});
when(forwarder.forwardProtobuf(eq("metrics"), any())).thenReturn(new byte[0]);
server = NettyServerBuilder.forPort(0)
.addService(ServerInterceptors.intercept(
new OtlpGrpcServerConfig.MetricsService(forwarder, guard),
new OtlpGrpcServerConfig.BearerTokenInterceptor(validator)))
.build().start();
channel = NettyChannelBuilder.forAddress("127.0.0.1", server.getPort()).usePlaintext().build();
}
@AfterEach
void tearDown() throws Exception {
channel.shutdownNow().awaitTermination(5, TimeUnit.SECONDS);
server.shutdownNow().awaitTermination(5, TimeUnit.SECONDS);
}
@Test
void shouldAcceptAuthorizedOtlpGrpcMetricsRequest() {
Metadata headers = new Metadata();
headers.put(Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER), "Bearer probe-token");
ExportMetricsServiceResponse response = MetricsServiceGrpc.newBlockingStub(channel)
.withInterceptors(MetadataUtils.newAttachHeadersInterceptor(headers))
.export(ExportMetricsServiceRequest.getDefaultInstance());
assertThat(response).isEqualTo(ExportMetricsServiceResponse.getDefaultInstance());
}
@Test
void shouldRejectMissingOtlpGrpcBearerToken() {
assertThatThrownBy(() -> MetricsServiceGrpc.newBlockingStub(channel)
.export(ExportMetricsServiceRequest.getDefaultInstance()))
.isInstanceOfSatisfying(StatusRuntimeException.class,
exception -> assertThat(exception.getStatus().getCode())
.isEqualTo(Status.Code.UNAUTHENTICATED));
}
@Test
void shouldMapGrpcFailuresWithoutHidingClientAndCapacityErrors() {
assertThat(OtlpGrpcServerConfig.toGrpcStatus(new IllegalArgumentException("payload")))
.isEqualTo(Status.INVALID_ARGUMENT);
assertThat(OtlpGrpcServerConfig.toGrpcStatus(new SignalQueryRejectedException("OTLP_WRITE")))
.isEqualTo(Status.RESOURCE_EXHAUSTED);
assertThat(OtlpGrpcServerConfig.toGrpcStatus(new IllegalStateException("storage")))
.isEqualTo(Status.UNAVAILABLE);
}
}
@@ -18,20 +18,21 @@
package org.apache.hertzbeat.log.controller;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.hertzbeat.common.constants.CommonConstants;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.apache.hertzbeat.log.service.SignalWorkloadGuard;
import org.apache.hertzbeat.warehouse.store.history.tsdb.HistoryDataReader;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -57,8 +58,24 @@ class LogQueryControllerTest {
@BeforeEach
void setUp() {
this.logQueryController = new LogQueryController(historyDataReader);
this.mockMvc = MockMvcBuilders.standaloneSetup(logQueryController).build();
lenient().when(historyDataReader.supportsLogQuery()).thenReturn(true);
this.logQueryController = new LogQueryController(historyDataReader, new SignalWorkloadGuard());
this.mockMvc = MockMvcBuilders.standaloneSetup(logQueryController)
.setControllerAdvice(new SignalWorkloadExceptionHandler())
.build();
}
@Test
void shouldReturnFriendlyFailureWhenLogQueryUnsupported() throws Exception {
when(historyDataReader.supportsLogQuery()).thenReturn(false);
for (String path : List.of("/api/logs/list", "/api/logs/stats/overview",
"/api/logs/stats/trace-coverage", "/api/logs/stats/trend")) {
mockMvc.perform(MockMvcRequestBuilders.get(path))
.andExpect(status().isOk())
.andExpect(jsonPath("$.code").value((int) CommonConstants.FAIL_CODE))
.andExpect(jsonPath("$.msg").isNotEmpty());
}
}
@Test
@@ -86,11 +103,8 @@ class LogQueryControllerTest {
List<LogEntry> mockLogs = Arrays.asList(logEntry1, logEntry2);
when(historyDataReader.countLogsByMultipleConditions(anyLong(), anyLong(), any(),
any(), any(), any(), any())).thenReturn(2L);
when(historyDataReader.queryLogsByMultipleConditionsWithPagination(anyLong(), anyLong(),
any(), any(), any(), any(), any(), anyInt(), anyInt()))
.thenReturn(mockLogs);
when(historyDataReader.countObservabilityLogs(any())).thenReturn(2L);
when(historyDataReader.queryObservabilityLogs(any(), any(), any())).thenReturn(mockLogs);
mockMvc.perform(
MockMvcRequestBuilders
@@ -104,7 +118,6 @@ class LogQueryControllerTest {
.param("pageIndex", "0")
.param("pageSize", "20")
)
.andDo(print())
.andExpect(status().isOk())
.andExpect(jsonPath("$.code").value((int) CommonConstants.SUCCESS_CODE))
.andExpect(jsonPath("$.data.content").isArray())
@@ -126,11 +139,8 @@ class LogQueryControllerTest {
.build()
);
when(historyDataReader.countLogsByMultipleConditions(any(), any(), any(),
any(), any(), any(), any())).thenReturn(1L);
when(historyDataReader.queryLogsByMultipleConditionsWithPagination(any(), any(),
any(), any(), any(), any(), any(), eq(0), eq(20)))
.thenReturn(mockLogs);
when(historyDataReader.countObservabilityLogs(any())).thenReturn(1L);
when(historyDataReader.queryObservabilityLogs(any(), any(), any())).thenReturn(mockLogs);
mockMvc.perform(
MockMvcRequestBuilders
@@ -143,27 +153,36 @@ class LogQueryControllerTest {
}
@Test
void testOverviewStatsWithMixedSeverityLogs() throws Exception {
// Create logs with different severity levels according to OpenTelemetry standard
List<LogEntry> mockLogs = Arrays.asList(
// TRACE (1-4)
LogEntry.builder().severityNumber(2).build(),
// DEBUG (5-8)
LogEntry.builder().severityNumber(6).build(),
// INFO (9-12)
LogEntry.builder().severityNumber(9).build(),
LogEntry.builder().severityNumber(10).build(),
// WARN (13-16)
LogEntry.builder().severityNumber(14).build(),
// ERROR (17-20)
LogEntry.builder().severityNumber(17).build(),
LogEntry.builder().severityNumber(18).build(),
// FATAL (21-24)
LogEntry.builder().severityNumber(21).build()
);
void shouldBoundLogPaginationBeforeReadingStorage() throws Exception {
when(historyDataReader.countObservabilityLogs(any())).thenReturn(0L);
when(historyDataReader.queryObservabilityLogs(any(), any(), any())).thenReturn(List.of());
when(historyDataReader.queryLogsByMultipleConditions(any(), any(), any(),
any(), any(), any(), any())).thenReturn(mockLogs);
mockMvc.perform(MockMvcRequestBuilders.get("/api/logs/list")
.param("pageIndex", "-1")
.param("pageSize", "1000"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.data.number").value(0))
.andExpect(jsonPath("$.data.size").value(200));
verify(historyDataReader).queryObservabilityLogs(any(), eq(0), eq(200));
}
@Test
void shouldReturnBadRequestForInvalidResourceExpression() throws Exception {
when(historyDataReader.countObservabilityLogs(any()))
.thenThrow(new IllegalArgumentException("Invalid resource filter expression"));
mockMvc.perform(MockMvcRequestBuilders.get("/api/logs/list")
.param("resource", "bad-expression"))
.andExpect(status().isBadRequest())
.andExpect(jsonPath("$.code").value((int) CommonConstants.FAIL_CODE));
}
@Test
void testOverviewStatsWithMixedSeverityLogs() throws Exception {
when(historyDataReader.queryLogOverviewAggregate(any())).thenReturn(Map.of(
"totalCount", 8L, "traceCount", 1L, "debugCount", 1L, "infoCount", 2L,
"warnCount", 1L, "errorCount", 2L, "fatalCount", 1L));
mockMvc.perform(
MockMvcRequestBuilders
@@ -182,13 +201,7 @@ class LogQueryControllerTest {
@Test
void testOverviewStatsWithTimeRange() throws Exception {
List<LogEntry> mockLogs = Arrays.asList(
LogEntry.builder().severityNumber(9).build(),
LogEntry.builder().severityNumber(17).build()
);
when(historyDataReader.queryLogsByMultipleConditions(eq(1734005477000L), eq(1734005478000L),
any(), any(), any(), any(), any())).thenReturn(mockLogs);
when(historyDataReader.queryLogOverviewAggregate(any())).thenReturn(Map.of("totalCount", 2L));
mockMvc.perform(
MockMvcRequestBuilders
@@ -203,21 +216,8 @@ class LogQueryControllerTest {
@Test
void testTraceCoverageStats() throws Exception {
List<LogEntry> mockLogs = Arrays.asList(
// With both trace and span
LogEntry.builder().traceId("trace1").spanId("span1").build(),
LogEntry.builder().traceId("trace2").spanId("span2").build(),
// With trace only
LogEntry.builder().traceId("trace3").spanId("").build(),
// With span only
LogEntry.builder().traceId("").spanId("span4").build(),
// Without trace info
LogEntry.builder().traceId("").spanId("").build(),
LogEntry.builder().build() // null values
);
when(historyDataReader.queryLogsByMultipleConditions(any(), any(), any(),
any(), any(), any(), any())).thenReturn(mockLogs);
when(historyDataReader.queryLogOverviewAggregate(any())).thenReturn(Map.of("traceCoverage", Map.of(
"withTrace", 3L, "withoutTrace", 3L, "withSpan", 3L, "withBothTraceAndSpan", 2L)));
mockMvc.perform(
MockMvcRequestBuilders
@@ -233,18 +233,8 @@ class LogQueryControllerTest {
@Test
void testTrendStats() throws Exception {
// Create logs with timestamps that fall into different hours
List<LogEntry> mockLogs = Arrays.asList(
// 2023-12-12 10:00 (1734005477630000000L nano = 1734005477630L ms)
LogEntry.builder().timeUnixNano(1734005477630000000L).build(),
// Same hour
LogEntry.builder().timeUnixNano(1734005477640000000L).build(),
// Next hour: 2023-12-12 11:00 (1734009077630000000L nano = 1734009077630L ms)
LogEntry.builder().timeUnixNano(1734009077630000000L).build()
);
when(historyDataReader.queryLogsByMultipleConditions(any(), any(), any(),
any(), any(), any(), any())).thenReturn(mockLogs);
when(historyDataReader.queryLogTrendAggregate(any())).thenReturn(Map.of(
"2023-12-12 10:00", 2L, "2023-12-12 11:00", 1L));
mockMvc.perform(
MockMvcRequestBuilders
@@ -257,13 +247,7 @@ class LogQueryControllerTest {
@Test
void testTrendStatsWithNullTimestamp() throws Exception {
List<LogEntry> mockLogs = Arrays.asList(
LogEntry.builder().timeUnixNano(1734005477630000000L).build(),
LogEntry.builder().timeUnixNano(null).build() // This should be filtered out
);
when(historyDataReader.queryLogsByMultipleConditions(any(), any(), any(),
any(), any(), any(), any())).thenReturn(mockLogs);
when(historyDataReader.queryLogTrendAggregate(any())).thenReturn(Map.of("2023-12-12 10:00", 1L));
mockMvc.perform(
MockMvcRequestBuilders
@@ -17,9 +17,11 @@
package org.apache.hertzbeat.log.controller;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyLong;
import static org.mockito.Mockito.verify;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.request;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
import org.apache.hertzbeat.log.notice.LogSseFilterCriteria;
@@ -35,6 +37,7 @@ import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.MediaType;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.MvcResult;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
/**
@@ -75,6 +78,28 @@ class LogSseControllerTest {
Assertions.assertNull(capturedCriteria.getSpanId());
}
@Test
void testSubscribeCommitsInitialEvent() throws Exception {
LogSseManager realEmitterManager = new LogSseManager();
try {
MockMvc realMockMvc = MockMvcBuilders
.standaloneSetup(new LogSseController(realEmitterManager))
.build();
MvcResult result = realMockMvc.perform(get("/api/logs/sse/subscribe")
.accept(MediaType.TEXT_EVENT_STREAM_VALUE))
.andExpect(status().isOk())
.andExpect(request().asyncStarted())
.andReturn();
assertTrue(result.getResponse().isCommitted());
assertTrue(result.getResponse().getContentAsString().contains(":connected"));
assertTrue(result.getResponse().getContentAsString().contains("retry:5000"));
} finally {
realEmitterManager.shutdown();
}
}
@Test
void testSubscribeWithMultipleFilters() throws Exception {
// Given: Multiple filter parameters
@@ -110,4 +135,4 @@ class LogSseControllerTest {
.accept(MediaType.TEXT_EVENT_STREAM_VALUE))
.andExpect(status().is4xxClientError());
}
}
}
@@ -0,0 +1,85 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.log.controller;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
import org.apache.hertzbeat.common.constants.CommonConstants;
import org.apache.hertzbeat.log.service.ThreeSignalQueryService;
import org.apache.hertzbeat.warehouse.store.history.tsdb.HistoryDataReader;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
/**
* Unit test for {@link ObservabilityCapabilityController}
*/
@ExtendWith(MockitoExtension.class)
class ObservabilityCapabilityControllerTest {
private MockMvc mockMvc;
@Mock
private HistoryDataReader historyDataReader;
@Mock
private ObjectProvider<ThreeSignalQueryService> threeSignalQueryService;
@Mock
private ThreeSignalQueryService queryService;
@BeforeEach
void setUp() {
this.mockMvc = MockMvcBuilders.standaloneSetup(
new ObservabilityCapabilityController(historyDataReader, threeSignalQueryService)).build();
}
@Test
void shouldReportAllUnsupportedWithoutCapableStorage() throws Exception {
when(historyDataReader.supportsLogQuery()).thenReturn(false);
when(threeSignalQueryService.getIfAvailable()).thenReturn(null);
mockMvc.perform(MockMvcRequestBuilders.get("/api/observability/capability"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.code").value((int) CommonConstants.SUCCESS_CODE))
.andExpect(jsonPath("$.data.log").value(false))
.andExpect(jsonPath("$.data.trace").value(false))
.andExpect(jsonPath("$.data.metric").value(false));
}
@Test
void shouldReportAllSupportedWithCapableStorage() throws Exception {
when(historyDataReader.supportsLogQuery()).thenReturn(true);
when(threeSignalQueryService.getIfAvailable()).thenReturn(queryService);
mockMvc.perform(MockMvcRequestBuilders.get("/api/observability/capability"))
.andExpect(status().isOk())
.andExpect(jsonPath("$.code").value((int) CommonConstants.SUCCESS_CODE))
.andExpect(jsonPath("$.data.log").value(true))
.andExpect(jsonPath("$.data.trace").value(true))
.andExpect(jsonPath("$.data.metric").value(true));
}
}

Some files were not shown because too many files have changed in this diff Show More