Compare commits

...
Author SHA1 Message Date
Yang Chen 1fb359a428 Merge branch 'master' into update-38e39e 2025-12-19 14:13:52 +08:00
Logic ad9a4ab100 [improve] xugu.md and app-xugu.yml (#3918) 2025-12-17 18:51:58 +08:00
KOYR a806ca1319 [Feature] Enhance draggable UI elements in alert settings (#3915) (#3917) 2025-12-17 09:33:23 +08:00
tomsun28 a3e3f754b2 chore: update docker compose hertzbeat version
Signed-off-by: tomsun28 <tomsun28@outlook.com>
2025-12-17 00:08:01 +08:00
P_PeacefulandTomsun28 aac5bafe43 [bugfix]: AI conversation message loading issue (#3911)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-16 23:54:38 +08:00
P_PeacefulandLogic 668106d445 [bugfix] AI prompt classpath error (#3912)
Co-authored-by: Logic <zqr10159@dromara.org>
2025-12-16 11:36:23 +08:00
a74671abdd [improve]Improve UI handling for label overflow (#3913)
Co-authored-by: 半半 <kongranyi@cai-inc.com>
Co-authored-by: Logic <zqr10159@dromara.org>
2025-12-16 11:03:15 +08:00
Duansg 120d494c47 [fix]Fixed private collector tasks not taking effect (#3914) 2025-12-16 10:51:39 +08:00
DeleiGuo 5b1d01e0c5 [bugfix] Modify the search predicate from 'host' to 'instance' field (#3910) 2025-12-13 10:25:52 +08:00
Yang ChenandCopilot 950138a340 [feat] Add an SQL editor and prevent SQL injection (#3900)
Signed-off-by: Yang Chen <1597081640@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-12 17:52:31 +08:00
zhaoyangplus 93d8f92165 feat(db): Added support for monitoring the Xugu database (#3899) 2025-12-12 11:32:40 +08:00
LogicandCopilot 488df340a1 [Fix]: Use a connection pool to avoid concurrent access to duckdb files. (#3898)
Signed-off-by: Logic <zqr10159@126.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-10 15:38:30 +08:00
DuansgandTomsun28 dc51ce7bef [fix]Fixed position offset issue during deserialization of ArrowUtil (#3897)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-09 21:46:36 +08:00
Logicandaias00 862f32e699 [Feature]: enhance logging configuration with asynchronous console appender (#3895)
Co-authored-by: aias00 <liuhongyu@apache.org>
2025-12-09 21:15:42 +08:00
DeleiGuo d0f6a04661 [docs] Add new committer blog (#3894) 2025-12-09 09:46:19 +08:00
shownandTomsun28 cb3fad6a2c feat: optimize hertzbeat ai module (#3891)
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
Signed-off-by: shown <yuluo08290126@gmail.com>
Signed-off-by: tomsun28 <tomsun28@outlook.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-08 22:55:44 +08:00
Tomsun28andshown c88c232889 feat: support mcp streamable http protocol and upgrade spring ai version (#3890)
Signed-off-by: tomsun28 <tomsun28@outlook.com>
Co-authored-by: shown <yuluo08290126@gmail.com>
2025-12-08 22:12:35 +08:00
6b03e4bb5f chore: add .editorconfig for consistent coding style (#3875)
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
2025-12-07 00:02:07 +08:00
7727d63451 [add] Add parameter validators and their corresponding tests (#3888)
Signed-off-by: aias00 <liuhongyu@apache.org>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-06 19:34:47 +08:00
Duansg 3e567e5be6 [improve]Improve issues related to tag conflicts (#3889) 2025-12-05 22:02:43 +08:00
Tomsun28andaias00 4ae6ba9a3c feat: update add monitors pop selected menus modal (#3878)
Signed-off-by: tomsun28 <tomsun28@outlook.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
2025-12-04 23:55:10 +08:00
360abcc61d [feature] Using duckdb instead of jpa to store history data (#3883)
Signed-off-by: Logic <zqr10159@126.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-04 23:51:04 +08:00
P_PeacefulandTomsun28 41e57db032 [bugfix] notice policy lost Gotify receiver type (#3886)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-03 22:44:20 +08:00
DuansgandTomsun28 b62137d185 [improve]Improve performance issues caused by too many metrics (#3879)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-03 22:30:26 +08:00
DuansgandTomsun28 2bd635e1d4 [fix]Fix some systemic issues (#3884)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-12-03 22:25:02 +08:00
Yang Chen 8e8ab80aa0 Revert "[refractor] Using Spring jdbc to query Greptime log" (#3885) 2025-12-03 12:32:45 +08:00
LogicandYang Chen 7f4c2e0487 [refractor] Using Spring jdbc to query Greptime log (#3880)
Signed-off-by: Yang Chen <1597081640@qq.com>
Co-authored-by: Yang Chen <1597081640@qq.com>
2025-12-03 12:11:08 +08:00
e52e715a53 improvement: optimize log stream for high TPS scenarios (#3876)
Signed-off-by: Yang Chen <1597081640@qq.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Duansg <siguoduan@gmail.com>
2025-12-03 11:18:49 +08:00
e3e7377d86 [refactor] Refactor Monitor host field to instance and update related logic (#3863)
Signed-off-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-01 21:45:59 +08:00
P_PeacefulandTomsun28 625dbe9624 [bugfix] redis cluster monitor master-slave relationship is inconsistent (#3874)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-11-30 19:39:40 +08:00
Duansgandaias00 42f0a74edf [fix] adding trusted domain verification (#3866)
Co-authored-by: aias00 <liuhongyu@apache.org>
2025-11-27 20:38:00 +08:00
Tomsun28 cc9f53d71a chore: update sureness yml (#3870) 2025-11-27 20:10:46 +08:00
DeleiGuo 2f122dfb05 [docs] fix the missing documents references in the sidebar (#3867) 2025-11-26 16:52:43 +08:00
f7b0010180 fix: Remove redundant metrics data send in KafkaCommonDataQueue (#3854)
Co-authored-by: Calvin <zhengqiwei@apache.org>
Co-authored-by: Duansg <siguoduan@gmail.com>
2025-11-24 22:11:49 +08:00
2c0fecaa62 [improve] Added simple exponential backoff strategy (#3860)
Signed-off-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Yang Chen <1597081640@qq.com>
2025-11-21 16:28:21 +08:00
548966906c [bugfix] align tooltip position for mute icon with other icons (#3862)
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-11-20 22:22:56 +08:00
6473546f56 [feat] Add DB2 database monitoring support (#3857)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: Calvin <zhengqiwei@apache.org>
Co-authored-by: Duansg <siguoduan@gmail.com>
2025-11-20 22:16:52 +08:00
035d63c880 [feature] Support FeiShu/Lark App Alert Notification (#3856)
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: Calvin <zhengqiwei@apache.org>
2025-11-20 00:14:57 +08:00
Yang ChenandCalvin d56176dfa5 feat: add new docker-compose deployment solution (#3859)
Co-authored-by: Calvin <zhengqiwei@apache.org>
2025-11-19 23:49:13 +08:00
Yang ChenandCalvin c1815f5107 fix: allow access to /log/** path to resolve unauthorized error on refresh (#3858)
Co-authored-by: Calvin <zhengqiwei@apache.org>
2025-11-18 01:33:29 +08:00
7ac175e726 [Improve]Unify UI border-radius to 6px for a more professional look (#3853)
Co-authored-by: Duansg <siguoduan@gmail.com>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
2025-11-17 23:02:21 +08:00
eb5e843251 [feat] Add webhook authentication support for alert notifications (#3797)
Co-authored-by: Sherlock Yin <sherlock.yin1994@gmail.com>
Co-authored-by: yunfan24 <yunfan24@outlook.com>
Co-authored-by: aias00 <liuhongyu@apache.org>
Co-authored-by: Calvin <zhengqiwei@apache.org>
Co-authored-by: kangli <likang@apache.org>
Co-authored-by: 刘俊辉 <liu.junhui3@iwhalecloud.com>
2025-11-17 22:59:35 +08:00
starryCoderandDuansg bc098aa4d2 [bugfix]manager: Avoid NPE when metrics data queue is empty (#3848)
Co-authored-by: Duansg <siguoduan@gmail.com>
2025-11-14 16:09:49 +08:00
Tomsun28 bcd394b47f [docs] update doc fix wrong manager (#3850)
Signed-off-by: tomsun28 <tomsun28@outlook.com>
2025-11-08 20:33:42 +08:00
Duansg fd2202fcaf [doc] add new committer blog (#3847) 2025-11-06 21:37:17 +08:00
a7629e317a Modify the issue of incorrect signature in Alibaba's SMS sending (#3788)
Co-authored-by: Logic <zqr10159@dromara.org>
Co-authored-by: Tomsun28 <tomsun28@outlook.com>
Co-authored-by: shown <yuluo08290126@gmail.com>
Co-authored-by: yunfan24 <yunfan24@outlook.com>
Co-authored-by: Calvin <zhengqiwei@apache.org>
Co-authored-by: aias00 <liuhongyu@apache.org>
Co-authored-by: Duansg <siguoduan@gmail.com>
2025-11-06 10:38:48 +08:00
DeleiGuoandshown 3a378f8710 [feature] support the security settings for robot signatures in DingTalk (#3841)
Co-authored-by: shown <yuluo08290126@gmail.com>
2025-11-03 11:43:08 +08:00
Tomsun28 cf59907fcd fix the labels limit error (#3842)
Signed-off-by: tomsun28 <tomsun28@outlook.com>
2025-11-02 20:56:21 +08:00
291 changed files with 10321 additions and 2833 deletions
+3 -4
View File
@@ -20,9 +20,11 @@ github:
homepage: https://hertzbeat.apache.org/
labels:
- ai
- agent
- llm
- monitoring
- logs
- monitor
- monitoring
- notifications
- alerting
- self-hosted
@@ -34,12 +36,9 @@ github:
- uptime
- status
- status-page
- devops
- server
- linux
- database
- mysql
- cloud
enabled_merge_buttons:
squash: true
merge: false
+55
View File
@@ -0,0 +1,55 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# EditorConfig is awesome: https://EditorConfig.org
# top-most EditorConfig file
root = true
[*]
charset = utf-8
end_of_line = lf
indent_style = space
# The line max length is 120, match checkstyle rules.
max_line_length = 120
insert_final_newline = true
trim_trailing_whitespace = true
[*.json]
tab_width = 2
indent_size = 2
[*.{yml,yaml}]
tab_width = 2
indent_size = 2
[*.xml]
ij_xml_attribute_wrap = off
ij_xml_text_wrap = off
ij_xml_keep_blank_lines = 1
[pom.xml]
indent_size = 2
[*.java]
indent_size = 4
tab_width = 4
[*.js]
indent_size = 2
[*.ts]
indent_size = 2
+10 -10
View File
@@ -90,24 +90,24 @@ e2e:
backend:
- changed-files:
- any-glob-to-any-file:
- "alerter/**/*"
- "collector/**/*"
- "common/**/*"
- "grafana/**/*"
- "manager/**/*"
- "push/**/*"
- "remoting/**/*"
- "warehouse/**/*"
- "hertzbeat-alerter/**/*"
- "hertzbeat-collector/**/*"
- "hertzbeat-common/**/*"
- "hertzbeat-grafana/**/*"
- "hertzbeat-manager/**/*"
- "hertzbeat-push/**/*"
- "hertzbeat-remoting/**/*"
- "hertzbeat-warehouse/**/*"
collector:
- changed-files:
- any-glob-to-any-file:
- "collector/**/*"
- "hertzbeat-collector/**/*"
monitoring-template:
- changed-files:
- any-glob-to-any-file:
- "manager/src/main/resources/define/**/*"
- "hertzbeat-manager/src/main/resources/define/**/*"
HIP:
- changed-files:
+3 -3
View File
@@ -4,7 +4,7 @@ image:
tasks:
- name: Run backend
before: cd hertzbeat-manager
before: cd hertzbeat-startup
command: |
gp sync-await setup-backend
mvn spring-boot:run
@@ -13,7 +13,7 @@ tasks:
before: cd web-app
command: |
gp sync-await setup-frontend
yarn start --public-host "`gp url 4200`"
pnpm start --public-host "`gp url 4200`"
openMode: split-right
- name: Setup backend
@@ -25,7 +25,7 @@ tasks:
- name: Setup frontend
init: |
cd web-app && yarn install
cd web-app && pnpm install
command: |
gp sync-done setup-frontend
exit 0
+1
View File
@@ -39,6 +39,7 @@ header:
- '**/*.js'
- '**/*.less'
- '**/*.txt'
- '**/*.st'
- '**/target/**'
- '.gitattributes'
- '**/.gitignore'
+1 -1
View File
@@ -31,7 +31,7 @@
"MD025": true,
"MD029": true,
"MD033": {
"allowed_elements": ["table", "tr", "td", "a", "img", "sub", "b", "br", "img", "tbody", "mark", "font"]
"allowed_elements": ["table", "tr", "td", "a", "img", "sub", "b", "br", "img", "tbody", "mark", "font", "Tabs", "TabItem"]
},
"MD036": false,
"MD040": true,
+8 -30
View File
@@ -20,27 +20,27 @@
- 💬**Suggestion**: You can also make some suggestions to the project or community to promote the healthy development of the community;
More see [Contribution Types](https://allcontributors.org/docs/en/emoji-key)
More see [Contribution Types](https://allcontributors.org/emoji-key)
Even small corrections to typos are very welcome :)
### Getting HertzBeat up and running
> To get HertzBeat code running on your development tools, and able to debug with breakpoints.
> This is a front-end and back-end separation project. To start the local code, the back-end [manager](manager) and the front-end [web-app](web-app) must be started separately.
> This is a front-end and back-end separation project. To start the local code, the back-end [startup](hertzbeat-startup) and the front-end [web-app](web-app) must be started separately.
#### Backend start
1. Requires `maven3+`, `java17` and `lombok` environments
2. (Optional) Modify the configuration file: `manager/src/main/resources/application.yml`
2. (Optional) Modify the configuration file: `hertzbeat-startup/src/main/resources/application.yml`
3. Execute under the project root directory: `mvn clean install -DskipTests`
4. Add VM Options: `--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`
5. Start `springboot manager` service: `manager/src/main/java/org/apache/hertzbeat/hertzbeat-manager/Manager.java`
5. Start `springboot startup` service: `hertzbeat-startup/src/main/java/org/apache/hertzbeat/startup/HertzBeatApplication.java`
#### Frontend start
@@ -131,17 +131,6 @@ Add WeChat account `ahertzbeat` to pull you into the WeChat group.
## 🥐 Architecture
- **[manager](https://github.com/apache/hertzbeat/tree/master/hertzbeat-manager)** Provide monitoring management, system management basic services.
> Provides monitoring management, monitoring configuration management, system user management, etc.
- **[collector](https://github.com/apache/hertzbeat/tree/master/collector)** Provide metrics data collection services.
> Use common protocols to remotely collect and obtain peer-to-peer metrics data.
- **[warehouse](https://github.com/apache/hertzbeat/tree/master/warehouse)** Provide monitoring data warehousing services.
> Metrics data management, data query, calculation and statistics.
- **[alerter](https://github.com/apache/hertzbeat/tree/master/hertzbeat-alerter)** Provide alert service.
> Alarm calculation trigger, monitoring status linkage, alarm configuration, and alarm notification.
- **[web-app](https://github.com/apache/hertzbeat/tree/master/web-app)** Provide web ui.
> Angular Web UI.
![hertzBeat](home/static/img/docs/hertzbeat-arch.png)
<br>
@@ -172,26 +161,26 @@ Add WeChat account `ahertzbeat` to pull you into the WeChat group.
- 💬**建议**:也可以对项目或者社区提出一些建议,促进社区的良性发展;
更多贡献方式参见 [Contribution Types](https://allcontributors.org/docs/en/emoji-key)
更多贡献方式参见 [Contribution Types](https://allcontributors.org/emoji-key)
即便是小到错别字的修正我们也都非常欢迎 :)
### 让 HertzBeat 运行起来
> 让 HertzBeat 的代码在您的开发工具上运行起来,并且能够断点调试。
> 此为前后端分离项目,本地代码启动需将后端[manager](manager)和前端[web-app](web-app)分别启动生效。
> 此为前后端分离项目,本地代码启动需将后端[hertzbeat-startup](hertzbeat-startup)和前端[web-app](web-app)分别启动生效。
#### 后端启动
1. 需要 `maven3+`, `java17` 和 `lombok` 环境
2. (可选)修改配置文件配置信息-`manager/src/main/resources/application.yml`
2. (可选)修改配置文件配置信息-`hertzbeat-startup/src/main/resources/application.yml`
3. 在项目根目录运行编译: `mvn clean install -DskipTests`
4. 在 `jvm` 加入参数 `--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`
5. 启动`springboot manager`服务 `manager/src/main/java/org/apache/hertzbeat/hertzbeat-manager/Manager.java`
5. 启动`springboot startup`服务 `hertzbeat-startup/src/main/java/org/apache/hertzbeat/startup/HertzBeatApplication.java`
#### 前端启动
@@ -282,15 +271,4 @@ Add WeChat account `ahertzbeat` to pull you into the WeChat group.
### 模块
- **[manager](https://github.com/apache/hertzbeat/tree/master/hertzbeat-manager)** 提供监控管理,系统管理基础服务
> 提供对监控的管理,监控应用配置的管理,系统用户租户后台管理等。
- **[collector](https://github.com/apache/hertzbeat/tree/master/collector)** 提供监控数据采集服务
> 使用通用协议远程采集获取对端指标数据。
- **[warehouse](https://github.com/apache/hertzbeat/tree/master/warehouse)** 提供监控数据仓储服务
> 采集指标结果数据管理,数据落盘,查询,计算统计。
- **[alerter](https://github.com/apache/hertzbeat/tree/master/hertzbeat-alerter)** 提供告警服务
> 告警计算触发,任务状态联动,告警配置,告警通知。
- **[web-app](https://github.com/apache/hertzbeat/tree/master/web-app)** 提供可视化控制台页面
> 监控告警系统可视化控制台前端
![hertzBeat](home/static/img/docs/hertzbeat-arch.png)
+2 -2
View File
@@ -155,8 +155,8 @@ Detailed config refer to [Install HertzBeat via Package](https://hertzbeat.apach
##### 3Start via source code
1. Local source code debugging needs to start the back-end project `manager` and the front-end project `web-app`.
2. Backendneed `maven3+`, `java17`, `lombok`, add VM options in IDE: ` --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED `, then start the `manager` service.
1. Local source code debugging needs to start the back-end project `hertzbeat-startup` and the front-end project `web-app`.
2. Backendneed `maven3+`, `java17`, `lombok`, add VM options in IDE: ` --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED `, then start the `hertzbeat-startup` service.
3. Webneed `nodejs npm angular-cli` environment, Run `ng serve --open` in `web-app` directory after backend startup.
4. Access `http://localhost:4200` to start, default account: `admin/hertzbeat`
+2 -2
View File
@@ -152,8 +152,8 @@
##### 方式三:本地代码启动
1. 此为前后端分离项目,本地代码调试需要分别启动后端工程 `manager` 和前端工程 `web-app`
2. 后端:需要 `maven3+`, `java17` 和 `lombok` 环境,修改 `YML` 配置信息,添加JVM参数`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`后启动 `manager` 服务即可。
1. 此为前后端分离项目,本地代码调试需要分别启动后端工程 `hertzbeat-startup` 和前端工程 `web-app`
2. 后端:需要 `maven3+`, `java17` 和 `lombok` 环境,修改 `YML` 配置信息,添加JVM参数`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`后启动 `hertzbeat-startup` 服务即可。
3. 前端:需要 `nodejs npm angular-cli`环境,待本地后端启动后,在 `web-app` 目录下启动 `ng serve --open`
4. 浏览器访问 `http://localhost:4200` 即可开始,默认账号密码 `admin/hertzbeat`
+2 -2
View File
@@ -155,8 +155,8 @@
##### 方式3:ローカルの実行
1. ローカルの実行には、バックエンドのプロジェクト`manager`とフロントエンドのプロジェクト`web-app`を起動する必要があります。
2. バックエンド:`maven3+`、`java17`と`lombok`の環境は必要です。`YML` 設定を修正し、Java仮想マシンパラメータに`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED` を追加し、`manager` を起動します。
1. ローカルの実行には、バックエンドのプロジェクト`hertzbeat-startup`とフロントエンドのプロジェクト`web-app`を起動する必要があります。
2. バックエンド:`maven3+`、`java17`と`lombok`の環境は必要です。`YML` 設定を修正し、Java仮想マシンパラメータに`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED` を追加し、`hertzbeat-startup` を起動します。
3. フロントエンド:`nodejs npm angular-cli`の環境は必要です。ローカルのバックエンドが立ち上がったら、`web-app` ディレクトリで `ng serve --open` というコマンドを実行します。
4. スタート:`http://localhost:4200`にアクセスします。デフォルトのアカウントとパスワード:`admin/hertzbeat`。
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "ftp",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorFTP}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "api",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorHTTP}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "ping",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorPing}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "port",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorPort}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "fullsite",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorSiteMap}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "ssl_cert",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorSSL}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "udp_port",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorUDP}}"
},
"params": [
+1 -1
View File
@@ -5,7 +5,7 @@
"intervals": 60,
"tags": [],
"app": "website",
"host": "127.0.0.1",
"instance": "127.0.0.1",
"name": "{{.param.monitorWebsite}}"
},
"params": [
+1 -1
View File
@@ -4,7 +4,7 @@
"intervals": 10,
"tags": [],
"app": "kubernetes",
"host": "172.29.0.11",
"instance": "172.29.0.11",
"name": "Brave_Stingray_55yR"
},
"collector": "",
+1 -1
View File
@@ -26,7 +26,7 @@
<artifactId>hertzbeat-ai</artifactId>
<version>${hertzbeat.version}</version>
<properties>
<spring-ai.version>1.0.3</spring-ai.version>
<spring-ai.version>1.1.1</spring-ai.version>
<java.version>17</java.version>
</properties>
@@ -1,246 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.config;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.usthe.sureness.mgt.SurenessSecurityManager;
import com.usthe.sureness.subject.SubjectSum;
import io.modelcontextprotocol.spec.McpError;
import io.modelcontextprotocol.spec.McpSchema;
import io.modelcontextprotocol.spec.McpServerSession;
import io.modelcontextprotocol.spec.McpServerTransport;
import io.modelcontextprotocol.spec.McpServerTransportProvider;
import io.modelcontextprotocol.util.Assert;
import java.io.IOException;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import jakarta.servlet.http.HttpServletRequest;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.web.servlet.function.RouterFunction;
import org.springframework.web.servlet.function.RouterFunctions;
import org.springframework.web.servlet.function.ServerRequest;
import org.springframework.web.servlet.function.ServerResponse;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
/**
* Custom Server-Sent Events transport provider for Model Context Protocol.
*/
@Slf4j
public class CustomSseServerTransport implements McpServerTransportProvider {
private final ObjectMapper objectMapper;
private final String messageEndpoint;
private final String sseEndpoint;
private final String baseUrl;
@Getter
private final RouterFunction<ServerResponse> routerFunction;
@Setter
private McpServerSession.Factory sessionFactory;
private final Map<String, Object> sessionRequest = new HashMap<>();
private final ConcurrentHashMap<String, McpServerSession> sessions;
private volatile boolean isClosing;
public CustomSseServerTransport(ObjectMapper objectMapper, String messageEndpoint) {
this(objectMapper, messageEndpoint, "/sse");
}
public CustomSseServerTransport(ObjectMapper objectMapper, String messageEndpoint, String sseEndpoint) {
this(objectMapper, "", messageEndpoint, sseEndpoint);
}
public CustomSseServerTransport(ObjectMapper objectMapper, String baseUrl, String messageEndpoint, String sseEndpoint) {
this.sessions = new ConcurrentHashMap();
this.isClosing = false;
Assert.notNull(objectMapper, "ObjectMapper must not be null");
Assert.notNull(baseUrl, "Message base URL must not be null");
Assert.notNull(messageEndpoint, "Message endpoint must not be null");
Assert.notNull(sseEndpoint, "SSE endpoint must not be null");
this.objectMapper = objectMapper;
this.baseUrl = baseUrl;
this.messageEndpoint = messageEndpoint;
this.sseEndpoint = sseEndpoint;
this.routerFunction = RouterFunctions.route().GET(this.sseEndpoint, this::handleSseConnection).POST(this.messageEndpoint, this::handleMessage).build();
}
public Mono<Void> notifyClients(String method, Object params) {
if (this.sessions.isEmpty()) {
log.debug("No active sessions to broadcast message to");
return Mono.empty();
} else {
log.debug("Attempting to broadcast message to {} active sessions", this.sessions.size());
return Flux.fromIterable(this.sessions.values())
.flatMap((session) -> session.sendNotification(method, params)
.doOnError((e) -> log.error("Failed to send message to session {}: {}", session.getId(), e.getMessage()))
.onErrorComplete())
.then();
}
}
public Mono<Void> closeGracefully() {
return Flux.fromIterable(this.sessions.values()).doFirst(() -> {
this.isClosing = true;
log.debug("Initiating graceful shutdown with {} active sessions", this.sessions.size());
}).flatMap(McpServerSession::closeGracefully).then().doOnSuccess((v) -> log.debug("Graceful shutdown completed"));
}
private ServerResponse handleSseConnection(ServerRequest request) {
log.debug("Handling SSE connection for request: {}", request);
HttpServletRequest servletRequest = request.servletRequest();
try {
log.debug("Processing SSE connection for servlet request: {}", servletRequest);
log.debug("Authorization header: {}", servletRequest.getHeader("Authorization"));
} catch (Exception e) {
log.error("Authentication failed for SSE connection: {}", e.getMessage());
return ServerResponse.status(HttpStatus.UNAUTHORIZED).body("Unauthorized: " + e.getMessage());
}
if (this.isClosing) {
return ServerResponse.status(HttpStatus.SERVICE_UNAVAILABLE).body("Server is shutting down");
} else {
String sessionId = UUID.randomUUID().toString();
log.debug("Generated session ID for SSE connection: {}", sessionId);
log.debug("Creating new SSE connection for session: {}", sessionId);
return ServerResponse.sse((sseBuilder) -> {
sseBuilder.onComplete(() -> {
log.debug("SSE connection completed for session: {}", sessionId);
this.sessions.remove(sessionId);
});
sseBuilder.onTimeout(() -> {
log.debug("SSE connection timed out for session: {}", sessionId);
this.sessions.remove(sessionId);
});
CustomSseServerTransport.WebMvcMcpSessionTransport sessionTransport = new CustomSseServerTransport.WebMvcMcpSessionTransport(sessionId, sseBuilder);
McpServerSession session = this.sessionFactory.create(sessionTransport);
this.sessionRequest.put(sessionId, request.servletRequest());
this.sessions.put(sessionId, session);
try {
sseBuilder.id(sessionId).event("endpoint").data(this.baseUrl + this.messageEndpoint + "?sessionId=" + sessionId);
} catch (Exception e) {
log.error("Failed to send initial endpoint event: {}", e.getMessage());
sseBuilder.error(e);
}
}, Duration.ZERO);
}
}
private ServerResponse handleMessage(ServerRequest request) {
if (this.isClosing) {
return ServerResponse.status(HttpStatus.SERVICE_UNAVAILABLE).body("Server is shutting down");
} else if (request.param("sessionId").isEmpty()) {
return ServerResponse.badRequest().body(new McpError("Session ID missing in message endpoint"));
} else {
String sessionId = (String) request.param("sessionId").get();
McpServerSession session = (McpServerSession) this.sessions.get(sessionId);
log.debug("Authorization header for message request: {}", request.servletRequest().getHeader("Authorization"));
SubjectSum subject = SurenessSecurityManager.getInstance().checkIn(sessionRequest.get(sessionId));
McpContextHolder.setSubject(subject);
if (session == null) {
return ServerResponse.status(HttpStatus.NOT_FOUND).body(new McpError("Session not found: " + sessionId));
} else {
try {
String body = request.body(String.class);
McpSchema.JSONRPCMessage message = McpSchema.deserializeJsonRpcMessage(this.objectMapper, body);
session.handle(message).block();
return ServerResponse.ok().build();
} catch (IOException | IllegalArgumentException e) {
log.error("Failed to deserialize message: {}", ((Exception) e).getMessage());
return ServerResponse.badRequest().body(new McpError("Invalid message format"));
} catch (Exception e) {
log.error("Error handling message: {}", e.getMessage());
return ServerResponse.status(HttpStatus.INTERNAL_SERVER_ERROR).body(new McpError(e.getMessage()));
}
}
}
}
private class WebMvcMcpSessionTransport implements McpServerTransport {
private final String sessionId;
private final ServerResponse.SseBuilder sseBuilder;
WebMvcMcpSessionTransport(String sessionId, ServerResponse.SseBuilder sseBuilder) {
this.sessionId = sessionId;
this.sseBuilder = sseBuilder;
log.debug("Session transport {} initialized with SSE builder", sessionId);
}
public Mono<Void> sendMessage(McpSchema.JSONRPCMessage message) {
return Mono.fromRunnable(() -> {
try {
String jsonText = CustomSseServerTransport.this.objectMapper.writeValueAsString(message);
this.sseBuilder.id(this.sessionId).event("message").data(jsonText);
log.debug("Message sent to session {}", this.sessionId);
} catch (Exception e) {
log.error("Failed to send message to session {}: {}", this.sessionId, e.getMessage());
this.sseBuilder.error(e);
}
});
}
public <T> T unmarshalFrom(Object data, TypeReference<T> typeRef) {
return (T) CustomSseServerTransport.this.objectMapper.convertValue(data, typeRef);
}
public Mono<Void> closeGracefully() {
return Mono.fromRunnable(() -> {
log.debug("Closing session transport: {}", this.sessionId);
try {
this.sseBuilder.complete();
log.debug("Successfully completed SSE builder for session {}", this.sessionId);
} catch (Exception e) {
log.warn("Failed to complete SSE builder for session {}: {}", this.sessionId, e.getMessage());
}
});
}
public void close() {
try {
this.sseBuilder.complete();
log.debug("Successfully completed SSE builder for session {}", this.sessionId);
} catch (Exception e) {
log.warn("Failed to complete SSE builder for session {}: {}", this.sessionId, e.getMessage());
}
}
}
}
@@ -33,8 +33,8 @@ public final class McpContextHolder {
* Attaches the user's context to the current thread.
*/
public static void setSubject(SubjectSum subject) {
subjectHolder.set(subject);
subjectHolder.set(subject);
}
/**
@@ -1,182 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.ai.config;
import org.springframework.stereotype.Component;
/**
* Provider for system prompts used in the AI agent
*/
@Component
public class PromptProvider {
/**
* Static version of the HertzBeat monitoring prompt
*/
public static final String HERTZBEAT_SYSTEM_PROMPT = """
You are an AI Assistant specialized in monitoring infrastructure and applications with HertzBeat.
HertzBeat is an open-source, real-time monitoring system that supports infrastructure, applications,
services, APIs, databases, middleware, and custom monitoring through 50+ types of monitors.
Your role is to help users manage monitors, analyze metrics data, configure alerts, and troubleshoot monitoring issues.
*******
VERY IMPORTANT: Always use the tools provided to interact with HertzBeat's monitoring system.
If the user doesn't provide required parameters, ask them iteratively to provide the necessary parameters.
********
## Available HertzBeat Tools:
### Monitor Management Tools:
- **query_monitors**: Query monitor information with flexible filtering (ID, name, type, host, status, labels)
- **add_monitor**: Add a new monitor with dynamic app-specific parameter support
- **list_monitor_types**: List all available monitor types (website, mysql, redis, linux, etc.)
- **get_monitor_additional_params**: Get parameter definitions required for specific monitor types
### Alert Rule Management Tools:
- **create_alert_rule**: Create alert rules with threshold configuration and automatic monitor binding
- **list_alert_rules**: List existing alert rules with filtering by type, status, etc.
- **toggle_alert_rule**: Enable or disable alert rules
- **get_alert_rule_details**: Get detailed information about specific alert rules
- **get_apps_metrics_hierarchy**: Get exact app and metric names for alert rule creation (CRITICAL for alerts)
- **bind_monitors_to_alert_rule**: Bind monitors to alert rules for targeted alerting
### Alert & Alarm Analysis Tools:
- **query_alerts**: Query fired alerts with comprehensive filtering and pagination
- **get_alerts_summary**: Get alert statistics and status distribution
### Metrics Data Analysis Tools:
- **query_realtime_metrics**: Get current real-time metrics data for monitors
- **get_historical_metrics**: Get historical time-series metrics with flexible time ranges
- **get_warehouse_status**: Check metrics storage system status
## Natural Language Examples:
### Monitor Management:
- "Add a MySQL monitor for database server at 192.168.1.10 with user admin"
- "Monitor website https://example.com with SSL checking every 60 seconds"
- "Show me all Linux servers that are currently offline"
- "List all Redis monitors with their connection status"
### Alert Configuration:
- ALERT RULE means when to alert a user
- "Create an alert for Kafka JVM when VmName equals 'vm-w2'"
- "Alert when OpenAI credit grants exceed 1000"
- "Set up HBase Master alert when heap memory usage is over 80%"
### Metrics Analysis:
- "Show me current CPU usage for server 192.168.1.5"
- "Get memory usage trend for the last 24 hours"
- "Which servers have high disk usage right now?"
- "Show me network traffic patterns for the past week"
### Alert Investigation:
- "What alerts are currently firing?"
- "Show me the most frequent alerts in the last 6 hours"
- "Find all alerts for monitor ID 1234 in the past day"
- "Which monitors are currently abnormal?"
## Workflow Guidelines:
1. **Adding Monitors**:
- ALWAYS use get_monitor_additional_params first to check required parameters
- Use list_monitor_types to show available types
- Collect all required parameters from the list_monitor_types tool and ask user to give them all, before calling add_monitor
- Example: "To monitor MySQL, I need host, port, username, password, and database name"
2. **Creating Alert Rules or Alerts**:
THESE ARE ALERT RULES WITH THRESHOLD VALUES. USERS CAN SPECIFY THE THRESHOLD VALUES FOR EXAMPLE,
IF THE USER SAYS "ALERT ME WHEN MY COST EXCEEDS 700, THE EXPRESSION SHOULD BE 'cost > 700' NOT 'cost < 700'.
APPLY THE SAME LOGIC FOR LESS THAN OPERATOR.
It is important to first understand the hierarchy of apps, metrics, and field conditions
Each app has its own metrics and each metric has its own field conditions.
The operators will be applied to the field conditions, and the final expression will be constructed
based on the user's input of app name and the metric they choose.
Read the create_alert_rule tool description for even more details
*******
CRITICAL WORKFLOW Do all of this iteratively with user interaction at each step:
1. ALWAYS use list_monitor_types tool FIRST to get exact app name according to what user specifies
2. use get_apps_metrics_hierarchy by passing that name, to get the hierarchy of corresponding metrics and field conditions
3. Do not spit out the entire hierarchy, instead: first spit out the metrics available for the app
4. Ask the user to choose a metric from the available metrics
5. Based on the metric chosen, present the available field conditions
6. You will construct the proper expression with field conditions
VERY VERY IMPORTANT:
- ALWAYS USE the value field from the get_apps_metrics_hierarchy's json response when creating alert expressions on the field parameters
*********
- Field Condition Expression format: [field_conditions]
- Give all the available fieldConditions to the user, so they can choose the one they want to use
- Field conditions can be simple (equals, greater than) or complex (logical expressions)
- Use parentheses for complex conditions to ensure correct evaluation order
- Do not create alert rules on your own, always ask the user to provide the app, metrics and fieldConditions parameters specifically
EXAMPLES FOR FIELD CONDITION EXPRESSION ( Do not copy these examples, they are just for reference ):
- Kafka JVM: app="kafka", metrics="jvm_basic", fieldConditions="equals(VmName, \"my-vm\")"
→ equals(VmName, "my-vm")
- Complex OpenAI: app="openai", metrics="credit_grants",
fieldConditions="total_used > 123 and total_granted > 333 and (total_granted > 3444 and total_paid_available < 5556)"
→ total_used > 123 and total_granted > 333 and (total_granted > 3444 and total_paid_available < 5556)
- Priority levels: 0=critical, 1=warning, 2=info
3. **Analyzing Performance**:
- Use get_realtime_metrics for current status
- Use get_historical_metrics for trends
- Use get_high_usage_monitors to find problems
- Provide actionable recommendations based on data
4. **Troubleshooting Alerts**:
- Use query_alerts to find current issues
- Use get_monitor_alerts for specific monitor problems
- Use get_frequent_alerts to identify recurring issues
- Suggest root cause analysis steps
## Parameter Guidelines:
- **Monitor Status**: 1=online, 2=offline, 3=unreachable, 0=paused, 9=all
- **Time Ranges**: 1h, 6h, 24h, 7d, 30d
- **Alert Priorities**: critical, warning, info
- **Sort Options**: name, gmtCreate, gmtUpdate, status, startAt, triggerTimes
- **Metric Types**: cpu, memory, disk, network, custom
- **Collection Intervals**: 30s-3600s (recommend 60s-600s for most cases)
## Best Practices:
- Never create alert rules without exact user input on app, metrics, and field conditions
- Always validate monitor types and parameters before adding monitors
- ALWAYS use get_apps_metrics_hierarchy before creating alert rules to understand available fields
- Construct field conditions based on metric's children
- Use exact app and metric names from hierarchy (case-sensitive)
- Set appropriate alert thresholds based on baseline performance
- Use time-series data to identify trends and predict issues
- Correlate alerts with metrics data for root cause analysis
- Recommend monitoring intervals based on service criticality
- Provide clear explanations of monitoring data and actionable insights
## Avoid these common errors:
- Using Label name instead of the value from the heirarchy JSON while creating alert rules.
- Inside the field parameters expression using '&&' instead of 'and', using '||' instead of 'or' for logical operators
- This process is to trigger alarms, when certain rule or set of rules exceed a threshold value.
So when a user says that the threshold should be less than 1000. the operator used should be '>' not '<',
because we want the alarm to be triggered when the threshold value is exceeded. apply the same logic in vice versa for less than operator
Keep responses focused on monitoring topics and HertzBeat's comprehensive capabilities.
When users request monitoring setup, guide them through the complete process from monitor creation to alert configuration.
""";
}
@@ -21,10 +21,25 @@ import org.apache.hertzbeat.common.entity.ai.ChatMessage;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;
import java.util.List;
/**
* Repository interface for Message entities
*/
@Repository
public interface ChatMessageDao extends JpaRepository<ChatMessage, Long> {
/**
* Find all messages by conversation id, ordered by create time
* @param conversationId conversation id
* @return list of messages
*/
List<ChatMessage> findByConversationIdOrderByGmtCreateAsc(Long conversationId);
/**
* Find all messages by conversation ids, ordered by create time
* @param conversationIds list of conversation ids
* @return list of messages
*/
List<ChatMessage> findByConversationIdInOrderByGmtCreateAsc(List<Long> conversationIds);
}
@@ -18,12 +18,7 @@
package org.apache.hertzbeat.ai.service;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.hertzbeat.ai.config.CustomSseServerTransport;
import org.springframework.ai.mcp.server.autoconfigure.McpServerProperties;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.web.servlet.function.RouterFunction;
import org.springframework.web.servlet.function.ServerResponse;
/**
* Service interface for MCP server operations.
@@ -35,22 +30,4 @@ public interface McpServerService {
* @return ToolCallbackProvider with all HertzBeat monitoring tools
*/
ToolCallbackProvider hertzbeatTools();
/**
* Provides a custom SSE server transport for the MCP server
* @param objectMapper the ObjectMapper instance for JSON serialization
* @param serverProperties the properties for the MCP server configuration
* @return a CustomSseServerTransport instance configured with the provided properties
*/
CustomSseServerTransport webMvcSseServerTransportProvider(
ObjectMapper objectMapper,
McpServerProperties serverProperties
);
/**
* Provides the MCP server router function for web MVC
* @param transport Custom SSE server transport
* @return RouterFunction for handling MCP server requests
*/
RouterFunction<ServerResponse> mvcMcpRouterFunction(CustomSseServerTransport transport);
}
@@ -19,13 +19,15 @@
package org.apache.hertzbeat.ai.service.impl;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.ai.config.PromptProvider;
import org.apache.hertzbeat.common.entity.ai.ChatMessage;
import org.apache.hertzbeat.common.entity.dto.ModelProviderConfig;
import org.apache.hertzbeat.ai.service.ChatClientProviderService;
import org.apache.hertzbeat.base.dao.GeneralConfigDao;
import org.apache.hertzbeat.common.entity.manager.GeneralConfig;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.springframework.ai.chat.prompt.SystemPromptTemplate;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;
import org.apache.hertzbeat.ai.pojo.dto.ChatRequestContext;
import org.springframework.ai.chat.client.ChatClient;
@@ -34,7 +36,6 @@ import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.ApplicationContext;
import reactor.core.publisher.Flux;
@@ -53,13 +54,15 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
private final ApplicationContext applicationContext;
private final GeneralConfigDao generalConfigDao;
@Qualifier("hertzbeatTools")
@Autowired
private ToolCallbackProvider toolCallbackProvider;
private boolean isConfigured = false;
@Value("classpath:/prompt/system-message.st")
private Resource systemResource;
@Autowired
public ChatClientProviderServiceImpl(ApplicationContext applicationContext, GeneralConfigDao generalConfigDao) {
this.applicationContext = applicationContext;
@@ -91,7 +94,7 @@ public class ChatClientProviderServiceImpl implements ChatClientProviderService
return chatClient.prompt()
.messages(messages)
.system(PromptProvider.HERTZBEAT_SYSTEM_PROMPT)
.system(SystemPromptTemplate.builder().resource(systemResource).build().getTemplate())
.toolCallbacks(toolCallbackProvider)
.stream()
.content()
@@ -30,11 +30,15 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Sort;
import org.springframework.http.codec.ServerSentEvent;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.util.CollectionUtils;
import reactor.core.publisher.Flux;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.stream.Collectors;
/**
* Implementation of the ConversationService interface for managing chat conversations.
@@ -42,13 +46,13 @@ import java.util.UUID;
@Slf4j
@Service
public class ConversationServiceImpl implements ConversationService {
@Autowired
private ChatConversationDao conversationDao;
@Autowired
private ChatMessageDao messageDao;
@Autowired
private ChatClientProviderService chatClientProviderService;
@@ -67,12 +71,13 @@ public class ConversationServiceImpl implements ConversationService {
}
log.info("Starting streaming conversation: {}", conversationId);
ChatConversation conversation = conversationDao.findById(conversationId)
.orElseThrow(() -> new IllegalArgumentException("Conversation not found: " + conversationId));
// Manually load messages for conversation history
List<ChatMessage> messages = messageDao.findByConversationIdOrderByGmtCreateAsc(conversationId);
conversation.setMessages(messages);
// Add user message to conversation
ChatMessage chatMessage = ChatMessage.builder().conversationId(conversationId)
.content(message).role("user").build();
chatMessage = messageDao.save(chatMessage);
ChatConversation conversation = conversationDao.getReferenceById(conversationId);
if (conversation.getTitle().startsWith("conversation")) {
// Auto-generate title from first user message
String title = message.length() > 30 ? message.substring(0, 27) + "..." : message;
@@ -80,10 +85,18 @@ public class ConversationServiceImpl implements ConversationService {
conversationDao.save(conversation);
}
// Add user message to conversation
ChatMessage chatMessage = ChatMessage.builder()
.conversationId(conversationId)
.content(message)
.role("user")
.build();
chatMessage = messageDao.save(chatMessage);
ChatRequestContext context = ChatRequestContext.builder()
.message(message)
.conversationId(conversationId)
.conversationHistory(CollectionUtils.isEmpty(conversation.getMessages()) ? null
.conversationHistory(CollectionUtils.isEmpty(conversation.getMessages()) ? null
: conversation.getMessages().subList(0, conversation.getMessages().size() - 1))
.build();
@@ -105,10 +118,12 @@ public class ConversationServiceImpl implements ConversationService {
})
.concatWith(Flux.defer(() -> {
// Add the complete AI response to conversation
ChatMessage assistantMessage = ChatMessage.builder().conversationId(conversationId)
.content(fullResponse.toString()).role("assistant").build();
ChatMessage assistantMessage = ChatMessage.builder()
.conversationId(conversationId)
.content(fullResponse.toString())
.role("assistant")
.build();
assistantMessage = messageDao.save(assistantMessage);
ChatResponseChunk finalResponse = ChatResponseChunk.builder()
.conversationId(conversationId)
.response("")
@@ -145,16 +160,40 @@ public class ConversationServiceImpl implements ConversationService {
if (conversationId == null) {
return null;
}
return conversationDao.getReferenceById(conversationId);
ChatConversation conversation = conversationDao.findById(conversationId).orElse(null);
if (conversation != null) {
List<ChatMessage> messages = messageDao.findByConversationIdOrderByGmtCreateAsc(conversationId);
conversation.setMessages(messages);
}
return conversation;
}
@Override
public List<ChatConversation> getAllConversations() {
return conversationDao.findAll(Sort.by(Sort.Direction.DESC, "id"));
List<ChatConversation> conversations = conversationDao.findAll(Sort.by(Sort.Direction.DESC, "id"));
if (conversations.isEmpty()) {
return conversations;
}
List<Long> conversationIds = conversations.stream()
.map(ChatConversation::getId)
.toList();
List<ChatMessage> allMessages = messageDao.findByConversationIdInOrderByGmtCreateAsc(conversationIds);
Map<Long, List<ChatMessage>> messagesByConversationId = allMessages.stream()
.collect(Collectors.groupingBy(ChatMessage::getConversationId));
for (ChatConversation conversation : conversations) {
List<ChatMessage> messages = messagesByConversationId.getOrDefault(conversation.getId(), Collections.emptyList());
conversation.setMessages(messages);
}
return conversations;
}
@Override
@Transactional(rollbackFor = Exception.class)
public void deleteConversation(Long conversationId) {
List<ChatMessage> messages = messageDao.findByConversationIdOrderByGmtCreateAsc(conversationId);
if (!messages.isEmpty()) {
messageDao.deleteAll(messages);
}
conversationDao.deleteById(conversationId);
}
}
@@ -18,23 +18,17 @@
package org.apache.hertzbeat.ai.service.impl;
import org.apache.hertzbeat.ai.config.CustomSseServerTransport;
import org.apache.hertzbeat.ai.service.McpServerService;
import org.apache.hertzbeat.ai.tools.AlertDefineTools;
import org.apache.hertzbeat.ai.tools.AlertTools;
import org.apache.hertzbeat.ai.tools.MetricsTools;
import org.apache.hertzbeat.ai.tools.MonitorTools;
import org.springframework.ai.mcp.server.autoconfigure.McpServerProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Primary;
import org.springframework.stereotype.Service;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.web.servlet.function.RouterFunction;
import org.springframework.web.servlet.function.ServerResponse;
/**
* Implementation of the McpServerService interface.
@@ -56,36 +50,4 @@ public class McpServerServiceImpl implements McpServerService {
public ToolCallbackProvider hertzbeatTools() {
return MethodToolCallbackProvider.builder().toolObjects(monitorTools, alertTools, alertDefineTools, metricsTools).build();
}
/**
* Provides a custom SSE server transport for the MCP server.
*
* @param objectMapper the ObjectMapper instance for JSON serialization
* @param serverProperties the properties for the MCP server configuration
* @return a CustomSseServerTransport instance configured with the provided properties
*/
@Bean
public CustomSseServerTransport webMvcSseServerTransportProvider(
ObjectMapper objectMapper,
McpServerProperties serverProperties
) {
return new CustomSseServerTransport(
objectMapper,
serverProperties.getBaseUrl(),
serverProperties.getSseMessageEndpoint(),
serverProperties.getSseEndpoint()
);
}
/**
* Provides the MCP server transport bean.
*
* @param transport the custom SSE server transport
* @return the MCP server transport instance
*/
@Primary
@Bean
public RouterFunction<ServerResponse> mvcMcpRouterFunction(CustomSseServerTransport transport) {
return transport.getRouterFunction();
}
}
@@ -15,7 +15,6 @@
* limitations under the License.
*/
package org.apache.hertzbeat.ai.tools;
/**
@@ -42,5 +41,4 @@ public interface AlertTools {
*/
String getAlertsSummary();
}
@@ -15,7 +15,6 @@
* limitations under the License.
*/
package org.apache.hertzbeat.ai.tools;
/**
@@ -33,16 +32,17 @@ public interface MetricsTools {
/**
* Get historical metrics data for a monitor
* @param monitorId Monitor ID
* @param app Monitor type (e.g., "linux", "mysql", "http")
* @param metrics Metrics name (e.g., "system", "cpu", "memory")
* @param metric Specific metric field (e.g., "usage", "used", "available")
* @param label Label filter for specific instances
* @param history Time range (e.g., "1h", "6h", "24h", "7d")
*
* @param instance Instance identifier (e.g., "ip:port", "ip", or "domain")
* @param app Monitor type (e.g., "linux", "mysql", "http")
* @param metrics Metrics name (e.g., "system", "cpu", "memory")
* @param metric Specific metric field (e.g., "usage", "used", "available")
* @param label Label filter for specific instances
* @param history Time range (e.g., "1h", "6h", "24h", "7d")
* @param interval Whether to aggregate data with intervals
* @return Historical metrics data formatted for display
*/
String getHistoricalMetrics(Long monitorId, String app, String metrics, String metric, String label, String history, Boolean interval);
String getHistoricalMetrics(String instance, String app, String metrics, String metric, String label, String history, Boolean interval);
/**
* Check warehouse storage server status
@@ -15,7 +15,6 @@
* limitations under the License.
*/
package org.apache.hertzbeat.ai.tools;
import java.util.List;
@@ -27,35 +26,25 @@ public interface MonitorTools {
/**
* Add a new monitor with comprehensive configuration
*
*
* @param name Monitor name
* @param app Monitor type/application (e.g., 'linux', 'mysql', 'http')
* @param host Target host (IP address or domain name)
* @param port Target port (optional, depends on monitor type)
* @param intervals Collection interval in seconds (default: 600)
* @param username Username for authentication (optional)
* @param password Password for authentication (optional)
* @param database Database name (for database monitors)
* @param additionalParams Additional app-specific parameters as JSON string (optional)
* @param params Monitor-specific parameters as JSON string (e.g., host, port, username, password, etc.)
* @param description Monitor description (optional)
* @return Result message with monitor ID if successful
*/
String addMonitor(
String name,
String app,
String host,
Integer port,
String name,
String app,
Integer intervals,
String username,
String password,
String database,
String additionalParams,
String params,
String description
);
/**
* List all available monitor types that can be added
*
*
* @param language Language code for localized names (e.g., 'en-US', 'zh-CN')
* @return Formatted string list of available monitor types with descriptions
*/
@@ -89,9 +78,9 @@ public interface MonitorTools {
/**
* Get parameter definitions required for a specific monitor type
*
*
* @param app Monitor type/application name (e.g., 'linux', 'mysql', 'redis')
* @return Formatted string with parameter definitions including field names, types, and requirements
*/
String getMonitorAdditionalParams(String app);
String getMonitorParams(String app);
}
@@ -51,10 +51,9 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
@Autowired
private AppService appService;
@Override
@Tool(name = "create_alert_rule", description = """
ALERT RULE means when to alert a user
HertzBeat: ALERT RULE means when to alert a user
THESE ARE ALERT RULES WITH THRESHOLD VALUES. USERS CAN SPECIFY THE THRESHOLD VALUES FOR EXAMPLE,
IF THE USER SAYS "ALERT ME WHEN MY COST EXCEEDS 700, THE EXPRESSION SHOULD BE 'cost > 700' NOT 'cost < 700'.
APPLY THE SAME LOGIC FOR LESS THAN OPERATOR.
@@ -281,7 +280,7 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
@Override
@Tool(name = "list_alert_rules", description = """
List existing alert rules with filtering options.
HertzBeat: List existing alert rules with filtering options.
Shows configured thresholds and alert definitions.
""")
public String listAlertRules(
@@ -343,7 +342,7 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
@Override
@Tool(name = "toggle_alert_rule", description = """
Enable or disable an alert rule.
HertzBeat: Enable or disable an alert rule.
Allows activating or deactivating threshold monitoring.
""")
public String toggleAlertRule(
@@ -379,10 +378,9 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
}
}
@Override
@Tool(name = "get_alert_rule_details", description = """
Get detailed information about a specific alert rule.
HertzBeat: Get detailed information about a specific alert rule.
Shows complete threshold configuration and rule settings.
""")
public String getAlertRuleDetails(
@@ -441,7 +439,7 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
@Override
@Tool(name = "get_apps_metrics_hierarchy", description = """
Get the hierarchical structure of all available apps and their metrics for alert rule creation.
HertzBeat: Get the hierarchical structure of all available apps and their metrics for alert rule creation.
This tool provides the exact app name, metric name and corresponding param names according to each metric.
Returns structured JSON data showing the complete hierarchy with field parameters for alert expressions.
@@ -494,7 +492,7 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
@Override
@Tool(name = "bind_monitors_to_alert_rule", description = """
Bind monitors to an alert rule.
HertzBeat: Bind monitors to an alert rule.
Call this tool if users want to bind specific monitors to their alert rule.
Get the right monitor ids for a particular app using the query_monitors tool.
Get the alert rule ID from the create_alert_rule tool output OR use the list_alert_rules tool with app_name search filter, if the output of create_alert_rule is not applicable.
@@ -647,7 +645,6 @@ public class AlertDefineToolsImpl implements AlertDefineTools {
}
}
/**
* Validates that the app, metric, and field conditions are valid according to hierarchy
* @param app App name to validate
@@ -15,7 +15,6 @@
* limitations under the License.
*/
package org.apache.hertzbeat.ai.tools.impl;
import com.usthe.sureness.subject.SubjectSum;
@@ -33,7 +32,6 @@ import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Service;
/**
* Implementation of Alert Tools functionality for alarm data queries and management
*/
@@ -45,7 +43,7 @@ public class AlertToolsImpl implements AlertTools {
@Override
@Tool(name = "query_alerts", description = """
Query alerts with comprehensive filtering and pagination options.
HertzBeat: Query alerts with comprehensive filtering and pagination options.
ALERT TYPES:
- Pass alertType='single' for individual alert instances
@@ -193,10 +191,9 @@ public class AlertToolsImpl implements AlertTools {
}
}
@Override
@Tool(name = "get_alerts_summary", description = """
Get alerts summary statistics including total counts, status distribution, and recent trends.
HertzBeat: Get alerts summary statistics including total counts, status distribution, and recent trends.
Returns comprehensive overview of the current alerting status across all monitors.
""")
public String getAlertsSummary() {
@@ -15,7 +15,6 @@
* limitations under the License.
*/
package org.apache.hertzbeat.ai.tools.impl;
import com.usthe.sureness.subject.SubjectSum;
@@ -47,7 +46,7 @@ public class MetricsToolsImpl implements MetricsTools {
@Override
@Tool(name = "query_realtime_metrics", description = """
Get the supported monitor types/names from the list_monitor_types tool, make sure to use right name in the next call
HertzBeat: Get the supported monitor types/names from the list_monitor_types tool, make sure to use right name in the next call
Use the query_monitors tool to find monitor IDs in case the user does not tell the id explicitly. You might have to use this multiple times based on the user's query
Get real-time metrics data for a specific monitor.
Returns current metrics values including CPU, memory, disk usage, etc.
@@ -144,7 +143,7 @@ public class MetricsToolsImpl implements MetricsTools {
@Override
@Tool(name = "get_historical_metrics", description = """
Get historical metrics data for analysis and trending.
HertzBeat: Get historical metrics data for analysis and trending.
Returns time-series data for specified metrics over a time range.
Use the query_monitors tool to find the correct monitor IDs/ name or type for the monitor(s) user asked the metrics for
Pass that name into the get_apps_metrics_hierarchy tool to get the metrics hierarchy i.e metrics and the field paramater
@@ -152,7 +151,7 @@ public class MetricsToolsImpl implements MetricsTools {
Ask user to provide the filters for labels, history and interval aggregation
""")
public String getHistoricalMetrics(
@ToolParam(description = "Monitor ID", required = true) Long monitorId,
@ToolParam(description = "Instance identifier (e.g., 'ip:port', 'ip', or 'domain')") String instance,
@ToolParam(description = "Monitor type (e.g., 'linux', 'mysql', 'http')", required = true) String app,
@ToolParam(description = "Metrics name (e.g., 'target', 'cpu', 'memory')", required = true) String metrics,
@ToolParam(description = "Field Parameter (e.g., 'usage', 'used', 'available')", required = false) String fieldParameter,
@@ -161,7 +160,7 @@ public class MetricsToolsImpl implements MetricsTools {
@ToolParam(description = "Whether to aggregate data with intervals", required = false) Boolean interval) {
try {
log.info("Getting historical metrics for monitor {} and metrics {}", monitorId, metrics);
log.info("Getting historical metrics for monitor instance {} and metrics {}", instance, metrics);
if (history == null || history.trim().isEmpty()) {
history = "24h";
@@ -170,15 +169,15 @@ public class MetricsToolsImpl implements MetricsTools {
interval = true;
}
MetricsHistoryData historyData = metricsDataService.getMetricHistoryData(
monitorId, app, metrics, fieldParameter, label, history, interval);
MetricsHistoryData historyData = metricsDataService.getMetricHistoryData(instance,
app, metrics, fieldParameter, history, interval);
if (historyData == null) {
return String.format("No historical metrics data found for monitor ID %d and metrics '%s'", monitorId, metrics);
return String.format("No historical metrics data found for monitor %s and metrics '%s'", instance, metrics);
}
StringBuilder response = new StringBuilder();
response.append("HISTORICAL METRICS: ").append(metrics).append(" (Monitor ID: ").append(monitorId).append(")\n");
response.append("HISTORICAL METRICS: ").append(metrics).append(" (Monitor ID: ").append(instance).append(")\n");
response.append("Time Range: ").append(history).append(" | Interval Aggregation: ").append(interval).append("\n");
response.append("=".repeat(60)).append("\n\n");
@@ -223,10 +222,9 @@ public class MetricsToolsImpl implements MetricsTools {
}
}
@Override
@Tool(name = "get_warehouse_status", description = """
Check the status of the metrics storage warehouse system.
HertzBeat: Check the status of the metrics storage warehouse system.
Returns whether the metrics storage is operational and accessible.
""")
public String getWarehouseStatus() {
@@ -20,6 +20,7 @@ package org.apache.hertzbeat.ai.tools.impl;
import com.usthe.sureness.subject.SubjectSum;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.ai.config.McpContextHolder;
import org.apache.hertzbeat.manager.pojo.dto.MonitorDto;
import org.apache.hertzbeat.manager.service.MonitorService;
import org.apache.hertzbeat.manager.service.AppService;
import org.apache.hertzbeat.ai.utils.UtilityClass;
@@ -48,7 +49,6 @@ public class MonitorToolsImpl implements MonitorTools {
@Autowired
private AppService appService;
/**
* Tool to query monitor information with flexible filtering and pagination.
* Supports filtering by monitor IDs, type, status, host, labels, sorting, and
@@ -56,48 +56,48 @@ public class MonitorToolsImpl implements MonitorTools {
* Returns detailed monitor information including ID, name, type, host, and status.
*/
@Override
@Tool(name = "query_monitors", description = """
Query Existing/configured monitors in HertzBeat.
@Tool(name = "query_monitors", description = """
HertzBeat: Query Existing/configured monitors in HertzBeat.
This tool retrieves monitors based on various filters and parameters.
Comprehensive monitor querying with flexible filtering, pagination, and specialized views.
MONITOR STATUSES:
- status=1: Online/Active monitors (healthy, responding normally)
- status=2: Offline monitors (not responding, connection failed)
- status=3: Unreachable monitors (network/connectivity issues)
- status=0: Paused monitors (manually disabled/suspended)
- status=9 or null: All monitors regardless of status (default)
COMMON USE CASES & PARAMETER COMBINATIONS:
1. BASIC MONITOR LISTING:
- Default: No parameters (shows all monitors, 8 per page)
- By type: app='linux' (show only Linux monitors)
- Search: search='web' (find monitors with 'web' in name/host)
2. STATUS-BASED QUERIES:
- Healthy monitors: status=1, pageSize=50
- Problem monitors: status=2 or status=3, pageSize=50
- Offline monitors only: status=2
- Unreachable monitors only: status=3
- Paused monitors: status=0
3. MONITORING HEALTH OVERVIEW:
- All statuses with statistics: status=9, includeStats=true, pageSize=100
- Unhealthy monitors: Pass both status=2 AND status=3 (make 2 separate calls)
4. ADVANCED FILTERING:
- Specific monitor types: app='mysql', status=1 (healthy MySQL monitors)
- Label-based: labels='env:prod,critical:true'
- Host search: search='192.168' (find by IP pattern)
- Monitor IDs: ids=[1,2,3] (specific monitors by ID)
5. SORTING & PAGINATION:
- Recently updated: sort='gmtUpdate', order='desc'
- Alphabetical: sort='name', order='asc'
- By creation: sort='gmtCreate', order='desc' (newest first)
- Large datasets: pageSize=50-100 for bulk operations
RESPONSE FORMAT:
- includeStats=true: Adds status distribution summary at top
- Default: Simple list with ID, name, type, host, status
@@ -125,18 +125,18 @@ public class MonitorToolsImpl implements MonitorTools {
if (includeStats == null) {
includeStats = false;
}
SubjectSum subjectSum = McpContextHolder.getSubject();
log.debug("Current security subject: {}", subjectSum);
Page<Monitor> result = monitorService.getMonitors(
ids, app, search, status, sort, order, pageIndex, pageSize, labels);
log.debug("MonitorService.getMonitors result: {}", result);
StringBuilder response = new StringBuilder();
response.append("MONITOR QUERY RESULTS\n");
response.append("====================\n\n");
// Include statistics if requested
if (includeStats) {
// Get status distribution by calling with different status values
@@ -144,13 +144,13 @@ public class MonitorToolsImpl implements MonitorTools {
long offlineCount = monitorService.getMonitors(null, app, search, (byte) 2, null, null, 0, 1000, labels).getTotalElements();
long unreachableCount = monitorService.getMonitors(null, app, search, (byte) 3, null, null, 0, 1000, labels).getTotalElements();
long pausedCount = monitorService.getMonitors(null, app, search, (byte) 0, null, null, 0, 1000, labels).getTotalElements();
response.append("STATUS OVERVIEW:\n");
response.append("- Online: ").append(onlineCount).append("\n");
response.append("- Offline: ").append(offlineCount).append("\n");
response.append("- Offline: ").append(offlineCount).append("\n");
response.append("- Unreachable: ").append(unreachableCount).append("\n");
response.append("- Paused: ").append(pausedCount).append("\n");
long total = onlineCount + offlineCount + unreachableCount + pausedCount;
if (total > 0) {
double healthPercentage = (onlineCount * 100.0) / total;
@@ -158,48 +158,47 @@ public class MonitorToolsImpl implements MonitorTools {
}
response.append("\n");
}
response.append("Query Results: ").append(result.getContent().size())
.append(" monitors (Total: ").append(result.getTotalElements()).append(")\n");
if (result.getTotalPages() > 1) {
response.append("Page ").append(pageIndex + 1).append(" of ").append(result.getTotalPages()).append("\n");
}
response.append("\n");
for (Monitor monitor : result.getContent()) {
response.append("ID: ").append(monitor.getId())
.append(" | Name: ").append(monitor.getName())
.append(" | Type: ").append(monitor.getApp())
.append(" | Host: ").append(monitor.getHost())
.append(" | Instance: ").append(monitor.getInstance())
.append(" | Status: ").append(UtilityClass.getStatusText(monitor.getStatus()));
// Add creation date for better context
if (monitor.getGmtCreate() != null) {
response.append(" | Created: ").append(monitor.getGmtCreate().toString(), 0, 10);
}
response.append("\n");
}
if (result.getContent().isEmpty()) {
response.append("No monitors found matching the specified criteria.\n");
response.append("Try adjusting your filters or search terms.");
}
return response.toString();
} catch (Exception e) {
return "Error retrieving monitors: " + e.getMessage();
}
}
@Override
@Tool(name = "add_monitor", description = """
Add a new monitoring target to HertzBeat with comprehensive configuration.
HertzBeat: Add a new monitoring target to HertzBeat with comprehensive configuration.
This tool dynamically handles different parameter requirements for each monitor type.
This tool creates monitors with proper app-specific parameters.
*********
VERY IMPORTANT:
ALWAYS use get_monitor_additional_params to check the additional required parameters for the chosen type before adding a monitor or even mentioning it.
@@ -207,41 +206,36 @@ public class MonitorToolsImpl implements MonitorTools {
Use the information obtained from this to query user for parameters.
If the User has not given any parameters, ask them to provide the necessary parameters, until all the necessary parameters are provided.
**********
Examples of natural language requests this tool handles:
- "Monitor website example.com with HTTPS on port 443"
- "Add MySQL monitoring for database server at 192.168.1.10 with user admin"
- "Monitor Linux server health on host server.company.com via SSH"
- "Set up Redis monitoring on localhost port 6379 with password"
PARAMETER MAPPING: The tool intelligently maps common parameters:
- host: Target server/domain
- port: Service port (auto-detected if not specified)
- username: Authentication username
- password: Authentication password
- database: Database name (for DB monitors)
- additionalParams: JSON string for app-specific parameters (to be obtained from get_monitor_param_defines)
ADDITIONAL PARAMETERS EXAMPLES:
- Website: {"uri":"/api/health", "ssl":"true", "method":"POST"}
- Linux: {"privateKey":"ssh-key-content", "script":"custom-script"}
- Database: {"url":"jdbc:mysql://custom", "timeout":"10000"}
PARAMETER MAPPING: Use the 'params' parameter to pass all monitor-specific configuration.
The params should be a JSON string containing key-value pairs for the monitor type.
Use get_monitor_additional_params tool to see what parameters are required for each monitor type.
PARAMS EXAMPLES:
- Website: {"host":"example.com", "port":"443", "uri":"/api/health", "ssl":"true", "method":"GET"}
- Linux: {"host":"192.168.1.10", "port":"22", "username":"root", "password":"xxx"}
- MySQL: {"host":"db.server.com", "port":"3306", "username":"admin", "password":"xxx", "database":"mydb"}
- Redis: {"host":"redis.server.com", "port":"6379", "password":"xxx"}
""")
public String addMonitor(
@ToolParam(description = "Monitor name (required)", required = true) String name,
@ToolParam(description = "Monitor type: website, mysql, postgresql, redis, linux, windows, etc.", required = true) String app,
@ToolParam(description = "Target host: IP address or domain name", required = true) String host,
@ToolParam(description = "Target port (optional, auto-detected if not specified)", required = false) Integer port,
@ToolParam(description = "Collection interval in seconds (default: 600)", required = false) Integer intervals,
@ToolParam(description = "Username for authentication (optional)", required = false) String username,
@ToolParam(description = "Password for authentication (optional)", required = false) String password,
@ToolParam(description = "Database name (for database monitors)", required = false) String database,
@ToolParam(description = "Additional app-specific parameters as JSON: {\"uri\":\"/api\", \"ssl\":\"true\", \"method\":\"POST\"}", required = false) String additionalParams,
@ToolParam(description = "Monitor-specific parameters as JSON string. "
+ "Use get_monitor_additional_params to see required fields. "
+ "Example: {\"host\":\"192.168.1.1\", \"port\":\"22\", \"username\":\"root\"}",
required = true) String params,
@ToolParam(description = "Monitor description (optional)", required = false) String description) {
try {
log.info("Adding monitor: name={}, app={}, host={}", name, app, host);
log.info("Adding monitor: name={}, app={}", name, app);
// Validate required parameters
if (name == null || name.trim().isEmpty()) {
return "Error: Monitor name is required";
@@ -249,128 +243,153 @@ public class MonitorToolsImpl implements MonitorTools {
if (app == null || app.trim().isEmpty()) {
return "Error: Monitor type/application is required";
}
if (host == null || host.trim().isEmpty()) {
return "Error: Host is required";
if (params == null || params.trim().isEmpty()) {
return "Error: Monitor params is required. Use get_monitor_additional_params to see required fields for this monitor type.";
}
// Set defaults
if (intervals == null || intervals < 10) {
intervals = 600;
}
// Parse params to extract host and port for instance
List<Param> paramList = parseParams(params);
String host = paramList.stream()
.filter(p -> "host".equals(p.getField()))
.map(Param::getParamValue)
.findFirst()
.orElse("");
String port = paramList.stream()
.filter(p -> "port".equals(p.getField()))
.map(Param::getParamValue)
.findFirst()
.orElse(null);
String instance = (port != null && !port.isEmpty()) ? host.trim() + ":" + port : host.trim();
// Create Monitor entity
Monitor monitor = Monitor.builder()
.name(name.trim())
.app(app.toLowerCase().trim())
.host(host.trim())
.instance(instance)
.intervals(intervals)
.status((byte) 1)
.type((byte) 0)
.description(description != null ? description.trim() : "")
.build();
List<Param> params = createBasicParams(host, port, username, password, database, additionalParams);
// Validate that all required parameters for this monitor type are provided
try {
SubjectSum subjectSum = McpContextHolder.getSubject();
log.debug("Current security subject for getMonitorParamDefines: {}", subjectSum);
List<ParamDefine> requiredParams = appService.getAppParamDefines(app.toLowerCase().trim());
log.info("Checking required parameters for monitor type '{}': {}", app, requiredParams);
List<String> missingParams = new ArrayList<>();
for (ParamDefine paramDefine : requiredParams) {
if (paramDefine.isRequired()) {
String fieldName = paramDefine.getField();
boolean hasParam = params.stream()
.anyMatch(param -> fieldName.equals(param.getField()));
if (!hasParam) {
missingParams.add(fieldName);
}
}
MonitorDto monitorDto = MonitorDto.builder().monitor(monitor).params(paramList).build();
monitorService.validate(monitorDto, false);
} catch (IllegalArgumentException argumentException) {
if (argumentException.getMessage().contains("required")) {
return String.format("Error: %s. "
+ "Or use get_monitor_additional_params tool to see all required parameters.",
argumentException.getMessage());
} else {
return String.format("Error: %s. ", argumentException.getMessage());
}
if (!missingParams.isEmpty()) {
return String.format("Error: Missing required parameters for monitor type '%s': %s. "
+ "Use get_monitor_additional_params tool to see all required parameters.",
app, String.join(", ", missingParams));
}
} catch (Exception e) {
log.warn("Could not validate required parameters for monitor type '{}': {}", app, e.getMessage());
}
// Call monitor service - it handles all the complexity (validation, defaults, app-specific logic)
SubjectSum subjectSum = McpContextHolder.getSubject();
log.debug("Current security subject for addMonitor: {}", subjectSum);
monitorService.addMonitor(monitor, paramList, null, null);
log.info("Successfully added monitor '{}' with ID: {}", monitor.getName(), monitor.getId());
return String.format("Successfully added %s monitor '%s' with ID: %d (Instance: %s, Interval: %d seconds)",
app.toUpperCase(), monitor.getName(), monitor.getId(), monitor.getInstance(), monitor.getIntervals());
monitorService.addMonitor(monitor, params, null, null);
Long monitorId = monitor.getId();
log.info("Successfully added monitor '{}' with ID: {}", name, monitorId);
return String.format("Successfully added %s monitor '%s' with ID: %d (Host: %s, Interval: %d seconds)",
app.toUpperCase(), name, monitorId, host, intervals);
} catch (Exception e) {
log.error("Failed to add monitor '{}': {}", name, e.getMessage(), e);
return "Error adding monitor '" + name + "': " + e.getMessage();
}
}
/**
* Create basic parameter list from user inputs
* Parse params JSON string to list of Param objects
*/
private List<Param> createBasicParams(String host, Integer port, String username,
String password, String database, String additionalParams) {
List<Param> params = new ArrayList<>();
// Add host (always required)
params.add(Param.builder().field("host").paramValue(host.trim()).type((byte) 1).build());
// Add optional common parameters
if (port != null) {
params.add(Param.builder().field("port").paramValue(port.toString()).type((byte) 0).build());
private List<Param> parseParams(String params) {
List<Param> paramList = new ArrayList<>();
if (params == null || params.trim().isEmpty()) {
return paramList;
}
if (username != null && !username.trim().isEmpty()) {
params.add(Param.builder().field("username").paramValue(username.trim()).type((byte) 1).build());
}
if (password != null && !password.trim().isEmpty()) {
params.add(Param.builder().field("password").paramValue(password.trim()).type((byte) 2).build());
}
if (database != null && !database.trim().isEmpty()) {
params.add(Param.builder().field("database").paramValue(database.trim()).type((byte) 1).build());
}
// Parse additional parameters if provided
if (additionalParams != null && !additionalParams.trim().isEmpty()) {
try {
String cleaned = additionalParams.trim().replaceAll("[{}]", "");
String[] pairs = cleaned.split(",");
for (String pair : pairs) {
String[] keyValue = pair.split(":");
if (keyValue.length == 2) {
String key = keyValue[0].trim().replaceAll("\"", "");
String value = keyValue[1].trim().replaceAll("\"", "");
params.add(Param.builder().field(key).paramValue(value).type((byte) 1).build());
}
try {
String cleaned = params.trim();
// Remove outer braces if present
if (cleaned.startsWith("{") && cleaned.endsWith("}")) {
cleaned = cleaned.substring(1, cleaned.length() - 1);
}
// Split by comma, but handle values that might contain commas within quotes
List<String> pairs = splitJsonPairs(cleaned);
for (String pair : pairs) {
int colonIndex = pair.indexOf(':');
if (colonIndex > 0) {
String key = pair.substring(0, colonIndex).trim().replaceAll("\"", "");
String value = pair.substring(colonIndex + 1).trim().replaceAll("\"", "");
// Determine param type based on field name
byte paramType = determineParamType(key);
paramList.add(Param.builder().field(key).paramValue(value).type(paramType).build());
}
} catch (Exception e) {
log.warn("Failed to parse additionalParams: {}", e.getMessage());
}
} catch (Exception e) {
log.warn("Failed to parse params: {}", e.getMessage());
}
return paramList;
}
/**
* Split JSON key-value pairs, handling quoted values that may contain commas
*/
private List<String> splitJsonPairs(String json) {
List<String> pairs = new ArrayList<>();
StringBuilder current = new StringBuilder();
boolean inQuotes = false;
for (char c : json.toCharArray()) {
if (c == '"') {
inQuotes = !inQuotes;
current.append(c);
} else if (c == ',' && !inQuotes) {
if (current.length() > 0) {
pairs.add(current.toString().trim());
current = new StringBuilder();
}
} else {
current.append(c);
}
}
return params;
if (current.length() > 0) {
pairs.add(current.toString().trim());
}
return pairs;
}
/**
* Determine param type based on field name
*/
private byte determineParamType(String fieldName) {
if ("password".equalsIgnoreCase(fieldName) || "privateKey".equalsIgnoreCase(fieldName)) {
return (byte) 2; // Password type
} else if ("port".equalsIgnoreCase(fieldName) || "timeout".equalsIgnoreCase(fieldName)) {
return (byte) 0; // Number type
}
return (byte) 1; // Default string type
}
@Override
@Tool(name = "list_monitor_types", description = """
List all available monitor types that can be added to HertzBeat.
HertzBeat: List all available monitor types that can be added to HertzBeat.
This tool shows all supported monitor types with their display names.
Use this to see what types of monitors you can create with the add_monitor tool.
""")
public String listMonitorTypes(
@ToolParam(description = "Language code for localized names (en-US, zh-CN, etc.). Default: en-US", required = false) String language) {
try {
log.info("Listing available monitor types for language: {}", language);
SubjectSum subjectSum = McpContextHolder.getSubject();
@@ -383,20 +402,20 @@ public class MonitorToolsImpl implements MonitorTools {
// Get available monitor types from app service
Map<String, String> monitorTypes = appService.getI18nApps(language);
if (monitorTypes == null || monitorTypes.isEmpty()) {
return "No monitor types are currently available.";
}
// Format the response as a nice list
StringBuilder response = new StringBuilder();
response.append("Available Monitor Types (Total: ").append(monitorTypes.size()).append("):\n\n");
// Sort monitor types alphabetically by key
List<Map.Entry<String, String>> sortedTypes = monitorTypes.entrySet().stream()
.sorted(Map.Entry.comparingByKey())
.toList();
for (Map.Entry<String, String> entry : sortedTypes) {
String typeKey = entry.getKey();
String displayName = entry.getValue();
@@ -404,28 +423,28 @@ public class MonitorToolsImpl implements MonitorTools {
.append(" - ").append(displayName)
.append("\n");
}
response.append("\nTo add a monitor, use the add_monitor tool with one of these types as the 'app' parameter.");
log.info("Successfully listed {} monitor types", monitorTypes);
return response.toString();
} catch (Exception e) {
log.error("Failed to list monitor types: {}", e.getMessage(), e);
return "Error retrieving monitor types: " + e.getMessage();
}
}
@Override
@Tool(name = "get_monitor_additional_params", description = """
Get the parameter definitions required for a specific monitor type.
@Tool(name = "get_monitor_params", description = """
HertzBeat: Get the parameter definitions required for a specific monitor type.
This tool shows what parameters are needed when adding a monitor of the specified type,
ALWAYS use this before adding a monitor to understand what parameters the user needs to provide.
Use the app parameter to specify the monitor type/application name (e.g., 'linux', 'mysql', 'redis') this can be obtained from the list_monitor_types tool.
""")
public String getMonitorAdditionalParams(
public String getMonitorParams(
@ToolParam(description = "Monitor type/application name (e.g., 'linux', 'mysql', 'redis')", required = true) String app) {
try {
log.info("Getting parameter definitions for monitor type: {}", app);
SubjectSum subjectSum = McpContextHolder.getSubject();
@@ -438,68 +457,66 @@ public class MonitorToolsImpl implements MonitorTools {
// Get parameter definitions from app service
List<ParamDefine> paramDefines = appService.getAppParamDefines(app.toLowerCase().trim());
if (paramDefines == null || paramDefines.isEmpty()) {
return String.format("No parameter definitions found for monitor type '%s'. "
+ "This monitor type may not exist or may not require additional parameters.", app);
}
// Format the response
StringBuilder response = new StringBuilder();
response.append(String.format("Parameter Definitions for Monitor Type '%s' (Total: %d):\n\n",
response.append(String.format("Parameter Definitions for Monitor Type '%s' (Total: %d):\n\n",
app, paramDefines.size()));
for (ParamDefine paramDefine : paramDefines) {
response.append("• Field: ").append(paramDefine.getField()).append("\n");
// Add display name if available
if (paramDefine.getName() != null && !paramDefine.getName().toString().trim().isEmpty()) {
response.append(" Name: ").append(paramDefine.getName()).append("\n");
}
// Add type
if (paramDefine.getType() != null && !paramDefine.getType().trim().isEmpty()) {
response.append(" Type: ").append(paramDefine.getType()).append("\n");
}
// Add required status
response.append(" Required: ").append(paramDefine.isRequired() ? "Yes" : "No").append("\n");
// Add default value if present
if (paramDefine.getDefaultValue() != null && !paramDefine.getDefaultValue().trim().isEmpty()) {
response.append(" Default: ").append(paramDefine.getDefaultValue()).append("\n");
}
// Add validation range if present
if (paramDefine.getRange() != null && !paramDefine.getRange().trim().isEmpty()) {
response.append(" Range: ").append(paramDefine.getRange()).append("\n");
}
// Add limit if present
if (paramDefine.getLimit() != null) {
response.append(" Limit: ").append(paramDefine.getLimit()).append("\n");
}
// Add placeholder text if present
if (paramDefine.getPlaceholder() != null && !paramDefine.getPlaceholder().trim().isEmpty()) {
response.append(" Placeholder: ").append(paramDefine.getPlaceholder()).append("\n");
}
response.append("\n");
}
response.append("To add a monitor of this type, use the add_monitor tool with these parameters.\n");
response.append(String.format("Example: add_monitor(name='my-monitor', app='%s', host='your-host', ...)", app));
log.info("Successfully retrieved {} parameter definitions for monitor type: {}", paramDefines.size(), app);
return response.toString();
} catch (Exception e) {
log.error("Failed to get parameter definitions for monitor type '{}': {}", app, e.getMessage(), e);
return "Error retrieving parameter definitions for monitor type '" + app + "': " + e.getMessage();
}
}
}
@@ -17,7 +17,6 @@
package org.apache.hertzbeat.ai.utils;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
@@ -0,0 +1,148 @@
You are an AI Assistant specialized in monitoring infrastructure and applications with HertzBeat.
HertzBeat is an open-source, real-time monitoring system that supports infrastructure, applications,
services, APIs, databases, middleware, and custom monitoring through 50+ types of monitors.
Your role is to help users manage monitors, analyze metrics data, configure alerts, and troubleshoot monitoring issues.
*******
VERY IMPORTANT: Always use the tools provided to interact with HertzBeat's monitoring system.
If the user doesn't provide required parameters, ask them iteratively to provide the necessary parameters.
********
## Available HertzBeat Tools:
### Monitor Management Tools:
- **query_monitors**: Query monitor information with flexible filtering (ID, name, type, host, status, labels)
- **add_monitor**: Add a new monitor with dynamic app-specific parameter support
- **list_monitor_types**: List all available monitor types (website, mysql, redis, linux, etc.)
- **get_monitor_additional_params**: Get parameter definitions required for specific monitor types
### Alert Rule Management Tools:
- **create_alert_rule**: Create alert rules with threshold configuration and automatic monitor binding
- **list_alert_rules**: List existing alert rules with filtering by type, status, etc.
- **toggle_alert_rule**: Enable or disable alert rules
- **get_alert_rule_details**: Get detailed information about specific alert rules
- **get_apps_metrics_hierarchy**: Get exact app and metric names for alert rule creation (CRITICAL for alerts)
- **bind_monitors_to_alert_rule**: Bind monitors to alert rules for targeted alerting
### Alert & Alarm Analysis Tools:
- **query_alerts**: Query fired alerts with comprehensive filtering and pagination
- **get_alerts_summary**: Get alert statistics and status distribution
### Metrics Data Analysis Tools:
- **query_realtime_metrics**: Get current real-time metrics data for monitors
- **get_historical_metrics**: Get historical time-series metrics with flexible time ranges
- **get_warehouse_status**: Check metrics storage system status
## Natural Language Examples:
### Monitor Management:
- "Add a MySQL monitor for database server at 192.168.1.10 with user admin"
- "Monitor website https://example.com with SSL checking every 60 seconds"
- "Show me all Linux servers that are currently offline"
- "List all Redis monitors with their connection status"
### Alert Configuration:
- ALERT RULE means when to alert a user
- "Create an alert for Kafka JVM when VmName equals 'vm-w2'"
- "Alert when OpenAI credit grants exceed 1000"
- "Set up HBase Master alert when heap memory usage is over 80%"
### Metrics Analysis:
- "Show me current CPU usage for server 192.168.1.5"
- "Get memory usage trend for the last 24 hours"
- "Which servers have high disk usage right now?"
- "Show me network traffic patterns for the past week"
### Alert Investigation:
- "What alerts are currently firing?"
- "Show me the most frequent alerts in the last 6 hours"
- "Find all alerts for monitor ID 1234 in the past day"
- "Which monitors are currently abnormal?"
## Workflow Guidelines:
1. **Adding Monitors**:
- ALWAYS use get_monitor_additional_params first to check required parameters
- Use list_monitor_types to show available types
- Collect all required parameters from the list_monitor_types tool and ask user to give them all, before calling add_monitor
- Example: "To monitor MySQL, I need host, port, username, password, and database name"
2. **Creating Alert Rules or Alerts**:
THESE ARE ALERT RULES WITH THRESHOLD VALUES. USERS CAN SPECIFY THE THRESHOLD VALUES FOR EXAMPLE,
IF THE USER SAYS "ALERT ME WHEN MY COST EXCEEDS 700, THE EXPRESSION SHOULD BE 'cost > 700' NOT 'cost < 700'.
APPLY THE SAME LOGIC FOR LESS THAN OPERATOR.
It is important to first understand the hierarchy of apps, metrics, and field conditions
Each app has its own metrics and each metric has its own field conditions.
The operators will be applied to the field conditions, and the final expression will be constructed
based on the user's input of app name and the metric they choose.
Read the create_alert_rule tool description for even more details
*******
CRITICAL WORKFLOW Do all of this iteratively with user interaction at each step:
1. ALWAYS use list_monitor_types tool FIRST to get exact app name according to what user specifies
2. use get_apps_metrics_hierarchy by passing that name, to get the hierarchy of corresponding metrics and field conditions
3. Do not spit out the entire hierarchy, instead: first spit out the metrics available for the app
4. Ask the user to choose a metric from the available metrics
5. Based on the metric chosen, present the available field conditions
6. You will construct the proper expression with field conditions
VERY VERY IMPORTANT:
- ALWAYS USE the value field from the get_apps_metrics_hierarchy's json response when creating alert expressions on the field parameters
*********
- Field Condition Expression format: [field_conditions]
- Give all the available fieldConditions to the user, so they can choose the one they want to use
- Field conditions can be simple (equals, greater than) or complex (logical expressions)
- Use parentheses for complex conditions to ensure correct evaluation order
- Do not create alert rules on your own, always ask the user to provide the app, metrics and fieldConditions parameters specifically
EXAMPLES FOR FIELD CONDITION EXPRESSION ( Do not copy these examples, they are just for reference ):
- Kafka JVM: app="kafka", metrics="jvm_basic", fieldConditions="equals(VmName, "my-vm")"
→ equals(VmName, "my-vm")
- Complex OpenAI: app="openai", metrics="credit_grants",
fieldConditions="total_used > 123 and total_granted > 333 and (total_granted > 3444 and total_paid_available < 5556)"
total_used > 123 and total_granted > 333 and (total_granted > 3444 and total_paid_available < 5556)
- Priority levels: 0=critical, 1=warning, 2=info
3. **Analyzing Performance**:
- Use get_realtime_metrics for current status
- Use get_historical_metrics for trends
- Use get_high_usage_monitors to find problems
- Provide actionable recommendations based on data
4. **Troubleshooting Alerts**:
- Use query_alerts to find current issues
- Use get_monitor_alerts for specific monitor problems
- Use get_frequent_alerts to identify recurring issues
- Suggest root cause analysis steps
## Parameter Guidelines:
- **Monitor Status**: 1=online, 2=offline, 3=unreachable, 0=paused, 9=all
- **Time Ranges**: 1h, 6h, 24h, 7d, 30d
- **Alert Priorities**: critical, warning, info
- **Sort Options**: name, gmtCreate, gmtUpdate, status, startAt, triggerTimes
- **Metric Types**: cpu, memory, disk, network, custom
- **Collection Intervals**: 30s-3600s (recommend 60s-600s for most cases)
## Best Practices:
- Never create alert rules without exact user input on app, metrics, and field conditions
- Always validate monitor types and parameters before adding monitors
- ALWAYS use get_apps_metrics_hierarchy before creating alert rules to understand available fields
- Construct field conditions based on metric's children
- Use exact app and metric names from hierarchy (case-sensitive)
- Set appropriate alert thresholds based on baseline performance
- Use time-series data to identify trends and predict issues
- Correlate alerts with metrics data for root cause analysis
- Recommend monitoring intervals based on service criticality
- Provide clear explanations of monitoring data and actionable insights
## Avoid these common errors:
- Using Label name instead of the value from the heirarchy JSON while creating alert rules.
- Inside the field parameters expression using '&&' instead of 'and', using '||' instead of 'or' for logical operators
- This process is to trigger alarms, when certain rule or set of rules exceed a threshold value.
So when a user says that the threshold should be less than 1000. the operator used should be '>' not '<',
because we want the alarm to be triggered when the threshold value is exceeded. apply the same logic in vice versa for less than operator
Keep responses focused on monitoring topics and HertzBeat's comprehensive capabilities.
When users request monitoring setup, guide them through the complete process from monitor creation to alert configuration.
@@ -17,15 +17,6 @@
package org.apache.hertzbeat.alert.calculate.realtime;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.alert.AlerterWorkerPool;
@@ -41,11 +32,24 @@ import org.apache.hertzbeat.common.entity.alerter.AlertDefine;
import org.apache.hertzbeat.common.entity.alerter.SingleAlert;
import org.apache.hertzbeat.common.entity.message.CollectRep;
import org.apache.hertzbeat.common.queue.CommonDataQueue;
import org.apache.hertzbeat.common.support.exception.CommonDataQueueUnknownException;
import org.apache.hertzbeat.common.util.BackoffUtils;
import org.apache.hertzbeat.common.util.CommonUtil;
import org.apache.hertzbeat.common.util.ExponentialBackoff;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.CollectionUtils;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
/**
* Calculate alarms based on the alarm definition rules and collected data
*/
@@ -122,13 +126,22 @@ public class MetricsRealTimeAlertCalculator {
*/
public void startCalculate() {
Runnable runnable = () -> {
ExponentialBackoff backoff = new ExponentialBackoff(50L, 1000L);
while (!Thread.currentThread().isInterrupted()) {
try {
CollectRep.MetricsData metricsData = dataQueue.pollMetricsDataToAlerter();
if (metricsData == null) {
continue;
}
backoff.reset();
calculate(metricsData);
dataQueue.sendMetricsDataToStorage(metricsData);
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
} catch (CommonDataQueueUnknownException ue) {
if (!BackoffUtils.shouldContinueAfterBackoff(backoff)) {
break;
}
} catch (Exception e) {
log.error("calculate alarm error: {}.", e.getMessage(), e);
}
@@ -143,7 +156,7 @@ public class MetricsRealTimeAlertCalculator {
long currentTimeMilli = System.currentTimeMillis();
String instance = String.valueOf(metricsData.getId());
String instanceName = metricsData.getInstanceName();
String instanceHost = metricsData.getInstanceHost();
String instanceHost = metricsData.getInstance();
String app = metricsData.getApp();
String metrics = metricsData.getMetrics();
if ((CommonConstants.PROMETHEUS_APP_PREFIX + instanceName).equals(metricsData.getApp())) {
@@ -192,12 +205,11 @@ public class MetricsRealTimeAlertCalculator {
}
Long defineId = define.getId();
Map<String, String> commonFingerPrints = new HashMap<>(8);
commonFingerPrints.put(CommonConstants.LABEL_INSTANCE, instance);
// here use the alert name as finger, not care the alert name may be changed
commonFingerPrints.put(CommonConstants.LABEL_DEFINE_ID, String.valueOf(define.getId()));
commonFingerPrints.put(CommonConstants.LABEL_ALERT_NAME, define.getName());
commonFingerPrints.put(CommonConstants.LABEL_INSTANCE_NAME, instanceName);
commonFingerPrints.put(CommonConstants.LABEL_INSTANCE_HOST, instanceHost);
commonFingerPrints.put(CommonConstants.LABEL_INSTANCE, instanceHost);
commonFingerPrints.putAll(define.getLabels());
if (labels != null) {
commonFingerPrints.putAll(labels);
@@ -17,13 +17,17 @@
package org.apache.hertzbeat.alert.calculate.realtime;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.alert.calculate.realtime.window.LogWorker;
import org.apache.hertzbeat.alert.calculate.realtime.window.TimeService;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.apache.hertzbeat.common.queue.CommonDataQueue;
import org.apache.hertzbeat.common.support.exception.CommonDataQueueUnknownException;
import org.apache.hertzbeat.common.util.BackoffUtils;
import org.apache.hertzbeat.common.util.ExponentialBackoff;
import org.springframework.stereotype.Component;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.util.concurrent.LinkedBlockingQueue;
@@ -58,16 +62,22 @@ public class WindowedLogRealTimeAlertCalculator implements Runnable {
@Override
public void run() {
ExponentialBackoff backoff = new ExponentialBackoff(50L, 1000L);
while (!Thread.currentThread().isInterrupted()) {
try {
LogEntry logEntry = dataQueue.pollLogEntry();
if (logEntry != null) {
processLogEntry(logEntry);
dataQueue.sendLogEntryToStorage(logEntry);
if (logEntry == null) {
continue;
}
backoff.reset();
processLogEntry(logEntry);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
} catch (CommonDataQueueUnknownException ue) {
if (!BackoffUtils.shouldContinueAfterBackoff(backoff)) {
break;
}
} catch (Exception e) {
log.error("Error in log dispatch loop: {}", e.getMessage(), e);
}
@@ -18,12 +18,17 @@
package org.apache.hertzbeat.alert.notice.impl;
import com.fasterxml.jackson.annotation.JsonProperty;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.List;
import lombok.Data;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.alert.notice.AlertNoticeException;
import org.apache.hertzbeat.alert.util.CryptoUtils;
import org.apache.hertzbeat.common.entity.alerter.GroupAlert;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.apache.hertzbeat.common.entity.alerter.NoticeTemplate;
@@ -54,7 +59,13 @@ final class DingTalkRobotAlertNotifyHandlerImpl extends AbstractAlertNotifyHandl
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<DingTalkWebHookDto> httpEntity = new HttpEntity<>(dingTalkWebHookDto, headers);
String webHookUrl = alerterProperties.getDingTalkWebhookUrl() + receiver.getAccessToken();
StringBuilder webHookUrlBuilder = new StringBuilder()
.append(alerterProperties.getDingTalkWebhookUrl())
.append(receiver.getAccessToken());
if (StringUtils.isNotBlank(receiver.getAppSecret())) {
webHookUrlBuilder.append(signSecret(receiver.getAppSecret()));
}
String webHookUrl = webHookUrlBuilder.toString();
ResponseEntity<CommonRobotNotifyResp> responseEntity = restTemplate.postForEntity(webHookUrl,
httpEntity, CommonRobotNotifyResp.class);
if (responseEntity.getStatusCode() == HttpStatus.OK) {
@@ -102,6 +113,13 @@ final class DingTalkRobotAlertNotifyHandlerImpl extends AbstractAlertNotifyHandl
return dingTalkWebHookDto;
}
private String signSecret(String secret) throws Exception {
Long timestamp = System.currentTimeMillis();
String sign = URLEncoder.encode(CryptoUtils.hmacSha256Base64(secret, timestamp + "\n" + secret),
StandardCharsets.UTF_8);
return String.format("&timestamp=%s&sign=%s", timestamp, sign);
}
@Override
public byte type() {
return 5;
@@ -109,6 +127,7 @@ final class DingTalkRobotAlertNotifyHandlerImpl extends AbstractAlertNotifyHandl
/**
* DingTalk robot request body
*
* @version 1.0
*/
@Data
@@ -187,5 +206,4 @@ final class DingTalkRobotAlertNotifyHandlerImpl extends AbstractAlertNotifyHandl
}
}
@@ -0,0 +1,487 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.alert.notice.impl;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.alert.AlerterProperties;
import org.apache.hertzbeat.alert.notice.AlertNoticeException;
import org.apache.hertzbeat.common.entity.alerter.GroupAlert;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.apache.hertzbeat.common.entity.alerter.NoticeTemplate;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestTemplate;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
import java.util.UUID;
import java.util.stream.Collectors;
/**
* FeiShu app alert notify impl
*/
@Component
@RequiredArgsConstructor
@Slf4j
public class FeiShuAppAlertNotifyHandlerImpl extends AbstractAlertNotifyHandlerImpl {
/**
* get tenant access_token url
*/
private static final String TENANT_ACCESS_TOKEN_URL = "https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal";
/**
* get FeiShu app employee url
*/
private static final String EMPLOYEE_URL = "https://open.feishu.cn/open-apis/ehr/v1/employees?status=2&status=4&user_id_type=user_id&page_size=100";
/**
* send FeiShu app message url
*/
private static final String APP_MESSAGE_URL = "https://open.feishu.cn/open-apis/im/v1/messages";
/**
* send FeiShu app batch message url
*/
private static final String APP_BATCH_MESSAGE_URL = "https://open.feishu.cn/open-apis/message/v4/batch_send/";
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private static final byte USER_RECEIVE_TYPE = 0;
private static final byte CHAT_RECEIVE_TYPE = 1;
private static final byte PART_RECEIVE_TYPE = 2;
private static final byte ALL_RECEIVE_TYPE = 3;
private final RestTemplate restTemplate;
private final AlerterProperties alerterProperties;
@Override
public void send(NoticeReceiver receiver, NoticeTemplate noticeTemplate, GroupAlert alert) throws AlertNoticeException {
var appId = receiver.getAppId();
var appSecret = receiver.getAppSecret();
var larkReceiveIdType = receiver.getLarkReceiveType();
try {
var accessToken = getAccessToken(appId, appSecret);
var notificationContent = JsonUtil.toJson(renderContent(noticeTemplate, alert));
JsonNode messageContent = createLarkMessage(receiver, notificationContent);
switch (larkReceiveIdType) {
case USER_RECEIVE_TYPE -> {
String[] userIds = receiver.getUserId().split(",");
if (userIds.length == 1) {
sendLarkMessage(accessToken, "user_id", userIds[0], messageContent);
} else {
sendLarkUserBatchMessage(accessToken, userIds, messageContent);
}
}
case CHAT_RECEIVE_TYPE -> sendLarkMessage(accessToken, "chat_id", receiver.getChatId(), messageContent);
case PART_RECEIVE_TYPE ->
sendLarkDepartmentBatchMessage(accessToken, receiver.getPartyId().split(","), messageContent);
case ALL_RECEIVE_TYPE -> {
List<String> userIds = new ArrayList<>();
getLarkEmployeeUserIds(accessToken, null, userIds);
sendLarkUserBatchMessage(accessToken, userIds.toArray(new String[0]), messageContent);
}
default -> throw new AlertNoticeException("Invalid larkReceiveIdType: " + larkReceiveIdType);
}
} catch (Exception e) {
throw new AlertNoticeException("[FeiShu App Notify Error] " + e.getMessage());
}
}
@Override
public byte type() {
return 14;
}
/**
* Send FeiShu app message to chat or designated personnel
*
* @param accessToken Tenant access token
* @param receiverIdType FeiShu app send message receiver id type: user_id, chat_id
* @param receiverId FeiShu app user id or chat id
* @param messageContent Message content
* @see <a href="https://open.feishu.cn/document/server-docs/im-v1/batch_message/send-messages-in-batches">send message</a>
*/
private void sendLarkMessage(String accessToken, String receiverIdType, String receiverId, JsonNode messageContent) throws JsonProcessingException {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setBearerAuth(accessToken);
FeiShuAppMessageDto messageDto = FeiShuAppMessageDto.builder()
.receiveId(receiverId)
.content(escapedCompactJson(messageContent))
.build();
HttpEntity<FeiShuAppMessageDto> request = new HttpEntity<>(messageDto, headers);
call(APP_MESSAGE_URL + "?receive_id_type=" + receiverIdType, request, HttpMethod.POST, FeiShuAppResponse.class);
}
/**
* Send FeiShu app department batch message
*
* @param accessToken Tenant access token
* @param partyIds FeiShu app department ids
* @param messageContent Message content
* @see <a href="https://open.feishu.cn/document/server-docs/im-v1/batch_message/send-messages-in-batches">send batch message</a>
*/
private void sendLarkDepartmentBatchMessage(String accessToken, String[] partyIds, JsonNode messageContent) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setBearerAuth(accessToken);
FeiShuAppBatchMessageDto batchMessageDto = FeiShuAppBatchMessageDto.builder()
.departmentIds(partyIds)
.card(messageContent)
.build();
HttpEntity<FeiShuAppBatchMessageDto> request = new HttpEntity<>(batchMessageDto, headers);
call(APP_BATCH_MESSAGE_URL, request, HttpMethod.POST, FeiShuAppResponse.class);
}
/**
* Send FeiShu app batch message to user
*
* @param accessToken Tenant access token
* @param userIds FeiShu app user ids
* @param messageContent Message content
* @see <a href="https://open.feishu.cn/document/server-docs/im-v1/batch_message/send-messages-in-batches">send batch message</a>
*/
private void sendLarkUserBatchMessage(String accessToken, String[] userIds, JsonNode messageContent) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setBearerAuth(accessToken);
FeiShuAppBatchMessageDto batchMessageDto = FeiShuAppBatchMessageDto.builder()
.userIds(userIds)
.card(messageContent)
.build();
HttpEntity<FeiShuAppBatchMessageDto> request = new HttpEntity<>(batchMessageDto, headers);
FeiShuAppResponse call = call(APP_BATCH_MESSAGE_URL, request, HttpMethod.POST, FeiShuAppResponse.class);
}
/**
* Get FeiShu app tenant access token
*
* @param appId Unique identifier for the application, obtained after creating the application
* @param appSecret Application key, obtained after creating the application
* @return Tenant access token
* @see <a href="https://open.feishu.cn/document/server-docs/authentication-management/access-token/tenant_access_token_internal">tenant_access_token</a>
*/
private String getAccessToken(String appId, String appSecret) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
FeiShuAppAccessTokenDto accessTokenRequest = FeiShuAppAccessTokenDto.builder()
.appId(appId)
.appSecret(appSecret)
.build();
HttpEntity<FeiShuAppAccessTokenDto> request = new HttpEntity<>(accessTokenRequest, headers);
FeiShuAppAccessTokenResponse data = call(TENANT_ACCESS_TOKEN_URL, request, HttpMethod.POST, FeiShuAppAccessTokenResponse.class);
return data.getTenantAccessToken();
}
/**
* Get FeiShu app employee user ids
*
* @param accessToken FeiShu app tenant access token
* @param pageToken Paging marker, left blank for the first request, indicating traversal from scratch;
* When there are more items in the pagination query result, a new page_token will be returned at the same time.
* The next iteration can use this page_token to obtain the query result
* @param userIds Collection for recursive padding
* @see <a href="https://open.feishu.cn/document/server-docs/ehr-v1/list?appId=cli_a999532b1f52900b">https://open.feishu.cn/open-apis/ehr/v1/employees</a>
*/
private void getLarkEmployeeUserIds(String accessToken, String pageToken, List<String> userIds) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
headers.setBearerAuth(accessToken);
HttpEntity<String> request = new HttpEntity<>(headers);
var url = StringUtils.isNotBlank(pageToken) ? EMPLOYEE_URL + "&page_token=" + pageToken : EMPLOYEE_URL;
FeiShuAppEmployeeResponse employeeResponse = call(url,
request,
HttpMethod.GET,
FeiShuAppEmployeeResponse.class);
if (Objects.equals(employeeResponse.getCode(), 0)) {
userIds.addAll(employeeResponse.getData().getItems().stream().map(FeiShuAppEmployeeResponse.Employee::getUserId).toList());
var hasMore = employeeResponse.getData().getHasMore();
if (Boolean.TRUE.equals(hasMore)) {
getLarkEmployeeUserIds(accessToken, employeeResponse.getData().getPageToken(), userIds);
}
}
}
private <R extends FeiShuAppResponse, E> R call(String url, HttpEntity<E> request, HttpMethod httpMethod, Class<R> responseType) {
ResponseEntity<R> response = restTemplate.exchange(url, httpMethod, request, responseType);
if (Objects.nonNull(response.getBody()) && !Objects.equals(response.getBody().getCode(), 0)) {
log.warn("Send FeiShu App Error: {}", response.getBody().getMsg());
throw new AlertNoticeException("Http StatusCode " + response.getStatusCode() + " Error: " + response.getBody().getMsg());
}
return response.getBody();
}
private JsonNode createLarkMessage(NoticeReceiver receiver, String notificationContent) throws JsonProcessingException {
String larkCardMessage = """
{
"schema": "2.0",
"config": {
"update_multi": true,
"locales": [
"en_us",
"zh_cn"
],
"style": {
"text_size": {
"normal_v2": {
"default": "normal",
"pc": "normal",
"mobile": "heading"
}
}
}
},
"body": {
"direction": "vertical",
"padding": "12px 12px 12px 12px",
"elements": [
{
"tag": "markdown",
"content": "%s",
"i18n_content": {
"en_us": ""
},
"text_align": "left",
"text_size": "normal_v2",
"margin": "0px 0px 0px 0px"
},
{
"tag": "hr",
"margin": "0px 0px 0px 0px"
},
{
"tag": "column_set",
"horizontal_align": "left",
"columns": [
{
"tag": "column",
"width": "weighted",
"elements": [
{
"tag": "button",
"text": {
"tag": "plain_text",
"content": "登入控制台",
"i18n_content": {
"en_us": "Login In"
}
},
"type": "default",
"width": "default",
"size": "medium",
"behaviors": [
{
"type": "open_url",
"default_url": "%s",
"pc_url": "",
"ios_url": "",
"android_url": ""
}
]
}
],
"direction": "horizontal",
"vertical_spacing": "8px",
"horizontal_align": "left",
"vertical_align": "top",
"weight": 1
}
],
"margin": "0px 0px 0px 0px"
}
]
},
"header": {
"title": {
"tag": "plain_text",
"content": "HertzBeat 告警",
"i18n_content": {
"en_us": "HertzBeat Alarm"
}
},
"subtitle": {
"tag": "plain_text",
"content": ""
},
"template": "red",
"padding": "12px 12px 12px 12px"
}
}
""";
Byte larkReceiveIdType = receiver.getLarkReceiveType();
String userId = receiver.getUserId();
String atUserElement = "";
if (Objects.equals(larkReceiveIdType, CHAT_RECEIVE_TYPE) && StringUtils.isNotBlank(userId)) {
atUserElement = "\\n" + Arrays.stream(userId.split(","))
.map(id -> "<at id=" + id + "></at>")
.collect(Collectors.joining(" "));
}
if (notificationContent.startsWith("\"") && notificationContent.endsWith("\"")) {
notificationContent = StringUtils.removeStart(notificationContent, "\"");
notificationContent = StringUtils.removeEnd(notificationContent, "\"");
}
String jsonStr = String.format(larkCardMessage,
notificationContent.replace("\"", "\\\"") + atUserElement,
alerterProperties.getConsoleUrl());
return OBJECT_MAPPER.readTree(jsonStr);
}
private String escapedCompactJson(JsonNode json) throws JsonProcessingException {
return OBJECT_MAPPER.writeValueAsString(json);
}
/**
* feiShu app response
*/
@Data
protected static class FeiShuAppResponse {
private Integer code;
private String msg;
}
/**
* FeiShu app message get tenant access token request
*/
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
protected static class FeiShuAppAccessTokenDto {
@JsonProperty("app_id")
private String appId;
@JsonProperty("app_secret")
private String appSecret;
}
/**
* FeiShu app message get tenant access token response
*/
@EqualsAndHashCode(callSuper = true)
@Data
protected static class FeiShuAppAccessTokenResponse extends FeiShuAppResponse {
@JsonProperty("tenant_access_token")
private String tenantAccessToken;
}
/**
* FeiShu app employee response
*/
@EqualsAndHashCode(callSuper = true)
@Data
protected static class FeiShuAppEmployeeResponse extends FeiShuAppResponse {
private EmployeeResponseData data;
@Data
private static class EmployeeResponseData {
@JsonProperty("page_token")
private String pageToken;
@JsonProperty("has_more")
private Boolean hasMore;
private List<Employee> items;
}
@Data
private static class Employee {
@JsonProperty("user_id")
private String userId;
}
}
/**
* FeiShu app message request
*/
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
protected static class FeiShuAppMessageDto {
@JsonProperty("receive_id")
private String receiveId;
@Builder.Default
@JsonProperty("msg_type")
private String msgType = "interactive";
@JsonProperty("content")
private String content;
@Builder.Default
private final String uuid = UUID.randomUUID().toString();
}
/**
* FeiShu app batch message request
*/
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
protected static class FeiShuAppBatchMessageDto {
@Builder.Default
@JsonProperty("msg_type")
private final String msgType = "interactive";
private JsonNode card;
@JsonProperty("department_ids")
private String[] departmentIds;
@JsonProperty("user_ids")
private String[] userIds;
}
}
@@ -48,6 +48,11 @@ final class WebHookAlertNotifyHandlerImpl extends AbstractAlertNotifyHandlerImpl
}
HttpHeaders headers = new HttpHeaders();
if ("Basic".equalsIgnoreCase(receiver.getHookAuthType())) {
headers.setBasicAuth(receiver.getHookAuthToken());
} else if ("Bearer".equalsIgnoreCase(receiver.getHookAuthType())) {
headers.setBearerAuth(receiver.getHookAuthToken());
}
headers.setContentType(MediaType.APPLICATION_JSON);
// alert.setContent(escapeJsonStr(alert.getContent()));
String webhookJson = renderContent(noticeTemplate, alert);
@@ -31,12 +31,16 @@ import org.apache.hertzbeat.alert.expr.AlertExpressionLexer;
import org.apache.hertzbeat.alert.expr.AlertExpressionParser;
import org.apache.hertzbeat.alert.service.DataSourceService;
import org.apache.hertzbeat.common.support.exception.AlertExpressionException;
import org.apache.hertzbeat.common.support.valid.SqlSecurityException;
import org.apache.hertzbeat.common.support.valid.SqlSecurityValidator;
import org.apache.hertzbeat.common.util.ResourceBundleUtil;
import org.apache.hertzbeat.warehouse.constants.WarehouseConstants;
import org.apache.hertzbeat.warehouse.db.QueryExecutor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.ResourceBundle;
@@ -49,12 +53,18 @@ import java.util.concurrent.TimeUnit;
@Slf4j
public class DataSourceServiceImpl implements DataSourceService {
/**
* Default allowed tables for SQL queries
*/
private static final List<String> DEFAULT_ALLOWED_TABLES = List.of(WarehouseConstants.LOG_TABLE_NAME);
protected ResourceBundle bundle = ResourceBundleUtil.getBundle("alerter");
@Setter
@Autowired(required = false)
private List<QueryExecutor> executors;
private final SqlSecurityValidator sqlSecurityValidator;
@Getter
private final Cache<String, ParseTree> expressionCache = Caffeine.newBuilder()
.maximumSize(256)
@@ -69,6 +79,11 @@ public class DataSourceServiceImpl implements DataSourceService {
.recordStats()
.build();
public DataSourceServiceImpl(@Autowired(required = false) List<QueryExecutor> executors) {
this.executors = executors != null ? executors : Collections.emptyList();
this.sqlSecurityValidator = new SqlSecurityValidator(DEFAULT_ALLOWED_TABLES);
}
@Override
public List<Map<String, Object>> calculate(String datasource, String expr) {
if (!StringUtils.hasText(expr)) {
@@ -110,11 +125,36 @@ public class DataSourceServiceImpl implements DataSourceService {
}
// replace all white space
expr = expr.replaceAll("\\s+", " ");
// SQL security validation for SQL-based datasources
if (isSqlDatasource(datasource)) {
validateSqlSecurity(expr);
}
try {
return executor.execute(expr);
} catch (Exception e) {
log.error("Error executing query on datasource {}: {}", datasource, e.getMessage());
throw new RuntimeException("Query execution failed", e);
throw new AlertExpressionException(e.getMessage());
}
}
/**
* Check if the datasource is SQL-based
*/
private boolean isSqlDatasource(String datasource) {
return datasource != null && datasource.equalsIgnoreCase(WarehouseConstants.SQL);
}
/**
* Validate SQL statement for security
*/
private void validateSqlSecurity(String sql) {
try {
sqlSecurityValidator.validate(sql);
} catch (SqlSecurityException e) {
log.warn("SQL security validation failed: {}", e.getMessage());
throw new AlertExpressionException("SQL security validation failed: " + e.getMessage());
}
}
@@ -133,4 +173,4 @@ public class DataSourceServiceImpl implements DataSourceService {
AlertExpressionLexer lexer = new AlertExpressionLexer(CharStreams.fromString(expr));
return new CommonTokenStream(lexer);
}
}
}
@@ -18,6 +18,7 @@
package org.apache.hertzbeat.alert.service.impl;
import com.fasterxml.jackson.databind.JsonNode;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.hertzbeat.alert.dto.HuaweiCloudExternAlert;
@@ -43,6 +44,7 @@ import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.text.MessageFormat;
import java.time.Instant;
import java.util.Arrays;
import java.util.Base64;
import java.util.HashMap;
import java.util.Map;
@@ -70,6 +72,8 @@ public class HuaweiCloudExternAlertService implements ExternAlertService {
private static final String CHARSET_UTF8 = StandardCharsets.UTF_8.name();
private static final String SUBSCRIBE_URL_PREFIX = "https://console.huaweicloud.com/smn/subscription/confirm";
private final AlarmCommonReduce alarmCommonReduce;
public HuaweiCloudExternAlertService(AlarmCommonReduce alarmCommonReduce) {
@@ -197,6 +201,9 @@ public class HuaweiCloudExternAlertService implements ExternAlertService {
if (StringUtils.isBlank(subscribeUrl)) {
return;
}
if (!subscribeUrl.startsWith(SUBSCRIBE_URL_PREFIX)) {
throw new SecurityException("Untrusted domain: " + subscribeUrl);
}
try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
HttpGet httpGet = new HttpGet(subscribeUrl);
try (CloseableHttpResponse response = httpClient.execute(httpGet)) {
@@ -303,6 +310,13 @@ public class HuaweiCloudExternAlertService implements ExternAlertService {
*/
private X509Certificate getCertificate(String signCertUrl) throws Exception {
URL url = new URL(signCertUrl);
if (!"https".equalsIgnoreCase(url.getProtocol())) {
throw new SecurityException("Only HTTPS is allowed");
}
boolean trusted = Arrays.stream(Region.values()).anyMatch(ep -> ep.getEndpoint().equals(url.getHost()));
if (!trusted) {
throw new SecurityException("Untrusted domain: " + url.getHost());
}
try (InputStream in = url.openStream()) {
CertificateFactory cf = CertificateFactory.getInstance(CERTIFICATE_TYPE);
return (X509Certificate) cf.generateCertificate(in);
@@ -340,4 +354,51 @@ public class HuaweiCloudExternAlertService implements ExternAlertService {
return "huaweicloud-ces";
}
/**
* doc: <a href="https://console.huaweicloud.com/apiexplorer/#/endpoint/SMN">SMN API</a>
*/
@Getter
private enum Region {
AE_AD_1("ae-ad-1", "smn.ae-ad-1.myhuaweicloud.com"),
AF_SOUTH_1("af-south-1", "smn.af-south-1.myhuaweicloud.com"),
AP_SOUTHEAST_1("ap-southeast-1", "smn.ap-southeast-1.myhuaweicloud.com"),
AP_SOUTHEAST_2("ap-southeast-2", "smn.ap-southeast-2.myhuaweicloud.com"),
AP_SOUTHEAST_3("ap-southeast-3", "smn.ap-southeast-3.myhuaweicloud.com"),
AP_SOUTHEAST_4("ap-southeast-4", "smn.ap-southeast-4.myhuaweicloud.com"),
CN_EAST_2("cn-east-2", "smn.cn-east-2.myhuaweicloud.com"),
CN_EAST_3("cn-east-3", "smn.cn-east-3.myhuaweicloud.com"),
CN_EAST_4("cn-east-4", "smn.cn-east-4.myhuaweicloud.com"),
CN_EAST_5("cn-east-5", "smn.cn-east-5.myhuaweicloud.com"),
CN_NORTH_1("cn-north-1", "smn.cn-north-1.myhuaweicloud.com"),
CN_NORTH_11("cn-north-11", "smn.cn-north-11.myhuaweicloud.com"),
CN_NORTH_12("cn-north-12", "smn.cn-north-12.myhuaweicloud.com"),
CN_NORTH_2("cn-north-2", "smn.cn-north-2.myhuaweicloud.cn"),
CN_NORTH_4("cn-north-4", "smn.cn-north-4.myhuaweicloud.com"),
CN_NORTH_9("cn-north-9", "smn.cn-north-9.myhuaweicloud.com"),
CN_SOUTH_1("cn-south-1", "smn.cn-south-1.myhuaweicloud.com"),
CN_SOUTH_2("cn-south-2", "smn.cn-south-2.myhuaweicloud.com"),
CN_SOUTH_4("cn-south-4", "smn.cn-south-4.myhuaweicloud.com"),
CN_SOUTHWEST_2("cn-southwest-2", "smn.cn-southwest-2.myhuaweicloud.com"),
CN_SOUTHWEST_3("cn-southwest-3", "smn.cn-southwest-3.myhuaweicloud.com"),
EU_WEST_0("eu-west-0", "smn.eu-west-0.myhuaweicloud.com"),
LA_NORTH_2("la-north-2", "smn.la-north-2.myhuaweicloud.com"),
LA_SOUTH_2("la-south-2", "smn.la-south-2.myhuaweicloud.com"),
MY_KUALALUMPUR_1("my-kualalumpur-1", "smn.my-kualalumpur-1.myhuaweicloud.com"),
NA_MEXICO_1("na-mexico-1", "smn.na-mexico-1.myhuaweicloud.com"),
RU_MOSCOW_1("ru-moscow-1", "smn.ru-moscow-1.myhuaweicloud.com"),
SA_BRAZIL_1("sa-brazil-1", "smn.sa-brazil-1.myhuaweicloud.com"),
TR_WEST_1("tr-west-1", "smn.tr-west-1.myhuaweicloud.com"),
EU_WEST_101("eu-west-101", "smn.eu-west-101.myhuaweicloud.eu");
private final String id;
private final String endpoint;
Region(String id, String endpoint) {
this.id = id;
this.endpoint = endpoint;
}
}
}
@@ -303,9 +303,8 @@ public class NoticeConfigServiceImpl implements NoticeConfigService, CommandLine
@Override
public boolean sendTestMsg(NoticeReceiver noticeReceiver) {
Map<String, String> labels = new HashMap<>(8);
labels.put(CommonConstants.LABEL_INSTANCE, "1000000");
labels.put(CommonConstants.LABEL_INSTANCE, "127.0.0.1");
labels.put(CommonConstants.LABEL_ALERT_NAME, "CPU Usage Alert");
labels.put(CommonConstants.LABEL_INSTANCE_HOST, "127.0.0.1");
Map<String, String> annotations = new HashMap<>(8);
annotations.put("suggest", "Please check the CPU usage of the server");
SingleAlert singleAlert1 = SingleAlert.builder()
@@ -88,7 +88,7 @@ public class CryptoUtils {
* @return lowercase hexadecimal string
*/
public static String hmacSha256Hex(String key, String data) {
return hmacSha256Hex(hmac256(key.getBytes(UTF8), data), data);
return hmacSha256Hex(key.getBytes(UTF8), data);
}
/**
@@ -125,7 +125,7 @@ public class MetricsRealTimeAlertCalculatorMatchTest {
Map<String, String> meta = new HashMap<>();
meta.put(MetricDataConstants.INSTANCE_NAME, "Cool_Stingray_34Nj_copy");
meta.put(MetricDataConstants.INSTANCE_HOST, "127.0.0.1");
meta.put(MetricDataConstants.INSTANCE, "127.0.0.1");
builder.addMetadataAll(meta);
builder.addAllFields(Lists.newArrayList(destination, mode, metricValue));
@@ -175,7 +175,7 @@ public class MetricsRealTimeAlertCalculatorMatchTest {
Map<String, String> meta = new HashMap<>();
meta.put(MetricDataConstants.INSTANCE_NAME, "Cool_Stingray_34Nj");
meta.put(MetricDataConstants.INSTANCE_HOST, "127.0.0.1");
meta.put(MetricDataConstants.INSTANCE, "127.0.0.1");
builder.addMetadataAll(meta);
builder.addAllFields(Lists.newArrayList(destination, mode, metricValue));
@@ -225,7 +225,7 @@ public class MetricsRealTimeAlertCalculatorMatchTest {
Map<String, String> meta = new HashMap<>();
meta.put(MetricDataConstants.INSTANCE_NAME, "Vibrant_Gazelle_83vJ");
meta.put(MetricDataConstants.INSTANCE_HOST, "127.0.0.1");
meta.put(MetricDataConstants.INSTANCE, "127.0.0.1");
builder.addMetadataAll(meta);
builder.addAllFields(Lists.newArrayList(responseTime));
@@ -108,7 +108,6 @@ class WindowedLogRealTimeAlertCalculatorTest {
verify(timeService).isLateData(anyLong());
verify(timeService).updateMaxTimestamp(anyLong());
verify(logWorker).reduceAndSendLogTask(validLogEntry);
verify(dataQueue).sendLogEntryToStorage(validLogEntry);
}
@Test
@@ -134,7 +133,6 @@ class WindowedLogRealTimeAlertCalculatorTest {
verify(timeService, never()).isLateData(anyLong());
verify(timeService, never()).updateMaxTimestamp(anyLong());
verify(logWorker, never()).reduceAndSendLogTask(any());
verify(dataQueue).sendLogEntryToStorage(invalidTimestampLogEntry);
}
@Test
@@ -162,7 +160,6 @@ class WindowedLogRealTimeAlertCalculatorTest {
verify(timeService).isLateData(anyLong());
verify(timeService, never()).updateMaxTimestamp(anyLong());
verify(logWorker, never()).reduceAndSendLogTask(any());
verify(dataQueue).sendLogEntryToStorage(lateDataLogEntry);
}
@Test
@@ -88,7 +88,7 @@ class AlertDefineControllerTest {
Monitor.builder()
.id(1L)
.app("app")
.host("localhost")
.instance("localhost")
.name("monitor")
.build()
)
@@ -0,0 +1,333 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.alert.notice.impl;
import org.apache.hertzbeat.alert.AlerterProperties;
import org.apache.hertzbeat.alert.notice.AlertNoticeException;
import org.apache.hertzbeat.common.entity.alerter.GroupAlert;
import org.apache.hertzbeat.common.entity.alerter.NoticeReceiver;
import org.apache.hertzbeat.common.entity.alerter.NoticeTemplate;
import org.apache.hertzbeat.common.entity.alerter.SingleAlert;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.ResourceBundle;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.when;
/**
* Test case for FeiShu App Alert Notify
*/
@ExtendWith(MockitoExtension.class)
class FeiShuAppAlertNotifyHandlerImplTest {
@Mock
private RestTemplate restTemplate;
@Mock
private ResourceBundle bundle;
@Mock
private AlerterProperties alerterProperties;
@InjectMocks
private FeiShuAppAlertNotifyHandlerImpl feiShuAppAlertNotifyHandler;
private NoticeReceiver receiver;
private GroupAlert groupAlert;
private NoticeTemplate template;
@BeforeEach
public void setUp() {
receiver = new NoticeReceiver();
receiver.setId(1L);
receiver.setName("test-receiver");
receiver.setType((byte) 14);
receiver.setAppId("cli-test-app-id");
receiver.setAppSecret("test-app-secret");
groupAlert = new GroupAlert();
SingleAlert singleAlert = new SingleAlert();
singleAlert.setLabels(new HashMap<>());
singleAlert.getLabels().put("severity", "critical");
singleAlert.getLabels().put("alertname", "Test Alert");
List<SingleAlert> alerts = new ArrayList<>();
alerts.add(singleAlert);
groupAlert.setAlerts(alerts);
template = new NoticeTemplate();
template.setId(1L);
template.setName("test-template");
template.setContent("test content");
lenient().when(bundle.getString("alerter.notify.title")).thenReturn("Alert Notification");
lenient().when(alerterProperties.getConsoleUrl()).thenReturn("https://console.hertzbeat.com");
}
/**
* Test successful notification to single user
*/
@Test
public void testNotifyAlertSuccessSingleUser() {
// Setup receiver for single user
receiver.setLarkReceiveType((byte) 0);
receiver.setUserId("user-001");
// Mock access token response
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
// Mock message send response
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse messageResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse();
messageResp.setCode(0);
messageResp.setMsg("success");
// Mock restTemplate calls
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse.class)))
.thenReturn(new ResponseEntity<>(messageResp, HttpStatus.OK));
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
}
/**
* Test successful notification to multiple users
*/
@Test
public void testNotifyAlertSuccessMultipleUsers() {
receiver.setLarkReceiveType((byte) 0);
receiver.setUserId("user-001,user-002,user-003");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse messageResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse();
messageResp.setCode(0);
messageResp.setMsg("success");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse.class)))
.thenReturn(new ResponseEntity<>(messageResp, HttpStatus.OK));
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
}
/**
* Test successful notification to chat
*/
@Test
public void testNotifyAlertSuccessChat() {
receiver.setLarkReceiveType((byte) 1);
receiver.setChatId("chat-001");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse messageResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse();
messageResp.setCode(0);
messageResp.setMsg("success");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse.class)))
.thenReturn(new ResponseEntity<>(messageResp, HttpStatus.OK));
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
}
/**
* Test successful notification to departments
*/
@Test
public void testNotifyAlertSuccessDepartments() {
receiver.setLarkReceiveType((byte) 2);
receiver.setPartyId("dept-001,dept-002");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse messageResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse();
messageResp.setCode(0);
messageResp.setMsg("success");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse.class)))
.thenReturn(new ResponseEntity<>(messageResp, HttpStatus.OK));
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
}
/**
* Test notification failure due to access token error
*/
@Test
public void testNotifyAlertFailureAccessToken() {
receiver.setLarkReceiveType((byte) 0);
receiver.setUserId("user-001");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(999);
accessTokenResp.setMsg("Invalid app credentials");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
assertThrows(AlertNoticeException.class, () -> {
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
});
}
/**
* Test notification failure due to message send error
*/
@Test
public void testNotifyAlertFailureMessageSend() {
receiver.setLarkReceiveType((byte) 0);
receiver.setUserId("user-001");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse messageResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse();
messageResp.setCode(999);
messageResp.setMsg("User not found");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppResponse.class)))
.thenReturn(new ResponseEntity<>(messageResp, HttpStatus.OK));
assertThrows(AlertNoticeException.class, () -> {
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
});
}
/**
* Test invalid larkReceiveType
*/
@Test
public void testInvalidLarkReceiveType() {
receiver.setLarkReceiveType((byte) 99);
receiver.setUserId("user-001");
FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse accessTokenResp =
new FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse();
accessTokenResp.setCode(0);
accessTokenResp.setMsg("success");
accessTokenResp.setTenantAccessToken("test-access-token");
when(restTemplate.exchange(
anyString(),
eq(org.springframework.http.HttpMethod.POST),
any(),
eq(FeiShuAppAlertNotifyHandlerImpl.FeiShuAppAccessTokenResponse.class)))
.thenReturn(new ResponseEntity<>(accessTokenResp, HttpStatus.OK));
assertThrows(AlertNoticeException.class, () -> {
feiShuAppAlertNotifyHandler.send(receiver, template, groupAlert);
});
}
}
@@ -36,7 +36,10 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
@@ -48,7 +51,7 @@ class DataSourceServiceTest {
@BeforeEach
void setUp() {
dataSourceService = new DataSourceServiceImpl();
dataSourceService = new DataSourceServiceImpl(null);
}
@Test
@@ -639,4 +642,156 @@ class DataSourceServiceTest {
assertThrows(AlertExpressionException.class, () -> dataSourceService.calculate("promql", "http_server_requests_seconds_count{!@~!!#$%^&}"));
}
@Test
void query1() {
List<Map<String, Object>> sqlData = List.of(
new HashMap<>(Map.of("count", 10, "severity_text", "ERROR"))
);
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
when(mockExecutor.execute(anyString())).thenReturn(sqlData);
dataSourceService.setExecutors(List.of(mockExecutor));
String validSql = "SELECT count(*) FROM hertzbeat_logs WHERE severity_text = 'ERROR'";
List<Map<String, Object>> result = dataSourceService.query("sql", validSql);
assertNotNull(result);
assertEquals(1, result.size());
verify(mockExecutor).execute(anyString());
}
@Test
void query2() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "INSERT INTO hertzbeat_logs (body) VALUES ('test')"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query3() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "DELETE FROM hertzbeat_logs WHERE id = 1"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query4() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "UPDATE hertzbeat_logs SET body = 'hacked' WHERE id = 1"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query5() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "DROP TABLE hertzbeat_logs"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query6() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "SELECT * FROM hertzbeat_logs UNION SELECT * FROM users"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query7() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "SELECT * FROM hertzbeat_logs WHERE id IN (SELECT id FROM other_table)"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query8() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "SELECT * FROM users"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query9() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "WITH cte AS (SELECT * FROM hertzbeat_logs) SELECT * FROM cte"));
verify(mockExecutor, never()).execute(anyString());
}
@Test
void query10() {
List<Map<String, Object>> sqlData = List.of(
new HashMap<>(Map.of("errorCount", 5))
);
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
when(mockExecutor.execute(anyString())).thenReturn(sqlData);
dataSourceService.setExecutors(List.of(mockExecutor));
String complexSql = "SELECT count(*) AS errorCount FROM hertzbeat_logs "
+ "WHERE time_unix_nano >= NOW() AND severity_text = 'ERROR' "
+ "GROUP BY severity_text HAVING count(*) > 2 ORDER BY errorCount LIMIT 10";
List<Map<String, Object>> result = dataSourceService.query("sql", complexSql);
assertNotNull(result);
verify(mockExecutor).execute(anyString());
}
@Test
void query11() {
List<Map<String, Object>> prometheusData = List.of(
new HashMap<>(Map.of("__value__", 100.0))
);
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("promql")).thenReturn(true);
when(mockExecutor.execute(anyString())).thenReturn(prometheusData);
dataSourceService.setExecutors(List.of(mockExecutor));
List<Map<String, Object>> result = dataSourceService.query("promql", "node_cpu_seconds_total > 50");
assertNotNull(result);
verify(mockExecutor).execute(anyString());
}
@Test
void query12() {
QueryExecutor mockExecutor = Mockito.mock(QueryExecutor.class);
when(mockExecutor.support("sql")).thenReturn(true);
dataSourceService.setExecutors(List.of(mockExecutor));
assertThrows(AlertExpressionException.class,
() -> dataSourceService.query("sql", "SELEC * FORM hertzbeat_logs"));
verify(mockExecutor, never()).execute(anyString());
}
}
@@ -146,7 +146,42 @@ public class HuaweiCloudExternAlertServiceTest {
verify(alarmCommonReduce, never()).reduceAndSendAlarm(any(SingleAlert.class));
}
@Test
void testSubscriptionUrlFail() {
HuaweiCloudExternAlert externAlert = new HuaweiCloudExternAlert();
externAlert.setSubscribeUrl("https://console.huaweicloud.com/smn/subscription/confirm?token=477a784601d744e4ab9ab83986502d31c4b938"
+ "0ec0b64392b134e517c3aa17eb7b3a12dc9f3b4ab495e61c4dee654b435d7223ea934345bf8ae8901cef912b1d&topic_urn=urn:smn:cn-north-4"
+ ":477a784601d744e4ab9ab83986502d31:CES_notification_group_bngJ2aMpX&region=cn-north-4");
externAlert.setSignature("ottf37C/2RdDgqimRQMIBU6i7XjUfPPMU760jJn71wwP3825YPoIT22uw2A9399rkm9Jrt1qUEFrDLuA5yHFLd5n/XoM4FghIgyFn7VIfgpuVM31a+co78s"
+ "YBiZ1egOCE/AwFm2oygRhfIceUj9Kw9vmc06el9TXY6RtE5tAEF6qEmICtTh45KwtCO/WRs3DY72dQi5hm0w7/tktS4WFZ1iP4LHt5eCwFvnH0u29Y96cJNI0fLUQxI5MkhgjK"
+ "77JkFK7UT6ZYJZhzgSp/B7OQGStOQx+3Duvx4T4CzccZQM3sca81Z0B0GFGWeVXuEHyCPLsayY/Iz+5Tco51elT8w==");
externAlert.setTopicUrn("urn:smn:cn-north-4:477a784601d744e4ab9ab83986502d31:CES_notification_group_bngJ2aMpX");
externAlert.setMessageId("242fac183d3a4936b5ead6c725a32ed0");
externAlert.setType("SubscriptionConfirmation");
externAlert.setMessage("You are invited to subscribe to topic: urn:smn:cn-north-4:477a784601d744e4ab9ab83986502d31:"
+ "CES_notification_group_bngJ2aMpX. To confirm this subscription, please visit the subscribe_url included in this message. The subscribe_url is valid only within 48 hours.");
externAlert.setSigningCertUrl("https://xxxx.myhuaweicloud.com/smn/SMN_cn-north-4_b98100ca131b4116ab8ee7ccedbaae99.pem");
externAlert.setTimestamp("2025-06-07T15:07:14Z");
externAlertService.addExternAlert(JsonUtil.toJson(externAlert));
verify(alarmCommonReduce, never()).reduceAndSendAlarm(any(SingleAlert.class));
}
@Test
void testCertUrlFailed() {
HuaweiCloudExternAlert externAlert = new HuaweiCloudExternAlert();
externAlert.setSignature("TImrLoeb0tV1JZJSPyA0rpC9mNqH3MmhwQ4tgpuHHa+JztfGVZFvkU//OthKKhzpDAoYiXOYG9DbzXCLb"
+ "vaGePIRITakoynYyYr9zZIpdx9jXhQNlgF8np1+t0JxNeoIq0DYWgH52tsodwqOm+OnmkcHwCRo/1rFv85KrKAaX2gy3sNwX"
+ "w1hKnAwAw0mJlxHHSf/N3+7j6GoxCNV7fN9K4CpJiLMGNvUa7zVmG0U9mPvt/7Lac155kPPQ9lYyeL7vVI0e4sfRbuQruz3E"
+ "0ZP40TKx0afoeR0/Bx/IoZzRP1La7pKlbEISvkcM7TqW/IOGQTkhVsQ32RFRxZWO2snw==");
externAlert.setSubject("DCS Redis “dcs-h4tv” ");
externAlert.setTopicUrn("urn:smn:cn-north-4:477a784601d744e4ab9ab83986502d31:CES_notification_group_bngJ2aMpX");
externAlert.setMessageId("d3672d737bb742cf8c2aa3f0fd72d4d1");
externAlert.setType("Notification");
externAlert.setMessage("{}");
externAlert.setSigningCertUrl("https://xxx.myhuaweicloud.com/failedUrl");
externAlert.setTimestamp("2025-06-07T15:12:09Z");
externAlertService.addExternAlert(JsonUtil.toJson(externAlert));
verify(alarmCommonReduce, never()).reduceAndSendAlarm(any(SingleAlert.class));
}
}
@@ -42,7 +42,7 @@ public class CryptoUtilsTest {
@Test
void testHmacSha256Hex() {
String signature = CryptoUtils.hmacSha256Hex("your-real-key", "your-real-data");;
assertEquals("41878ccd7ecd795a2dd7ec39be7f33fed4be3ec75f5307689e39dd6f41fdbaac", signature);
assertEquals("f09adf5f4bf94eddecf0f7c8f39a3a8dc7f95ccdc2faf2e5330bc5a78e4bba95", signature);
}
}
@@ -83,6 +83,10 @@
<groupId>com.dameng</groupId>
<artifactId>DmJdbcDriver18</artifactId>
</dependency>
<dependency>
<groupId>com.xugudb</groupId>
<artifactId>xugu-jdbc</artifactId>
</dependency>
<!-- postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
@@ -109,6 +113,12 @@
<artifactId>orai18n</artifactId>
<scope>provided</scope>
</dependency>
<!-- db2 -->
<dependency>
<groupId>com.ibm.db2</groupId>
<artifactId>jcc</artifactId>
<scope>provided</scope>
</dependency>
<!--redis-->
<dependency>
<groupId>io.lettuce</groupId>
@@ -152,7 +162,6 @@
</dependency>
<!--plc-->
<dependency>
<groupId>org.apache.plc4x</groupId>
@@ -172,6 +172,14 @@ public class JdbcCommonCollect extends AbstractCollect {
".*en\\s*([/\\\\]|\\\\n|/n|\\n)\\s*crypt.*",
".*ci\\s*([/\\\\]|\\\\n|/n|\\n)\\s*pher.*"
});
// DB2 bypass mode
PLATFORM_BYPASS_PATTERNS.put("db2", new String[]{
".*security\\s*([/\\\\]|\\\\n|/n|\\n)\\s*mechanism.*",
".*client\\s*([/\\\\]|\\\\n|/n|\\n)\\s*accounting.*",
".*ssl\\s*([/\\\\]|\\\\n|/n|\\n)\\s*connection.*",
".*use\\s*([/\\\\]|\\\\n|/n|\\n)\\s*client\\s*([/\\\\]|\\\\n|/n|\\n)\\s*information.*"
});
}
private final GlobalConnectionCache connectionCommonCache = GlobalConnectionCache.getInstance();
@@ -521,6 +529,8 @@ public class JdbcCommonCollect extends AbstractCollect {
case "mysql", "mariadb" -> "jdbc:mysql://" + host + ":" + port
+ "/" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase())
+ "?useUnicode=true&characterEncoding=utf-8&useSSL=false";
case "xugu" -> "jdbc:xugu://" + host + ":" + port
+ "/" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase());
case "postgresql" -> "jdbc:postgresql://" + host + ":" + port
+ "/" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase());
case "clickhouse" -> "jdbc:clickhouse://" + host + ":" + port
@@ -531,6 +541,8 @@ public class JdbcCommonCollect extends AbstractCollect {
case "oracle" -> "jdbc:oracle:thin:@" + host + ":" + port
+ "/" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase());
case "dm" -> "jdbc:dm://" + host + ":" + port;
case "db2" -> "jdbc:db2://" + host + ":" + port
+ "/" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase());
case "testcontainers" -> "jdbc:tc:" + host + ":" + port
+ ":///" + (jdbcProtocol.getDatabase() == null ? "" : jdbcProtocol.getDatabase()) + "?user=root&password=root";
default -> throw new IllegalArgumentException("Not support database platform: " + jdbcProtocol.getPlatform());
@@ -67,6 +67,8 @@ public class RedisCommonCollectImpl extends AbstractCollect {
private static final String CLUSTER = "3";
private static final String SINGLE = "1";
private static final String CLUSTER_INFO = "cluster";
private static final String UNIQUE_IDENTITY = "identity";
@@ -140,7 +142,7 @@ public class RedisCommonCollectImpl extends AbstractCollect {
* @return data
*/
private List<Map<String, String>> getClusterRedisInfo(Metrics metrics) throws GeneralSecurityException, IOException {
Map<String, StatefulRedisClusterConnection<String, String>> connectionMap = getConnectionList(metrics.getRedis());
Map<String, StatefulRedisConnection<String, String>> connectionMap = getConnectionList(metrics.getRedis());
List<Map<String, String>> list = new ArrayList<>(connectionMap.size());
connectionMap.forEach((identity, connection) ->{
String info = connection.sync().info(metrics.getName());
@@ -214,16 +216,23 @@ public class RedisCommonCollectImpl extends AbstractCollect {
* @param redisProtocol protocol
* @return connection map
*/
private Map<String, StatefulRedisClusterConnection<String, String>> getConnectionList(RedisProtocol redisProtocol) throws GeneralSecurityException, IOException {
private Map<String, StatefulRedisConnection<String, String>> getConnectionList(RedisProtocol redisProtocol) throws GeneralSecurityException, IOException {
// first connection
StatefulRedisClusterConnection<String, String> connection = getClusterConnection(redisProtocol);
Partitions partitions = connection.getPartitions();
Map<String, StatefulRedisClusterConnection<String, String>> clusterConnectionMap = new HashMap<>(partitions.size());
Map<String, StatefulRedisConnection<String, String>> clusterConnectionMap = new HashMap<>(partitions.size());
for (RedisClusterNode partition : partitions) {
RedisURI uri = partition.getUri();
redisProtocol.setHost(uri.getHost());
redisProtocol.setPort(String.valueOf(uri.getPort()));
StatefulRedisClusterConnection<String, String> clusterConnection = getClusterConnection(redisProtocol);
RedisProtocol singleRedisProtocol = RedisProtocol.builder()
.host(uri.getHost())
.port(String.valueOf(uri.getPort()))
.username(redisProtocol.getUsername())
.password(redisProtocol.getPassword())
.pattern(SINGLE)
.timeout(redisProtocol.getTimeout())
.sshTunnel(redisProtocol.getSshTunnel())
.build();
StatefulRedisConnection<String, String> clusterConnection = getSingleConnection(singleRedisProtocol);
clusterConnectionMap.put(doUri(uri.getHost(), uri.getPort()), clusterConnection);
}
return clusterConnectionMap;
@@ -19,10 +19,13 @@ package org.apache.hertzbeat.collector.collect.redis;
import static org.apache.hertzbeat.common.constants.CommonConstants.TYPE_STRING;
import static org.junit.jupiter.api.Assertions.assertEquals;
import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.api.sync.RedisCommands;
import io.lettuce.core.cluster.RedisClusterClient;
import io.lettuce.core.cluster.api.StatefulRedisClusterConnection;
import io.lettuce.core.cluster.api.sync.RedisAdvancedClusterCommands;
import io.lettuce.core.cluster.models.partitions.Partitions;
import io.lettuce.core.cluster.models.partitions.RedisClusterNode;
import io.lettuce.core.resource.ClientResources;
@@ -37,6 +40,7 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
@@ -51,13 +55,19 @@ public class RedisClusterCollectImplTest {
@Mock
private StatefulRedisClusterConnection<String, String> connection;
private StatefulRedisClusterConnection<String, String> clusterConnection;
@Mock
private RedisAdvancedClusterCommands<String, String> cmd;
private StatefulRedisConnection<String, String> singleConnection;
@Mock
private RedisClusterClient client;
private RedisCommands<String, String> cmd;
@Mock
private RedisClusterClient clusterClient;
@Mock
private RedisClient singleClient;
@BeforeEach
void setUp() {
@@ -65,8 +75,10 @@ public class RedisClusterCollectImplTest {
@AfterEach
void setDown() {
connection.close();
client.shutdown();
clusterConnection.close();
singleConnection.close();
clusterClient.shutdown();
singleClient.shutdown();
}
@Test
@@ -110,10 +122,14 @@ public class RedisClusterCollectImplTest {
metrics.setAliasFields(aliasField);
metrics.setFields(fields);
Mockito.mockStatic(RedisClusterClient.class).when(() -> RedisClusterClient.create(Mockito.any(ClientResources.class),
Mockito.any(RedisURI.class))).thenReturn(client);
Mockito.when(client.connect()).thenReturn(connection);
MockedStatic<RedisClusterClient> redisClusterClientMockedStatic = Mockito.mockStatic(RedisClusterClient.class);
redisClusterClientMockedStatic.when(() -> RedisClusterClient.create(Mockito.any(ClientResources.class),
Mockito.any(RedisURI.class))).thenReturn(clusterClient);
Mockito.when(clusterClient.connect()).thenReturn(clusterConnection);
MockedStatic<RedisClient> redisClientMockedStatic = Mockito.mockStatic(RedisClient.class);
redisClientMockedStatic.when(() -> RedisClient.create(Mockito.any(ClientResources.class),
Mockito.any(RedisURI.class))).thenReturn(singleClient);
Mockito.when(singleClient.connect()).thenReturn(singleConnection);
Partitions partitions = new Partitions();
RedisClusterNode node = new RedisClusterNode();
@@ -125,9 +141,9 @@ public class RedisClusterCollectImplTest {
node2.setUri(RedisURI.create("redis://" + uri2));
partitions.add(node2);
Mockito.when(connection.getPartitions()).thenReturn(partitions);
Mockito.when(clusterConnection.getPartitions()).thenReturn(partitions);
Mockito.when(connection.sync()).thenReturn(cmd);
Mockito.when(singleConnection.sync()).thenReturn(cmd);
Mockito.when(cmd.info(metrics.getName())).thenReturn(info);
Mockito.when(cmd.clusterInfo()).thenReturn(clusterInfo);
@@ -147,6 +163,8 @@ public class RedisClusterCollectImplTest {
assertEquals(row.getColumns(2), uri2);
}
}
redisClusterClientMockedStatic.close();
redisClientMockedStatic.close();
}
}
@@ -18,7 +18,7 @@
<configuration scan="true">
<springProperty scope="context" name="application_name" source="spring.application.name" defaultValue="collector"/>
<!-- Output logs to ConsoleAppender -->
<appender name="ConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<appender name="CONSOLE_RAW" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<!--<pattern>%d %p (%file:%line\)- %m%n</pattern>-->
<!-- Format output: %d for date, %thread for thread name, %-5level for level with 5 character width, %msg for log message, %n for new line -->
@@ -27,6 +27,13 @@
</encoder>
</appender>
<appender name="ConsoleAppender" class="ch.qos.logback.classic.AsyncAppender">
<discardingThreshold>0</discardingThreshold>
<queueSize>512</queueSize>
<includeCallerData>true</includeCallerData>
<appender-ref ref="CONSOLE_RAW"/>
</appender>
<appender name="SystemOutFileAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!-- Rolling policy for loggers, by date and by size -->
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
@@ -94,6 +101,7 @@
<!-- Production environment configuration -->
<springProfile name="prod">
<root level="INFO">
<appender-ref ref="ConsoleAppender"/>
<appender-ref ref="SystemOutFileAppender"/>
<appender-ref ref="ErrOutFileAppender"/>
</root>
@@ -101,6 +109,7 @@
<springProfile name="cluster">
<root level="INFO">
<appender-ref ref="ConsoleAppender"/>
<appender-ref ref="SystemOutFileAppender"/>
<appender-ref ref="ErrOutFileAppender"/>
</root>
@@ -35,9 +35,9 @@ import org.apache.hertzbeat.remoting.netty.NettyRemotingProcessor;
*/
@Slf4j
public class GoOnlineProcessor implements NettyRemotingProcessor {
private TimerDispatch timerDispatch;
@Override
public ClusterMsg.Message handle(ChannelHandlerContext ctx, ClusterMsg.Message message) {
if (this.timerDispatch == null) {
@@ -53,7 +53,9 @@ public class GoOnlineProcessor implements NettyRemotingProcessor {
AesUtil.setDefaultSecretKey(serverInfo.getAesSecret());
}
}
timerDispatch.goOnline();
if (ClusterMsg.Direction.REQUEST.equals(message.getDirection())) {
timerDispatch.goOnline();
}
log.info("receive online message and handle success");
return ClusterMsg.Message.newBuilder()
.setIdentity(message.getIdentity())
@@ -25,6 +25,8 @@ import org.apache.hertzbeat.common.queue.CommonDataQueue;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Configuration;
import java.util.List;
/**
* for collector instance
* send collect response data by netty
@@ -108,4 +110,24 @@ public class NettyDataQueue implements CommonDataQueue {
public LogEntry pollLogEntryToStorage() throws InterruptedException {
return null;
}
@Override
public void sendLogEntryToAlertBatch(List<LogEntry> logEntries) {
}
@Override
public List<LogEntry> pollLogEntryToAlertBatch(int maxBatchSize) throws InterruptedException {
return List.of();
}
@Override
public void sendLogEntryToStorageBatch(List<LogEntry> logEntries) {
}
@Override
public List<LogEntry> pollLogEntryToStorageBatch(int maxBatchSize) throws InterruptedException {
return List.of();
}
}
@@ -0,0 +1,105 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.collector.dispatch.entrance.processor;
import com.google.common.collect.Lists;
import com.google.protobuf.ByteString;
import io.netty.channel.ChannelHandlerContext;
import org.apache.hertzbeat.collector.timer.TimerDispatch;
import org.apache.hertzbeat.collector.timer.TimerDispatcher;
import org.apache.hertzbeat.common.entity.job.Job;
import org.apache.hertzbeat.common.entity.job.Metrics;
import org.apache.hertzbeat.common.entity.message.ClusterMsg;
import org.apache.hertzbeat.common.support.SpringContextHolder;
import org.apache.hertzbeat.common.util.JsonUtil;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mock;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
import java.lang.reflect.Field;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* Test for GoOnlineProcessor
*/
class GoOnlineProcessorTest {
private GoOnlineProcessor goOnlineProcessor;
private TimerDispatcher timerDispatcher;
@Mock
private ChannelHandlerContext channelHandlerContext;
private MockedStatic<SpringContextHolder> springContextHolderMockedStatic;
@BeforeEach
void setUp() {
MockitoAnnotations.openMocks(this);
goOnlineProcessor = new GoOnlineProcessor();
timerDispatcher = new TimerDispatcher();
springContextHolderMockedStatic = Mockito.mockStatic(SpringContextHolder.class);
springContextHolderMockedStatic.when(() -> SpringContextHolder.getBean(TimerDispatch.class)).thenReturn(timerDispatcher);
}
@AfterEach
void tearDown() throws Exception {
springContextHolderMockedStatic.close();
timerDispatcher.destroy();
}
@Test
void verifyTaskMapPreservation() throws Exception {
Job job = Job.builder()
.app("test")
.id(12345L)
.metrics(Lists.newArrayList(Metrics.builder().interval(100L).build()))
.configmap(Lists.newArrayList())
.isCyclic(true)
.build();
timerDispatcher.addJob(job, null);
Field cyclicTaskMapField = TimerDispatcher.class.getDeclaredField("currentCyclicTaskMap");
cyclicTaskMapField.setAccessible(true);
Map<?, ?> currentCyclicTaskMap = (Map<?, ?>) cyclicTaskMapField.get(timerDispatcher);
assertEquals(1, currentCyclicTaskMap.size(), "Task map should have 1 job initially");
ClusterMsg.Message responseMsg = ClusterMsg.Message.newBuilder()
.setType(ClusterMsg.MessageType.GO_ONLINE)
.setDirection(ClusterMsg.Direction.RESPONSE)
.setMsg(ByteString.copyFromUtf8(JsonUtil.toJson(job)))
.setIdentity("test-identity")
.build();
goOnlineProcessor.handle(channelHandlerContext, responseMsg);
assertEquals(1, currentCyclicTaskMap.size(), "Task map should still have 1 job after receiving RESPONSE");
ClusterMsg.Message requestMsg = ClusterMsg.Message.newBuilder()
.setType(ClusterMsg.MessageType.GO_ONLINE)
.setDirection(ClusterMsg.Direction.REQUEST)
.setMsg(ByteString.copyFromUtf8(JsonUtil.toJson(job)))
.setIdentity("test-identity")
.build();
goOnlineProcessor.handle(channelHandlerContext, requestMsg);
assertEquals(0, currentCyclicTaskMap.size(), "Task map should be empty after receiving REQUEST");
}
}
+5 -1
View File
@@ -171,7 +171,7 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-netty</artifactId>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
@@ -183,6 +183,10 @@
<version>${javaparser.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.github.jsqlparser</groupId>
<artifactId>jsqlparser</artifactId>
</dependency>
</dependencies>
</project>
@@ -125,6 +125,11 @@ public class CommonProperties {
*/
private String logEntryToStorageQueueName;
/**
* Timeout for blocking wait in seconds (defaults to 1 second if not configured)
*/
private Long waitTimeout;
}
/**
@@ -97,11 +97,6 @@ public interface CommonConstants {
*/
String LABEL_ALERT_NAME = "alertname";
/**
* label key: instance host
*/
String LABEL_INSTANCE_HOST = "instancehost";
/**
* label key: instance name
*/
@@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.constants;
/**
* HertzBeat Reserved Keywords Enum
*/
public enum HertzBeatKeywordsEnum {
INSTANCE("instance", "metric_instance");
private final String keyword;
private final String alias;
HertzBeatKeywordsEnum(String keyword, String alias) {
this.keyword = keyword;
this.alias = alias;
}
public String getKeyword() {
return keyword;
}
public String getAlias() {
return alias;
}
}
@@ -38,5 +38,5 @@ public interface MetricDataConstants {
String CODE = "code";
String MSG = "msg";
String INSTANCE_NAME = "instancename";
String INSTANCE_HOST = "instancehost";
String INSTANCE = "instance";
}
@@ -68,12 +68,13 @@ public class NoticeReceiver {
private String name;
@Schema(title = "Notification information method: 0-SMS 1-Email 2-webhook 3-WeChat Official Account 4-Enterprise WeChat Robot "
+ "5-DingTalk Robot 6-FeiShu Robot 7-Telegram Bot 8-SlackWebHook 9-Discord Bot 10-Enterprise WeChat app message",
+ "5-DingTalk Robot 6-FeiShu Robot 7-Telegram Bot 8-SlackWebHook 9-Discord Bot 10-Enterprise WeChat app message "
+ "11-Slack 12-Discord 13-Gotify 14-FeiShu app message",
description = "Notification information method: "
+ "0-SMS 1-Email 2-webhook 3-WeChat Official Account "
+ "4-Enterprise WeChat Robot 5-DingTalk Robot 6-FeiShu Robot "
+ "7-Telegram Bot 8-SlackWebHook 9-Discord Bot 10-Enterprise "
+ "WeChat app message",
+ "WeChat app message 11-Slack 12-Discord 13-Gotify 14-FeiShu app message",
accessMode = READ_WRITE)
@Min(0)
@NotNull(message = "type can not null")
@@ -98,6 +99,20 @@ public class NoticeReceiver {
@Column(length = 1000)
private String hookUrl;
@Schema(title = "Auth Type: WebHook Authorization Type",
description = "Auth Type: WebHook Authorization Type, one of the 'None', 'Basic' and 'Bearer'",
example = "None", accessMode = READ_ONLY)
@Size(max = 300)
@Column(length = 300)
private String hookAuthType;
@Schema(title = "Auth Token: WebHook Authorization Token",
description = "Auth Token: WebHook Authorization Token",
example = "YWxpY2U6c3VwZXJtYW4", accessMode = READ_WRITE)
@Size(max = 300)
@Column(length = 300)
private String hookAuthToken;
@Schema(title = "openId : The notification method is valid for WeChat official account, enterprise WeChat robot or FlyBook robot",
description = "openId : The notification method is valid for WeChat official account, enterprise WeChat robot or FlyBook robot",
example = "343432", accessMode = READ_WRITE)
@@ -105,6 +120,12 @@ public class NoticeReceiver {
@Column(length = 300)
private String wechatId;
@Schema(title = "FeiShu app id : The notification method is valid for FeiShu app message",
description = "FeiShu app id : The notification method is valid for FeiShu app message",
example = "34823984635647", accessMode = READ_WRITE)
@Size(max = 255)
private String appId;
@Schema(title = "Access token : The notification method is valid for DingTalk robot",
description = "Access token : The notification method is valid for DingTalk robot",
example = "34823984635647", accessMode = READ_WRITE)
@@ -127,11 +148,21 @@ public class NoticeReceiver {
example = "779294123", accessMode = READ_WRITE)
private String tgMessageThreadId;
@Schema(title = "FeiShu app message receiveType: 0-user 1-chat 2-party 3-all",
description = "FeiShu app message receiveType: 0-user 1-chat 2-party 3-all",
example = "1", accessMode = READ_WRITE)
private Byte larkReceiveType;
@Schema(title = "DingTalk,FeiShu,WeWork user id: The notification method is valid for DingTalk,FeiShu,WeWork Bot",
description = "DingTalk,FeiShu,WeWork user id: The notification method is valid for DingTalk,FeiShu,WeWork Bot",
example = "779294123", accessMode = READ_WRITE)
private String userId;
@Schema(title = "FeiShu app message chatId: The notification method is valid for FeiShu app message",
description = "FeiShu app message chatId: The notification method is valid for FeiShu app message",
example = "779294123", accessMode = READ_WRITE)
private String chatId;
@Schema(title = "URL address: The notification method is valid for Slack",
description = "URL address: The notification method is valid for Slack",
example = "https://hooks.slack.com/services/XXXX/XXXX/XXXX", accessMode = READ_WRITE)
@@ -18,13 +18,14 @@
package org.apache.hertzbeat.common.entity.dto;
import io.swagger.v3.oas.annotations.media.Schema;
import java.util.List;
import java.util.Map;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
import java.util.Map;
/**
* Metric History Range Query Data
*/
@@ -35,8 +36,8 @@ import lombok.NoArgsConstructor;
@Schema(description = "Metric History Range Query Data")
public class MetricsHistoryData {
@Schema(title = "Monitoring Task ID")
private Long id;
@Schema(title = "Monitor Instance (e.g., ip:port or domain)")
private String instance;
@Schema(title = "Monitoring Type")
private String app;
@@ -52,7 +52,7 @@ import org.springframework.data.jpa.domain.support.AuditingEntityListener;
@Entity
@Table(name = "hzb_monitor", indexes = {
@Index(name = "monitor_query_index", columnList = "app"),
@Index(name = "monitor_query_index", columnList = "host"),
@Index(name = "monitor_query_index", columnList = "instance"),
@Index(name = "monitor_query_index", columnList = "name")
})
@Data
@@ -82,10 +82,10 @@ public class Monitor {
@Size(max = 100)
private String scrape;
@Schema(title = "peer host: ipv4, ipv6, domain name", example = "192.167.25.11", accessMode = READ_WRITE)
@Schema(title = "the monitor target: ip/domain+port or ip/domain", example = "192.167.25.11:8081", accessMode = READ_WRITE)
@Size(max = 100)
@HostValid
private String host;
private String instance;
@Schema(title = "Monitoring of the acquisition interval time in seconds", example = "600", accessMode = READ_WRITE)
@Min(10)
@@ -225,9 +225,9 @@ public final class CollectRep {
return metadata.getOrDefault(MetricDataConstants.INSTANCE_NAME, null);
}
public String getInstanceHost() {
public String getInstance() {
Map<String, String> metadata = getMetadata();
return metadata.getOrDefault(MetricDataConstants.INSTANCE_HOST, null);
return metadata.getOrDefault(MetricDataConstants.INSTANCE, null);
}
public Map<String, String> getLabels() {
@@ -501,8 +501,8 @@ public final class CollectRep {
return metadata.getOrDefault(MetricDataConstants.INSTANCE_NAME, null);
}
public String getInstanceHost() {
return metadata.getOrDefault(MetricDataConstants.INSTANCE_HOST, null);
public String getInstance() {
return metadata.getOrDefault(MetricDataConstants.INSTANCE, null);
}
public Map<String, String> getLabels() {
@@ -37,7 +37,7 @@ import lombok.NoArgsConstructor;
*/
@Entity
@Table(name = "hzb_history", indexes = {
@Index(name = "history_query_index", columnList = "monitorId"),
@Index(name = "history_query_index", columnList = "instance"),
@Index(name = "history_query_index", columnList = "app"),
@Index(name = "history_query_index", columnList = "metrics"),
@Index(name = "history_query_index", columnList = "metric")
@@ -54,8 +54,8 @@ public class History {
@Schema(description = "Metric data history entity primary key index ID", example = "87584674384", accessMode = READ_ONLY)
private Long id;
@Schema(title = "Monitoring Id", example = "87432674336", accessMode = READ_WRITE)
private Long monitorId;
@Schema(title = "Monitoring instance", example = "127.0.0.1:8080", accessMode = READ_WRITE)
private String instance;
@Schema(title = "Monitoring Type mysql oracle db2")
private String app;
@@ -67,7 +67,7 @@ public class History {
private String metric;
@Column(length = 5000)
private String instance;
private String metricLabels;
@Schema(title = "Metric Type 0: Number 1String")
private Byte metricType;
@@ -17,6 +17,7 @@
package org.apache.hertzbeat.common.queue;
import java.util.List;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.apache.hertzbeat.common.entity.message.CollectRep;
@@ -91,4 +92,32 @@ public interface CommonDataQueue {
* @throws InterruptedException when poll timeout
*/
LogEntry pollLogEntryToStorage() throws InterruptedException;
/**
* send batch log entries to alert queue
* @param logEntries list of log entry data
*/
void sendLogEntryToAlertBatch(List<LogEntry> logEntries);
/**
* poll batch log entries from alert queue
* @param maxBatchSize maximum number of entries to poll
* @return list of log entry data
* @throws InterruptedException when poll timeout
*/
List<LogEntry> pollLogEntryToAlertBatch(int maxBatchSize) throws InterruptedException;
/**
* send batch log entries to storage queue
* @param logEntries list of log entry data
*/
void sendLogEntryToStorageBatch(List<LogEntry> logEntries);
/**
* poll batch log entries from storage queue
* @param maxBatchSize maximum number of entries to poll
* @return list of log entry data
* @throws InterruptedException when poll timeout
*/
List<LogEntry> pollLogEntryToStorageBatch(int maxBatchSize) throws InterruptedException;
}
@@ -17,9 +17,12 @@
package org.apache.hertzbeat.common.queue.impl;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.constants.DataQueueConstants;
import org.apache.hertzbeat.common.entity.log.LogEntry;
@@ -117,6 +120,48 @@ public class InMemoryCommonDataQueue implements CommonDataQueue, DisposableBean
return logEntryToStorageQueue.take();
}
@Override
public void sendLogEntryToAlertBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
for (LogEntry logEntry : logEntries) {
logEntryQueue.offer(logEntry);
}
}
@Override
public List<LogEntry> pollLogEntryToAlertBatch(int maxBatchSize) throws InterruptedException {
List<LogEntry> batch = new ArrayList<>(maxBatchSize);
LogEntry first = logEntryQueue.poll(1, TimeUnit.SECONDS);
if (first != null) {
batch.add(first);
logEntryQueue.drainTo(batch, maxBatchSize - 1);
}
return batch;
}
@Override
public void sendLogEntryToStorageBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
for (LogEntry logEntry : logEntries) {
logEntryToStorageQueue.offer(logEntry);
}
}
@Override
public List<LogEntry> pollLogEntryToStorageBatch(int maxBatchSize) throws InterruptedException {
List<LogEntry> batch = new ArrayList<>(maxBatchSize);
LogEntry first = logEntryToStorageQueue.poll(1, TimeUnit.SECONDS);
if (first != null) {
batch.add(first);
logEntryToStorageQueue.drainTo(batch, maxBatchSize - 1);
}
return batch;
}
@Override
public void destroy() {
metricsDataToAlertQueue.clear();
@@ -18,8 +18,10 @@
package org.apache.hertzbeat.common.queue.impl;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.locks.ReentrantLock;
@@ -33,6 +35,7 @@ import org.apache.hertzbeat.common.serialize.KafkaLogEntryDeserializer;
import org.apache.hertzbeat.common.serialize.KafkaLogEntrySerializer;
import org.apache.hertzbeat.common.serialize.KafkaMetricsDataDeserializer;
import org.apache.hertzbeat.common.serialize.KafkaMetricsDataSerializer;
import org.apache.hertzbeat.common.support.exception.CommonDataQueueUnknownException;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
@@ -117,7 +120,7 @@ public class KafkaCommonDataQueue implements CommonDataQueue, DisposableBean {
Map<String, Object> metricsToStorageConsumerConfig = new HashMap<>(consumerConfig);
metricsToStorageConsumerConfig.put("group.id", "metrics-persistent-consumer");
metricsDataToStorageConsumer = new KafkaConsumer<>(metricsToStorageConsumerConfig, new LongDeserializer(), new KafkaMetricsDataDeserializer());
metricsDataToStorageConsumer.subscribe(Collections.singletonList(kafka.getMetricsDataTopic()));
metricsDataToStorageConsumer.subscribe(Collections.singletonList(kafka.getMetricsDataToStorageTopic()));
Map<String, Object> serviceDiscoveryDataConsumerConfig = new HashMap<>(consumerConfig);
serviceDiscoveryDataConsumerConfig.put("group.id", "service-discovery-data-consumer");
@@ -176,6 +179,7 @@ public class KafkaCommonDataQueue implements CommonDataQueue, DisposableBean {
dataConsumer.commitAsync();
} catch (Exception e) {
log.error(e.getMessage());
throw new CommonDataQueueUnknownException(e.getMessage(), e);
} finally {
lock.unlock();
}
@@ -259,6 +263,92 @@ public class KafkaCommonDataQueue implements CommonDataQueue, DisposableBean {
return genericPollDataFunction(logEntryToStorageQueue, logEntryToStorageConsumer, logEntryToStorageLock);
}
@Override
public void sendLogEntryToAlertBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
if (logEntryProducer != null) {
try {
for (LogEntry logEntry : logEntries) {
ProducerRecord<Long, LogEntry> record = new ProducerRecord<>(kafka.getLogEntryDataTopic(), logEntry);
logEntryProducer.send(record);
}
} catch (Exception e) {
log.error("Failed to send LogEntry batch to Kafka: {}", e.getMessage());
for (LogEntry logEntry : logEntries) {
logEntryQueue.offer(logEntry);
}
}
} else {
log.warn("logEntryProducer is not enabled, using memory queue");
for (LogEntry logEntry : logEntries) {
logEntryQueue.offer(logEntry);
}
}
}
@Override
public List<LogEntry> pollLogEntryToAlertBatch(int maxBatchSize) throws InterruptedException {
return genericBatchPollDataFunction(logEntryQueue, logEntryConsumer, logEntryLock, maxBatchSize);
}
@Override
public void sendLogEntryToStorageBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
if (logEntryProducer != null) {
try {
for (LogEntry logEntry : logEntries) {
ProducerRecord<Long, LogEntry> record = new ProducerRecord<>(kafka.getLogEntryDataToStorageTopic(), logEntry);
logEntryProducer.send(record);
}
} catch (Exception e) {
log.error("Failed to send LogEntry batch to storage via Kafka: {}", e.getMessage());
for (LogEntry logEntry : logEntries) {
logEntryToStorageQueue.offer(logEntry);
}
}
} else {
log.warn("logEntryProducer is not enabled, using memory queue for storage");
for (LogEntry logEntry : logEntries) {
logEntryToStorageQueue.offer(logEntry);
}
}
}
@Override
public List<LogEntry> pollLogEntryToStorageBatch(int maxBatchSize) throws InterruptedException {
return genericBatchPollDataFunction(logEntryToStorageQueue, logEntryToStorageConsumer, logEntryToStorageLock, maxBatchSize);
}
public <T> List<T> genericBatchPollDataFunction(LinkedBlockingQueue<T> dataQueue, KafkaConsumer<Long, T> dataConsumer,
ReentrantLock lock, int maxBatchSize) throws InterruptedException {
List<T> batch = new ArrayList<>(maxBatchSize);
lock.lockInterruptibly();
try {
dataQueue.drainTo(batch, maxBatchSize);
if (batch.size() >= maxBatchSize) {
return batch;
}
ConsumerRecords<Long, T> records = dataConsumer.poll(Duration.ofSeconds(1));
for (ConsumerRecord<Long, T> record : records) {
if (batch.size() < maxBatchSize) {
batch.add(record.value());
} else {
dataQueue.offer(record.value());
}
}
dataConsumer.commitAsync();
} catch (Exception e) {
log.error(e.getMessage());
} finally {
lock.unlock();
}
return batch;
}
@Override
public void destroy() throws Exception {
if (metricsDataProducer != null) {
@@ -17,6 +17,9 @@
package org.apache.hertzbeat.common.queue.impl;
import java.util.ArrayList;
import java.util.List;
import io.lettuce.core.KeyValue;
import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
@@ -29,10 +32,13 @@ import org.apache.hertzbeat.common.entity.message.CollectRep;
import org.apache.hertzbeat.common.queue.CommonDataQueue;
import org.apache.hertzbeat.common.serialize.RedisLogEntryCodec;
import org.apache.hertzbeat.common.serialize.RedisMetricsDataCodec;
import org.apache.hertzbeat.common.support.exception.CommonDataQueueUnknownException;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Configuration;
import java.util.Objects;
/**
* common data queue implement redis.
*/
@@ -56,6 +62,7 @@ public class RedisCommonDataQueue implements CommonDataQueue, DisposableBean {
private final String logEntryQueueName;
private final String logEntryToStorageQueueName;
private final CommonProperties.RedisProperties redisProperties;
private final Long waitTimeout;
public RedisCommonDataQueue(CommonProperties properties) {
@@ -83,36 +90,23 @@ public class RedisCommonDataQueue implements CommonDataQueue, DisposableBean {
this.metricsDataQueueNameToAlerter = redisProperties.getMetricsDataQueueNameToAlerter();
this.logEntryQueueName = redisProperties.getLogEntryQueueName();
this.logEntryToStorageQueueName = redisProperties.getLogEntryToStorageQueueName();
this.waitTimeout = Objects.requireNonNullElse(redisProperties.getWaitTimeout(), 1L);
}
@Override
public CollectRep.MetricsData pollMetricsDataToAlerter() {
try {
return syncCommands.rpop(metricsDataQueueNameToAlerter);
} catch (Exception e) {
log.error(e.getMessage());
return null;
}
public CollectRep.MetricsData pollMetricsDataToAlerter() throws InterruptedException {
return genericBlockingPollFunction(metricsDataQueueNameToAlerter, syncCommands);
}
@Override
public CollectRep.MetricsData pollMetricsDataToStorage() throws InterruptedException {
try {
return syncCommands.rpop(metricsDataQueueNameToStorage);
} catch (Exception e) {
log.error(e.getMessage());
return null;
}
return genericBlockingPollFunction(metricsDataQueueNameToStorage, syncCommands);
}
@Override
public CollectRep.MetricsData pollServiceDiscoveryData() throws InterruptedException {
try {
return syncCommands.rpop(metricsDataQueueNameForServiceDiscovery);
} catch (Exception e) {
log.error(e.getMessage());
return null;
}
return genericBlockingPollFunction(metricsDataQueueNameForServiceDiscovery, syncCommands);
}
@Override
@@ -153,12 +147,7 @@ public class RedisCommonDataQueue implements CommonDataQueue, DisposableBean {
@Override
public LogEntry pollLogEntry() throws InterruptedException {
try {
return logEntrySyncCommands.rpop(logEntryQueueName);
} catch (Exception e) {
log.error("Failed to poll LogEntry from Redis: {}", e.getMessage());
throw new InterruptedException("Failed to poll LogEntry from Redis");
}
return genericBlockingPollFunction(logEntryQueueName, logEntrySyncCommands);
}
@Override
@@ -172,12 +161,43 @@ public class RedisCommonDataQueue implements CommonDataQueue, DisposableBean {
@Override
public LogEntry pollLogEntryToStorage() throws InterruptedException {
try {
return logEntrySyncCommands.rpop(logEntryToStorageQueueName);
} catch (Exception e) {
log.error("Failed to poll LogEntry from storage via Redis: {}", e.getMessage());
throw new InterruptedException("Failed to poll LogEntry from storage via Redis");
return genericBlockingPollFunction(logEntryToStorageQueueName, logEntrySyncCommands);
}
@Override
@SuppressWarnings("unchecked")
public void sendLogEntryToAlertBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
try {
logEntrySyncCommands.lpush(logEntryQueueName, logEntries.toArray(new LogEntry[0]));
} catch (Exception e) {
log.error("Failed to send LogEntry batch to Redis: {}", e.getMessage());
}
}
@Override
public List<LogEntry> pollLogEntryToAlertBatch(int maxBatchSize) throws InterruptedException {
return genericBatchPollFunction(logEntryQueueName, logEntrySyncCommands, maxBatchSize);
}
@Override
@SuppressWarnings("unchecked")
public void sendLogEntryToStorageBatch(List<LogEntry> logEntries) {
if (logEntries == null || logEntries.isEmpty()) {
return;
}
try {
logEntrySyncCommands.lpush(logEntryToStorageQueueName, logEntries.toArray(new LogEntry[0]));
} catch (Exception e) {
log.error("Failed to send LogEntry batch to storage via Redis: {}", e.getMessage());
}
}
@Override
public List<LogEntry> pollLogEntryToStorageBatch(int maxBatchSize) throws InterruptedException {
return genericBatchPollFunction(logEntryToStorageQueueName, logEntrySyncCommands, maxBatchSize);
}
@Override
@@ -187,4 +207,36 @@ public class RedisCommonDataQueue implements CommonDataQueue, DisposableBean {
redisClient.shutdown();
}
private <T> T genericBlockingPollFunction(String key, RedisCommands<String, T> commands) throws InterruptedException {
try {
// Use BRPOP for blocking pop with the configured timeout.
// If data arrives, it returns immediately; if it times out, it returns null.
KeyValue<String, T> keyData = commands.brpop(waitTimeout, key);
if (keyData != null) {
return keyData.getValue();
} else {
// Returns null on timeout
return null;
}
} catch (Exception e) {
log.error("Redis BRPOP failed: {}", e.getMessage());
throw new CommonDataQueueUnknownException(e.getMessage(), e);
}
}
private List<LogEntry> genericBatchPollFunction(String key, RedisCommands<String, LogEntry> commands, int maxBatchSize) {
List<LogEntry> batch = new ArrayList<>(maxBatchSize);
try {
List<LogEntry> elements = commands.rpop(key, maxBatchSize);
if (elements != null) {
batch.addAll(elements);
}
} catch (Exception e) {
log.error("Redis batch poll failed: {}", e.getMessage());
}
return batch;
}
}
@@ -0,0 +1,39 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.support.exception;
/**
* Common data queue unknown exception
*/
public class CommonDataQueueUnknownException extends RuntimeException {
public CommonDataQueueUnknownException() {
}
public CommonDataQueueUnknownException(String message) {
super(message);
}
public CommonDataQueueUnknownException(String message, Throwable cause) {
super(message, cause);
}
public CommonDataQueueUnknownException(Throwable cause) {
super(cause);
}
}
@@ -44,7 +44,21 @@ public class HostParamValidator implements ConstraintValidator<HostValid, String
value = value.replaceFirst(PATTERN_HTTPS, BLANK);
}
return IpDomainUtil.validateIpDomain(value);
String hostPart = value;
if (value.contains(":")) {
// if contains multiple ":", it may be IPv6 with port
if (value.lastIndexOf(":") > value.indexOf(":") && value.contains("[")) {
int portIndex = value.lastIndexOf(":");
hostPart = value.substring(0, portIndex);
} else if (value.split(":").length == 2) {
// it is IPv4 or domain with port
String[] parts = value.split(":");
hostPart = parts[0];
}
}
return IpDomainUtil.validateIpDomain(hostPart);
}
}
@@ -0,0 +1,32 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.support.valid;
/**
* SQL security validation exception
*/
public class SqlSecurityException extends RuntimeException {
public SqlSecurityException(String message) {
super(message);
}
public SqlSecurityException(String message, Throwable cause) {
super(message, cause);
}
}
@@ -0,0 +1,150 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.support.valid;
import lombok.extern.slf4j.Slf4j;
import net.sf.jsqlparser.JSQLParserException;
import net.sf.jsqlparser.parser.CCJSqlParserUtil;
import net.sf.jsqlparser.statement.Statement;
import net.sf.jsqlparser.statement.select.LateralSubSelect;
import net.sf.jsqlparser.statement.select.ParenthesedSelect;
import net.sf.jsqlparser.statement.select.Select;
import net.sf.jsqlparser.statement.select.SetOperationList;
import net.sf.jsqlparser.statement.select.WithItem;
import net.sf.jsqlparser.util.TablesNamesFinder;
import org.springframework.util.CollectionUtils;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
/**
* SQL Security Validator using JSqlParser 5.1+.
* Security Policy:
* 1. Only SELECT statements are allowed.
* 2. All referenced tables must be in the whitelist.
* 3. Subqueries, UNION, CTE, LATERAL are blocked.
*/
@Slf4j
public class SqlSecurityValidator {
private final Set<String> allowedTables;
public SqlSecurityValidator(Collection<String> allowedTables) {
if (CollectionUtils.isEmpty(allowedTables)) {
this.allowedTables = new HashSet<>();
} else {
this.allowedTables = allowedTables.stream()
.map(this::normalizeIdentifier)
.collect(Collectors.toSet());
}
}
public void validate(String sql) throws SqlSecurityException {
if (sql == null || sql.trim().isEmpty()) {
throw new SqlSecurityException("SQL statement cannot be empty");
}
Statement statement;
try {
statement = CCJSqlParserUtil.parse(sql);
} catch (JSQLParserException e) {
log.warn("Failed to parse SQL: {}", sql, e);
throw new SqlSecurityException("Invalid SQL syntax: " + e.getMessage(), e);
}
if (!(statement instanceof Select select)) {
throw new SqlSecurityException("Only SELECT statements are allowed.");
}
// Check for CTE at top level
if (select.getWithItemsList() != null && !select.getWithItemsList().isEmpty()) {
throw new SqlSecurityException("CTE (WITH clause) is not allowed");
}
// Use custom TablesNamesFinder that throws on dangerous structures
SecurityTablesNamesFinder finder = new SecurityTablesNamesFinder();
List<String> tables;
try {
tables = finder.getTableList(statement);
} catch (SecurityViolationException e) {
throw new SqlSecurityException(e.getMessage());
}
validateTables(tables);
}
private void validateTables(List<String> tables) throws SqlSecurityException {
if (CollectionUtils.isEmpty(tables)) {
return;
}
if (allowedTables.isEmpty()) {
throw new SqlSecurityException("No access allowed: whitelist is empty.");
}
for (String table : tables) {
String normalizedTable = normalizeIdentifier(table);
if (!allowedTables.contains(normalizedTable)) {
throw new SqlSecurityException("Access to table '" + table + "' is not allowed. "
+ "Allowed tables: " + allowedTables);
}
}
}
private String normalizeIdentifier(String identifier) {
if (identifier == null) {
return "";
}
return identifier.replace("\"", "").replace("`", "").replace("'", "").toLowerCase();
}
private static class SecurityViolationException extends RuntimeException {
SecurityViolationException(String message) {
super(message);
}
}
/**
* Custom TablesNamesFinder that throws exceptions on dangerous SQL structures.
* Extends TablesNamesFinder with proper generic type to avoid raw type warnings.
*/
private static class SecurityTablesNamesFinder extends TablesNamesFinder<Void> {
@Override
public Void visit(ParenthesedSelect parenthesedSelect, Object context) {
throw new SecurityViolationException("Subqueries are not allowed");
}
@Override
public Void visit(SetOperationList setOpList, Object context) {
throw new SecurityViolationException("UNION and set operations are not allowed");
}
@Override
public Void visit(LateralSubSelect lateralSubSelect, Object context) {
throw new SecurityViolationException("LATERAL subqueries are not allowed");
}
@Override
public Void visit(WithItem withItem, Object context) {
throw new SecurityViolationException("CTE (WITH clause) is not allowed");
}
}
}
@@ -17,14 +17,6 @@
package org.apache.hertzbeat.common.util;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.nio.channels.Channels;
import java.util.ArrayList;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
@@ -32,12 +24,21 @@ import org.apache.arrow.vector.ipc.ArrowStreamReader;
import org.apache.arrow.vector.ipc.ArrowStreamWriter;
import org.apache.hertzbeat.common.entity.message.CollectRep;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.util.ArrayList;
import java.util.List;
/**
* Arrow data serialization and deserialization utility class
*/
@Slf4j
public final class ArrowUtil {
private ArrowUtil() {
}
@@ -53,18 +54,22 @@ public final class ArrowUtil {
public static byte[] serializeMultipleRoots(List<VectorSchemaRoot> roots) {
try (ByteArrayOutputStream out = new ByteArrayOutputStream();
DataOutputStream dataOut = new DataOutputStream(out)) {
dataOut.writeInt(roots.size());
for (VectorSchemaRoot root : roots) {
ArrowStreamWriter writer = new ArrowStreamWriter(
root,
null,
Channels.newChannel(out));
writer.start();
writer.writeBatch();
writer.end();
writer.close();
root.close();
// Use a temporary stream to obtain the precise byte length of a single root,
// write the length, and resolve the pre-read issue.
try (ByteArrayOutputStream tempOut = new ByteArrayOutputStream()) {
try (ArrowStreamWriter writer = new ArrowStreamWriter(root, null, Channels.newChannel(tempOut))) {
writer.start();
writer.writeBatch();
writer.end();
}
int size = tempOut.size();
dataOut.writeInt(size);
dataOut.flush();
tempOut.writeTo(out);
}
}
return out.toByteArray();
} catch (IOException e) {
@@ -85,15 +90,20 @@ public final class ArrowUtil {
*/
public static List<VectorSchemaRoot> deserializeMultipleRoots(byte[] data) {
List<VectorSchemaRoot> roots = new ArrayList<>();
try (ByteArrayInputStream in = new ByteArrayInputStream(data);
DataInputStream dataIn = new DataInputStream(in)) {
int rootCount = dataIn.readInt();
ByteBuffer buffer = ByteBuffer.wrap(data);
try {
int rootCount = buffer.getInt();
RootAllocator allocator = new RootAllocator();
for (int i = 0; i < rootCount; i++) {
int length = buffer.getInt();
// Split the InputStream to prevent the Reader from reading beyond its bounds.
ByteArrayInputStream rootIn = new ByteArrayInputStream(data, buffer.position(), length);
buffer.position(buffer.position() + length);
ArrowStreamReader reader = new ArrowStreamReader(
Channels.newChannel(in),
Channels.newChannel(rootIn),
allocator);
VectorSchemaRoot root = reader.getVectorSchemaRoot();
reader.loadNextBatch();
@@ -147,11 +157,15 @@ public final class ArrowUtil {
*/
public static byte[] serializeMetricsData(List<CollectRep.MetricsData> metricsDataList) {
List<VectorSchemaRoot> roots = new ArrayList<>(metricsDataList.size());
for (CollectRep.MetricsData metricsData : metricsDataList) {
VectorSchemaRoot root = metricsData.toVectorSchemaRootAndRelease();
roots.add(root);
try {
for (CollectRep.MetricsData metricsData : metricsDataList) {
VectorSchemaRoot root = metricsData.toVectorSchemaRootAndRelease();
roots.add(root);
}
return serializeMultipleRoots(roots);
} finally {
roots.forEach(VectorSchemaRoot::close);
}
return serializeMultipleRoots(roots);
}
}
@@ -0,0 +1,50 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import java.util.concurrent.TimeUnit;
/**
* Backoff utility class.
*/
public final class BackoffUtils {
/** Private constructor to prevent instantiation */
private BackoffUtils() {}
/**
* Sleeps for the next delay specified by the ExponentialBackoff instance.
* If the thread is interrupted during sleep, it resets the interrupt status
* and returns false to indicate that the operation should not continue.
*
* @param backoff the ExponentialBackoff instance to get the next delay from
* @return true if the sleep completed without interruption, false otherwise
*/
public static boolean shouldContinueAfterBackoff(ExponentialBackoff backoff) {
if (Thread.currentThread().isInterrupted()) {
return false;
}
try {
TimeUnit.MILLISECONDS.sleep(backoff.nextDelay());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return false;
}
return true;
}
}
@@ -0,0 +1,66 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
/**
* Exponential backoff utility class.
*
* <p>Provides exponentially increasing delays starting from an initial value,
* doubling with each call to {@link #nextDelay()} until reaching the maximum value.
* Call {@link #reset()} to restart from the initial delay.</p>
*
* <p>Note: This class is <b>not</b> thread-safe. Each thread should use its own instance.</p>
*/
public final class ExponentialBackoff {
private final long initial;
private final long max;
private long current;
public ExponentialBackoff(long initial, long max) {
if (initial <= 0 || max < initial) {
throw new IllegalArgumentException("Invalid exponential backoff params");
}
this.initial = initial;
this.max = max;
this.current = initial;
}
/**
* Returns the current delay value and advances to the next delay.
* The delay doubles on each call until it reaches the maximum value.
*
* @return the delay in milliseconds to wait before the next retry
*/
public long nextDelay() {
long delay = this.current;
if (this.current <= this.max / 2) {
this.current = this.current * 2;
} else {
this.current = this.max;
}
return delay;
}
/**
* Resets the backoff to the initial delay value.
* Should be called when an operation succeeds after previous failures.
*/
public void reset() {
this.current = this.initial;
}
}
@@ -0,0 +1,51 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import org.apache.hertzbeat.common.constants.HertzBeatKeywordsEnum;
import java.util.Arrays;
import java.util.Map;
import java.util.stream.Collectors;
/**
* Validate JEXL rules
*/
public class HertzBeatKeywordsUtil {
private static final Map<String, HertzBeatKeywordsEnum> KEYWORDS =
Arrays.stream(HertzBeatKeywordsEnum.values())
.collect(Collectors.toMap(HertzBeatKeywordsEnum::getKeyword, e -> e));
/**
* Verify if the field matches any reserved keywords
*
* @param field Field name to verify
*/
public static void verifyKeywords(String field) {
HertzBeatKeywordsEnum keyword = KEYWORDS.get(field);
if (null == keyword) {
return;
}
throw new IllegalArgumentException(
String.format("Field matches keyword `%s`, please set alias `%s`.",
keyword.getKeyword(), keyword.getAlias()));
}
}
@@ -25,6 +25,7 @@ import java.util.Enumeration;
import java.util.regex.Pattern;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.constants.NetworkConstants;
import org.apache.hertzbeat.common.constants.SignConstants;
import org.apache.http.conn.util.InetAddressUtils;
import org.springframework.util.StringUtils;
@@ -33,7 +34,7 @@ import org.springframework.util.StringUtils;
*/
@Slf4j
public final class IpDomainUtil {
private static final Pattern DOMAIN_PATTERN =
Pattern.compile("^[-\\w]+(\\.[-\\w]+)*$");
@@ -81,6 +82,23 @@ public final class IpDomainUtil {
return DOMAIN_SCHEMA.matcher(domainIp).matches();
}
/**
* if instance has the port with mark
* @param instance instance ip:port
* @return true if has
*/
public static boolean isHasPortWithMark(String instance) {
if (instance == null || !StringUtils.hasText(instance)) {
return false;
}
String[] parts = instance.split(SignConstants.DOUBLE_MARK);
if (parts.length >= 2) {
String port = parts[parts.length - 1];
return CommonUtil.isNumeric(port);
}
return false;
}
/**
* get localhost IP.
* @return ip
@@ -106,7 +124,7 @@ public final class IpDomainUtil {
}
return null;
}
/**
* check IP address type.
* @param ipDomain ip domain
@@ -118,7 +136,7 @@ public final class IpDomainUtil {
}
return NetworkConstants.IPV4;
}
/**
* get current local host name.
* @return hostname
@@ -126,7 +144,7 @@ public final class IpDomainUtil {
public static String getCurrentHostName() {
try {
InetAddress inetAddress = InetAddress.getLocalHost();
return inetAddress.getHostName();
return inetAddress.getHostName();
} catch (UnknownHostException e) {
return null;
}
@@ -27,6 +27,7 @@ import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.api.sync.RedisCommands;
import io.lettuce.core.KeyValue;
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
@@ -78,7 +79,7 @@ class RedisCommonDataQueueTest {
when(redisProperties.getLogEntryToStorageQueueName()).thenReturn("logEntryToStorageQueue");
when(redisProperties.getRedisHost()).thenReturn("localhost");
when(redisProperties.getRedisPort()).thenReturn(6379);
when(redisProperties.getWaitTimeout()).thenReturn(1L);
try (MockedStatic<RedisClient> mockedRedisClient = mockStatic(RedisClient.class)) {
mockedRedisClient.when(() -> RedisClient.create(any(RedisURI.class))).thenReturn(redisClient);
when(redisClient.connect(any(RedisMetricsDataCodec.class))).thenReturn(connection);
@@ -95,11 +96,13 @@ class RedisCommonDataQueueTest {
CollectRep.MetricsData metricsData = CollectRep.MetricsData.newBuilder()
.setMetrics("test metrics")
.build();
String queueName = "metricsDataQueueToAlerter";
when(syncCommands.rpop("metricsDataQueueToAlerter")).thenReturn(metricsData);
when(syncCommands.brpop(1L, queueName)).thenReturn(KeyValue.just(queueName, metricsData));
CollectRep.MetricsData actualMetricsData = redisCommonDataQueue.pollMetricsDataToAlerter();
assertEquals(metricsData, actualMetricsData);
verify(syncCommands).brpop(1L, queueName);
}
@Test
@@ -163,13 +166,14 @@ class RedisCommonDataQueueTest {
.severityText("WARN")
.body("Test warning log message")
.build();
String queueName = "logEntryQueue";
when(logEntrySyncCommands.rpop("logEntryQueue")).thenReturn(expectedLogEntry);
when(logEntrySyncCommands.brpop(1L, queueName)).thenReturn(KeyValue.just(queueName, expectedLogEntry));
LogEntry result = redisCommonDataQueue.pollLogEntry();
assertEquals(expectedLogEntry, result);
verify(logEntrySyncCommands).rpop("logEntryQueue");
verify(logEntrySyncCommands).brpop(1L, "logEntryQueue");
}
@Test
@@ -181,13 +185,13 @@ class RedisCommonDataQueueTest {
.severityText("FATAL")
.body("Critical error log for storage")
.build();
String queueName = "logEntryToStorageQueue";
when(logEntrySyncCommands.rpop("logEntryToStorageQueue")).thenReturn(expectedLogEntry);
when(logEntrySyncCommands.brpop(1L, queueName)).thenReturn(KeyValue.just(queueName, expectedLogEntry));
LogEntry result = redisCommonDataQueue.pollLogEntryToStorage();
assertEquals(expectedLogEntry, result);
verify(logEntrySyncCommands).rpop("logEntryToStorageQueue");
verify(logEntrySyncCommands).brpop(1L, "logEntryToStorageQueue");
}
@Test
@@ -0,0 +1,267 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.support.valid;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.util.Arrays;
import java.util.Collections;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertThrows;
/**
* Test for {@link SqlSecurityValidator}
*/
class SqlSecurityValidatorTest {
private SqlSecurityValidator validator;
@BeforeEach
void setUp() {
validator = new SqlSecurityValidator(Arrays.asList("hertzbeat_logs", "app_logs", "access_logs"));
}
@Test
void testValidSelectStatement() {
assertDoesNotThrow(() -> validator.validate("SELECT * FROM hertzbeat_logs"));
assertDoesNotThrow(() -> validator.validate("SELECT id, message FROM hertzbeat_logs WHERE level = 'ERROR'"));
assertDoesNotThrow(() -> validator.validate("SELECT COUNT(*) FROM app_logs"));
assertDoesNotThrow(() -> validator.validate("select * from HERTZBEAT_LOGS")); // case insensitive
}
@Test
void testSelectWithJoin() {
assertDoesNotThrow(() -> validator.validate(
"SELECT a.id, b.message FROM hertzbeat_logs a JOIN app_logs b ON a.id = b.id"));
}
@Test
void testSelectWithSubqueryNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs WHERE id IN (SELECT id FROM app_logs)"));
}
@Test
void testSelectWithSubqueryInFromNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM (SELECT * FROM hertzbeat_logs) AS subq"));
}
@Test
void testEmptySql() {
assertThrows(SqlSecurityException.class, () -> validator.validate(null));
assertThrows(SqlSecurityException.class, () -> validator.validate(""));
assertThrows(SqlSecurityException.class, () -> validator.validate(" "));
}
@Test
void testInvalidSqlSyntax() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FORM hertzbeat_logs")); // typo: FORM instead of FROM
}
@Test
void testInsertNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("INSERT INTO hertzbeat_logs (message) VALUES ('test')"));
}
@Test
void testUpdateNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("UPDATE hertzbeat_logs SET message = 'test' WHERE id = 1"));
}
@Test
void testDeleteNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("DELETE FROM hertzbeat_logs WHERE id = 1"));
}
@Test
void testDropNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("DROP TABLE hertzbeat_logs"));
}
@Test
void testTruncateNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("TRUNCATE TABLE hertzbeat_logs"));
}
@Test
void testAlterNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("ALTER TABLE hertzbeat_logs ADD COLUMN new_col VARCHAR(100)"));
}
@Test
void testCreateNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("CREATE TABLE new_table (id INT)"));
}
@Test
void testUnauthorizedTable() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM users"));
}
@Test
void testUnauthorizedTableInJoin() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs JOIN users ON hertzbeat_logs.user_id = users.id"));
}
@Test
void testUnauthorizedTableInSubquery() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs WHERE user_id IN (SELECT id FROM users)"));
}
@Test
void testTableWithQuotes() {
assertDoesNotThrow(() -> validator.validate("SELECT * FROM \"hertzbeat_logs\""));
assertDoesNotThrow(() -> validator.validate("SELECT * FROM `hertzbeat_logs`"));
}
@Test
void testEmptyAllowedTables() {
SqlSecurityValidator emptyValidator = new SqlSecurityValidator(Collections.emptyList());
assertThrows(SqlSecurityException.class,
() -> emptyValidator.validate("SELECT * FROM any_table"));
}
@Test
void testNullAllowedTables() {
SqlSecurityValidator nullValidator = new SqlSecurityValidator(null);
assertThrows(SqlSecurityException.class,
() -> nullValidator.validate("SELECT * FROM any_table"));
}
@Test
void testComplexSelectWithAggregation() {
assertDoesNotThrow(() -> validator.validate(
"SELECT level, COUNT(*) as cnt FROM hertzbeat_logs GROUP BY level HAVING COUNT(*) > 10 ORDER BY cnt DESC LIMIT 100"));
}
@Test
void testSelectWithUnionNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs UNION SELECT * FROM app_logs"));
}
@Test
void testSelectWithUnionAllNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs UNION ALL SELECT * FROM app_logs"));
}
@Test
void testSelectWithIntersectNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs INTERSECT SELECT * FROM app_logs"));
}
@Test
void testSelectWithExceptNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs EXCEPT SELECT * FROM app_logs"));
}
@Test
void testLateralSubqueryNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM hertzbeat_logs, LATERAL (SELECT * FROM app_logs) AS t"));
}
@Test
void testWithClauseNotAllowed() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("WITH cte AS (SELECT * FROM hertzbeat_logs) SELECT * FROM cte"));
}
@Test
void testSqlInjectionAttemptDropTable() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("DROP TABLE users"));
}
@Test
void testSqlInjectionAttemptUnauthorizedTable() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("SELECT * FROM users"));
}
@Test
void testSqlInjectionAttemptDeleteFrom() {
assertThrows(SqlSecurityException.class,
() -> validator.validate("DELETE FROM hertzbeat_logs WHERE 1=1"));
}
@Test
void testBypassInSelectItems() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT (SELECT password FROM secret_table) FROM hertzbeat_logs"));
}
@Test
void testBypassInWhereClauseAnd() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE 1=1 AND id IN (SELECT id FROM secret_table)"));
}
@Test
void testBypassInFunction() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE id = abs((SELECT count(*) FROM secret_table))"));
}
@Test
void testBypassInCaseWhen() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE status = (CASE WHEN (SELECT 1 FROM secret_table)=1 THEN 1 ELSE 0 END)"));
}
@Test
void testBypassWithAndExpression() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE 1=1 AND id = (SELECT id FROM secret_table)"));
}
@Test
void testBypassWithGreaterThan() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE id > (SELECT count(*) FROM secret_table)"));
}
@Test
void testBypassWithBetween() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE id BETWEEN 1 AND (SELECT id FROM secret_table)"));
}
@Test
void testBypassWithMathOperations() {
assertThrows(SqlSecurityException.class, () -> validator.validate(
"SELECT * FROM hertzbeat_logs WHERE id = 1 + (SELECT id FROM secret_table)"));
}
}
@@ -0,0 +1,123 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import com.google.common.collect.Lists;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.BigIntVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.apache.arrow.vector.types.pojo.Schema;
import org.apache.hertzbeat.common.entity.message.CollectRep;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/**
* Test case for {@link ArrowUtil}
*/
class ArrowUtilTest {
@Test
void testSerializeAndDeserializeMultipleRoots() {
RootAllocator allocator = new RootAllocator();
List<VectorSchemaRoot> roots = new ArrayList<>();
// Create first root
Field field1 = new Field("field1", FieldType.nullable(new ArrowType.Int(64, true)), null);
Schema schema1 = new Schema(Collections.singletonList(field1));
VectorSchemaRoot root1 = VectorSchemaRoot.create(schema1, allocator);
BigIntVector vector1 = (BigIntVector) root1.getVector("field1");
vector1.allocateNew(10);
vector1.setSafe(0, 100L);
vector1.setValueCount(1);
root1.setRowCount(1);
roots.add(root1);
// Create second root
Field field2 = new Field("field2", FieldType.nullable(new ArrowType.Int(64, true)), null);
Schema schema2 = new Schema(Collections.singletonList(field2));
VectorSchemaRoot root2 = VectorSchemaRoot.create(schema2, allocator);
BigIntVector vector2 = (BigIntVector) root2.getVector("field2");
vector2.allocateNew(10);
vector2.setSafe(0, 200L);
vector2.setValueCount(1);
root2.setRowCount(1);
roots.add(root2);
// Serialize
byte[] data = ArrowUtil.serializeMultipleRoots(roots);
// Deserialize
List<VectorSchemaRoot> deserializedRoots = ArrowUtil.deserializeMultipleRoots(data);
Assertions.assertEquals(2, deserializedRoots.size());
VectorSchemaRoot resultRoot1 = deserializedRoots.get(0);
Assertions.assertEquals(1, resultRoot1.getRowCount());
Assertions.assertEquals(100L, ((BigIntVector) resultRoot1.getVector("field1")).get(0));
VectorSchemaRoot resultRoot2 = deserializedRoots.get(1);
Assertions.assertEquals(1, resultRoot2.getRowCount());
Assertions.assertEquals(200L, ((BigIntVector) resultRoot2.getVector("field2")).get(0));
// Cleanup
roots.forEach(VectorSchemaRoot::close);
deserializedRoots.forEach(VectorSchemaRoot::close);
allocator.close();
}
@Test
void testSerializeAndDeserializeMetricsData() {
CollectRep.MetricsData metricsData = CollectRep.MetricsData.newBuilder()
.setId(1L)
.setApp("linux")
.setMetrics("cpu")
.setTime(System.currentTimeMillis())
.addField(CollectRep.Field.newBuilder().setName("usage").setType(1).build())
.addValueRow(CollectRep.ValueRow.newBuilder().addColumn("50.5").build())
.build();
CollectRep.MetricsData metricsData1 = CollectRep.MetricsData.newBuilder()
.setId(1L)
.setApp("linux_1")
.setMetrics("cpu")
.setTime(System.currentTimeMillis())
.addField(CollectRep.Field.newBuilder().setName("usage").setType(1).build())
.addValueRow(CollectRep.ValueRow.newBuilder().addColumn("60.5").build())
.build();
List<CollectRep.MetricsData> list = Lists.newArrayList(metricsData, metricsData1);
// Serialize
byte[] data = ArrowUtil.serializeMetricsData(list);
// Deserialize
List<CollectRep.MetricsData> deserializedList = ArrowUtil.deserializeMetricsData(data);
Assertions.assertEquals(2, deserializedList.size());
CollectRep.MetricsData result = deserializedList.get(0);
Assertions.assertEquals("linux", result.getApp());
Assertions.assertEquals(1, result.getValues().size());
Assertions.assertEquals("50.5", result.getValues().get(0).getColumns(0));
result = deserializedList.get(1);
Assertions.assertEquals("linux_1", result.getApp());
Assertions.assertEquals(1, result.getValues().size());
Assertions.assertEquals("60.5", result.getValues().get(0).getColumns(0));
}
}
@@ -0,0 +1,68 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* Test case for {@link BackoffUtils}
*/
class BackoffUtilsTest {
@Test
void shouldContinueAfterBackoff() {
ExponentialBackoff backoff = new ExponentialBackoff(10L, 100L);
boolean shouldContinue = BackoffUtils.shouldContinueAfterBackoff(backoff);
assertTrue(shouldContinue);
}
@Test
void shouldNotContinueWhenInterrupted() {
Thread.currentThread().interrupt();
ExponentialBackoff backoff = new ExponentialBackoff(10L, 100L);
boolean shouldContinue = BackoffUtils.shouldContinueAfterBackoff(backoff);
assertFalse(shouldContinue);
}
@Test
void shouldHandleInterruptedException() throws InterruptedException {
final Thread mainThread = Thread.currentThread();
ExponentialBackoff backoff = new ExponentialBackoff(1000L, 2000L);
Thread interruptingThread = new Thread(() -> {
try {
// Give the main thread some time to enter the sleep
Thread.sleep(200);
mainThread.interrupt();
} catch (InterruptedException ignored) {
}
});
interruptingThread.start();
boolean shouldContinue = BackoffUtils.shouldContinueAfterBackoff(backoff);
interruptingThread.join();
assertFalse(shouldContinue);
assertTrue(Thread.currentThread().isInterrupted());
}
}
@@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
/**
* Test case for {@link ExponentialBackoff}
*/
class ExponentialBackoffTest {
@Test
void testProgressionAndCap() {
ExponentialBackoff backoff = new ExponentialBackoff(50L, 1000L);
Assertions.assertEquals(50L, backoff.nextDelay());
Assertions.assertEquals(100L, backoff.nextDelay());
Assertions.assertEquals(200L, backoff.nextDelay());
Assertions.assertEquals(400L, backoff.nextDelay());
Assertions.assertEquals(800L, backoff.nextDelay());
Assertions.assertEquals(1000L, backoff.nextDelay());
Assertions.assertEquals(1000L, backoff.nextDelay());
}
@Test
void testReset() {
ExponentialBackoff backoff = new ExponentialBackoff(50L, 1000L);
Assertions.assertEquals(50L, backoff.nextDelay());
Assertions.assertEquals(100L, backoff.nextDelay());
backoff.reset();
Assertions.assertEquals(50L, backoff.nextDelay());
}
@Test
void testInvalidParams() {
Assertions.assertThrows(IllegalArgumentException.class, () -> new ExponentialBackoff(0L, 1000L));
Assertions.assertThrows(IllegalArgumentException.class, () -> new ExponentialBackoff(50L, 10L));
}
}
@@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.common.util;
import org.apache.hertzbeat.common.constants.HertzBeatKeywordsEnum;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
/**
* Test case for {@link HertzBeatKeywordsUtil}
*/
class HertzBeatKeywordsUtilTest {
@Test
void testVerifyKeywords_WithReservedKeyword_Instance() {
IllegalArgumentException exception = assertThrows(
IllegalArgumentException.class,
() -> HertzBeatKeywordsUtil.verifyKeywords("instance")
);
String expectedMessage = String.format(
"Field matches keyword `%s`, please set alias `%s`.",
HertzBeatKeywordsEnum.INSTANCE.getKeyword(),
HertzBeatKeywordsEnum.INSTANCE.getAlias()
);
assertEquals(expectedMessage, exception.getMessage());
}
@Test
void testVerifyKeywords_WithValidField() {
assertDoesNotThrow(() -> HertzBeatKeywordsUtil.verifyKeywords("cpu_usage"));
assertDoesNotThrow(() -> HertzBeatKeywordsUtil.verifyKeywords("memory_used"));
assertDoesNotThrow(() -> HertzBeatKeywordsUtil.verifyKeywords("response_time"));
assertDoesNotThrow(() -> HertzBeatKeywordsUtil.verifyKeywords("metric_instance"));
assertDoesNotThrow(() -> HertzBeatKeywordsUtil.verifyKeywords(HertzBeatKeywordsEnum.INSTANCE.getAlias()));
}
}
@@ -102,7 +102,7 @@ public class LogPeriodicAlertE2eTest {
@DynamicPropertySource
static void greptimeProps(DynamicPropertyRegistry r) {
// Configure GreptimeDB storage
r.add("warehouse.store.jpa.enabled", () -> "false");
r.add("warehouse.store.duckdb.enabled", () -> "false");
r.add("warehouse.store.greptime.enabled", () -> "true");
r.add("warehouse.store.greptime.http-endpoint", () -> "http://localhost:" + greptimedb.getMappedPort(GREPTIME_HTTP_PORT));
r.add("warehouse.store.greptime.grpc-endpoints", () -> "localhost:" + greptimedb.getMappedPort(GREPTIME_GRPC_PORT));
@@ -86,7 +86,7 @@ public class GreptimeLogStorageE2eTest {
@DynamicPropertySource
static void greptimeProps(DynamicPropertyRegistry r) {
r.add("warehouse.store.jpa.enabled", () -> "false");
r.add("warehouse.store.duckdb.enabled", () -> "false");
r.add("warehouse.store.greptime.enabled", () -> "true");
r.add("warehouse.store.greptime.http-endpoint", () -> "http://localhost:" + greptimedb.getMappedPort(GREPTIME_HTTP_PORT));
r.add("warehouse.store.greptime.grpc-endpoints", () -> "localhost:" + greptimedb.getMappedPort(GREPTIME_GRPC_PORT));
+11
View File
@@ -28,6 +28,10 @@
<artifactId>hertzbeat-log</artifactId>
<name>${project.artifactId}</name>
<properties>
<awaitility.version>4.2.0</awaitility.version>
</properties>
<dependencies>
<!-- common -->
<dependency>
@@ -67,6 +71,13 @@
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
</dependency>
<!-- awaitility for async testing -->
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>${awaitility.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
@@ -60,7 +60,7 @@ public class LogIngestionController {
@PostMapping("/ingest/{protocol}")
public ResponseEntity<Message<Void>> ingestExternLog(@PathVariable("protocol") String protocol,
@RequestBody String content) {
log.info("Receive extern log from protocol: {}, content length: {}", protocol, content == null ? 0 : content.length());
log.debug("Receive extern log from protocol: {}, content length: {}", protocol, content == null ? 0 : content.length());
if (!StringUtils.hasText(protocol)) {
protocol = DEFAULT_PROTOCOL; // Default to OTLP if no protocol specified
}
@@ -19,34 +19,75 @@
package org.apache.hertzbeat.log.notice;
import jakarta.annotation.PreDestroy;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
/**
* SSE manager for log
* SSE manager for log with batch processing support for high TPS scenarios
*/
@Component
@Slf4j
@Getter
public class LogSseManager {
private static final long BATCH_INTERVAL_MS = 200;
private static final int MAX_BATCH_SIZE = 1000;
private static final int MAX_QUEUE_SIZE = 10000;
private final Map<Long, SseSubscriber> emitters = new ConcurrentHashMap<>();
private final Queue<LogEntry> logQueue = new ConcurrentLinkedQueue<>();
private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r, "sse-batch-scheduler");
t.setDaemon(true);
return t;
});
private final ExecutorService senderPool = Executors.newCachedThreadPool(r -> {
Thread t = new Thread(r, "sse-sender");
t.setDaemon(true);
return t;
});
private final AtomicLong queueSize = new AtomicLong(0);
public LogSseManager() {
scheduler.scheduleAtFixedRate(this::flushBatch, BATCH_INTERVAL_MS, BATCH_INTERVAL_MS, TimeUnit.MILLISECONDS);
}
@PreDestroy
public void shutdown() {
scheduler.shutdown();
senderPool.shutdown();
try {
scheduler.awaitTermination(2, TimeUnit.SECONDS);
senderPool.awaitTermination(2, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
scheduler.shutdownNow();
senderPool.shutdownNow();
}
/**
* Create a new SSE emitter for a client with specified filters
* @param clientId The unique identifier for the client
* @param filters The filters to apply to the log data
* @return The SSE emitter
*/
public SseEmitter createEmitter(Long clientId, LogSseFilterCriteria filters) {
SseEmitter emitter = new SseEmitter(Long.MAX_VALUE);
@@ -54,55 +95,115 @@ public class LogSseManager {
emitter.onTimeout(() -> removeEmitter(clientId));
emitter.onError((ex) -> removeEmitter(clientId));
SseSubscriber subscriber = new SseSubscriber(emitter, filters);
emitters.put(clientId, subscriber);
emitters.put(clientId, new SseSubscriber(emitter, filters));
return emitter;
}
/**
* Broadcast log data to all subscribers
* @param logEntry The log data to broadcast
* Queue log entry for batch processing
*/
@Async
public void broadcast(LogEntry logEntry) {
emitters.forEach((clientId, subscriber) -> {
try {
// Check if the log entry matches the subscriber's filter criteria
if (subscriber.filters == null || subscriber.filters.matches(logEntry)) {
subscriber.emitter.send(SseEmitter.event()
.id(String.valueOf(System.currentTimeMillis()))
.name("LOG_EVENT")
.data(logEntry));
}
} catch (IOException | IllegalStateException e) {
subscriber.emitter.complete();
removeEmitter(clientId);
} catch (Exception exception) {
log.error("Failed to broadcast log to client: {}", exception.getMessage());
subscriber.emitter.complete();
removeEmitter(clientId);
if (queueSize.incrementAndGet() > MAX_QUEUE_SIZE) {
queueSize.decrementAndGet();
return;
}
boolean offered = logQueue.offer(logEntry);
if (!offered) {
queueSize.decrementAndGet();
log.warn("Failed to enqueue log entry: {}", logEntry);
}
}
/**
* Flush queued logs to all subscribers in batch
*/
private void flushBatch() {
try {
if (logQueue.isEmpty() || emitters.isEmpty()) {
return;
}
});
List<LogEntry> batch = new ArrayList<>(MAX_BATCH_SIZE);
LogEntry entry;
while (batch.size() < MAX_BATCH_SIZE && (entry = logQueue.poll()) != null) {
batch.add(entry);
queueSize.decrementAndGet();
}
if (batch.isEmpty()) {
return;
}
// Send to each subscriber in parallel
for (Map.Entry<Long, SseSubscriber> e : emitters.entrySet()) {
Long clientId = e.getKey();
SseSubscriber subscriber = e.getValue();
List<LogEntry> filtered = filterLogs(batch, subscriber.filters);
if (!filtered.isEmpty()) {
senderPool.submit(() -> sendToSubscriber(clientId, subscriber.emitter, filtered));
}
}
} catch (Exception e) {
log.error("Error in flushBatch: {}", e.getMessage(), e);
}
}
private void sendToSubscriber(Long clientId, SseEmitter emitter, List<LogEntry> logs) {
try {
long batchTimestamp = System.currentTimeMillis();
int sequenceNumber = 0;
for (LogEntry logEntry : logs) {
String eventId = batchTimestamp + "-" + sequenceNumber++;
emitter.send(SseEmitter.event()
.id(eventId)
.name("LOG_EVENT")
.data(logEntry));
}
} catch (IOException | IllegalStateException e) {
safeComplete(clientId, emitter);
} catch (Exception e) {
log.error("Failed to send to client {}: {}", clientId, e.getMessage());
safeComplete(clientId, emitter);
}
}
private void safeComplete(Long clientId, SseEmitter emitter) {
try {
emitter.complete();
} catch (Exception ignored) {
}
removeEmitter(clientId);
}
private List<LogEntry> filterLogs(List<LogEntry> logs, LogSseFilterCriteria filters) {
if (filters == null) {
return logs;
}
List<LogEntry> filtered = new ArrayList<>();
for (LogEntry log : logs) {
if (filters.matches(log)) {
filtered.add(log);
}
}
return filtered;
}
private void removeEmitter(Long clientId) {
emitters.remove(clientId);
}
public long getQueueSize() {
return queueSize.get();
}
/**
* SSE subscriber
* SseSubscriber for SseEmitter and LogSseFilterCriteria
*/
@Data
@AllArgsConstructor
@NoArgsConstructor
public static class SseSubscriber {
/**
* The SSE emitter for streaming log events
*/
private SseEmitter emitter;
/**
* The filters for streaming log events
*/
private LogSseFilterCriteria filters;
}
}
@@ -69,13 +69,9 @@ public class OtlpLogProtocolAdapter implements LogProtocolAdapter {
// Extract LogEntry instances from the request
List<LogEntry> logEntries = extractLogEntries(request);
log.debug("Successfully extracted {} log entries from OTLP payload {}", logEntries.size(), content);
logEntries.forEach(entry -> {
commonDataQueue.sendLogEntry(entry);
logSseManager.broadcast(entry);
log.info("Log entry sent to queue: {}", entry);
});
commonDataQueue.sendLogEntryToStorageBatch(logEntries);
commonDataQueue.sendLogEntryToAlertBatch(logEntries);
logEntries.forEach(logSseManager::broadcast);
} catch (InvalidProtocolBufferException e) {
log.error("Failed to parse OTLP log payload: {}", e.getMessage());
throw new IllegalArgumentException("Invalid OTLP log content", e);
@@ -20,19 +20,21 @@
package org.apache.hertzbeat.log.notice;
import org.apache.hertzbeat.common.entity.log.LogEntry;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Spy;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import static org.awaitility.Awaitility.await;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.atLeastOnce;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
@@ -41,14 +43,21 @@ import static org.mockito.Mockito.verify;
/**
* Unit tests for {@link LogSseManager}.
*/
@ExtendWith(MockitoExtension.class)
class LogSseManagerTest {
@Spy
private LogSseManager logSseManager;
private static final Long CLIENT_ID = 1L;
@BeforeEach
void setUp() {
logSseManager = new LogSseManager();
}
@AfterEach
void tearDown() {
logSseManager.shutdown();
}
@Test
void shouldCreateAndStoreEmitter() {
// When: Creating a new emitter for a client
@@ -73,12 +82,14 @@ class LogSseManagerTest {
// When: An "INFO" log is broadcast
logSseManager.broadcast(infoLog);
// Then: The log should be sent to the client
verify(mockEmitter).send(any(SseEmitter.SseEventBuilder.class));
// Then: The log should be sent to the client (wait for batch processing)
await().atMost(500, TimeUnit.MILLISECONDS).untilAsserted(() ->
verify(mockEmitter, atLeastOnce()).send(any(SseEmitter.SseEventBuilder.class))
);
}
@Test
void shouldNotBroadcastLogWhenFilterDoesNotMatch() throws IOException {
void shouldNotBroadcastLogWhenFilterDoesNotMatch() throws IOException, InterruptedException {
// Given: A client with a filter for "ERROR" logs
LogSseFilterCriteria filters = new LogSseFilterCriteria();
filters.setSeverityText("ERROR");
@@ -90,6 +101,9 @@ class LogSseManagerTest {
// When: An "INFO" log is broadcast
logSseManager.broadcast(infoLog);
// Wait for batch processing
Thread.sleep(300);
// Then: The log should NOT be sent to the client
verify(mockEmitter, never()).send(any(SseEmitter.SseEventBuilder.class));
}
@@ -105,12 +119,14 @@ class LogSseManagerTest {
// When: Any log is broadcast
logSseManager.broadcast(anyLog);
// Then: The log should be sent to the client
verify(mockEmitter).send(any(SseEmitter.SseEventBuilder.class));
// Then: The log should be sent to the client (wait for batch processing)
await().atMost(500, TimeUnit.MILLISECONDS).untilAsserted(() ->
verify(mockEmitter, atLeastOnce()).send(any(SseEmitter.SseEventBuilder.class))
);
}
@Test
void shouldBroadcastOnlyToMatchingSubscribers() throws IOException {
void shouldBroadcastOnlyToMatchingSubscribers() throws IOException, InterruptedException {
// Given: Two clients with different filters
LogSseFilterCriteria infoFilter = new LogSseFilterCriteria();
infoFilter.setSeverityText("INFO");
@@ -127,8 +143,12 @@ class LogSseManagerTest {
// When: An "INFO" log is broadcast
logSseManager.broadcast(infoLog);
// Wait for batch processing
await().atMost(500, TimeUnit.MILLISECONDS).untilAsserted(() ->
verify(infoEmitter, atLeastOnce()).send(any(SseEmitter.SseEventBuilder.class))
);
// Then: The log is sent only to the client subscribed to "INFO" logs
verify(infoEmitter).send(any(SseEmitter.SseEventBuilder.class));
verify(errorEmitter, never()).send(any(SseEmitter.SseEventBuilder.class));
}
@@ -146,8 +166,10 @@ class LogSseManagerTest {
logSseManager.broadcast(log);
// Then: The failing emitter should be completed and removed
verify(mockEmitter).complete();
assertFalse(logSseManager.getEmitters().containsKey(CLIENT_ID));
await().atMost(500, TimeUnit.MILLISECONDS).untilAsserted(() -> {
verify(mockEmitter).complete();
assertFalse(logSseManager.getEmitters().containsKey(CLIENT_ID));
});
}
/**
@@ -44,6 +44,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoInteractions;
@@ -85,12 +86,16 @@ class OtlpLogProtocolAdapterTest {
adapter.ingest(otlpPayload);
ArgumentCaptor<LogEntry> logEntryCaptor = ArgumentCaptor.forClass(LogEntry.class);
verify(commonDataQueue, times(1)).sendLogEntry(logEntryCaptor.capture());
verify(logSseManager, times(1)).broadcast(logEntryCaptor.capture());
ArgumentCaptor<List<LogEntry>> listCaptor = ArgumentCaptor.forClass(List.class);
verify(commonDataQueue, times(1)).sendLogEntryToStorageBatch(listCaptor.capture());
verify(commonDataQueue, times(1)).sendLogEntryToAlertBatch(anyList());
verify(logSseManager, times(1)).broadcast(any(LogEntry.class));
LogEntry capturedEntry = logEntryCaptor.getValue();
assertNotNull(capturedEntry);
List<LogEntry> capturedList = listCaptor.getValue();
assertNotNull(capturedList);
assertEquals(1, capturedList.size());
LogEntry capturedEntry = capturedList.get(0);
assertEquals("test-service", capturedEntry.getResource().get("service_name"));
assertEquals("test-version", capturedEntry.getResource().get("service_version"));
assertEquals("test-scope", capturedEntry.getInstrumentationScope().getName());
@@ -106,8 +111,13 @@ class OtlpLogProtocolAdapterTest {
adapter.ingest(otlpPayload);
verify(commonDataQueue, times(2)).sendLogEntry(any(LogEntry.class));
ArgumentCaptor<List<LogEntry>> listCaptor = ArgumentCaptor.forClass(List.class);
verify(commonDataQueue, times(1)).sendLogEntryToStorageBatch(listCaptor.capture());
verify(commonDataQueue, times(1)).sendLogEntryToAlertBatch(anyList());
verify(logSseManager, times(2)).broadcast(any(LogEntry.class));
List<LogEntry> capturedList = listCaptor.getValue();
assertEquals(2, capturedList.size());
}
@Test
@@ -116,15 +126,16 @@ class OtlpLogProtocolAdapterTest {
adapter.ingest(otlpPayload);
verify(commonDataQueue, times(1)).sendLogEntry(any(LogEntry.class));
ArgumentCaptor<List<LogEntry>> listCaptor = ArgumentCaptor.forClass(List.class);
verify(commonDataQueue, times(1)).sendLogEntryToStorageBatch(listCaptor.capture());
verify(commonDataQueue, times(1)).sendLogEntryToAlertBatch(anyList());
verify(logSseManager, times(1)).broadcast(any(LogEntry.class));
ArgumentCaptor<LogEntry> logEntryCaptor = ArgumentCaptor.forClass(LogEntry.class);
verify(commonDataQueue).sendLogEntry(logEntryCaptor.capture());
LogEntry capturedEntry = logEntryCaptor.getValue();
assertNotNull(capturedEntry);
List<LogEntry> capturedList = listCaptor.getValue();
assertNotNull(capturedList);
assertEquals(1, capturedList.size());
LogEntry capturedEntry = capturedList.get(0);
Map<String, Object> attributes = capturedEntry.getAttributes();
assertEquals("string_value", attributes.get("string_attr"));
assertEquals(true, attributes.get("bool_attr"));
@@ -145,12 +156,15 @@ class OtlpLogProtocolAdapterTest {
adapter.ingest(otlpPayload);
verify(commonDataQueue, times(1)).sendLogEntry(any(LogEntry.class));
ArgumentCaptor<List<LogEntry>> listCaptor = ArgumentCaptor.forClass(List.class);
verify(commonDataQueue, times(1)).sendLogEntryToStorageBatch(listCaptor.capture());
verify(commonDataQueue, times(1)).sendLogEntryToAlertBatch(anyList());
verify(logSseManager, times(1)).broadcast(any(LogEntry.class));
ArgumentCaptor<LogEntry> logEntryCaptor = ArgumentCaptor.forClass(LogEntry.class);
verify(commonDataQueue).sendLogEntry(logEntryCaptor.capture());
List<LogEntry> capturedList = listCaptor.getValue();
assertEquals(1, capturedList.size());
LogEntry capturedEntry = logEntryCaptor.getValue();
LogEntry capturedEntry = capturedList.get(0);
assertEquals("1234567890abcdef1234567890abcdef", capturedEntry.getTraceId());
assertEquals("1234567890abcdef", capturedEntry.getSpanId());
assertEquals(1, capturedEntry.getTraceFlags());
@@ -170,7 +184,15 @@ class OtlpLogProtocolAdapterTest {
adapter.ingest(otlpPayload);
verifyNoInteractions(commonDataQueue, logSseManager);
ArgumentCaptor<List<LogEntry>> listCaptor = ArgumentCaptor.forClass(List.class);
verify(commonDataQueue, times(1)).sendLogEntryToStorageBatch(listCaptor.capture());
verify(commonDataQueue, times(1)).sendLogEntryToAlertBatch(anyList());
List<LogEntry> capturedList = listCaptor.getValue();
assertNotNull(capturedList);
assertEquals(0, capturedList.size());
verifyNoInteractions(logSseManager);
}
private String createValidOtlpLogPayload() throws Exception {
@@ -18,13 +18,6 @@
package org.apache.hertzbeat.manager.component.sd;
import com.google.common.collect.Maps;
import java.time.LocalDateTime;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.hertzbeat.common.constants.CommonConstants;
import org.apache.hertzbeat.common.entity.arrow.RowWrapper;
@@ -34,6 +27,9 @@ import org.apache.hertzbeat.common.entity.manager.MonitorBind;
import org.apache.hertzbeat.common.entity.manager.Param;
import org.apache.hertzbeat.common.entity.message.CollectRep;
import org.apache.hertzbeat.common.queue.CommonDataQueue;
import org.apache.hertzbeat.common.support.exception.CommonDataQueueUnknownException;
import org.apache.hertzbeat.common.util.BackoffUtils;
import org.apache.hertzbeat.common.util.ExponentialBackoff;
import org.apache.hertzbeat.manager.dao.CollectorMonitorBindDao;
import org.apache.hertzbeat.manager.dao.MonitorBindDao;
import org.apache.hertzbeat.manager.dao.MonitorDao;
@@ -43,6 +39,14 @@ import org.apache.hertzbeat.manager.service.MonitorService;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;
import java.time.LocalDateTime;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
/**
* Service Discovery Worker
*/
@@ -80,8 +84,13 @@ public class ServiceDiscoveryWorker implements InitializingBean {
private class SdUpdateTask implements Runnable {
@Override
public void run() {
ExponentialBackoff backoff = new ExponentialBackoff(50L, 1000L);
while (!Thread.currentThread().isInterrupted()) {
try (final CollectRep.MetricsData metricsData = dataQueue.pollServiceDiscoveryData()) {
if (metricsData == null) {
continue;
}
backoff.reset();
Long monitorId = metricsData.getId();
final Monitor mainMonitor = monitorDao.findById(monitorId).orElse(null);
if (mainMonitor == null) {
@@ -114,13 +123,14 @@ public class ServiceDiscoveryWorker implements InitializingBean {
.filter(p -> !p.isEmpty())
.orElse(defaultPort);
final String keyStr = host + ":" + port;
final String instance = port.isEmpty() ? host : host + ":" + port;
if (subMonitorBindMap.containsKey(keyStr)) {
subMonitorBindMap.remove(keyStr);
continue;
}
Monitor newMonitor = mainMonitor.clone();
newMonitor.setId(null);
newMonitor.setHost(host);
newMonitor.setInstance(instance);
newMonitor.setName(newMonitor.getName() + "-" + host + ":" + port);
newMonitor.setScrape(CommonConstants.SCRAPE_STATIC);
newMonitor.setGmtCreate(LocalDateTime.now());
@@ -153,6 +163,10 @@ public class ServiceDiscoveryWorker implements InitializingBean {
final Set<Long> needCancelMonitorIdSet = subMonitorBindMap.values().stream()
.map(MonitorBind::getMonitorId).collect(Collectors.toSet());
monitorService.deleteMonitors(needCancelMonitorIdSet);
} catch (CommonDataQueueUnknownException ue) {
if (!BackoffUtils.shouldContinueAfterBackoff(backoff)) {
break;
}
} catch (Exception exception) {
log.error(exception.getMessage(), exception);
}
@@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hertzbeat.manager.component.validator;
import org.apache.hertzbeat.common.entity.manager.Param;
import org.apache.hertzbeat.common.entity.manager.ParamDefine;
/**
* Parameter validator interface
*/
public interface ParamValidator {
/**
* Check if the validator supports the given parameter type
*
* @param type parameter type
* @return true if supported
*/
boolean support(String type);
/**
* Validate the parameter
*
* @param paramDefine parameter definition
* @param param parameter actual value
* @throws IllegalArgumentException if validation fails
*/
void validate(ParamDefine paramDefine, Param param) throws IllegalArgumentException;
}

Some files were not shown because too many files have changed in this diff Show More