Compare commits
305
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
eac34cc170 | ||
|
|
894b57bd3d | ||
|
|
8c92ce2caa | ||
|
|
ab6cf73527 | ||
|
|
47a9ac5642 | ||
|
|
a81ba0c95c | ||
|
|
a63f493dc0 | ||
|
|
e9a579759e | ||
|
|
9168f4c796 | ||
|
|
53b2288345 | ||
|
|
1b20d9a9e5 | ||
|
|
8c6dca4fcf | ||
|
|
64eeea95bb | ||
|
|
f0fcca0810 | ||
|
|
1a7a00c7d1 | ||
|
|
ef84593760 | ||
|
|
50c40095f3 | ||
|
|
e740300481 | ||
|
|
04c3c2e810 | ||
|
|
da9134bda4 | ||
|
|
8d9e936047 | ||
|
|
f6f94821bc | ||
|
|
d1a17c5c91 | ||
|
|
1a5f1f0288 | ||
|
|
2d37cf10b9 | ||
|
|
cb518114dc | ||
|
|
4e3fdd9a64 | ||
|
|
81b004fbb4 | ||
|
|
99ce616479 | ||
|
|
2ee8878da1 | ||
|
|
fa569937f2 | ||
|
|
8d29a621be | ||
|
|
bcbe242a29 | ||
|
|
c5c41de1b1 | ||
|
|
3b7bf92edf | ||
|
|
36f23ecc3e | ||
|
|
3d35c22424 | ||
|
|
740bf010e8 | ||
|
|
ea488d20c4 | ||
|
|
9a51e665e5 | ||
|
|
b55d26dca7 | ||
|
|
57d82f0636 | ||
|
|
3a0b52ac88 | ||
|
|
e1590952ec | ||
|
|
ccdf5958be | ||
|
|
788cca5ab9 | ||
|
|
5c1ecef9f7 | ||
|
|
7436176b27 | ||
|
|
c96256c9a2 | ||
|
|
fad2760ca2 | ||
|
|
e3f9e1219e | ||
|
|
c2a76420cc | ||
|
|
0680ce2e97 | ||
|
|
67dd21edfe | ||
|
|
df3ea169b0 | ||
|
|
fc4fc419cb | ||
|
|
41c577da90 | ||
|
|
db01f12463 | ||
|
|
5f86e2b10f | ||
|
|
22bf5b537b | ||
|
|
c655685acd | ||
|
|
155e93b9d8 | ||
|
|
b71cdf693e | ||
|
|
f68c4134d6 | ||
|
|
c140ca35fb | ||
|
|
c96a366a4e | ||
|
|
042aa2d865 | ||
|
|
ce29ae7ee5 | ||
|
|
b1334308a0 | ||
|
|
9f09039a23 | ||
|
|
dcb541f048 | ||
|
|
55351fcbaa | ||
|
|
19a44e8d9d | ||
|
|
b017102e0e | ||
|
|
30ed711131 | ||
|
|
bd43216cf8 | ||
|
|
738c11e0e3 | ||
|
|
4e916b0f4b | ||
|
|
ca39f422d0 | ||
|
|
b28ca9c0a8 | ||
|
|
69320e750a | ||
|
|
6980d89883 | ||
|
|
54048480fb | ||
|
|
c7fc73bcb6 | ||
|
|
61dc03e9f8 | ||
|
|
a5012fe1af | ||
|
|
8c97106902 | ||
|
|
86553c490d | ||
|
|
a204a91b06 | ||
|
|
12483fe75a | ||
|
|
6d27133ff4 | ||
|
|
f7793f487a | ||
|
|
f2f4a1cf5e | ||
|
|
b984f4bfc7 | ||
|
|
e4bcc0e50e | ||
|
|
2fa114838d | ||
|
|
07b947da01 | ||
|
|
32efb130d7 | ||
|
|
75c032c4f9 | ||
|
|
1372a653cf | ||
|
|
5d58600e44 | ||
|
|
beb0d76136 | ||
|
|
16f8439a3f | ||
|
|
1c662080e5 | ||
|
|
d554e26fa0 | ||
|
|
bc1ec02233 | ||
|
|
e8563a2379 | ||
|
|
30fa8f1284 | ||
|
|
400b34e8cd | ||
|
|
ecfda8617c | ||
|
|
4ba6540714 | ||
|
|
72c72612fd | ||
|
|
ef6c53e7e2 | ||
|
|
9587f9b93b | ||
|
|
17b9cbc982 | ||
|
|
9ed9cf9861 | ||
|
|
92a7972be1 | ||
|
|
5f13c9ffa2 | ||
|
|
bc03a69884 | ||
|
|
7ef6475a53 | ||
|
|
c2027ed102 | ||
|
|
82340f596c | ||
|
|
64ba5e91a5 | ||
|
|
a52b1909ca | ||
|
|
f8aa7e9094 | ||
|
|
1bc24bfbd0 | ||
|
|
5cc0fa3b17 | ||
|
|
9b2d33b6e1 | ||
|
|
93a3369eb6 | ||
|
|
617f8e7d5d | ||
|
|
24952af5c3 | ||
|
|
801b1a0a8e | ||
|
|
17c39b11ba | ||
|
|
b47cdaf217 | ||
|
|
e5c3fed1a9 | ||
|
|
be78295e11 | ||
|
|
485d2d8815 | ||
|
|
75d4f7e101 | ||
|
|
ca1851fb99 | ||
|
|
bf01999222 | ||
|
|
906cc24ba1 | ||
|
|
3de5a8f715 | ||
|
|
e60948b604 | ||
|
|
d08b100e74 | ||
|
|
490cba2c0b | ||
|
|
072b6b4ed4 | ||
|
|
bb0d715eae | ||
|
|
cd6a2f73da | ||
|
|
efa1de0513 | ||
|
|
a1e48c53e3 | ||
|
|
7d9543105d | ||
|
|
66ab68c00c | ||
|
|
ab526ca5c4 | ||
|
|
34063f7fb2 | ||
|
|
5e54aee590 | ||
|
|
0d5c8fc765 | ||
|
|
08cc6dbdca | ||
|
|
c5d64c00a9 | ||
|
|
9121b7d3c5 | ||
|
|
975096b676 | ||
|
|
f6520cf06b | ||
|
|
ff87a6bcff | ||
|
|
5da6e34d5c | ||
|
|
d2ab612754 | ||
|
|
01e0723d65 | ||
|
|
236bfc3786 | ||
|
|
83e90f4f2a | ||
|
|
72b42ccd87 | ||
|
|
2d2f3e22b1 | ||
|
|
cbdfaa69ef | ||
|
|
65104b79ed | ||
|
|
133269e69e | ||
|
|
a080664e1a | ||
|
|
8e37018d78 | ||
|
|
2632974c52 | ||
|
|
7cfd0e3621 | ||
|
|
26df387f0f | ||
|
|
3bdabda6a8 | ||
|
|
26d1f11c1b | ||
|
|
09374d2596 | ||
|
|
c0351b8d76 | ||
|
|
a02a69f2bd | ||
|
|
59f1825016 | ||
|
|
c22db301b6 | ||
|
|
5b975c55a7 | ||
|
|
38b4dc8946 | ||
|
|
787421fa8b | ||
|
|
3225f02bf9 | ||
|
|
73a42021c5 | ||
|
|
65f6e95cce | ||
|
|
3ea5ec74f7 | ||
|
|
f366328a5a | ||
|
|
2137e2710c | ||
|
|
53e4c82ef2 | ||
|
|
32b703cdfa | ||
|
|
6061010e00 | ||
|
|
d3bd2c5fcf | ||
|
|
6c3fb66711 | ||
|
|
22946625d1 | ||
|
|
33e6ace661 | ||
|
|
4097c01d15 | ||
|
|
d338162e7c | ||
|
|
1039a60fec | ||
|
|
9a188b4e06 | ||
|
|
34533041de | ||
|
|
754fe7b111 | ||
|
|
50e9e6feb2 | ||
|
|
629050dd53 | ||
|
|
e7bdc26219 | ||
|
|
a35bddcc8e | ||
|
|
5485fb328e | ||
|
|
42e50f626f | ||
|
|
bcf800e7bf | ||
|
|
ed59373005 | ||
|
|
96120c721e | ||
|
|
b4e1673684 | ||
|
|
3f9e236329 | ||
|
|
544debd055 | ||
|
|
3fb7fe7451 | ||
|
|
dabb9a2fc3 | ||
|
|
46a5e90fd9 | ||
|
|
b4a3eeb130 | ||
|
|
809cffc5f6 | ||
|
|
00d8e9783d | ||
|
|
80860c224d | ||
|
|
3fd9c0a79f | ||
|
|
30842ca21e | ||
|
|
6a5a6e014a | ||
|
|
a35b0e2f10 | ||
|
|
f318dbfdf6 | ||
|
|
6dda19a23f | ||
|
|
3c03fedd0f | ||
|
|
1dc5343113 | ||
|
|
54146d19e1 | ||
|
|
aafcac8a0b | ||
|
|
55e3e3eaef | ||
|
|
8487eee472 | ||
|
|
098eb9f8fe | ||
|
|
ff1f9618a6 | ||
|
|
1371d35f70 | ||
|
|
d6daf9cfb0 | ||
|
|
155279cd7b | ||
|
|
329e9c74e1 | ||
|
|
c83c5d1819 | ||
|
|
7ac827b597 | ||
|
|
f6afdefa9e | ||
|
|
6641150217 | ||
|
|
dee608b093 | ||
|
|
6e2df5578e | ||
|
|
066df492ca | ||
|
|
a9b77e4c98 | ||
|
|
7666c5e8df | ||
|
|
7d239d5d78 | ||
|
|
024e7f41c1 | ||
|
|
5419d044fe | ||
|
|
8583cb3e45 | ||
|
|
f0c9e588ac | ||
|
|
78f319ec4c | ||
|
|
82f86a3ff1 | ||
|
|
cd6bbb6706 | ||
|
|
9fa49565ab | ||
|
|
7c464ebc28 | ||
|
|
7c1dca3ee4 | ||
|
|
acb7e5cc19 | ||
|
|
a375618ef9 | ||
|
|
937abd5215 | ||
|
|
561d9002b7 | ||
|
|
4150aab4a9 | ||
|
|
04d0937779 | ||
|
|
d763fa33c6 | ||
|
|
1a2ca67bd7 | ||
|
|
21f2539ce6 | ||
|
|
66de85112d | ||
|
|
f0db8135e7 | ||
|
|
e2aed2e9e0 | ||
|
|
f75321f4a4 | ||
|
|
5891b49f5d | ||
|
|
f09515c95c | ||
|
|
b8428efe3c | ||
|
|
3b3fa878db | ||
|
|
0631ad67af | ||
|
|
74b776cdc0 | ||
|
|
d9f2f4aa0d | ||
|
|
05d1851d99 | ||
|
|
e09ec84f45 | ||
|
|
2484343ade | ||
|
|
d6b70be5ac | ||
|
|
07022e1276 | ||
|
|
643b6e9c64 | ||
|
|
9af6dfaf1d | ||
|
|
e1b1354a90 | ||
|
|
235d4d3378 | ||
|
|
f76d5cd921 | ||
|
|
c4bf7775a2 | ||
|
|
30cc3d56a2 | ||
|
|
a301cf5c6c | ||
|
|
c2effde235 | ||
|
|
928300f3cd | ||
|
|
dd19fd4332 | ||
|
|
ad3e8d6332 | ||
|
|
de093d73c6 | ||
|
|
c64dab42aa | ||
|
|
de583090dd | ||
|
|
24cdfc37ea | ||
|
|
3944752098 |
@@ -0,0 +1,39 @@
|
||||
========================================================
|
||||
DataX 是阿里云 DataWorks数据集成 的开源版本,在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
|
||||
|
||||
DataX is an open source offline data synchronization tool / platform widely used in Alibaba group and other companies. DataX implements efficient data synchronization between heterogeneous data sources including mysql, Oracle, oceanbase, sqlserver, postgre, HDFS, hive, ads, HBase, tablestore (OTS), maxcompute (ODPs), hologres, DRDS, etc.
|
||||
|
||||
Copyright 1999-2022 Alibaba Group Holding Ltd.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
===================================================================
|
||||
文级别引用,按许可证
|
||||
This product contains various third-party components under other open source licenses.
|
||||
This section summarizes those components and their licenses.
|
||||
GNU Lesser General Public License
|
||||
--------------------------------------
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/CliQuery.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/Connection4TSDB.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DataPoint4TSDB.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DumpSeries.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBConnection.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBDump.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Constant.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Key.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReader.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReaderErrorCode.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/HttpUtils.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TSDBUtils.java
|
||||
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TimeUtils.java
|
||||
===================================================================
|
||||
@@ -1,11 +1,14 @@
|
||||

|
||||
|
||||
|
||||
|
||||
# DataX
|
||||
|
||||
DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台,实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据同步功能。
|
||||
DataX 是阿里云 [DataWorks数据集成](https://www.aliyun.com/product/bigdata/ide) 的开源版本,在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
|
||||
|
||||
# DataX 商业版本
|
||||
阿里云DataWorks数据集成是DataX团队在阿里云上的商业化产品,致力于提供复杂网络环境下、丰富的异构数据源之间高速稳定的数据移动能力,以及繁杂业务背景下的数据同步解决方案。目前已经支持云上近3000家客户,单日同步数据超过3万亿条。DataWorks数据集成目前支持离线50+种数据源,可以进行整库迁移、批量上云、增量同步、分库分表等各类同步解决方案。2020年更新实时同步能力,2020年更新实时同步能力,支持10+种数据源的读写任意组合。提供MySQL,Oracle等多种数据源到阿里云MaxCompute,Hologres等大数据引擎的一键全增量同步解决方案。
|
||||
|
||||
商业版本参见: https://www.aliyun.com/product/bigdata/ide
|
||||
|
||||
|
||||
# Features
|
||||
@@ -22,7 +25,8 @@ DataX本身作为数据同步框架,将不同数据源的同步抽象为从源
|
||||
|
||||
# Quick Start
|
||||
|
||||
##### Download [DataX下载地址](http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz)
|
||||
##### Download [DataX下载地址](https://datax-opensource.oss-cn-hangzhou.aliyuncs.com/20220530/datax.tar.gz)
|
||||
|
||||
|
||||
##### 请点击:[Quick Start](https://github.com/alibaba/DataX/blob/master/userGuid.md)
|
||||
|
||||
@@ -36,6 +40,7 @@ DataX目前已经有了比较全面的插件体系,主流的RDBMS数据库、N
|
||||
| ------------ | ---------- | :-------: | :-------: |:-------: |
|
||||
| RDBMS 关系型数据库 | MySQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md)|
|
||||
| | Oracle | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md)|
|
||||
| | OceanBase | √ | √ |[读](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase) 、[写](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase)|
|
||||
| | SQLServer | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md)|
|
||||
| | PostgreSQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md)|
|
||||
| | DRDS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md)|
|
||||
@@ -43,13 +48,13 @@ DataX目前已经有了比较全面的插件体系,主流的RDBMS数据库、N
|
||||
| 阿里云数仓数据存储 | ODPS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/odpsreader/doc/odpsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/odpswriter/doc/odpswriter.md)|
|
||||
| | ADS | | √ |[写](https://github.com/alibaba/DataX/blob/master/adswriter/doc/adswriter.md)|
|
||||
| | OSS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/ossreader/doc/ossreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/osswriter/doc/osswriter.md)|
|
||||
| | OCS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/ocsreader/doc/ocsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)|
|
||||
| | OCS | | √ |[写](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)|
|
||||
| NoSQL数据存储 | OTS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/otsreader/doc/otsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/otswriter/doc/otswriter.md)|
|
||||
| | Hbase0.94 | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase094xreader/doc/hbase094xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase094xwriter/doc/hbase094xwriter.md)|
|
||||
| | Hbase1.1 | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase11xreader/doc/hbase11xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase11xwriter/doc/hbase11xwriter.md)|
|
||||
| | Phoenix4.x | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase11xsqlreader/doc/hbase11xsqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase11xsqlwriter/doc/hbase11xsqlwriter.md)|
|
||||
| | Phoenix5.x | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase20xsqlreader/doc/hbase20xsqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase20xsqlwriter/doc/hbase20xsqlwriter.md)|
|
||||
| | MongoDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mongoreader/doc/mongoreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mongowriter/doc/mongowriter.md)|
|
||||
| | MongoDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mongodbreader/doc/mongodbreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mongodbwriter/doc/mongodbwriter.md)|
|
||||
| | Hive | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md)|
|
||||
| | Cassandra | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/cassandrareader/doc/cassandrareader.md) 、[写](https://github.com/alibaba/DataX/blob/master/cassandrawriter/doc/cassandrawriter.md)|
|
||||
| 无结构化数据存储 | TxtFile | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/txtfilereader/doc/txtfilereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/txtfilewriter/doc/txtfilewriter.md)|
|
||||
@@ -57,16 +62,48 @@ DataX目前已经有了比较全面的插件体系,主流的RDBMS数据库、N
|
||||
| | HDFS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md)|
|
||||
| | Elasticsearch | | √ |[写](https://github.com/alibaba/DataX/blob/master/elasticsearchwriter/doc/elasticsearchwriter.md)|
|
||||
| 时间序列数据库 | OpenTSDB | √ | |[读](https://github.com/alibaba/DataX/blob/master/opentsdbreader/doc/opentsdbreader.md)|
|
||||
| | TSDB | | √ |[写](https://github.com/alibaba/DataX/blob/master/tsdbwriter/doc/tsdbhttpwriter.md)|
|
||||
| | TSDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tsdbreader/doc/tsdbreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/tsdbwriter/doc/tsdbhttpwriter.md)|
|
||||
| | TDengine | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tdenginereader/doc/tdenginereader-CN.md) 、[写](https://github.com/alibaba/DataX/blob/master/tdenginewriter/doc/tdenginewriter-CN.md)|
|
||||
|
||||
# 阿里云DataWorks数据集成
|
||||
|
||||
目前DataX的已有能力已经全部融和进阿里云的数据集成,并且比DataX更加高效、安全,同时数据集成具备DataX不具备的其它高级特性和功能。可以理解为数据集成是DataX的全面升级的商业化用版本,为企业可以提供稳定、可靠、安全的数据传输服务。与DataX相比,数据集成主要有以下几大突出特点:
|
||||
|
||||
支持实时同步:
|
||||
|
||||
- 功能简介:https://help.aliyun.com/document_detail/181912.html
|
||||
- 支持的数据源:https://help.aliyun.com/document_detail/146778.html
|
||||
- 支持数据处理:https://help.aliyun.com/document_detail/146777.html
|
||||
|
||||
离线同步数据源种类大幅度扩充:
|
||||
|
||||
- 新增比如:DB2、Kafka、Hologres、MetaQ、SAPHANA、达梦等等,持续扩充中
|
||||
- 离线同步支持的数据源:https://help.aliyun.com/document_detail/137670.html
|
||||
- 具备同步解决方案:
|
||||
- 解决方案系统:https://help.aliyun.com/document_detail/171765.html
|
||||
- 一键全增量:https://help.aliyun.com/document_detail/175676.html
|
||||
- 整库迁移:https://help.aliyun.com/document_detail/137809.html
|
||||
- 批量上云:https://help.aliyun.com/document_detail/146671.html
|
||||
- 更新更多能力请访问:https://help.aliyun.com/document_detail/137663.html
|
||||
|
||||
|
||||
# 我要开发新的插件
|
||||
|
||||
请点击:[DataX插件开发宝典](https://github.com/alibaba/DataX/blob/master/dataxPluginDev.md)
|
||||
|
||||
# 重要版本更新说明
|
||||
|
||||
DataX 后续计划月度迭代更新,也欢迎感兴趣的同学提交 Pull requests,月度更新内容会介绍介绍如下。
|
||||
|
||||
- [datax_v202205](https://github.com/alibaba/DataX/releases/tag/datax_v202205)
|
||||
- 涉及通道能力更新(MaxCompute、Hologres、OSS、Tdengine等)、安全漏洞更新、通用打包更新等
|
||||
|
||||
|
||||
# 项目成员
|
||||
|
||||
核心Contributions: 光戈、一斅、祁然、云时
|
||||
核心Contributions: 言柏 、枕水、秋奇、青砾、一斅、云时
|
||||
|
||||
感谢天烬、巴真、静行对DataX做出的贡献。
|
||||
感谢天烬、光戈、祁然、巴真、静行对DataX做出的贡献。
|
||||
|
||||
# License
|
||||
|
||||
@@ -108,7 +145,10 @@ This software is free to use under the Apache License [Apache license](https://g
|
||||
8. 对高并发、高稳定可用性、高性能、大数据处理有过实际项目及产品经验者优先考虑;
|
||||
9. 有大数据产品、云产品、中间件技术解决方案者优先考虑。
|
||||
````
|
||||
钉钉用户群:23169395
|
||||
|
||||
用户咨询支持:
|
||||
|
||||
钉钉群目前暂时受到了一些管控策略影响,建议大家有问题优先在这里提交问题 Issue,DataX研发和社区会定期回答Issue中的问题,知识库丰富后也能帮助到后来的使用者。
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -65,9 +65,9 @@ COPY命令将数据写入ADB PG数据库中。
|
||||
"writer": {
|
||||
"name": "adbpgwriter",
|
||||
"parameter": {
|
||||
"username": "username",
|
||||
"password": "password",
|
||||
"host": "host",
|
||||
"username": "",
|
||||
"password": "",
|
||||
"host": "127.0.0.1",
|
||||
"port": "1234",
|
||||
"database": "database",
|
||||
"schema": "schema",
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<artifactId>clickhousewriter</artifactId>
|
||||
<name>clickhousewriter</name>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>ru.yandex.clickhouse</groupId>
|
||||
<artifactId>clickhouse-jdbc</artifactId>
|
||||
<version>0.2.4</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-core</artifactId>
|
||||
<version>${datax-project-version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<version>${datax-project-version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>plugin-rdbms-util</artifactId>
|
||||
<version>${datax-project-version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<build>
|
||||
<resources>
|
||||
<resource>
|
||||
<directory>src/main/java</directory>
|
||||
<includes>
|
||||
<include>**/*.properties</include>
|
||||
</includes>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<!-- compiler plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>${jdk-version}</source>
|
||||
<target>${jdk-version}</target>
|
||||
<encoding>${project-sourceEncoding}</encoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- assembly plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<configuration>
|
||||
<descriptors>
|
||||
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||
</descriptors>
|
||||
<finalName>datax</finalName>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>dwzip</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>single</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
<assembly
|
||||
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||
<id></id>
|
||||
<formats>
|
||||
<format>dir</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<fileSets>
|
||||
<fileSet>
|
||||
<directory>src/main/resources</directory>
|
||||
<includes>
|
||||
<include>plugin.json</include>
|
||||
<include>plugin_job_template.json</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/writer/clickhousewriter</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>target/</directory>
|
||||
<includes>
|
||||
<include>clickhousewriter-0.0.1-SNAPSHOT.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/writer/clickhousewriter</outputDirectory>
|
||||
</fileSet>
|
||||
</fileSets>
|
||||
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<useProjectArtifact>false</useProjectArtifact>
|
||||
<outputDirectory>plugin/writer/clickhousewriter/libs</outputDirectory>
|
||||
<scope>runtime</scope>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
+329
@@ -0,0 +1,329 @@
|
||||
package com.alibaba.datax.plugin.writer.clickhousewriter;
|
||||
|
||||
import com.alibaba.datax.common.element.Column;
|
||||
import com.alibaba.datax.common.element.StringColumn;
|
||||
import com.alibaba.datax.common.exception.CommonErrorCode;
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||
import com.alibaba.datax.common.spi.Writer;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.rdbms.util.DBUtilErrorCode;
|
||||
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||
import com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter;
|
||||
import com.alibaba.fastjson.JSON;
|
||||
import com.alibaba.fastjson.JSONArray;
|
||||
|
||||
import java.sql.Array;
|
||||
import java.sql.Connection;
|
||||
import java.sql.PreparedStatement;
|
||||
import java.sql.SQLException;
|
||||
import java.sql.Timestamp;
|
||||
import java.sql.Types;
|
||||
import java.util.List;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
public class ClickhouseWriter extends Writer {
|
||||
private static final DataBaseType DATABASE_TYPE = DataBaseType.ClickHouse;
|
||||
|
||||
public static class Job extends Writer.Job {
|
||||
private Configuration originalConfig = null;
|
||||
private CommonRdbmsWriter.Job commonRdbmsWriterMaster;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.originalConfig = super.getPluginJobConf();
|
||||
this.commonRdbmsWriterMaster = new CommonRdbmsWriter.Job(DATABASE_TYPE);
|
||||
this.commonRdbmsWriterMaster.init(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void prepare() {
|
||||
this.commonRdbmsWriterMaster.prepare(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Configuration> split(int mandatoryNumber) {
|
||||
return this.commonRdbmsWriterMaster.split(this.originalConfig, mandatoryNumber);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsWriterMaster.post(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsWriterMaster.destroy(this.originalConfig);
|
||||
}
|
||||
}
|
||||
|
||||
public static class Task extends Writer.Task {
|
||||
private Configuration writerSliceConfig;
|
||||
|
||||
private CommonRdbmsWriter.Task commonRdbmsWriterSlave;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.writerSliceConfig = super.getPluginJobConf();
|
||||
|
||||
this.commonRdbmsWriterSlave = new CommonRdbmsWriter.Task(DATABASE_TYPE) {
|
||||
@Override
|
||||
protected PreparedStatement fillPreparedStatementColumnType(PreparedStatement preparedStatement, int columnIndex, int columnSqltype, Column column) throws SQLException {
|
||||
try {
|
||||
if (column.getRawData() == null) {
|
||||
preparedStatement.setNull(columnIndex + 1, columnSqltype);
|
||||
return preparedStatement;
|
||||
}
|
||||
|
||||
java.util.Date utilDate;
|
||||
switch (columnSqltype) {
|
||||
case Types.CHAR:
|
||||
case Types.NCHAR:
|
||||
case Types.CLOB:
|
||||
case Types.NCLOB:
|
||||
case Types.VARCHAR:
|
||||
case Types.LONGVARCHAR:
|
||||
case Types.NVARCHAR:
|
||||
case Types.LONGNVARCHAR:
|
||||
preparedStatement.setString(columnIndex + 1, column
|
||||
.asString());
|
||||
break;
|
||||
|
||||
case Types.TINYINT:
|
||||
case Types.SMALLINT:
|
||||
case Types.INTEGER:
|
||||
case Types.BIGINT:
|
||||
case Types.DECIMAL:
|
||||
case Types.FLOAT:
|
||||
case Types.REAL:
|
||||
case Types.DOUBLE:
|
||||
String strValue = column.asString();
|
||||
if (emptyAsNull && "".equals(strValue)) {
|
||||
preparedStatement.setNull(columnIndex + 1, columnSqltype);
|
||||
} else {
|
||||
switch (columnSqltype) {
|
||||
case Types.TINYINT:
|
||||
case Types.SMALLINT:
|
||||
case Types.INTEGER:
|
||||
preparedStatement.setInt(columnIndex + 1, column.asBigInteger().intValue());
|
||||
break;
|
||||
case Types.BIGINT:
|
||||
preparedStatement.setLong(columnIndex + 1, column.asLong());
|
||||
break;
|
||||
case Types.DECIMAL:
|
||||
preparedStatement.setBigDecimal(columnIndex + 1, column.asBigDecimal());
|
||||
break;
|
||||
case Types.REAL:
|
||||
case Types.FLOAT:
|
||||
preparedStatement.setFloat(columnIndex + 1, column.asDouble().floatValue());
|
||||
break;
|
||||
case Types.DOUBLE:
|
||||
preparedStatement.setDouble(columnIndex + 1, column.asDouble());
|
||||
break;
|
||||
}
|
||||
}
|
||||
break;
|
||||
|
||||
case Types.DATE:
|
||||
if (this.resultSetMetaData.getRight().get(columnIndex)
|
||||
.equalsIgnoreCase("year")) {
|
||||
if (column.asBigInteger() == null) {
|
||||
preparedStatement.setString(columnIndex + 1, null);
|
||||
} else {
|
||||
preparedStatement.setInt(columnIndex + 1, column.asBigInteger().intValue());
|
||||
}
|
||||
} else {
|
||||
java.sql.Date sqlDate = null;
|
||||
try {
|
||||
utilDate = column.asDate();
|
||||
} catch (DataXException e) {
|
||||
throw new SQLException(String.format(
|
||||
"Date 类型转换错误:[%s]", column));
|
||||
}
|
||||
|
||||
if (null != utilDate) {
|
||||
sqlDate = new java.sql.Date(utilDate.getTime());
|
||||
}
|
||||
preparedStatement.setDate(columnIndex + 1, sqlDate);
|
||||
}
|
||||
break;
|
||||
|
||||
case Types.TIME:
|
||||
java.sql.Time sqlTime = null;
|
||||
try {
|
||||
utilDate = column.asDate();
|
||||
} catch (DataXException e) {
|
||||
throw new SQLException(String.format(
|
||||
"Date 类型转换错误:[%s]", column));
|
||||
}
|
||||
|
||||
if (null != utilDate) {
|
||||
sqlTime = new java.sql.Time(utilDate.getTime());
|
||||
}
|
||||
preparedStatement.setTime(columnIndex + 1, sqlTime);
|
||||
break;
|
||||
|
||||
case Types.TIMESTAMP:
|
||||
Timestamp sqlTimestamp = null;
|
||||
if (column instanceof StringColumn && column.asString() != null) {
|
||||
String timeStampStr = column.asString();
|
||||
// JAVA TIMESTAMP 类型入参必须是 "2017-07-12 14:39:00.123566" 格式
|
||||
String pattern = "^\\d+-\\d+-\\d+ \\d+:\\d+:\\d+.\\d+";
|
||||
boolean isMatch = Pattern.matches(pattern, timeStampStr);
|
||||
if (isMatch) {
|
||||
sqlTimestamp = Timestamp.valueOf(timeStampStr);
|
||||
preparedStatement.setTimestamp(columnIndex + 1, sqlTimestamp);
|
||||
break;
|
||||
}
|
||||
}
|
||||
try {
|
||||
utilDate = column.asDate();
|
||||
} catch (DataXException e) {
|
||||
throw new SQLException(String.format(
|
||||
"Date 类型转换错误:[%s]", column));
|
||||
}
|
||||
|
||||
if (null != utilDate) {
|
||||
sqlTimestamp = new Timestamp(
|
||||
utilDate.getTime());
|
||||
}
|
||||
preparedStatement.setTimestamp(columnIndex + 1, sqlTimestamp);
|
||||
break;
|
||||
|
||||
case Types.BINARY:
|
||||
case Types.VARBINARY:
|
||||
case Types.BLOB:
|
||||
case Types.LONGVARBINARY:
|
||||
preparedStatement.setBytes(columnIndex + 1, column
|
||||
.asBytes());
|
||||
break;
|
||||
|
||||
case Types.BOOLEAN:
|
||||
preparedStatement.setInt(columnIndex + 1, column.asBigInteger().intValue());
|
||||
break;
|
||||
|
||||
// warn: bit(1) -> Types.BIT 可使用setBoolean
|
||||
// warn: bit(>1) -> Types.VARBINARY 可使用setBytes
|
||||
case Types.BIT:
|
||||
if (this.dataBaseType == DataBaseType.MySql) {
|
||||
Boolean asBoolean = column.asBoolean();
|
||||
if (asBoolean != null) {
|
||||
preparedStatement.setBoolean(columnIndex + 1, asBoolean);
|
||||
} else {
|
||||
preparedStatement.setNull(columnIndex + 1, Types.BIT);
|
||||
}
|
||||
} else {
|
||||
preparedStatement.setString(columnIndex + 1, column.asString());
|
||||
}
|
||||
break;
|
||||
|
||||
default:
|
||||
boolean isHandled = fillPreparedStatementColumnType4CustomType(preparedStatement,
|
||||
columnIndex, columnSqltype, column);
|
||||
if (isHandled) {
|
||||
break;
|
||||
}
|
||||
throw DataXException
|
||||
.asDataXException(
|
||||
DBUtilErrorCode.UNSUPPORTED_TYPE,
|
||||
String.format(
|
||||
"您的配置文件中的列配置信息有误. 因为DataX 不支持数据库写入这种字段类型. 字段名:[%s], 字段类型:[%d], 字段Java类型:[%s]. 请修改表中该字段的类型或者不同步该字段.",
|
||||
this.resultSetMetaData.getLeft()
|
||||
.get(columnIndex),
|
||||
this.resultSetMetaData.getMiddle()
|
||||
.get(columnIndex),
|
||||
this.resultSetMetaData.getRight()
|
||||
.get(columnIndex)));
|
||||
}
|
||||
return preparedStatement;
|
||||
} catch (DataXException e) {
|
||||
// fix类型转换或者溢出失败时,将具体哪一列打印出来
|
||||
if (e.getErrorCode() == CommonErrorCode.CONVERT_NOT_SUPPORT ||
|
||||
e.getErrorCode() == CommonErrorCode.CONVERT_OVER_FLOW) {
|
||||
throw DataXException
|
||||
.asDataXException(
|
||||
e.getErrorCode(),
|
||||
String.format(
|
||||
"类型转化错误. 字段名:[%s], 字段类型:[%d], 字段Java类型:[%s]. 请修改表中该字段的类型或者不同步该字段.",
|
||||
this.resultSetMetaData.getLeft()
|
||||
.get(columnIndex),
|
||||
this.resultSetMetaData.getMiddle()
|
||||
.get(columnIndex),
|
||||
this.resultSetMetaData.getRight()
|
||||
.get(columnIndex)));
|
||||
} else {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private Object toJavaArray(Object val) {
|
||||
if (null == val) {
|
||||
return null;
|
||||
} else if (val instanceof JSONArray) {
|
||||
Object[] valArray = ((JSONArray) val).toArray();
|
||||
for (int i = 0; i < valArray.length; i++) {
|
||||
valArray[i] = this.toJavaArray(valArray[i]);
|
||||
}
|
||||
return valArray;
|
||||
} else {
|
||||
return val;
|
||||
}
|
||||
}
|
||||
|
||||
boolean fillPreparedStatementColumnType4CustomType(PreparedStatement ps,
|
||||
int columnIndex, int columnSqltype,
|
||||
Column column) throws SQLException {
|
||||
switch (columnSqltype) {
|
||||
case Types.OTHER:
|
||||
if (this.resultSetMetaData.getRight().get(columnIndex).startsWith("Tuple")) {
|
||||
throw DataXException
|
||||
.asDataXException(ClickhouseWriterErrorCode.TUPLE_NOT_SUPPORTED_ERROR, ClickhouseWriterErrorCode.TUPLE_NOT_SUPPORTED_ERROR.getDescription());
|
||||
} else {
|
||||
ps.setString(columnIndex + 1, column.asString());
|
||||
}
|
||||
return true;
|
||||
|
||||
case Types.ARRAY:
|
||||
Connection conn = ps.getConnection();
|
||||
List<Object> values = JSON.parseArray(column.asString(), Object.class);
|
||||
for (int i = 0; i < values.size(); i++) {
|
||||
values.set(i, this.toJavaArray(values.get(i)));
|
||||
}
|
||||
Array array = conn.createArrayOf("String", values.toArray());
|
||||
ps.setArray(columnIndex + 1, array);
|
||||
return true;
|
||||
|
||||
default:
|
||||
break;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
this.commonRdbmsWriterSlave.init(this.writerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void prepare() {
|
||||
this.commonRdbmsWriterSlave.prepare(this.writerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startWrite(RecordReceiver recordReceiver) {
|
||||
this.commonRdbmsWriterSlave.startWrite(recordReceiver, this.writerSliceConfig, super.getTaskPluginCollector());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsWriterSlave.post(this.writerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsWriterSlave.destroy(this.writerSliceConfig);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package com.alibaba.datax.plugin.writer.clickhousewriter;
|
||||
|
||||
import com.alibaba.datax.common.spi.ErrorCode;
|
||||
|
||||
public enum ClickhouseWriterErrorCode implements ErrorCode {
|
||||
TUPLE_NOT_SUPPORTED_ERROR("ClickhouseWriter-00", "不支持TUPLE类型导入."),
|
||||
;
|
||||
|
||||
private final String code;
|
||||
private final String description;
|
||||
|
||||
private ClickhouseWriterErrorCode(String code, String description) {
|
||||
this.code = code;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getCode() {
|
||||
return this.code;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getDescription() {
|
||||
return this.description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s].", this.code, this.description);
|
||||
}
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "clickhousewriter",
|
||||
"class": "com.alibaba.datax.plugin.writer.clickhousewriter.ClickhouseWriter",
|
||||
"description": "useScene: prod. mechanism: Jdbc connection using the database, execute insert sql.",
|
||||
"developer": "jiye.tjy"
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "clickhousewriter",
|
||||
"parameter": {
|
||||
"username": "username",
|
||||
"password": "password",
|
||||
"column": ["col1", "col2", "col3"],
|
||||
"connection": [
|
||||
{
|
||||
"jdbcUrl": "jdbc:clickhouse://<host>:<port>[/<database>]",
|
||||
"table": ["table1", "table2"]
|
||||
}
|
||||
],
|
||||
"preSql": [],
|
||||
"postSql": [],
|
||||
|
||||
"batchSize": 65536,
|
||||
"batchByteSize": 134217728,
|
||||
"dryRun": false,
|
||||
"writeMode": "insert"
|
||||
}
|
||||
}
|
||||
@@ -61,6 +61,14 @@
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<resources>
|
||||
<resource>
|
||||
<directory>src/main/java</directory>
|
||||
<includes>
|
||||
<include>**/*.properties</include>
|
||||
</includes>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
|
||||
@@ -92,7 +92,13 @@ public class BoolColumn extends Column {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
throw DataXException.asDataXException(
|
||||
|
||||
@@ -75,6 +75,12 @@ public class BytesColumn extends Column {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Boolean asBoolean() {
|
||||
|
||||
@@ -55,6 +55,8 @@ public abstract class Column {
|
||||
public abstract String asString();
|
||||
|
||||
public abstract Date asDate();
|
||||
|
||||
public abstract Date asDate(String dateFormat);
|
||||
|
||||
public abstract byte[] asBytes();
|
||||
|
||||
|
||||
@@ -22,6 +22,11 @@ public final class ColumnCast {
|
||||
throws ParseException {
|
||||
return StringCast.asDate(column);
|
||||
}
|
||||
|
||||
public static Date string2Date(final StringColumn column, String dateFormat)
|
||||
throws ParseException {
|
||||
return StringCast.asDate(column, dateFormat);
|
||||
}
|
||||
|
||||
public static byte[] string2Bytes(final StringColumn column)
|
||||
throws UnsupportedEncodingException {
|
||||
@@ -115,6 +120,16 @@ class StringCast {
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
static Date asDate(final StringColumn column, String dateFormat) throws ParseException {
|
||||
ParseException e;
|
||||
try {
|
||||
return FastDateFormat.getInstance(dateFormat, StringCast.timeZoner).parse(column.asString());
|
||||
} catch (ParseException ignored) {
|
||||
e = ignored;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
static byte[] asBytes(final StringColumn column)
|
||||
throws UnsupportedEncodingException {
|
||||
|
||||
@@ -89,6 +89,11 @@ public class DateColumn extends Column {
|
||||
|
||||
return new Date((Long)this.getRawData());
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
return asDate();
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
|
||||
@@ -132,6 +132,12 @@ public class DoubleColumn extends Column {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
throw DataXException.asDataXException(
|
||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
|
||||
@@ -125,6 +125,11 @@ public class LongColumn extends Column {
|
||||
}
|
||||
return new Date(this.asLong());
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
return this.asDate();
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
package com.alibaba.datax.common.element;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Created by jingxing on 14-8-24.
|
||||
*/
|
||||
@@ -20,4 +22,8 @@ public interface Record {
|
||||
|
||||
public int getMemorySize();
|
||||
|
||||
public void setMeta(Map<String, String> meta);
|
||||
|
||||
public Map<String, String> getMeta();
|
||||
|
||||
}
|
||||
|
||||
@@ -149,6 +149,16 @@ public class StringColumn extends Column {
|
||||
String.format("String[\"%s\"]不能转为Date .", this.asString()));
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
try {
|
||||
return ColumnCast.string2Date(this, dateFormat);
|
||||
} catch (Exception e) {
|
||||
throw DataXException.asDataXException(CommonErrorCode.CONVERT_NOT_SUPPORT,
|
||||
String.format("String[\"%s\"]不能转为Date .", this.asString()));
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
|
||||
@@ -16,6 +16,10 @@ public class DataXException extends RuntimeException {
|
||||
this.errorCode = errorCode;
|
||||
}
|
||||
|
||||
public DataXException(String errorMessage) {
|
||||
super(errorMessage);
|
||||
}
|
||||
|
||||
private DataXException(ErrorCode errorCode, String errorMessage, Throwable cause) {
|
||||
super(errorCode.toString() + " - " + getMessage(errorMessage) + " - " + getMessage(cause), cause);
|
||||
|
||||
@@ -26,6 +30,10 @@ public class DataXException extends RuntimeException {
|
||||
return new DataXException(errorCode, message);
|
||||
}
|
||||
|
||||
public static DataXException asDataXException(String message) {
|
||||
return new DataXException(message);
|
||||
}
|
||||
|
||||
public static DataXException asDataXException(ErrorCode errorCode, String message, Throwable cause) {
|
||||
if (cause instanceof DataXException) {
|
||||
return (DataXException) cause;
|
||||
|
||||
@@ -3,6 +3,8 @@ package com.alibaba.datax.common.plugin;
|
||||
import com.alibaba.datax.common.base.BaseObject;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
||||
//作业的config
|
||||
private Configuration pluginJobConf;
|
||||
@@ -15,6 +17,8 @@ public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
||||
|
||||
private String peerPluginName;
|
||||
|
||||
private List<Configuration> readerPluginSplitConf;
|
||||
|
||||
@Override
|
||||
public String getPluginName() {
|
||||
assert null != this.pluginConf;
|
||||
@@ -84,4 +88,12 @@ public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
||||
public void postHandler(Configuration jobConfiguration){
|
||||
|
||||
}
|
||||
|
||||
public List<Configuration> getReaderPluginSplitConf(){
|
||||
return this.readerPluginSplitConf;
|
||||
}
|
||||
|
||||
public void setReaderPluginSplitConf(List<Configuration> readerPluginSplitConf){
|
||||
this.readerPluginSplitConf = readerPluginSplitConf;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
package com.alibaba.datax.common.util;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
public class ConfigurationUtil {
|
||||
private static final List<String> SENSITIVE_KEYS = Arrays.asList("password", "accessKey", "securityToken",
|
||||
"AccessKeyId", "AccessKeySecert", "AccessKeySecret", "clientPassword");
|
||||
|
||||
public static Configuration filterSensitive(Configuration origin) {
|
||||
// shell 任务configuration metric 可能为null。
|
||||
if (origin == null) {
|
||||
return origin;
|
||||
}
|
||||
// 确保不影响入参的对象
|
||||
Configuration configuration = origin.clone();
|
||||
Set<String> keys = configuration.getKeys();
|
||||
for (final String key : keys) {
|
||||
boolean isSensitive = false;
|
||||
for (String sensitiveKey : SENSITIVE_KEYS) {
|
||||
if (StringUtils.endsWithIgnoreCase(key, sensitiveKey)) {
|
||||
isSensitive = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (isSensitive && configuration.get(key) instanceof String) {
|
||||
configuration.set(key, configuration.getString(key).replaceAll(".", "*"));
|
||||
}
|
||||
|
||||
}
|
||||
return configuration;
|
||||
}
|
||||
}
|
||||
Regular → Executable
+86
-212
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* (C) 2010-2014 Alibaba Group Holding Limited.
|
||||
* (C) 2010-2022 Alibaba Group Holding Limited.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
@@ -14,342 +14,216 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.alibaba.datax.plugin.reader.odpsreader.util;
|
||||
package com.alibaba.datax.common.util;
|
||||
|
||||
import javax.crypto.Cipher;
|
||||
import javax.crypto.SecretKey;
|
||||
import javax.crypto.SecretKeyFactory;
|
||||
import javax.crypto.spec.DESKeySpec;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.security.SecureRandom;
|
||||
|
||||
/**
|
||||
* * DES加解密,支持与delphi交互(字符串编码需统一为UTF-8)
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @author wym
|
||||
*
|
||||
*
|
||||
* DES加解密,支持与delphi交互(字符串编码需统一为UTF-8)
|
||||
* 将这个工具类抽取到 common 中,方便后续代码复用
|
||||
*/
|
||||
|
||||
public class DESCipher {
|
||||
|
||||
private static Logger LOGGER = LoggerFactory.getLogger(DESCipher.class);
|
||||
/**
|
||||
* * 密钥
|
||||
*
|
||||
*
|
||||
* 密钥
|
||||
*/
|
||||
|
||||
public static final String KEY = "u4Gqu4Z8";
|
||||
|
||||
public static final String KEY = "";
|
||||
private final static String DES = "DES";
|
||||
|
||||
/**
|
||||
* * 加密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 明文(字节)
|
||||
*
|
||||
* * @param key
|
||||
*
|
||||
* * 密钥,长度必须是8的倍数
|
||||
*
|
||||
* * @return 密文(字节)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 加密
|
||||
* @param src 明文(字节)
|
||||
* @param key 密钥,长度必须是8的倍数
|
||||
* @return 密文(字节)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public static byte[] encrypt(byte[] src, byte[] key) throws Exception {
|
||||
|
||||
// DES算法要求有一个可信任的随机数源
|
||||
|
||||
SecureRandom sr = new SecureRandom();
|
||||
|
||||
|
||||
// 从原始密匙数据创建DESKeySpec对象
|
||||
|
||||
DESKeySpec dks = new DESKeySpec(key);
|
||||
|
||||
|
||||
// 创建一个密匙工厂,然后用它把DESKeySpec转换成
|
||||
|
||||
// 一个SecretKey对象
|
||||
|
||||
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
||||
|
||||
SecretKey securekey = keyFactory.generateSecret(dks);
|
||||
|
||||
|
||||
// Cipher对象实际完成加密操作
|
||||
|
||||
Cipher cipher = Cipher.getInstance(DES);
|
||||
|
||||
// 用密匙初始化Cipher对象
|
||||
|
||||
cipher.init(Cipher.ENCRYPT_MODE, securekey, sr);
|
||||
|
||||
// 现在,获取数据并加密
|
||||
|
||||
// 正式执行加密操作
|
||||
|
||||
return cipher.doFinal(src);
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 解密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 密文(字节)
|
||||
*
|
||||
* * @param key
|
||||
*
|
||||
* * 密钥,长度必须是8的倍数
|
||||
*
|
||||
* * @return 明文(字节)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* * 解密
|
||||
* * @param src
|
||||
* * 密文(字节)
|
||||
* * @param key
|
||||
* * 密钥,长度必须是8的倍数
|
||||
* * @return 明文(字节)
|
||||
* * @throws Exception
|
||||
*/
|
||||
|
||||
public static byte[] decrypt(byte[] src, byte[] key) throws Exception {
|
||||
|
||||
// DES算法要求有一个可信任的随机数源
|
||||
|
||||
SecureRandom sr = new SecureRandom();
|
||||
|
||||
// 从原始密匙数据创建一个DESKeySpec对象
|
||||
|
||||
DESKeySpec dks = new DESKeySpec(key);
|
||||
|
||||
// 创建一个密匙工厂,然后用它把DESKeySpec对象转换成
|
||||
|
||||
// 一个SecretKey对象
|
||||
|
||||
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
||||
|
||||
SecretKey securekey = keyFactory.generateSecret(dks);
|
||||
|
||||
// Cipher对象实际完成解密操作
|
||||
|
||||
Cipher cipher = Cipher.getInstance(DES);
|
||||
|
||||
// 用密匙初始化Cipher对象
|
||||
|
||||
cipher.init(Cipher.DECRYPT_MODE, securekey, sr);
|
||||
|
||||
// 现在,获取数据并解密
|
||||
|
||||
// 正式执行解密操作
|
||||
|
||||
return cipher.doFinal(src);
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 加密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 明文(字节)
|
||||
*
|
||||
* * @return 密文(字节)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 加密
|
||||
* @param src * 明文(字节)
|
||||
* @return 密文(字节)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public static byte[] encrypt(byte[] src) throws Exception {
|
||||
|
||||
return encrypt(src, KEY.getBytes());
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 解密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 密文(字节)
|
||||
*
|
||||
* * @return 明文(字节)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 解密
|
||||
* @param src 密文(字节)
|
||||
* @return 明文(字节)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public static byte[] decrypt(byte[] src) throws Exception {
|
||||
|
||||
return decrypt(src, KEY.getBytes());
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 加密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 明文(字符串)
|
||||
*
|
||||
* * @return 密文(16进制字符串)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 加密
|
||||
* @param src 明文(字符串)
|
||||
* @return 密文(16进制字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public final static String encrypt(String src) {
|
||||
|
||||
try {
|
||||
|
||||
return byte2hex(encrypt(src.getBytes(), KEY.getBytes()));
|
||||
|
||||
} catch (Exception e) {
|
||||
|
||||
e.printStackTrace();
|
||||
|
||||
LOGGER.warn(e.getMessage(), e);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 加密
|
||||
* @param src 明文(字符串)
|
||||
* @param encryptKey 加密用的秘钥
|
||||
* @return 密文(16进制字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
public final static String encrypt(String src, String encryptKey) {
|
||||
try {
|
||||
return byte2hex(encrypt(src.getBytes(), encryptKey.getBytes()));
|
||||
} catch (Exception e) {
|
||||
LOGGER.warn(e.getMessage(), e);
|
||||
}
|
||||
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 解密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 密文(字符串)
|
||||
*
|
||||
* * @return 明文(字符串)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 解密
|
||||
* @param src 密文(字符串)
|
||||
* @return 明文(字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public final static String decrypt(String src) {
|
||||
try {
|
||||
|
||||
return new String(decrypt(hex2byte(src.getBytes()), KEY.getBytes()));
|
||||
|
||||
} catch (Exception e) {
|
||||
|
||||
e.printStackTrace();
|
||||
|
||||
LOGGER.warn(e.getMessage(), e);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* 解密
|
||||
* @param src 密文(字符串)
|
||||
* @param decryptKey 解密用的秘钥
|
||||
* @return 明文(字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
public final static String decrypt(String src, String decryptKey) {
|
||||
try {
|
||||
return new String(decrypt(hex2byte(src.getBytes()), decryptKey.getBytes()));
|
||||
} catch (Exception e) {
|
||||
LOGGER.warn(e.getMessage(), e);
|
||||
}
|
||||
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 加密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 明文(字节)
|
||||
*
|
||||
* * @return 密文(16进制字符串)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 加密
|
||||
* @param src
|
||||
* 明文(字节)
|
||||
* @return 密文(16进制字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public static String encryptToString(byte[] src) throws Exception {
|
||||
|
||||
return encrypt(new String(src));
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* * 解密
|
||||
*
|
||||
* *
|
||||
*
|
||||
* * @param src
|
||||
*
|
||||
* * 密文(字节)
|
||||
*
|
||||
* * @return 明文(字符串)
|
||||
*
|
||||
* * @throws Exception
|
||||
*
|
||||
*
|
||||
* 解密
|
||||
* @param src 密文(字节)
|
||||
* @return 明文(字符串)
|
||||
* @throws Exception
|
||||
*/
|
||||
|
||||
public static String decryptToString(byte[] src) throws Exception {
|
||||
|
||||
return decrypt(new String(src));
|
||||
|
||||
}
|
||||
|
||||
public static String byte2hex(byte[] b) {
|
||||
|
||||
String hs = "";
|
||||
|
||||
String stmp = "";
|
||||
|
||||
for (int n = 0; n < b.length; n++) {
|
||||
|
||||
stmp = (Integer.toHexString(b[n] & 0XFF));
|
||||
|
||||
if (stmp.length() == 1)
|
||||
|
||||
hs = hs + "0" + stmp;
|
||||
|
||||
else
|
||||
|
||||
hs = hs + stmp;
|
||||
|
||||
}
|
||||
|
||||
return hs.toUpperCase();
|
||||
|
||||
}
|
||||
|
||||
public static byte[] hex2byte(byte[] b) {
|
||||
|
||||
if ((b.length % 2) != 0)
|
||||
|
||||
throw new IllegalArgumentException("长度不是偶数");
|
||||
|
||||
throw new IllegalArgumentException("The length is not an even number");
|
||||
byte[] b2 = new byte[b.length / 2];
|
||||
|
||||
for (int n = 0; n < b.length; n += 2) {
|
||||
|
||||
String item = new String(b, n, 2);
|
||||
|
||||
b2[n / 2] = (byte) Integer.parseInt(item, 16);
|
||||
|
||||
}
|
||||
return b2;
|
||||
|
||||
}
|
||||
|
||||
/*
|
||||
* public static void main(String[] args) { try { String src = "cheetah";
|
||||
* String crypto = DESCipher.encrypt(src); System.out.println("密文[" + src +
|
||||
* "]:" + crypto); System.out.println("解密后:" + DESCipher.decrypt(crypto)); }
|
||||
* catch (Exception e) { e.printStackTrace(); } }
|
||||
*/
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package com.alibaba.datax.common.util;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
||||
public class DataXCaseEnvUtil {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(DataXCaseEnvUtil.class);
|
||||
|
||||
// datax回归测试效率提升
|
||||
private static String DATAX_AUTOTEST_RETRY_TIME = System.getenv("DATAX_AUTOTEST_RETRY_TIME");
|
||||
private static String DATAX_AUTOTEST_RETRY_INTERVAL = System.getenv("DATAX_AUTOTEST_RETRY_INTERVAL");
|
||||
private static String DATAX_AUTOTEST_RETRY_EXPONENTIAL = System.getenv("DATAX_AUTOTEST_RETRY_EXPONENTIAL");
|
||||
|
||||
public static int getRetryTimes(int retryTimes) {
|
||||
int actualRetryTimes = DATAX_AUTOTEST_RETRY_TIME != null ? Integer.valueOf(DATAX_AUTOTEST_RETRY_TIME) : retryTimes;
|
||||
// LOGGER.info("The actualRetryTimes is {}", actualRetryTimes);
|
||||
return actualRetryTimes;
|
||||
}
|
||||
|
||||
public static long getRetryInterval(long retryInterval) {
|
||||
long actualRetryInterval = DATAX_AUTOTEST_RETRY_INTERVAL != null ? Long.valueOf(DATAX_AUTOTEST_RETRY_INTERVAL) : retryInterval;
|
||||
// LOGGER.info("The actualRetryInterval is {}", actualRetryInterval);
|
||||
return actualRetryInterval;
|
||||
}
|
||||
|
||||
public static boolean getRetryExponential(boolean retryExponential) {
|
||||
boolean actualRetryExponential = DATAX_AUTOTEST_RETRY_EXPONENTIAL != null ? Boolean.valueOf(DATAX_AUTOTEST_RETRY_EXPONENTIAL) : retryExponential;
|
||||
// LOGGER.info("The actualRetryExponential is {}", actualRetryExponential);
|
||||
return actualRetryExponential;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package com.alibaba.datax.common.util;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
|
||||
public class IdAndKeyRollingUtil {
|
||||
private static Logger LOGGER = LoggerFactory.getLogger(IdAndKeyRollingUtil.class);
|
||||
public static final String SKYNET_ACCESSID = "SKYNET_ACCESSID";
|
||||
public static final String SKYNET_ACCESSKEY = "SKYNET_ACCESSKEY";
|
||||
|
||||
public final static String ACCESS_ID = "accessId";
|
||||
public final static String ACCESS_KEY = "accessKey";
|
||||
|
||||
public static String parseAkFromSkynetAccessKey() {
|
||||
Map<String, String> envProp = System.getenv();
|
||||
String skynetAccessID = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSID);
|
||||
String skynetAccessKey = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSKEY);
|
||||
String accessKey = null;
|
||||
// follow 原有的判断条件
|
||||
// 环境变量中,如果存在SKYNET_ACCESSID/SKYNET_ACCESSKEy(只要有其中一个变量,则认为一定是两个都存在的!
|
||||
// if (StringUtils.isNotBlank(skynetAccessID) ||
|
||||
// StringUtils.isNotBlank(skynetAccessKey)) {
|
||||
// 检查严格,只有加密串不为空的时候才进去,不过 之前能跑的加密串都不应该为空
|
||||
if (StringUtils.isNotBlank(skynetAccessKey)) {
|
||||
LOGGER.info("Try to get accessId/accessKey from environment SKYNET_ACCESSKEY.");
|
||||
accessKey = DESCipher.decrypt(skynetAccessKey);
|
||||
if (StringUtils.isBlank(accessKey)) {
|
||||
// 环境变量里面有,但是解析不到
|
||||
throw DataXException.asDataXException(String.format(
|
||||
"Failed to get the [accessId]/[accessKey] from the environment variable. The [accessId]=[%s]",
|
||||
skynetAccessID));
|
||||
}
|
||||
}
|
||||
if (StringUtils.isNotBlank(accessKey)) {
|
||||
LOGGER.info("Get accessId/accessKey from environment variables SKYNET_ACCESSKEY successfully.");
|
||||
}
|
||||
return accessKey;
|
||||
}
|
||||
|
||||
public static String getAccessIdAndKeyFromEnv(Configuration originalConfig) {
|
||||
String accessId = null;
|
||||
Map<String, String> envProp = System.getenv();
|
||||
accessId = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSID);
|
||||
String accessKey = null;
|
||||
if (StringUtils.isBlank(accessKey)) {
|
||||
// 老的没有出异常,只是获取不到ak
|
||||
accessKey = IdAndKeyRollingUtil.parseAkFromSkynetAccessKey();
|
||||
}
|
||||
|
||||
if (StringUtils.isNotBlank(accessKey)) {
|
||||
// 确认使用这个的都是 accessId、accessKey的命名习惯
|
||||
originalConfig.set(IdAndKeyRollingUtil.ACCESS_ID, accessId);
|
||||
originalConfig.set(IdAndKeyRollingUtil.ACCESS_KEY, accessKey);
|
||||
}
|
||||
return accessKey;
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
@@ -136,4 +137,25 @@ public final class ListUtil {
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
public static Boolean checkIfHasSameValue(List<String> listA, List<String> listB) {
|
||||
if (null == listA || listA.isEmpty() || null == listB || listB.isEmpty()) {
|
||||
return false;
|
||||
}
|
||||
|
||||
for (String oneValue : listA) {
|
||||
if (listB.contains(oneValue)) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
public static boolean checkIfAllSameValue(List<String> listA, List<String> listB) {
|
||||
if (null == listA || listA.isEmpty() || null == listB || listB.isEmpty() || listA.size() != listB.size()) {
|
||||
return false;
|
||||
}
|
||||
return new HashSet<>(listA).containsAll(new HashSet<>(listB));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
|
||||
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
|
||||
|
||||
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
|
||||
|
||||
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
|
||||
|
||||
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1}, STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
@@ -0,0 +1,53 @@
|
||||
very_like_yixiao=1{0}2{1}3
|
||||
|
||||
|
||||
configuration.1=Configuration information error. The configuration file [{0}] you provided does not exist. Please check your configuration files.
|
||||
configuration.2=Configuration information error. Failed to read the configuration file [{0}] you provided. Error reason: {1}. Please check the permission settings of your configuration files.
|
||||
configuration.3=Please check your configuration files. Failed to read the configuration file you provided. Error reason: {0}. Please check the permission settings of your configuration files.
|
||||
configuration.4=The configuration file you provided contains errors. [{0}] is a required parameter and cannot be empty or blank.
|
||||
configuration.5=The configuration file you provided contains errors. [{0}] is a required parameter and cannot be empty or blank.
|
||||
configuration.6=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the character type: {1}. Please check your configuration and make corrections.
|
||||
configuration.7=The configuration information you provided contains errors. The value [{1}] obtained from [{0}] cannot be converted to the Bool type. Please check the source table configuration and make corrections.
|
||||
configuration.8=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the integer type: {1}. Please check your configuration and make corrections.
|
||||
configuration.9=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the integer type: {1}. Please check your configuration and make corrections.
|
||||
configuration.10=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the floating-point type: {1}. Please check your configuration and make corrections.
|
||||
configuration.11=The Key [{0}] for the configuration file does not exist. This is a code programming error. Please contact the DataX team.
|
||||
configuration.12=The value [{0}] cannot adapt to the [{1}] you provided. This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.13=The path cannot be null. This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.14=The path [{0}] has an invalid value type [{1}]. This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.15=This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.16=The configuration file you provided contains errors. The path [{0}] requires you to configure a Map object in JSON format, but the actual type found on the node is [{1}]. Please check your configuration and make corrections.
|
||||
configuration.17=The configuration file you provided contains errors. The value of the path [{0}] is null and DataX cannot recognize the configuration. Please check your configuration and make corrections.
|
||||
configuration.18=The configuration file you provided contains errors. The path [{0}] requires you to configure a Map object in JSON format, but the actual type found on the node is [{1}]. Please check your configuration and make corrections.
|
||||
configuration.19=System programming error. The list subscript must be of the numeric type, but the actual type found on this node is [{0}]. This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.20=System programming error. This exception represents a system programming error. Please contact the DataX developer team.
|
||||
configuration.21=System programming error. Invalid path [{0}]. No spaces are allowed between path layers.
|
||||
configuration.22=Configuration information error. The configuration information you provided is not in a legal JSON format. JSON cannot be blank. Please provide the configuration information in the standard JSON format.
|
||||
configuration.23=Configuration information error. The configuration information you provided is not in a valid JSON format: {0}. Please provide the configuration information in the standard JSON format.
|
||||
|
||||
|
||||
listutil.1=The job configuration you provided contains errors. The list cannot be empty.
|
||||
listutil.2=The job configuration you provided contains errors. The list cannot be empty.
|
||||
listutil.3=The job configuration information you provided contains errors. String: [{0}] is not allowed to be repeated in the list: [{1}].
|
||||
listutil.4=The job configuration you provided contains errors. The list cannot be empty.
|
||||
listutil.5=The job configuration you provided contains errors. The list cannot be empty.
|
||||
listutil.6=The job configuration information you provided contains errors. String: [{0}] does not exist in the list: [{1}].
|
||||
listutil.7=The job configuration you provided contains errors. The list cannot be empty.
|
||||
listutil.8=The job configuration you provided contains errors. The list cannot be empty.
|
||||
|
||||
|
||||
rangesplitutil.1=The slice number cannot be less than 1. Here: [expectSliceNumber]=[{0}].
|
||||
rangesplitutil.2=The left or right intervals of BigInteger character strings cannot be null when they are sliced. Here: [left]=[{0}], [right]=[{1}].
|
||||
rangesplitutil.3=The [bigInteger] parameter cannot be null.
|
||||
rangesplitutil.4=Only ASCII character strings are supported for character string slicing, but the [{0}] character string is not of the ASCII type.
|
||||
rangesplitutil.5=The [bigInteger] parameter cannot be null.
|
||||
rangesplitutil.6=Only ASCII character strings are supported for character string slicing, but the [{0}] character string is not of the ASCII type.
|
||||
|
||||
|
||||
retryutil.1=System programming error. The “callable” input parameter cannot be null.
|
||||
retryutil.2=System programming error. The “retrytime[%d]” input parameter cannot be less than 1.
|
||||
retryutil.3=Exception when calling callable. Exception Msg: {0}
|
||||
retryutil.4=Exception when calling callable. Retry Attempt: {0} will start soon. {1} attempts in total. This attempt planned to wait for [{2}]ms, and actually waited for [{3}]ms. Exception Msg: [{4}].
|
||||
|
||||
httpclientutil.1=Request address: {0}. Request method: {1}. STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=The remote interface returns -1. We will try again
|
||||
@@ -0,0 +1,53 @@
|
||||
very_like_yixiao=1{0}2{1}3
|
||||
|
||||
|
||||
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
|
||||
|
||||
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
|
||||
|
||||
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
|
||||
|
||||
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
@@ -0,0 +1,54 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
|
||||
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
|
||||
|
||||
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
|
||||
|
||||
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
|
||||
|
||||
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
@@ -0,0 +1,104 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
|
||||
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
|
||||
|
||||
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
|
||||
|
||||
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
|
||||
|
||||
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||
|
||||
very_like_yixiao=一{0}二{1}三
|
||||
|
||||
|
||||
configuration.1=配置資訊錯誤,您提供的配置檔案[{0}]不存在. 請檢查您的配置檔案.
|
||||
configuration.2=配置資訊錯誤. 您提供配置檔案[{0}]讀取失敗,錯誤原因: {1}. 請檢查您的配置檔案的權限設定.
|
||||
configuration.3=請檢查您的配置檔案. 您提供的配置檔案讀取失敗,錯誤原因: {0}. 請檢查您的配置檔案的權限設定.
|
||||
configuration.4=您提供配置檔案有誤,[{0}]是必填參數,不允許為空或者留白 .
|
||||
configuration.5=您提供配置檔案有誤,[{0}]是必填參數,不允許為空或者留白 .
|
||||
configuration.6=任務讀取配置檔案出錯. 因為配置檔案路徑[{0}] 值不合法,期望是字符類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.7=您提供的配置資訊有誤,因為從[{0}]獲取的值[{1}]無法轉換為bool類型. 請檢查源表的配置並且做出相應的修改.
|
||||
configuration.8=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.9=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.10=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是浮點類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.11=配置檔案對應Key[{0}]並不存在,該情況是代碼編程錯誤. 請聯絡DataX團隊的同學.
|
||||
configuration.12=值[{0}]無法適配您提供[{1}], 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!
|
||||
configuration.13=Path不能為null,該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.14=路徑[{0}]出現不合法值類型[{1}],該異常代表系統編程錯誤, 請聯絡DataX開發團隊! .
|
||||
configuration.15=該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.16=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象,但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||
configuration.17=您提供的配置檔案有誤. 路徑[{0}]值為null,datax無法識別該配置. 請檢查您的配置並作出修改.
|
||||
configuration.18=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象,但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||
configuration.19=系統編程錯誤,清單下標必須為數字類型,但該節點發現實際類型是[{0}] ,該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.20=系統編程錯誤, 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!.
|
||||
configuration.21=系統編程錯誤, 路徑[{0}]不合法, 路徑層次之間不能出現空白字符 .
|
||||
configuration.22=配置資訊錯誤. 因為您提供的配置資訊不是合法的JSON格式, JSON不能為空白. 請按照標準json格式提供配置資訊.
|
||||
configuration.23=配置資訊錯誤. 您提供的配置資訊不是合法的JSON格式: {0}. 請按照標準json格式提供配置資訊.
|
||||
|
||||
|
||||
listutil.1=您提供的作業配置有誤,List不能為空.
|
||||
listutil.2=您提供的作業配置有誤, List不能為空.
|
||||
listutil.3=您提供的作業配置資訊有誤, String:[{0}]不允許重複出現在清單中: [{1}].
|
||||
listutil.4=您提供的作業配置有誤, List不能為空.
|
||||
listutil.5=您提供的作業配置有誤, List不能為空.
|
||||
listutil.6=您提供的作業配置資訊有誤, String:[{0}]不存在於清單中:[{1}].
|
||||
listutil.7=您提供的作業配置有誤, List不能為空.
|
||||
listutil.8=您提供的作業配置有誤, List不能為空.
|
||||
|
||||
|
||||
rangesplitutil.1=切分份數不能小於1. 此處:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=對 BigInteger 進行切分時,其左右區間不能為 null. 此處:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=參數 bigInteger 不能為空.
|
||||
rangesplitutil.4=根據字符串進行切分時僅支援 ASCII 字符串,而字符串:[{0}]非 ASCII 字符串.
|
||||
rangesplitutil.5=參數 bigInteger 不能為空.
|
||||
rangesplitutil.6=根據字符串進行切分時僅支援 ASCII 字符串,而字符串:[{0}]非 ASCII 字符串.
|
||||
|
||||
|
||||
retryutil.1=系統編程錯誤, 入參callable不能為空 !
|
||||
retryutil.2=系統編程錯誤, 入參retrytime[%d]不能小於1 !
|
||||
retryutil.3=Exception when calling callable, 異常Msg:{0}
|
||||
retryutil.4=Exception when calling callable, 即將嘗試執行第{0}次重試,共計重試{1}次.本次重試計劃等待[{2}]ms,實際等待[{3}]ms, 異常Msg:[{4}]
|
||||
|
||||
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||
@@ -0,0 +1,104 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
|
||||
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||
|
||||
|
||||
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||
|
||||
|
||||
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||
|
||||
|
||||
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||
|
||||
very_like_yixiao=一{0}二{1}三
|
||||
|
||||
|
||||
configuration.1=配置資訊錯誤,您提供的配置檔案[{0}]不存在. 請檢查您的配置檔案.
|
||||
configuration.2=配置資訊錯誤. 您提供配置檔案[{0}]讀取失敗,錯誤原因: {1}. 請檢查您的配置檔案的權限設定.
|
||||
configuration.3=請檢查您的配置檔案. 您提供的配置檔案讀取失敗,錯誤原因: {0}. 請檢查您的配置檔案的權限設定.
|
||||
configuration.4=您提供配置檔案有誤,[{0}]是必填參數,不允許為空或者留白 .
|
||||
configuration.5=您提供配置檔案有誤,[{0}]是必填參數,不允許為空或者留白 .
|
||||
configuration.6=任務讀取配置檔案出錯. 因為配置檔案路徑[{0}] 值不合法,期望是字符類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.7=您提供的配置資訊有誤,因為從[{0}]獲取的值[{1}]無法轉換為bool類型. 請檢查源表的配置並且做出相應的修改.
|
||||
configuration.8=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.9=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.10=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是浮點類型: {1}. 請檢查您的配置並作出修改.
|
||||
configuration.11=配置檔案對應Key[{0}]並不存在,該情況是代碼編程錯誤. 請聯絡DataX團隊的同學.
|
||||
configuration.12=值[{0}]無法適配您提供[{1}], 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!
|
||||
configuration.13=Path不能為null,該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.14=路徑[{0}]出現不合法值類型[{1}],該異常代表系統編程錯誤, 請聯絡DataX開發團隊! .
|
||||
configuration.15=該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.16=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象,但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||
configuration.17=您提供的配置檔案有誤. 路徑[{0}]值為null,datax無法識別該配置. 請檢查您的配置並作出修改.
|
||||
configuration.18=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象,但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||
configuration.19=系統編程錯誤,清單下標必須為數字類型,但該節點發現實際類型是[{0}] ,該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||
configuration.20=系統編程錯誤, 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!.
|
||||
configuration.21=系統編程錯誤, 路徑[{0}]不合法, 路徑層次之間不能出現空白字符 .
|
||||
configuration.22=配置資訊錯誤. 因為您提供的配置資訊不是合法的JSON格式, JSON不能為空白. 請按照標準json格式提供配置資訊.
|
||||
configuration.23=配置資訊錯誤. 您提供的配置資訊不是合法的JSON格式: {0}. 請按照標準json格式提供配置資訊.
|
||||
|
||||
|
||||
listutil.1=您提供的作業配置有誤,List不能為空.
|
||||
listutil.2=您提供的作業配置有誤, List不能為空.
|
||||
listutil.3=您提供的作業配置資訊有誤, String:[{0}]不允許重複出現在清單中: [{1}].
|
||||
listutil.4=您提供的作業配置有誤, List不能為空.
|
||||
listutil.5=您提供的作業配置有誤, List不能為空.
|
||||
listutil.6=您提供的作業配置資訊有誤, String:[{0}]不存在於清單中:[{1}].
|
||||
listutil.7=您提供的作業配置有誤, List不能為空.
|
||||
listutil.8=您提供的作業配置有誤, List不能為空.
|
||||
|
||||
|
||||
rangesplitutil.1=切分份數不能小於1. 此處:expectSliceNumber=[{0}].
|
||||
rangesplitutil.2=對 BigInteger 進行切分時,其左右區間不能為 null. 此處:left=[{0}],right=[{1}].
|
||||
rangesplitutil.3=參數 bigInteger 不能為空.
|
||||
rangesplitutil.4=根據字符串進行切分時僅支援 ASCII 字符串,而字符串:[{0}]非 ASCII 字符串.
|
||||
rangesplitutil.5=參數 bigInteger 不能為空.
|
||||
rangesplitutil.6=根據字符串進行切分時僅支援 ASCII 字符串,而字符串:[{0}]非 ASCII 字符串.
|
||||
|
||||
|
||||
retryutil.1=系統編程錯誤, 入參callable不能為空 !
|
||||
retryutil.2=系統編程錯誤, 入參retrytime[%d]不能小於1 !
|
||||
retryutil.3=Exception when calling callable, 異常Msg:{0}
|
||||
retryutil.4=Exception when calling callable, 即將嘗試執行第{0}次重試,共計重試{1}次.本次重試計劃等待[{2}]ms,實際等待[{3}]ms, 異常Msg:[{4}]
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
@@ -0,0 +1,207 @@
|
||||
package com.alibaba.datax.common.util;
|
||||
|
||||
import java.text.MessageFormat;
|
||||
import java.util.HashMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.MissingResourceException;
|
||||
import java.util.ResourceBundle;
|
||||
import java.util.TimeZone;
|
||||
|
||||
import org.apache.commons.lang3.LocaleUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
|
||||
public class MessageSource {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(MessageSource.class);
|
||||
private static Map<String, ResourceBundle> resourceBundleCache = new HashMap<String, ResourceBundle>();
|
||||
public static Locale locale = null;
|
||||
public static TimeZone timeZone = null;
|
||||
private ResourceBundle resourceBundle = null;
|
||||
|
||||
private MessageSource(ResourceBundle resourceBundle) {
|
||||
this.resourceBundle = resourceBundle;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param baseName
|
||||
* demo: javax.servlet.http.LocalStrings
|
||||
*
|
||||
* @throws MissingResourceException
|
||||
* - if no resource bundle for the specified base name can be
|
||||
* found
|
||||
* */
|
||||
public static MessageSource loadResourceBundle(String baseName) {
|
||||
return loadResourceBundle(baseName, MessageSource.locale,
|
||||
MessageSource.timeZone);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param clazz
|
||||
* 根据其获取package name
|
||||
* */
|
||||
public static <T> MessageSource loadResourceBundle(Class<T> clazz) {
|
||||
return loadResourceBundle(clazz.getPackage().getName());
|
||||
}
|
||||
|
||||
/**
|
||||
* @param clazz
|
||||
* 根据其获取package name
|
||||
* */
|
||||
public static <T> MessageSource loadResourceBundle(Class<T> clazz,
|
||||
Locale locale, TimeZone timeZone) {
|
||||
return loadResourceBundle(clazz.getPackage().getName(), locale,
|
||||
timeZone);
|
||||
}
|
||||
|
||||
/**
|
||||
* warn:
|
||||
* ok: ResourceBundle.getBundle("xxx.LocalStrings", Locale.getDefault(), LoadUtil.getJarLoader(PluginType.WRITER, "odpswriter"))
|
||||
* error: ResourceBundle.getBundle("xxx.LocalStrings", Locale.getDefault(), LoadUtil.getJarLoader(PluginType.WRITER, "odpswriter"))
|
||||
* @param baseName
|
||||
* demo: javax.servlet.http.LocalStrings
|
||||
*
|
||||
* @throws MissingResourceException
|
||||
* - if no resource bundle for the specified base name can be
|
||||
* found
|
||||
*
|
||||
* */
|
||||
public static MessageSource loadResourceBundle(String baseName,
|
||||
Locale locale, TimeZone timeZone) {
|
||||
ResourceBundle resourceBundle = null;
|
||||
if (null == locale) {
|
||||
locale = LocaleUtils.toLocale("en_US");
|
||||
}
|
||||
if (null == timeZone) {
|
||||
timeZone = TimeZone.getDefault();
|
||||
}
|
||||
String resourceBaseName = String.format("%s.LocalStrings", baseName);
|
||||
LOG.debug(
|
||||
"initEnvironment MessageSource.locale[{}], MessageSource.timeZone[{}]",
|
||||
MessageSource.locale, MessageSource.timeZone);
|
||||
LOG.debug(
|
||||
"loadResourceBundle with locale[{}], timeZone[{}], baseName[{}]",
|
||||
locale, timeZone, resourceBaseName);
|
||||
// warn: 这个map的维护需要考虑Local吗, no?
|
||||
if (!MessageSource.resourceBundleCache.containsKey(resourceBaseName)) {
|
||||
ClassLoader clazzLoader = Thread.currentThread()
|
||||
.getContextClassLoader();
|
||||
LOG.debug("loadResourceBundle classLoader:{}", clazzLoader);
|
||||
resourceBundle = ResourceBundle.getBundle(resourceBaseName, locale,
|
||||
clazzLoader);
|
||||
MessageSource.resourceBundleCache.put(resourceBaseName,
|
||||
resourceBundle);
|
||||
} else {
|
||||
resourceBundle = MessageSource.resourceBundleCache
|
||||
.get(resourceBaseName);
|
||||
}
|
||||
|
||||
return new MessageSource(resourceBundle);
|
||||
}
|
||||
|
||||
public static <T> boolean unloadResourceBundle(Class<T> clazz) {
|
||||
String baseName = clazz.getPackage().getName();
|
||||
String resourceBaseName = String.format("%s.LocalStrings", baseName);
|
||||
if (!MessageSource.resourceBundleCache.containsKey(resourceBaseName)) {
|
||||
return false;
|
||||
} else {
|
||||
MessageSource.resourceBundleCache.remove(resourceBaseName);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
public static <T> MessageSource reloadResourceBundle(Class<T> clazz) {
|
||||
MessageSource.unloadResourceBundle(clazz);
|
||||
return MessageSource.loadResourceBundle(clazz);
|
||||
}
|
||||
|
||||
public static void setEnvironment(Locale locale, TimeZone timeZone) {
|
||||
// warn: 设置默认? @2018.03.21 将此处注释移除,否则在国际化多时区下会遇到问题
|
||||
Locale.setDefault(locale);
|
||||
TimeZone.setDefault(timeZone);
|
||||
MessageSource.locale = locale;
|
||||
MessageSource.timeZone = timeZone;
|
||||
LOG.info("use Locale: {} timeZone: {}", locale, timeZone);
|
||||
}
|
||||
|
||||
public static void init(final Configuration configuration) {
|
||||
Locale locale2Set = Locale.getDefault();
|
||||
String localeStr = configuration.getString("common.column.locale", "zh_CN");// 默认操作系统的
|
||||
if (StringUtils.isNotBlank(localeStr)) {
|
||||
try {
|
||||
locale2Set = LocaleUtils.toLocale(localeStr);
|
||||
} catch (Exception e) {
|
||||
LOG.warn("ignored locale parse exception: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
TimeZone timeZone2Set = TimeZone.getDefault();
|
||||
String timeZoneStr = configuration.getString("common.column.timeZone");// 默认操作系统的
|
||||
if (StringUtils.isNotBlank(timeZoneStr)) {
|
||||
try {
|
||||
timeZone2Set = TimeZone.getTimeZone(timeZoneStr);
|
||||
} catch (Exception e) {
|
||||
LOG.warn("ignored timezone parse exception: {}", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
LOG.info("JVM TimeZone: {}, Locale: {}", timeZone2Set.getID(), locale2Set);
|
||||
MessageSource.setEnvironment(locale2Set, timeZone2Set);
|
||||
}
|
||||
|
||||
public static void clearCache() {
|
||||
MessageSource.resourceBundleCache.clear();
|
||||
}
|
||||
|
||||
public String message(String code) {
|
||||
return this.messageWithDefaultMessage(code, null);
|
||||
}
|
||||
|
||||
public String message(String code, String args1) {
|
||||
return this.messageWithDefaultMessage(code, null,
|
||||
new Object[] { args1 });
|
||||
}
|
||||
|
||||
public String message(String code, String args1, String args2) {
|
||||
return this.messageWithDefaultMessage(code, null, new Object[] { args1,
|
||||
args2 });
|
||||
}
|
||||
|
||||
public String message(String code, String args1, String args2, String args3) {
|
||||
return this.messageWithDefaultMessage(code, null, new Object[] { args1,
|
||||
args2, args3 });
|
||||
}
|
||||
|
||||
// 上面几个重载可以应对大多数情况, 避免使用这个可以提高性能的
|
||||
public String message(String code, Object... args) {
|
||||
return this.messageWithDefaultMessage(code, null, args);
|
||||
}
|
||||
|
||||
public String messageWithDefaultMessage(String code, String defaultMessage) {
|
||||
return this.messageWithDefaultMessage(code, defaultMessage,
|
||||
new Object[] {});
|
||||
}
|
||||
|
||||
/**
|
||||
* @param args
|
||||
* MessageFormat会依次调用对应对象的toString方法
|
||||
* */
|
||||
public String messageWithDefaultMessage(String code, String defaultMessage,
|
||||
Object... args) {
|
||||
String messageStr = null;
|
||||
try {
|
||||
messageStr = this.resourceBundle.getString(code);
|
||||
} catch (MissingResourceException e) {
|
||||
messageStr = defaultMessage;
|
||||
}
|
||||
if (null != messageStr && null != args && args.length > 0) {
|
||||
// warn: see loadResourceBundle set default locale
|
||||
return MessageFormat.format(messageStr, args);
|
||||
} else {
|
||||
return messageStr;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
@@ -206,4 +206,27 @@ public final class RangeSplitUtil {
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* List拆分工具函数,主要用于reader插件的split拆分逻辑
|
||||
* */
|
||||
public static <T> List<List<T>> doListSplit(List<T> objects, int adviceNumber) {
|
||||
List<List<T>> splitLists = new ArrayList<List<T>>();
|
||||
if (null == objects) {
|
||||
return splitLists;
|
||||
}
|
||||
long[] splitPoint = RangeSplitUtil.doLongSplit(0, objects.size(), adviceNumber);
|
||||
for (int startIndex = 0; startIndex < splitPoint.length - 1; startIndex++) {
|
||||
List<T> objectsForTask = new ArrayList<T>();
|
||||
int endIndex = startIndex + 1;
|
||||
for (long i = splitPoint[startIndex]; i < splitPoint[endIndex]; i++) {
|
||||
objectsForTask.add(objects.get((int) i));
|
||||
}
|
||||
if (!objectsForTask.isEmpty()) {
|
||||
splitLists.add(objectsForTask);
|
||||
}
|
||||
}
|
||||
return splitLists;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
+10
-2
@@ -41,12 +41,12 @@
|
||||
<dependency>
|
||||
<groupId>org.apache.httpcomponents</groupId>
|
||||
<artifactId>httpclient</artifactId>
|
||||
<version>4.4</version>
|
||||
<version>4.5</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.httpcomponents</groupId>
|
||||
<artifactId>fluent-hc</artifactId>
|
||||
<version>4.4</version>
|
||||
<version>4.5</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
@@ -100,6 +100,14 @@
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<resources>
|
||||
<resource>
|
||||
<directory>src/main/java</directory>
|
||||
<includes>
|
||||
<include>**/*.properties</include>
|
||||
</includes>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
|
||||
+64
-50
@@ -1,23 +1,26 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding:utf-8 -*-
|
||||
|
||||
import sys
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
import re
|
||||
import socket
|
||||
import json
|
||||
from optparse import OptionParser
|
||||
from optparse import OptionGroup
|
||||
from string import Template
|
||||
import codecs
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from optparse import OptionGroup
|
||||
from optparse import OptionParser
|
||||
from string import Template
|
||||
|
||||
ispy2 = sys.version_info.major == 2
|
||||
|
||||
def isWindows():
|
||||
return platform.system() == 'Windows'
|
||||
|
||||
|
||||
DATAX_HOME = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
DATAX_VERSION = 'DATAX-OPENSOURCE-3.0'
|
||||
@@ -52,13 +55,19 @@ def getLocalIp():
|
||||
|
||||
def suicide(signum, e):
|
||||
global child_process
|
||||
print >> sys.stderr, "[Error] DataX receive unexpected signal %d, starts to suicide." % (signum)
|
||||
if ispy2:
|
||||
print >> sys.stderr, "[Error] DataX receive unexpected signal %d, starts to suicide." % (signum)
|
||||
else:
|
||||
print("[Error] DataX receive unexpected signal %d, starts to suicide." % (signum), sys.stderr)
|
||||
|
||||
if child_process:
|
||||
child_process.send_signal(signal.SIGQUIT)
|
||||
time.sleep(1)
|
||||
child_process.kill()
|
||||
print >> sys.stderr, "DataX Process was killed ! you did ?"
|
||||
if ispy2:
|
||||
print >> sys.stderr, "DataX Process was killed ! you did ?"
|
||||
else:
|
||||
print("DataX Process was killed ! you did ?", sys.stderr)
|
||||
sys.exit(RET_STATE["KILL"])
|
||||
|
||||
|
||||
@@ -92,10 +101,10 @@ def getOptionParser():
|
||||
'if you have mutiple parameters: -p"-DtableName=your-table-name -DcolumnName=your-column-name".'
|
||||
'Note: you should config in you job tableName with ${tableName}.')
|
||||
prodEnvOptionGroup.add_option("-r", "--reader", metavar="<parameter used in view job config[reader] template>",
|
||||
action="store", dest="reader",type="string",
|
||||
action="store", dest="reader", type="string",
|
||||
help='View job config[reader] template, eg: mysqlreader,streamreader')
|
||||
prodEnvOptionGroup.add_option("-w", "--writer", metavar="<parameter used in view job config[writer] template>",
|
||||
action="store", dest="writer",type="string",
|
||||
action="store", dest="writer", type="string",
|
||||
help='View job config[writer] template, eg: mysqlwriter,streamwriter')
|
||||
parser.add_option_group(prodEnvOptionGroup)
|
||||
|
||||
@@ -108,45 +117,50 @@ def getOptionParser():
|
||||
parser.add_option_group(devEnvOptionGroup)
|
||||
return parser
|
||||
|
||||
|
||||
def generateJobConfigTemplate(reader, writer):
|
||||
readerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n" % (reader,reader,reader)
|
||||
writerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n " % (writer,writer,writer)
|
||||
print readerRef
|
||||
print writerRef
|
||||
readerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n" % (
|
||||
reader, reader, reader)
|
||||
writerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n " % (
|
||||
writer, writer, writer)
|
||||
print(readerRef)
|
||||
print(writerRef)
|
||||
jobGuid = 'Please save the following configuration as a json file and use\n python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json \nto run the job.\n'
|
||||
print jobGuid
|
||||
jobTemplate={
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": ""
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {},
|
||||
"writer": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
print(jobGuid)
|
||||
jobTemplate = {
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": ""
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {},
|
||||
"writer": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
readerTemplatePath = "%s/plugin/reader/%s/plugin_job_template.json" % (DATAX_HOME,reader)
|
||||
writerTemplatePath = "%s/plugin/writer/%s/plugin_job_template.json" % (DATAX_HOME,writer)
|
||||
readerTemplatePath = "%s/plugin/reader/%s/plugin_job_template.json" % (DATAX_HOME, reader)
|
||||
writerTemplatePath = "%s/plugin/writer/%s/plugin_job_template.json" % (DATAX_HOME, writer)
|
||||
try:
|
||||
readerPar = readPluginTemplate(readerTemplatePath);
|
||||
except Exception, e:
|
||||
print "Read reader[%s] template error: can\'t find file %s" % (reader,readerTemplatePath)
|
||||
readerPar = readPluginTemplate(readerTemplatePath)
|
||||
except:
|
||||
print("Read reader[%s] template error: can\'t find file %s" % (reader, readerTemplatePath))
|
||||
try:
|
||||
writerPar = readPluginTemplate(writerTemplatePath);
|
||||
except Exception, e:
|
||||
print "Read writer[%s] template error: : can\'t find file %s" % (writer,writerTemplatePath)
|
||||
jobTemplate['job']['content'][0]['reader'] = readerPar;
|
||||
jobTemplate['job']['content'][0]['writer'] = writerPar;
|
||||
print json.dumps(jobTemplate, indent=4, sort_keys=True)
|
||||
writerPar = readPluginTemplate(writerTemplatePath)
|
||||
except:
|
||||
print("Read writer[%s] template error: : can\'t find file %s" % (writer, writerTemplatePath))
|
||||
jobTemplate['job']['content'][0]['reader'] = readerPar
|
||||
jobTemplate['job']['content'][0]['writer'] = writerPar
|
||||
print(json.dumps(jobTemplate, indent=4, sort_keys=True))
|
||||
|
||||
|
||||
def readPluginTemplate(plugin):
|
||||
with open(plugin, 'r') as f:
|
||||
return json.load(f)
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def isUrl(path):
|
||||
if not path:
|
||||
@@ -168,7 +182,7 @@ def buildStartCommand(options, args):
|
||||
|
||||
if options.remoteDebug:
|
||||
tempJVMCommand = tempJVMCommand + " " + REMOTE_DEBUG_CONFIG
|
||||
print 'local ip: ', getLocalIp()
|
||||
print('local ip: ', getLocalIp())
|
||||
|
||||
if options.loglevel:
|
||||
tempJVMCommand = tempJVMCommand + " " + ("-Dloglevel=%s" % (options.loglevel))
|
||||
@@ -198,11 +212,11 @@ def buildStartCommand(options, args):
|
||||
|
||||
|
||||
def printCopyright():
|
||||
print '''
|
||||
print('''
|
||||
DataX (%s), From Alibaba !
|
||||
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
|
||||
|
||||
''' % DATAX_VERSION
|
||||
''' % DATAX_VERSION)
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
@@ -211,7 +225,7 @@ if __name__ == "__main__":
|
||||
parser = getOptionParser()
|
||||
options, args = parser.parse_args(sys.argv[1:])
|
||||
if options.reader is not None and options.writer is not None:
|
||||
generateJobConfigTemplate(options.reader,options.writer)
|
||||
generateJobConfigTemplate(options.reader, options.writer)
|
||||
sys.exit(RET_STATE['OK'])
|
||||
if len(args) != 1:
|
||||
parser.print_help()
|
||||
|
||||
@@ -174,6 +174,9 @@ def parsePluginName(jdbcUrl, pluginType):
|
||||
db2Regex = re.compile('jdbc:(db2)://.*')
|
||||
if (db2Regex.match(jdbcUrl)):
|
||||
name = 'db2'
|
||||
kingbaseesRegex = re.compile('jdbc:(kingbase8)://.*')
|
||||
if (kingbaseesRegex.match(jdbcUrl)):
|
||||
name = 'kingbasees'
|
||||
return "%s%s" % (name, pluginType)
|
||||
|
||||
def renderDataXJson(paramsDict, readerOrWriter = 'reader', channel = 1):
|
||||
|
||||
@@ -6,6 +6,7 @@ import com.alibaba.datax.common.spi.ErrorCode;
|
||||
import com.alibaba.datax.common.statistics.PerfTrace;
|
||||
import com.alibaba.datax.common.statistics.VMInfo;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.common.util.MessageSource;
|
||||
import com.alibaba.datax.core.job.JobContainer;
|
||||
import com.alibaba.datax.core.taskgroup.TaskGroupContainer;
|
||||
import com.alibaba.datax.core.util.ConfigParser;
|
||||
@@ -73,7 +74,7 @@ public class Engine {
|
||||
boolean traceEnable = allConf.getBool(CoreConstant.DATAX_CORE_CONTAINER_TRACE_ENABLE, true);
|
||||
boolean perfReportEnable = allConf.getBool(CoreConstant.DATAX_CORE_REPORT_DATAX_PERFLOG, true);
|
||||
|
||||
//standlone模式的datax shell任务不进行汇报
|
||||
//standalone模式的 datax shell任务不进行汇报
|
||||
if(instanceId == -1){
|
||||
perfReportEnable = false;
|
||||
}
|
||||
@@ -135,6 +136,9 @@ public class Engine {
|
||||
RUNTIME_MODE = cl.getOptionValue("mode");
|
||||
|
||||
Configuration configuration = ConfigParser.parse(jobPath);
|
||||
// 绑定i18n信息
|
||||
MessageSource.init(configuration);
|
||||
MessageSource.reloadResourceBundle(Configuration.class);
|
||||
|
||||
long jobId;
|
||||
if (!"-1".equalsIgnoreCase(jobIdString)) {
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
very_like_yixiao=1{0}2{1}3
|
||||
|
||||
engine.1=A valid job ID must be provided in the URL for the non-standalone mode.
|
||||
engine.2=\n\nThrough the intelligent analysis by DataX, the most likely error reason of this task is: \n{0}
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
very_like_yixiao=1{0}2{1}3
|
||||
|
||||
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||
|
||||
very_like_yixiao=一{0}二{1}三
|
||||
|
||||
engine.1=非 standalone 模式必須在 URL 中提供有效的 jobId.
|
||||
engine.2=\n\n經DataX智能分析,該任務最可能的錯誤原因是:\n{0}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||
|
||||
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||
|
||||
very_like_yixiao=一{0}二{1}三
|
||||
|
||||
engine.1=非 standalone 模式必須在 URL 中提供有效的 jobId.
|
||||
engine.2=\n\n經DataX智能分析,該任務最可能的錯誤原因是:\n{0}
|
||||
|
||||
@@ -427,7 +427,7 @@ public class JobContainer extends AbstractContainer {
|
||||
Long channelLimitedByteSpeed = this.configuration
|
||||
.getLong(CoreConstant.DATAX_CORE_TRANSPORT_CHANNEL_SPEED_BYTE);
|
||||
if (channelLimitedByteSpeed == null || channelLimitedByteSpeed <= 0) {
|
||||
DataXException.asDataXException(
|
||||
throw DataXException.asDataXException(
|
||||
FrameworkErrorCode.CONFIG_ERROR,
|
||||
"在有总bps限速条件下,单个channel的bps值不能为空,也不能为非正数");
|
||||
}
|
||||
@@ -448,7 +448,7 @@ public class JobContainer extends AbstractContainer {
|
||||
Long channelLimitedRecordSpeed = this.configuration.getLong(
|
||||
CoreConstant.DATAX_CORE_TRANSPORT_CHANNEL_SPEED_RECORD);
|
||||
if (channelLimitedRecordSpeed == null || channelLimitedRecordSpeed <= 0) {
|
||||
DataXException.asDataXException(FrameworkErrorCode.CONFIG_ERROR,
|
||||
throw DataXException.asDataXException(FrameworkErrorCode.CONFIG_ERROR,
|
||||
"在有总tps限速条件下,单个channel的tps值不能为空,也不能为非正数");
|
||||
}
|
||||
|
||||
|
||||
+19
@@ -11,15 +11,18 @@ import java.math.BigInteger;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Date;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
public class DirtyRecord implements Record {
|
||||
private List<Column> columns = new ArrayList<Column>();
|
||||
private Map<String, String> meta;
|
||||
|
||||
public static DirtyRecord asDirtyRecord(final Record record) {
|
||||
DirtyRecord result = new DirtyRecord();
|
||||
for (int i = 0; i < record.getColumnNumber(); i++) {
|
||||
result.addColumn(record.getColumn(i));
|
||||
}
|
||||
result.setMeta(record.getMeta());
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -65,6 +68,16 @@ public class DirtyRecord implements Record {
|
||||
"该方法不支持!");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMeta(Map<String, String> meta) {
|
||||
this.meta = meta;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, String> getMeta() {
|
||||
return this.meta;
|
||||
}
|
||||
|
||||
public List<Column> getColumns() {
|
||||
return columns;
|
||||
}
|
||||
@@ -119,6 +132,12 @@ class DirtyColumn extends Column {
|
||||
throw DataXException.asDataXException(FrameworkErrorCode.RUNTIME_ERROR,
|
||||
"该方法不支持!");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Date asDate(String dateFormat) {
|
||||
throw DataXException.asDataXException(FrameworkErrorCode.RUNTIME_ERROR,
|
||||
"该方法不支持!");
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] asBytes() {
|
||||
|
||||
@@ -27,6 +27,8 @@ public class DefaultRecord implements Record {
|
||||
// 首先是Record本身需要的内存
|
||||
private int memorySize = ClassSize.DefaultRecordHead;
|
||||
|
||||
private Map<String, String> meta;
|
||||
|
||||
public DefaultRecord() {
|
||||
this.columns = new ArrayList<Column>(RECORD_AVERGAE_COLUMN_NUMBER);
|
||||
}
|
||||
@@ -83,6 +85,16 @@ public class DefaultRecord implements Record {
|
||||
return memorySize;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMeta(Map<String, String> meta) {
|
||||
this.meta = meta;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, String> getMeta() {
|
||||
return this.meta;
|
||||
}
|
||||
|
||||
private void decrByteSize(final Column column) {
|
||||
if (null == column) {
|
||||
return;
|
||||
|
||||
@@ -3,6 +3,8 @@ package com.alibaba.datax.core.transport.record;
|
||||
import com.alibaba.datax.common.element.Column;
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 作为标示 生产者已经完成生产的标志
|
||||
*
|
||||
@@ -41,6 +43,16 @@ public class TerminateRecord implements Record {
|
||||
return 0;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMeta(Map<String, String> meta) {
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, String> getMeta() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setColumn(int i, Column column) {
|
||||
return;
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||
|
||||
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1}, STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
configparser.1=Failed to load the plug-in [{0},{1}]. We will retry in 1s... Exception: {2}
|
||||
configparser.2=Failed to obtain the job configuration information: {0}
|
||||
configparser.3=Failed to obtain the job configuration information: {0}
|
||||
configparser.4=Failed to obtain the job configuration information: {0}
|
||||
configparser.5=Failed to load the plug-in. Loading of the specific plug-in:{0} is not completed
|
||||
configparser.6=Failed to load the plug-in. A duplicate plug-in: {0} exists
|
||||
|
||||
dataxserviceutil.1=Exception in creating signature. NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=Exception in creating signature. InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=Exception in creating signature. UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=The percentage of dirty data should be limited to within [0.0, 1.0]
|
||||
errorrecordchecker.2=The number of dirty data entries should now be a nonnegative integer
|
||||
errorrecordchecker.3=Check for the number of dirty data entries has not passed. The limit is [{0}] entries, but [{1}] entries have been captured.
|
||||
errorrecordchecker.4=Check for the percentage of dirty data has not passed. The limit is [{0}], but [{1}] of dirty data has been captured.
|
||||
|
||||
|
||||
errorcode.install_error=Error in installing DataX engine. Please contact your O&M team to solve the problem.
|
||||
errorcode.argument_error=Error in running DataX engine. This problem is generally caused by an internal programming error. Please contact the DataX developer team to solve the problem.
|
||||
errorcode.runtime_error=The DataX engine encountered an error during running. For the specific cause, refer to the error diagnosis after DataX stops running.
|
||||
errorcode.config_error=Error in DataX engine configuration. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||
errorcode.secret_error=Error in DataX engine encryption or decryption. This problem is generally caused by a DataX key configuration error. Please contact your O&M team to solve the problem.
|
||||
errorcode.hook_load_error=Error in loading the external hook. This problem is generally caused by the DataX installation.
|
||||
errorcode.hook_fail_error=Error in executing the external hook
|
||||
errorcode.plugin_install_error=Error in installing DataX plug-in. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||
errorcode.plugin_not_found=Error in DataX plug-in configuration. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||
errorcode.plugin_init_error=Error in DataX plug-in initialization. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||
errorcode.plugin_runtime_error=The DataX plug-in encountered an error during running. For the specific cause, refer to the error diagnosis after DataX stops running.
|
||||
errorcode.plugin_dirty_data_limit_exceed=The dirty data transmitted by DataX exceeds user expectations. This error often occurs when a lot dirty data exists in the source data. Please carefully check the dirty data log information reported by DataX, or you can tune up the dirty data threshold value.
|
||||
errorcode.plugin_split_error=Error in DataX plug-in slicing. This problem is generally caused by a programming error in some DataX plug-in. Please contact the DataX developer team to solve the problem.
|
||||
errorcode.kill_job_timeout_error=The kill task times out. Please contact the PE to solve the problem
|
||||
errorcode.start_taskgroup_error=Failed to start the task group. Please contact the DataX developer team to solve the problem
|
||||
errorcode.call_datax_service_failed=Error in requesting DataX Service.
|
||||
errorcode.call_remote_failed=Remote call failure
|
||||
errorcode.killed_exit_value=The job has received a Kill command.
|
||||
|
||||
|
||||
httpclientutil.1=Request address: {0}. Request method: {1}. STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=The remote interface returns -1. We will try again
|
||||
|
||||
|
||||
secretutil.1=System programing error. Unsupported encryption type
|
||||
secretutil.2=System programing error. Unsupported encryption type
|
||||
secretutil.3=RSA encryption error
|
||||
secretutil.4=RSA decryption error
|
||||
secretutil.5=Triple DES encryption error
|
||||
secretutil.6=RSA decryption error
|
||||
secretutil.7=Error in building Triple DES key
|
||||
secretutil.8=DataX configuration requires encryption and decryption, but unable to find the key configuration file
|
||||
secretutil.9=Error in reading the encryption and decryption configuration file
|
||||
secretutil.10=The version of the DataX-configured key is [{0}], but there is no configuration in the system. Error in task key configuration. The key version you configured does not exist
|
||||
secretutil.11=The version of the DataX-configured key is [{0}], but there is no configuration in the system. There may be an error in task key configuration, or a problem in system maintenance
|
||||
secretutil.12=The version of the DataX-configured key is [{0}], but there is no configuration in the system. Error in task key configuration. The key version you configured does not exist
|
||||
secretutil.13=The version of the DataX-configured key is [{0}], but there is no configuration in the system. There may be an error in task key configuration, or a problem in system maintenance
|
||||
secretutil.14=DataX configuration requires encryption and decryption, but some key in the configured key version [{0}] is empty
|
||||
secretutil.15=DataX configuration requires encryption and decryption, but some configured public/private key pairs are empty and the version is [{0}]
|
||||
secretutil.16=DataX configuration requires encryption and decryption, but the encryption and decryption configuration cannot be found
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||
|
||||
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||
|
||||
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||
|
||||
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
configparser.1=\u5916\u639B\u7A0B\u5F0F[{0},{1}]\u8F09\u5165\u5931\u6557\uFF0C1s\u5F8C\u91CD\u8A66... Exception:{2}
|
||||
configparser.2=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.3=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.4=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.5=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u5916\u639B\u7A0B\u5F0F\u8F09\u5165:{0}
|
||||
configparser.6=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557,\u5B58\u5728\u91CD\u8907\u5916\u639B\u7A0B\u5F0F:{0}
|
||||
|
||||
dataxserviceutil.1=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u9650\u5236\u61C9\u8A72\u5728[0.0, 1.0]\u4E4B\u9593
|
||||
errorrecordchecker.2=\u9AD2\u6578\u64DA\u689D\u6578\u73FE\u5728\u61C9\u8A72\u70BA\u975E\u8CA0\u6574\u6578
|
||||
errorrecordchecker.3=\u9AD2\u6578\u64DA\u689D\u6578\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\u689D\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u4E86[{1}]\u689D.
|
||||
errorrecordchecker.4=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88DD\u932F\u8AA4, \u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u904B\u884C\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BC\u5167\u90E8\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u904B\u884C\u904E\u7A0B\u51FA\u932F\uFF0C\u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u932F\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.hook_load_error=\u8F09\u5165\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4\uFF0C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u57F7\u884C\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4
|
||||
errorcode.plugin_install_error=DataX\u5916\u639B\u7A0B\u5F0F\u5B89\u88DD\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_not_found=DataX\u5916\u639B\u7A0B\u5F0F\u914D\u7F6E\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_init_error=DataX\u5916\u639B\u7A0B\u5F0F\u521D\u59CB\u5316\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_runtime_error=DataX\u5916\u639B\u7A0B\u5F0F\u904B\u884C\u6642\u51FA\u932F, \u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u50B3\u8F38\u9AD2\u6578\u64DA\u8D85\u904E\u7528\u6236\u9810\u671F\uFF0C\u8A72\u932F\u8AA4\u901A\u5E38\u662F\u7531\u65BC\u6E90\u7AEF\u6578\u64DA\u5B58\u5728\u8F03\u591A\u696D\u52D9\u9AD2\u6578\u64DA\u5C0E\u81F4\uFF0C\u8ACB\u4ED4\u7D30\u6AA2\u67E5DataX\u5F59\u5831\u7684\u9AD2\u6578\u64DA\u65E5\u8A8C\u8CC7\u8A0A, \u6216\u8005\u60A8\u53EF\u4EE5\u9069\u7576\u8ABF\u5927\u9AD2\u6578\u64DA\u95BE\u503C .
|
||||
errorcode.plugin_split_error=DataX\u5916\u639B\u7A0B\u5F0F\u5207\u5206\u51FA\u932F, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5404\u500B\u5916\u639B\u7A0B\u5F0F\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52D9\u903E\u6642\uFF0C\u8ACB\u806F\u7D61PE\u89E3\u6C7A
|
||||
errorcode.start_taskgroup_error=taskGroup\u555F\u52D5\u5931\u6557,\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||
errorcode.call_datax_service_failed=\u8ACB\u6C42 DataX Service \u51FA\u932F.
|
||||
errorcode.call_remote_failed=\u9060\u7A0B\u8ABF\u7528\u5931\u6557
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||
secretutil.2=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u932F
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u932F
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u932F
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u932F
|
||||
secretutil.7=\u69CB\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u932F
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u5BC6\u9470\u7684\u914D\u7F6E\u6A94\u6848
|
||||
secretutil.9=\u8B80\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6A94\u6848\u51FA\u932F
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u9470\u70BA\u7A7A\u7684\u60C5\u6CC1
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u9470\u5C0D\u5B58\u5728\u70BA\u7A7A\u7684\u60C5\u6CC1\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||
|
||||
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
configparser.1=\u5916\u639B\u7A0B\u5F0F[{0},{1}]\u8F09\u5165\u5931\u6557\uFF0C1s\u5F8C\u91CD\u8A66... Exception:{2}
|
||||
configparser.2=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.3=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.4=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||
configparser.5=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u5916\u639B\u7A0B\u5F0F\u8F09\u5165:{0}
|
||||
configparser.6=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557,\u5B58\u5728\u91CD\u8907\u5916\u639B\u7A0B\u5F0F:{0}
|
||||
|
||||
dataxserviceutil.1=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38NoSuchAlgorithmException, [{0}]
|
||||
dataxserviceutil.2=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38InvalidKeyException, [{0}]
|
||||
dataxserviceutil.3=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38UnsupportedEncodingException, [{0}]
|
||||
|
||||
errorrecordchecker.1=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u9650\u5236\u61C9\u8A72\u5728[0.0, 1.0]\u4E4B\u9593
|
||||
errorrecordchecker.2=\u9AD2\u6578\u64DA\u689D\u6578\u73FE\u5728\u61C9\u8A72\u70BA\u975E\u8CA0\u6574\u6578
|
||||
errorrecordchecker.3=\u9AD2\u6578\u64DA\u689D\u6578\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\u689D\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u4E86[{1}]\u689D.
|
||||
errorrecordchecker.4=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u5230[{1}].
|
||||
|
||||
|
||||
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88DD\u932F\u8AA4, \u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.argument_error=DataX\u5F15\u64CE\u904B\u884C\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BC\u5167\u90E8\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A .
|
||||
errorcode.runtime_error=DataX\u5F15\u64CE\u904B\u884C\u904E\u7A0B\u51FA\u932F\uFF0C\u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u932F\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.hook_load_error=\u8F09\u5165\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4\uFF0C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u5F15\u8D77\u7684
|
||||
errorcode.hook_fail_error=\u57F7\u884C\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4
|
||||
errorcode.plugin_install_error=DataX\u5916\u639B\u7A0B\u5F0F\u5B89\u88DD\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_not_found=DataX\u5916\u639B\u7A0B\u5F0F\u914D\u7F6E\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_init_error=DataX\u5916\u639B\u7A0B\u5F0F\u521D\u59CB\u5316\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||
errorcode.plugin_runtime_error=DataX\u5916\u639B\u7A0B\u5F0F\u904B\u884C\u6642\u51FA\u932F, \u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||
errorcode.plugin_dirty_data_limit_exceed=DataX\u50B3\u8F38\u9AD2\u6578\u64DA\u8D85\u904E\u7528\u6236\u9810\u671F\uFF0C\u8A72\u932F\u8AA4\u901A\u5E38\u662F\u7531\u65BC\u6E90\u7AEF\u6578\u64DA\u5B58\u5728\u8F03\u591A\u696D\u52D9\u9AD2\u6578\u64DA\u5C0E\u81F4\uFF0C\u8ACB\u4ED4\u7D30\u6AA2\u67E5DataX\u5F59\u5831\u7684\u9AD2\u6578\u64DA\u65E5\u8A8C\u8CC7\u8A0A, \u6216\u8005\u60A8\u53EF\u4EE5\u9069\u7576\u8ABF\u5927\u9AD2\u6578\u64DA\u95BE\u503C .
|
||||
errorcode.plugin_split_error=DataX\u5916\u639B\u7A0B\u5F0F\u5207\u5206\u51FA\u932F, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5404\u500B\u5916\u639B\u7A0B\u5F0F\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||
errorcode.kill_job_timeout_error=kill \u4EFB\u52D9\u903E\u6642\uFF0C\u8ACB\u806F\u7D61PE\u89E3\u6C7A
|
||||
errorcode.start_taskgroup_error=taskGroup\u555F\u52D5\u5931\u6557,\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||
errorcode.call_datax_service_failed=\u8ACB\u6C42 DataX Service \u51FA\u932F.
|
||||
errorcode.call_remote_failed=\u9060\u7A0B\u8ABF\u7528\u5931\u6557
|
||||
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||
|
||||
|
||||
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||
|
||||
|
||||
secretutil.1=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||
secretutil.2=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||
secretutil.3=rsa\u52A0\u5BC6\u51FA\u932F
|
||||
secretutil.4=rsa\u89E3\u5BC6\u51FA\u932F
|
||||
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u932F
|
||||
secretutil.6=rsa\u89E3\u5BC6\u51FA\u932F
|
||||
secretutil.7=\u69CB\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u932F
|
||||
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u5BC6\u9470\u7684\u914D\u7F6E\u6A94\u6848
|
||||
secretutil.9=\u8B80\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6A94\u6848\u51FA\u932F
|
||||
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u9470\u70BA\u7A7A\u7684\u60C5\u6CC1
|
||||
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u9470\u5C0D\u5B58\u5728\u70BA\u7A7A\u7684\u60C5\u6CC1\uFF0C\u7248\u672C[{0}]
|
||||
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||
|
||||
+4
-4
@@ -111,7 +111,7 @@ public class SomeReader extends Reader {
|
||||
```
|
||||
|
||||
`Job`接口功能如下:
|
||||
- `init`: Job对象初始化工作,测试可以通过`super.getPluginJobConf()`获取与本插件相关的配置。读插件获得配置中`reader`部分,写插件获得`writer`部分。
|
||||
- `init`: Job对象初始化工作,此时可以通过`super.getPluginJobConf()`获取与本插件相关的配置。读插件获得配置中`reader`部分,写插件获得`writer`部分。
|
||||
- `prepare`: 全局准备工作,比如odpswriter清空目标表。
|
||||
- `split`: 拆分`Task`。参数`adviceNumber`框架建议的拆分数,一般是运行时所配置的并发度。值返回的是`Task`的配置列表。
|
||||
- `post`: 全局的后置工作,比如mysqlwriter同步完影子表后的rename操作。
|
||||
@@ -155,7 +155,7 @@ public class SomeReader extends Reader {
|
||||
```
|
||||
|
||||
- `name`: 插件名称,大小写敏感。框架根据用户在配置文件中指定的名称来搜寻插件。 **十分重要** 。
|
||||
- `class`: 入口类的全限定名称,框架通过反射穿件入口类的实例。**十分重要** 。
|
||||
- `class`: 入口类的全限定名称,框架通过反射插件入口类的实例。**十分重要** 。
|
||||
- `description`: 描述信息。
|
||||
- `developer`: 开发人员。
|
||||
|
||||
@@ -435,7 +435,7 @@ DataX的内部类型在实现上会选用不同的java类型:
|
||||
|
||||
#### 如何处理脏数据
|
||||
|
||||
在`Reader.Task`和`Writer.Task`中,功过`AbstractTaskPlugin.getPluginCollector()`可以拿到一个`TaskPluginCollector`,它提供了一系列`collectDirtyRecord`的方法。当脏数据出现时,只需要调用合适的`collectDirtyRecord`方法,把被认为是脏数据的`Record`传入即可。
|
||||
在`Reader.Task`和`Writer.Task`中,通过`AbstractTaskPlugin.getTaskPluginCollector()`可以拿到一个`TaskPluginCollector`,它提供了一系列`collectDirtyRecord`的方法。当脏数据出现时,只需要调用合适的`collectDirtyRecord`方法,把被认为是脏数据的`Record`传入即可。
|
||||
|
||||
用户可以在任务的配置中指定脏数据限制条数或者百分比限制,当脏数据超出限制时,框架会结束同步任务,退出。插件需要保证脏数据都被收集到,其他工作交给框架就好。
|
||||
|
||||
@@ -468,4 +468,4 @@ DataX的内部类型在实现上会选用不同的java类型:
|
||||
- 测试参数集(多组),系统参数(比如并发数),插件参数(比如batchSize)
|
||||
- 不同参数下同步速度(Rec/s, MB/s),机器负载(load, cpu)等,对数据源压力(load, cpu, mem等)。
|
||||
6. **约束限制**:是否存在其他的使用限制条件。
|
||||
7. **FQA**:用户经常会遇到的问题。
|
||||
7. **FAQ**:用户经常会遇到的问题。
|
||||
|
||||
@@ -50,7 +50,7 @@ DRDS的插件目前DataX只适配了Mysql引擎的场景,DRDS对于DataX而言
|
||||
// 数据库连接密码
|
||||
"password": "root",
|
||||
"column": [
|
||||
"id","name"
|
||||
"id","name"
|
||||
],
|
||||
"connection": [
|
||||
{
|
||||
|
||||
+1
-1
@@ -42,7 +42,7 @@
|
||||
<dependency>
|
||||
<groupId>mysql</groupId>
|
||||
<artifactId>mysql-connector-java</artifactId>
|
||||
<version>5.1.34</version>
|
||||
<version>${mysql.driver.version}</version>
|
||||
</dependency>
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -44,7 +44,7 @@
|
||||
<dependency>
|
||||
<groupId>mysql</groupId>
|
||||
<artifactId>mysql-connector-java</artifactId>
|
||||
<version>5.1.34</version>
|
||||
<version>${mysql.driver.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<version>4.11</version>
|
||||
<version>4.13.1</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
@@ -63,6 +63,7 @@ FtpWriter实现了从DataX协议转为FTP文件功能,FTP文件本身是无结
|
||||
"nullFormat": "null",
|
||||
"dateFormat": "yyyy-MM-dd",
|
||||
"fileFormat": "csv",
|
||||
"suffix": ".csv",
|
||||
"header": []
|
||||
}
|
||||
}
|
||||
@@ -200,6 +201,14 @@ FtpWriter实现了从DataX协议转为FTP文件功能,FTP文件本身是无结
|
||||
* 必选:否 <br />
|
||||
|
||||
* 默认值:text <br />
|
||||
|
||||
* **suffix**
|
||||
|
||||
* 描述:最后输出文件的后缀,当前支持 ".text"以及".csv"
|
||||
|
||||
* 必选:否 <br />
|
||||
|
||||
* 默认值:"" <br />
|
||||
|
||||
* **header**
|
||||
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
|
||||
# DataX GDBReader
|
||||
|
||||
## 1. 快速介绍
|
||||
|
||||
GDBReader插件实现读取GDB实例数据的功能,通过`Gremlin Client`连接远程GDB实例,按配置提供的`label`生成查询DSL,遍历点或边数据,包括属性数据,并将数据写入到Record中给到Writer使用。
|
||||
|
||||
## 2. 实现原理
|
||||
|
||||
GDBReader使用`Gremlin Client`连接GDB实例,按`label`分不同Task取点或边数据。
|
||||
单个Task中按`label`遍历点或边的id,再切分范围分多次请求查询点或边和属性数据,最后将点或边数据根据配置转换成指定格式记录发送给下游写插件。
|
||||
|
||||
GDBReader按`label`切分多个Task并发,同一个`label`的数据批量异步获取来加快读取速度。如果配置读取的`label`列表为空,任务启动前会从GDB查询所有`label`再切分Task。
|
||||
|
||||
## 3. 功能说明
|
||||
|
||||
GDB中点和边不同,读取需要区分点和边点配置。
|
||||
|
||||
### 3.1 点配置样例
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 1
|
||||
}
|
||||
"errorLimit": {
|
||||
"record": 1
|
||||
}
|
||||
},
|
||||
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "gdbreader",
|
||||
"parameter": {
|
||||
"host": "10.218.145.24",
|
||||
"port": 8182,
|
||||
"username": "***",
|
||||
"password": "***",
|
||||
"fetchBatchSize": 100,
|
||||
"rangeSplitSize": 1000,
|
||||
"labelType": "VERTEX",
|
||||
"labels": ["label1", "label2"],
|
||||
"column": [
|
||||
{
|
||||
"name": "id",
|
||||
"type": "string",
|
||||
"columnType": "primaryKey"
|
||||
},
|
||||
{
|
||||
"name": "label",
|
||||
"type": "string",
|
||||
"columnType": "primaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "age",
|
||||
"type": "int",
|
||||
"columnType": "vertexProperty"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"print": true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 边配置样例
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 1
|
||||
},
|
||||
"errorLimit": {
|
||||
"record": 1
|
||||
}
|
||||
},
|
||||
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "gdbreader",
|
||||
"parameter": {
|
||||
"host": "10.218.145.24",
|
||||
"port": 8182,
|
||||
"username": "***",
|
||||
"password": "***",
|
||||
"fetchBatchSize": 100,
|
||||
"rangeSplitSize": 1000,
|
||||
"labelType": "EDGE",
|
||||
"labels": ["label1", "label2"],
|
||||
"column": [
|
||||
{
|
||||
"name": "id",
|
||||
"type": "string",
|
||||
"columnType": "primaryKey"
|
||||
},
|
||||
{
|
||||
"name": "label",
|
||||
"type": "string",
|
||||
"columnType": "primaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "srcId",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "srcLabel",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "dstId",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "dstLabel",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"type": "string",
|
||||
"columnType": "edgeProperty"
|
||||
},
|
||||
{
|
||||
"name": "weight",
|
||||
"type": "double",
|
||||
"columnType": "edgeProperty"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"print": true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 参数说明
|
||||
|
||||
* **host**
|
||||
* 描述:GDB实例连接地址,对应'实例管理'->'基本信息'页面的网络地址
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **port**
|
||||
* 描述:GDB实例连接地址对应的端口
|
||||
* 必选:是
|
||||
* 默认值:8182
|
||||
|
||||
* **username**
|
||||
* 描述:GDB实例账号名
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **password**
|
||||
* 描述:GDB实例账号名对应的密码
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **fetchBatchSize**
|
||||
* 描述:一次GDB请求读取点或边的数量,响应包含点或边以及属性
|
||||
* 必选:是
|
||||
* 默认值:100
|
||||
|
||||
* **rangeSplitSize**
|
||||
* 描述:id遍历,一次遍历请求扫描的id个数
|
||||
* 必选:是
|
||||
* 默认值:10 \* fetchBatchSize
|
||||
|
||||
* **labels**
|
||||
* 描述:标签数组,即需要导出的点或边标签,支持读取多个标签,用数组表示。如果留空([]),表示GDB中所有点或边标签
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **labelType**
|
||||
* 描述:数据标签类型,支持点、边两种枚举值
|
||||
* VERTEX:表示点
|
||||
* EDGE:表示边
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **column**
|
||||
* 描述:点或边字段映射关系配置
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **column -> name**
|
||||
* 描述:点或边映射关系的字段名,指定属性时表示读取的属性名,读取其他字段时会被忽略
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **column -> type**
|
||||
* 描述:点或边映射关系的字段类型
|
||||
* id, label在GDB中都是string类型,配置非string类型时可能会转换失败
|
||||
* 普通属性支持基础类型,包括int, long, float, double, boolean, string
|
||||
* GDBReader尽量将读取到的数据转换成配置要求的类型,但转换失败会导致该条记录错误
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
* **column -> columnType**
|
||||
* 描述:GDB点或边数据到列数据的映射关系,支持以下枚举值:
|
||||
* primaryKey: 表示该字段是点或边的id
|
||||
* primaryLabel: 表示该字段是点或边的label
|
||||
* srcPrimaryKey: 表示该字段是边关联的起点id,只在读取边时使用
|
||||
* srcPrimaryLabel: 表示该字段是边关联的起点label,只在读取边时使用
|
||||
* dstPrimaryKey: 表示该字段是边关联的终点id,只在读取边时使用
|
||||
* dstPrimaryLabel: 表示该字段是边关联的终点label,只在读取边时使用
|
||||
* vertexProperty: 表示该字段是点的属性,只在读取点时使用,应用到SET属性时只读取其中的一个属性值
|
||||
* vertexJsonProperty: 表示该字段是点的属性集合,只在读取点时使用。属性集合使用JSON格式输出,包含所有的属性,不能与其他vertexProperty配置一起使用
|
||||
* edgeProperty: 表示该字段是边的属性,只在读取边时使用
|
||||
* edgeJsonProperty: 表示该字段是边的属性集合,只在读取边时使用。属性集合使用JSON格式输出,包含所有的属性,不能与其他edgeProperty配置一起使用
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
* vertexJsonProperty格式示例,新增`c`字段区分SET属性,但是SET属性只包含单个属性值时会标记成普通属性
|
||||
```
|
||||
{"properties":[
|
||||
{"k":"name","t","string","v":"Jack","c":"set"},
|
||||
{"k":"name","t","string","v":"Luck","c":"set"},
|
||||
{"k":"age","t","int","v":"20","c":"single"}
|
||||
]}
|
||||
```
|
||||
* edgeJsonProperty格式示例,边不支持多值属性
|
||||
```
|
||||
{"properties":[
|
||||
{"k":"created_at","t","long","v":"153498653"},
|
||||
{"k":"weight","t","double","v":"3.14"}
|
||||
]}
|
||||
|
||||
## 4 性能报告
|
||||
(TODO)
|
||||
|
||||
## 5 使用约束
|
||||
无
|
||||
|
||||
## 6 FAQ
|
||||
无
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>gdbreader</artifactId>
|
||||
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<version>${datax-project-version}</version>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-core</artifactId>
|
||||
<version>${datax-project-version}</version>
|
||||
<scope>test</scope>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.tinkerpop</groupId>
|
||||
<artifactId>gremlin-driver</artifactId>
|
||||
<version>3.4.1</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>1.18.8</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-api</artifactId>
|
||||
<version>5.4.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.junit.jupiter</groupId>
|
||||
<artifactId>junit-jupiter-engine</artifactId>
|
||||
<version>5.4.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<!-- compiler plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>1.6</source>
|
||||
<target>1.6</target>
|
||||
<encoding>${project-sourceEncoding}</encoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- test case plugin -->
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<version>2.22.0</version>
|
||||
<configuration>
|
||||
<includes>
|
||||
<include>**/*Test*.class</include>
|
||||
</includes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- assembly plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<configuration>
|
||||
<descriptors>
|
||||
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||
</descriptors>
|
||||
<finalName>datax</finalName>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>dwzip</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>single</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>8</source>
|
||||
<target>8</target>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@@ -0,0 +1,35 @@
|
||||
<assembly
|
||||
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||
<id></id>
|
||||
<formats>
|
||||
<format>dir</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<fileSets>
|
||||
<fileSet>
|
||||
<directory>src/main/resources</directory>
|
||||
<includes>
|
||||
<include>plugin.json</include>
|
||||
<include>plugin_job_template.json</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/gdbreader</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>target/</directory>
|
||||
<includes>
|
||||
<include>gdbreader-0.0.1-SNAPSHOT.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/gdbreader</outputDirectory>
|
||||
</fileSet>
|
||||
</fileSets>
|
||||
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<useProjectArtifact>false</useProjectArtifact>
|
||||
<outputDirectory>plugin/reader/gdbreader/libs</outputDirectory>
|
||||
<scope>runtime</scope>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
@@ -0,0 +1,231 @@
|
||||
package com.alibaba.datax.plugin.reader.gdbreader;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.plugin.RecordSender;
|
||||
import com.alibaba.datax.common.spi.Reader;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.mapping.DefaultGdbMapper;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.mapping.MappingRule;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.mapping.MappingRuleFactory;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.model.GdbElement;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.model.GdbGraph;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.model.ScriptGdbGraph;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.util.ConfigHelper;
|
||||
import org.apache.tinkerpop.gremlin.driver.ResultSet;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
|
||||
public class GdbReader extends Reader {
|
||||
private final static int DEFAULT_FETCH_BATCH_SIZE = 200;
|
||||
private static GdbGraph graph;
|
||||
private static Key.ExportType exportType;
|
||||
|
||||
/**
|
||||
* Job 中的方法仅执行一次,Task 中方法会由框架启动多个 Task 线程并行执行。
|
||||
* <p/>
|
||||
* 整个 Reader 执行流程是:
|
||||
* <pre>
|
||||
* Job类init-->prepare-->split
|
||||
*
|
||||
* Task类init-->prepare-->startRead-->post-->destroy
|
||||
* Task类init-->prepare-->startRead-->post-->destroy
|
||||
*
|
||||
* Job类post-->destroy
|
||||
* </pre>
|
||||
*/
|
||||
public static class Job extends Reader.Job {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(Job.class);
|
||||
|
||||
private Configuration jobConfig = null;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.jobConfig = super.getPluginJobConf();
|
||||
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:通常在这里对用户的配置进行校验:是否缺失必填项?有无错误值?有没有无关配置项?...
|
||||
* 并给出清晰的报错/警告提示。校验通常建议采用静态工具类进行,以保证本类结构清晰。
|
||||
*/
|
||||
|
||||
ConfigHelper.assertGdbClient(jobConfig);
|
||||
ConfigHelper.assertLabels(jobConfig);
|
||||
try {
|
||||
exportType = Key.ExportType.valueOf(jobConfig.getString(Key.EXPORT_TYPE));
|
||||
} catch (NullPointerException | IllegalArgumentException e) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.BAD_CONFIG_VALUE, Key.EXPORT_TYPE);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void prepare() {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:如果 Job 中有需要进行数据同步之前的处理,可以在此处完成,如果没有必要则可以直接去掉。
|
||||
*/
|
||||
|
||||
try {
|
||||
graph = new ScriptGdbGraph(jobConfig, exportType);
|
||||
} catch (RuntimeException e) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.FAIL_CLIENT_CONNECT, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Configuration> split(int adviceNumber) {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:通常采用工具静态类完成把 Job 配置切分成多个 Task 配置的工作。
|
||||
* 这里的 adviceNumber 是框架根据用户的同步速度的要求建议的切分份数,仅供参考,不是强制必须切分的份数。
|
||||
*/
|
||||
List<String> labels = ConfigHelper.assertLabels(jobConfig);
|
||||
|
||||
/**
|
||||
* 配置label列表为空时,尝试查询GDB中所有label,添加到读取列表
|
||||
*/
|
||||
if (labels.isEmpty()) {
|
||||
try {
|
||||
labels.addAll(graph.getLabels().keySet());
|
||||
} catch (RuntimeException ex) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.FAIL_FETCH_LABELS, ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
if (labels.isEmpty()) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.FAIL_FETCH_LABELS, "none labels to read");
|
||||
}
|
||||
|
||||
return ConfigHelper.splitConfig(jobConfig, labels);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:如果 Job 中有需要进行数据同步之后的后续处理,可以在此处完成。
|
||||
*/
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:通常配合 Job 中的 post() 方法一起完成 Job 的资源释放。
|
||||
*/
|
||||
try {
|
||||
graph.close();
|
||||
} catch (Exception ex) {
|
||||
LOG.error("Failed to close client : {}", ex);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
public static class Task extends Reader.Task {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
||||
private static MappingRule rule;
|
||||
private Configuration taskConfig;
|
||||
private String fetchLabel = null;
|
||||
|
||||
private int rangeSplitSize;
|
||||
private int fetchBatchSize;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.taskConfig = super.getPluginJobConf();
|
||||
|
||||
/**
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:此处通过对 taskConfig 配置的读取,进而初始化一些资源为 startRead()做准备。
|
||||
*/
|
||||
fetchLabel = taskConfig.getString(Key.LABEL);
|
||||
fetchBatchSize = taskConfig.getInt(Key.FETCH_BATCH_SIZE, DEFAULT_FETCH_BATCH_SIZE);
|
||||
rangeSplitSize = taskConfig.getInt(Key.RANGE_SPLIT_SIZE, fetchBatchSize * 10);
|
||||
rule = MappingRuleFactory.getInstance().create(taskConfig, exportType);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void prepare() {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:如果 Job 中有需要进行数据同步之后的处理,可以在此处完成,如果没有必要则可以直接去掉。
|
||||
*/
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startRead(RecordSender recordSender) {
|
||||
/**
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:此处适当封装确保简洁清晰完成数据读取工作。
|
||||
*/
|
||||
|
||||
String start = "";
|
||||
while (true) {
|
||||
List<String> ids;
|
||||
try {
|
||||
ids = graph.fetchIds(fetchLabel, start, rangeSplitSize);
|
||||
if (ids.isEmpty()) {
|
||||
break;
|
||||
}
|
||||
start = ids.get(ids.size() - 1);
|
||||
} catch (Exception ex) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.FAIL_FETCH_IDS, ex.getMessage());
|
||||
}
|
||||
|
||||
// send range fetch async
|
||||
int count = ids.size();
|
||||
List<ResultSet> resultSets = new LinkedList<>();
|
||||
for (int pos = 0; pos < count; pos += fetchBatchSize) {
|
||||
int rangeSize = Math.min(fetchBatchSize, count - pos);
|
||||
String endId = ids.get(pos + rangeSize - 1);
|
||||
String beginId = ids.get(pos);
|
||||
|
||||
List<String> propNames = rule.isHasProperty() ? rule.getPropertyNames() : null;
|
||||
try {
|
||||
resultSets.add(graph.fetchElementsAsync(fetchLabel, beginId, endId, propNames));
|
||||
} catch (Exception ex) {
|
||||
// just print error logs and continues
|
||||
LOG.error("failed to request label: {}, start: {}, end: {}, e: {}", fetchLabel, beginId, endId, ex);
|
||||
}
|
||||
}
|
||||
|
||||
// get range fetch dsl results
|
||||
resultSets.forEach(results -> {
|
||||
try {
|
||||
List<GdbElement> elements = graph.getElement(results);
|
||||
elements.forEach(element -> {
|
||||
Record record = recordSender.createRecord();
|
||||
DefaultGdbMapper.getMapper(rule).accept(element, record);
|
||||
recordSender.sendToWriter(record);
|
||||
});
|
||||
recordSender.flush();
|
||||
} catch (Exception ex) {
|
||||
LOG.error("failed to send records e {}", ex);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
/**
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:如果 Task 中有需要进行数据同步之后的后续处理,可以在此处完成。
|
||||
*/
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
/**
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:通常配合Task 中的 post() 方法一起完成 Task 的资源释放。
|
||||
*/
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
package com.alibaba.datax.plugin.reader.gdbreader;
|
||||
|
||||
import com.alibaba.datax.common.spi.ErrorCode;
|
||||
|
||||
public enum GdbReaderErrorCode implements ErrorCode {
|
||||
/**
|
||||
*
|
||||
*/
|
||||
BAD_CONFIG_VALUE("GdbReader-00", "The value you configured is invalid."),
|
||||
FAIL_CLIENT_CONNECT("GdbReader-02", "GDB connection is abnormal."),
|
||||
UNSUPPORTED_TYPE("GdbReader-03", "Unsupported data type conversion."),
|
||||
FAIL_FETCH_LABELS("GdbReader-04", "Error pulling all labels, it is recommended to configure the specified label pull."),
|
||||
FAIL_FETCH_IDS("GdbReader-05", "Pull range id error."),
|
||||
;
|
||||
|
||||
private final String code;
|
||||
private final String description;
|
||||
|
||||
private GdbReaderErrorCode(String code, String description) {
|
||||
this.code = code;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getCode() {
|
||||
return this.code;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getDescription() {
|
||||
return this.description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code,
|
||||
this.description);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package com.alibaba.datax.plugin.reader.gdbreader;
|
||||
|
||||
public final class Key {
|
||||
|
||||
/**
|
||||
* 此处声明插件用到的需要插件使用者提供的配置项
|
||||
*/
|
||||
public final static String HOST = "host";
|
||||
public final static String PORT = "port";
|
||||
public final static String USERNAME = "username";
|
||||
public static final String PASSWORD = "password";
|
||||
|
||||
public static final String LABEL = "labels";
|
||||
public static final String EXPORT_TYPE = "labelType";
|
||||
|
||||
public static final String RANGE_SPLIT_SIZE = "RangeSplitSize";
|
||||
public static final String FETCH_BATCH_SIZE = "fetchBatchSize";
|
||||
|
||||
public static final String COLUMN = "column";
|
||||
public static final String COLUMN_NAME = "name";
|
||||
public static final String COLUMN_TYPE = "type";
|
||||
public static final String COLUMN_NODE_TYPE = "columnType";
|
||||
|
||||
public enum ExportType {
|
||||
/**
|
||||
* Import vertices
|
||||
*/
|
||||
VERTEX,
|
||||
/**
|
||||
* Import edges
|
||||
*/
|
||||
EDGE
|
||||
}
|
||||
|
||||
public enum ColumnType {
|
||||
/**
|
||||
* vertex or edge id
|
||||
*/
|
||||
primaryKey,
|
||||
|
||||
/**
|
||||
* vertex or edge label
|
||||
*/
|
||||
primaryLabel,
|
||||
|
||||
/**
|
||||
* vertex property
|
||||
*/
|
||||
vertexProperty,
|
||||
|
||||
/**
|
||||
* collects all vertex property to Json list
|
||||
*/
|
||||
vertexJsonProperty,
|
||||
|
||||
/**
|
||||
* start vertex id of edge
|
||||
*/
|
||||
srcPrimaryKey,
|
||||
|
||||
/**
|
||||
* start vertex label of edge
|
||||
*/
|
||||
srcPrimaryLabel,
|
||||
|
||||
/**
|
||||
* end vertex id of edge
|
||||
*/
|
||||
dstPrimaryKey,
|
||||
|
||||
/**
|
||||
* end vertex label of edge
|
||||
*/
|
||||
dstPrimaryLabel,
|
||||
|
||||
/**
|
||||
* edge property
|
||||
*/
|
||||
edgeProperty,
|
||||
|
||||
/**
|
||||
* collects all edge property to Json list
|
||||
*/
|
||||
edgeJsonProperty,
|
||||
}
|
||||
}
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.mapping;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.model.GdbElement;
|
||||
import org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceProperty;
|
||||
import org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceVertexProperty;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.BiConsumer;
|
||||
import java.util.function.Function;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public class DefaultGdbMapper {
|
||||
|
||||
public static BiConsumer<GdbElement, Record> getMapper(MappingRule rule) {
|
||||
return (gdbElement, record) -> rule.getColumns().forEach(columnMappingRule -> {
|
||||
Object value = null;
|
||||
ValueType type = columnMappingRule.getValueType();
|
||||
String name = columnMappingRule.getName();
|
||||
Map<String, Object> props = gdbElement.getProperties();
|
||||
|
||||
switch (columnMappingRule.getColumnType()) {
|
||||
case dstPrimaryKey:
|
||||
value = gdbElement.getTo();
|
||||
break;
|
||||
case srcPrimaryKey:
|
||||
value = gdbElement.getFrom();
|
||||
break;
|
||||
case primaryKey:
|
||||
value = gdbElement.getId();
|
||||
break;
|
||||
case primaryLabel:
|
||||
value = gdbElement.getLabel();
|
||||
break;
|
||||
case dstPrimaryLabel:
|
||||
value = gdbElement.getToLabel();
|
||||
break;
|
||||
case srcPrimaryLabel:
|
||||
value = gdbElement.getFromLabel();
|
||||
break;
|
||||
case vertexProperty:
|
||||
value = forVertexOnePropertyValue().apply(props.get(name));
|
||||
break;
|
||||
case edgeProperty:
|
||||
value = forEdgePropertyValue().apply(props.get(name));
|
||||
break;
|
||||
case edgeJsonProperty:
|
||||
value = forEdgeJsonProperties().apply(props);
|
||||
break;
|
||||
case vertexJsonProperty:
|
||||
value = forVertexJsonProperties().apply(props);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
record.addColumn(type.applyObject(value));
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* parser ReferenceProperty value for edge
|
||||
*
|
||||
* @return property value
|
||||
*/
|
||||
private static Function<Object, Object> forEdgePropertyValue() {
|
||||
return prop -> {
|
||||
if (prop instanceof ReferenceProperty) {
|
||||
return ((ReferenceProperty) prop).value();
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* parser ReferenceVertexProperty value for vertex
|
||||
*
|
||||
* @return the first property value in list
|
||||
*/
|
||||
private static Function<Object, Object> forVertexOnePropertyValue() {
|
||||
return props -> {
|
||||
if (props instanceof List<?>) {
|
||||
// get the first one property if more than one
|
||||
Object o = ((List) props).get(0);
|
||||
if (o instanceof ReferenceVertexProperty) {
|
||||
return ((ReferenceVertexProperty) o).value();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* parser all edge properties to json string
|
||||
*
|
||||
* @return json string
|
||||
*/
|
||||
private static Function<Map<String, Object>, String> forEdgeJsonProperties() {
|
||||
return props -> "{\"properties\":[" +
|
||||
props.entrySet().stream().filter(p -> p.getValue() instanceof ReferenceProperty)
|
||||
.map(p -> "{\"k\":\"" + ((ReferenceProperty) p.getValue()).key() + "\"," +
|
||||
"\"t\":\"" + ((ReferenceProperty) p.getValue()).value().getClass().getSimpleName().toLowerCase() + "\"," +
|
||||
"\"v\":\"" + String.valueOf(((ReferenceProperty) p.getValue()).value()) + "\"}")
|
||||
.collect(Collectors.joining(",")) +
|
||||
"]}";
|
||||
}
|
||||
|
||||
/**
|
||||
* parser all vertex properties to json string, include set-property
|
||||
*
|
||||
* @return json string
|
||||
*/
|
||||
private static Function<Map<String, Object>, String> forVertexJsonProperties() {
|
||||
return props -> "{\"properties\":[" +
|
||||
props.entrySet().stream().filter(p -> p.getValue() instanceof List<?>)
|
||||
.map(p -> forVertexPropertyStr().apply((List<?>) p.getValue()))
|
||||
.collect(Collectors.joining(",")) +
|
||||
"]}";
|
||||
}
|
||||
|
||||
/**
|
||||
* parser one vertex property to json string item, set 'cardinality'
|
||||
*
|
||||
* @return json string item
|
||||
*/
|
||||
private static Function<List<?>, String> forVertexPropertyStr() {
|
||||
return vp -> {
|
||||
final String setFlag = vp.size() > 1 ? "set" : "single";
|
||||
return vp.stream().filter(p -> p instanceof ReferenceVertexProperty)
|
||||
.map(p -> "{\"k\":\"" + ((ReferenceVertexProperty) p).key() + "\"," +
|
||||
"\"t\":\"" + ((ReferenceVertexProperty) p).value().getClass().getSimpleName().toLowerCase() + "\"," +
|
||||
"\"v\":\"" + String.valueOf(((ReferenceVertexProperty) p).value()) + "\"," +
|
||||
"\"c\":\"" + setFlag + "\"}")
|
||||
.collect(Collectors.joining(","));
|
||||
};
|
||||
}
|
||||
}
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.mapping;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.GdbReaderErrorCode;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key.ColumnType;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key.ExportType;
|
||||
import lombok.Data;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
@Data
|
||||
public class MappingRule {
|
||||
private boolean hasRelation = false;
|
||||
private boolean hasProperty = false;
|
||||
private ExportType type = ExportType.VERTEX;
|
||||
|
||||
/**
|
||||
* property names for property key-value
|
||||
*/
|
||||
private List<String> propertyNames = new ArrayList<>();
|
||||
|
||||
private List<ColumnMappingRule> columns = new ArrayList<>();
|
||||
|
||||
void addColumn(ColumnType columnType, ValueType type, String name) {
|
||||
ColumnMappingRule rule = new ColumnMappingRule();
|
||||
rule.setColumnType(columnType);
|
||||
rule.setName(name);
|
||||
rule.setValueType(type);
|
||||
|
||||
if (columnType == ColumnType.vertexProperty || columnType == ColumnType.edgeProperty) {
|
||||
propertyNames.add(name);
|
||||
hasProperty = true;
|
||||
}
|
||||
|
||||
boolean hasTo = columnType == ColumnType.dstPrimaryKey || columnType == ColumnType.dstPrimaryLabel;
|
||||
boolean hasFrom = columnType == ColumnType.srcPrimaryKey || columnType == ColumnType.srcPrimaryLabel;
|
||||
if (hasTo || hasFrom) {
|
||||
hasRelation = true;
|
||||
}
|
||||
|
||||
columns.add(rule);
|
||||
}
|
||||
|
||||
void addJsonColumn(ColumnType columnType) {
|
||||
ColumnMappingRule rule = new ColumnMappingRule();
|
||||
rule.setColumnType(columnType);
|
||||
rule.setName("json");
|
||||
rule.setValueType(ValueType.STRING);
|
||||
|
||||
if (!propertyNames.isEmpty()) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.BAD_CONFIG_VALUE, "JsonProperties should be only property");
|
||||
}
|
||||
|
||||
columns.add(rule);
|
||||
hasProperty = true;
|
||||
}
|
||||
|
||||
@Data
|
||||
protected static class ColumnMappingRule {
|
||||
private String name = null;
|
||||
|
||||
private ValueType valueType = null;
|
||||
|
||||
private ColumnType columnType = null;
|
||||
}
|
||||
}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.mapping;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.GdbReaderErrorCode;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key.ColumnType;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key.ExportType;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.util.ConfigHelper;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/20
|
||||
*/
|
||||
|
||||
public class MappingRuleFactory {
|
||||
private static final MappingRuleFactory instance = new MappingRuleFactory();
|
||||
|
||||
public static MappingRuleFactory getInstance() {
|
||||
return instance;
|
||||
}
|
||||
|
||||
public MappingRule create(Configuration config, ExportType exportType) {
|
||||
MappingRule rule = new MappingRule();
|
||||
|
||||
rule.setType(exportType);
|
||||
List<Configuration> configurationList = config.getListConfiguration(Key.COLUMN);
|
||||
for (Configuration column : configurationList) {
|
||||
ColumnType columnType;
|
||||
try {
|
||||
columnType = ColumnType.valueOf(column.getString(Key.COLUMN_NODE_TYPE));
|
||||
} catch (NullPointerException | IllegalArgumentException e) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.BAD_CONFIG_VALUE, Key.COLUMN_NODE_TYPE);
|
||||
}
|
||||
|
||||
if (exportType == ExportType.VERTEX) {
|
||||
// only id/label/property column allow when vertex
|
||||
ConfigHelper.assertConfig(Key.COLUMN_NODE_TYPE, () ->
|
||||
columnType == ColumnType.primaryKey || columnType == ColumnType.primaryLabel
|
||||
|| columnType == ColumnType.vertexProperty || columnType == ColumnType.vertexJsonProperty);
|
||||
} else if (exportType == ExportType.EDGE) {
|
||||
// edge
|
||||
ConfigHelper.assertConfig(Key.COLUMN_NODE_TYPE, () ->
|
||||
columnType == ColumnType.primaryKey || columnType == ColumnType.primaryLabel
|
||||
|| columnType == ColumnType.srcPrimaryKey || columnType == ColumnType.srcPrimaryLabel
|
||||
|| columnType == ColumnType.dstPrimaryKey || columnType == ColumnType.dstPrimaryLabel
|
||||
|| columnType == ColumnType.edgeProperty || columnType == ColumnType.edgeJsonProperty);
|
||||
}
|
||||
|
||||
if (columnType == ColumnType.edgeProperty || columnType == ColumnType.vertexProperty) {
|
||||
String name = column.getString(Key.COLUMN_NAME);
|
||||
ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
|
||||
ConfigHelper.assertConfig(Key.COLUMN_NAME, () -> name != null);
|
||||
if (propType == null) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.UNSUPPORTED_TYPE, Key.COLUMN_TYPE);
|
||||
}
|
||||
rule.addColumn(columnType, propType, name);
|
||||
} else if (columnType == ColumnType.vertexJsonProperty || columnType == ColumnType.edgeJsonProperty) {
|
||||
rule.addJsonColumn(columnType);
|
||||
} else {
|
||||
rule.addColumn(columnType, ValueType.STRING, null);
|
||||
}
|
||||
}
|
||||
return rule;
|
||||
}
|
||||
}
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.mapping;
|
||||
|
||||
import com.alibaba.datax.common.element.BoolColumn;
|
||||
import com.alibaba.datax.common.element.Column;
|
||||
import com.alibaba.datax.common.element.DoubleColumn;
|
||||
import com.alibaba.datax.common.element.LongColumn;
|
||||
import com.alibaba.datax.common.element.StringColumn;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public enum ValueType {
|
||||
/**
|
||||
* transfer gdb element object value to DataX Column data
|
||||
* <p>
|
||||
* int, long -> LongColumn
|
||||
* float, double -> DoubleColumn
|
||||
* bool -> BooleanColumn
|
||||
* string -> StringColumn
|
||||
*/
|
||||
INT(Integer.class, "int", ValueTypeHolder::longColumnMapper),
|
||||
INTEGER(Integer.class, "integer", ValueTypeHolder::longColumnMapper),
|
||||
LONG(Long.class, "long", ValueTypeHolder::longColumnMapper),
|
||||
DOUBLE(Double.class, "double", ValueTypeHolder::doubleColumnMapper),
|
||||
FLOAT(Float.class, "float", ValueTypeHolder::doubleColumnMapper),
|
||||
BOOLEAN(Boolean.class, "boolean", ValueTypeHolder::boolColumnMapper),
|
||||
STRING(String.class, "string", ValueTypeHolder::stringColumnMapper),
|
||||
;
|
||||
|
||||
private Class<?> type = null;
|
||||
private String shortName = null;
|
||||
private Function<Object, Column> columnFunc = null;
|
||||
|
||||
ValueType(Class<?> type, String name, Function<Object, Column> columnFunc) {
|
||||
this.type = type;
|
||||
this.shortName = name;
|
||||
this.columnFunc = columnFunc;
|
||||
|
||||
ValueTypeHolder.shortName2type.put(shortName, this);
|
||||
}
|
||||
|
||||
public static ValueType fromShortName(String name) {
|
||||
return ValueTypeHolder.shortName2type.get(name);
|
||||
}
|
||||
|
||||
public Column applyObject(Object value) {
|
||||
if (value == null) {
|
||||
return null;
|
||||
}
|
||||
return columnFunc.apply(value);
|
||||
}
|
||||
|
||||
private static class ValueTypeHolder {
|
||||
private static Map<String, ValueType> shortName2type = new HashMap<>();
|
||||
|
||||
private static LongColumn longColumnMapper(Object o) {
|
||||
long v;
|
||||
if (o instanceof Integer) {
|
||||
v = (int) o;
|
||||
} else if (o instanceof Long) {
|
||||
v = (long) o;
|
||||
} else if (o instanceof String) {
|
||||
v = Long.valueOf((String) o);
|
||||
} else {
|
||||
throw new RuntimeException("Failed to cast " + o.getClass() + " to Long");
|
||||
}
|
||||
|
||||
return new LongColumn(v);
|
||||
}
|
||||
|
||||
private static DoubleColumn doubleColumnMapper(Object o) {
|
||||
double v;
|
||||
if (o instanceof Integer) {
|
||||
v = (double) (int) o;
|
||||
} else if (o instanceof Long) {
|
||||
v = (double) (long) o;
|
||||
} else if (o instanceof Float) {
|
||||
v = (double) (float) o;
|
||||
} else if (o instanceof Double) {
|
||||
v = (double) o;
|
||||
} else if (o instanceof String) {
|
||||
v = Double.valueOf((String) o);
|
||||
} else {
|
||||
throw new RuntimeException("Failed to cast " + o.getClass() + " to Double");
|
||||
}
|
||||
|
||||
return new DoubleColumn(v);
|
||||
}
|
||||
|
||||
private static BoolColumn boolColumnMapper(Object o) {
|
||||
boolean v;
|
||||
if (o instanceof Integer) {
|
||||
v = ((int) o != 0);
|
||||
} else if (o instanceof Long) {
|
||||
v = ((long) o != 0);
|
||||
} else if (o instanceof Boolean) {
|
||||
v = (boolean) o;
|
||||
} else if (o instanceof String) {
|
||||
v = Boolean.valueOf((String) o);
|
||||
} else {
|
||||
throw new RuntimeException("Failed to cast " + o.getClass() + " to Boolean");
|
||||
}
|
||||
|
||||
return new BoolColumn(v);
|
||||
}
|
||||
|
||||
private static StringColumn stringColumnMapper(Object o) {
|
||||
if (o instanceof String) {
|
||||
return new StringColumn((String) o);
|
||||
} else {
|
||||
return new StringColumn(String.valueOf(o));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.model;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key;
|
||||
import org.apache.tinkerpop.gremlin.driver.Client;
|
||||
import org.apache.tinkerpop.gremlin.driver.Cluster;
|
||||
import org.apache.tinkerpop.gremlin.driver.RequestOptions;
|
||||
import org.apache.tinkerpop.gremlin.driver.Result;
|
||||
import org.apache.tinkerpop.gremlin.driver.ResultSet;
|
||||
import org.apache.tinkerpop.gremlin.driver.ser.Serializers;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public abstract class AbstractGdbGraph implements GdbGraph {
|
||||
final static int DEFAULT_TIMEOUT = 30000;
|
||||
private static final Logger log = LoggerFactory.getLogger(AbstractGdbGraph.class);
|
||||
private Client client;
|
||||
|
||||
AbstractGdbGraph() {
|
||||
}
|
||||
|
||||
AbstractGdbGraph(Configuration config) {
|
||||
log.info("init graphdb client");
|
||||
String host = config.getString(Key.HOST);
|
||||
int port = config.getInt(Key.PORT);
|
||||
String username = config.getString(Key.USERNAME);
|
||||
String password = config.getString(Key.PASSWORD);
|
||||
|
||||
try {
|
||||
Cluster cluster = Cluster.build(host).port(port).credentials(username, password)
|
||||
.serializer(Serializers.GRAPHBINARY_V1D0)
|
||||
.maxContentLength(1024 * 1024)
|
||||
.resultIterationBatchSize(64)
|
||||
.create();
|
||||
client = cluster.connect().init();
|
||||
|
||||
warmClient();
|
||||
} catch (RuntimeException e) {
|
||||
log.error("Failed to connect to GDB {}:{}, due to {}", host, port, e);
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
protected List<Result> runInternal(String dsl, Map<String, Object> params) throws Exception {
|
||||
return runInternalAsync(dsl, params).all().get(DEFAULT_TIMEOUT + 1000, TimeUnit.MILLISECONDS);
|
||||
}
|
||||
|
||||
protected ResultSet runInternalAsync(String dsl, Map<String, Object> params) throws Exception {
|
||||
RequestOptions.Builder options = RequestOptions.build().timeout(DEFAULT_TIMEOUT);
|
||||
if (params != null && !params.isEmpty()) {
|
||||
params.forEach(options::addParameter);
|
||||
}
|
||||
return client.submitAsync(dsl, options.create()).get(DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS);
|
||||
}
|
||||
|
||||
private void warmClient() {
|
||||
try {
|
||||
runInternal("g.V('test')", null);
|
||||
log.info("warm graphdb client over");
|
||||
} catch (Exception e) {
|
||||
log.error("warmClient error");
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws Exception {
|
||||
if (client != null) {
|
||||
log.info("close graphdb client");
|
||||
client.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.model;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
@Data
|
||||
public class GdbElement {
|
||||
String id = null;
|
||||
String label = null;
|
||||
String to = null;
|
||||
String from = null;
|
||||
String toLabel = null;
|
||||
String fromLabel = null;
|
||||
|
||||
Map<String, Object> properties = new HashMap<>();
|
||||
|
||||
public GdbElement() {
|
||||
}
|
||||
|
||||
public GdbElement(String id, String label) {
|
||||
this.id = id;
|
||||
this.label = label;
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.model;
|
||||
|
||||
import org.apache.tinkerpop.gremlin.driver.ResultSet;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public interface GdbGraph extends AutoCloseable {
|
||||
|
||||
/**
|
||||
* Get All labels of GraphDB
|
||||
*
|
||||
* @return labels map included numbers
|
||||
*/
|
||||
Map<String, Long> getLabels();
|
||||
|
||||
/**
|
||||
* Get the Ids list of special 'label', size up to 'limit'
|
||||
*
|
||||
* @param label is Label of Vertex or Edge
|
||||
* @param start of Ids range to get
|
||||
* @param limit size of Ids list
|
||||
* @return Ids list
|
||||
*/
|
||||
List<String> fetchIds(String label, String start, long limit);
|
||||
|
||||
/**
|
||||
* Fetch element in async mode, just send query dsl to server
|
||||
*
|
||||
* @param label node label to filter
|
||||
* @param start range begin(included)
|
||||
* @param end range end(included)
|
||||
* @param propNames propKey list to fetch
|
||||
* @return future to get result later
|
||||
*/
|
||||
ResultSet fetchElementsAsync(String label, String start, String end, List<String> propNames);
|
||||
|
||||
/**
|
||||
* Get get element from Response @{ResultSet}
|
||||
*
|
||||
* @param results Response of Server
|
||||
* @return element sets
|
||||
*/
|
||||
List<GdbElement> getElement(ResultSet results);
|
||||
|
||||
/**
|
||||
* close graph client
|
||||
*
|
||||
* @throws Exception if fails
|
||||
*/
|
||||
@Override
|
||||
void close() throws Exception;
|
||||
}
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.model;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key.ExportType;
|
||||
import org.apache.tinkerpop.gremlin.driver.Result;
|
||||
import org.apache.tinkerpop.gremlin.driver.ResultSet;
|
||||
import org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceEdge;
|
||||
import org.apache.tinkerpop.gremlin.structure.util.reference.ReferenceVertex;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public class ScriptGdbGraph extends AbstractGdbGraph {
|
||||
private static final Logger log = LoggerFactory.getLogger(ScriptGdbGraph.class);
|
||||
|
||||
private final static String LABEL = "GDB___LABEL";
|
||||
private final static String START_ID = "GDB___ID";
|
||||
private final static String END_ID = "GDB___ID_END";
|
||||
private final static String LIMIT = "GDB___LIMIT";
|
||||
|
||||
private final static String FETCH_VERTEX_IDS_DSL = "g.V().hasLabel(" + LABEL + ").has(id, gt(" + START_ID + ")).limit(" + LIMIT + ").id()";
|
||||
private final static String FETCH_EDGE_IDS_DSL = "g.E().hasLabel(" + LABEL + ").has(id, gt(" + START_ID + ")).limit(" + LIMIT + ").id()";
|
||||
|
||||
private final static String FETCH_VERTEX_LABELS_DSL = "g.V().groupCount().by(label)";
|
||||
private final static String FETCH_EDGE_LABELS_DSL = "g.E().groupCount().by(label)";
|
||||
|
||||
/**
|
||||
* fetch node range [START_ID, END_ID]
|
||||
*/
|
||||
private final static String FETCH_RANGE_VERTEX_DSL = "g.V().hasLabel(" + LABEL + ").has(id, gte(" + START_ID + ")).has(id, lte(" + END_ID + "))";
|
||||
private final static String FETCH_RANGE_EDGE_DSL = "g.E().hasLabel(" + LABEL + ").has(id, gte(" + START_ID + ")).has(id, lte(" + END_ID + "))";
|
||||
private final static String PART_WITH_PROP_DSL = ".as('a').project('node', 'props').by(select('a')).by(select('a').propertyMap(";
|
||||
|
||||
private final ExportType exportType;
|
||||
|
||||
public ScriptGdbGraph(ExportType exportType) {
|
||||
super();
|
||||
this.exportType = exportType;
|
||||
}
|
||||
|
||||
public ScriptGdbGraph(Configuration config, ExportType exportType) {
|
||||
super(config);
|
||||
this.exportType = exportType;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<String> fetchIds(final String label, final String start, long limit) {
|
||||
Map<String, Object> params = new HashMap<String, Object>(3) {{
|
||||
put(LABEL, label);
|
||||
put(START_ID, start);
|
||||
put(LIMIT, limit);
|
||||
}};
|
||||
String fetchDsl = exportType == ExportType.VERTEX ? FETCH_VERTEX_IDS_DSL : FETCH_EDGE_IDS_DSL;
|
||||
|
||||
List<String> ids = new ArrayList<>();
|
||||
try {
|
||||
List<Result> results = runInternal(fetchDsl, params);
|
||||
|
||||
// transfer result to id string
|
||||
results.forEach(id -> ids.add(id.getString()));
|
||||
} catch (Exception e) {
|
||||
log.error("fetch range node failed, label {}, start {}", label, start);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
@Override
|
||||
public ResultSet fetchElementsAsync(final String label, final String start, final String end, final List<String> propNames) {
|
||||
Map<String, Object> params = new HashMap<>(3);
|
||||
params.put(LABEL, label);
|
||||
params.put(START_ID, start);
|
||||
params.put(END_ID, end);
|
||||
|
||||
String prefixDsl = exportType == ExportType.VERTEX ? FETCH_RANGE_VERTEX_DSL : FETCH_RANGE_EDGE_DSL;
|
||||
StringBuilder fetchDsl = new StringBuilder(prefixDsl);
|
||||
if (propNames != null) {
|
||||
fetchDsl.append(PART_WITH_PROP_DSL);
|
||||
for (int i = 0; i < propNames.size(); i++) {
|
||||
String propName = "GDB___PK" + String.valueOf(i);
|
||||
params.put(propName, propNames.get(i));
|
||||
|
||||
fetchDsl.append(propName);
|
||||
if (i != propNames.size() - 1) {
|
||||
fetchDsl.append(", ");
|
||||
}
|
||||
}
|
||||
fetchDsl.append("))");
|
||||
}
|
||||
|
||||
try {
|
||||
return runInternalAsync(fetchDsl.toString(), params);
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to fetch range node startId {}, end {} , e {}", start, end, e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
@SuppressWarnings("unchecked")
|
||||
public List<GdbElement> getElement(ResultSet results) {
|
||||
List<GdbElement> elements = new LinkedList<>();
|
||||
try {
|
||||
List<Result> resultList = results.all().get(DEFAULT_TIMEOUT + 1000, TimeUnit.MILLISECONDS);
|
||||
|
||||
resultList.forEach(n -> {
|
||||
Object o = n.getObject();
|
||||
GdbElement element = new GdbElement();
|
||||
if (o instanceof Map) {
|
||||
// project response
|
||||
Object node = ((Map) o).get("node");
|
||||
Object props = ((Map) o).get("props");
|
||||
|
||||
mapNodeToElement(node, element);
|
||||
mapPropToElement((Map<String, Object>) props, element);
|
||||
} else {
|
||||
// range node response
|
||||
mapNodeToElement(n.getObject(), element);
|
||||
}
|
||||
if (element.getId() != null) {
|
||||
elements.add(element);
|
||||
}
|
||||
});
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to get node: {}", e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
return elements;
|
||||
}
|
||||
|
||||
private void mapNodeToElement(Object node, GdbElement element) {
|
||||
if (node instanceof ReferenceVertex) {
|
||||
ReferenceVertex v = (ReferenceVertex) node;
|
||||
|
||||
element.setId((String) v.id());
|
||||
element.setLabel(v.label());
|
||||
} else if (node instanceof ReferenceEdge) {
|
||||
ReferenceEdge e = (ReferenceEdge) node;
|
||||
|
||||
element.setId((String) e.id());
|
||||
element.setLabel(e.label());
|
||||
element.setTo((String) e.inVertex().id());
|
||||
element.setToLabel(e.inVertex().label());
|
||||
element.setFrom((String) e.outVertex().id());
|
||||
element.setFromLabel(e.outVertex().label());
|
||||
}
|
||||
}
|
||||
|
||||
private void mapPropToElement(Map<String, Object> props, GdbElement element) {
|
||||
element.setProperties(props);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, Long> getLabels() {
|
||||
String dsl = exportType == ExportType.VERTEX ? FETCH_VERTEX_LABELS_DSL : FETCH_EDGE_LABELS_DSL;
|
||||
|
||||
try {
|
||||
List<Result> results = runInternal(dsl, null);
|
||||
Map<String, Long> labelMap = new HashMap<>(2);
|
||||
|
||||
Map<?, ?> labels = results.get(0).get(Map.class);
|
||||
labels.forEach((k, v) -> {
|
||||
String label = (String) k;
|
||||
Long count = (Long) v;
|
||||
labelMap.put(label, count);
|
||||
});
|
||||
|
||||
return labelMap;
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to fetch label list, please give special labels and run again, e {}", e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.reader.gdbreader.util;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.GdbReaderErrorCode;
|
||||
import com.alibaba.datax.plugin.reader.gdbreader.Key;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/9/6
|
||||
*/
|
||||
|
||||
public interface ConfigHelper {
|
||||
static void assertConfig(String key, Supplier<Boolean> f) {
|
||||
if (!f.get()) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.BAD_CONFIG_VALUE, key);
|
||||
}
|
||||
}
|
||||
|
||||
static void assertHasContent(Configuration config, String key) {
|
||||
assertConfig(key, () -> StringUtils.isNotBlank(config.getString(key)));
|
||||
}
|
||||
|
||||
static void assertGdbClient(Configuration config) {
|
||||
assertHasContent(config, Key.HOST);
|
||||
assertConfig(Key.PORT, () -> config.getInt(Key.PORT) > 0);
|
||||
|
||||
assertHasContent(config, Key.USERNAME);
|
||||
assertHasContent(config, Key.PASSWORD);
|
||||
}
|
||||
|
||||
static List<String> assertLabels(Configuration config) {
|
||||
Object labels = config.get(Key.LABEL);
|
||||
if (!(labels instanceof List)) {
|
||||
throw DataXException.asDataXException(GdbReaderErrorCode.BAD_CONFIG_VALUE, "labels should be List");
|
||||
}
|
||||
|
||||
List<?> list = (List<?>) labels;
|
||||
List<String> configLabels = new ArrayList<>(0);
|
||||
list.forEach(n -> configLabels.add(String.valueOf(n)));
|
||||
|
||||
return configLabels;
|
||||
}
|
||||
|
||||
static List<Configuration> splitConfig(Configuration config, List<String> labels) {
|
||||
List<Configuration> configs = new ArrayList<>();
|
||||
for (String label : labels) {
|
||||
Configuration conf = config.clone();
|
||||
conf.set(Key.LABEL, label);
|
||||
|
||||
configs.add(conf);
|
||||
}
|
||||
return configs;
|
||||
}
|
||||
|
||||
static Configuration fromClasspath(String name) {
|
||||
try (InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream(name)) {
|
||||
return Configuration.from(is);
|
||||
} catch (IOException e) {
|
||||
throw new IllegalArgumentException("File not found: " + name);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "gdbreader",
|
||||
"class": "com.alibaba.datax.plugin.reader.gdbreader.GdbReader",
|
||||
"description": "useScene: prod. mechanism: connect GDB with gremlin-client, execute 'g.V().propertyMap() or g.E().propertyMap()' to get record",
|
||||
"developer": "alibaba"
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 1
|
||||
},
|
||||
"errorLimit": {
|
||||
"record": 1
|
||||
}
|
||||
},
|
||||
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "gdbreader",
|
||||
"parameter": {
|
||||
"host": "10.218.145.24",
|
||||
"port": 8182,
|
||||
"username": "***",
|
||||
"password": "***",
|
||||
"labelType": "EDGE",
|
||||
"labels": ["label1", "label2"],
|
||||
"column": [
|
||||
{
|
||||
"name": "id",
|
||||
"type": "string",
|
||||
"columnType": "primaryKey"
|
||||
},
|
||||
{
|
||||
"name": "label",
|
||||
"type": "string",
|
||||
"columnType": "primaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "srcId",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "srcLabel",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "dstId",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "dstLabel",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryLabel"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"type": "string",
|
||||
"columnType": "edgeProperty"
|
||||
},
|
||||
{
|
||||
"name": "weight",
|
||||
"type": "double",
|
||||
"columnType": "edgeProperty"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"print": true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
+45
-15
@@ -41,6 +41,14 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
{
|
||||
"random": "60,64",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"random": "100,1000",
|
||||
"type": "long"
|
||||
},
|
||||
{
|
||||
"random": "32,48",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"sliceRecordCount": 1000
|
||||
@@ -55,20 +63,32 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
"password": "***",
|
||||
"writeMode": "INSERT",
|
||||
"labelType": "VERTEX",
|
||||
"label": "${1}",
|
||||
"label": "#{1}",
|
||||
"idTransRule": "none",
|
||||
"session": true,
|
||||
"maxRecordsInBatch": 64,
|
||||
"column": [
|
||||
{
|
||||
"name": "id",
|
||||
"value": "${0}",
|
||||
"value": "#{0}",
|
||||
"type": "string",
|
||||
"columnType": "primaryKey"
|
||||
},
|
||||
{
|
||||
"name": "vertex_propKey",
|
||||
"value": "${2}",
|
||||
"value": "#{2}",
|
||||
"type": "string",
|
||||
"columnType": "vertexSetProperty"
|
||||
},
|
||||
{
|
||||
"name": "vertex_propKey",
|
||||
"value": "#{3}",
|
||||
"type": "long",
|
||||
"columnType": "vertexSetProperty"
|
||||
},
|
||||
{
|
||||
"name": "vertex_propKey2",
|
||||
"value": "#{4}",
|
||||
"type": "string",
|
||||
"columnType": "vertexProperty"
|
||||
}
|
||||
@@ -134,7 +154,7 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
"password": "***",
|
||||
"writeMode": "INSERT",
|
||||
"labelType": "EDGE",
|
||||
"label": "${3}",
|
||||
"label": "#{3}",
|
||||
"idTransRule": "none",
|
||||
"srcIdTransRule": "labelPrefix",
|
||||
"dstIdTransRule": "labelPrefix",
|
||||
@@ -144,25 +164,25 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
"column": [
|
||||
{
|
||||
"name": "id",
|
||||
"value": "${0}",
|
||||
"value": "#{0}",
|
||||
"type": "string",
|
||||
"columnType": "primaryKey"
|
||||
},
|
||||
{
|
||||
"name": "id",
|
||||
"value": "${1}",
|
||||
"value": "#{1}",
|
||||
"type": "string",
|
||||
"columnType": "srcPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "id",
|
||||
"value": "${2}",
|
||||
"value": "#{2}",
|
||||
"type": "string",
|
||||
"columnType": "dstPrimaryKey"
|
||||
},
|
||||
{
|
||||
"name": "edge_propKey",
|
||||
"value": "${4}",
|
||||
"value": "#{4}",
|
||||
"type": "string",
|
||||
"columnType": "edgeProperty"
|
||||
}
|
||||
@@ -199,7 +219,7 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
* 默认值:无
|
||||
|
||||
* **label**
|
||||
* 描述:类型名,即点/边名称; label支持从源列中读取,如${0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 描述:类型名,即点/边名称; label支持从源列中读取,如#{0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
@@ -211,12 +231,12 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
* 默认值:无
|
||||
|
||||
* **srcLabel**
|
||||
* 描述:当label为边时,表示起点的点名称;srcLabel支持从源列中读取,如${0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 描述:当label为边时,表示起点的点名称;srcLabel支持从源列中读取,如#{0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 必选:labelType为边,srcIdTransRule为none时可不填写,否则必填;
|
||||
* 默认值:无
|
||||
|
||||
* **dstLabel**
|
||||
* 描述:当label为边时,表示终点的点名称;dstLabel支持从源列中读取,如${0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 描述:当label为边时,表示终点的点名称;dstLabel支持从源列中读取,如#{0},表示取第一列字段作为label名。源列索引从0开始;
|
||||
* 必选:labelType为边,dstIdTransRule为none时可不填写,否则必填;
|
||||
* 默认值:无
|
||||
|
||||
@@ -271,9 +291,9 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
|
||||
* **column -> value**
|
||||
* 描述:点/边映射关系的字段值;
|
||||
* ${N}表示直接映射源端值,N为源端column索引,从0开始;${0}表示映射源端column第1个字段;
|
||||
* test-${0} 表示源端值做拼接转换,${0}值前/后可添加固定字符串;
|
||||
* ${0}-${1}表示做多字段拼接,也可在任意位置添加固定字符串,如test-${0}-test1-${1}-test2
|
||||
* #{N}表示直接映射源端值,N为源端column索引,从0开始;#{0}表示映射源端column第1个字段;
|
||||
* test-#{0} 表示源端值做拼接转换,#{0}值前/后可添加固定字符串;
|
||||
* #{0}-#{1}表示做多字段拼接,也可在任意位置添加固定字符串,如test-#{0}-test1-#{1}-test2
|
||||
* 必选:是
|
||||
* 默认值:无
|
||||
|
||||
@@ -290,6 +310,7 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
* primaryKey:表示该字段是主键id
|
||||
* 点枚举值:
|
||||
* vertexProperty:labelType为点时,表示该字段是点的普通属性
|
||||
* vertexSetProperty:labelType为点时,表示该字段是点的SET属性,value是SET属性中的一个属性值
|
||||
* vertexJsonProperty:labelType为点时,表示是点json属性,value结构请见备注**json properties示例**,点配置最多只允许出现一个json属性;
|
||||
* 边枚举值:
|
||||
* srcPrimaryKey:labelType为边时,表示该字段是起点主键id
|
||||
@@ -305,6 +326,14 @@ GDBWriter通过DataX框架获取Reader生成的协议数据,使用`g.addV/E(GD
|
||||
> {"k":"age","t":"int","v":"20"},
|
||||
> {"k":"sex","t":"string","v":"male"}
|
||||
> ]}
|
||||
>
|
||||
> # json格式同样支持给点添加SET属性,格式如下
|
||||
> {"properties":[
|
||||
> {"k":"name","t":"string","v":"tom","c":"set"},
|
||||
> {"k":"name","t":"string","v":"jack","c":"set"},
|
||||
> {"k":"age","t":"int","v":"20"},
|
||||
> {"k":"sex","t":"string","v":"male"}
|
||||
> ]}
|
||||
> ```
|
||||
|
||||
## 4 性能报告
|
||||
@@ -367,4 +396,5 @@ DataX压测机器
|
||||
- GDBWriter插件与用户查询DSL使用相同的GDB实例端口,导入时可能会影响查询性能
|
||||
|
||||
## FAQ
|
||||
无
|
||||
1. 使用SET属性需要升级GDB实例到`1.0.20`版本及以上。
|
||||
2. 边只支持普通单值属性,不能给边写SET属性数据。
|
||||
|
||||
@@ -1,10 +1,5 @@
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.*;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||
@@ -18,24 +13,33 @@ import com.alibaba.datax.plugin.writer.gdbwriter.mapping.MappingRule;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.mapping.MappingRuleFactory;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbElement;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbGraph;
|
||||
|
||||
import groovy.lang.Tuple2;
|
||||
import io.netty.util.concurrent.DefaultThreadFactory;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
public class GdbWriter extends Writer {
|
||||
private static final Logger log = LoggerFactory.getLogger(GdbWriter.class);
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Future;
|
||||
import java.util.concurrent.LinkedBlockingDeque;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.function.Function;
|
||||
|
||||
private static Function<Record, GdbElement> mapper = null;
|
||||
private static GdbGraph globalGraph = null;
|
||||
private static boolean session = false;
|
||||
public class GdbWriter extends Writer {
|
||||
private static final Logger log = LoggerFactory.getLogger(GdbWriter.class);
|
||||
|
||||
private static Function<Record, GdbElement> mapper = null;
|
||||
private static GdbGraph globalGraph = null;
|
||||
private static boolean session = false;
|
||||
|
||||
/**
|
||||
* Job 中的方法仅执行一次,Task 中方法会由框架启动多个 Task 线程并行执行。
|
||||
* <p/>
|
||||
* 整个 Writer 执行流程是:
|
||||
*
|
||||
* <pre>
|
||||
* Job类init-->prepare-->split
|
||||
*
|
||||
@@ -46,17 +50,16 @@ public class GdbWriter extends Writer {
|
||||
* </pre>
|
||||
*/
|
||||
public static class Job extends Writer.Job {
|
||||
private static final Logger LOG = LoggerFactory
|
||||
.getLogger(Job.class);
|
||||
private static final Logger LOG = LoggerFactory.getLogger(Job.class);
|
||||
|
||||
private Configuration jobConfig = null;
|
||||
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
LOG.info("GDB datax plugin writer job init begin ...");
|
||||
this.jobConfig = getPluginJobConf();
|
||||
GdbWriterConfig.of(this.jobConfig);
|
||||
LOG.info("GDB datax plugin writer job init end.");
|
||||
LOG.info("GDB datax plugin writer job init begin ...");
|
||||
this.jobConfig = getPluginJobConf();
|
||||
GdbWriterConfig.of(this.jobConfig);
|
||||
LOG.info("GDB datax plugin writer job init end.");
|
||||
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
@@ -71,37 +74,37 @@ public class GdbWriter extends Writer {
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:如果 Job 中有需要进行数据同步之前的处理,可以在此处完成,如果没有必要则可以直接去掉。
|
||||
*/
|
||||
super.prepare();
|
||||
super.prepare();
|
||||
|
||||
MappingRule rule = MappingRuleFactory.getInstance().createV2(jobConfig);
|
||||
final MappingRule rule = MappingRuleFactory.getInstance().createV2(this.jobConfig);
|
||||
|
||||
mapper = new DefaultGdbMapper().getMapper(rule);
|
||||
session = jobConfig.getBool(Key.SESSION_STATE, false);
|
||||
mapper = new DefaultGdbMapper(this.jobConfig).getMapper(rule);
|
||||
session = this.jobConfig.getBool(Key.SESSION_STATE, false);
|
||||
|
||||
/**
|
||||
* client connect check before task
|
||||
*/
|
||||
try {
|
||||
globalGraph = GdbGraphManager.instance().getGraph(jobConfig, false);
|
||||
} catch (RuntimeException e) {
|
||||
globalGraph = GdbGraphManager.instance().getGraph(this.jobConfig, false);
|
||||
} catch (final RuntimeException e) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.FAIL_CLIENT_CONNECT, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Configuration> split(int mandatoryNumber) {
|
||||
public List<Configuration> split(final int mandatoryNumber) {
|
||||
/**
|
||||
* 注意:此方法仅执行一次。
|
||||
* 最佳实践:通常采用工具静态类完成把 Job 配置切分成多个 Task 配置的工作。
|
||||
* 这里的 mandatoryNumber 是强制必须切分的份数。
|
||||
*/
|
||||
LOG.info("split begin...");
|
||||
List<Configuration> configurationList = new ArrayList<Configuration>();
|
||||
for (int i = 0; i < mandatoryNumber; i++) {
|
||||
configurationList.add(this.jobConfig.clone());
|
||||
}
|
||||
LOG.info("split end...");
|
||||
return configurationList;
|
||||
LOG.info("split begin...");
|
||||
final List<Configuration> configurationList = new ArrayList<Configuration>();
|
||||
for (int i = 0; i < mandatoryNumber; i++) {
|
||||
configurationList.add(this.jobConfig.clone());
|
||||
}
|
||||
LOG.info("split end...");
|
||||
return configurationList;
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -127,7 +130,7 @@ public class GdbWriter extends Writer {
|
||||
public static class Task extends Writer.Task {
|
||||
|
||||
private Configuration taskConfig;
|
||||
|
||||
|
||||
private int failed = 0;
|
||||
private int batchRecords;
|
||||
private ExecutorService submitService = null;
|
||||
@@ -139,24 +142,24 @@ public class GdbWriter extends Writer {
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:此处通过对 taskConfig 配置的读取,进而初始化一些资源为 startWrite()做准备。
|
||||
*/
|
||||
this.taskConfig = super.getPluginJobConf();
|
||||
batchRecords = taskConfig.getInt(Key.MAX_RECORDS_IN_BATCH, GdbWriterConfig.DEFAULT_RECORD_NUM_IN_BATCH);
|
||||
submitService = new ThreadPoolExecutor(1, 1, 0L,
|
||||
TimeUnit.MILLISECONDS, new LinkedBlockingDeque<>(), new DefaultThreadFactory("submit-dsl"));
|
||||
this.taskConfig = super.getPluginJobConf();
|
||||
this.batchRecords = this.taskConfig.getInt(Key.MAX_RECORDS_IN_BATCH, GdbWriterConfig.DEFAULT_RECORD_NUM_IN_BATCH);
|
||||
this.submitService = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingDeque<>(),
|
||||
new DefaultThreadFactory("submit-dsl"));
|
||||
|
||||
if (!session) {
|
||||
graph = globalGraph;
|
||||
} else {
|
||||
/**
|
||||
* 分批创建session client,由于服务端groovy编译性能的限制
|
||||
*/
|
||||
try {
|
||||
Thread.sleep((getTaskId()/10)*10000);
|
||||
} catch (Exception e) {
|
||||
// ...
|
||||
}
|
||||
graph = GdbGraphManager.instance().getGraph(taskConfig, session);
|
||||
}
|
||||
if (!session) {
|
||||
this.graph = globalGraph;
|
||||
} else {
|
||||
/**
|
||||
* 分批创建session client,由于服务端groovy编译性能的限制
|
||||
*/
|
||||
try {
|
||||
Thread.sleep((getTaskId() / 10) * 10000);
|
||||
} catch (final Exception e) {
|
||||
// ...
|
||||
}
|
||||
this.graph = GdbGraphManager.instance().getGraph(this.taskConfig, session);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -165,64 +168,69 @@ public class GdbWriter extends Writer {
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:如果 Task 中有需要进行数据同步之前的处理,可以在此处完成,如果没有必要则可以直接去掉。
|
||||
*/
|
||||
super.prepare();
|
||||
super.prepare();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startWrite(RecordReceiver recordReceiver) {
|
||||
public void startWrite(final RecordReceiver recordReceiver) {
|
||||
/**
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:此处适当封装确保简洁清晰完成数据写入工作。
|
||||
*/
|
||||
Record r;
|
||||
Future<Boolean> future = null;
|
||||
List<Tuple2<Record, GdbElement>> records = new ArrayList<>(batchRecords);
|
||||
Record r;
|
||||
Future<Boolean> future = null;
|
||||
List<Tuple2<Record, GdbElement>> records = new ArrayList<>(this.batchRecords);
|
||||
|
||||
while ((r = recordReceiver.getFromReader()) != null) {
|
||||
records.add(new Tuple2<>(r, mapper.apply(r)));
|
||||
while ((r = recordReceiver.getFromReader()) != null) {
|
||||
try {
|
||||
records.add(new Tuple2<>(r, mapper.apply(r)));
|
||||
} catch (final Exception ex) {
|
||||
getTaskPluginCollector().collectDirtyRecord(r, ex);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (records.size() >= batchRecords) {
|
||||
wait4Submit(future);
|
||||
if (records.size() >= this.batchRecords) {
|
||||
wait4Submit(future);
|
||||
|
||||
final List<Tuple2<Record, GdbElement>> batch = records;
|
||||
future = submitService.submit(() -> batchCommitRecords(batch));
|
||||
records = new ArrayList<>(batchRecords);
|
||||
}
|
||||
}
|
||||
final List<Tuple2<Record, GdbElement>> batch = records;
|
||||
future = this.submitService.submit(() -> batchCommitRecords(batch));
|
||||
records = new ArrayList<>(this.batchRecords);
|
||||
}
|
||||
}
|
||||
|
||||
wait4Submit(future);
|
||||
if (!records.isEmpty()) {
|
||||
final List<Tuple2<Record, GdbElement>> batch = records;
|
||||
future = submitService.submit(() -> batchCommitRecords(batch));
|
||||
wait4Submit(future);
|
||||
}
|
||||
wait4Submit(future);
|
||||
if (!records.isEmpty()) {
|
||||
final List<Tuple2<Record, GdbElement>> batch = records;
|
||||
future = this.submitService.submit(() -> batchCommitRecords(batch));
|
||||
wait4Submit(future);
|
||||
}
|
||||
}
|
||||
|
||||
private void wait4Submit(Future<Boolean> future) {
|
||||
if (future == null) {
|
||||
return;
|
||||
}
|
||||
private void wait4Submit(final Future<Boolean> future) {
|
||||
if (future == null) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
future.get();
|
||||
} catch (Exception e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
try {
|
||||
future.get();
|
||||
} catch (final Exception e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
}
|
||||
|
||||
private boolean batchCommitRecords(final List<Tuple2<Record, GdbElement>> records) {
|
||||
TaskPluginCollector collector = getTaskPluginCollector();
|
||||
try {
|
||||
List<Tuple2<Record, Exception>> errors = graph.add(records);
|
||||
errors.forEach(t -> collector.collectDirtyRecord(t.getFirst(), t.getSecond()));
|
||||
failed += errors.size();
|
||||
} catch (Exception e) {
|
||||
records.forEach(t -> collector.collectDirtyRecord(t.getFirst(), e));
|
||||
failed += records.size();
|
||||
}
|
||||
final TaskPluginCollector collector = getTaskPluginCollector();
|
||||
try {
|
||||
final List<Tuple2<Record, Exception>> errors = this.graph.add(records);
|
||||
errors.forEach(t -> collector.collectDirtyRecord(t.getFirst(), t.getSecond()));
|
||||
this.failed += errors.size();
|
||||
} catch (final Exception e) {
|
||||
records.forEach(t -> collector.collectDirtyRecord(t.getFirst(), e));
|
||||
this.failed += records.size();
|
||||
}
|
||||
|
||||
records.clear();
|
||||
return true;
|
||||
records.clear();
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -231,7 +239,7 @@ public class GdbWriter extends Writer {
|
||||
* 注意:此方法每个 Task 都会执行一次。
|
||||
* 最佳实践:如果 Task 中有需要进行数据同步之后的后续处理,可以在此处完成。
|
||||
*/
|
||||
log.info("Task done, dirty record count - {}", failed);
|
||||
log.info("Task done, dirty record count - {}", this.failed);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -241,9 +249,9 @@ public class GdbWriter extends Writer {
|
||||
* 最佳实践:通常配合Task 中的 post() 方法一起完成 Task 的资源释放。
|
||||
*/
|
||||
if (session) {
|
||||
graph.close();
|
||||
this.graph.close();
|
||||
}
|
||||
submitService.shutdown();
|
||||
this.submitService.shutdown();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
+1
-2
@@ -27,7 +27,6 @@ public enum GdbWriterErrorCode implements ErrorCode {
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code,
|
||||
this.description);
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code, this.description);
|
||||
}
|
||||
}
|
||||
@@ -6,136 +6,164 @@ public final class Key {
|
||||
* 此处声明插件用到的需要插件使用者提供的配置项
|
||||
*/
|
||||
|
||||
public final static String HOST = "host";
|
||||
public final static String PORT = "port";
|
||||
public final static String HOST = "host";
|
||||
public final static String PORT = "port";
|
||||
public final static String USERNAME = "username";
|
||||
public static final String PASSWORD = "password";
|
||||
public static final String PASSWORD = "password";
|
||||
|
||||
/**
|
||||
* import type and mode
|
||||
*/
|
||||
public static final String IMPORT_TYPE = "labelType";
|
||||
public static final String UPDATE_MODE = "writeMode";
|
||||
/**
|
||||
* import type and mode
|
||||
*/
|
||||
public static final String IMPORT_TYPE = "labelType";
|
||||
public static final String UPDATE_MODE = "writeMode";
|
||||
|
||||
/**
|
||||
* label prefix issue
|
||||
*/
|
||||
public static final String ID_TRANS_RULE = "idTransRule";
|
||||
public static final String SRC_ID_TRANS_RULE = "srcIdTransRule";
|
||||
public static final String DST_ID_TRANS_RULE = "dstIdTransRule";
|
||||
/**
|
||||
* label prefix issue
|
||||
*/
|
||||
public static final String ID_TRANS_RULE = "idTransRule";
|
||||
public static final String SRC_ID_TRANS_RULE = "srcIdTransRule";
|
||||
public static final String DST_ID_TRANS_RULE = "dstIdTransRule";
|
||||
|
||||
public static final String LABEL = "label";
|
||||
public static final String SRC_LABEL = "srcLabel";
|
||||
public static final String DST_LABEL = "dstLabel";
|
||||
public static final String LABEL = "label";
|
||||
public static final String SRC_LABEL = "srcLabel";
|
||||
public static final String DST_LABEL = "dstLabel";
|
||||
|
||||
public static final String MAPPING = "mapping";
|
||||
public static final String MAPPING = "mapping";
|
||||
|
||||
/**
|
||||
* column define in Gdb
|
||||
*/
|
||||
public static final String COLUMN = "column";
|
||||
public static final String COLUMN_NAME = "name";
|
||||
public static final String COLUMN_VALUE = "value";
|
||||
public static final String COLUMN_TYPE = "type";
|
||||
public static final String COLUMN_NODE_TYPE = "columnType";
|
||||
/**
|
||||
* column define in Gdb
|
||||
*/
|
||||
public static final String COLUMN = "column";
|
||||
public static final String COLUMN_NAME = "name";
|
||||
public static final String COLUMN_VALUE = "value";
|
||||
public static final String COLUMN_TYPE = "type";
|
||||
public static final String COLUMN_NODE_TYPE = "columnType";
|
||||
|
||||
/**
|
||||
* Gdb Vertex/Edge elements
|
||||
*/
|
||||
public static final String ID = "id";
|
||||
public static final String FROM = "from";
|
||||
public static final String TO = "to";
|
||||
public static final String PROPERTIES = "properties";
|
||||
public static final String PROP_KEY = "name";
|
||||
public static final String PROP_VALUE = "value";
|
||||
public static final String PROP_TYPE = "type";
|
||||
/**
|
||||
* Gdb Vertex/Edge elements
|
||||
*/
|
||||
public static final String ID = "id";
|
||||
public static final String FROM = "from";
|
||||
public static final String TO = "to";
|
||||
public static final String PROPERTIES = "properties";
|
||||
public static final String PROP_KEY = "name";
|
||||
public static final String PROP_VALUE = "value";
|
||||
public static final String PROP_TYPE = "type";
|
||||
|
||||
public static final String PROPERTIES_JSON_STR = "propertiesJsonStr";
|
||||
public static final String MAX_PROPERTIES_BATCH_NUM = "maxPropertiesBatchNumber";
|
||||
public static final String PROPERTIES_JSON_STR = "propertiesJsonStr";
|
||||
public static final String MAX_PROPERTIES_BATCH_NUM = "maxPropertiesBatchNumber";
|
||||
|
||||
/**
|
||||
* session less client configure for connect pool
|
||||
*/
|
||||
public static final String MAX_IN_PROCESS_PER_CONNECTION = "maxInProcessPerConnection";
|
||||
public static final String MAX_CONNECTION_POOL_SIZE = "maxConnectionPoolSize";
|
||||
public static final String MAX_SIMULTANEOUS_USAGE_PER_CONNECTION = "maxSimultaneousUsagePerConnection";
|
||||
/**
|
||||
* session less client configure for connect pool
|
||||
*/
|
||||
public static final String MAX_IN_PROCESS_PER_CONNECTION = "maxInProcessPerConnection";
|
||||
public static final String MAX_CONNECTION_POOL_SIZE = "maxConnectionPoolSize";
|
||||
public static final String MAX_SIMULTANEOUS_USAGE_PER_CONNECTION = "maxSimultaneousUsagePerConnection";
|
||||
|
||||
public static final String MAX_RECORDS_IN_BATCH = "maxRecordsInBatch";
|
||||
public static final String SESSION_STATE = "session";
|
||||
public static final String MAX_RECORDS_IN_BATCH = "maxRecordsInBatch";
|
||||
public static final String SESSION_STATE = "session";
|
||||
|
||||
public static enum ImportType {
|
||||
/**
|
||||
* Import vertices
|
||||
*/
|
||||
VERTEX,
|
||||
/**
|
||||
* Import edges
|
||||
*/
|
||||
EDGE;
|
||||
}
|
||||
|
||||
public static enum UpdateMode {
|
||||
/**
|
||||
* Insert new records, fail if exists
|
||||
*/
|
||||
INSERT,
|
||||
/**
|
||||
* Skip this record if exists
|
||||
*/
|
||||
SKIP,
|
||||
/**
|
||||
* Update property of this record if exists
|
||||
*/
|
||||
MERGE;
|
||||
}
|
||||
/**
|
||||
* request length limit, include gdb element string length GDB字段长度限制配置,可分别配置各字段的限制,超过限制的记录会当脏数据处理
|
||||
*/
|
||||
public static final String MAX_GDB_STRING_LENGTH = "maxStringLengthLimit";
|
||||
public static final String MAX_GDB_ID_LENGTH = "maxIdStringLengthLimit";
|
||||
public static final String MAX_GDB_LABEL_LENGTH = "maxLabelStringLengthLimit";
|
||||
public static final String MAX_GDB_PROP_KEY_LENGTH = "maxPropKeyStringLengthLimit";
|
||||
public static final String MAX_GDB_PROP_VALUE_LENGTH = "maxPropValueStringLengthLimit";
|
||||
|
||||
public static enum ColumnType {
|
||||
/**
|
||||
* vertex or edge id
|
||||
*/
|
||||
primaryKey,
|
||||
public static final String MAX_GDB_REQUEST_LENGTH = "maxRequestLengthLimit";
|
||||
|
||||
/**
|
||||
* vertex property
|
||||
*/
|
||||
vertexProperty,
|
||||
public static enum ImportType {
|
||||
/**
|
||||
* Import vertices
|
||||
*/
|
||||
VERTEX,
|
||||
/**
|
||||
* Import edges
|
||||
*/
|
||||
EDGE;
|
||||
}
|
||||
|
||||
/**
|
||||
* start vertex id of edge
|
||||
*/
|
||||
srcPrimaryKey,
|
||||
public static enum UpdateMode {
|
||||
/**
|
||||
* Insert new records, fail if exists
|
||||
*/
|
||||
INSERT,
|
||||
/**
|
||||
* Skip this record if exists
|
||||
*/
|
||||
SKIP,
|
||||
/**
|
||||
* Update property of this record if exists
|
||||
*/
|
||||
MERGE;
|
||||
}
|
||||
|
||||
/**
|
||||
* end vertex id of edge
|
||||
*/
|
||||
dstPrimaryKey,
|
||||
public static enum ColumnType {
|
||||
/**
|
||||
* vertex or edge id
|
||||
*/
|
||||
primaryKey,
|
||||
|
||||
/**
|
||||
* edge property
|
||||
*/
|
||||
edgeProperty,
|
||||
/**
|
||||
* vertex property
|
||||
*/
|
||||
vertexProperty,
|
||||
|
||||
/**
|
||||
* vertex json style property
|
||||
*/
|
||||
vertexJsonProperty,
|
||||
/**
|
||||
* vertex setProperty
|
||||
*/
|
||||
vertexSetProperty,
|
||||
|
||||
/**
|
||||
* edge json style property
|
||||
*/
|
||||
edgeJsonProperty
|
||||
}
|
||||
/**
|
||||
* start vertex id of edge
|
||||
*/
|
||||
srcPrimaryKey,
|
||||
|
||||
public static enum IdTransRule {
|
||||
/**
|
||||
* vertex or edge id with 'label' prefix
|
||||
*/
|
||||
labelPrefix,
|
||||
/**
|
||||
* end vertex id of edge
|
||||
*/
|
||||
dstPrimaryKey,
|
||||
|
||||
/**
|
||||
* vertex or edge id raw
|
||||
*/
|
||||
none
|
||||
}
|
||||
/**
|
||||
* edge property
|
||||
*/
|
||||
edgeProperty,
|
||||
|
||||
/**
|
||||
* vertex json style property
|
||||
*/
|
||||
vertexJsonProperty,
|
||||
|
||||
/**
|
||||
* edge json style property
|
||||
*/
|
||||
edgeJsonProperty
|
||||
}
|
||||
|
||||
public static enum IdTransRule {
|
||||
/**
|
||||
* vertex or edge id with 'label' prefix
|
||||
*/
|
||||
labelPrefix,
|
||||
|
||||
/**
|
||||
* vertex or edge id raw
|
||||
*/
|
||||
none
|
||||
}
|
||||
|
||||
public static enum PropertyType {
|
||||
/**
|
||||
* single Vertex Property
|
||||
*/
|
||||
single,
|
||||
|
||||
/**
|
||||
* set Vertex Property
|
||||
*/
|
||||
set
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
+22
-22
@@ -3,37 +3,37 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.client;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbGraph;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.ScriptGdbGraph;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
public class GdbGraphManager implements AutoCloseable {
|
||||
private static final GdbGraphManager instance = new GdbGraphManager();
|
||||
|
||||
private List<GdbGraph> graphs = new ArrayList<>();
|
||||
|
||||
public static GdbGraphManager instance() {
|
||||
return instance;
|
||||
}
|
||||
private static final GdbGraphManager INSTANCE = new GdbGraphManager();
|
||||
|
||||
public GdbGraph getGraph(Configuration config, boolean session) {
|
||||
GdbGraph graph = new ScriptGdbGraph(config, session);
|
||||
graphs.add(graph);
|
||||
return graph;
|
||||
}
|
||||
private List<GdbGraph> graphs = new ArrayList<>();
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
for(GdbGraph graph : graphs) {
|
||||
graph.close();
|
||||
}
|
||||
graphs.clear();
|
||||
}
|
||||
public static GdbGraphManager instance() {
|
||||
return INSTANCE;
|
||||
}
|
||||
|
||||
public GdbGraph getGraph(final Configuration config, final boolean session) {
|
||||
final GdbGraph graph = new ScriptGdbGraph(config, session);
|
||||
this.graphs.add(graph);
|
||||
return graph;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
for (final GdbGraph graph : this.graphs) {
|
||||
graph.close();
|
||||
}
|
||||
this.graphs.clear();
|
||||
}
|
||||
}
|
||||
|
||||
+26
-22
@@ -3,39 +3,43 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.client;
|
||||
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.util.ConfigHelper.assertConfig;
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.util.ConfigHelper.assertHasContent;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.util.ConfigHelper.*;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
public class GdbWriterConfig {
|
||||
public static final int DEFAULT_MAX_IN_PROCESS_PER_CONNECTION = 4;
|
||||
public static final int DEFAULT_MAX_CONNECTION_POOL_SIZE = 8;
|
||||
public static final int DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION = 8;
|
||||
public static final int DEFAULT_BATCH_PROPERTY_NUM = 30;
|
||||
public static final int DEFAULT_RECORD_NUM_IN_BATCH = 16;
|
||||
public static final int DEFAULT_MAX_IN_PROCESS_PER_CONNECTION = 4;
|
||||
public static final int DEFAULT_MAX_CONNECTION_POOL_SIZE = 8;
|
||||
public static final int DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION = 8;
|
||||
public static final int DEFAULT_BATCH_PROPERTY_NUM = 30;
|
||||
public static final int DEFAULT_RECORD_NUM_IN_BATCH = 16;
|
||||
|
||||
private Configuration config;
|
||||
public static final int MAX_STRING_LENGTH = 10240;
|
||||
public static final int MAX_REQUEST_LENGTH = 65535 - 1000;
|
||||
|
||||
private GdbWriterConfig(Configuration config) {
|
||||
this.config = config;
|
||||
private Configuration config;
|
||||
|
||||
validate();
|
||||
}
|
||||
private GdbWriterConfig(final Configuration config) {
|
||||
this.config = config;
|
||||
|
||||
private void validate() {
|
||||
assertHasContent(config, Key.HOST);
|
||||
assertConfig(Key.PORT, () -> config.getInt(Key.PORT) > 0);
|
||||
validate();
|
||||
}
|
||||
|
||||
assertHasContent(config, Key.USERNAME);
|
||||
assertHasContent(config, Key.PASSWORD);
|
||||
}
|
||||
|
||||
public static GdbWriterConfig of(Configuration config) {
|
||||
return new GdbWriterConfig(config);
|
||||
}
|
||||
public static GdbWriterConfig of(final Configuration config) {
|
||||
return new GdbWriterConfig(config);
|
||||
}
|
||||
|
||||
private void validate() {
|
||||
assertHasContent(this.config, Key.HOST);
|
||||
assertConfig(Key.PORT, () -> this.config.getInt(Key.PORT) > 0);
|
||||
|
||||
assertHasContent(this.config, Key.USERNAME);
|
||||
assertHasContent(this.config, Key.PASSWORD);
|
||||
}
|
||||
}
|
||||
|
||||
+164
-150
@@ -3,6 +3,8 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.mapping;
|
||||
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.Key.ImportType.VERTEX;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
@@ -12,179 +14,191 @@ import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.fastjson.JSONArray;
|
||||
import com.alibaba.fastjson.JSONObject;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbEdge;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbElement;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbVertex;
|
||||
import com.alibaba.fastjson.JSONArray;
|
||||
import com.alibaba.fastjson.JSONObject;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.Key.ImportType.VERTEX;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
@Slf4j
|
||||
public class DefaultGdbMapper implements GdbMapper {
|
||||
private static final Pattern STR_PATTERN = Pattern.compile("\\$\\{(\\d+)}");
|
||||
private static final Pattern NORMAL_PATTERN = Pattern.compile("^\\$\\{(\\d+)}$");
|
||||
private static final Pattern STR_DOLLAR_PATTERN = Pattern.compile("\\$\\{(\\d+)}");
|
||||
private static final Pattern NORMAL_DOLLAR_PATTERN = Pattern.compile("^\\$\\{(\\d+)}$");
|
||||
|
||||
@Override
|
||||
public Function<Record, GdbElement> getMapper(MappingRule rule) {
|
||||
return r -> {
|
||||
GdbElement e = (rule.getImportType() == VERTEX) ? new GdbVertex() : new GdbEdge();
|
||||
forElement(rule).accept(r, e);
|
||||
return e;
|
||||
private static final Pattern STR_NUM_PATTERN = Pattern.compile("#\\{(\\d+)}");
|
||||
private static final Pattern NORMAL_NUM_PATTERN = Pattern.compile("^#\\{(\\d+)}$");
|
||||
|
||||
public DefaultGdbMapper() {}
|
||||
|
||||
public DefaultGdbMapper(final Configuration config) {
|
||||
MapperConfig.getInstance().updateConfig(config);
|
||||
}
|
||||
|
||||
private static BiConsumer<Record, GdbElement> forElement(final MappingRule rule) {
|
||||
final boolean numPattern = rule.isNumPattern();
|
||||
final List<BiConsumer<Record, GdbElement>> properties = new ArrayList<>();
|
||||
for (final MappingRule.PropertyMappingRule propRule : rule.getProperties()) {
|
||||
final Function<Record, String> keyFunc = forStrColumn(numPattern, propRule.getKey());
|
||||
|
||||
if (propRule.getValueType() == ValueType.STRING) {
|
||||
final Function<Record, String> valueFunc = forStrColumn(numPattern, propRule.getValue());
|
||||
properties.add((r, e) -> {
|
||||
e.addProperty(keyFunc.apply(r), valueFunc.apply(r), propRule.getPType());
|
||||
});
|
||||
} else {
|
||||
final Function<Record, Object> valueFunc =
|
||||
forObjColumn(numPattern, propRule.getValue(), propRule.getValueType());
|
||||
properties.add((r, e) -> {
|
||||
e.addProperty(keyFunc.apply(r), valueFunc.apply(r), propRule.getPType());
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (rule.getPropertiesJsonStr() != null) {
|
||||
final Function<Record, String> jsonFunc = forStrColumn(numPattern, rule.getPropertiesJsonStr());
|
||||
properties.add((r, e) -> {
|
||||
final String propertiesStr = jsonFunc.apply(r);
|
||||
final JSONObject root = (JSONObject)JSONObject.parse(propertiesStr);
|
||||
final JSONArray propertiesList = root.getJSONArray("properties");
|
||||
|
||||
for (final Object object : propertiesList) {
|
||||
final JSONObject jsonObject = (JSONObject)object;
|
||||
final String key = jsonObject.getString("k");
|
||||
final String name = jsonObject.getString("v");
|
||||
final String type = jsonObject.getString("t");
|
||||
final String card = jsonObject.getString("c");
|
||||
|
||||
if (key == null || name == null) {
|
||||
continue;
|
||||
}
|
||||
addToProperties(e, key, name, type, card);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
final BiConsumer<Record, GdbElement> ret = (r, e) -> {
|
||||
final String label = forStrColumn(numPattern, rule.getLabel()).apply(r);
|
||||
String id = forStrColumn(numPattern, rule.getId()).apply(r);
|
||||
|
||||
if (rule.getImportType() == Key.ImportType.EDGE) {
|
||||
final String to = forStrColumn(numPattern, rule.getTo()).apply(r);
|
||||
final String from = forStrColumn(numPattern, rule.getFrom()).apply(r);
|
||||
if (to == null || from == null) {
|
||||
log.error("invalid record to: {} , from: {}", to, from);
|
||||
throw new IllegalArgumentException("to or from missed in edge");
|
||||
}
|
||||
((GdbEdge)e).setTo(to);
|
||||
((GdbEdge)e).setFrom(from);
|
||||
|
||||
// generate UUID for edge
|
||||
if (id == null) {
|
||||
id = UUID.randomUUID().toString();
|
||||
}
|
||||
}
|
||||
|
||||
if (id == null || label == null) {
|
||||
log.error("invalid record id: {} , label: {}", id, label);
|
||||
throw new IllegalArgumentException("id or label missed");
|
||||
}
|
||||
|
||||
e.setId(id);
|
||||
e.setLabel(label);
|
||||
|
||||
properties.forEach(p -> p.accept(r, e));
|
||||
};
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
private static BiConsumer<Record, GdbElement> forElement(MappingRule rule) {
|
||||
List<BiConsumer<Record, GdbElement>> properties = new ArrayList<>();
|
||||
for (MappingRule.PropertyMappingRule propRule : rule.getProperties()) {
|
||||
Function<Record, String> keyFunc = forStrColumn(propRule.getKey());
|
||||
private static Function<Record, Object> forObjColumn(final boolean numPattern, final String rule, final ValueType type) {
|
||||
final Pattern pattern = numPattern ? NORMAL_NUM_PATTERN : NORMAL_DOLLAR_PATTERN;
|
||||
final Matcher m = pattern.matcher(rule);
|
||||
if (m.matches()) {
|
||||
final int index = Integer.valueOf(m.group(1));
|
||||
return r -> type.applyColumn(r.getColumn(index));
|
||||
} else {
|
||||
return r -> type.fromStrFunc(rule);
|
||||
}
|
||||
}
|
||||
|
||||
if (propRule.getValueType() == ValueType.STRING) {
|
||||
final Function<Record, String> valueFunc = forStrColumn(propRule.getValue());
|
||||
properties.add((r, e) -> {
|
||||
String k = keyFunc.apply(r);
|
||||
String v = valueFunc.apply(r);
|
||||
if (k != null && v != null) {
|
||||
e.getProperties().put(k, v);
|
||||
}
|
||||
});
|
||||
} else {
|
||||
final Function<Record, Object> valueFunc = forObjColumn(propRule.getValue(), propRule.getValueType());
|
||||
properties.add((r, e) -> {
|
||||
String k = keyFunc.apply(r);
|
||||
Object v = valueFunc.apply(r);
|
||||
if (k != null && v != null) {
|
||||
e.getProperties().put(k, v);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
private static Function<Record, String> forStrColumn(final boolean numPattern, final String rule) {
|
||||
final List<BiConsumer<StringBuilder, Record>> list = new ArrayList<>();
|
||||
final Pattern pattern = numPattern ? STR_NUM_PATTERN : STR_DOLLAR_PATTERN;
|
||||
final Matcher m = pattern.matcher(rule);
|
||||
int last = 0;
|
||||
while (m.find()) {
|
||||
final String index = m.group(1);
|
||||
// as simple integer index.
|
||||
final int i = Integer.parseInt(index);
|
||||
|
||||
if (rule.getPropertiesJsonStr() != null) {
|
||||
Function<Record, String> jsonFunc = forStrColumn(rule.getPropertiesJsonStr());
|
||||
properties.add((r, e) -> {
|
||||
String propertiesStr = jsonFunc.apply(r);
|
||||
JSONObject root = (JSONObject)JSONObject.parse(propertiesStr);
|
||||
JSONArray propertiesList = root.getJSONArray("properties");
|
||||
final int tmp = last;
|
||||
final int start = m.start();
|
||||
list.add((sb, record) -> {
|
||||
sb.append(rule.subSequence(tmp, start));
|
||||
if (record.getColumn(i) != null && record.getColumn(i).getByteSize() > 0) {
|
||||
sb.append(record.getColumn(i).asString());
|
||||
}
|
||||
});
|
||||
|
||||
for (Object object : propertiesList) {
|
||||
JSONObject jsonObject = (JSONObject)object;
|
||||
String key = jsonObject.getString("k");
|
||||
String name = jsonObject.getString("v");
|
||||
String type = jsonObject.getString("t");
|
||||
last = m.end();
|
||||
}
|
||||
|
||||
if (key == null || name == null) {
|
||||
continue;
|
||||
}
|
||||
addToProperties(e, key, name, type);
|
||||
}
|
||||
});
|
||||
}
|
||||
final int tmp = last;
|
||||
list.add((sb, record) -> {
|
||||
sb.append(rule.subSequence(tmp, rule.length()));
|
||||
});
|
||||
|
||||
BiConsumer<Record, GdbElement> ret = (r, e) -> {
|
||||
String label = forStrColumn(rule.getLabel()).apply(r);
|
||||
String id = forStrColumn(rule.getId()).apply(r);
|
||||
return r -> {
|
||||
final StringBuilder sb = new StringBuilder();
|
||||
list.forEach(c -> c.accept(sb, r));
|
||||
final String res = sb.toString();
|
||||
return res.isEmpty() ? null : res;
|
||||
};
|
||||
}
|
||||
|
||||
if (rule.getImportType() == Key.ImportType.EDGE) {
|
||||
String to = forStrColumn(rule.getTo()).apply(r);
|
||||
String from = forStrColumn(rule.getFrom()).apply(r);
|
||||
if (to == null || from == null) {
|
||||
log.error("invalid record to: {} , from: {}", to, from);
|
||||
throw new IllegalArgumentException("to or from missed in edge");
|
||||
}
|
||||
((GdbEdge)e).setTo(to);
|
||||
((GdbEdge)e).setFrom(from);
|
||||
private static boolean addToProperties(final GdbElement e, final String key, final String value, final String type, final String card) {
|
||||
final Object pValue;
|
||||
final ValueType valueType = ValueType.fromShortName(type);
|
||||
|
||||
// generate UUID for edge
|
||||
if (id == null) {
|
||||
id = UUID.randomUUID().toString();
|
||||
}
|
||||
}
|
||||
if (valueType == ValueType.STRING) {
|
||||
pValue = value;
|
||||
} else if (valueType == ValueType.INT || valueType == ValueType.INTEGER) {
|
||||
pValue = Integer.valueOf(value);
|
||||
} else if (valueType == ValueType.LONG) {
|
||||
pValue = Long.valueOf(value);
|
||||
} else if (valueType == ValueType.DOUBLE) {
|
||||
pValue = Double.valueOf(value);
|
||||
} else if (valueType == ValueType.FLOAT) {
|
||||
pValue = Float.valueOf(value);
|
||||
} else if (valueType == ValueType.BOOLEAN) {
|
||||
pValue = Boolean.valueOf(value);
|
||||
} else {
|
||||
log.error("invalid property key {}, value {}, type {}", key, value, type);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (id == null || label == null) {
|
||||
log.error("invalid record id: {} , label: {}", id, label);
|
||||
throw new IllegalArgumentException("id or label missed");
|
||||
}
|
||||
// apply vertexSetProperty
|
||||
if (Key.PropertyType.set.name().equals(card) && (e instanceof GdbVertex)) {
|
||||
e.addProperty(key, pValue, Key.PropertyType.set);
|
||||
} else {
|
||||
e.addProperty(key, pValue);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
e.setId(id);
|
||||
e.setLabel(label);
|
||||
|
||||
properties.forEach(p -> p.accept(r, e));
|
||||
};
|
||||
return ret;
|
||||
}
|
||||
|
||||
static Function<Record, Object> forObjColumn(String rule, ValueType type) {
|
||||
Matcher m = NORMAL_PATTERN.matcher(rule);
|
||||
if (m.matches()) {
|
||||
int index = Integer.valueOf(m.group(1));
|
||||
return r -> type.applyColumn(r.getColumn(index));
|
||||
} else {
|
||||
return r -> type.fromStrFunc(rule);
|
||||
}
|
||||
}
|
||||
|
||||
static Function<Record, String> forStrColumn(String rule) {
|
||||
List<BiConsumer<StringBuilder, Record>> list = new ArrayList<>();
|
||||
Matcher m = STR_PATTERN.matcher(rule);
|
||||
int last = 0;
|
||||
while (m.find()) {
|
||||
String index = m.group(1);
|
||||
// as simple integer index.
|
||||
int i = Integer.parseInt(index);
|
||||
|
||||
final int tmp = last;
|
||||
final int start = m.start();
|
||||
list.add((sb, record) -> {
|
||||
sb.append(rule.subSequence(tmp, start));
|
||||
if(record.getColumn(i) != null && record.getColumn(i).getByteSize() > 0) {
|
||||
sb.append(record.getColumn(i).asString());
|
||||
}
|
||||
});
|
||||
|
||||
last = m.end();
|
||||
}
|
||||
|
||||
final int tmp = last;
|
||||
list.add((sb, record) -> {
|
||||
sb.append(rule.subSequence(tmp, rule.length()));
|
||||
});
|
||||
|
||||
return r -> {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
list.forEach(c -> c.accept(sb, r));
|
||||
String res = sb.toString();
|
||||
return res.isEmpty() ? null : res;
|
||||
};
|
||||
}
|
||||
|
||||
static boolean addToProperties(GdbElement e, String key, String value, String type) {
|
||||
ValueType valueType = ValueType.fromShortName(type);
|
||||
|
||||
if(valueType == ValueType.STRING) {
|
||||
e.getProperties().put(key, value);
|
||||
} else if (valueType == ValueType.INT) {
|
||||
e.getProperties().put(key, Integer.valueOf(value));
|
||||
} else if (valueType == ValueType.LONG) {
|
||||
e.getProperties().put(key, Long.valueOf(value));
|
||||
} else if (valueType == ValueType.DOUBLE) {
|
||||
e.getProperties().put(key, Double.valueOf(value));
|
||||
} else if (valueType == ValueType.FLOAT) {
|
||||
e.getProperties().put(key, Float.valueOf(value));
|
||||
} else if (valueType == ValueType.BOOLEAN) {
|
||||
e.getProperties().put(key, Boolean.valueOf(value));
|
||||
} else {
|
||||
log.error("invalid property key {}, value {}, type {}", key, value, type);
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
@Override
|
||||
public Function<Record, GdbElement> getMapper(final MappingRule rule) {
|
||||
return r -> {
|
||||
final GdbElement e = (rule.getImportType() == VERTEX) ? new GdbVertex() : new GdbEdge();
|
||||
forElement(rule).accept(r, e);
|
||||
return e;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -13,5 +13,5 @@ import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbElement;
|
||||
*
|
||||
*/
|
||||
public interface GdbMapper {
|
||||
Function<Record, GdbElement> getMapper(MappingRule rule);
|
||||
Function<Record, GdbElement> getMapper(MappingRule rule);
|
||||
}
|
||||
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public
|
||||
* License version 2 as published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.mapping;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.client.GdbWriterConfig;
|
||||
|
||||
/**
|
||||
* @author : Liu Jianping
|
||||
* @date : 2019/10/15
|
||||
*/
|
||||
|
||||
public class MapperConfig {
|
||||
private static MapperConfig instance = new MapperConfig();
|
||||
private int maxIdLength;
|
||||
private int maxLabelLength;
|
||||
private int maxPropKeyLength;
|
||||
private int maxPropValueLength;
|
||||
|
||||
private MapperConfig() {
|
||||
this.maxIdLength = GdbWriterConfig.MAX_STRING_LENGTH;
|
||||
this.maxLabelLength = GdbWriterConfig.MAX_STRING_LENGTH;
|
||||
this.maxPropKeyLength = GdbWriterConfig.MAX_STRING_LENGTH;
|
||||
this.maxPropValueLength = GdbWriterConfig.MAX_STRING_LENGTH;
|
||||
}
|
||||
|
||||
public static MapperConfig getInstance() {
|
||||
return instance;
|
||||
}
|
||||
|
||||
public void updateConfig(final Configuration config) {
|
||||
final int length = config.getInt(Key.MAX_GDB_STRING_LENGTH, GdbWriterConfig.MAX_STRING_LENGTH);
|
||||
|
||||
Integer sLength = config.getInt(Key.MAX_GDB_ID_LENGTH);
|
||||
this.maxIdLength = sLength == null ? length : sLength;
|
||||
|
||||
sLength = config.getInt(Key.MAX_GDB_LABEL_LENGTH);
|
||||
this.maxLabelLength = sLength == null ? length : sLength;
|
||||
|
||||
sLength = config.getInt(Key.MAX_GDB_PROP_KEY_LENGTH);
|
||||
this.maxPropKeyLength = sLength == null ? length : sLength;
|
||||
|
||||
sLength = config.getInt(Key.MAX_GDB_PROP_VALUE_LENGTH);
|
||||
this.maxPropValueLength = sLength == null ? length : sLength;
|
||||
}
|
||||
|
||||
public int getMaxIdLength() {
|
||||
return this.maxIdLength;
|
||||
}
|
||||
|
||||
public int getMaxLabelLength() {
|
||||
return this.maxLabelLength;
|
||||
}
|
||||
|
||||
public int getMaxPropKeyLength() {
|
||||
return this.maxPropKeyLength;
|
||||
}
|
||||
|
||||
public int getMaxPropValueLength() {
|
||||
return this.maxPropValueLength;
|
||||
}
|
||||
|
||||
}
|
||||
+22
-17
@@ -7,6 +7,7 @@ import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.ImportType;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.PropertyType;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
@@ -16,26 +17,30 @@ import lombok.Data;
|
||||
*/
|
||||
@Data
|
||||
public class MappingRule {
|
||||
private String id = null;
|
||||
private String id = null;
|
||||
|
||||
private String label = null;
|
||||
|
||||
private ImportType importType = null;
|
||||
|
||||
private String from = null;
|
||||
private String label = null;
|
||||
|
||||
private String to = null;
|
||||
private ImportType importType = null;
|
||||
|
||||
private List<PropertyMappingRule> properties = new ArrayList<>();
|
||||
private String from = null;
|
||||
|
||||
private String propertiesJsonStr = null;
|
||||
private String to = null;
|
||||
|
||||
@Data
|
||||
public static class PropertyMappingRule {
|
||||
private String key = null;
|
||||
|
||||
private String value = null;
|
||||
|
||||
private ValueType valueType = null;
|
||||
}
|
||||
private List<PropertyMappingRule> properties = new ArrayList<>();
|
||||
|
||||
private String propertiesJsonStr = null;
|
||||
|
||||
private boolean numPattern = false;
|
||||
|
||||
@Data
|
||||
public static class PropertyMappingRule {
|
||||
private String key = null;
|
||||
|
||||
private String value = null;
|
||||
|
||||
private ValueType valueType = null;
|
||||
|
||||
private PropertyType pType = PropertyType.single;
|
||||
}
|
||||
}
|
||||
|
||||
+162
-123
@@ -3,18 +3,21 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.mapping;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.GdbWriterErrorCode;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.ImportType;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.IdTransRule;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.ColumnType;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.IdTransRule;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.ImportType;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.mapping.MappingRule.PropertyMappingRule;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.util.ConfigHelper;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import java.util.List;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
@@ -22,66 +25,94 @@ import java.util.List;
|
||||
*/
|
||||
@Slf4j
|
||||
public class MappingRuleFactory {
|
||||
private static final MappingRuleFactory instance = new MappingRuleFactory();
|
||||
|
||||
public static final MappingRuleFactory getInstance() {
|
||||
return instance;
|
||||
}
|
||||
private static final MappingRuleFactory instance = new MappingRuleFactory();
|
||||
private static final Pattern STR_PATTERN = Pattern.compile("\\$\\{(\\d+)}");
|
||||
private static final Pattern STR_NUM_PATTERN = Pattern.compile("#\\{(\\d+)}");
|
||||
|
||||
@Deprecated
|
||||
public MappingRule create(Configuration config, ImportType type) {
|
||||
MappingRule rule = new MappingRule();
|
||||
rule.setId(config.getString(Key.ID));
|
||||
rule.setLabel(config.getString(Key.LABEL));
|
||||
if (type == ImportType.EDGE) {
|
||||
rule.setFrom(config.getString(Key.FROM));
|
||||
rule.setTo(config.getString(Key.TO));
|
||||
}
|
||||
|
||||
rule.setImportType(type);
|
||||
|
||||
List<Configuration> configurations = config.getListConfiguration(Key.PROPERTIES);
|
||||
if (configurations != null) {
|
||||
for (Configuration prop : config.getListConfiguration(Key.PROPERTIES)) {
|
||||
PropertyMappingRule propRule = new PropertyMappingRule();
|
||||
propRule.setKey(prop.getString(Key.PROP_KEY));
|
||||
propRule.setValue(prop.getString(Key.PROP_VALUE));
|
||||
propRule.setValueType(ValueType.fromShortName(prop.getString(Key.PROP_TYPE).toLowerCase()));
|
||||
rule.getProperties().add(propRule);
|
||||
}
|
||||
}
|
||||
|
||||
String propertiesJsonStr = config.getString(Key.PROPERTIES_JSON_STR, null);
|
||||
if (propertiesJsonStr != null) {
|
||||
rule.setPropertiesJsonStr(propertiesJsonStr);
|
||||
}
|
||||
|
||||
return rule;
|
||||
}
|
||||
|
||||
public MappingRule createV2(Configuration config) {
|
||||
try {
|
||||
ImportType type = ImportType.valueOf(config.getString(Key.IMPORT_TYPE));
|
||||
return createV2(config, type);
|
||||
} catch (NullPointerException e) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.CONFIG_ITEM_MISS, Key.IMPORT_TYPE);
|
||||
} catch (IllegalArgumentException e) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.BAD_CONFIG_VALUE, Key.IMPORT_TYPE);
|
||||
}
|
||||
public static MappingRuleFactory getInstance() {
|
||||
return instance;
|
||||
}
|
||||
|
||||
public MappingRule createV2(Configuration config, ImportType type) {
|
||||
MappingRule rule = new MappingRule();
|
||||
private static boolean isPattern(final String value, final MappingRule rule, final boolean checked) {
|
||||
if (checked) {
|
||||
return true;
|
||||
}
|
||||
|
||||
ConfigHelper.assertHasContent(config, Key.LABEL);
|
||||
rule.setLabel(config.getString(Key.LABEL));
|
||||
rule.setImportType(type);
|
||||
if (value == null || value.isEmpty()) {
|
||||
return false;
|
||||
}
|
||||
|
||||
IdTransRule srcTransRule = IdTransRule.none;
|
||||
Matcher m = STR_PATTERN.matcher(value);
|
||||
if (m.find()) {
|
||||
rule.setNumPattern(false);
|
||||
return true;
|
||||
}
|
||||
|
||||
m = STR_NUM_PATTERN.matcher(value);
|
||||
if (m.find()) {
|
||||
rule.setNumPattern(true);
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public MappingRule create(final Configuration config, final ImportType type) {
|
||||
final MappingRule rule = new MappingRule();
|
||||
rule.setId(config.getString(Key.ID));
|
||||
rule.setLabel(config.getString(Key.LABEL));
|
||||
if (type == ImportType.EDGE) {
|
||||
rule.setFrom(config.getString(Key.FROM));
|
||||
rule.setTo(config.getString(Key.TO));
|
||||
}
|
||||
|
||||
rule.setImportType(type);
|
||||
|
||||
final List<Configuration> configurations = config.getListConfiguration(Key.PROPERTIES);
|
||||
if (configurations != null) {
|
||||
for (final Configuration prop : config.getListConfiguration(Key.PROPERTIES)) {
|
||||
final PropertyMappingRule propRule = new PropertyMappingRule();
|
||||
propRule.setKey(prop.getString(Key.PROP_KEY));
|
||||
propRule.setValue(prop.getString(Key.PROP_VALUE));
|
||||
propRule.setValueType(ValueType.fromShortName(prop.getString(Key.PROP_TYPE).toLowerCase()));
|
||||
rule.getProperties().add(propRule);
|
||||
}
|
||||
}
|
||||
|
||||
final String propertiesJsonStr = config.getString(Key.PROPERTIES_JSON_STR, null);
|
||||
if (propertiesJsonStr != null) {
|
||||
rule.setPropertiesJsonStr(propertiesJsonStr);
|
||||
}
|
||||
|
||||
return rule;
|
||||
}
|
||||
|
||||
public MappingRule createV2(final Configuration config) {
|
||||
try {
|
||||
final ImportType type = ImportType.valueOf(config.getString(Key.IMPORT_TYPE));
|
||||
return createV2(config, type);
|
||||
} catch (final NullPointerException e) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.CONFIG_ITEM_MISS, Key.IMPORT_TYPE);
|
||||
} catch (final IllegalArgumentException e) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.BAD_CONFIG_VALUE, Key.IMPORT_TYPE);
|
||||
}
|
||||
}
|
||||
|
||||
public MappingRule createV2(final Configuration config, final ImportType type) {
|
||||
final MappingRule rule = new MappingRule();
|
||||
boolean patternChecked = false;
|
||||
|
||||
ConfigHelper.assertHasContent(config, Key.LABEL);
|
||||
rule.setLabel(config.getString(Key.LABEL));
|
||||
rule.setImportType(type);
|
||||
patternChecked = isPattern(rule.getLabel(), rule, patternChecked);
|
||||
|
||||
IdTransRule srcTransRule = IdTransRule.none;
|
||||
IdTransRule dstTransRule = IdTransRule.none;
|
||||
if (type == ImportType.EDGE) {
|
||||
ConfigHelper.assertHasContent(config, Key.SRC_ID_TRANS_RULE);
|
||||
ConfigHelper.assertHasContent(config, Key.DST_ID_TRANS_RULE);
|
||||
ConfigHelper.assertHasContent(config, Key.SRC_ID_TRANS_RULE);
|
||||
ConfigHelper.assertHasContent(config, Key.DST_ID_TRANS_RULE);
|
||||
|
||||
srcTransRule = IdTransRule.valueOf(config.getString(Key.SRC_ID_TRANS_RULE));
|
||||
dstTransRule = IdTransRule.valueOf(config.getString(Key.DST_ID_TRANS_RULE));
|
||||
@@ -94,88 +125,96 @@ public class MappingRuleFactory {
|
||||
ConfigHelper.assertHasContent(config, Key.DST_LABEL);
|
||||
}
|
||||
}
|
||||
ConfigHelper.assertHasContent(config, Key.ID_TRANS_RULE);
|
||||
IdTransRule transRule = IdTransRule.valueOf(config.getString(Key.ID_TRANS_RULE));
|
||||
ConfigHelper.assertHasContent(config, Key.ID_TRANS_RULE);
|
||||
final IdTransRule transRule = IdTransRule.valueOf(config.getString(Key.ID_TRANS_RULE));
|
||||
|
||||
List<Configuration> configurationList = config.getListConfiguration(Key.COLUMN);
|
||||
ConfigHelper.assertConfig(Key.COLUMN, () -> (configurationList != null && !configurationList.isEmpty()));
|
||||
for (Configuration column : configurationList) {
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_NAME);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_VALUE);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_TYPE);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_NODE_TYPE);
|
||||
final List<Configuration> configurationList = config.getListConfiguration(Key.COLUMN);
|
||||
ConfigHelper.assertConfig(Key.COLUMN, () -> (configurationList != null && !configurationList.isEmpty()));
|
||||
for (final Configuration column : configurationList) {
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_NAME);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_VALUE);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_TYPE);
|
||||
ConfigHelper.assertHasContent(column, Key.COLUMN_NODE_TYPE);
|
||||
|
||||
String columnValue = column.getString(Key.COLUMN_VALUE);
|
||||
ColumnType columnType = ColumnType.valueOf(column.getString(Key.COLUMN_NODE_TYPE));
|
||||
if (columnValue == null || columnValue.isEmpty()) {
|
||||
// only allow edge empty id
|
||||
ConfigHelper.assertConfig("empty column value",
|
||||
() -> (type == ImportType.EDGE && columnType == ColumnType.primaryKey));
|
||||
}
|
||||
final String columnValue = column.getString(Key.COLUMN_VALUE);
|
||||
final ColumnType columnType = ColumnType.valueOf(column.getString(Key.COLUMN_NODE_TYPE));
|
||||
if (columnValue == null || columnValue.isEmpty()) {
|
||||
// only allow edge empty id
|
||||
ConfigHelper.assertConfig("empty column value",
|
||||
() -> (type == ImportType.EDGE && columnType == ColumnType.primaryKey));
|
||||
}
|
||||
patternChecked = isPattern(columnValue, rule, patternChecked);
|
||||
|
||||
if (columnType == ColumnType.primaryKey) {
|
||||
ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key", () -> (propType == ValueType.STRING));
|
||||
if (columnType == ColumnType.primaryKey) {
|
||||
final ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key",
|
||||
() -> (propType == ValueType.STRING));
|
||||
|
||||
if (transRule == IdTransRule.labelPrefix) {
|
||||
rule.setId(config.getString(Key.LABEL) + columnValue);
|
||||
} else {
|
||||
rule.setId(columnValue);
|
||||
}
|
||||
} else if (columnType == ColumnType.edgeJsonProperty || columnType == ColumnType.vertexJsonProperty) {
|
||||
// only support one json property in column
|
||||
ConfigHelper.assertConfig("multi JsonProperty", () -> (rule.getPropertiesJsonStr() == null));
|
||||
|
||||
rule.setPropertiesJsonStr(columnValue);
|
||||
} else if (columnType == ColumnType.vertexProperty || columnType == ColumnType.edgeProperty) {
|
||||
PropertyMappingRule propertyMappingRule = new PropertyMappingRule();
|
||||
|
||||
propertyMappingRule.setKey(column.getString(Key.COLUMN_NAME));
|
||||
propertyMappingRule.setValue(columnValue);
|
||||
ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("unsupported property type", () -> propType != null);
|
||||
|
||||
propertyMappingRule.setValueType(propType);
|
||||
rule.getProperties().add(propertyMappingRule);
|
||||
} else if (columnType == ColumnType.srcPrimaryKey) {
|
||||
if (type != ImportType.EDGE) {
|
||||
continue;
|
||||
}
|
||||
|
||||
ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key", () -> (propType == ValueType.STRING));
|
||||
|
||||
if (srcTransRule == IdTransRule.labelPrefix) {
|
||||
rule.setFrom(config.getString(Key.SRC_LABEL) + columnValue);
|
||||
if (transRule == IdTransRule.labelPrefix) {
|
||||
rule.setId(config.getString(Key.LABEL) + columnValue);
|
||||
} else {
|
||||
rule.setFrom(columnValue);
|
||||
rule.setId(columnValue);
|
||||
}
|
||||
} else if (columnType == ColumnType.dstPrimaryKey) {
|
||||
} else if (columnType == ColumnType.edgeJsonProperty || columnType == ColumnType.vertexJsonProperty) {
|
||||
// only support one json property in column
|
||||
ConfigHelper.assertConfig("multi JsonProperty", () -> (rule.getPropertiesJsonStr() == null));
|
||||
|
||||
rule.setPropertiesJsonStr(columnValue);
|
||||
} else if (columnType == ColumnType.vertexProperty || columnType == ColumnType.edgeProperty
|
||||
|| columnType == ColumnType.vertexSetProperty) {
|
||||
final PropertyMappingRule propertyMappingRule = new PropertyMappingRule();
|
||||
|
||||
propertyMappingRule.setKey(column.getString(Key.COLUMN_NAME));
|
||||
propertyMappingRule.setValue(columnValue);
|
||||
final ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("unsupported property type", () -> propType != null);
|
||||
|
||||
if (columnType == ColumnType.vertexSetProperty) {
|
||||
propertyMappingRule.setPType(Key.PropertyType.set);
|
||||
}
|
||||
propertyMappingRule.setValueType(propType);
|
||||
rule.getProperties().add(propertyMappingRule);
|
||||
} else if (columnType == ColumnType.srcPrimaryKey) {
|
||||
if (type != ImportType.EDGE) {
|
||||
continue;
|
||||
}
|
||||
|
||||
ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key", () -> (propType == ValueType.STRING));
|
||||
final ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key",
|
||||
() -> (propType == ValueType.STRING));
|
||||
|
||||
if (srcTransRule == IdTransRule.labelPrefix) {
|
||||
rule.setFrom(config.getString(Key.SRC_LABEL) + columnValue);
|
||||
} else {
|
||||
rule.setFrom(columnValue);
|
||||
}
|
||||
} else if (columnType == ColumnType.dstPrimaryKey) {
|
||||
if (type != ImportType.EDGE) {
|
||||
continue;
|
||||
}
|
||||
|
||||
final ValueType propType = ValueType.fromShortName(column.getString(Key.COLUMN_TYPE));
|
||||
ConfigHelper.assertConfig("only string is allowed in primary key",
|
||||
() -> (propType == ValueType.STRING));
|
||||
|
||||
if (dstTransRule == IdTransRule.labelPrefix) {
|
||||
rule.setTo(config.getString(Key.DST_LABEL) + columnValue);
|
||||
} else {
|
||||
rule.setTo(columnValue);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (rule.getImportType() == ImportType.EDGE) {
|
||||
if (rule.getId() == null) {
|
||||
rule.setId("");
|
||||
log.info("edge id is missed, uuid be default");
|
||||
}
|
||||
ConfigHelper.assertConfig("to needed in edge", () -> (rule.getTo() != null));
|
||||
ConfigHelper.assertConfig("from needed in edge", () -> (rule.getFrom() != null));
|
||||
}
|
||||
ConfigHelper.assertConfig("id needed", () -> (rule.getId() != null));
|
||||
if (rule.getImportType() == ImportType.EDGE) {
|
||||
if (rule.getId() == null) {
|
||||
rule.setId("");
|
||||
log.info("edge id is missed, uuid be default");
|
||||
}
|
||||
ConfigHelper.assertConfig("to needed in edge", () -> (rule.getTo() != null));
|
||||
ConfigHelper.assertConfig("from needed in edge", () -> (rule.getFrom() != null));
|
||||
}
|
||||
ConfigHelper.assertConfig("id needed", () -> (rule.getId() != null));
|
||||
|
||||
return rule;
|
||||
}
|
||||
return rule;
|
||||
}
|
||||
}
|
||||
|
||||
+54
-48
@@ -8,6 +8,7 @@ import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
|
||||
import com.alibaba.datax.common.element.Column;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
/**
|
||||
@@ -16,56 +17,61 @@ import lombok.extern.slf4j.Slf4j;
|
||||
*/
|
||||
@Slf4j
|
||||
public enum ValueType {
|
||||
INT(Integer.class, "int", Column::asLong, Integer::valueOf),
|
||||
LONG(Long.class, "long", Column::asLong, Long::valueOf),
|
||||
DOUBLE(Double.class, "double", Column::asDouble, Double::valueOf),
|
||||
FLOAT(Float.class, "float", Column::asDouble, Float::valueOf),
|
||||
BOOLEAN(Boolean.class, "boolean", Column::asBoolean, Boolean::valueOf),
|
||||
STRING(String.class, "string", Column::asString, String::valueOf);
|
||||
/**
|
||||
* property value type
|
||||
*/
|
||||
INT(Integer.class, "int", Column::asLong, Integer::valueOf),
|
||||
INTEGER(Integer.class, "integer", Column::asLong, Integer::valueOf),
|
||||
LONG(Long.class, "long", Column::asLong, Long::valueOf),
|
||||
DOUBLE(Double.class, "double", Column::asDouble, Double::valueOf),
|
||||
FLOAT(Float.class, "float", Column::asDouble, Float::valueOf),
|
||||
BOOLEAN(Boolean.class, "boolean", Column::asBoolean, Boolean::valueOf),
|
||||
STRING(String.class, "string", Column::asString, String::valueOf);
|
||||
|
||||
private Class<?> type = null;
|
||||
private String shortName = null;
|
||||
private Function<Column, Object> columnFunc = null;
|
||||
private Function<String, Object> fromStrFunc = null;
|
||||
private Class<?> type = null;
|
||||
private String shortName = null;
|
||||
private Function<Column, Object> columnFunc = null;
|
||||
private Function<String, Object> fromStrFunc = null;
|
||||
|
||||
private ValueType(Class<?> type, String name, Function<Column, Object> columnFunc, Function<String, Object> fromStrFunc) {
|
||||
this.type = type;
|
||||
this.shortName = name;
|
||||
this.columnFunc = columnFunc;
|
||||
this.fromStrFunc = fromStrFunc;
|
||||
|
||||
ValueTypeHolder.shortName2type.put(name, this);
|
||||
}
|
||||
|
||||
public static ValueType fromShortName(String name) {
|
||||
return ValueTypeHolder.shortName2type.get(name);
|
||||
}
|
||||
private ValueType(final Class<?> type, final String name, final Function<Column, Object> columnFunc,
|
||||
final Function<String, Object> fromStrFunc) {
|
||||
this.type = type;
|
||||
this.shortName = name;
|
||||
this.columnFunc = columnFunc;
|
||||
this.fromStrFunc = fromStrFunc;
|
||||
|
||||
public Class<?> type() {
|
||||
return this.type;
|
||||
}
|
||||
|
||||
public String shortName() {
|
||||
return this.shortName;
|
||||
}
|
||||
|
||||
public Object applyColumn(Column column) {
|
||||
try {
|
||||
if (column == null) {
|
||||
return null;
|
||||
}
|
||||
return columnFunc.apply(column);
|
||||
} catch (Exception e) {
|
||||
log.error("applyColumn error {}, column {}", e.toString(), column);
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
public Object fromStrFunc(String str) {
|
||||
return fromStrFunc.apply(str);
|
||||
}
|
||||
ValueTypeHolder.shortName2type.put(name, this);
|
||||
}
|
||||
|
||||
private static class ValueTypeHolder {
|
||||
private static Map<String, ValueType> shortName2type = new HashMap<>();
|
||||
}
|
||||
public static ValueType fromShortName(final String name) {
|
||||
return ValueTypeHolder.shortName2type.get(name);
|
||||
}
|
||||
|
||||
public Class<?> type() {
|
||||
return this.type;
|
||||
}
|
||||
|
||||
public String shortName() {
|
||||
return this.shortName;
|
||||
}
|
||||
|
||||
public Object applyColumn(final Column column) {
|
||||
try {
|
||||
if (column == null) {
|
||||
return null;
|
||||
}
|
||||
return this.columnFunc.apply(column);
|
||||
} catch (final Exception e) {
|
||||
log.error("applyColumn error {}, column {}", e.toString(), column);
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
public Object fromStrFunc(final String str) {
|
||||
return this.fromStrFunc.apply(str);
|
||||
}
|
||||
|
||||
private static class ValueTypeHolder {
|
||||
private static Map<String, ValueType> shortName2type = new HashMap<>();
|
||||
}
|
||||
}
|
||||
|
||||
+112
-112
@@ -3,20 +3,24 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.model;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.client.GdbWriterConfig;
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.client.GdbWriterConfig.DEFAULT_BATCH_PROPERTY_NUM;
|
||||
import static com.alibaba.datax.plugin.writer.gdbwriter.client.GdbWriterConfig.MAX_REQUEST_LENGTH;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.tinkerpop.gremlin.driver.Client;
|
||||
import org.apache.tinkerpop.gremlin.driver.Cluster;
|
||||
import org.apache.tinkerpop.gremlin.driver.RequestOptions;
|
||||
import org.apache.tinkerpop.gremlin.driver.ResultSet;
|
||||
import org.apache.tinkerpop.gremlin.driver.ser.Serializers;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.client.GdbWriterConfig;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
@@ -24,128 +28,124 @@ import java.util.concurrent.TimeUnit;
|
||||
*/
|
||||
@Slf4j
|
||||
public abstract class AbstractGdbGraph implements GdbGraph {
|
||||
private final static int DEFAULT_TIMEOUT = 30000;
|
||||
private final static int DEFAULT_TIMEOUT = 30000;
|
||||
|
||||
protected Client client = null;
|
||||
protected Key.UpdateMode updateMode = Key.UpdateMode.INSERT;
|
||||
protected int propertiesBatchNum = GdbWriterConfig.DEFAULT_BATCH_PROPERTY_NUM;
|
||||
protected boolean session = false;
|
||||
protected Client client = null;
|
||||
protected Key.UpdateMode updateMode = Key.UpdateMode.INSERT;
|
||||
protected int propertiesBatchNum = DEFAULT_BATCH_PROPERTY_NUM;
|
||||
protected boolean session = false;
|
||||
protected int maxRequestLength = GdbWriterConfig.MAX_REQUEST_LENGTH;
|
||||
|
||||
protected AbstractGdbGraph() {}
|
||||
|
||||
protected AbstractGdbGraph() {}
|
||||
protected AbstractGdbGraph(final Configuration config, final boolean session) {
|
||||
initClient(config, session);
|
||||
}
|
||||
|
||||
protected AbstractGdbGraph(Configuration config, boolean session) {
|
||||
initClient(config, session);
|
||||
}
|
||||
protected void initClient(final Configuration config, final boolean session) {
|
||||
this.updateMode = Key.UpdateMode.valueOf(config.getString(Key.UPDATE_MODE, "INSERT"));
|
||||
log.info("init graphdb client");
|
||||
final String host = config.getString(Key.HOST);
|
||||
final int port = config.getInt(Key.PORT);
|
||||
final String username = config.getString(Key.USERNAME);
|
||||
final String password = config.getString(Key.PASSWORD);
|
||||
int maxDepthPerConnection =
|
||||
config.getInt(Key.MAX_IN_PROCESS_PER_CONNECTION, GdbWriterConfig.DEFAULT_MAX_IN_PROCESS_PER_CONNECTION);
|
||||
|
||||
protected void initClient(Configuration config, boolean session) {
|
||||
updateMode = Key.UpdateMode.valueOf(config.getString(Key.UPDATE_MODE, "INSERT"));
|
||||
log.info("init graphdb client");
|
||||
String host = config.getString(Key.HOST);
|
||||
int port = config.getInt(Key.PORT);
|
||||
String username = config.getString(Key.USERNAME);
|
||||
String password = config.getString(Key.PASSWORD);
|
||||
int maxDepthPerConnection = config.getInt(Key.MAX_IN_PROCESS_PER_CONNECTION,
|
||||
GdbWriterConfig.DEFAULT_MAX_IN_PROCESS_PER_CONNECTION);
|
||||
int maxConnectionPoolSize =
|
||||
config.getInt(Key.MAX_CONNECTION_POOL_SIZE, GdbWriterConfig.DEFAULT_MAX_CONNECTION_POOL_SIZE);
|
||||
|
||||
int maxConnectionPoolSize = config.getInt(Key.MAX_CONNECTION_POOL_SIZE,
|
||||
GdbWriterConfig.DEFAULT_MAX_CONNECTION_POOL_SIZE);
|
||||
int maxSimultaneousUsagePerConnection = config.getInt(Key.MAX_SIMULTANEOUS_USAGE_PER_CONNECTION,
|
||||
GdbWriterConfig.DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION);
|
||||
|
||||
int maxSimultaneousUsagePerConnection = config.getInt(Key.MAX_SIMULTANEOUS_USAGE_PER_CONNECTION,
|
||||
GdbWriterConfig.DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION);
|
||||
this.session = session;
|
||||
if (this.session) {
|
||||
maxConnectionPoolSize = GdbWriterConfig.DEFAULT_MAX_CONNECTION_POOL_SIZE;
|
||||
maxDepthPerConnection = GdbWriterConfig.DEFAULT_MAX_IN_PROCESS_PER_CONNECTION;
|
||||
maxSimultaneousUsagePerConnection = GdbWriterConfig.DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION;
|
||||
}
|
||||
|
||||
this.session = session;
|
||||
if (this.session) {
|
||||
maxConnectionPoolSize = GdbWriterConfig.DEFAULT_MAX_CONNECTION_POOL_SIZE;
|
||||
maxDepthPerConnection = GdbWriterConfig.DEFAULT_MAX_IN_PROCESS_PER_CONNECTION;
|
||||
maxSimultaneousUsagePerConnection = GdbWriterConfig.DEFAULT_MAX_SIMULTANEOUS_USAGE_PER_CONNECTION;
|
||||
}
|
||||
try {
|
||||
final Cluster cluster = Cluster.build(host).port(port).credentials(username, password)
|
||||
.serializer(Serializers.GRAPHBINARY_V1D0).maxContentLength(1048576)
|
||||
.maxInProcessPerConnection(maxDepthPerConnection).minInProcessPerConnection(0)
|
||||
.maxConnectionPoolSize(maxConnectionPoolSize).minConnectionPoolSize(maxConnectionPoolSize)
|
||||
.maxSimultaneousUsagePerConnection(maxSimultaneousUsagePerConnection).resultIterationBatchSize(64)
|
||||
.create();
|
||||
this.client = session ? cluster.connect(UUID.randomUUID().toString()).init() : cluster.connect().init();
|
||||
warmClient(maxConnectionPoolSize * maxDepthPerConnection);
|
||||
} catch (final RuntimeException e) {
|
||||
log.error("Failed to connect to GDB {}:{}, due to {}", host, port, e);
|
||||
throw e;
|
||||
}
|
||||
|
||||
try {
|
||||
Cluster cluster = Cluster.build(host).port(port).credentials(username, password)
|
||||
.serializer(Serializers.GRAPHBINARY_V1D0)
|
||||
.maxContentLength(1048576)
|
||||
.maxInProcessPerConnection(maxDepthPerConnection)
|
||||
.minInProcessPerConnection(0)
|
||||
.maxConnectionPoolSize(maxConnectionPoolSize)
|
||||
.minConnectionPoolSize(maxConnectionPoolSize)
|
||||
.maxSimultaneousUsagePerConnection(maxSimultaneousUsagePerConnection)
|
||||
.resultIterationBatchSize(64)
|
||||
.create();
|
||||
client = session ? cluster.connect(UUID.randomUUID().toString()).init() : cluster.connect().init();
|
||||
warmClient(maxConnectionPoolSize*maxDepthPerConnection);
|
||||
} catch (RuntimeException e) {
|
||||
log.error("Failed to connect to GDB {}:{}, due to {}", host, port, e);
|
||||
throw e;
|
||||
}
|
||||
this.propertiesBatchNum = config.getInt(Key.MAX_PROPERTIES_BATCH_NUM, DEFAULT_BATCH_PROPERTY_NUM);
|
||||
this.maxRequestLength = config.getInt(Key.MAX_GDB_REQUEST_LENGTH, MAX_REQUEST_LENGTH);
|
||||
}
|
||||
|
||||
propertiesBatchNum = config.getInt(Key.MAX_PROPERTIES_BATCH_NUM, GdbWriterConfig.DEFAULT_BATCH_PROPERTY_NUM);
|
||||
}
|
||||
/**
|
||||
* @param dsl
|
||||
* @param parameters
|
||||
*/
|
||||
protected void runInternal(final String dsl, final Map<String, Object> parameters) throws Exception {
|
||||
final RequestOptions.Builder options = RequestOptions.build().timeout(DEFAULT_TIMEOUT);
|
||||
if (parameters != null && !parameters.isEmpty()) {
|
||||
parameters.forEach(options::addParameter);
|
||||
}
|
||||
|
||||
final ResultSet results = this.client.submitAsync(dsl, options.create()).get(DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS);
|
||||
results.all().get(DEFAULT_TIMEOUT + 1000, TimeUnit.MILLISECONDS);
|
||||
}
|
||||
|
||||
/**
|
||||
* @param dsl
|
||||
* @param parameters
|
||||
*/
|
||||
protected void runInternal(String dsl, final Map<String, Object> parameters) throws Exception {
|
||||
RequestOptions.Builder options = RequestOptions.build().timeout(DEFAULT_TIMEOUT);
|
||||
if (parameters != null && !parameters.isEmpty()) {
|
||||
parameters.forEach(options::addParameter);
|
||||
}
|
||||
void beginTx() {
|
||||
if (!this.session) {
|
||||
return;
|
||||
}
|
||||
|
||||
ResultSet results = client.submitAsync(dsl, options.create()).get(DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS);
|
||||
results.all().get(DEFAULT_TIMEOUT + 1000, TimeUnit.MILLISECONDS);
|
||||
}
|
||||
final String dsl = "g.tx().open()";
|
||||
this.client.submit(dsl).all().join();
|
||||
}
|
||||
|
||||
void beginTx() {
|
||||
if (!session) {
|
||||
return;
|
||||
}
|
||||
void doCommit() {
|
||||
if (!this.session) {
|
||||
return;
|
||||
}
|
||||
|
||||
String dsl = "g.tx().open()";
|
||||
client.submit(dsl).all().join();
|
||||
}
|
||||
try {
|
||||
final String dsl = "g.tx().commit()";
|
||||
this.client.submit(dsl).all().join();
|
||||
} catch (final Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
void doCommit() {
|
||||
if (!session) {
|
||||
return;
|
||||
}
|
||||
void doRollback() {
|
||||
if (!this.session) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
String dsl = "g.tx().commit()";
|
||||
client.submit(dsl).all().join();
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
final String dsl = "g.tx().rollback()";
|
||||
this.client.submit(dsl).all().join();
|
||||
}
|
||||
|
||||
void doRollback() {
|
||||
if (!session) {
|
||||
return;
|
||||
}
|
||||
private void warmClient(final int num) {
|
||||
try {
|
||||
beginTx();
|
||||
runInternal("g.V('test')", null);
|
||||
doCommit();
|
||||
log.info("warm graphdb client over");
|
||||
} catch (final Exception e) {
|
||||
log.error("warmClient error");
|
||||
doRollback();
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
String dsl = "g.tx().rollback()";
|
||||
client.submit(dsl).all().join();
|
||||
}
|
||||
|
||||
private void warmClient(int num) {
|
||||
try {
|
||||
beginTx();
|
||||
runInternal("g.V('test')", null);
|
||||
doCommit();
|
||||
log.info("warm graphdb client over");
|
||||
} catch (Exception e) {
|
||||
log.error("warmClient error");
|
||||
doRollback();
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
if (client != null) {
|
||||
log.info("close graphdb client");
|
||||
client.close();
|
||||
}
|
||||
}
|
||||
@Override
|
||||
public void close() {
|
||||
if (this.client != null) {
|
||||
log.info("close graphdb client");
|
||||
this.client.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+28
-4
@@ -3,7 +3,8 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.model;
|
||||
|
||||
import lombok.Data;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.mapping.MapperConfig;
|
||||
|
||||
import lombok.EqualsAndHashCode;
|
||||
import lombok.ToString;
|
||||
|
||||
@@ -11,10 +12,33 @@ import lombok.ToString;
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper = true)
|
||||
@ToString(callSuper = true)
|
||||
public class GdbEdge extends GdbElement {
|
||||
private String from = null;
|
||||
private String to = null;
|
||||
private String from = null;
|
||||
private String to = null;
|
||||
|
||||
public String getFrom() {
|
||||
return this.from;
|
||||
}
|
||||
|
||||
public void setFrom(final String from) {
|
||||
final int maxIdLength = MapperConfig.getInstance().getMaxIdLength();
|
||||
if (from.length() > maxIdLength) {
|
||||
throw new IllegalArgumentException("from length over limit(" + maxIdLength + ")");
|
||||
}
|
||||
this.from = from;
|
||||
}
|
||||
|
||||
public String getTo() {
|
||||
return this.to;
|
||||
}
|
||||
|
||||
public void setTo(final String to) {
|
||||
final int maxIdLength = MapperConfig.getInstance().getMaxIdLength();
|
||||
if (to.length() > maxIdLength) {
|
||||
throw new IllegalArgumentException("to length over limit(" + maxIdLength + ")");
|
||||
}
|
||||
this.to = to;
|
||||
}
|
||||
}
|
||||
|
||||
+96
-7
@@ -3,18 +3,107 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.model;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key.PropertyType;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.mapping.MapperConfig;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
public class GdbElement {
|
||||
String id = null;
|
||||
String label = null;
|
||||
Map<String, Object> properties = new HashMap<>();
|
||||
private String id = null;
|
||||
private String label = null;
|
||||
private List<GdbProperty> properties = new LinkedList<>();
|
||||
|
||||
public String getId() {
|
||||
return this.id;
|
||||
}
|
||||
|
||||
public void setId(final String id) {
|
||||
final int maxIdLength = MapperConfig.getInstance().getMaxIdLength();
|
||||
if (id.length() > maxIdLength) {
|
||||
throw new IllegalArgumentException("id length over limit(" + maxIdLength + ")");
|
||||
}
|
||||
this.id = id;
|
||||
}
|
||||
|
||||
public String getLabel() {
|
||||
return this.label;
|
||||
}
|
||||
|
||||
public void setLabel(final String label) {
|
||||
final int maxLabelLength = MapperConfig.getInstance().getMaxLabelLength();
|
||||
if (label.length() > maxLabelLength) {
|
||||
throw new IllegalArgumentException("label length over limit(" + maxLabelLength + ")");
|
||||
}
|
||||
this.label = label;
|
||||
}
|
||||
|
||||
public List<GdbProperty> getProperties() {
|
||||
return this.properties;
|
||||
}
|
||||
|
||||
public void addProperty(final String propKey, final Object propValue, final PropertyType card) {
|
||||
if (propKey == null || propValue == null) {
|
||||
return;
|
||||
}
|
||||
|
||||
final int maxPropKeyLength = MapperConfig.getInstance().getMaxPropKeyLength();
|
||||
if (propKey.length() > maxPropKeyLength) {
|
||||
throw new IllegalArgumentException("property key length over limit(" + maxPropKeyLength + ")");
|
||||
}
|
||||
if (propValue instanceof String) {
|
||||
final int maxPropValueLength = MapperConfig.getInstance().getMaxPropValueLength();
|
||||
if (((String)propValue).length() > maxPropKeyLength) {
|
||||
throw new IllegalArgumentException("property value length over limit(" + maxPropValueLength + ")");
|
||||
}
|
||||
}
|
||||
|
||||
this.properties.add(new GdbProperty(propKey, propValue, card));
|
||||
}
|
||||
|
||||
public void addProperty(final String propKey, final Object propValue) {
|
||||
addProperty(propKey, propValue, PropertyType.single);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
final StringBuffer sb = new StringBuffer(this.id + "[" + this.label + "]{");
|
||||
this.properties.forEach(n -> {
|
||||
sb.append(n.cardinality.name());
|
||||
sb.append("[");
|
||||
sb.append(n.key);
|
||||
sb.append(" - ");
|
||||
sb.append(String.valueOf(n.value));
|
||||
sb.append("]");
|
||||
});
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
public static class GdbProperty {
|
||||
private String key;
|
||||
private Object value;
|
||||
private PropertyType cardinality;
|
||||
|
||||
private GdbProperty(final String key, final Object value, final PropertyType card) {
|
||||
this.key = key;
|
||||
this.value = value;
|
||||
this.cardinality = card;
|
||||
}
|
||||
|
||||
public PropertyType getCardinality() {
|
||||
return this.cardinality;
|
||||
}
|
||||
|
||||
public String getKey() {
|
||||
return this.key;
|
||||
}
|
||||
|
||||
public Object getValue() {
|
||||
return this.value;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+7
-6
@@ -3,18 +3,19 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.model;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import groovy.lang.Tuple2;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
|
||||
import groovy.lang.Tuple2;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
public interface GdbGraph extends AutoCloseable {
|
||||
List<Tuple2<Record, Exception>> add(List<Tuple2<Record, GdbElement>> records);
|
||||
List<Tuple2<Record, Exception>> add(List<Tuple2<Record, GdbElement>> records);
|
||||
|
||||
@Override
|
||||
void close();
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
|
||||
+181
-157
@@ -3,15 +3,17 @@
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.model;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Random;
|
||||
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.Key;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.util.GdbDuplicateIdException;
|
||||
import com.github.benmanes.caffeine.cache.Cache;
|
||||
import com.github.benmanes.caffeine.cache.Caffeine;
|
||||
|
||||
import groovy.lang.Tuple2;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
@@ -21,176 +23,198 @@ import lombok.extern.slf4j.Slf4j;
|
||||
*/
|
||||
@Slf4j
|
||||
public class ScriptGdbGraph extends AbstractGdbGraph {
|
||||
private static final String VAR_PREFIX = "GDB___";
|
||||
private static final String VAR_ID = VAR_PREFIX + "id";
|
||||
private static final String VAR_LABEL = VAR_PREFIX + "label";
|
||||
private static final String VAR_FROM = VAR_PREFIX + "from";
|
||||
private static final String VAR_TO = VAR_PREFIX + "to";
|
||||
private static final String VAR_PROP_KEY = VAR_PREFIX + "PK";
|
||||
private static final String VAR_PROP_VALUE = VAR_PREFIX + "PV";
|
||||
private static final String ADD_V_START = "g.addV(" + VAR_LABEL + ").property(id, " + VAR_ID + ")";
|
||||
private static final String ADD_E_START = "g.addE(" + VAR_LABEL + ").property(id, " + VAR_ID + ").from(V("
|
||||
+ VAR_FROM + ")).to(V(" + VAR_TO + "))";
|
||||
private static final String VAR_PREFIX = "GDB___";
|
||||
private static final String VAR_ID = VAR_PREFIX + "id";
|
||||
private static final String VAR_LABEL = VAR_PREFIX + "label";
|
||||
private static final String VAR_FROM = VAR_PREFIX + "from";
|
||||
private static final String VAR_TO = VAR_PREFIX + "to";
|
||||
private static final String VAR_PROP_KEY = VAR_PREFIX + "PK";
|
||||
private static final String VAR_PROP_VALUE = VAR_PREFIX + "PV";
|
||||
private static final String ADD_V_START = "g.addV(" + VAR_LABEL + ").property(id, " + VAR_ID + ")";
|
||||
private static final String ADD_E_START =
|
||||
"g.addE(" + VAR_LABEL + ").property(id, " + VAR_ID + ").from(V(" + VAR_FROM + ")).to(V(" + VAR_TO + "))";
|
||||
|
||||
private static final String UPDATE_V_START = "g.V("+VAR_ID+")";
|
||||
private static final String UPDATE_E_START = "g.E("+VAR_ID+")";
|
||||
private static final String UPDATE_V_START = "g.V(" + VAR_ID + ")";
|
||||
private static final String UPDATE_E_START = "g.E(" + VAR_ID + ")";
|
||||
|
||||
private Cache<Integer, String> propertyCache;
|
||||
private Random random;
|
||||
private Random random;
|
||||
|
||||
public ScriptGdbGraph() {
|
||||
propertyCache = Caffeine.newBuilder().maximumSize(1024).build();
|
||||
random = new Random();
|
||||
}
|
||||
public ScriptGdbGraph() {
|
||||
this.random = new Random();
|
||||
}
|
||||
|
||||
public ScriptGdbGraph(Configuration config, boolean session) {
|
||||
super(config, session);
|
||||
public ScriptGdbGraph(final Configuration config, final boolean session) {
|
||||
super(config, session);
|
||||
|
||||
propertyCache = Caffeine.newBuilder().maximumSize(1024).build();
|
||||
random = new Random();
|
||||
this.random = new Random();
|
||||
log.info("Init as ScriptGdbGraph.");
|
||||
}
|
||||
|
||||
log.info("Init as ScriptGdbGraph.");
|
||||
}
|
||||
/**
|
||||
* Apply list of {@link GdbElement} to GDB, return the failed records
|
||||
*
|
||||
* @param records
|
||||
* list of element to apply
|
||||
* @return
|
||||
*/
|
||||
@Override
|
||||
public List<Tuple2<Record, Exception>> add(final List<Tuple2<Record, GdbElement>> records) {
|
||||
final List<Tuple2<Record, Exception>> errors = new ArrayList<>();
|
||||
try {
|
||||
beginTx();
|
||||
for (final Tuple2<Record, GdbElement> elementTuple2 : records) {
|
||||
try {
|
||||
addInternal(elementTuple2.getSecond());
|
||||
} catch (final Exception e) {
|
||||
errors.add(new Tuple2<>(elementTuple2.getFirst(), e));
|
||||
}
|
||||
}
|
||||
doCommit();
|
||||
} catch (final Exception ex) {
|
||||
doRollback();
|
||||
throw new RuntimeException(ex);
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply list of {@link GdbElement} to GDB, return the failed records
|
||||
* @param records list of element to apply
|
||||
* @return
|
||||
*/
|
||||
@Override
|
||||
public List<Tuple2<Record, Exception>> add(List<Tuple2<Record, GdbElement>> records) {
|
||||
List<Tuple2<Record, Exception>> errors = new ArrayList<>();
|
||||
try {
|
||||
beginTx();
|
||||
for (Tuple2<Record, GdbElement> elementTuple2 : records) {
|
||||
try {
|
||||
addInternal(elementTuple2.getSecond());
|
||||
} catch (Exception e) {
|
||||
errors.add(new Tuple2<>(elementTuple2.getFirst(), e));
|
||||
}
|
||||
}
|
||||
doCommit();
|
||||
} catch (Exception ex) {
|
||||
doRollback();
|
||||
throw new RuntimeException(ex);
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
private void addInternal(final GdbElement element) {
|
||||
try {
|
||||
addInternal(element, false);
|
||||
} catch (final GdbDuplicateIdException e) {
|
||||
if (this.updateMode == Key.UpdateMode.SKIP) {
|
||||
log.debug("Skip duplicate id {}", element.getId());
|
||||
} else if (this.updateMode == Key.UpdateMode.INSERT) {
|
||||
throw new RuntimeException(e);
|
||||
} else if (this.updateMode == Key.UpdateMode.MERGE) {
|
||||
if (element.getProperties().isEmpty()) {
|
||||
return;
|
||||
}
|
||||
|
||||
private void addInternal(GdbElement element) {
|
||||
try {
|
||||
addInternal(element, false);
|
||||
} catch (GdbDuplicateIdException e) {
|
||||
if (updateMode == Key.UpdateMode.SKIP) {
|
||||
log.debug("Skip duplicate id {}", element.getId());
|
||||
} else if (updateMode == Key.UpdateMode.INSERT) {
|
||||
throw new RuntimeException(e);
|
||||
} else if (updateMode == Key.UpdateMode.MERGE) {
|
||||
if (element.getProperties().isEmpty()) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
addInternal(element, true);
|
||||
} catch (final GdbDuplicateIdException e1) {
|
||||
log.error("duplicate id {} while update...", element.getId());
|
||||
throw new RuntimeException(e1);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
addInternal(element, true);
|
||||
} catch (GdbDuplicateIdException e1) {
|
||||
log.error("duplicate id {} while update...", element.getId());
|
||||
throw new RuntimeException(e1);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
private void addInternal(final GdbElement element, final boolean update) throws GdbDuplicateIdException {
|
||||
boolean firstAdd = !update;
|
||||
final boolean isVertex = (element instanceof GdbVertex);
|
||||
final List<GdbElement.GdbProperty> params = element.getProperties();
|
||||
final List<GdbElement.GdbProperty> subParams = new ArrayList<>(this.propertiesBatchNum);
|
||||
|
||||
private void addInternal(GdbElement element, boolean update) throws GdbDuplicateIdException {
|
||||
Map<String, Object> params = element.getProperties();
|
||||
Map<String, Object> subParams = new HashMap<>(propertiesBatchNum);
|
||||
boolean firstAdd = !update;
|
||||
boolean isVertex = (element instanceof GdbVertex);
|
||||
final int idLength = element.getId().length();
|
||||
int attachLength = element.getLabel().length();
|
||||
if (element instanceof GdbEdge) {
|
||||
attachLength += ((GdbEdge)element).getFrom().length();
|
||||
attachLength += ((GdbEdge)element).getTo().length();
|
||||
}
|
||||
|
||||
for (Map.Entry<String, Object> entry : params.entrySet()) {
|
||||
subParams.put(entry.getKey(), entry.getValue());
|
||||
if (subParams.size() >= propertiesBatchNum) {
|
||||
setGraphDbElement(element, subParams, isVertex, firstAdd);
|
||||
firstAdd = false;
|
||||
subParams.clear();
|
||||
}
|
||||
}
|
||||
if (!subParams.isEmpty() || firstAdd) {
|
||||
setGraphDbElement(element, subParams, isVertex, firstAdd);
|
||||
}
|
||||
}
|
||||
int requestLength = idLength;
|
||||
for (final GdbElement.GdbProperty entry : params) {
|
||||
final String propKey = entry.getKey();
|
||||
final Object propValue = entry.getValue();
|
||||
|
||||
private Tuple2<String, Map<String, Object>> buildDsl(GdbElement element,
|
||||
Map<String, Object> properties,
|
||||
boolean isVertex, boolean firstAdd) {
|
||||
Map<String, Object> params = new HashMap<>();
|
||||
int appendLength = propKey.length();
|
||||
if (propValue instanceof String) {
|
||||
appendLength += ((String)propValue).length();
|
||||
}
|
||||
|
||||
String dslPropertyPart = propertyCache.get(properties.size(), keys -> {
|
||||
final StringBuilder sb = new StringBuilder();
|
||||
for (int i = 0; i < keys; i++) {
|
||||
sb.append(".property(").append(VAR_PROP_KEY).append(i)
|
||||
.append(", ").append(VAR_PROP_VALUE).append(i).append(")");
|
||||
}
|
||||
return sb.toString();
|
||||
});
|
||||
if (checkSplitDsl(firstAdd, requestLength, attachLength, appendLength, subParams.size())) {
|
||||
setGraphDbElement(element, subParams, isVertex, firstAdd);
|
||||
firstAdd = false;
|
||||
subParams.clear();
|
||||
requestLength = idLength;
|
||||
}
|
||||
|
||||
String dsl;
|
||||
if (isVertex) {
|
||||
dsl = (firstAdd ? ADD_V_START : UPDATE_V_START) + dslPropertyPart;
|
||||
} else {
|
||||
dsl = (firstAdd ? ADD_E_START : UPDATE_E_START) + dslPropertyPart;
|
||||
if (firstAdd) {
|
||||
params.put(VAR_FROM, ((GdbEdge)element).getFrom());
|
||||
params.put(VAR_TO, ((GdbEdge)element).getTo());
|
||||
}
|
||||
}
|
||||
requestLength += appendLength;
|
||||
subParams.add(entry);
|
||||
}
|
||||
if (!subParams.isEmpty() || firstAdd) {
|
||||
checkSplitDsl(firstAdd, requestLength, attachLength, 0, 0);
|
||||
setGraphDbElement(element, subParams, isVertex, firstAdd);
|
||||
}
|
||||
}
|
||||
|
||||
int index = 0;
|
||||
for (Map.Entry<String, Object> entry : properties.entrySet()) {
|
||||
params.put(VAR_PROP_KEY+index, entry.getKey());
|
||||
params.put(VAR_PROP_VALUE+index, entry.getValue());
|
||||
index++;
|
||||
}
|
||||
private boolean checkSplitDsl(final boolean firstAdd, final int requestLength, final int attachLength, final int appendLength,
|
||||
final int propNum) {
|
||||
final int length = firstAdd ? requestLength + attachLength : requestLength;
|
||||
if (length > this.maxRequestLength) {
|
||||
throw new IllegalArgumentException("request length over limit(" + this.maxRequestLength + ")");
|
||||
}
|
||||
return length + appendLength > this.maxRequestLength || propNum >= this.propertiesBatchNum;
|
||||
}
|
||||
|
||||
if (firstAdd) {
|
||||
params.put(VAR_LABEL, element.getLabel());
|
||||
}
|
||||
params.put(VAR_ID, element.getId());
|
||||
private Tuple2<String, Map<String, Object>> buildDsl(final GdbElement element, final List<GdbElement.GdbProperty> properties,
|
||||
final boolean isVertex, final boolean firstAdd) {
|
||||
final Map<String, Object> params = new HashMap<>();
|
||||
final StringBuilder sb = new StringBuilder();
|
||||
if (isVertex) {
|
||||
sb.append(firstAdd ? ADD_V_START : UPDATE_V_START);
|
||||
} else {
|
||||
sb.append(firstAdd ? ADD_E_START : UPDATE_E_START);
|
||||
}
|
||||
|
||||
return new Tuple2<>(dsl, params);
|
||||
}
|
||||
for (int i = 0; i < properties.size(); i++) {
|
||||
final GdbElement.GdbProperty prop = properties.get(i);
|
||||
|
||||
private void setGraphDbElement(GdbElement element, Map<String, Object> properties,
|
||||
boolean isVertex, boolean firstAdd) throws GdbDuplicateIdException {
|
||||
int retry = 10;
|
||||
int idleTime = random.nextInt(10) + 10;
|
||||
Tuple2<String, Map<String, Object>> elementDsl = buildDsl(element, properties, isVertex, firstAdd);
|
||||
sb.append(".property(");
|
||||
if (prop.getCardinality() == Key.PropertyType.set) {
|
||||
sb.append("set, ");
|
||||
}
|
||||
sb.append(VAR_PROP_KEY).append(i).append(", ").append(VAR_PROP_VALUE).append(i).append(")");
|
||||
|
||||
while (retry > 0) {
|
||||
try {
|
||||
runInternal(elementDsl.getFirst(), elementDsl.getSecond());
|
||||
log.debug("AddElement {}", element.getId());
|
||||
return;
|
||||
} catch (Exception e) {
|
||||
String cause = e.getCause() == null ? "" : e.getCause().toString();
|
||||
if (cause.contains("rejected from")) {
|
||||
retry--;
|
||||
try {
|
||||
Thread.sleep(idleTime);
|
||||
} catch (InterruptedException e1) {
|
||||
// ...
|
||||
}
|
||||
idleTime = Math.min(idleTime * 2, 2000);
|
||||
continue;
|
||||
} else if (firstAdd && cause.contains("GraphDB id exists")) {
|
||||
throw new GdbDuplicateIdException(e);
|
||||
}
|
||||
log.error("Add Failed id {}, dsl {}, params {}, e {}", element.getId(),
|
||||
elementDsl.getFirst(), elementDsl.getSecond(), e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
log.error("Add Failed id {}, dsl {}, params {}", element.getId(),
|
||||
elementDsl.getFirst(), elementDsl.getSecond());
|
||||
throw new RuntimeException("failed to queue new element to server");
|
||||
}
|
||||
params.put(VAR_PROP_KEY + i, prop.getKey());
|
||||
params.put(VAR_PROP_VALUE + i, prop.getValue());
|
||||
}
|
||||
|
||||
if (firstAdd) {
|
||||
params.put(VAR_LABEL, element.getLabel());
|
||||
if (!isVertex) {
|
||||
params.put(VAR_FROM, ((GdbEdge)element).getFrom());
|
||||
params.put(VAR_TO, ((GdbEdge)element).getTo());
|
||||
}
|
||||
}
|
||||
params.put(VAR_ID, element.getId());
|
||||
|
||||
return new Tuple2<>(sb.toString(), params);
|
||||
}
|
||||
|
||||
private void setGraphDbElement(final GdbElement element, final List<GdbElement.GdbProperty> properties, final boolean isVertex,
|
||||
final boolean firstAdd) throws GdbDuplicateIdException {
|
||||
int retry = 10;
|
||||
int idleTime = this.random.nextInt(10) + 10;
|
||||
final Tuple2<String, Map<String, Object>> elementDsl = buildDsl(element, properties, isVertex, firstAdd);
|
||||
|
||||
while (retry > 0) {
|
||||
try {
|
||||
runInternal(elementDsl.getFirst(), elementDsl.getSecond());
|
||||
log.debug("AddElement {}", element.getId());
|
||||
return;
|
||||
} catch (final Exception e) {
|
||||
final String cause = e.getCause() == null ? "" : e.getCause().toString();
|
||||
if (cause.contains("rejected from") || cause.contains("Timeout waiting to lock key")) {
|
||||
retry--;
|
||||
try {
|
||||
Thread.sleep(idleTime);
|
||||
} catch (final InterruptedException e1) {
|
||||
// ...
|
||||
}
|
||||
idleTime = Math.min(idleTime * 2, 2000);
|
||||
continue;
|
||||
} else if (firstAdd && cause.contains("GraphDB id exists")) {
|
||||
throw new GdbDuplicateIdException(e);
|
||||
}
|
||||
log.error("Add Failed id {}, dsl {}, params {}, e {}", element.getId(), elementDsl.getFirst(),
|
||||
elementDsl.getSecond(), e);
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}
|
||||
log.error("Add Failed id {}, dsl {}, params {}", element.getId(), elementDsl.getFirst(),
|
||||
elementDsl.getSecond());
|
||||
throw new RuntimeException("failed to queue new element to server");
|
||||
}
|
||||
}
|
||||
|
||||
+40
-36
@@ -7,53 +7,57 @@ import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.function.Supplier;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.alibaba.datax.plugin.writer.gdbwriter.GdbWriterErrorCode;
|
||||
import com.alibaba.fastjson.JSON;
|
||||
import com.alibaba.fastjson.JSONObject;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
/**
|
||||
* @author jerrywang
|
||||
*
|
||||
*/
|
||||
public interface ConfigHelper {
|
||||
static void assertConfig(String key, Supplier<Boolean> f) {
|
||||
if (!f.get()) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.BAD_CONFIG_VALUE, key);
|
||||
}
|
||||
}
|
||||
static void assertConfig(final String key, final Supplier<Boolean> f) {
|
||||
if (!f.get()) {
|
||||
throw DataXException.asDataXException(GdbWriterErrorCode.BAD_CONFIG_VALUE, key);
|
||||
}
|
||||
}
|
||||
|
||||
static void assertHasContent(Configuration config, String key) {
|
||||
assertConfig(key, () -> StringUtils.isNotBlank(config.getString(key)));
|
||||
}
|
||||
static void assertHasContent(final Configuration config, final String key) {
|
||||
assertConfig(key, () -> StringUtils.isNotBlank(config.getString(key)));
|
||||
}
|
||||
|
||||
/**
|
||||
* NOTE: {@code Configuration::get(String, Class<T>)} doesn't work.
|
||||
*
|
||||
* @param conf Configuration
|
||||
* @param key key path to configuration
|
||||
* @param cls Class of result type
|
||||
* @return the target configuration object of type T
|
||||
*/
|
||||
static <T> T getConfig(Configuration conf, String key, Class<T> cls) {
|
||||
JSONObject j = (JSONObject) conf.get(key);
|
||||
return JSON.toJavaObject(j, cls);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a configuration from the specified file on the classpath.
|
||||
*
|
||||
* @param name file name
|
||||
* @return Configuration instance.
|
||||
*/
|
||||
static Configuration fromClasspath(String name) {
|
||||
try (InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream(name)) {
|
||||
return Configuration.from(is);
|
||||
} catch (IOException e) {
|
||||
throw new IllegalArgumentException("File not found: " + name);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* NOTE: {@code Configuration::get(String, Class<T>)} doesn't work.
|
||||
*
|
||||
* @param conf
|
||||
* Configuration
|
||||
* @param key
|
||||
* key path to configuration
|
||||
* @param cls
|
||||
* Class of result type
|
||||
* @return the target configuration object of type T
|
||||
*/
|
||||
static <T> T getConfig(final Configuration conf, final String key, final Class<T> cls) {
|
||||
final JSONObject j = (JSONObject)conf.get(key);
|
||||
return JSON.toJavaObject(j, cls);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a configuration from the specified file on the classpath.
|
||||
*
|
||||
* @param name
|
||||
* file name
|
||||
* @return Configuration instance.
|
||||
*/
|
||||
static Configuration fromClasspath(final String name) {
|
||||
try (final InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream(name)) {
|
||||
return Configuration.from(is);
|
||||
} catch (final IOException e) {
|
||||
throw new IllegalArgumentException("File not found: " + name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+9
-10
@@ -1,9 +1,8 @@
|
||||
/*
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
* (C) 2019-present Alibaba Group Holding Limited.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License version 2 as
|
||||
* published by the Free Software Foundation.
|
||||
* This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public
|
||||
* License version 2 as published by the Free Software Foundation.
|
||||
*/
|
||||
package com.alibaba.datax.plugin.writer.gdbwriter.util;
|
||||
|
||||
@@ -13,11 +12,11 @@ package com.alibaba.datax.plugin.writer.gdbwriter.util;
|
||||
*/
|
||||
|
||||
public class GdbDuplicateIdException extends Exception {
|
||||
public GdbDuplicateIdException(Exception e) {
|
||||
super(e);
|
||||
}
|
||||
public GdbDuplicateIdException(Exception e) {
|
||||
super(e);
|
||||
}
|
||||
|
||||
public GdbDuplicateIdException() {
|
||||
super();
|
||||
}
|
||||
public GdbDuplicateIdException() {
|
||||
super();
|
||||
}
|
||||
}
|
||||
|
||||
+8
@@ -34,6 +34,14 @@ import java.util.Map;
|
||||
public class HbaseSQLHelper {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(HbaseSQLHelper.class);
|
||||
|
||||
static {
|
||||
try {
|
||||
Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
|
||||
} catch (Throwable t) {
|
||||
throw new RuntimeException("faild load org.apache.phoenix.jdbc.PhoenixDriver", t);
|
||||
}
|
||||
}
|
||||
|
||||
public static org.apache.hadoop.conf.Configuration generatePhoenixConf(HbaseSQLReaderConfig readerConfig) {
|
||||
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
|
||||
|
||||
|
||||
@@ -2,6 +2,6 @@
|
||||
"name": "hbase11xsqlreader",
|
||||
"class": "com.alibaba.datax.plugin.reader.hbase11xsqlreader.HbaseSQLReader",
|
||||
"description": "useScene: prod. mechanism: Scan to read data.",
|
||||
"developer": "liwei.li, bug reported to : liwei.li@alibaba-inc.com"
|
||||
"developer": "alibaba"
|
||||
}
|
||||
|
||||
|
||||
@@ -203,19 +203,20 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上,Hba
|
||||
* 描述:要写入的hbase字段。index:指定该列对应reader端column的索引,从0开始;name:指定hbase表中的列,必须为 列族:列名 的格式;type:指定写入数据类型,用于转换HBase byte[]。配置格式如下:
|
||||
|
||||
```
|
||||
"column": [
|
||||
{
|
||||
"index":1,
|
||||
"name": "cf1:q1",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"index":2,
|
||||
"name": "cf1:q2",
|
||||
"type": "string"
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
"column": [
|
||||
{
|
||||
"index":1,
|
||||
"name": "cf1:q1",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"index":2,
|
||||
"name": "cf1:q2",
|
||||
"type": "string"
|
||||
}
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
* 必选:是<br />
|
||||
@@ -227,17 +228,17 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上,Hba
|
||||
* 描述:要写入的hbase的rowkey列。index:指定该列对应reader端column的索引,从0开始,若为常量index为-1;type:指定写入数据类型,用于转换HBase byte[];value:配置常量,常作为多个字段的拼接符。hbasewriter会将rowkeyColumn中所有列按照配置顺序进行拼接作为写入hbase的rowkey,不能全为常量。配置格式如下:
|
||||
|
||||
```
|
||||
"rowkeyColumn": [
|
||||
{
|
||||
"index":0,
|
||||
"type":"string"
|
||||
},
|
||||
{
|
||||
"index":-1,
|
||||
"type":"string",
|
||||
"value":"_"
|
||||
}
|
||||
]
|
||||
"rowkeyColumn": [
|
||||
{
|
||||
"index":0,
|
||||
"type":"string"
|
||||
},
|
||||
{
|
||||
"index":-1,
|
||||
"type":"string",
|
||||
"value":"_"
|
||||
}
|
||||
]
|
||||
|
||||
```
|
||||
|
||||
@@ -250,19 +251,19 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上,Hba
|
||||
* 描述:指定写入hbase的时间戳。支持:当前时间、指定时间列,指定时间,三者选一。若不配置表示用当前时间。index:指定对应reader端column的索引,从0开始,需保证能转换为long,若是Date类型,会尝试用yyyy-MM-dd HH:mm:ss和yyyy-MM-dd HH:mm:ss SSS去解析;若为指定时间index为-1;value:指定时间的值,long值。配置格式如下:
|
||||
|
||||
```
|
||||
"versionColumn":{
|
||||
"index":1
|
||||
}
|
||||
"versionColumn":{
|
||||
"index":1
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
或者
|
||||
|
||||
```
|
||||
"versionColumn":{
|
||||
"index":-1,
|
||||
"value":123456789
|
||||
}
|
||||
"versionColumn":{
|
||||
"index":-1,
|
||||
"value":123456789
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
@@ -58,7 +58,9 @@ hbase20xsqlreader插件实现了从Phoenix(HBase SQL)读取数据,对应版本
|
||||
* **queryServerAddress**
|
||||
|
||||
* 描述:hbase20xsqlreader需要通过Phoenix轻客户端去连接Phoenix QueryServer,因此这里需要填写对应QueryServer地址。
|
||||
|
||||
增强版/Lindorm 用户若需透传user, password参数,可以在queryServerAddress后增加对应可选属性.
|
||||
格式参考:http://127.0.0.1:8765;user=root;password=root
|
||||
|
||||
* 必选:是 <br />
|
||||
|
||||
* 默认值:无 <br />
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user