Compare commits

...
136 Commits
Author SHA1 Message Date
agapple f745c698e9 [maven-release-plugin] prepare for next development iteration 2016-12-30 14:57:19 +08:00
agapple 5f4bbab722 [maven-release-plugin] prepare release canal-1.0.23 2016-12-30 14:57:07 +08:00
agapple a483f03711 fixed #227 , loadBinlogChecksum NPE 2016-12-30 14:51:22 +08:00
agapple a1207060bf format 2016-12-30 14:32:52 +08:00
agapple 5055fdb556 Merge pull request #245 from dangxia/master
issues#244
2016-12-30 14:30:00 +08:00
XuehuiHe a69b69c724 issues#244 2016-12-30 11:39:02 +08:00
agapple 90e6d34fb4 Merge pull request #243 from deeplinux/master
原来的example内的startup.bat脚本直接运行时报语法错误
2016-12-29 13:45:32 +08:00
deeplinux 907ecd43e4 修改批处理中的!=为NEQ 2016-12-28 08:06:18 +08:00
deeplinux c57a921a1a 修正批处理中的语法错误 2016-12-27 17:57:07 +08:00
agapple 83fee4ffbb Merge pull request #231 from alexwangx/alexwang
fixed issue #227 , fixed Mysql5.1.73 throw NullPointerException
2016-12-02 18:06:36 +08:00
alexwang bc7927fe75 fixed issue #227 , fixed Mysql5.1.73 throw NullPointerException 2016-12-01 17:39:51 +08:00
agapple e67f27018c fixed support json inteAddress 2016-11-28 14:11:12 +08:00
agapple 3f38ad6f49 fixed issue #227 , fixed binlog_checksum variables 2016-11-28 14:10:57 +08:00
agapple 96dca0b9e6 fixed issue #136 , support rds no primary key && mysql.ha_health_check 2016-11-17 17:52:52 +08:00
agapple ad2a96b3f1 fixed json type cast exception 2016-11-10 11:12:35 +08:00
agapple 6284a429a0 fixed lazy 2016-10-11 21:37:39 +08:00
agapple 83d8a39d16 add ServerRunningMonitor.init 2016-10-11 19:39:55 +08:00
agapple 4898829384 Merge pull request #213 from lulu2panpan/master
修复issue #212
2016-10-11 19:25:06 +08:00
agapple 69a7e6d083 Merge pull request #208 from jianan789/master
mysql5.6.29'@@global.binlog_checksum'带单引号导致master退出,优化binlogFormat判断
2016-10-11 19:16:34 +08:00
agapple a5d6c3b747 Merge pull request #210 from xujianhai/patch-1
不好的设计
2016-10-11 19:14:22 +08:00
agapple 86378cad32 Merge pull request #209 from lcybo/master
Fix a minor fault: Canal instance reload again after startup.
2016-10-11 19:11:30 +08:00
lubiao 8626c6ee24 1、SessionHandler类中执行SUBSCRIPTION时,应该先尝试启动runningMonitor再调用embeddedServer的subscribe方法
2、embeddedServer的subscribe方法中应该首先进行checkStart操作,否则可能引起metaManager的脏启动
3、SessionHandler中增加日志
2016-10-11 18:46:30 +08:00
lubiao 350f8637c3 1、BinLogFileQueue的listBinlogFiles方法中对binlog文件名进行更严格的验证,必须以数字结尾
2、LocalBinLogConnection类中解决如下bug
	2.1第二个dump方法中增加代码"decoder.handle(LogEvent.FORMAT_DESCRIPTION_EVENT);",解决"((QueryLogEvent)event).getQuery()"语句出现乱码的问题(BEGIN后面多了4个字节),出现乱码不会进入if语句,导致找到的binlog位点不准确
	2.2第一个dump方法中,被注释的decode代码才是正确的,和DirectLogFetcher不同,此处需要构造子循环进行遍历
	2.3第一个dump方法中,将每个binlog-file的首个event设置为RotateLogEvent,否则会导致所有event的binlogFileName都是默认的mysql-bin.000001。这么改完之后,和DirectLogFetcher的行为相契合,后者在每个binlogfile的首个event就是RotateLogEvent
3、LocalBinlogEventParser中增加元数据处理的相关代码,不加的话,所有的event都不带列名,基本上没法儿使用
2016-10-11 17:28:24 +08:00
李建安 96454d4387 修复开启checksum后rotateLogEvent数据包解析时binlog name乱码 2016-10-10 18:30:09 +08:00
徐建海 8a293e4b0c Update AbstractEventParser.java 2016-10-06 19:13:50 +08:00
徐建海 006dd63646 不好的设计 2016-10-06 15:48:05 +08:00
Chuanyi Li b17c11868c Fix a minor fault: Canal instance reload again after startup. 2016-10-02 13:43:17 +08:00
李建安 0c9eecd758 mysql5.6.29'@@global.binlog_checksum'带单引号导致master退出,优化binlogFormat判断 2016-09-29 17:21:59 +08:00
agapple 5d9d5b1e44 fixed testcase 2016-09-29 11:43:35 +08:00
agapple b23e814c48 Merge pull request #203 from kevinKaiF/master
fixed issue # 201
2016-09-29 10:31:29 +08:00
zikaifeng b349b400cf 修复issue # 201 2016-09-27 16:22:48 +08:00
agapple c86fba6092 Merge pull request #199 from MarkLinHz/master
upgrade protobuf version
2016-09-19 14:03:20 +08:00
yinxiu 9742fdc9d9 upgrade protobuf version 2016-09-19 12:24:06 +08:00
agapple c99b4b8cd1 fixed NPE #200 2016-09-07 12:37:21 +08:00
agapple aa8ea96572 fixed issue #195 2016-08-26 18:25:52 +08:00
agapple f6cf88a6bd support 5.7 2016-08-04 22:53:59 +08:00
agapple 14a5762fbb support 5.7 2016-08-04 22:53:22 +08:00
agapple eda24df3fe fixed kill CONNECTION exception 2016-07-26 10:59:09 +08:00
agapple 3a5a4b06a9 [maven-release-plugin] prepare for next development iteration 2016-06-30 16:59:17 +08:00
agapple ba173fcd48 [maven-release-plugin] prepare release canal-1.0.22 2016-06-30 16:59:05 +08:00
agapple 69170f58df fixed issue #172, support rds 2016-06-30 16:45:42 +08:00
agapple c3e38e50e2 fixed issue #187 ,support MySQL 5.7 Json 2016-06-30 16:23:05 +08:00
agapple ae05a3f39f fixed issue #172, 精确识别主备切换,尝试基于时间戳重新定位位点,确保不丢数据 2016-06-28 22:58:55 +08:00
agapple 6eafc73a65 Merge pull request #177 from chenyishun/master
RDS主从库切换时候异常处理
2016-05-10 18:06:11 +08:00
Yishun.Chen 63b3c4ab54 IOException限制retry次数,超过次数后跳过binlog解析 2016-05-03 16:30:38 +08:00
Yishun.Chen ecba855bf9 IOException限制retry次数,超过次数后跳过binlog解析 2016-05-03 16:30:01 +08:00
Yishun.Chen b599846470 IOException限制retry次数,超过次数后跳过binlog解析 2016-05-03 16:29:23 +08:00
agapple 8aefd7e79b fixed issue #171, getNextAddress 2016-04-29 17:44:06 +08:00
agapple 4e76a4bd6e fixed issue #176 , handle running deleted 2016-04-29 17:43:16 +08:00
agapple f1a1cb26cc fixed issue #171, compiler error 2016-04-29 11:29:16 +08:00
agapple 4f1e3bafe7 fixed issue #171 , client HA bug , modify SimpleCanalConnector 2016-04-29 11:26:37 +08:00
agapple 6a03ffaf7b Merge pull request #173 from lulu2panpan/master
Canal客户端高可用bug修复
2016-04-29 11:16:24 +08:00
lulu2panpan d7d969915e Update ClientRunningMonitor.java 2016-04-29 09:50:37 +08:00
lulu2panpan 4898ed7b83 Update SimpleCanalConnector.java 2016-04-29 09:40:27 +08:00
lulu2panpan 11e1d1d600 Update CanalConnectors.java 2016-04-29 09:31:40 +08:00
lulu2panpan e614c6141d Update ClusterCanalConnector.java 2016-04-29 09:28:48 +08:00
lulu2panpan 8ded94a09e Update ClusterCanalConnector.java 2016-04-29 09:28:12 +08:00
agapple 38cc81b7da fixed issue #162 , fixed IDE CanalLauncher 2016-04-07 11:25:59 +08:00
agapple 533c5cb8f6 fixed issue #119 , fixed batch data changeColumns 2016-04-05 23:42:08 +08:00
agapple 5932e1e31f edit url 2016-03-05 18:46:21 +08:00
agapple 70dda221f1 add yugong 2016-03-05 18:44:57 +08:00
agapple c32a5d0882 fixed issue #139 , KILL CONNECTION binlogdump thread 2016-03-01 22:19:02 +08:00
agapple fbd99acac2 Merge pull request #145 from OpenZYK/master
添加rowdata过滤
2016-02-25 18:14:20 +08:00
zhangyukun 51cfa7fa96 Update AbstractMysqlEventParser.java 2016-02-25 13:56:07 +08:00
zhangyukun 652101035e 添加Rows过滤 2016-02-25 11:44:22 +08:00
zhangyukun 1813c1eff9 Update canal.properties 2016-02-25 11:39:10 +08:00
zhangyukun 31ea9fa8f0 新增canal.instance.filter.rows过滤 2016-02-25 11:38:09 +08:00
zhangyukun 369acd8eed 新增canal.instance.filter.rows过滤 2016-02-25 11:37:45 +08:00
zhangyukun 891eb960ca 新增canal.instance.filter.rows过滤 2016-02-25 11:37:17 +08:00
zhangyukun 2e5a165b88 新增canal.instance.filter.rows过滤
为parseRowsEvent方法添加过滤
2016-02-25 11:35:46 +08:00
zhangyukun 6b9c1305ff 添加dml过滤判断
修复parseRowsEvent方法中没有dml过滤的判断而导致dml操作的数据总是会输出到store中的bug
2016-02-24 21:18:01 +08:00
agapple 4f504aab31 fixed testcase compiler error 2016-02-24 17:05:14 +08:00
agapple dc28c68bf0 fixed format 2016-02-23 17:05:12 +08:00
agapple 38f2f98da3 Merge pull request #144 from dingjun84/master
注销父类已经有的属性,避免super.start()调用了没有在子类初始化的同名属性
2016-02-23 17:03:20 +08:00
dingjun efebbe301a 注销父类已经有的属性,避免super.start()调用了没有在子类初始化的同名属性 2016-02-23 10:40:42 +08:00
agapple bf10b16138 migrate 2016-02-04 11:22:36 +08:00
agapple 37478f3409 Merge pull request #138 from MarkLinHz/master
make canal server become a single instance
2016-02-04 11:02:23 +08:00
yinxiu d2f98cda87 remove duplicate code 2016-01-14 10:59:40 +08:00
yinxiu ed74d934d3 add setAlarmHandler for otter 2016-01-11 16:15:41 +08:00
yinxiu 7b29828f60 add AbstractCanalInstance and remove duplicate code 2016-01-08 20:32:08 +08:00
yinxiu 9164fe4ca7 make canal server become a single instance 2016-01-07 17:23:58 +08:00
agapple 51ac311aa1 fixed git ignore 2015-12-31 14:31:55 +08:00
agapple 643bd5cb71 [maven-release-plugin] prepare for next development iteration 2015-12-30 16:23:30 +08:00
agapple 8375ea4b7d [maven-release-plugin] prepare release canal-1.0.21 2015-12-30 16:23:20 +08:00
agapple 72ceb58fa3 fixed issue #135 , isUpdate bugfix 2015-12-02 14:37:06 +08:00
agapple 44f2b74862 fixed issue #130 , support mysql5.6 time2/timestamp2/datetime2 javaType 2015-11-13 14:16:12 +08:00
agapple 5a34195201 Merge pull request #128 from likeloveluck/simple_ddl_pattern
fix rename table ddl parser tablename and schemaname error
2015-09-07 15:49:12 +08:00
jiawei d976932177 fix rename table ddl parser tablename and schemaname error 2015-09-01 12:17:41 +08:00
agapple d887c8f763 Merge pull request #127 from dozer47528/master
fix not skip filter and read the error data
2015-08-04 11:24:14 +08:00
Dozer 197a1446b7 fix not skip filter and read the error data 2015-07-31 17:18:42 +08:00
agapple 155c00b256 Merge pull request #124 from smilingleo/master
fixed a hard coded mysql host ip address
2015-07-28 13:51:44 +08:00
Leo b16f4844a1 fixed a hard coded mysql host ip address 2015-07-23 10:42:03 +08:00
agapple f2f2bfb538 add rename testcase 2015-07-17 14:43:15 +08:00
agapple 5f6580399c fixed format 2015-07-17 14:41:39 +08:00
agapple 10930b9e3d Merge pull request #122 from likeloveluck/rename_bug_fix
修复rename table命令中包含多个表,解析失败问题
2015-07-17 14:33:38 +08:00
jiawei d1afeb7540 change result reference 2015-07-16 19:25:46 +08:00
jiawei 119af9fca1 change rename operate 2015-07-15 19:20:30 +08:00
agapple f0ce4b569e Merge pull request #121 from huyanping/patch-1
Update README.md
2015-07-14 16:59:24 +08:00
Jenner 4459b8acdf Update README.md 2015-07-14 14:42:13 +08:00
agapple bb5d5749bd Merge pull request #120 from smilingleo/master
增加了一个脚本来自动获取mysql master binlog信息,并自动更新instance配置文件
2015-07-05 11:10:56 +08:00
smilingleo 169baafb8b correct format 2015-07-03 10:32:40 +08:00
smilingleo 199f30de87 add a script to fetch mysql master binlog file and sync position automatically 2015-07-03 10:29:58 +08:00
smilingleo 87c14f0b51 Revert "test"
This reverts commit f35d965f63.
2015-07-03 10:08:03 +08:00
smilingleo f35d965f63 test 2015-07-03 10:02:10 +08:00
agapple 9956b3557f [maven-release-plugin] prepare for next development iteration 2015-06-30 10:23:54 +08:00
agapple 452fae15fc [maven-release-plugin] prepare release canal-1.0.20 2015-06-30 10:23:45 +08:00
agapple 3d6174f857 fixed issue #119 , support binlog row image noblob/minimal 2015-06-29 23:52:37 +08:00
agapple 21abbbfe47 fixed issue #116 , process create index bug 2015-05-05 10:52:56 +08:00
agapple 89056490c3 upgrade fastjson version 2015-04-30 11:25:24 +08:00
agapple c935f06344 remove compiler warning 2015-04-30 11:20:25 +08:00
agapple eeaf5ce910 fixed issue #116 , process empty array 2015-04-30 11:06:10 +08:00
agapple c3f2ad08cd fixed warning 2015-04-28 21:33:28 +08:00
agapple 7ebb3fd1ab fixed code template and format 2015-04-28 21:11:41 +08:00
agapple ea889cb004 Merge pull request #115 from wenerme/master
Typo,javadoc 修正, 更新 guava 到18.0
2015-04-28 20:26:41 +08:00
wener 596b060649 cleanup code, extract CanalService 2015-04-27 14:57:01 +08:00
wener fc9b616f4a 添加遗漏的 MigrateMap 2015-04-24 09:36:47 +08:00
wener 6a0733bbab update guava to 18.0, magriate compute map by package level implement 2015-04-23 22:15:57 +08:00
wener bdf3b0c861 typo, javadoc,code cleanup 2015-04-23 17:25:36 +08:00
wener d01e82ce3c fix typo embeded -> embedded 2015-04-23 16:23:34 +08:00
agapple ae521195c5 Merge pull request #113 from WalterInSH/master
add missing api parameter doc
2015-04-10 12:39:38 +08:00
Walter 5529dad14b add missing api parameter doc 2015-04-09 17:35:28 +08:00
agapple b0143520eb fixed issue #112 , add mysql 5.6 useconds parse 2015-03-30 18:04:26 +08:00
agapple 5b56a027e5 Merge pull request #110 from frew/master
FileLogFetcher's fetch() method should fix up position and origin even i...
2015-01-30 20:54:22 +08:00
agapple 50d4b18556 Update README.md 2015-01-30 14:27:06 +08:00
Fred Wulff f9edeca6d2 FileLogFetcher's fetch() method should fix up position and origin even if we weren't successful at fetching anything from the file 2015-01-29 11:51:34 -08:00
agapple 5ade057113 fixed issue #106 , logger exception 2014-12-26 20:23:59 +08:00
agapple befc31146a fixed parse instance 2014-12-26 11:29:00 +08:00
agapple ac4970ef3a fixed issue #105 , seek method add format eventType parse 2014-12-17 22:08:36 +08:00
agapple 13bf7c85a0 fixed search by time 2014-12-17 20:57:12 +08:00
agapple a2ae8cda50 remove setEventFilter 2014-12-10 13:48:50 +08:00
agapple 42b164f860 Merge pull request #104 from mczcl/master
修复了重复订阅没有真正修改过滤器的问题
2014-12-10 13:37:48 +08:00
zelin.miao aac4d95b19 修复了重复订阅没有真正修改过滤器的问题 2014-12-05 23:56:42 +08:00
agapple 0e283bb52d Merge pull request #102 from diguage/master
修正插件的声明
2014-10-21 11:05:52 +08:00
D瓜哥 9ad328c5ad 增加插件maven-jar-plugin的版本声明 2014-10-14 16:22:39 +08:00
D瓜哥 5d1e952c6d 给插件maven-source-plugin\maven-compiler-plugin增加版本声明,删除重复的插件声明 2014-10-14 16:22:09 +08:00
agapple 4ba814da26 [maven-release-plugin] prepare for next development iteration 2014-09-23 16:10:57 +08:00
175 changed files with 16700 additions and 8498 deletions
+1
View File
@@ -13,3 +13,4 @@ otter.properties
jtester.properties
.idea/
*.iml
.DS_Store
+13 -2
View File
@@ -9,7 +9,7 @@
<h1>背景</h1>
<p style="font-size: 14px;"> 早期,阿里巴巴B2B公司因为存在杭州和美国双机房部署,存在跨机房同步的业务需求。不过早期的数据库同步业务,主要是基于trigger的方式获取增量变更,不过从2010年开始,阿里系公司开始逐步的尝试基于数据库的日志解析,获取增量变更进行同步,由此衍生出了增量订阅&amp;消费的业务,从此开启了一段新纪元。</p>
<p style="font-size: 14px;"> ps. 目前内部版本已经支持mysql和oracle部分版本的日志解析,当前的canal开源版本支持5.6及以下的版本(阿里内部mysql 5.6.10, mysql 5.5.18和5.1.40/48)</p>
<p style="font-size: 14px;"> ps. 目前内部版本已经支持mysql和oracle部分版本的日志解析,当前的canal开源版本支持5.7及以下的版本(阿里内部mysql 5.7.13, 5.6.10, mysql 5.5.18和5.1.40/48)</p>
<p style="font-size: 14px;"> </p>
<p style="font-size: 14px;">基于日志增量订阅&amp;消费支持的业务:</p>
<ol style="font-size: 14px;">
@@ -36,7 +36,7 @@
<li>slave重做中继日志中的事件,将改变反映它自己的数据。</li>
</ol>
<h3>canal的工作原理:</h3>
<p><img width="590" src="http://dl.iteye.com/upload/attachment/0080/3107/c87b67ba-394c-3086-9577-9db05be04c95.jpg" alt="" height="273">
<p><img width="590" src="https://camo.githubusercontent.com/46c626b4cde399db43b2634a7911a04aecf273a0/687474703a2f2f646c2e69746579652e636f6d2f75706c6f61642f6174746163686d656e742f303038302f333130372f63383762363762612d333934632d333038362d393537372d3964623035626530346339352e6a7067" alt="" height="273">
<p>原理相对比较简单:</p>
<ol>
<li>canal模拟mysql slave的交互协议,伪装自己为mysql slave,向mysql master发送dump协议</li>
@@ -62,6 +62,17 @@ See the wiki page for : <a href="https://github.com/alibaba/canal/wiki" >wiki文
<li><a href="https://github.com/alibaba/canal/releases">Download</a></li>
</ul>
<h1>相关资料</h1>
* ADC阿里技术嘉年华分享ppt (放在google docs上,可能需要翻墙): <a href="https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit?usp=sharing">ppt下载</href>
* [与阿里巴巴的RocketMQ配合使用](https://github.com/alibaba/RocketMQ)
<h1>相关开源</h1>
<ol>
<li>阿里巴巴分布式数据库同步系统(解决中美异地机房)<a href="http://github.com/alibaba/otter">http://github.com/alibaba/otter</a></li>
<li>阿里巴巴去Oracle数据迁移同步工具(目标支持MySQL/DRDS)<a href="http://github.com/alibaba/yugong">http://github.com/alibaba/yugong</a></li>
</ol>
<h1>问题反馈</h1>
<ol>
<li>qq交流群: 161559791 </li>
+1 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -19,14 +19,14 @@ public interface CanalConnector {
*
* @throws CanalClientException
*/
public void connect() throws CanalClientException;
void connect() throws CanalClientException;
/**
* 释放链接
*
* @throws CanalClientException
*/
public void disconnect() throws CanalClientException;
void disconnect() throws CanalClientException;
/**
* 检查下链接是否合法
@@ -43,7 +43,7 @@ public interface CanalConnector {
*
* @throws CanalClientException
*/
public boolean checkValid() throws CanalClientException;
boolean checkValid() throws CanalClientException;
/**
* 客户端订阅,重复订阅时会更新对应的filter信息
@@ -56,7 +56,6 @@ public interface CanalConnector {
* TODO: 后续可以考虑,如果本次提交的filter不为空,在执行过滤时,是对canal server filter + 本次filter的交集处理,达到只取1份binlog数据,多个客户端消费不同的表
* </pre>
*
* @param clientIdentity
* @throws CanalClientException
*/
void subscribe(String filter) throws CanalClientException;
@@ -64,7 +63,6 @@ public interface CanalConnector {
/**
* 客户端订阅,不提交客户端filter,以服务端的filter为准
*
* @param clientIdentity
* @throws CanalClientException
*/
void subscribe() throws CanalClientException;
@@ -72,7 +70,6 @@ public interface CanalConnector {
/**
* 取消订阅
*
* @param clientIdentity
* @throws CanalClientException
*/
void unsubscribe() throws CanalClientException;
@@ -140,14 +137,14 @@ public interface CanalConnector {
void ack(long batchId) throws CanalClientException;
/**
* 回滚到未进行 {@link ack} 的地方,指定回滚具体的batchId
* 回滚到未进行 {@link #ack} 的地方,指定回滚具体的batchId
*
* @throws CanalClientException
*/
void rollback(long batchId) throws CanalClientException;
/**
* 回滚到未进行 {@link ack} 的地方,下次fetch的时候,可以从最后一个没有 {@link ack} 的地方开始拿
* 回滚到未进行 {@link #ack} 的地方,下次fetch的时候,可以从最后一个没有 {@link #ack} 的地方开始拿
*
* @throws CanalClientException
*/
@@ -19,8 +19,9 @@ public class CanalConnectors {
/**
* 创建单链接的客户端链接
*
*
* @param address
* @param destination
* @param username
* @param password
* @return
@@ -34,36 +35,38 @@ public class CanalConnectors {
/**
* 创建带cluster模式的客户端链接,自动完成failover切换
*
*
* @param addresses
* @param destination
* @param username
* @param password
* @return
*/
public static CanalConnector newClusterConnector(List<? extends SocketAddress> addresses, String destination,
String username, String password) {
ClusterCanalConnector canalConnector = new ClusterCanalConnector(username, password, destination,
new SimpleNodeAccessStrategy(addresses));
ClusterCanalConnector canalConnector = new ClusterCanalConnector(username,
password,
destination,
new SimpleNodeAccessStrategy(addresses));
canalConnector.setSoTimeout(30 * 1000);
return canalConnector;
}
/**
* 创建带cluster模式的客户端链接,自动完成failover切换,服务器列表自动扫描
*
*
* @param zkServers
* @param destination
* @param username
* @param password
* @return
*/
public static CanalConnector newClusterConnector(String zkServers, String destination, String username,
String password) {
ClusterCanalConnector canalConnector = new ClusterCanalConnector(
username,
password,
destination,
new ClusterNodeAccessStrategy(
destination,
ZkClientx.getZkClient(zkServers)));
ClusterCanalConnector canalConnector = new ClusterCanalConnector(username,
password,
destination,
new ClusterNodeAccessStrategy(destination, ZkClientx.getZkClient(zkServers)));
canalConnector.setSoTimeout(30 * 1000);
return canalConnector;
}
@@ -10,5 +10,7 @@ import java.net.SocketAddress;
*/
public interface CanalNodeAccessStrategy {
SocketAddress currentNode();
SocketAddress nextNode();
}
@@ -41,11 +41,17 @@ public class ClusterCanalConnector implements CanalConnector {
public void connect() throws CanalClientException {
while (currentConnector == null) {
SocketAddress nextAddress = this.accessStrategy.nextNode();
int times = 0;
while (true) {
try {
currentConnector = new SimpleCanalConnector(nextAddress, username, password, destination);
currentConnector = new SimpleCanalConnector(null, username, password, destination) {
@Override
public SocketAddress getNextAddress() {
return accessStrategy.nextNode();
}
};
currentConnector.setSoTimeout(soTimeout);
if (filter != null) {
currentConnector.setFilter(filter);
@@ -57,7 +63,7 @@ public class ClusterCanalConnector implements CanalConnector {
currentConnector.connect();
break;
} catch (Exception e) {
logger.warn("failed to connect to:{} after retry {} times", nextAddress, times);
logger.warn("failed to connect to:{} after retry {} times", accessStrategy.currentNode(), times);
currentConnector.disconnect();
currentConnector = null;
// retry for #retryTimes for each node when trying to
@@ -62,6 +62,11 @@ public class ClusterNodeAccessStrategy implements CanalNodeAccessStrategy {
initRunning(this.zkClient.readData(runningPath, true));
}
@Override
public SocketAddress currentNode() {
return nextNode();
}
public SocketAddress nextNode() {
if (runningAddress != null) {// 如果服务已经启动,直接选择当前正在工作的节点
return runningAddress;
@@ -109,4 +114,5 @@ public class ClusterNodeAccessStrategy implements CanalNodeAccessStrategy {
return zkClient;
}
}
@@ -102,7 +102,7 @@ public class SimpleCanalConnector implements CanalConnector {
rollback();
}
}
connected = true;
}
@@ -125,6 +125,10 @@ public class SimpleCanalConnector implements CanalConnector {
try {
channel = SocketChannel.open();
channel.socket().setSoTimeout(soTimeout);
SocketAddress address = getAddress();
if (address == null) {
address = getNextAddress();
}
channel.connect(address);
Packet p = Packet.parseFrom(readNextPacket(channel));
if (p.getVersion() != 1) {
@@ -411,7 +415,6 @@ public class SimpleCanalConnector implements CanalConnector {
}
}
private void waitClientRunning() {
try {
if (zkClientx != null) {
@@ -435,6 +438,10 @@ public class SimpleCanalConnector implements CanalConnector {
}
}
public SocketAddress getNextAddress() {
return null;
}
public SocketAddress getAddress() {
return address;
}
@@ -32,4 +32,9 @@ public class SimpleNodeAccessStrategy implements CanalNodeAccessStrategy {
}
}
@Override
public SocketAddress currentNode() {
return nodes.get(index);
}
}
@@ -1,6 +1,7 @@
package com.alibaba.otter.canal.client.impl.running;
import java.net.InetSocketAddress;
import java.text.MessageFormat;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
@@ -20,6 +21,7 @@ import com.alibaba.otter.canal.common.utils.BooleanMutex;
import com.alibaba.otter.canal.common.utils.JsonUtils;
import com.alibaba.otter.canal.common.zookeeper.ZkClientx;
import com.alibaba.otter.canal.common.zookeeper.ZookeeperPathUtils;
import com.alibaba.otter.canal.protocol.exception.CanalClientException;
/**
* clinet running控制
@@ -62,6 +64,8 @@ public class ClientRunningMonitor extends AbstractCanalLifeCycle {
public void handleDataDeleted(String dataPath) throws Exception {
MDC.put("destination", destination);
mutex.set(false);
// 触发一下退出,可能是人为干预的释放操作或者网络闪断引起的session expired timeout
processActiveExit();
if (!release && activeData != null && isMine(activeData.getAddress())) {
// 如果上一次active的状态就是本机,则即时触发一下active抢占
initRunning();
@@ -96,7 +100,11 @@ public class ClientRunningMonitor extends AbstractCanalLifeCycle {
releaseRunning(); // 尝试一下release
}
public void initRunning() {
// 改动记录:
// 1,在方法上加synchronized关键字,保证同步顺序执行;
// 2,判断Zk上已经存在的activeData是否是本机,是的话把mutex重置为true,否则会导致死锁
// 3,增加异常处理,保证出现异常时,running节点能被删除,否则会导致死锁
public synchronized void initRunning() {
if (!isStart()) {
return;
}
@@ -116,11 +124,22 @@ public class ClientRunningMonitor extends AbstractCanalLifeCycle {
initRunning();
} else {
activeData = JsonUtils.unmarshalFromByte(bytes, ClientRunningData.class);
// 如果发现已经存在,判断一下是否自己,避免活锁
if (activeData.getAddress().contains(":") && isMine(activeData.getAddress())) {
mutex.set(true);
}
}
} catch (ZkNoNodeException e) {
zkClient.createPersistent(ZookeeperPathUtils.getClientIdNodePath(this.destination, clientData.getClientId()),
true); // 尝试创建父节点
initRunning();
} catch (Throwable t) {
logger.error(MessageFormat.format("There is an error when execute initRunning method, with destination [{0}].",
destination),
t);
// 出现任何异常尝试release
releaseRunning();
throw new CanalClientException("something goes wrong in initRunning method. ", t);
}
}
+291
View File
@@ -0,0 +1,291 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<profiles version="12">
<profile kind="CodeFormatterProfile" name="canal-format" version="12">
<setting id="org.eclipse.jdt.core.formatter.comment.insert_new_line_before_root_tags" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.disabling_tag" value="@formatter:off"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_annotation" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_type_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_type_arguments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_anonymous_type_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_case" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_brace_in_array_initializer" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.new_lines_at_block_boundaries" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_annotation_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_closing_brace_in_array_initializer" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_annotation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_field" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_while" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.use_on_off_tags" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_parens_in_annotation_type_member_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_else_in_if_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_prefix_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.keep_else_statement_on_same_line" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_ellipsis" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.insert_new_line_for_parameter" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_annotation_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_breaks_compare_to_cases" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_at_in_annotation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_multiple_fields" value="16"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_expressions_in_array_initializer" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_conditional_expression" value="4"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_for" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_binary_operator" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_question_in_wildcard" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_array_initializer" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_parens_in_enum_constant" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_finally_in_try_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_local_variable" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_catch_in_try_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_while" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_after_package" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_type_parameters" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.continuation_indentation" value="2"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_postfix_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_method_invocation" value="84"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_angle_bracket_in_type_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_superinterfaces" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_new_chunk" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_binary_operator" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_package" value="0"/>
<setting id="org.eclipse.jdt.core.compiler.source" value="1.7"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_enum_constant_arguments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_constructor_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_line_comments" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_closing_angle_bracket_in_type_arguments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_enum_declarations" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.join_wrapped_lines" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_block" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_explicit_constructor_call" value="84"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_method_invocation_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_member_type" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.align_type_members_on_columns" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_enum_constant" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_for" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_method_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_selector_in_method_invocation" value="84"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_switch" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_unary_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_case" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.indent_parameter_description" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_method_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_switch" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_enum_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_angle_bracket_in_type_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.clear_blank_lines_in_block_comment" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.lineSplit" value="120"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_if" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_brackets_in_array_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_parenthesized_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_explicitconstructorcall_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_constructor_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_first_class_body_declaration" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_method" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indentation.size" value="4"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_parens_in_method_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.enabling_tag" value="@formatter:on"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_enum_constant" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_superclass_in_type_declaration" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_assignment" value="0"/>
<setting id="org.eclipse.jdt.core.compiler.problem.assertIdentifier" value="error"/>
<setting id="org.eclipse.jdt.core.formatter.tabulation.char" value="space"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_constructor_declaration_parameters" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_semicolon_in_try_resources" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_prefix_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_statements_compare_to_body" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_method" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.wrap_outer_expressions_when_nested" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.format_guardian_clause_on_one_line" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_for" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_cast" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_parameters_in_constructor_declaration" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_labeled_statement" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_annotation_type_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_method_body" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_method_declaration" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_method_invocation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_try" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_bracket_in_array_allocation_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_enum_constant" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_annotation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_at_in_annotation_type_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_method_declaration_throws" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_if" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_switch" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_method_declaration_throws" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_parenthesized_expression_in_return" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_annotation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_question_in_conditional" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_question_in_wildcard" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_try" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_bracket_in_array_allocation_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.preserve_white_space_between_code_and_line_comments" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_parenthesized_expression_in_throw" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_type_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.compiler.problem.enumIdentifier" value="error"/>
<setting id="org.eclipse.jdt.core.formatter.indent_switchstatements_compare_to_switch" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_ellipsis" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_block" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_for_inits" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_method_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.compact_else_if" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.wrap_before_or_operator_multicatch" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_array_initializer" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_for_increments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.format_line_comment_starting_on_first_column" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_bracket_in_array_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_field" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.indent_root_tags" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_enum_constant" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_enum_declarations" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_union_type_in_multicatch" value="16"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_explicitconstructorcall_arguments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_switch" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_method_declaration_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_superinterfaces" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_allocation_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.tabulation.size" value="4"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_bracket_in_array_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_opening_brace_in_array_initializer" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_closing_brace_in_block" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_bracket_in_array_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_enum_constant" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_angle_bracket_in_type_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_constructor_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_constructor_declaration_throws" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_if" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.clear_blank_lines_in_javadoc_comment" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_throws_clause_in_constructor_declaration" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_assignment_operator" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_assignment_operator" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_empty_lines" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_synchronized" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_closing_paren_in_cast" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_method_declaration_parameters" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_block_in_case" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.number_of_empty_lines_to_preserve" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_method_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_catch" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_constructor_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_method_invocation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_bracket_in_array_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_qualified_allocation_expression" value="84"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_and_in_type_parameter" value="insert"/>
<setting id="org.eclipse.jdt.core.compiler.compliance" value="1.7"/>
<setting id="org.eclipse.jdt.core.formatter.continuation_indentation_for_array_initializer" value="2"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_brackets_in_array_allocation_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_at_in_annotation_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_allocation_expression" value="84"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_cast" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_unary_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_angle_bracket_in_parameterized_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_anonymous_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.keep_empty_array_initializer_on_one_line" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_enum_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.keep_imple_if_on_one_line" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_constructor_declaration_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_closing_angle_bracket_in_type_parameters" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_at_end_of_file_if_missing" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_for" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_labeled_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_parameterized_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_superinterfaces_in_type_declaration" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_binary_expression" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_enum_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_type" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_while" value="do not insert"/>
<setting id="org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode" value="enabled"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_try" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.put_empty_statement_on_new_line" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_label" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_parameter" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_angle_bracket_in_type_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_parens_in_method_invocation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_before_while_in_do_statement" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_enum_constant" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_javadoc_comments" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.comment.line_length" value="80"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_after_annotation_on_package" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_between_import_groups" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_enum_constant_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_semicolon" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_constructor_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.number_of_blank_lines_at_beginning_of_method_body" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_conditional" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_body_declarations_compare_to_type_header" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_annotation_type_member_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.wrap_before_binary_operator" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.indent_body_declarations_compare_to_enum_declaration_header" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_between_type_declarations" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_synchronized" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_statements_compare_to_block" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_superinterfaces_in_enum_declaration" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.join_lines_in_comments" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_question_in_conditional" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_multiple_field_declarations" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_compact_if" value="16"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_for_inits" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_switchstatements_compare_to_cases" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_array_initializer" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_default" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_and_in_type_parameter" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_parens_in_constructor_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_before_imports" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_assert" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_html" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_throws_clause_in_method_declaration" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_angle_bracket_in_type_parameters" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_bracket_in_array_allocation_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_anonymous_type_declaration" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_colon_in_conditional" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_angle_bracket_in_parameterized_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_for" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_postfix_operator" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_source_code" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_synchronized" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_allocation_expression" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_constructor_declaration_throws" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_parameters_in_method_declaration" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_brace_in_array_initializer" value="insert"/>
<setting id="org.eclipse.jdt.core.compiler.codegen.targetPlatform" value="1.7"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_resources_in_try" value="80"/>
<setting id="org.eclipse.jdt.core.formatter.use_tabs_only_for_leading_indentations" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_arguments_in_annotation" value="0"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_header" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.comment.format_block_comments" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_enum_constant" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.alignment_for_enum_constants" value="18"/>
<setting id="org.eclipse.jdt.core.formatter.insert_new_line_in_empty_block" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_body_declarations_compare_to_annotation_declaration_header" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_parenthesized_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_parenthesized_expression" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_catch" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_multiple_local_declarations" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_switch" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_comma_in_for_increments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_method_invocation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_colon_in_assert" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.brace_position_for_type_declaration" value="end_of_line"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_brace_in_array_initializer" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_between_empty_braces_in_array_initializer" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_opening_paren_in_method_declaration" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_semicolon_in_for" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_paren_in_catch" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_angle_bracket_in_parameterized_type_reference" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_multiple_field_declarations" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_closing_paren_in_annotation" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_parameterized_type_reference" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_method_invocation_arguments" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.comment.new_lines_at_javadoc_boundaries" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.blank_lines_after_imports" value="1"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_comma_in_multiple_local_declarations" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.indent_body_declarations_compare_to_enum_constant_header" value="true"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_after_semicolon_in_for" value="insert"/>
<setting id="org.eclipse.jdt.core.formatter.never_indent_line_comments_on_first_column" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_semicolon_in_try_resources" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.insert_space_before_opening_angle_bracket_in_type_arguments" value="do not insert"/>
<setting id="org.eclipse.jdt.core.formatter.never_indent_block_comments_on_first_column" value="false"/>
<setting id="org.eclipse.jdt.core.formatter.keep_then_statement_on_same_line" value="true"/>
</profile>
</profiles>
+27
View File
@@ -0,0 +1,27 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?><templates><template autoinsert="true" context="gettercomment_context" deleted="false" description="Comment for getter method" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.gettercomment" name="gettercomment">/**
* @return the ${bare_field_name}
*/</template><template autoinsert="true" context="settercomment_context" deleted="false" description="Comment for setter method" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.settercomment" name="settercomment">/**
* @param ${param} the ${bare_field_name} to set
*/</template><template autoinsert="true" context="constructorcomment_context" deleted="false" description="Comment for created constructors" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.constructorcomment" name="constructorcomment">/**
* ${tags}
*/</template><template autoinsert="false" context="filecomment_context" deleted="false" description="Comment for created Java files" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.filecomment" name="filecomment"/><template autoinsert="false" context="typecomment_context" deleted="false" description="Comment for created types" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.typecomment" name="typecomment">/**
* @author ${user} ${date} ${time}
* @since 1.0.0
*/</template><template autoinsert="true" context="fieldcomment_context" deleted="false" description="Comment for fields" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.fieldcomment" name="fieldcomment">/**
*
*/</template><template autoinsert="true" context="methodcomment_context" deleted="false" description="Comment for non-overriding methods" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.methodcomment" name="methodcomment">/**
* ${tags}
*/</template><template autoinsert="false" context="overridecomment_context" deleted="false" description="Comment for overriding methods" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.overridecomment" name="overridecomment"/><template autoinsert="true" context="delegatecomment_context" deleted="false" description="Comment for delegate methods" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.delegatecomment" name="delegatecomment">/**
* ${tags}
* ${see_to_target}
*/</template><template autoinsert="false" context="newtype_context" deleted="false" description="Newly created files" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.newtype" name="newtype">${filecomment}
${package_declaration}
${typecomment}
${type_declaration}</template><template autoinsert="true" context="classbody_context" deleted="false" description="Code in new class type bodies" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.classbody" name="classbody">
</template><template autoinsert="true" context="interfacebody_context" deleted="false" description="Code in new interface type bodies" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.interfacebody" name="interfacebody">
</template><template autoinsert="true" context="enumbody_context" deleted="false" description="Code in new enum type bodies" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.enumbody" name="enumbody">
</template><template autoinsert="true" context="annotationbody_context" deleted="false" description="Code in new annotation type bodies" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.annotationbody" name="annotationbody">
</template><template autoinsert="true" context="catchblock_context" deleted="false" description="Code in new catch blocks" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.catchblock" name="catchblock">// ${todo} Auto-generated catch block
${exception_var}.printStackTrace();</template><template autoinsert="true" context="methodbody_context" deleted="false" description="Code in created method stubs" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.methodbody" name="methodbody">// ${todo} Auto-generated method stub
${body_statement}</template><template autoinsert="true" context="constructorbody_context" deleted="false" description="Code in created constructor stubs" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.constructorbody" name="constructorbody">${body_statement}
// ${todo} Auto-generated constructor stub</template><template autoinsert="true" context="getterbody_context" deleted="false" description="Code in created getters" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.getterbody" name="getterbody">return ${field};</template><template autoinsert="true" context="setterbody_context" deleted="false" description="Code in created setters" enabled="true" id="org.eclipse.jdt.ui.text.codetemplates.setterbody" name="setterbody">${field} = ${param};</template></templates>
+1 -2
View File
@@ -3,10 +3,9 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.common</artifactId>
<packaging>jar</packaging>
<name>canal common module for otter ${project.version}</name>
@@ -6,9 +6,9 @@ package com.alibaba.otter.canal.common;
*/
public interface CanalLifeCycle {
public void start();
void start();
public void stop();
void stop();
public boolean isStart();
boolean isStart();
}
@@ -14,8 +14,7 @@ public interface CanalAlarmHandler extends CanalLifeCycle {
* 发送对应destination的报警
*
* @param destination
* @param title
* @param msg
*/
public void sendAlarm(String destination, String msg);
void sendAlarm(String destination, String msg);
}
@@ -38,7 +38,8 @@ public class AddressUtils {
private static boolean isValidHostAddress(InetAddress address) {
if (address == null || address.isLoopbackAddress()) return false;
String name = address.getHostAddress();
return (name != null && !EMPTY_IP.equals(name) && !LOCALHOST_IP.equals(name) && IP_PATTERN.matcher(name).matches());
return (name != null && !EMPTY_IP.equals(name) && !LOCALHOST_IP.equals(name) && IP_PATTERN.matcher(name)
.matches());
}
public static String getHostIp() {
@@ -1,12 +1,19 @@
package com.alibaba.otter.canal.common.utils;
import java.io.IOException;
import java.lang.reflect.Type;
import java.net.Inet4Address;
import java.net.Inet6Address;
import java.net.InetAddress;
import java.util.Arrays;
import java.util.List;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.TypeReference;
import com.alibaba.fastjson.serializer.JSONSerializer;
import com.alibaba.fastjson.serializer.ObjectSerializer;
import com.alibaba.fastjson.serializer.PropertyFilter;
import com.alibaba.fastjson.serializer.SerializeConfig;
import com.alibaba.fastjson.serializer.SerializeWriter;
import com.alibaba.fastjson.serializer.SerializerFeature;
@@ -17,6 +24,12 @@ import com.alibaba.fastjson.serializer.SerializerFeature;
*/
public class JsonUtils {
static {
SerializeConfig.getGlobalInstance().put(InetAddress.class, InetAddressSerializer.instance);
SerializeConfig.getGlobalInstance().put(Inet4Address.class, InetAddressSerializer.instance);
SerializeConfig.getGlobalInstance().put(Inet6Address.class, InetAddressSerializer.instance);
}
public static <T> T unmarshalFromByte(byte[] bytes, Class<T> targetClass) {
return (T) JSON.parseObject(bytes, targetClass);// 默认为UTF-8
}
@@ -70,4 +83,21 @@ public class JsonUtils {
out.close();
}
}
public static class InetAddressSerializer implements ObjectSerializer {
public static InetAddressSerializer instance = new InetAddressSerializer();
public void write(JSONSerializer serializer, Object object, Object fieldName, Type fieldType)
throws IOException {
if (object == null) {
serializer.writeNull();
return;
}
InetAddress address = (InetAddress) object;
// 优先使用name
serializer.write(address.getHostName());
}
}
}
@@ -54,6 +54,7 @@ public final class UriUtils {
}
String query = uri.getRawQuery();
HashMap<String, String> params = new HashMap<String, String>();
@SuppressWarnings("resource")
Scanner scan = new Scanner(query);
scan.useDelimiter(SPLIT);
while (scan.hasNext()) {
@@ -12,7 +12,7 @@ import org.I0Itec.zkclient.serialize.ZkSerializer;
import org.apache.zookeeper.CreateMode;
import com.google.common.base.Function;
import com.google.common.collect.MapMaker;
import com.google.common.collect.MigrateMap;
/**
* 使用自定义的ZooKeeperx for zk connection
@@ -23,7 +23,7 @@ import com.google.common.collect.MapMaker;
public class ZkClientx extends ZkClient {
// 对于zkclient进行一次缓存,避免一个jvm内部使用多个zk connection
private static Map<String, ZkClientx> clients = new MapMaker().makeComputingMap(new Function<String, ZkClientx>() {
private static Map<String, ZkClientx> clients = MigrateMap.makeComputingMap(new Function<String, ZkClientx>() {
public ZkClientx apply(String servers) {
return new ZkClientx(servers);
@@ -62,10 +62,12 @@ public class ZkClientx extends ZkClient {
* Create a persistent Sequential node.
*
* @param path
* @param createParents if true all parent dirs are created as well and no {@link ZkNodeExistsException} is thrown
* in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the ZooKeeper event thread
* @param createParents if true all parent dirs are created as well and no
* {@link ZkNodeExistsException} is thrown in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a
* required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the
* ZooKeeper event thread
* @throws ZkException if any ZooKeeper exception occurred
* @throws RuntimeException if any other exception occurs
*/
@@ -89,10 +91,12 @@ public class ZkClientx extends ZkClient {
*
* @param path
* @param data
* @param createParents if true all parent dirs are created as well and no {@link ZkNodeExistsException} is thrown
* in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the ZooKeeper event thread
* @param createParents if true all parent dirs are created as well and no
* {@link ZkNodeExistsException} is thrown in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a
* required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the
* ZooKeeper event thread
* @throws ZkException if any ZooKeeper exception occurred
* @throws RuntimeException if any other exception occurs
*/
@@ -118,10 +122,12 @@ public class ZkClientx extends ZkClient {
*
* @param path
* @param data
* @param createParents if true all parent dirs are created as well and no {@link ZkNodeExistsException} is thrown
* in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the ZooKeeper event thread
* @param createParents if true all parent dirs are created as well and no
* {@link ZkNodeExistsException} is thrown in case the path already exists
* @throws ZkInterruptedException if operation was interrupted, or a
* required reconnection got interrupted
* @throws IllegalArgumentException if called from anything except the
* ZooKeeper event thread
* @throws ZkException if any ZooKeeper exception occurred
* @throws RuntimeException if any other exception occurs
*/
@@ -15,8 +15,8 @@ import org.apache.zookeeper.ClientCnxn;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.ZooKeeper.States;
import org.apache.zookeeper.client.ConnectStringParser;
import org.apache.zookeeper.client.HostProvider;
@@ -39,7 +39,7 @@ public class ZooKeeperx implements IZkConnection {
private static final Field clientCnxnField = ReflectionUtils.findField(ZooKeeper.class, "cnxn");
private static final Field hostProviderField = ReflectionUtils.findField(ClientCnxn.class, "hostProvider");
private static final Field serverAddressesField = ReflectionUtils.findField(StaticHostProvider.class,
"serverAddresses");
"serverAddresses");
private static final int DEFAULT_SESSION_TIMEOUT = 90000;
private ZooKeeper _zk = null;
@@ -159,9 +159,8 @@ public class ZooKeeperx implements IZkConnection {
// 强制获取zk中的地址信息
ClientCnxn cnxn = (ClientCnxn) ReflectionUtils.getField(clientCnxnField, zk);
HostProvider hostProvider = (HostProvider) ReflectionUtils.getField(hostProviderField, cnxn);
List<InetSocketAddress> serverAddrs = (List<InetSocketAddress>) ReflectionUtils.getField(
serverAddressesField,
hostProvider);
List<InetSocketAddress> serverAddrs = (List<InetSocketAddress>) ReflectionUtils.getField(serverAddressesField,
hostProvider);
// 添加第二组集群列表
serverAddrs.addAll(new ConnectStringParser(cluster).getServerAddresses());
}
@@ -120,8 +120,10 @@ public class ZookeeperPathUtils {
}
public static String getBatchMarkWithIdPath(String destinationName, short clientId, Long batchId) {
return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH, destinationName,
String.valueOf(clientId), getBatchMarkNode(batchId));
return MessageFormat.format(DESTINATION_CLIENTID_BATCH_MARK_WITH_ID_PATH,
destinationName,
String.valueOf(clientId),
getBatchMarkNode(batchId));
}
public static String getCursorPath(String destination, short clientId) {
@@ -87,6 +87,10 @@ public class ServerRunningMonitor extends AbstractCanalLifeCycle {
}
public void init() {
processStart();
}
public void start() {
super.start();
processStart();
@@ -11,7 +11,8 @@ import java.util.Map;
public class ServerRunningMonitors {
private static ServerRunningData serverData;
private static Map runningMonitors; // <String, ServerRunningMonitor>
private static Map runningMonitors; // <String,
// ServerRunningMonitor>
public static ServerRunningData getServerData() {
return serverData;
@@ -0,0 +1,19 @@
package com.google.common.collect;
import java.util.concurrent.ConcurrentMap;
import com.google.common.base.Function;
public class MigrateMap {
@SuppressWarnings("deprecation")
public static <K, V> ConcurrentMap<K, V> makeComputingMap(MapMaker maker,
Function<? super K, ? extends V> computingFunction) {
return maker.makeComputingMap(computingFunction);
}
@SuppressWarnings("deprecation")
public static <K, V> ConcurrentMap<K, V> makeComputingMap(Function<? super K, ? extends V> computingFunction) {
return new MapMaker().makeComputingMap(computingFunction);
}
}
+1 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -6,24 +6,24 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
/**
* An utility class implements MySQL/Java charsets conversion. you can see com.mysql.jdbc.CharsetMapping.
* An utility class implements MySQL/Java charsets conversion. you can see
* com.mysql.jdbc.CharsetMapping.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
*/
public final class CharsetConversion
{
public final class CharsetConversion {
static final Log logger = LogFactory.getLog(CharsetConversion.class);
static final class Entry
{
static final class Entry {
protected final int charsetId;
protected final String mysqlCharset;
protected final String mysqlCollation;
protected final String javaCharset;
Entry(final int id, String mysqlCharset, // NL
String mysqlCollation, String javaCharset)
{
String mysqlCollation, String javaCharset){
this.charsetId = id;
this.mysqlCharset = mysqlCharset;
this.mysqlCollation = mysqlCollation;
@@ -34,38 +34,31 @@ public final class CharsetConversion
// Character set data used in lookups. The array will be sparse.
static final Entry[] entries = new Entry[0xff];
static Entry getEntry(final int id)
{
if (id >= 0 && id < entries.length)
{
static Entry getEntry(final int id) {
if (id >= 0 && id < entries.length) {
return entries[id];
}
else
{
} else {
throw new IllegalArgumentException("Invalid charset id: " + id);
}
}
// Loads character set information.
static void putEntry(final int charsetId, String mysqlCharset,
String mysqlCollation, String javaCharset)
{
static void putEntry(final int charsetId, String mysqlCharset, String mysqlCollation, String javaCharset) {
entries[charsetId] = new Entry(charsetId, mysqlCharset, // NL
mysqlCollation, javaCharset);
mysqlCollation,
javaCharset);
}
// Loads character set information.
@Deprecated
static void putEntry(final int charsetId, String mysqlCharset,
String mysqlCollation)
{
static void putEntry(final int charsetId, String mysqlCharset, String mysqlCollation) {
entries[charsetId] = new Entry(charsetId, mysqlCharset, // NL
mysqlCollation, /* Unknown java charset */null);
mysqlCollation, /* Unknown java charset */
null);
}
// Load character set data statically.
static
{
static {
putEntry(1, "big5", "big5_chinese_ci", "Big5");
putEntry(2, "latin2", "latin2_czech_cs", "ISO8859_2");
putEntry(3, "dec8", "dec8_swedish_ci", "ISO8859_1");
@@ -135,7 +128,7 @@ public final class CharsetConversion
putEntry(69, "dec8", "dec8_bin", "US-ASCII");
putEntry(70, "greek", "greek_bin", "ISO8859_7");
putEntry(71, "hebrew", "hebrew_bin", "ISO8859_8");
putEntry(72, "hp8", "hp8_bin" , "US-ASCII");
putEntry(72, "hp8", "hp8_bin", "US-ASCII");
putEntry(73, "keybcs2", "keybcs2_bin", "Cp895");
putEntry(74, "koi8r", "koi8r_bin", "KOI8_R");
putEntry(75, "koi8u", "koi8u_bin", "KOI8_U");
@@ -274,16 +267,12 @@ public final class CharsetConversion
/**
* Return defined charset name for mysql.
*/
public static String getCharset(final int id)
{
public static String getCharset(final int id) {
Entry entry = getEntry(id);
if (entry != null)
{
if (entry != null) {
return entry.mysqlCharset;
}
else
{
} else {
logger.warn("Unexpect mysql charset: " + id);
return null;
}
@@ -292,16 +281,12 @@ public final class CharsetConversion
/**
* Return defined collaction name for mysql.
*/
public static String getCollation(final int id)
{
public static String getCollation(final int id) {
Entry entry = getEntry(id);
if (entry != null)
{
if (entry != null) {
return entry.mysqlCollation;
}
else
{
} else {
logger.warn("Unexpect mysql charset: " + id);
return null;
}
@@ -310,55 +295,41 @@ public final class CharsetConversion
/**
* Return converted charset name for java.
*/
public static String getJavaCharset(final int id)
{
public static String getJavaCharset(final int id) {
Entry entry = getEntry(id);
if (entry != null)
{
if (entry.javaCharset != null)
{
if (entry != null) {
if (entry.javaCharset != null) {
return entry.javaCharset;
}
else
{
logger.warn("Unknown java charset for: id = " + id
+ ", name = " + entry.mysqlCharset + ", coll = "
+ entry.mysqlCollation);
} else {
logger.warn("Unknown java charset for: id = " + id + ", name = " + entry.mysqlCharset + ", coll = "
+ entry.mysqlCollation);
return null;
}
}
else
{
} else {
logger.warn("Unexpect mysql charset: " + id);
return null;
}
}
public static void main(String[] args)
{
for (int i = 0; i < entries.length; i++)
{
public static void main(String[] args) {
for (int i = 0; i < entries.length; i++) {
Entry entry = entries[i];
System.out.print(i);
System.out.print(',');
System.out.print(' ');
if (entry != null)
{
if (entry != null) {
System.out.print(entry.mysqlCharset);
System.out.print(',');
System.out.print(' ');
System.out.print(entry.javaCharset);
if (entry.javaCharset != null)
{
if (entry.javaCharset != null) {
System.out.print(',');
System.out.print(' ');
System.out.print(Charset.forName(entry.javaCharset).name());
}
}
else
{
} else {
System.out.print("null");
}
System.out.println();
@@ -21,16 +21,13 @@ import org.apache.commons.logging.LogFactory;
* DirectLogFetcher fetcher = new DirectLogFetcher();
* fetcher.open(conn, file, 0, 13);
*
* while (fetcher.fetch())
* {
* while (fetcher.fetch()) {
* LogEvent event;
* do
* {
* do {
* event = decoder.decode(fetcher, context);
*
* // process log event.
* }
* while (event != null);
* } while (event != null);
* }
* // connection closed.
* </pre>
@@ -38,97 +35,79 @@ import org.apache.commons.logging.LogFactory;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class DirectLogFetcher extends LogFetcher
{
protected static final Log logger = LogFactory.getLog(DirectLogFetcher.class);
public final class DirectLogFetcher extends LogFetcher {
protected static final Log logger = LogFactory.getLog(DirectLogFetcher.class);
/** Command to dump binlog */
public static final byte COM_BINLOG_DUMP = 18;
public static final byte COM_BINLOG_DUMP = 18;
/** Packet header sizes */
public static final int NET_HEADER_SIZE = 4;
public static final int SQLSTATE_LENGTH = 5;
public static final int NET_HEADER_SIZE = 4;
public static final int SQLSTATE_LENGTH = 5;
/** Packet offsets */
public static final int PACKET_LEN_OFFSET = 0;
public static final int PACKET_SEQ_OFFSET = 3;
public static final int PACKET_LEN_OFFSET = 0;
public static final int PACKET_SEQ_OFFSET = 3;
/** Maximum packet length */
public static final int MAX_PACKET_LENGTH = (256 * 256 * 256 - 1);
public static final int MAX_PACKET_LENGTH = (256 * 256 * 256 - 1);
/** BINLOG_DUMP options */
public static final int BINLOG_DUMP_NON_BLOCK = 1;
public static final int BINLOG_DUMP_NON_BLOCK = 1;
public static final int BINLOG_SEND_ANNOTATE_ROWS_EVENT = 2;
private Connection conn;
private OutputStream mysqlOutput;
private InputStream mysqlInput;
public DirectLogFetcher()
{
public DirectLogFetcher(){
super(DEFAULT_INITIAL_CAPACITY, DEFAULT_GROWTH_FACTOR);
}
public DirectLogFetcher(final int initialCapacity)
{
public DirectLogFetcher(final int initialCapacity){
super(initialCapacity, DEFAULT_GROWTH_FACTOR);
}
public DirectLogFetcher(final int initialCapacity, final float growthFactor)
{
public DirectLogFetcher(final int initialCapacity, final float growthFactor){
super(initialCapacity, growthFactor);
}
private static final Object unwrapConnection(Object conn, Class<?> connClazz)
throws IOException
{
while (!connClazz.isInstance(conn))
{
try
{
private static final Object unwrapConnection(Object conn, Class<?> connClazz) throws IOException {
while (!connClazz.isInstance(conn)) {
try {
Class<?> connProxy = Class.forName("org.springframework.jdbc.datasource.ConnectionProxy");
if (connProxy.isInstance(conn))
{
if (connProxy.isInstance(conn)) {
conn = invokeMethod(conn, connProxy, "getTargetConnection");
continue;
}
}
catch (ClassNotFoundException e)
{
// org.springframework.jdbc.datasource.ConnectionProxy not found.
} catch (ClassNotFoundException e) {
// org.springframework.jdbc.datasource.ConnectionProxy not
// found.
}
try
{
try {
Class<?> connProxy = Class.forName("org.apache.commons.dbcp.DelegatingConnection");
if (connProxy.isInstance(conn))
{
if (connProxy.isInstance(conn)) {
conn = getDeclaredField(conn, connProxy, "_conn");
continue;
}
}
catch (ClassNotFoundException e)
{
} catch (ClassNotFoundException e) {
// org.apache.commons.dbcp.DelegatingConnection not found.
}
try
{
if (conn instanceof java.sql.Wrapper)
{
try {
if (conn instanceof java.sql.Wrapper) {
Class<?> connIface = Class.forName("com.mysql.jdbc.Connection");
conn = ((java.sql.Wrapper) conn).unwrap(connIface);
continue;
}
}
catch (ClassNotFoundException e)
{
} catch (ClassNotFoundException e) {
// com.mysql.jdbc.Connection not found.
}
catch (SQLException e)
{
logger.warn("Unwrap " + conn.getClass().getName() + " to "
+ connClazz.getName() + " failed: " + e.getMessage(), e);
} catch (SQLException e) {
logger.warn("Unwrap " + conn.getClass().getName() + " to " + connClazz.getName() + " failed: "
+ e.getMessage(),
e);
}
return null;
@@ -136,127 +115,85 @@ public final class DirectLogFetcher extends LogFetcher
return conn;
}
private static final Object invokeMethod(Object obj, Class<?> objClazz,
String name)
{
try
{
private static final Object invokeMethod(Object obj, Class<?> objClazz, String name) {
try {
Method method = objClazz.getMethod(name, (Class<?>[]) null);
return method.invoke(obj, (Object[]) null);
}
catch (NoSuchMethodException e)
{
throw new IllegalArgumentException("No such method: \'" + name
+ "\' @ " + objClazz.getName(), e);
}
catch (IllegalAccessException e)
{
throw new IllegalArgumentException("Cannot invoke method: \'"
+ name + "\' @ " + objClazz.getName(), e);
}
catch (InvocationTargetException e)
{
throw new IllegalArgumentException("Invoke method failed: \'"
+ name + "\' @ " + objClazz.getName(),
e.getTargetException());
} catch (NoSuchMethodException e) {
throw new IllegalArgumentException("No such method: \'" + name + "\' @ " + objClazz.getName(), e);
} catch (IllegalAccessException e) {
throw new IllegalArgumentException("Cannot invoke method: \'" + name + "\' @ " + objClazz.getName(), e);
} catch (InvocationTargetException e) {
throw new IllegalArgumentException("Invoke method failed: \'" + name + "\' @ " + objClazz.getName(),
e.getTargetException());
}
}
private static final Object getDeclaredField(Object obj, Class<?> objClazz,
String name)
{
try
{
private static final Object getDeclaredField(Object obj, Class<?> objClazz, String name) {
try {
Field field = objClazz.getDeclaredField(name);
field.setAccessible(true);
return field.get(obj);
}
catch (NoSuchFieldException e)
{
throw new IllegalArgumentException("No such field: \'" + name
+ "\' @ " + objClazz.getName(), e);
}
catch (IllegalAccessException e)
{
throw new IllegalArgumentException("Cannot get field: \'" + name
+ "\' @ " + objClazz.getName(), e);
} catch (NoSuchFieldException e) {
throw new IllegalArgumentException("No such field: \'" + name + "\' @ " + objClazz.getName(), e);
} catch (IllegalAccessException e) {
throw new IllegalArgumentException("Cannot get field: \'" + name + "\' @ " + objClazz.getName(), e);
}
}
/**
* Connect MySQL master to fetch binlog.
*/
public void open(Connection conn, String fileName, final int serverId)
throws IOException
{
public void open(Connection conn, String fileName, final int serverId) throws IOException {
open(conn, fileName, BIN_LOG_HEADER_SIZE, serverId, false);
}
/**
* Connect MySQL master to fetch binlog.
*/
public void open(Connection conn, String fileName, final int serverId,
boolean nonBlocking) throws IOException
{
public void open(Connection conn, String fileName, final int serverId, boolean nonBlocking) throws IOException {
open(conn, fileName, BIN_LOG_HEADER_SIZE, serverId, nonBlocking);
}
/**
* Connect MySQL master to fetch binlog.
*/
public void open(Connection conn, String fileName, final long filePosition,
final int serverId) throws IOException
{
public void open(Connection conn, String fileName, final long filePosition, final int serverId) throws IOException {
open(conn, fileName, filePosition, serverId, false);
}
/**
* Connect MySQL master to fetch binlog.
*/
public void open(Connection conn, String fileName, long filePosition,
final int serverId, boolean nonBlocking) throws IOException
{
try
{
public void open(Connection conn, String fileName, long filePosition, final int serverId, boolean nonBlocking)
throws IOException {
try {
this.conn = conn;
Class<?> connClazz = Class.forName("com.mysql.jdbc.ConnectionImpl");
Object unwrapConn = unwrapConnection(conn, connClazz);
if (unwrapConn == null)
{
throw new IOException("Unable to unwrap "
+ conn.getClass().getName()
+ " to com.mysql.jdbc.ConnectionImpl");
if (unwrapConn == null) {
throw new IOException("Unable to unwrap " + conn.getClass().getName()
+ " to com.mysql.jdbc.ConnectionImpl");
}
// Get underlying IO streams for network communications.
Object connIo = getDeclaredField(unwrapConn, connClazz, "io");
if (connIo == null)
{
throw new IOException("Get null field:"
+ conn.getClass().getName() + "#io");
if (connIo == null) {
throw new IOException("Get null field:" + conn.getClass().getName() + "#io");
}
mysqlOutput = (OutputStream) getDeclaredField(connIo,
connIo.getClass(), "mysqlOutput");
mysqlInput = (InputStream) getDeclaredField(connIo,
connIo.getClass(), "mysqlInput");
mysqlOutput = (OutputStream) getDeclaredField(connIo, connIo.getClass(), "mysqlOutput");
mysqlInput = (InputStream) getDeclaredField(connIo, connIo.getClass(), "mysqlInput");
if (filePosition == 0)
filePosition = BIN_LOG_HEADER_SIZE;
if (filePosition == 0) filePosition = BIN_LOG_HEADER_SIZE;
sendBinlogDump(fileName, filePosition, serverId, nonBlocking);
position = 0;
}
catch (IOException e)
{
} catch (IOException e) {
close(); /* Do cleanup */
logger.error("Error on COM_BINLOG_DUMP: file = " + fileName
+ ", position = " + filePosition);
logger.error("Error on COM_BINLOG_DUMP: file = " + fileName + ", position = " + filePosition);
throw e;
}
catch (ClassNotFoundException e)
{
} catch (ClassNotFoundException e) {
close(); /* Do cleanup */
throw new IOException(
"Unable to load com.mysql.jdbc.ConnectionImpl", e);
throw new IOException("Unable to load com.mysql.jdbc.ConnectionImpl", e);
}
}
@@ -265,8 +202,7 @@ public final class DirectLogFetcher extends LogFetcher
*
* @param b the byte to put in the buffer
*/
protected final void putByte(byte b)
{
protected final void putByte(byte b) {
ensureCapacity(position + 1);
buffer[position++] = b;
@@ -277,8 +213,7 @@ public final class DirectLogFetcher extends LogFetcher
*
* @param i16 the integer to put in the buffer
*/
protected final void putInt16(int i16)
{
protected final void putInt16(int i16) {
ensureCapacity(position + 2);
byte[] buf = buffer;
@@ -291,8 +226,7 @@ public final class DirectLogFetcher extends LogFetcher
*
* @param i32 the integer to put in the buffer
*/
protected final void putInt32(long i32)
{
protected final void putInt32(long i32) {
ensureCapacity(position + 4);
byte[] buf = buffer;
@@ -307,8 +241,7 @@ public final class DirectLogFetcher extends LogFetcher
*
* @param s the value to put in the buffer
*/
protected final void putString(String s)
{
protected final void putString(String s) {
ensureCapacity(position + (s.length() * 2) + 1);
System.arraycopy(s.getBytes(), 0, buffer, position, s.length());
@@ -316,10 +249,8 @@ public final class DirectLogFetcher extends LogFetcher
buffer[position++] = 0;
}
protected final void sendBinlogDump(String fileName,
final long filePosition, final int serverId, boolean nonBlocking)
throws IOException
{
protected final void sendBinlogDump(String fileName, final long filePosition, final int serverId,
boolean nonBlocking) throws IOException {
position = NET_HEADER_SIZE;
putByte(COM_BINLOG_DUMP);
@@ -327,7 +258,7 @@ public final class DirectLogFetcher extends LogFetcher
int binlog_flags = nonBlocking ? BINLOG_DUMP_NON_BLOCK : 0;
binlog_flags |= BINLOG_SEND_ANNOTATE_ROWS_EVENT;
putInt16(binlog_flags); // binlog_flags
putInt32(serverId); // slave's server-id
putInt32(serverId); // slave's server-id
putString(fileName);
final byte[] buf = buffer;
@@ -345,13 +276,10 @@ public final class DirectLogFetcher extends LogFetcher
*
* @see com.taobao.tddl.dbsync.binlog.LogFetcher#fetch()
*/
public boolean fetch() throws IOException
{
try
{
public boolean fetch() throws IOException {
try {
// Fetching packet header from input.
if (!fetch0(0, NET_HEADER_SIZE))
{
if (!fetch0(0, NET_HEADER_SIZE)) {
logger.warn("Reached end of input stream while fetching header");
return false;
}
@@ -359,61 +287,48 @@ public final class DirectLogFetcher extends LogFetcher
// Fetching the first packet(may a multi-packet).
int netlen = getUint24(PACKET_LEN_OFFSET);
int netnum = getUint8(PACKET_SEQ_OFFSET);
if (!fetch0(NET_HEADER_SIZE, netlen))
{
logger.warn("Reached end of input stream: packet #" + netnum
+ ", len = " + netlen);
if (!fetch0(NET_HEADER_SIZE, netlen)) {
logger.warn("Reached end of input stream: packet #" + netnum + ", len = " + netlen);
return false;
}
// Detecting error code.
final int mark = getUint8(NET_HEADER_SIZE);
if (mark != 0)
{
if (mark != 0) {
if (mark == 255) // error from master
{
// Indicates an error, for example trying to fetch from wrong
// Indicates an error, for example trying to fetch from
// wrong
// binlog position.
position = NET_HEADER_SIZE + 1;
final int errno = getInt16();
String sqlstate = forward(1).getFixString(SQLSTATE_LENGTH);
String errmsg = getFixString(limit - position);
throw new IOException("Received error packet:"
+ " errno = " + errno + ", sqlstate = " + sqlstate
+ " errmsg = " + errmsg);
}
else if (mark == 254)
{
throw new IOException("Received error packet:" + " errno = " + errno + ", sqlstate = " + sqlstate
+ " errmsg = " + errmsg);
} else if (mark == 254) {
// Indicates end of stream. It's not clear when this would
// be sent.
logger.warn("Received EOF packet from server, apparent"
+ " master disconnected.");
logger.warn("Received EOF packet from server, apparent" + " master disconnected.");
return false;
}
else
{
} else {
// Should not happen.
throw new IOException("Unexpected response " + mark
+ " while fetching binlog: packet #" + netnum
+ ", len = " + netlen);
throw new IOException("Unexpected response " + mark + " while fetching binlog: packet #" + netnum
+ ", len = " + netlen);
}
}
// The first packet is a multi-packet, concatenate the packets.
while (netlen == MAX_PACKET_LENGTH)
{
if (!fetch0(0, NET_HEADER_SIZE))
{
while (netlen == MAX_PACKET_LENGTH) {
if (!fetch0(0, NET_HEADER_SIZE)) {
logger.warn("Reached end of input stream while fetching header");
return false;
}
netlen = getUint24(PACKET_LEN_OFFSET);
netnum = getUint8(PACKET_SEQ_OFFSET);
if (!fetch0(limit, netlen))
{
logger.warn("Reached end of input stream: packet #"
+ netnum + ", len = " + netlen);
if (!fetch0(limit, netlen)) {
logger.warn("Reached end of input stream: packet #" + netnum + ", len = " + netlen);
return false;
}
}
@@ -423,43 +338,32 @@ public final class DirectLogFetcher extends LogFetcher
position = origin;
limit -= origin;
return true;
}
catch (SocketTimeoutException e)
{
} catch (SocketTimeoutException e) {
close(); /* Do cleanup */
logger.error("Socket timeout expired, closing connection", e);
throw e;
}
catch (InterruptedIOException e)
{
} catch (InterruptedIOException e) {
close(); /* Do cleanup */
logger.warn("I/O interrupted while reading from client socket", e);
throw e;
}
catch (IOException e)
{
} catch (IOException e) {
close(); /* Do cleanup */
logger.error("I/O error while reading from client socket", e);
throw e;
}
}
private final boolean fetch0(final int off, final int len)
throws IOException
{
private final boolean fetch0(final int off, final int len) throws IOException {
ensureCapacity(off + len);
for (int count, n = 0; n < len; n += count)
{
if (0 > (count = mysqlInput.read(buffer, off + n, len - n)))
{
for (int count, n = 0; n < len; n += count) {
if (0 > (count = mysqlInput.read(buffer, off + n, len - n))) {
// Reached end of input stream
return false;
}
}
if (limit < off + len)
limit = off + len;
if (limit < off + len) limit = off + len;
return true;
}
@@ -468,19 +372,14 @@ public final class DirectLogFetcher extends LogFetcher
*
* @see com.taobao.tddl.dbsync.binlog.LogFetcher#close()
*/
public void close() throws IOException
{
try
{
if (conn != null)
conn.close();
public void close() throws IOException {
try {
if (conn != null) conn.close();
conn = null;
mysqlInput = null;
mysqlOutput = null;
}
catch (SQLException e)
{
} catch (SQLException e) {
logger.warn("Unable to close connection", e);
}
}
@@ -129,11 +129,11 @@ public final class FileLogFetcher extends LogFetcher {
}
} else if (limit > 0) {
System.arraycopy(buffer, origin, buffer, 0, limit);
position -= origin;
origin = 0;
final int len = fin.read(buffer, limit, buffer.length - limit);
if (len >= 0) {
limit += len;
position -= origin;
origin = 0;
/* More binlog to fetch */
return true;
@@ -0,0 +1,402 @@
package com.taobao.tddl.dbsync.binlog;
/**
* 处理下MySQL json二进制转化为可读的字符串
*
* @author agapple 2016年6月30日 上午11:26:17
* @since 1.0.22
*/
public class JsonConversion {
// JSON TYPE
public static final int JSONB_TYPE_SMALL_OBJECT = 0x0;
public static final int JSONB_TYPE_LARGE_OBJECT = 0x1;
public static final int JSONB_TYPE_SMALL_ARRAY = 0x2;
public static final int JSONB_TYPE_LARGE_ARRAY = 0x3;
public static final int JSONB_TYPE_LITERAL = 0x4;
public static final int JSONB_TYPE_INT16 = 0x5;
public static final int JSONB_TYPE_UINT16 = 0x6;
public static final int JSONB_TYPE_INT32 = 0x7;
public static final int JSONB_TYPE_UINT32 = 0x8;
public static final int JSONB_TYPE_INT64 = 0x9;
public static final int JSONB_TYPE_UINT64 = 0xA;
public static final int JSONB_TYPE_DOUBLE = 0xB;
public static final int JSONB_TYPE_STRING = 0xC;
public static final int JSONB_TYPE_OPAQUE = 0xF;
public static final char JSONB_NULL_LITERAL = '\0';
public static final char JSONB_TRUE_LITERAL = '\1';
public static final char JSONB_FALSE_LITERAL = '\2';
/*
* The size of offset or size fields in the small and the large storage
* format for JSON objects and JSON arrays.
*/
public static final int SMALL_OFFSET_SIZE = 2;
public static final int LARGE_OFFSET_SIZE = 4;
/*
* The size of key entries for objects when using the small storage format
* or the large storage format. In the small format it is 4 bytes (2 bytes
* for key length and 2 bytes for key offset). In the large format it is 6
* (2 bytes for length, 4 bytes for offset).
*/
public static final int KEY_ENTRY_SIZE_SMALL = (2 + SMALL_OFFSET_SIZE);
public static final int KEY_ENTRY_SIZE_LARGE = (2 + LARGE_OFFSET_SIZE);
/*
* The size of value entries for objects or arrays. When using the small
* storage format, the entry size is 3 (1 byte for type, 2 bytes for
* offset). When using the large storage format, it is 5 (1 byte for type, 4
* bytes for offset).
*/
public static final int VALUE_ENTRY_SIZE_SMALL = (1 + SMALL_OFFSET_SIZE);
public static final int VALUE_ENTRY_SIZE_LARGE = (1 + LARGE_OFFSET_SIZE);
public static Json_Value parse_value(int type, LogBuffer buffer, long len) {
buffer = buffer.duplicate(buffer.position(), (int) len);
switch (type) {
case JSONB_TYPE_SMALL_OBJECT:
return parse_array_or_object(Json_enum_type.OBJECT, buffer, len, false);
case JSONB_TYPE_LARGE_OBJECT:
return parse_array_or_object(Json_enum_type.OBJECT, buffer, len, true);
case JSONB_TYPE_SMALL_ARRAY:
return parse_array_or_object(Json_enum_type.ARRAY, buffer, len, false);
case JSONB_TYPE_LARGE_ARRAY:
return parse_array_or_object(Json_enum_type.ARRAY, buffer, len, true);
default:
return parse_scalar(type, buffer, len);
}
}
private static Json_Value parse_array_or_object(Json_enum_type type, LogBuffer buffer, long len, boolean large) {
long offset_size = large ? LARGE_OFFSET_SIZE : SMALL_OFFSET_SIZE;
if (len < 2 * offset_size) {
throw new IllegalArgumentException("illegal json data");
}
long element_count = read_offset_or_size(buffer, large);
long bytes = read_offset_or_size(buffer, large);
if (bytes > len) {
throw new IllegalArgumentException("illegal json data");
}
long header_size = 2 * offset_size;
if (type == Json_enum_type.OBJECT) {
header_size += element_count * (large ? KEY_ENTRY_SIZE_LARGE : KEY_ENTRY_SIZE_SMALL);
}
header_size += element_count * (large ? VALUE_ENTRY_SIZE_LARGE : VALUE_ENTRY_SIZE_SMALL);
if (header_size > bytes) {
throw new IllegalArgumentException("illegal json data");
}
return new Json_Value(type, buffer.rewind(), element_count, bytes, large);
}
private static long read_offset_or_size(LogBuffer buffer, boolean large) {
return large ? buffer.getUint32() : buffer.getUint16();
}
private static Json_Value parse_scalar(int type, LogBuffer buffer, long len) {
switch (type) {
case JSONB_TYPE_LITERAL:
/* purecov: inspected */
int data = buffer.getUint8();
switch (data) {
case JSONB_NULL_LITERAL:
return new Json_Value(Json_enum_type.LITERAL_NULL);
case JSONB_TRUE_LITERAL:
return new Json_Value(Json_enum_type.LITERAL_TRUE);
case JSONB_FALSE_LITERAL:
return new Json_Value(Json_enum_type.LITERAL_FALSE);
default:
throw new IllegalArgumentException("illegal json data");
}
case JSONB_TYPE_INT16:
return new Json_Value(Json_enum_type.INT, buffer.getInt16());
case JSONB_TYPE_INT32:
return new Json_Value(Json_enum_type.INT, buffer.getInt32());
case JSONB_TYPE_INT64:
return new Json_Value(Json_enum_type.INT, buffer.getLong64());
case JSONB_TYPE_UINT16:
return new Json_Value(Json_enum_type.UINT, buffer.getUint16());
case JSONB_TYPE_UINT32:
return new Json_Value(Json_enum_type.UINT, buffer.getUint32());
case JSONB_TYPE_UINT64:
return new Json_Value(Json_enum_type.UINT, buffer.getUlong64());
case JSONB_TYPE_DOUBLE:
return new Json_Value(Json_enum_type.DOUBLE, Double.valueOf(buffer.getDouble64()));
case JSONB_TYPE_STRING:
int max_bytes = (int) Math.min(len, 5);
long tlen = 0;
long str_len = 0;
long n = 0;
byte[] datas = buffer.getData(max_bytes);
for (int i = 0; i < max_bytes; i++) {
// Get the next 7 bits of the length.
tlen |= (datas[i] & 0x7f) << (7 * i);
if ((datas[i] & 0x80) == 0) {
// The length shouldn't exceed 32 bits.
if (tlen > 4294967296L) {
throw new IllegalArgumentException("illegal json data");
}
// This was the last byte. Return successfully.
n = i + 1;
str_len = tlen;
break;
}
}
if (str_len == 0 || len < n + str_len) {
throw new IllegalArgumentException("illegal json data");
}
return new Json_Value(Json_enum_type.STRING, buffer.rewind()
.forward((int) n)
.getFixString((int) str_len));
case JSONB_TYPE_OPAQUE:
/*
* There should always be at least one byte, which tells the
* field type of the opaque value.
*/
// The type is encoded as a uint8 that maps to an
// enum_field_types.
int type_byte = buffer.getUint8();
int position = buffer.position();
// Then there's the length of the value.
int q_max_bytes = (int) Math.min(len, 5);
long q_tlen = 0;
long q_str_len = 0;
long q_n = 0;
byte[] q_datas = buffer.getData(q_max_bytes);
for (int i = 0; i < q_max_bytes; i++) {
// Get the next 7 bits of the length.
q_tlen |= (q_datas[i] & 0x7f) << (7 * i);
if ((q_datas[i] & 0x80) == 0) {
// The length shouldn't exceed 32 bits.
if (q_tlen > 4294967296L) {
throw new IllegalArgumentException("illegal json data");
}
// This was the last byte. Return successfully.
q_n = i + 1;
q_str_len = q_tlen;
break;
}
}
if (q_str_len == 0 || len < q_n + q_str_len) {
throw new IllegalArgumentException("illegal json data");
}
return new Json_Value(type_byte, buffer.position(position).forward((int) q_n), q_str_len);
default:
throw new IllegalArgumentException("illegal json data");
}
}
public static class Json_Value {
Json_enum_type m_type;
int m_field_type;
LogBuffer m_data;
long m_element_count;
long m_length;
String m_string_value;
Number m_int_value;
double m_double_value;
boolean m_large;
public Json_Value(Json_enum_type t){
this.m_type = t;
}
public Json_Value(Json_enum_type t, Number val){
this.m_type = t;
if (t == Json_enum_type.DOUBLE) {
this.m_double_value = val.doubleValue();
} else {
this.m_int_value = val;
}
}
public Json_Value(Json_enum_type t, String value){
this.m_type = t;
this.m_string_value = value;
}
public Json_Value(int field_type, LogBuffer data, long bytes){
this.m_type = Json_enum_type.OPAQUE; // 不确定类型
this.m_field_type = field_type;
this.m_data = data;
this.m_length = bytes;
}
public Json_Value(Json_enum_type t, LogBuffer data, long element_count, long bytes, boolean large){
this.m_type = t;
this.m_data = data;
this.m_element_count = element_count;
this.m_length = bytes;
this.m_large = large;
}
public String key(int i) {
m_data.rewind();
int offset_size = m_large ? LARGE_OFFSET_SIZE : SMALL_OFFSET_SIZE;
int key_entry_size = m_large ? KEY_ENTRY_SIZE_LARGE : KEY_ENTRY_SIZE_SMALL;
int entry_offset = 2 * offset_size + key_entry_size * i;
// The offset of the key is the first part of the key
// entry.
m_data.forward(entry_offset);
long key_offset = read_offset_or_size(m_data, m_large);
// The length of the key is the second part of the
// entry, always two
// bytes.
long key_length = m_data.getUint16();
return m_data.rewind().forward((int) key_offset).getFixString((int) key_length);
}
public Json_Value element(int i) {
m_data.rewind();
int offset_size = m_large ? LARGE_OFFSET_SIZE : SMALL_OFFSET_SIZE;
int key_entry_size = m_large ? KEY_ENTRY_SIZE_LARGE : KEY_ENTRY_SIZE_SMALL;
int value_entry_size = m_large ? VALUE_ENTRY_SIZE_LARGE : VALUE_ENTRY_SIZE_SMALL;
int first_entry_offset = 2 * offset_size;
if (m_type == Json_enum_type.OBJECT) {
first_entry_offset += m_element_count * key_entry_size;
}
int entry_offset = first_entry_offset + value_entry_size * i;
int type = m_data.forward(entry_offset).getUint8();
if (type == JSONB_TYPE_INT16 || type == JSONB_TYPE_UINT16 || type == JSONB_TYPE_LITERAL
|| (m_large && (type == JSONB_TYPE_INT32 || type == JSONB_TYPE_UINT32))) {
return parse_scalar(type, m_data, value_entry_size - 1);
}
int value_offset = (int) read_offset_or_size(m_data, m_large);
return parse_value(type, m_data.rewind().forward(value_offset), (int) m_length - value_offset);
}
public StringBuilder toJsonString(StringBuilder buf) {
switch (m_type) {
case OBJECT:
buf.append("{");
for (int i = 0; i < m_element_count; ++i) {
if (i > 0) {
buf.append(", ");
}
buf.append('"').append(key(i)).append('"');
buf.append(": ");
element(i).toJsonString(buf);
}
buf.append("}");
break;
case ARRAY:
buf.append("[");
for (int i = 0; i < m_element_count; ++i) {
if (i > 0) {
buf.append(", ");
}
element(i).toJsonString(buf);
}
buf.append("]");
break;
case DOUBLE:
buf.append(Double.valueOf(m_double_value).toString());
break;
case INT:
buf.append(m_int_value.toString());
break;
case UINT:
buf.append(m_int_value.toString());
break;
case LITERAL_FALSE:
buf.append("false");
break;
case LITERAL_TRUE:
buf.append("true");
break;
case LITERAL_NULL:
buf.append("NULL");
break;
case OPAQUE:
String text = null;
if (m_field_type == LogEvent.MYSQL_TYPE_NEWDECIMAL) {
int precision = m_data.getInt8();
int scale = m_data.getInt8();
text = m_data.getDecimal(precision, scale).toPlainString();
buf.append(text);
} else if (m_field_type == LogEvent.MYSQL_TYPE_TIME) {
long packed_value = m_data.getLong64();
if (packed_value == 0) {
text = "00:00:00";
} else {
long ultime = Math.abs(packed_value);
long intpart = ultime >> 24;
int frac = (int) (ultime % (1L << 24));
text = String.format("%s%02d:%02d:%02d",
packed_value >= 0 ? "" : "-",
(int) ((intpart >> 12) % (1 << 10)),
(int) ((intpart >> 6) % (1 << 6)),
(int) (intpart % (1 << 6)));
text = text + "." + usecondsToStr(frac, 6);
}
buf.append('"').append(text).append('"');
} else if (m_field_type == LogEvent.MYSQL_TYPE_DATE || m_field_type == LogEvent.MYSQL_TYPE_DATETIME
|| m_field_type == LogEvent.MYSQL_TYPE_TIMESTAMP) {
long packed_value = m_data.getLong64();
if (packed_value == 0) {
text = "0000-00-00 00:00:00";
} else {
// 构造TimeStamp只处理到秒
long ultime = Math.abs(packed_value);
long intpart = ultime >> 24;
int frac = (int) (ultime % (1L << 24));
long ymd = intpart >> 17;
long ym = ymd >> 5;
long hms = intpart % (1 << 17);
text = String.format("%04d-%02d-%02d %02d:%02d:%02d",
(int) (ym / 13),
(int) (ym % 13),
(int) (ymd % (1 << 5)),
(int) (hms >> 12),
(int) ((hms >> 6) % (1 << 6)),
(int) (hms % (1 << 6)));
text = text + "." + usecondsToStr(frac, 6);
}
buf.append('"').append(text).append('"');
} else {
text = m_data.getFixString((int) m_length);
buf.append('"').append(text).append('"');
}
break;
case STRING:
buf.append('"').append(m_string_value).append('"');
break;
case ERROR:
throw new IllegalArgumentException("illegal json data");
}
return buf;
}
}
public static enum Json_enum_type {
OBJECT, ARRAY, STRING, INT, UINT, DOUBLE, LITERAL_NULL, LITERAL_TRUE, LITERAL_FALSE, OPAQUE, ERROR
}
private static String usecondsToStr(int frac, int meta) {
String sec = String.valueOf(frac);
if (meta > 6) {
throw new IllegalArgumentException("unknow useconds meta : " + meta);
}
if (sec.length() < 6) {
StringBuilder result = new StringBuilder(6);
int len = 6 - sec.length();
for (; len > 0; len--) {
result.append('0');
}
result.append(sec);
sec = result.toString();
}
return sec.substring(0, meta);
}
}
File diff suppressed because it is too large Load Diff
@@ -7,69 +7,56 @@ import com.taobao.tddl.dbsync.binlog.event.FormatDescriptionLogEvent;
import com.taobao.tddl.dbsync.binlog.event.TableMapLogEvent;
/**
* TODO: Document Me!!
*
* NOTE: Log context will NOT write multi-threaded.
* TODO: Document Me!! NOTE: Log context will NOT write multi-threaded.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class LogContext
{
public final class LogContext {
private final Map<Long, TableMapLogEvent> mapOfTable = new HashMap<Long, TableMapLogEvent>();
private FormatDescriptionLogEvent formatDescription;
private LogPosition logPosition;
public LogContext()
{
public LogContext(){
this.formatDescription = FormatDescriptionLogEvent.FORMAT_DESCRIPTION_EVENT_5_x;
}
public LogContext(FormatDescriptionLogEvent descriptionEvent)
{
public LogContext(FormatDescriptionLogEvent descriptionEvent){
this.formatDescription = descriptionEvent;
}
public final LogPosition getLogPosition()
{
public final LogPosition getLogPosition() {
return logPosition;
}
public final void setLogPosition(LogPosition logPosition)
{
public final void setLogPosition(LogPosition logPosition) {
this.logPosition = logPosition;
}
public final FormatDescriptionLogEvent getFormatDescription()
{
public final FormatDescriptionLogEvent getFormatDescription() {
return formatDescription;
}
public final void setFormatDescription(
FormatDescriptionLogEvent formatDescription)
{
public final void setFormatDescription(FormatDescriptionLogEvent formatDescription) {
this.formatDescription = formatDescription;
}
public final void putTable(TableMapLogEvent mapEvent)
{
public final void putTable(TableMapLogEvent mapEvent) {
mapOfTable.put(Long.valueOf(mapEvent.getTableId()), mapEvent);
}
public final TableMapLogEvent getTable(final long tableId)
{
public final TableMapLogEvent getTable(final long tableId) {
return mapOfTable.get(Long.valueOf(tableId));
}
public final void clearAllTables()
{
public final void clearAllTables() {
mapOfTable.clear();
}
public void reset()
{
public void reset() {
formatDescription = FormatDescriptionLogEvent.FORMAT_DESCRIPTION_EVENT_5_x;
mapOfTable.clear();
@@ -61,28 +61,24 @@ import com.taobao.tddl.dbsync.binlog.event.mariadb.MariaGtidLogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class LogDecoder
{
public final class LogDecoder {
protected static final Log logger = LogFactory.getLog(LogDecoder.class);
protected final BitSet handleSet = new BitSet(LogEvent.ENUM_END_EVENT);
public LogDecoder()
{
public LogDecoder(){
}
public LogDecoder(final int fromIndex, final int toIndex)
{
public LogDecoder(final int fromIndex, final int toIndex){
handleSet.set(fromIndex, toIndex);
}
public final void handle(final int fromIndex, final int toIndex)
{
public final void handle(final int fromIndex, final int toIndex) {
handleSet.set(fromIndex, toIndex);
}
public final void handle(final int flagIndex)
{
public final void handle(final int flagIndex) {
handleSet.set(flagIndex);
}
@@ -90,48 +86,32 @@ public final class LogDecoder
* Decoding an event from binary-log buffer.
*
* @return <code>UknownLogEvent</code> if event type is unknown or skipped,
* <code>null</code> if buffer is not including a full event.
* <code>null</code> if buffer is not including a full event.
*/
public LogEvent decode(LogBuffer buffer, LogContext context)
throws IOException
{
public LogEvent decode(LogBuffer buffer, LogContext context) throws IOException {
final int limit = buffer.limit();
if (limit >= FormatDescriptionLogEvent.LOG_EVENT_HEADER_LEN)
{
LogHeader header = new LogHeader(buffer,
context.getFormatDescription());
if (limit >= FormatDescriptionLogEvent.LOG_EVENT_HEADER_LEN) {
LogHeader header = new LogHeader(buffer, context.getFormatDescription());
final int len = header.getEventLen();
if (limit >= len)
{
if (limit >= len) {
LogEvent event;
/* Checking binary-log's header */
if (handleSet.get(header.getType()))
{
if (handleSet.get(header.getType())) {
buffer.limit(len);
try
{
try {
/* Decoding binary-log to event */
event = decode(buffer, header, context);
}
catch (IOException e)
{
if (logger.isWarnEnabled())
logger.warn("Decoding "
+ LogEvent.getTypeName(header.getType())
+ " failed from: "
+ context.getLogPosition(), e);
} catch (IOException e) {
if (logger.isWarnEnabled()) logger.warn("Decoding " + LogEvent.getTypeName(header.getType())
+ " failed from: " + context.getLogPosition(), e);
throw e;
}
finally
{
} finally {
buffer.limit(limit); /* Restore limit */
}
}
else
{
} else {
/* Ignore unsupported binary-log. */
event = new UnknownLogEvent(header);
}
@@ -152,338 +132,275 @@ public final class LogDecoder
*
* @return <code>UknownLogEvent</code> if event type is unknown or skipped.
*/
public static LogEvent decode(LogBuffer buffer, LogHeader header,
LogContext context) throws IOException
{
public static LogEvent decode(LogBuffer buffer, LogHeader header, LogContext context) throws IOException {
FormatDescriptionLogEvent descriptionEvent = context.getFormatDescription();
LogPosition logPosition = context.getLogPosition();
int checksumAlg = LogEvent.BINLOG_CHECKSUM_ALG_UNDEF;
if (header.getType() != LogEvent.FORMAT_DESCRIPTION_EVENT) {
checksumAlg = descriptionEvent.header.getChecksumAlg();
}else {
} else {
// 如果是format事件自己,也需要处理checksum
checksumAlg = header.getChecksumAlg();
}
if (checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_OFF && checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_UNDEF) {
// remove checksum bytes
buffer.limit(header.getEventLen() - LogEvent.BINLOG_CHECKSUM_LEN);
}
switch (header.getType())
{
case LogEvent.QUERY_EVENT:
{
QueryLogEvent event = new QueryLogEvent(header, buffer,
descriptionEvent);
switch (header.getType()) {
case LogEvent.QUERY_EVENT: {
QueryLogEvent event = new QueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.XID_EVENT:
{
XidLogEvent event = new XidLogEvent(header, buffer,
descriptionEvent);
case LogEvent.XID_EVENT: {
XidLogEvent event = new XidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.TABLE_MAP_EVENT:
{
TableMapLogEvent mapEvent = new TableMapLogEvent(header,
buffer, descriptionEvent);
case LogEvent.TABLE_MAP_EVENT: {
TableMapLogEvent mapEvent = new TableMapLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
context.putTable(mapEvent);
return mapEvent;
}
case LogEvent.WRITE_ROWS_EVENT_V1:
{
RowsLogEvent event = new WriteRowsLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.UPDATE_ROWS_EVENT_V1:
{
RowsLogEvent event = new UpdateRowsLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.DELETE_ROWS_EVENT_V1:
{
RowsLogEvent event = new DeleteRowsLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.ROTATE_EVENT:
{
RotateLogEvent event = new RotateLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition = new LogPosition(event.getFilename(),
event.getPosition());
context.setLogPosition(logPosition);
return event;
}
case LogEvent.LOAD_EVENT:
case LogEvent.NEW_LOAD_EVENT:
{
LoadLogEvent event = new LoadLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.SLAVE_EVENT: /* can never happen (unused event) */
{
if (logger.isWarnEnabled())
logger.warn("Skipping unsupported SLAVE_EVENT from: "
+ context.getLogPosition());
break;
}
case LogEvent.CREATE_FILE_EVENT:
{
CreateFileLogEvent event = new CreateFileLogEvent(header,
buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.APPEND_BLOCK_EVENT:
{
AppendBlockLogEvent event = new AppendBlockLogEvent(header,
buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.DELETE_FILE_EVENT:
{
DeleteFileLogEvent event = new DeleteFileLogEvent(header,
buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.EXEC_LOAD_EVENT:
{
ExecuteLoadLogEvent event = new ExecuteLoadLogEvent(header,
buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.START_EVENT_V3:
{
/* This is sent only by MySQL <=4.x */
StartLogEventV3 event = new StartLogEventV3(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.STOP_EVENT:
{
StopLogEvent event = new StopLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.INTVAR_EVENT:
{
IntvarLogEvent event = new IntvarLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.RAND_EVENT:
{
RandLogEvent event = new RandLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.USER_VAR_EVENT:
{
UserVarLogEvent event = new UserVarLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.FORMAT_DESCRIPTION_EVENT:
{
descriptionEvent = new FormatDescriptionLogEvent(header,
buffer, descriptionEvent);
context.setFormatDescription(descriptionEvent);
return descriptionEvent;
}
case LogEvent.PRE_GA_WRITE_ROWS_EVENT:
{
if (logger.isWarnEnabled())
logger.warn("Skipping unsupported PRE_GA_WRITE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Write_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.PRE_GA_UPDATE_ROWS_EVENT:
{
if (logger.isWarnEnabled())
logger.warn("Skipping unsupported PRE_GA_UPDATE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Update_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.PRE_GA_DELETE_ROWS_EVENT:
{
if (logger.isWarnEnabled())
logger.warn("Skipping unsupported PRE_GA_DELETE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Delete_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.BEGIN_LOAD_QUERY_EVENT:
{
BeginLoadQueryLogEvent event = new BeginLoadQueryLogEvent(
header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.EXECUTE_LOAD_QUERY_EVENT:
{
ExecuteLoadQueryLogEvent event = new ExecuteLoadQueryLogEvent(
header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.INCIDENT_EVENT:
{
IncidentLogEvent event = new IncidentLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.HEARTBEAT_LOG_EVENT:
{
HeartbeatLogEvent event = new HeartbeatLogEvent(header, buffer,
descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.IGNORABLE_LOG_EVENT:
{
IgnorableLogEvent event = new IgnorableLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.ROWS_QUERY_LOG_EVENT:
{
RowsQueryLogEvent event = new RowsQueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.WRITE_ROWS_EVENT: {
case LogEvent.WRITE_ROWS_EVENT_V1: {
RowsLogEvent event = new WriteRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.UPDATE_ROWS_EVENT: {
case LogEvent.UPDATE_ROWS_EVENT_V1: {
RowsLogEvent event = new UpdateRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.DELETE_ROWS_EVENT: {
case LogEvent.DELETE_ROWS_EVENT_V1: {
RowsLogEvent event = new DeleteRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.GTID_LOG_EVENT:
case LogEvent.ANONYMOUS_GTID_LOG_EVENT:
case LogEvent.ROTATE_EVENT: {
RotateLogEvent event = new RotateLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition = new LogPosition(event.getFilename(), event.getPosition());
context.setLogPosition(logPosition);
return event;
}
case LogEvent.LOAD_EVENT:
case LogEvent.NEW_LOAD_EVENT: {
LoadLogEvent event = new LoadLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.SLAVE_EVENT: /* can never happen (unused event) */
{
if (logger.isWarnEnabled()) logger.warn("Skipping unsupported SLAVE_EVENT from: "
+ context.getLogPosition());
break;
}
case LogEvent.CREATE_FILE_EVENT: {
CreateFileLogEvent event = new CreateFileLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.APPEND_BLOCK_EVENT: {
AppendBlockLogEvent event = new AppendBlockLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.DELETE_FILE_EVENT: {
DeleteFileLogEvent event = new DeleteFileLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.EXEC_LOAD_EVENT: {
ExecuteLoadLogEvent event = new ExecuteLoadLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.START_EVENT_V3: {
/* This is sent only by MySQL <=4.x */
StartLogEventV3 event = new StartLogEventV3(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.STOP_EVENT: {
StopLogEvent event = new StopLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.INTVAR_EVENT: {
IntvarLogEvent event = new IntvarLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.RAND_EVENT: {
RandLogEvent event = new RandLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.USER_VAR_EVENT: {
UserVarLogEvent event = new UserVarLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.FORMAT_DESCRIPTION_EVENT: {
descriptionEvent = new FormatDescriptionLogEvent(header, buffer, descriptionEvent);
context.setFormatDescription(descriptionEvent);
return descriptionEvent;
}
case LogEvent.PRE_GA_WRITE_ROWS_EVENT: {
if (logger.isWarnEnabled()) logger.warn("Skipping unsupported PRE_GA_WRITE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Write_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.PRE_GA_UPDATE_ROWS_EVENT: {
if (logger.isWarnEnabled()) logger.warn("Skipping unsupported PRE_GA_UPDATE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Update_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.PRE_GA_DELETE_ROWS_EVENT: {
if (logger.isWarnEnabled()) logger.warn("Skipping unsupported PRE_GA_DELETE_ROWS_EVENT from: "
+ context.getLogPosition());
// ev = new Delete_rows_log_event_old(buf, event_len,
// description_event);
break;
}
case LogEvent.BEGIN_LOAD_QUERY_EVENT: {
BeginLoadQueryLogEvent event = new BeginLoadQueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.EXECUTE_LOAD_QUERY_EVENT: {
ExecuteLoadQueryLogEvent event = new ExecuteLoadQueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.INCIDENT_EVENT: {
IncidentLogEvent event = new IncidentLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.HEARTBEAT_LOG_EVENT: {
HeartbeatLogEvent event = new HeartbeatLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.IGNORABLE_LOG_EVENT: {
IgnorableLogEvent event = new IgnorableLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.ROWS_QUERY_LOG_EVENT: {
RowsQueryLogEvent event = new RowsQueryLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.WRITE_ROWS_EVENT: {
RowsLogEvent event = new WriteRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.UPDATE_ROWS_EVENT: {
RowsLogEvent event = new UpdateRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.DELETE_ROWS_EVENT: {
RowsLogEvent event = new DeleteRowsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
event.fillTable(context);
return event;
}
case LogEvent.GTID_LOG_EVENT:
case LogEvent.ANONYMOUS_GTID_LOG_EVENT: {
GtidLogEvent event = new GtidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.PREVIOUS_GTIDS_LOG_EVENT:
{
case LogEvent.PREVIOUS_GTIDS_LOG_EVENT: {
PreviousGtidsLogEvent event = new PreviousGtidsLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.ANNOTATE_ROWS_EVENT:
{
AnnotateRowsEvent event = new AnnotateRowsEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.BINLOG_CHECKPOINT_EVENT:
{
BinlogCheckPointLogEvent event = new BinlogCheckPointLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.GTID_EVENT:
{
MariaGtidLogEvent event = new MariaGtidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.GTID_LIST_EVENT:
{
MariaGtidListLogEvent event = new MariaGtidListLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
default:
/*
Create an object of Ignorable_log_event for unrecognized sub-class.
So that SLAVE SQL THREAD will only update the position and continue.
*/
if((buffer.getUint16(LogEvent.FLAGS_OFFSET) & LogEvent.LOG_EVENT_IGNORABLE_F) > 0){
IgnorableLogEvent event = new IgnorableLogEvent(header, buffer, descriptionEvent);
case LogEvent.ANNOTATE_ROWS_EVENT: {
AnnotateRowsEvent event = new AnnotateRowsEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}else {
if (logger.isWarnEnabled())
logger.warn("Skipping unrecognized binlog event "
+ LogEvent.getTypeName(header.getType()) + " from: "
+ context.getLogPosition());
}
case LogEvent.BINLOG_CHECKPOINT_EVENT: {
BinlogCheckPointLogEvent event = new BinlogCheckPointLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.GTID_EVENT: {
MariaGtidLogEvent event = new MariaGtidLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
case LogEvent.GTID_LIST_EVENT: {
MariaGtidListLogEvent event = new MariaGtidListLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
}
default:
/*
* Create an object of Ignorable_log_event for unrecognized
* sub-class. So that SLAVE SQL THREAD will only update the
* position and continue.
*/
if ((buffer.getUint16(LogEvent.FLAGS_OFFSET) & LogEvent.LOG_EVENT_IGNORABLE_F) > 0) {
IgnorableLogEvent event = new IgnorableLogEvent(header, buffer, descriptionEvent);
/* updating position in context */
logPosition.position = header.getLogPos();
return event;
} else {
if (logger.isWarnEnabled()) logger.warn("Skipping unrecognized binlog event "
+ LogEvent.getTypeName(header.getType()) + " from: "
+ context.getLogPosition());
}
}
/* updating position in context */
@@ -7,239 +7,218 @@ import com.taobao.tddl.dbsync.binlog.event.LogHeader;
/**
* Binary log event definitions. This includes generic code common to all types
* of log events, as well as specific code for each type of log event.
*
* - All numbers, whether they are 16-, 24-, 32-, or 64-bit numbers, are stored
* in little endian, i.e., the least significant byte first, unless otherwise
* specified.
*
* representation of unsigned integers, called Packed Integer. A Packed Integer
* has the capacity of storing up to 8-byte integers, while small integers still
* can use 1, 3, or 4 bytes. The value of the first byte determines how to read
* the number, according to the following table:
*
* of log events, as well as specific code for each type of log event. - All
* numbers, whether they are 16-, 24-, 32-, or 64-bit numbers, are stored in
* little endian, i.e., the least significant byte first, unless otherwise
* specified. representation of unsigned integers, called Packed Integer. A
* Packed Integer has the capacity of storing up to 8-byte integers, while small
* integers still can use 1, 3, or 4 bytes. The value of the first byte
* determines how to read the number, according to the following table:
* <table>
* <caption>Format of Packed Integer</caption>
*
* <tr>
* <th>First byte</th>
* <th>Format</th>
* </tr>
*
* <tr>
* <td>0-250</td>
* <td>The first byte is the number (in the range 0-250), and no more bytes are
* used.</td>
* </tr>
*
* <tr>
* <td>252</td>
* <td>Two more bytes are used. The number is in the range 251-0xffff.</td>
* </tr>
*
* <tr>
* <td>253</td>
* <td>Three more bytes are used. The number is in the range 0xffff-0xffffff.</td>
* </tr>
*
* <tr>
* <td>254</td>
* <td>Eight more bytes are used. The number is in the range
* 0xffffff-0xffffffffffffffff.</td>
* </tr>
*
* </table>
*
* - Strings are stored in various formats. The format of each string is
* documented separately.
*
* @see mysql-5.1.60/sql/log_event.h
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public abstract class LogEvent
{
public abstract class LogEvent {
/*
* 3 is MySQL 4.x; 4 is MySQL 5.0.0.
* Compared to version 3, version 4 has:
* - a different Start_log_event, which includes info about the binary log
* 3 is MySQL 4.x; 4 is MySQL 5.0.0. Compared to version 3, version 4 has: -
* a different Start_log_event, which includes info about the binary log
* (sizes of headers); this info is included for better compatibility if the
* master's MySQL version is different from the slave's.
* - all events have a unique ID (the triplet (server_id, timestamp at server
* start, other) to be sure an event is not executed more than once in a
* multimaster setup, example:
* M1
* / \
* v v
* M2 M3
* \ /
* v v
* S
* if a query is run on M1, it will arrive twice on S, so we need that S
* remembers the last unique ID it has processed, to compare and know if the
* event should be skipped or not. Example of ID: we already have the server id
* (4 bytes), plus:
* master's MySQL version is different from the slave's. - all events have a
* unique ID (the triplet (server_id, timestamp at server start, other) to
* be sure an event is not executed more than once in a multimaster setup,
* example: M1 / \ v v M2 M3 \ / v v S if a query is run on M1, it will
* arrive twice on S, so we need that S remembers the last unique ID it has
* processed, to compare and know if the event should be skipped or not.
* Example of ID: we already have the server id (4 bytes), plus:
* timestamp_when_the_master_started (4 bytes), a counter (a sequence number
* which increments every time we write an event to the binlog) (3 bytes).
* Q: how do we handle when the counter is overflowed and restarts from 0 ?
*
* - Query and Load (Create or Execute) events may have a more precise
* timestamp (with microseconds), number of matched/affected/warnings rows
* and fields of session variables: SQL_MODE,
* FOREIGN_KEY_CHECKS, UNIQUE_CHECKS, SQL_AUTO_IS_NULL, the collations and
* charsets, the PASSWORD() version (old/new/...).
* timestamp (with microseconds), number of matched/affected/warnings rows
* and fields of session variables: SQL_MODE, FOREIGN_KEY_CHECKS,
* UNIQUE_CHECKS, SQL_AUTO_IS_NULL, the collations and charsets, the
* PASSWORD() version (old/new/...).
*/
public static final int BINLOG_VERSION = 4;
public static final int BINLOG_VERSION = 4;
/* Default 5.0 server version */
public static final String SERVER_VERSION = "5.0";
public static final String SERVER_VERSION = "5.0";
/**
* Event header offsets; these point to places inside the fixed header.
*/
public static final int EVENT_TYPE_OFFSET = 4;
public static final int SERVER_ID_OFFSET = 5;
public static final int EVENT_LEN_OFFSET = 9;
public static final int LOG_POS_OFFSET = 13;
public static final int FLAGS_OFFSET = 17;
public static final int EVENT_TYPE_OFFSET = 4;
public static final int SERVER_ID_OFFSET = 5;
public static final int EVENT_LEN_OFFSET = 9;
public static final int LOG_POS_OFFSET = 13;
public static final int FLAGS_OFFSET = 17;
/* event-specific post-header sizes */
// where 3.23, 4.x and 5.0 agree
public static final int QUERY_HEADER_MINIMAL_LEN = (4 + 4 + 1 + 2);
public static final int QUERY_HEADER_MINIMAL_LEN = (4 + 4 + 1 + 2);
// where 5.0 differs: 2 for len of N-bytes vars.
public static final int QUERY_HEADER_LEN = (QUERY_HEADER_MINIMAL_LEN + 2);
public static final int QUERY_HEADER_LEN = (QUERY_HEADER_MINIMAL_LEN + 2);
/* Enumeration type for the different types of log events. */
public static final int UNKNOWN_EVENT = 0;
public static final int START_EVENT_V3 = 1;
public static final int QUERY_EVENT = 2;
public static final int STOP_EVENT = 3;
public static final int ROTATE_EVENT = 4;
public static final int INTVAR_EVENT = 5;
public static final int LOAD_EVENT = 6;
public static final int SLAVE_EVENT = 7;
public static final int CREATE_FILE_EVENT = 8;
public static final int APPEND_BLOCK_EVENT = 9;
public static final int EXEC_LOAD_EVENT = 10;
public static final int DELETE_FILE_EVENT = 11;
public static final int UNKNOWN_EVENT = 0;
public static final int START_EVENT_V3 = 1;
public static final int QUERY_EVENT = 2;
public static final int STOP_EVENT = 3;
public static final int ROTATE_EVENT = 4;
public static final int INTVAR_EVENT = 5;
public static final int LOAD_EVENT = 6;
public static final int SLAVE_EVENT = 7;
public static final int CREATE_FILE_EVENT = 8;
public static final int APPEND_BLOCK_EVENT = 9;
public static final int EXEC_LOAD_EVENT = 10;
public static final int DELETE_FILE_EVENT = 11;
/**
* NEW_LOAD_EVENT is like LOAD_EVENT except that it has a longer sql_ex,
* allowing multibyte TERMINATED BY etc; both types share the same class
* (Load_log_event)
*/
public static final int NEW_LOAD_EVENT = 12;
public static final int RAND_EVENT = 13;
public static final int USER_VAR_EVENT = 14;
public static final int FORMAT_DESCRIPTION_EVENT = 15;
public static final int XID_EVENT = 16;
public static final int BEGIN_LOAD_QUERY_EVENT = 17;
public static final int EXECUTE_LOAD_QUERY_EVENT = 18;
public static final int NEW_LOAD_EVENT = 12;
public static final int RAND_EVENT = 13;
public static final int USER_VAR_EVENT = 14;
public static final int FORMAT_DESCRIPTION_EVENT = 15;
public static final int XID_EVENT = 16;
public static final int BEGIN_LOAD_QUERY_EVENT = 17;
public static final int EXECUTE_LOAD_QUERY_EVENT = 18;
public static final int TABLE_MAP_EVENT = 19;
public static final int TABLE_MAP_EVENT = 19;
/**
* These event numbers were used for 5.1.0 to 5.1.15 and are therefore
* obsolete.
*/
public static final int PRE_GA_WRITE_ROWS_EVENT = 20;
public static final int PRE_GA_UPDATE_ROWS_EVENT = 21;
public static final int PRE_GA_DELETE_ROWS_EVENT = 22;
public static final int PRE_GA_WRITE_ROWS_EVENT = 20;
public static final int PRE_GA_UPDATE_ROWS_EVENT = 21;
public static final int PRE_GA_DELETE_ROWS_EVENT = 22;
/**
* These event numbers are used from 5.1.16 and forward
*/
public static final int WRITE_ROWS_EVENT_V1 = 23;
public static final int UPDATE_ROWS_EVENT_V1 = 24;
public static final int DELETE_ROWS_EVENT_V1 = 25;
public static final int WRITE_ROWS_EVENT_V1 = 23;
public static final int UPDATE_ROWS_EVENT_V1 = 24;
public static final int DELETE_ROWS_EVENT_V1 = 25;
/**
* Something out of the ordinary happened on the master
*/
public static final int INCIDENT_EVENT = 26;
public static final int INCIDENT_EVENT = 26;
/**
* Heartbeat event to be send by master at its idle time to ensure master's online status to slave
* Heartbeat event to be send by master at its idle time to ensure master's
* online status to slave
*/
public static final int HEARTBEAT_LOG_EVENT = 27;
public static final int HEARTBEAT_LOG_EVENT = 27;
/**
* In some situations, it is necessary to send over ignorable data to the slave: data that a slave can handle in
* case there is code for handling it, but which can be ignored if it is not recognized.
* In some situations, it is necessary to send over ignorable data to the
* slave: data that a slave can handle in case there is code for handling
* it, but which can be ignored if it is not recognized.
*/
public static final int IGNORABLE_LOG_EVENT = 28;
public static final int ROWS_QUERY_LOG_EVENT = 29;
public static final int IGNORABLE_LOG_EVENT = 28;
public static final int ROWS_QUERY_LOG_EVENT = 29;
/** Version 2 of the Row events */
public static final int WRITE_ROWS_EVENT = 30;
public static final int UPDATE_ROWS_EVENT = 31;
public static final int DELETE_ROWS_EVENT = 32;
public static final int WRITE_ROWS_EVENT = 30;
public static final int UPDATE_ROWS_EVENT = 31;
public static final int DELETE_ROWS_EVENT = 32;
public static final int GTID_LOG_EVENT = 33;
public static final int ANONYMOUS_GTID_LOG_EVENT = 34;
public static final int GTID_LOG_EVENT = 33;
public static final int ANONYMOUS_GTID_LOG_EVENT = 34;
public static final int PREVIOUS_GTIDS_LOG_EVENT = 35;
public static final int PREVIOUS_GTIDS_LOG_EVENT = 35;
// mariaDb 5.5.34
/* New MySQL/Sun events are to be added right above this comment */
public static final int MYSQL_EVENTS_END = 36;
public static final int MYSQL_EVENTS_END = 36;
public static final int MARIA_EVENTS_BEGIN = 160;
public static final int MARIA_EVENTS_BEGIN = 160;
/* New Maria event numbers start from here */
public static final int ANNOTATE_ROWS_EVENT = 160;
public static final int ANNOTATE_ROWS_EVENT = 160;
/*
Binlog checkpoint event. Used for XA crash recovery on the master, not used
in replication.
A binlog checkpoint event specifies a binlog file such that XA crash
recovery can start from that file - and it is guaranteed to find all XIDs
that are prepared in storage engines but not yet committed.
* Binlog checkpoint event. Used for XA crash recovery on the master, not
* used in replication. A binlog checkpoint event specifies a binlog file
* such that XA crash recovery can start from that file - and it is
* guaranteed to find all XIDs that are prepared in storage engines but not
* yet committed.
*/
public static final int BINLOG_CHECKPOINT_EVENT = 161;
public static final int BINLOG_CHECKPOINT_EVENT = 161;
/*
Gtid event. For global transaction ID, used to start a new event group,
instead of the old BEGIN query event, and also to mark stand-alone
events.
* Gtid event. For global transaction ID, used to start a new event group,
* instead of the old BEGIN query event, and also to mark stand-alone
* events.
*/
public static final int GTID_EVENT = 162;
public static final int GTID_EVENT = 162;
/*
Gtid list event. Logged at the start of every binlog, to record the
current replication state. This consists of the last GTID seen for
each replication domain.
* Gtid list event. Logged at the start of every binlog, to record the
* current replication state. This consists of the last GTID seen for each
* replication domain.
*/
public static final int GTID_LIST_EVENT = 163;
public static final int GTID_LIST_EVENT = 163;
/** end marker */
public static final int ENUM_END_EVENT = 164;
public static final int ENUM_END_EVENT = 164;
/**
1 byte length, 1 byte format
Length is total length in bytes, including 2 byte header
Length values 0 and 1 are currently invalid and reserved.
*/
public static final int EXTRA_ROW_INFO_LEN_OFFSET = 0;
public static final int EXTRA_ROW_INFO_FORMAT_OFFSET = 1;
public static final int EXTRA_ROW_INFO_HDR_BYTES = 2;
public static final int EXTRA_ROW_INFO_MAX_PAYLOAD = (255 - EXTRA_ROW_INFO_HDR_BYTES);
* 1 byte length, 1 byte format Length is total length in bytes, including 2
* byte header Length values 0 and 1 are currently invalid and reserved.
*/
public static final int EXTRA_ROW_INFO_LEN_OFFSET = 0;
public static final int EXTRA_ROW_INFO_FORMAT_OFFSET = 1;
public static final int EXTRA_ROW_INFO_HDR_BYTES = 2;
public static final int EXTRA_ROW_INFO_MAX_PAYLOAD = (255 - EXTRA_ROW_INFO_HDR_BYTES);
// Events are without checksum though its generator
public static final int BINLOG_CHECKSUM_ALG_OFF = 0;
public static final int BINLOG_CHECKSUM_ALG_OFF = 0;
// is checksum-capable New Master (NM).
// CRC32 of zlib algorithm.
public static final int BINLOG_CHECKSUM_ALG_CRC32 = 1;
public static final int BINLOG_CHECKSUM_ALG_CRC32 = 1;
// the cut line: valid alg range is [1, 0x7f].
public static final int BINLOG_CHECKSUM_ALG_ENUM_END = 2;
public static final int BINLOG_CHECKSUM_ALG_ENUM_END = 2;
// special value to tag undetermined yet checksum
public static final int BINLOG_CHECKSUM_ALG_UNDEF = 255;
public static final int BINLOG_CHECKSUM_ALG_UNDEF = 255;
// or events from checksum-unaware servers
public static final int CHECKSUM_CRC32_SIGNATURE_LEN = 4;
public static final int BINLOG_CHECKSUM_ALG_DESC_LEN = 1;
public static final int CHECKSUM_CRC32_SIGNATURE_LEN = 4;
public static final int BINLOG_CHECKSUM_ALG_DESC_LEN = 1;
/**
* defined statically while there is just one alg implemented
*/
public static final int BINLOG_CHECKSUM_LEN = CHECKSUM_CRC32_SIGNATURE_LEN;
public static final int BINLOG_CHECKSUM_LEN = CHECKSUM_CRC32_SIGNATURE_LEN;
/* MySQL or old MariaDB slave with no announced capability. */
public static final int MARIA_SLAVE_CAPABILITY_UNKNOWN = 0;
@@ -258,124 +237,122 @@ public abstract class LogEvent
/* Our capability. */
public static final int MARIA_SLAVE_CAPABILITY_MINE = MARIA_SLAVE_CAPABILITY_GTID;
/**
For an event, 'e', carrying a type code, that a slave,
's', does not recognize, 's' will check 'e' for
LOG_EVENT_IGNORABLE_F, and if the flag is set, then 'e'
is ignored. Otherwise, 's' acknowledges that it has
found an unknown event in the relay log.
*/
public static final int LOG_EVENT_IGNORABLE_F = 0x80;
/** enum_field_types */
public static final int MYSQL_TYPE_DECIMAL = 0;
public static final int MYSQL_TYPE_TINY = 1;
public static final int MYSQL_TYPE_SHORT = 2;
public static final int MYSQL_TYPE_LONG = 3;
public static final int MYSQL_TYPE_FLOAT = 4;
public static final int MYSQL_TYPE_DOUBLE = 5;
public static final int MYSQL_TYPE_NULL = 6;
public static final int MYSQL_TYPE_TIMESTAMP = 7;
public static final int MYSQL_TYPE_LONGLONG = 8;
public static final int MYSQL_TYPE_INT24 = 9;
public static final int MYSQL_TYPE_DATE = 10;
public static final int MYSQL_TYPE_TIME = 11;
public static final int MYSQL_TYPE_DATETIME = 12;
public static final int MYSQL_TYPE_YEAR = 13;
public static final int MYSQL_TYPE_NEWDATE = 14;
public static final int MYSQL_TYPE_VARCHAR = 15;
public static final int MYSQL_TYPE_BIT = 16;
public static final int MYSQL_TYPE_TIMESTAMP2 = 17;
public static final int MYSQL_TYPE_DATETIME2 = 18;
public static final int MYSQL_TYPE_TIME2 = 19;
public static final int MYSQL_TYPE_NEWDECIMAL = 246;
public static final int MYSQL_TYPE_ENUM = 247;
public static final int MYSQL_TYPE_SET = 248;
public static final int MYSQL_TYPE_TINY_BLOB = 249;
public static final int MYSQL_TYPE_MEDIUM_BLOB = 250;
public static final int MYSQL_TYPE_LONG_BLOB = 251;
public static final int MYSQL_TYPE_BLOB = 252;
public static final int MYSQL_TYPE_VAR_STRING = 253;
public static final int MYSQL_TYPE_STRING = 254;
public static final int MYSQL_TYPE_GEOMETRY = 255;
public static String getTypeName(final int type)
{
switch (type)
{
case START_EVENT_V3:
return "Start_v3";
case STOP_EVENT:
return "Stop";
case QUERY_EVENT:
return "Query";
case ROTATE_EVENT:
return "Rotate";
case INTVAR_EVENT:
return "Intvar";
case LOAD_EVENT:
return "Load";
case NEW_LOAD_EVENT:
return "New_load";
case SLAVE_EVENT:
return "Slave";
case CREATE_FILE_EVENT:
return "Create_file";
case APPEND_BLOCK_EVENT:
return "Append_block";
case DELETE_FILE_EVENT:
return "Delete_file";
case EXEC_LOAD_EVENT:
return "Exec_load";
case RAND_EVENT:
return "RAND";
case XID_EVENT:
return "Xid";
case USER_VAR_EVENT:
return "User var";
case FORMAT_DESCRIPTION_EVENT:
return "Format_desc";
case TABLE_MAP_EVENT:
return "Table_map";
case PRE_GA_WRITE_ROWS_EVENT:
return "Write_rows_event_old";
case PRE_GA_UPDATE_ROWS_EVENT:
return "Update_rows_event_old";
case PRE_GA_DELETE_ROWS_EVENT:
return "Delete_rows_event_old";
case WRITE_ROWS_EVENT_V1:
return "Write_rows_v1";
case UPDATE_ROWS_EVENT_V1:
return "Update_rows_v1";
case DELETE_ROWS_EVENT_V1:
return "Delete_rows_v1";
case BEGIN_LOAD_QUERY_EVENT:
return "Begin_load_query";
case EXECUTE_LOAD_QUERY_EVENT:
return "Execute_load_query";
case INCIDENT_EVENT:
return "Incident";
case HEARTBEAT_LOG_EVENT:
return "Heartbeat";
case IGNORABLE_LOG_EVENT:
return "Ignorable";
case ROWS_QUERY_LOG_EVENT:
return "Rows_query";
case WRITE_ROWS_EVENT:
return "Write_rows";
case UPDATE_ROWS_EVENT:
return "Update_rows";
case DELETE_ROWS_EVENT:
return "Delete_rows";
case GTID_LOG_EVENT:
return "Gtid";
case ANONYMOUS_GTID_LOG_EVENT:
return "Anonymous_Gtid";
case PREVIOUS_GTIDS_LOG_EVENT:
return "Previous_gtids";
default:
return "Unknown"; /* impossible */
/**
* For an event, 'e', carrying a type code, that a slave, 's', does not
* recognize, 's' will check 'e' for LOG_EVENT_IGNORABLE_F, and if the flag
* is set, then 'e' is ignored. Otherwise, 's' acknowledges that it has
* found an unknown event in the relay log.
*/
public static final int LOG_EVENT_IGNORABLE_F = 0x80;
/** enum_field_types */
public static final int MYSQL_TYPE_DECIMAL = 0;
public static final int MYSQL_TYPE_TINY = 1;
public static final int MYSQL_TYPE_SHORT = 2;
public static final int MYSQL_TYPE_LONG = 3;
public static final int MYSQL_TYPE_FLOAT = 4;
public static final int MYSQL_TYPE_DOUBLE = 5;
public static final int MYSQL_TYPE_NULL = 6;
public static final int MYSQL_TYPE_TIMESTAMP = 7;
public static final int MYSQL_TYPE_LONGLONG = 8;
public static final int MYSQL_TYPE_INT24 = 9;
public static final int MYSQL_TYPE_DATE = 10;
public static final int MYSQL_TYPE_TIME = 11;
public static final int MYSQL_TYPE_DATETIME = 12;
public static final int MYSQL_TYPE_YEAR = 13;
public static final int MYSQL_TYPE_NEWDATE = 14;
public static final int MYSQL_TYPE_VARCHAR = 15;
public static final int MYSQL_TYPE_BIT = 16;
public static final int MYSQL_TYPE_TIMESTAMP2 = 17;
public static final int MYSQL_TYPE_DATETIME2 = 18;
public static final int MYSQL_TYPE_TIME2 = 19;
public static final int MYSQL_TYPE_JSON = 245;
public static final int MYSQL_TYPE_NEWDECIMAL = 246;
public static final int MYSQL_TYPE_ENUM = 247;
public static final int MYSQL_TYPE_SET = 248;
public static final int MYSQL_TYPE_TINY_BLOB = 249;
public static final int MYSQL_TYPE_MEDIUM_BLOB = 250;
public static final int MYSQL_TYPE_LONG_BLOB = 251;
public static final int MYSQL_TYPE_BLOB = 252;
public static final int MYSQL_TYPE_VAR_STRING = 253;
public static final int MYSQL_TYPE_STRING = 254;
public static final int MYSQL_TYPE_GEOMETRY = 255;
public static String getTypeName(final int type) {
switch (type) {
case START_EVENT_V3:
return "Start_v3";
case STOP_EVENT:
return "Stop";
case QUERY_EVENT:
return "Query";
case ROTATE_EVENT:
return "Rotate";
case INTVAR_EVENT:
return "Intvar";
case LOAD_EVENT:
return "Load";
case NEW_LOAD_EVENT:
return "New_load";
case SLAVE_EVENT:
return "Slave";
case CREATE_FILE_EVENT:
return "Create_file";
case APPEND_BLOCK_EVENT:
return "Append_block";
case DELETE_FILE_EVENT:
return "Delete_file";
case EXEC_LOAD_EVENT:
return "Exec_load";
case RAND_EVENT:
return "RAND";
case XID_EVENT:
return "Xid";
case USER_VAR_EVENT:
return "User var";
case FORMAT_DESCRIPTION_EVENT:
return "Format_desc";
case TABLE_MAP_EVENT:
return "Table_map";
case PRE_GA_WRITE_ROWS_EVENT:
return "Write_rows_event_old";
case PRE_GA_UPDATE_ROWS_EVENT:
return "Update_rows_event_old";
case PRE_GA_DELETE_ROWS_EVENT:
return "Delete_rows_event_old";
case WRITE_ROWS_EVENT_V1:
return "Write_rows_v1";
case UPDATE_ROWS_EVENT_V1:
return "Update_rows_v1";
case DELETE_ROWS_EVENT_V1:
return "Delete_rows_v1";
case BEGIN_LOAD_QUERY_EVENT:
return "Begin_load_query";
case EXECUTE_LOAD_QUERY_EVENT:
return "Execute_load_query";
case INCIDENT_EVENT:
return "Incident";
case HEARTBEAT_LOG_EVENT:
return "Heartbeat";
case IGNORABLE_LOG_EVENT:
return "Ignorable";
case ROWS_QUERY_LOG_EVENT:
return "Rows_query";
case WRITE_ROWS_EVENT:
return "Write_rows";
case UPDATE_ROWS_EVENT:
return "Update_rows";
case DELETE_ROWS_EVENT:
return "Delete_rows";
case GTID_LOG_EVENT:
return "Gtid";
case ANONYMOUS_GTID_LOG_EVENT:
return "Anonymous_Gtid";
case PREVIOUS_GTIDS_LOG_EVENT:
return "Previous_gtids";
default:
return "Unknown"; /* impossible */
}
}
@@ -383,16 +360,14 @@ public abstract class LogEvent
protected final LogHeader header;
protected LogEvent(LogHeader header)
{
protected LogEvent(LogHeader header){
this.header = header;
}
/**
* Return event header.
*/
public final LogHeader getHeader()
{
public final LogHeader getHeader() {
return header;
}
@@ -400,16 +375,14 @@ public abstract class LogEvent
* The total size of this event, in bytes. In other words, this is the sum
* of the sizes of Common-Header, Post-Header, and Body.
*/
public final int getEventLen()
{
public final int getEventLen() {
return header.getEventLen();
}
/**
* Server ID of the server that created the event.
*/
public final long getServerId()
{
public final long getServerId() {
return header.getServerId();
}
@@ -420,18 +393,15 @@ public abstract class LogEvent
* file. In a relay log, this is the position of the next event in the
* master's binlog.
*/
public final long getLogPos()
{
public final long getLogPos() {
return header.getLogPos();
}
/**
* The time when the query started, in seconds since 1970.
*/
public final long getWhen()
{
public final long getWhen() {
return header.getWhen();
}
}
@@ -29,8 +29,8 @@ import java.util.Arrays;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public abstract class LogFetcher extends LogBuffer implements Closeable
{
public abstract class LogFetcher extends LogBuffer implements Closeable {
/** Default initial capacity. */
public static final int DEFAULT_INITIAL_CAPACITY = 8192;
@@ -42,18 +42,15 @@ public abstract class LogFetcher extends LogBuffer implements Closeable
protected final float factor;
public LogFetcher()
{
public LogFetcher(){
this(DEFAULT_INITIAL_CAPACITY, DEFAULT_GROWTH_FACTOR);
}
public LogFetcher(final int initialCapacity)
{
public LogFetcher(final int initialCapacity){
this(initialCapacity, DEFAULT_GROWTH_FACTOR);
}
public LogFetcher(final int initialCapacity, final float growthFactor)
{
public LogFetcher(final int initialCapacity, final float growthFactor){
this.buffer = new byte[initialCapacity];
this.factor = growthFactor;
}
@@ -65,15 +62,12 @@ public abstract class LogFetcher extends LogBuffer implements Closeable
*
* @param minCapacity the desired minimum capacity
*/
protected final void ensureCapacity(final int minCapacity)
{
protected final void ensureCapacity(final int minCapacity) {
final int oldCapacity = buffer.length;
if (minCapacity > oldCapacity)
{
if (minCapacity > oldCapacity) {
int newCapacity = (int) (oldCapacity * factor);
if (newCapacity < minCapacity)
newCapacity = minCapacity;
if (newCapacity < minCapacity) newCapacity = minCapacity;
buffer = Arrays.copyOf(buffer, newCapacity);
}
@@ -6,8 +6,8 @@ package com.taobao.tddl.dbsync.binlog;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public class LogPosition implements Cloneable, Comparable<LogPosition>
{
public class LogPosition implements Cloneable, Comparable<LogPosition> {
/* binlog file's name */
protected String fileName;
@@ -19,8 +19,7 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
*
* @param fileName file name for binlog files: mysql-bin.000001
*/
public LogPosition(String fileName)
{
public LogPosition(String fileName){
this.fileName = fileName;
this.position = 0L;
}
@@ -30,8 +29,7 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
*
* @param fileName file name for binlog files: mysql-bin.000001
*/
public LogPosition(String fileName, final long position)
{
public LogPosition(String fileName, final long position){
this.fileName = fileName;
this.position = position;
}
@@ -39,31 +37,24 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
/**
* Binlog position copy init.
*/
public LogPosition(LogPosition source)
{
public LogPosition(LogPosition source){
this.fileName = source.fileName;
this.position = source.position;
}
public final String getFileName()
{
public final String getFileName() {
return fileName;
}
public final long getPosition()
{
public final long getPosition() {
return position;
}
/* Clone binlog position without CloneNotSupportedException */
public LogPosition clone()
{
try
{
public LogPosition clone() {
try {
return (LogPosition) super.clone();
}
catch (CloneNotSupportedException e)
{
} catch (CloneNotSupportedException e) {
// Never happend
return null;
}
@@ -72,12 +63,10 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
/**
* Compares with the specified fileName and position.
*/
public final int compareTo(String fileName, final long position)
{
public final int compareTo(String fileName, final long position) {
final int val = this.fileName.compareTo(fileName);
if (val == 0)
{
if (val == 0) {
return (int) (this.position - position);
}
return val;
@@ -88,12 +77,10 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
*
* @see java.lang.Comparable#compareTo(java.lang.Object)
*/
public int compareTo(LogPosition o)
{
public int compareTo(LogPosition o) {
final int val = fileName.compareTo(o.fileName);
if (val == 0)
{
if (val == 0) {
return (int) (position - o.position);
}
return val;
@@ -104,13 +91,10 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
*
* @see java.lang.Object#equals(java.lang.Object)
*/
public boolean equals(Object obj)
{
if (obj instanceof LogPosition)
{
public boolean equals(Object obj) {
if (obj instanceof LogPosition) {
LogPosition pos = ((LogPosition) obj);
return fileName.equals(pos.fileName)
&& (this.position == pos.position);
return fileName.equals(pos.fileName) && (this.position == pos.position);
}
return false;
}
@@ -120,8 +104,7 @@ public class LogPosition implements Cloneable, Comparable<LogPosition>
*
* @see java.lang.Object#toString()
*/
public String toString()
{
public String toString() {
return fileName + ':' + position;
}
}
@@ -9,8 +9,8 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public class AppendBlockLogEvent extends LogEvent
{
public class AppendBlockLogEvent extends LogEvent {
private final LogBuffer blockBuf;
private final int blockLen;
@@ -19,9 +19,7 @@ public class AppendBlockLogEvent extends LogEvent
/* AB = "Append Block" */
public static final int AB_FILE_ID_OFFSET = 0;
public AppendBlockLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public AppendBlockLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
@@ -36,18 +34,15 @@ public class AppendBlockLogEvent extends LogEvent
blockBuf = buffer.duplicate(blockLen);
}
public final long getFileId()
{
public final long getFileId() {
return fileId;
}
public final LogBuffer getBuffer()
{
public final LogBuffer getBuffer() {
return blockBuf;
}
public final byte[] getData()
{
public final byte[] getData() {
return blockBuf.getData();
}
}
@@ -10,11 +10,9 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class BeginLoadQueryLogEvent extends AppendBlockLogEvent
{
public BeginLoadQueryLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public final class BeginLoadQueryLogEvent extends AppendBlockLogEvent {
public BeginLoadQueryLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header, buffer, descriptionEvent);
}
}
@@ -8,8 +8,8 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class CreateFileLogEvent extends LoadLogEvent
{
public final class CreateFileLogEvent extends LoadLogEvent {
protected LogBuffer blockBuf;
protected int blockLen;
protected long fileId;
@@ -20,9 +20,7 @@ public final class CreateFileLogEvent extends LoadLogEvent
public static final int CF_FILE_ID_OFFSET = 0;
public static final int CF_DATA_OFFSET = FormatDescriptionLogEvent.CREATE_FILE_HEADER_LEN;
public CreateFileLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public CreateFileLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header, buffer, descriptionEvent);
final int headerLen = descriptionEvent.commonHeaderLen;
@@ -30,44 +28,37 @@ public final class CreateFileLogEvent extends LoadLogEvent
final int createFileHeaderLen = descriptionEvent.postHeaderLen[CREATE_FILE_EVENT - 1];
copyLogEvent(buffer,
((header.type == LOAD_EVENT) ? (loadHeaderLen + headerLen)
: (headerLen + loadHeaderLen + createFileHeaderLen)),
descriptionEvent);
((header.type == LOAD_EVENT) ? (loadHeaderLen + headerLen) : (headerLen + loadHeaderLen + createFileHeaderLen)),
descriptionEvent);
if (descriptionEvent.binlogVersion != 1)
{
fileId = buffer.getUint32(headerLen + loadHeaderLen
+ CF_FILE_ID_OFFSET);
if (descriptionEvent.binlogVersion != 1) {
fileId = buffer.getUint32(headerLen + loadHeaderLen + CF_FILE_ID_OFFSET);
/*
Note that it's ok to use get_data_size() below, because it is computed
with values we have already read from this event (because we called
copy_log_event()); we are not using slave's format info to decode
master's format, we are really using master's format info.
Anyway, both formats should be identical (except the common_header_len)
as these Load events are not changed between 4.0 and 5.0 (as logging of
LOAD DATA INFILE does not use Load_log_event in 5.0).
*/
* Note that it's ok to use get_data_size() below, because it is
* computed with values we have already read from this event
* (because we called copy_log_event()); we are not using slave's
* format info to decode master's format, we are really using
* master's format info. Anyway, both formats should be identical
* (except the common_header_len) as these Load events are not
* changed between 4.0 and 5.0 (as logging of LOAD DATA INFILE does
* not use Load_log_event in 5.0).
*/
blockLen = buffer.limit() - buffer.position();
blockBuf = buffer.duplicate(blockLen);
}
else
{
} else {
initedFromOld = true;
}
}
public final long getFileId()
{
public final long getFileId() {
return fileId;
}
public final LogBuffer getBuffer()
{
public final LogBuffer getBuffer() {
return blockBuf;
}
public final byte[] getData()
{
public final byte[] getData() {
return blockBuf.getData();
}
}
@@ -9,25 +9,22 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class DeleteFileLogEvent extends LogEvent
{
public final class DeleteFileLogEvent extends LogEvent {
private final long fileId;
/* DF = "Delete File" */
public static final int DF_FILE_ID_OFFSET = 0;
public DeleteFileLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public DeleteFileLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
buffer.position(commonHeaderLen + DF_FILE_ID_OFFSET);
fileId = buffer.getUint32(); // DF_FILE_ID_OFFSET
fileId = buffer.getUint32(); // DF_FILE_ID_OFFSET
}
public final long getFileId()
{
public final long getFileId() {
return fileId;
}
}
@@ -9,11 +9,9 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class DeleteRowsLogEvent extends RowsLogEvent
{
public DeleteRowsLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public final class DeleteRowsLogEvent extends RowsLogEvent {
public DeleteRowsLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header, buffer, descriptionEvent);
}
}
@@ -9,25 +9,22 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class ExecuteLoadLogEvent extends LogEvent
{
public final class ExecuteLoadLogEvent extends LogEvent {
private final long fileId;
/* EL = "Execute Load" */
public static final int EL_FILE_ID_OFFSET = 0;
public ExecuteLoadLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public ExecuteLoadLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
buffer.position(commonHeaderLen + EL_FILE_ID_OFFSET);
fileId = buffer.getUint32(); // EL_FILE_ID_OFFSET
fileId = buffer.getUint32(); // EL_FILE_ID_OFFSET
}
public final long getFileId()
{
public final long getFileId() {
return fileId;
}
}
@@ -8,7 +8,6 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* Event responsible for LOAD DATA execution, it similar to Query_log_event but
* before executing the query it substitutes original filename in LOAD DATA
* query with name of temporary file.
*
* <ul>
* <li>4 bytes. The ID of the file to load.</li>
* <li>4 bytes. The start position within the statement for filename
@@ -22,8 +21,8 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class ExecuteLoadQueryLogEvent extends QueryLogEvent
{
public final class ExecuteLoadQueryLogEvent extends QueryLogEvent {
/** file_id of temporary file */
private long fileId;
@@ -53,9 +52,8 @@ public final class ExecuteLoadQueryLogEvent extends QueryLogEvent
public static final int ELQ_FN_POS_END_OFFSET = ELQ_FILE_ID_OFFSET + 8;
public static final int ELQ_DUP_HANDLING_OFFSET = ELQ_FILE_ID_OFFSET + 12;
public ExecuteLoadQueryLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent) throws IOException
{
public ExecuteLoadQueryLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent)
throws IOException{
super(header, buffer, descriptionEvent);
buffer.position(descriptionEvent.commonHeaderLen + ELQ_FILE_ID_OFFSET);
@@ -66,36 +64,27 @@ public final class ExecuteLoadQueryLogEvent extends QueryLogEvent
dupHandling = buffer.getInt8(); // ELQ_DUP_HANDLING_OFFSET
final int len = query.length();
if (fnPosStart > len || fnPosEnd > len
|| dupHandling > LOAD_DUP_REPLACE)
{
throw new IOException(String.format(
"Invalid ExecuteLoadQueryLogEvent: fn_pos_start=%d, "
+ "fn_pos_end=%d, dup_handling=%d", fnPosStart,
fnPosEnd, dupHandling));
if (fnPosStart > len || fnPosEnd > len || dupHandling > LOAD_DUP_REPLACE) {
throw new IOException(String.format("Invalid ExecuteLoadQueryLogEvent: fn_pos_start=%d, "
+ "fn_pos_end=%d, dup_handling=%d", fnPosStart, fnPosEnd, dupHandling));
}
}
public final int getFilenamePosStart()
{
public final int getFilenamePosStart() {
return fnPosStart;
}
public final int getFilenamePosEnd()
{
public final int getFilenamePosEnd() {
return fnPosEnd;
}
public final String getFilename()
{
if (query != null)
return query.substring(fnPosStart, fnPosEnd).trim();
public final String getFilename() {
if (query != null) return query.substring(fnPosStart, fnPosEnd).trim();
return null;
}
public final long getFileId()
{
public final long getFileId() {
return fileId;
}
}
@@ -5,8 +5,8 @@ import java.io.IOException;
import com.taobao.tddl.dbsync.binlog.LogBuffer;
/**
* For binlog version 4. This event is saved by threads which read it, as they need it for future use (to decode the
* ordinary events).
* For binlog version 4. This event is saved by threads which read it, as they
* need it for future use (to decode the ordinary events).
*
* @see mysql-5.1.60/sql/log_event.cc - Format_description_log_event
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
@@ -15,23 +15,24 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
public final class FormatDescriptionLogEvent extends StartLogEventV3 {
/**
* The number of types we handle in Format_description_log_event (UNKNOWN_EVENT is not to be handled, it does not
* exist in binlogs, it does not have a format).
* The number of types we handle in Format_description_log_event
* (UNKNOWN_EVENT is not to be handled, it does not exist in binlogs, it
* does not have a format).
*/
public static final int LOG_EVENT_TYPES = (ENUM_END_EVENT - 1);
public static final int LOG_EVENT_TYPES = (ENUM_END_EVENT - 1);
public static final int ST_COMMON_HEADER_LEN_OFFSET = (ST_SERVER_VER_OFFSET + ST_SERVER_VER_LEN + 4);
public static final int ST_COMMON_HEADER_LEN_OFFSET = (ST_SERVER_VER_OFFSET + ST_SERVER_VER_LEN + 4);
public static final int OLD_HEADER_LEN = 13;
public static final int LOG_EVENT_HEADER_LEN = 19;
public static final int LOG_EVENT_MINIMAL_HEADER_LEN = 19;
public static final int OLD_HEADER_LEN = 13;
public static final int LOG_EVENT_HEADER_LEN = 19;
public static final int LOG_EVENT_MINIMAL_HEADER_LEN = 19;
/* event-specific post-header sizes */
public static final int STOP_HEADER_LEN = 0;
public static final int LOAD_HEADER_LEN = (4 + 4 + 4 + 1 + 1 + 4);
public static final int SLAVE_HEADER_LEN = 0;
public static final int START_V3_HEADER_LEN = (2 + ST_SERVER_VER_LEN + 4);
public static final int ROTATE_HEADER_LEN = 8; // this
public static final int STOP_HEADER_LEN = 0;
public static final int LOAD_HEADER_LEN = (4 + 4 + 4 + 1 + 1 + 4);
public static final int SLAVE_HEADER_LEN = 0;
public static final int START_V3_HEADER_LEN = (2 + ST_SERVER_VER_LEN + 4);
public static final int ROTATE_HEADER_LEN = 8; // this
// is
// FROZEN
// (the
@@ -39,47 +40,48 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
// post-header
// is
// frozen)
public static final int INTVAR_HEADER_LEN = 0;
public static final int CREATE_FILE_HEADER_LEN = 4;
public static final int APPEND_BLOCK_HEADER_LEN = 4;
public static final int EXEC_LOAD_HEADER_LEN = 4;
public static final int DELETE_FILE_HEADER_LEN = 4;
public static final int NEW_LOAD_HEADER_LEN = LOAD_HEADER_LEN;
public static final int RAND_HEADER_LEN = 0;
public static final int USER_VAR_HEADER_LEN = 0;
public static final int FORMAT_DESCRIPTION_HEADER_LEN = (START_V3_HEADER_LEN + 1 + LOG_EVENT_TYPES);
public static final int XID_HEADER_LEN = 0;
public static final int BEGIN_LOAD_QUERY_HEADER_LEN = APPEND_BLOCK_HEADER_LEN;
public static final int ROWS_HEADER_LEN_V1 = 8;
public static final int TABLE_MAP_HEADER_LEN = 8;
public static final int EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN = (4 + 4 + 4 + 1);
public static final int EXECUTE_LOAD_QUERY_HEADER_LEN = (QUERY_HEADER_LEN + EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN);
public static final int INCIDENT_HEADER_LEN = 2;
public static final int HEARTBEAT_HEADER_LEN = 0;
public static final int IGNORABLE_HEADER_LEN = 0;
public static final int ROWS_HEADER_LEN_V2 = 10;
public static final int ANNOTATE_ROWS_HEADER_LEN = 0;
public static final int BINLOG_CHECKPOINT_HEADER_LEN = 4;
public static final int GTID_HEADER_LEN = 19;
public static final int GTID_LIST_HEADER_LEN = 4;
public static final int INTVAR_HEADER_LEN = 0;
public static final int CREATE_FILE_HEADER_LEN = 4;
public static final int APPEND_BLOCK_HEADER_LEN = 4;
public static final int EXEC_LOAD_HEADER_LEN = 4;
public static final int DELETE_FILE_HEADER_LEN = 4;
public static final int NEW_LOAD_HEADER_LEN = LOAD_HEADER_LEN;
public static final int RAND_HEADER_LEN = 0;
public static final int USER_VAR_HEADER_LEN = 0;
public static final int FORMAT_DESCRIPTION_HEADER_LEN = (START_V3_HEADER_LEN + 1 + LOG_EVENT_TYPES);
public static final int XID_HEADER_LEN = 0;
public static final int BEGIN_LOAD_QUERY_HEADER_LEN = APPEND_BLOCK_HEADER_LEN;
public static final int ROWS_HEADER_LEN_V1 = 8;
public static final int TABLE_MAP_HEADER_LEN = 8;
public static final int EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN = (4 + 4 + 4 + 1);
public static final int EXECUTE_LOAD_QUERY_HEADER_LEN = (QUERY_HEADER_LEN + EXECUTE_LOAD_QUERY_EXTRA_HEADER_LEN);
public static final int INCIDENT_HEADER_LEN = 2;
public static final int HEARTBEAT_HEADER_LEN = 0;
public static final int IGNORABLE_HEADER_LEN = 0;
public static final int ROWS_HEADER_LEN_V2 = 10;
public static final int ANNOTATE_ROWS_HEADER_LEN = 0;
public static final int BINLOG_CHECKPOINT_HEADER_LEN = 4;
public static final int GTID_HEADER_LEN = 19;
public static final int GTID_LIST_HEADER_LEN = 4;
public static final int POST_HEADER_LENGTH = 11;
public static final int POST_HEADER_LENGTH = 11;
public static final int BINLOG_CHECKSUM_ALG_DESC_LEN = 1;
public static final int[] checksumVersionSplit = { 5, 6, 1 };
public static final long checksumVersionProduct = (checksumVersionSplit[0] * 256 + checksumVersionSplit[1])
* 256 + checksumVersionSplit[2];
/**
* The size of the fixed header which _all_ events have (for binlogs written by this version, this is equal to
* LOG_EVENT_HEADER_LEN), except FORMAT_DESCRIPTION_EVENT and ROTATE_EVENT (those have a header of size
* The size of the fixed header which _all_ events have (for binlogs written
* by this version, this is equal to LOG_EVENT_HEADER_LEN), except
* FORMAT_DESCRIPTION_EVENT and ROTATE_EVENT (those have a header of size
* LOG_EVENT_MINIMAL_HEADER_LEN).
*/
protected final int commonHeaderLen;
protected int numberOfEventTypes;
protected final int commonHeaderLen;
protected int numberOfEventTypes;
/** The list of post-headers' lengthes */
protected final short[] postHeaderLen;
protected int[] serverVersionSplit = new int[3];
protected final short[] postHeaderLen;
protected int[] serverVersionSplit = new int[3];
public FormatDescriptionLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent)
throws IOException{
@@ -105,12 +107,15 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
calcServerVersionSplit();
long calc = getVersionProduct();
if (calc >= checksumVersionProduct) {
/* the last bytes are the checksum alg desc and value (or value's room) */
/*
* the last bytes are the checksum alg desc and value (or value's
* room)
*/
numberOfEventTypes -= BINLOG_CHECKSUM_ALG_DESC_LEN;
}
if (logger.isInfoEnabled()) logger.info("common_header_len= " + commonHeaderLen + ", number_of_event_types= "
+ numberOfEventTypes);
+ numberOfEventTypes);
}
/** MySQL 5.0 format descriptions. */
@@ -135,10 +140,15 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
throw new IOException("Unknown binlog version: " + binlogVersion);
}
}
public FormatDescriptionLogEvent(final int binlogVersion,int binlogChecksum){
this(binlogVersion);
this.header.checksumAlg = binlogChecksum;
}
public FormatDescriptionLogEvent(final int binlogVersion){
this.binlogVersion = binlogVersion;
postHeaderLen = new short[ENUM_END_EVENT];
/* identify binlog format */
switch (binlogVersion) {
@@ -147,7 +157,10 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
commonHeaderLen = LOG_EVENT_HEADER_LEN;
numberOfEventTypes = LOG_EVENT_TYPES;
/* Note: all event types must explicitly fill in their lengths here. */
/*
* Note: all event types must explicitly fill in their lengths
* here.
*/
postHeaderLen[START_EVENT_V3 - 1] = START_V3_HEADER_LEN;
postHeaderLen[QUERY_EVENT - 1] = QUERY_HEADER_LEN;
postHeaderLen[STOP_EVENT - 1] = STOP_HEADER_LEN;
@@ -171,11 +184,14 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
postHeaderLen[UPDATE_ROWS_EVENT_V1 - 1] = ROWS_HEADER_LEN_V1;
postHeaderLen[DELETE_ROWS_EVENT_V1 - 1] = ROWS_HEADER_LEN_V1;
/*
* We here have the possibility to simulate a master of before we changed the table map id to be stored
* in 6 bytes: when it was stored in 4 bytes (=> post_header_len was 6). This is used to test backward
* compatibility. This code can be removed after a few months (today is Dec 21st 2005), when we know
* that the 4-byte masters are not deployed anymore (check with Tomas Ulin first!), and the accompanying
* test (rpl_row_4_bytes) too.
* We here have the possibility to simulate a master of before
* we changed the table map id to be stored in 6 bytes: when it
* was stored in 4 bytes (=> post_header_len was 6). This is
* used to test backward compatibility. This code can be removed
* after a few months (today is Dec 21st 2005), when we know
* that the 4-byte masters are not deployed anymore (check with
* Tomas Ulin first!), and the accompanying test
* (rpl_row_4_bytes) too.
*/
postHeaderLen[HEARTBEAT_LOG_EVENT - 1] = 0;
postHeaderLen[IGNORABLE_LOG_EVENT - 1] = IGNORABLE_HEADER_LEN;
@@ -195,17 +211,19 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
case 3: /* 4.0.x x>=2 */
/*
* We build an artificial (i.e. not sent by the master) event, which describes what those old master
* versions send.
* We build an artificial (i.e. not sent by the master) event,
* which describes what those old master versions send.
*/
serverVersion = "4.0";
commonHeaderLen = LOG_EVENT_MINIMAL_HEADER_LEN;
/*
* The first new event in binlog version 4 is Format_desc. So any event type after that does not exist
* in older versions. We use the events known by version 3, even if version 1 had only a subset of them
* (this is not a problem: it uses a few bytes for nothing but unifies code; it does not make the slave
* detect less corruptions).
* The first new event in binlog version 4 is Format_desc. So
* any event type after that does not exist in older versions.
* We use the events known by version 3, even if version 1 had
* only a subset of them (this is not a problem: it uses a few
* bytes for nothing but unifies code; it does not make the
* slave detect less corruptions).
*/
numberOfEventTypes = FORMAT_DESCRIPTION_EVENT - 1;
@@ -222,17 +240,19 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
case 1: /* 3.23 */
/*
* We build an artificial (i.e. not sent by the master) event, which describes what those old master
* versions send.
* We build an artificial (i.e. not sent by the master) event,
* which describes what those old master versions send.
*/
serverVersion = "3.23";
commonHeaderLen = OLD_HEADER_LEN;
/*
* The first new event in binlog version 4 is Format_desc. So any event type after that does not exist
* in older versions. We use the events known by version 3, even if version 1 had only a subset of them
* (this is not a problem: it uses a few bytes for nothing but unifies code; it does not make the slave
* detect less corruptions).
* The first new event in binlog version 4 is Format_desc. So
* any event type after that does not exist in older versions.
* We use the events known by version 3, even if version 1 had
* only a subset of them (this is not a problem: it uses a few
* bytes for nothing but unifies code; it does not make the
* slave detect less corruptions).
*/
numberOfEventTypes = FORMAT_DESCRIPTION_EVENT - 1;
@@ -251,7 +271,7 @@ public final class FormatDescriptionLogEvent extends StartLogEventV3 {
commonHeaderLen = 0;
}
}
public void calcServerVersionSplit() {
doServerVersionSplit(serverVersion, serverVersionSplit);
}
@@ -6,19 +6,19 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* <pre>
* Base class for ignorable log events. Events deriving from
this class can be safely ignored by slaves that cannot
recognize them. Newer slaves, will be able to read and
handle them. This has been designed to be an open-ended
architecture, so adding new derived events shall not harm
the old slaves that support ignorable log event mechanism
(they will just ignore unrecognized ignorable events).
@note The only thing that makes an event ignorable is that it has
the LOG_EVENT_IGNORABLE_F flag set. It is not strictly necessary
that ignorable event types derive from Ignorable_log_event; they may
just as well derive from Log_event and pass LOG_EVENT_IGNORABLE_F as
argument to the Log_event constructor.
</pre>
* this class can be safely ignored by slaves that cannot
* recognize them. Newer slaves, will be able to read and
* handle them. This has been designed to be an open-ended
* architecture, so adding new derived events shall not harm
* the old slaves that support ignorable log event mechanism
* (they will just ignore unrecognized ignorable events).
*
* @note The only thing that makes an event ignorable is that it has
* the LOG_EVENT_IGNORABLE_F flag set. It is not strictly necessary
* that ignorable event types derive from Ignorable_log_event; they may
* just as well derive from Log_event and pass LOG_EVENT_IGNORABLE_F as
* argument to the Log_event constructor.
* </pre>
*
* @author jianghang 2013-4-8 上午12:36:29
* @version 1.0.3
@@ -5,12 +5,9 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* Class representing an incident, an occurance out of the ordinary, that
* happened on the master.
*
* The event is used to inform the slave that something out of the ordinary
* happened on the master that might cause the database to be in an inconsistent
* state.
*
* happened on the master. The event is used to inform the slave that something
* out of the ordinary happened on the master that might cause the database to
* be in an inconsistent state.
* <table id="IncidentFormat">
* <caption>Incident event format</caption>
* <tr>
@@ -38,8 +35,8 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class IncidentLogEvent extends LogEvent
{
public final class IncidentLogEvent extends LogEvent {
public static final int INCIDENT_NONE = 0;
/** There are possibly lost events in the replication stream */
@@ -51,9 +48,7 @@ public final class IncidentLogEvent extends LogEvent
private final int incident;
private final String message;
public IncidentLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public IncidentLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
@@ -61,10 +56,9 @@ public final class IncidentLogEvent extends LogEvent
buffer.position(commonHeaderLen);
final int incidentNumber = buffer.getUint16();
if (incidentNumber >= INCIDENT_COUNT || incidentNumber <= INCIDENT_NONE)
{
if (incidentNumber >= INCIDENT_COUNT || incidentNumber <= INCIDENT_NONE) {
// If the incident is not recognized, this binlog event is
// invalid. If we set incident_number to INCIDENT_NONE, the
// invalid. If we set incident_number to INCIDENT_NONE, the
// invalidity will be detected by is_valid().
incident = INCIDENT_NONE;
message = null;
@@ -76,13 +70,11 @@ public final class IncidentLogEvent extends LogEvent
message = buffer.getString();
}
public final int getIncident()
{
public final int getIncident() {
return incident;
}
public final String getMessage()
{
public final String getMessage() {
return message;
}
}
@@ -6,54 +6,43 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* An Intvar_log_event will be created just before a Query_log_event, if the
* query uses one of the variables LAST_INSERT_ID or INSERT_ID. Each
* Intvar_log_event holds the value of one of these variables.
*
* Binary Format
*
* The Post-Header for this event type is empty. The Body has two components:
*
* Intvar_log_event holds the value of one of these variables. Binary Format The
* Post-Header for this event type is empty. The Body has two components:
* <table>
* <caption>Body for Intvar_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>type</td>
* <td>1 byte enumeration</td>
* <td>One byte identifying the type of variable stored. Currently, two
* identifiers are supported: LAST_INSERT_ID_EVENT==1 and INSERT_ID_EVENT==2.</td>
* </tr>
*
* <tr>
* <td>value</td>
* <td>8 byte unsigned integer</td>
* <td>The value of the variable.</td>
* </tr>
*
* </table>
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class IntvarLogEvent extends LogEvent
{
public final class IntvarLogEvent extends LogEvent {
/**
* Fixed data part: Empty
*
* <p>
* Variable data part:
*
* <ul>
* <li>1 byte. A value indicating the variable type: LAST_INSERT_ID_EVENT =
* 1 or INSERT_ID_EVENT = 2.</li>
* <li>8 bytes. An unsigned integer indicating the value to be used for the
* LAST_INSERT_ID() invocation or AUTO_INCREMENT column.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private final long value;
@@ -68,31 +57,25 @@ public final class IntvarLogEvent extends LogEvent
public static final int LAST_INSERT_ID_EVENT = 1;
public static final int INSERT_ID_EVENT = 2;
public IntvarLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public IntvarLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
/* The Post-Header is empty. The Varible Data part begins immediately. */
buffer.position(descriptionEvent.commonHeaderLen
+ descriptionEvent.postHeaderLen[INTVAR_EVENT - 1]
+ I_TYPE_OFFSET);
buffer.position(descriptionEvent.commonHeaderLen + descriptionEvent.postHeaderLen[INTVAR_EVENT - 1]
+ I_TYPE_OFFSET);
type = buffer.getInt8(); // I_TYPE_OFFSET
value = buffer.getLong64(); // !uint8korr(buf + I_VAL_OFFSET);
}
public final int getType()
{
public final int getType() {
return type;
}
public final long getValue()
{
public final long getValue() {
return value;
}
public final String getQuery()
{
public final String getQuery() {
return "SET INSERT_ID = " + value;
}
}
@@ -26,16 +26,13 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* </pre>
*
* Binary Format: The Post-Header consists of the following six components.
*
* <table>
* <caption>Post-Header for Load_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>slave_proxy_id</td>
* <td>4 byte unsigned integer</td>
@@ -45,71 +42,57 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* temporary table local to the client. The slave_proxy_id is used to
* distinguish temporary tables that belong to different clients.</td>
* </tr>
*
* <tr>
* <td>exec_time</td>
* <td>4 byte unsigned integer</td>
* <td>The time from when the query started to when it was logged in the binlog,
* in seconds.</td>
* </tr>
*
* <tr>
* <td>skip_lines</td>
* <td>4 byte unsigned integer</td>
* <td>The number on line (14) above, if present, or 0 if line (14) is left out.
* </td>
* </tr>
*
* <tr>
* <td>table_name_len</td>
* <td>1 byte unsigned integer</td>
* <td>The length of 'table_name' on line (4) above.</td>
* </tr>
*
* <tr>
* <td>db_len</td>
* <td>1 byte unsigned integer</td>
* <td>The length of 'db' on line (1) above.</td>
* </tr>
*
* <tr>
* <td>num_fields</td>
* <td>4 byte unsigned integer</td>
* <td>The number n of fields on line (15) above.</td>
* </tr>
* </table>
*
* The Body contains the following components.
*
* <table>
* <caption>Body of Load_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>sql_ex</td>
* <td>variable length</td>
*
* <td>Describes the part of the query on lines (3) and (5)&ndash;(13) above.
* More precisely, it stores the five strings (on lines) field_term (6),
* enclosed (7), escaped (8), line_term (11), and line_start (12); as well as a
* bitfield indicating the presence of the keywords REPLACE (3), IGNORE (3), and
* OPTIONALLY (7).
*
* The data is stored in one of two formats, called "old" and "new". The type
* field of Common-Header determines which of these two formats is used: type
* LOAD_EVENT means that the old format is used, and type NEW_LOAD_EVENT means
* that the new format is used. When MySQL writes a Load_log_event, it uses the
* new format if at least one of the five strings is two or more bytes long.
* Otherwise (i.e., if all strings are 0 or 1 bytes long), the old format is
* used.
*
* The new and old format differ in the way the five strings are stored.
*
* OPTIONALLY (7). The data is stored in one of two formats, called "old" and
* "new". The type field of Common-Header determines which of these two formats
* is used: type LOAD_EVENT means that the old format is used, and type
* NEW_LOAD_EVENT means that the new format is used. When MySQL writes a
* Load_log_event, it uses the new format if at least one of the five strings is
* two or more bytes long. Otherwise (i.e., if all strings are 0 or 1 bytes
* long), the old format is used. The new and old format differ in the way the
* five strings are stored.
* <ul>
* <li>In the new format, the strings are stored in the order field_term,
* enclosed, escaped, line_term, line_start. Each string consists of a length (1
@@ -117,7 +100,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* combination of the following flags is stored in 1 byte: REPLACE_FLAG==0x4,
* IGNORE_FLAG==0x8, and OPT_ENCLOSED_FLAG==0x2. If a flag is set, it indicates
* the presence of the corresponding keyword in the SQL query.
*
* <li>In the old format, we know that each string has length 0 or 1. Therefore,
* only the first byte of each string is stored. The order of the strings is the
* same as in the new format. These five bytes are followed by the same 1 byte
@@ -128,18 +110,15 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* FIELD_TERM_EMPTY==0x1, ENCLOSED_EMPTY==0x2, LINE_TERM_EMPTY==0x4,
* LINE_START_EMPTY==0x8, ESCAPED_EMPTY==0x10.
* </ul>
*
* Thus, the size of the new format is 6 bytes + the sum of the sizes of the
* five strings. The size of the old format is always 7 bytes.</td>
* </tr>
*
* <tr>
* <td>field_lens</td>
* <td>num_fields 1 byte unsigned integers</td>
* <td>An array of num_fields integers representing the length of each field in
* the query. (num_fields is from the Post-Header).</td>
* </tr>
*
* <tr>
* <td>fields</td>
* <td>num_fields null-terminated strings</td>
@@ -148,7 +127,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* the num_fields array.) The total length of all strings equals to the sum of
* all field_lens, plus num_fields bytes for all the trailing zeros.</td>
* </tr>
*
* <tr>
* <td>table_name</td>
* <td>null-terminated string of length table_len+1 bytes</td>
@@ -156,31 +134,27 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* trailing zero is actually redundant since the table_len is known from
* Post-Header.)</td>
* </tr>
*
* <tr>
* <td>db</td>
* <td>null-terminated string of length db_len+1 bytes</td>
* <td>The 'db' from the query, as a null-terminated string. (The trailing zero
* is actually redundant since the db_len is known from Post-Header.)</td>
* </tr>
*
* <tr>
* <td>file_name</td>
* <td>variable length string without trailing zero, extending to the end of the
* event (determined by the length field of the Common-Header)</td>
* <td>The 'file_name' from the query.</td>
* </tr>
*
* </table>
*
* This event type is understood by current versions, but only generated by
* MySQL 3.23 and earlier.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public class LoadLogEvent extends LogEvent
{
public class LoadLogEvent extends LogEvent {
private String table;
private String db;
private String fname;
@@ -210,11 +184,10 @@ public class LoadLogEvent extends LogEvent
public static final int L_DATA_OFFSET = FormatDescriptionLogEvent.LOAD_HEADER_LEN;
/*
These are flags and structs to handle all the LOAD DATA INFILE options (LINES
TERMINATED etc).
DUMPFILE_FLAG is probably useless (DUMPFILE is a clause of SELECT, not of LOAD
DATA).
*/
* These are flags and structs to handle all the LOAD DATA INFILE options
* (LINES TERMINATED etc). DUMPFILE_FLAG is probably useless (DUMPFILE is a
* clause of SELECT, not of LOAD DATA).
*/
public static final int DUMPFILE_FLAG = 0x1;
public static final int OPT_ENCLOSED_FLAG = 0x2;
public static final int REPLACE_FLAG = 0x4;
@@ -226,9 +199,7 @@ public class LoadLogEvent extends LogEvent
public static final int LINE_START_EMPTY = 0x8;
public static final int ESCAPED_EMPTY = 0x10;
public LoadLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public LoadLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int loadHeaderLen = FormatDescriptionLogEvent.LOAD_HEADER_LEN;
@@ -236,18 +207,16 @@ public class LoadLogEvent extends LogEvent
* I (Guilhem) manually tested replication of LOAD DATA INFILE for
* 3.23->5.0, 4.0->5.0 and 5.0->5.0 and it works.
*/
copyLogEvent(buffer, ((header.type == LOAD_EVENT) ? loadHeaderLen
+ descriptionEvent.commonHeaderLen : loadHeaderLen
+ FormatDescriptionLogEvent.LOG_EVENT_HEADER_LEN),
descriptionEvent);
copyLogEvent(buffer,
((header.type == LOAD_EVENT) ? loadHeaderLen + descriptionEvent.commonHeaderLen : loadHeaderLen
+ FormatDescriptionLogEvent.LOG_EVENT_HEADER_LEN),
descriptionEvent);
}
/**
* @see mysql-5.1.60/sql/log_event.cc - Load_log_event::copy_log_event
*/
protected final void copyLogEvent(LogBuffer buffer, final int bodyOffset,
FormatDescriptionLogEvent descriptionEvent)
{
protected final void copyLogEvent(LogBuffer buffer, final int bodyOffset, FormatDescriptionLogEvent descriptionEvent) {
/* this is the beginning of the post-header */
buffer.position(descriptionEvent.commonHeaderLen + L_EXEC_TIME_OFFSET);
@@ -262,8 +231,7 @@ public class LoadLogEvent extends LogEvent
* Sql_ex.init() on success returns the pointer to the first byte after
* the sql_ex structure, which is the start of field lengths array.
*/
if (header.type != LOAD_EVENT /* use_new_format */)
{
if (header.type != LOAD_EVENT /* use_new_format */) {
/*
* The code below assumes that buf will not disappear from under our
* feet during the lifetime of the event. This assumption holds true
@@ -279,9 +247,7 @@ public class LoadLogEvent extends LogEvent
escaped = buffer.getString();
optFlags = buffer.getInt8();
emptyFlags = 0;
}
else
{
} else {
fieldTerm = buffer.getFixString(1);
enclosed = buffer.getFixString(1);
lineTerm = buffer.getFixString(1);
@@ -290,23 +256,17 @@ public class LoadLogEvent extends LogEvent
optFlags = buffer.getUint8();
emptyFlags = buffer.getUint8();
if ((emptyFlags & FIELD_TERM_EMPTY) != 0)
fieldTerm = null;
if ((emptyFlags & ENCLOSED_EMPTY) != 0)
enclosed = null;
if ((emptyFlags & LINE_TERM_EMPTY) != 0)
lineTerm = null;
if ((emptyFlags & LINE_START_EMPTY) != 0)
lineStart = null;
if ((emptyFlags & ESCAPED_EMPTY) != 0)
escaped = null;
if ((emptyFlags & FIELD_TERM_EMPTY) != 0) fieldTerm = null;
if ((emptyFlags & ENCLOSED_EMPTY) != 0) enclosed = null;
if ((emptyFlags & LINE_TERM_EMPTY) != 0) lineTerm = null;
if ((emptyFlags & LINE_START_EMPTY) != 0) lineStart = null;
if ((emptyFlags & ESCAPED_EMPTY) != 0) escaped = null;
}
final int fieldLenPos = buffer.position();
buffer.forward(numFields);
fields = new String[numFields];
for (int i = 0; i < numFields; i++)
{
for (int i = 0; i < numFields; i++) {
final int fieldLen = buffer.getUint8(fieldLenPos + i);
fields[i] = buffer.getFixString(fieldLen + 1);
}
@@ -314,78 +274,65 @@ public class LoadLogEvent extends LogEvent
table = buffer.getFixString(tableNameLen + 1);
db = buffer.getFixString(dbLen + 1);
// null termination is accomplished by the caller
// null termination is accomplished by the caller
final int from = buffer.position();
final int end = from + buffer.limit();
int found = from;
for (; (found < end) && buffer.getInt8(found) != '\0'; found++)
/* empty loop */;
fname = buffer.getString(found);
buffer.forward(1); // The + 1 is for \0 terminating fname
buffer.forward(1); // The + 1 is for \0 terminating fname
}
public final String getTable()
{
public final String getTable() {
return table;
}
public final String getDb()
{
public final String getDb() {
return db;
}
public final String getFname()
{
public final String getFname() {
return fname;
}
public final int getSkipLines()
{
public final int getSkipLines() {
return skipLines;
}
public final String[] getFields()
{
public final String[] getFields() {
return fields;
}
public final String getFieldTerm()
{
public final String getFieldTerm() {
return fieldTerm;
}
public final String getLineTerm()
{
public final String getLineTerm() {
return lineTerm;
}
public final String getLineStart()
{
public final String getLineStart() {
return lineStart;
}
public final String getEnclosed()
{
public final String getEnclosed() {
return enclosed;
}
public final String getEscaped()
{
public final String getEscaped() {
return escaped;
}
public final int getOptFlags()
{
public final int getOptFlags() {
return optFlags;
}
public final int getEmptyFlags()
{
public final int getEmptyFlags() {
return emptyFlags;
}
public final long getExecTime()
{
public final long getExecTime() {
return execTime;
}
}
@@ -11,41 +11,34 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* different format and length even for different events of the same type. The
* binary formats of Post-Header and Body are documented separately in each
* subclass. The binary format of Common-Header is as follows.
*
* <table>
* <caption>Common-Header</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>timestamp</td>
* <td>4 byte unsigned integer</td>
* <td>The time when the query started, in seconds since 1970.</td>
* </tr>
*
* <tr>
* <td>type</td>
* <td>1 byte enumeration</td>
* <td>See enum #Log_event_type.</td>
* </tr>
*
* <tr>
* <td>server_id</td>
* <td>4 byte unsigned integer</td>
* <td>Server ID of the server that created the event.</td>
* </tr>
*
* <tr>
* <td>total_size</td>
* <td>4 byte unsigned integer</td>
* <td>The total size of this event, in bytes. In other words, this is the sum
* of the sizes of Common-Header, Post-Header, and Body.</td>
* </tr>
*
* <tr>
* <td>master_position</td>
* <td>4 byte unsigned integer</td>
@@ -54,24 +47,21 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* the position of the next event, in bytes from the beginning of the file. In a
* relay log, this is the position of the next event in the master's binlog.</td>
* </tr>
*
* <tr>
* <td>flags</td>
* <td>2 byte bitfield</td>
* <td>See Log_event::flags.</td>
* </tr>
* </table>
*
* Summing up the numbers above, we see that the total size of the common header
* is 19 bytes.
*
* @see mysql-5.1.60/sql/log_event.cc
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class LogHeader
{
public final class LogHeader {
protected final int type;
/**
@@ -111,37 +101,32 @@ public final class LogHeader
* LOG_EVENT_SUPPRESS_USE_F for notes.
*/
protected int flags;
/**
/**
* The value is set by caller of FD constructor and
* Log_event::write_header() for the rest.
* In the FD case it's propagated into the last byte
* of post_header_len[] at FD::write().
* On the slave side the value is assigned from post_header_len[last]
* of the last seen FD event.
* Log_event::write_header() for the rest. In the FD case it's propagated
* into the last byte of post_header_len[] at FD::write(). On the slave side
* the value is assigned from post_header_len[last] of the last seen FD
* event.
*/
protected int checksumAlg;
/**
Placeholder for event checksum while writing to binlog.
*/
protected long crc; // ha_checksum
* Placeholder for event checksum while writing to binlog.
*/
protected long crc; // ha_checksum
/* for Start_event_v3 */
public LogHeader(final int type)
{
public LogHeader(final int type){
this.type = type;
}
public LogHeader(LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public LogHeader(LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
when = buffer.getUint32();
type = buffer.getUint8(); // LogEvent.EVENT_TYPE_OFFSET;
serverId = buffer.getUint32(); // LogEvent.SERVER_ID_OFFSET;
eventLen = (int) buffer.getUint32(); // LogEvent.EVENT_LEN_OFFSET;
if (descriptionEvent.binlogVersion == 1)
{
if (descriptionEvent.binlogVersion == 1) {
logPos = 0;
flags = 0;
return;
@@ -150,50 +135,45 @@ public final class LogHeader
/* 4.0 or newer */
logPos = buffer.getUint32(); // LogEvent.LOG_POS_OFFSET
/*
If the log is 4.0 (so here it can only be a 4.0 relay log read by
the SQL thread or a 4.0 master binlog read by the I/O thread),
log_pos is the beginning of the event: we transform it into the end
of the event, which is more useful.
But how do you know that the log is 4.0: you know it if
description_event is version 3 *and* you are not reading a
Format_desc (remember that mysqlbinlog starts by assuming that 5.0
logs are in 4.0 format, until it finds a Format_desc).
*/
if (descriptionEvent.binlogVersion == 3
&& type < LogEvent.FORMAT_DESCRIPTION_EVENT && logPos != 0)
{
* If the log is 4.0 (so here it can only be a 4.0 relay log read by the
* SQL thread or a 4.0 master binlog read by the I/O thread), log_pos is
* the beginning of the event: we transform it into the end of the
* event, which is more useful. But how do you know that the log is 4.0:
* you know it if description_event is version 3 *and* you are not
* reading a Format_desc (remember that mysqlbinlog starts by assuming
* that 5.0 logs are in 4.0 format, until it finds a Format_desc).
*/
if (descriptionEvent.binlogVersion == 3 && type < LogEvent.FORMAT_DESCRIPTION_EVENT && logPos != 0) {
/*
If log_pos=0, don't change it. log_pos==0 is a marker to mean
"don't change rli->group_master_log_pos" (see
inc_group_relay_log_pos()). As it is unreal log_pos, adding the
event len's is nonsense. For example, a fake Rotate event should
not have its log_pos (which is 0) changed or it will modify
Exec_master_log_pos in SHOW SLAVE STATUS, displaying a nonsense
value of (a non-zero offset which does not exist in the master's
binlog, so which will cause problems if the user uses this value
in CHANGE MASTER).
*/
* If log_pos=0, don't change it. log_pos==0 is a marker to mean
* "don't change rli->group_master_log_pos" (see
* inc_group_relay_log_pos()). As it is unreal log_pos, adding the
* event len's is nonsense. For example, a fake Rotate event should
* not have its log_pos (which is 0) changed or it will modify
* Exec_master_log_pos in SHOW SLAVE STATUS, displaying a nonsense
* value of (a non-zero offset which does not exist in the master's
* binlog, so which will cause problems if the user uses this value
* in CHANGE MASTER).
*/
logPos += eventLen; /* purecov: inspected */
}
flags = buffer.getUint16(); // LogEvent.FLAGS_OFFSET
if ((type == LogEvent.FORMAT_DESCRIPTION_EVENT)
|| (type == LogEvent.ROTATE_EVENT))
{
if ((type == LogEvent.FORMAT_DESCRIPTION_EVENT) || (type == LogEvent.ROTATE_EVENT)) {
/*
These events always have a header which stops here (i.e. their
header is FROZEN).
*/
* These events always have a header which stops here (i.e. their
* header is FROZEN).
*/
/*
Initialization to zero of all other Log_event members as they're
not specified. Currently there are no such members; in the future
there will be an event UID (but Format_description and Rotate
don't need this UID, as they are not propagated through
--log-slave-updates (remember the UID is used to not play a query
twice when you have two masters which are slaves of a 3rd master).
Then we are done.
*/
* Initialization to zero of all other Log_event members as they're
* not specified. Currently there are no such members; in the future
* there will be an event UID (but Format_description and Rotate
* don't need this UID, as they are not propagated through
* --log-slave-updates (remember the UID is used to not play a query
* twice when you have two masters which are slaves of a 3rd
* master). Then we are done.
*/
if (type == LogEvent.FORMAT_DESCRIPTION_EVENT) {
int commonHeaderLen = buffer.getUint8(FormatDescriptionLogEvent.LOG_EVENT_MINIMAL_HEADER_LEN
+ FormatDescriptionLogEvent.ST_COMMON_HEADER_LEN_OFFSET);
@@ -211,27 +191,24 @@ public final class LogHeader
}
return;
}
/*
CRC verification by SQL and Show-Binlog-Events master side.
The caller has to provide @description_event->checksum_alg to
be the last seen FD's (A) descriptor.
If event is FD the descriptor is in it.
Notice, FD of the binlog can be only in one instance and therefore
Show-Binlog-Events executing master side thread needs just to know
the only FD's (A) value - whereas RL can contain more.
In the RL case, the alg is kept in FD_e (@description_event) which is reset
to the newer read-out event after its execution with possibly new alg descriptor.
Therefore in a typical sequence of RL:
{FD_s^0, FD_m, E_m^1} E_m^1
will be verified with (A) of FD_m.
See legends definition on MYSQL_BIN_LOG::relay_log_checksum_alg docs
lines (log.h).
Notice, a pre-checksum FD version forces alg := BINLOG_CHECKSUM_ALG_UNDEF.
*/
checksumAlg = descriptionEvent.getHeader().checksumAlg; // fetch checksum alg
* CRC verification by SQL and Show-Binlog-Events master side. The
* caller has to provide @description_event->checksum_alg to be the last
* seen FD's (A) descriptor. If event is FD the descriptor is in it.
* Notice, FD of the binlog can be only in one instance and therefore
* Show-Binlog-Events executing master side thread needs just to know
* the only FD's (A) value - whereas RL can contain more. In the RL
* case, the alg is kept in FD_e (@description_event) which is reset to
* the newer read-out event after its execution with possibly new alg
* descriptor. Therefore in a typical sequence of RL: {FD_s^0, FD_m,
* E_m^1} E_m^1 will be verified with (A) of FD_m. See legends
* definition on MYSQL_BIN_LOG::relay_log_checksum_alg docs lines
* (log.h). Notice, a pre-checksum FD version forces alg :=
* BINLOG_CHECKSUM_ALG_UNDEF.
*/
checksumAlg = descriptionEvent.getHeader().checksumAlg; // fetch
// checksum alg
processCheckSum(buffer);
/* otherwise, go on with reading the header from buf (nothing now) */
}
@@ -239,8 +216,7 @@ public final class LogHeader
/**
* The different types of log events.
*/
public final int getType()
{
public final int getType() {
return type;
}
@@ -251,8 +227,7 @@ public final class LogHeader
* file. In a relay log, this is the position of the next event in the
* master's binlog.
*/
public final long getLogPos()
{
public final long getLogPos() {
return logPos;
}
@@ -260,24 +235,21 @@ public final class LogHeader
* The total size of this event, in bytes. In other words, this is the sum
* of the sizes of Common-Header, Post-Header, and Body.
*/
public final int getEventLen()
{
public final int getEventLen() {
return eventLen;
}
/**
* The time when the query started, in seconds since 1970.
*/
public final long getWhen()
{
public final long getWhen() {
return when;
}
/**
* Server ID of the server that created the event.
*/
public final long getServerId()
{
public final long getServerId() {
return serverId;
}
@@ -286,25 +258,21 @@ public final class LogHeader
* LOG_EVENT_FORCED_ROTATE_F, LOG_EVENT_THREAD_SPECIFIC_F, and
* LOG_EVENT_SUPPRESS_USE_F for notes.
*/
public final int getFlags()
{
public final int getFlags() {
return flags;
}
public long getCrc() {
return crc;
}
public int getChecksumAlg() {
return checksumAlg;
}
private void processCheckSum(LogBuffer buffer) {
if (checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_OFF &&
checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_UNDEF){
crc = buffer.getUint32(eventLen - LogEvent.BINLOG_CHECKSUM_LEN);
if (checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_OFF && checksumAlg != LogEvent.BINLOG_CHECKSUM_ALG_UNDEF) {
crc = buffer.getUint32(eventLen - LogEvent.BINLOG_CHECKSUM_LEN);
}
}
}
@@ -4,16 +4,14 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
*
* @author jianghang 2013-4-8 上午12:36:29
* @version 1.0.3
* @since mysql 5.6
*/
public class PreviousGtidsLogEvent extends LogEvent{
public class PreviousGtidsLogEvent extends LogEvent {
public PreviousGtidsLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
// do nothing , just for mysql gtid search function
}
}
@@ -9,19 +9,14 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* A Query_log_event is created for each query that modifies the database,
* unless the query is logged row-based.
*
* The Post-Header has five components:
*
* unless the query is logged row-based. The Post-Header has five components:
* <table>
* <caption>Post-Header for Query_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>slave_proxy_id</td>
* <td>4 byte unsigned integer</td>
@@ -31,20 +26,17 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* temporary table local to the client. The slave_proxy_id is used to
* distinguish temporary tables that belong to different clients.</td>
* </tr>
*
* <tr>
* <td>exec_time</td>
* <td>4 byte unsigned integer</td>
* <td>The time from when the query started to when it was logged in the binlog,
* in seconds.</td>
* </tr>
*
* <tr>
* <td>db_len</td>
* <td>1 byte integer</td>
* <td>The length of the name of the currently selected database.</td>
* </tr>
*
* <tr>
* <td>error_code</td>
* <td>2 byte unsigned integer</td>
@@ -52,7 +44,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* fail with the same error code, except for the error codes ER_DB_CREATE_EXISTS
* == 1007 and ER_DB_DROP_EXISTS == 1008.</td>
* </tr>
*
* <tr>
* <td>status_vars_len</td>
* <td>2 byte unsigned integer</td>
@@ -60,18 +51,14 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* query_log_event_status_vars "below".</td>
* </tr>
* </table>
*
* The Body has the following components:
*
* <table>
* <caption>Body for Query_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>query_log_event_status_vars status_vars</td>
* <td>status_vars_len bytes</td>
@@ -81,16 +68,13 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* Table_query_log_event_status_vars "below". MySQL always writes events in the
* order defined below; however, it is capable of reading them in any order.</td>
* </tr>
*
* <tr>
* <td>db</td>
* <td>db_len+1</td>
* <td>The currently selected database, as a null-terminated string.
*
* (The trailing zero is redundant since the length is already known; it is
* db_len from Post-Header.)</td>
* <td>The currently selected database, as a null-terminated string. (The
* trailing zero is redundant since the length is already known; it is db_len
* from Post-Header.)</td>
* </tr>
*
* <tr>
* <td>query</td>
* <td>variable length string without trailing zero, extending to the end of the
@@ -98,20 +82,16 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>The SQL query.</td>
* </tr>
* </table>
*
* The following table lists the status variables that may appear in the
* status_vars field. Table_query_log_event_status_vars
*
* <table>
* <caption>Status variables for Query_log_event</caption>
*
* <tr>
* <th>Status variable</th>
* <th>1 byte identifier</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>flags2</td>
* <td>Q_FLAGS2_CODE == 0</td>
@@ -121,16 +101,12 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* identifies those options that need to be written to the binlog (not all do).
* Specifically, OPTIONS_WRITTEN_TO_BIN_LOG equals (OPTION_AUTO_IS_NULL |
* OPTION_NO_FOREIGN_KEY_CHECKS | OPTION_RELAXED_UNIQUE_CHECKS |
* OPTION_NOT_AUTOCOMMIT), or 0x0c084000 in hex.
*
* These flags correspond to the SQL variables SQL_AUTO_IS_NULL,
* FOREIGN_KEY_CHECKS, UNIQUE_CHECKS, and AUTOCOMMIT, documented in the
* "SET Syntax" section of the MySQL Manual.
*
* This field is always written to the binlog in version >= 5.0, and never
* written in version < 5.0.</td>
* OPTION_NOT_AUTOCOMMIT), or 0x0c084000 in hex. These flags correspond to the
* SQL variables SQL_AUTO_IS_NULL, FOREIGN_KEY_CHECKS, UNIQUE_CHECKS, and
* AUTOCOMMIT, documented in the "SET Syntax" section of the MySQL Manual. This
* field is always written to the binlog in version >= 5.0, and never written in
* version < 5.0.</td>
* </tr>
*
* <tr>
* <td>sql_mode</td>
* <td>Q_SQL_MODE_CODE == 1</td>
@@ -177,11 +153,9 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* All these flags are replicated from the server. However, all flags except
* MODE_NO_DIR_IN_CREATE are honored by the slave; the slave always preserves
* its old value of MODE_NO_DIR_IN_CREATE. For a rationale, see comment in
* Query_log_event::do_apply_event in log_event.cc.
*
* This field is always written to the binlog.</td>
* Query_log_event::do_apply_event in log_event.cc. This field is always written
* to the binlog.</td>
* </tr>
*
* <tr>
* <td>catalog</td>
* <td>Q_CATALOG_NZ_CODE == 6</td>
@@ -189,23 +163,17 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* characters (at most 255 bytes)</td>
* <td>Stores the client's current catalog. Every database belongs to a catalog,
* the same way that every table belongs to a database. Currently, there is only
* one catalog, "std".
*
* This field is written if the length of the catalog is > 0; otherwise it is
* not written.</td>
* one catalog, "std". This field is written if the length of the catalog is >
* 0; otherwise it is not written.</td>
* </tr>
*
* <tr>
* <td>auto_increment</td>
* <td>Q_AUTO_INCREMENT == 3</td>
* <td>two 2 byte unsigned integers, totally 2+2=4 bytes</td>
*
* <td>The two variables auto_increment_increment and auto_increment_offset, in
* that order. For more information, see "System variables" in the MySQL manual.
*
* This field is written if auto_increment > 1. Otherwise, it is not written.</td>
* </tr>
*
* <tr>
* <td>charset</td>
* <td>Q_CHARSET_CODE == 4</td>
@@ -216,33 +184,22 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* collation_connection identifies the character set and collation that the
* master converts the query to when it receives it; this is useful when
* comparing literal strings. collation_server is the default character set and
* collation used when a new database is created.
*
* See also "Connection Character Sets and Collations" in the MySQL 5.1 manual.
*
* All three variables are codes identifying a (character set, collation) pair.
* To see which codes map to which pairs, run the query "SELECT id,
* character_set_name, collation_name FROM COLLATIONS".
*
* Cf. Q_CHARSET_DATABASE_CODE below.
*
* This field is always written.</td>
* collation used when a new database is created. See also
* "Connection Character Sets and Collations" in the MySQL 5.1 manual. All three
* variables are codes identifying a (character set, collation) pair. To see
* which codes map to which pairs, run the query "SELECT id, character_set_name,
* collation_name FROM COLLATIONS". Cf. Q_CHARSET_DATABASE_CODE below. This
* field is always written.</td>
* </tr>
*
* <tr>
* <td>time_zone</td>
* <td>Q_TIME_ZONE_CODE == 5</td>
* <td>Variable-length string: the length in bytes (1 byte) followed by the
* characters (at most 255 bytes).
* <td>The time_zone of the master.
*
* See also "System Variables" and "MySQL Server Time Zone Support" in the MySQL
* manual.
*
* This field is written if the length of the time zone string is > 0;
* otherwise, it is not written.</td>
* <td>The time_zone of the master. See also "System Variables" and
* "MySQL Server Time Zone Support" in the MySQL manual. This field is written
* if the length of the time zone string is > 0; otherwise, it is not written.</td>
* </tr>
*
* <tr>
* <td>lc_time_names_number</td>
* <td>Q_LC_TIME_NAMES_CODE == 7</td>
@@ -251,84 +208,68 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* to languages is defined in sql_locale.cc. This field is written if it is not
* 0, i.e., if the locale is not en_US.</td>
* </tr>
*
* <tr>
* <td>charset_database_number</td>
* <td>Q_CHARSET_DATABASE_CODE == 8</td>
* <td>2 byte integer</td>
*
* <td>The value of the collation_database system variable (in the source code
* stored in thd->variables.collation_database), which holds the code for a
* (character set, collation) pair as described above (see Q_CHARSET_CODE).
*
* collation_database was used in old versions (???WHEN). Its value was loaded
* when issuing a "use db" query and could be changed by issuing a
* "SET collation_database=xxx" query. It used to affect the "LOAD DATA INFILE"
* and "CREATE TABLE" commands.
*
* In newer versions, "CREATE TABLE" has been changed to take the character set
* from the database of the created table, rather than the character set of the
* current database. This makes a difference when creating a table in another
* database than the current one. "LOAD DATA INFILE" has not yet changed to do
* this, but there are plans to eventually do it, and to make collation_database
* read-only.
*
* This field is written if it is not 0.</td>
* and "CREATE TABLE" commands. In newer versions, "CREATE TABLE" has been
* changed to take the character set from the database of the created table,
* rather than the character set of the current database. This makes a
* difference when creating a table in another database than the current one.
* "LOAD DATA INFILE" has not yet changed to do this, but there are plans to
* eventually do it, and to make collation_database read-only. This field is
* written if it is not 0.</td>
* </tr>
* <tr>
* <td>table_map_for_update</td>
* <td>Q_TABLE_MAP_FOR_UPDATE_CODE == 9</td>
* <td>8 byte integer</td>
*
* <td>The value of the table map that is to be updated by the multi-table
* update query statement. Every bit of this variable represents a table, and is
* set to 1 if the corresponding table is to be updated by this statement.
*
* The value of this variable is set when executing a multi-table update
* statement and used by slave to apply filter rules without opening all the
* tables on slave. This is required because some tables may not exist on slave
* because of the filter rules.</td>
* set to 1 if the corresponding table is to be updated by this statement. The
* value of this variable is set when executing a multi-table update statement
* and used by slave to apply filter rules without opening all the tables on
* slave. This is required because some tables may not exist on slave because of
* the filter rules.</td>
* </tr>
* </table>
*
* Query_log_event_notes_on_previous_versions Notes on Previous Versions
*
* Status vars were introduced in version 5.0. To read earlier versions
* correctly, check the length of the Post-Header.
*
* The status variable Q_CATALOG_CODE == 2 existed in MySQL 5.0.x, where
* 0<=x<=3. It was identical to Q_CATALOG_CODE, except that the string had a
* trailing '\0'. The '\0' was removed in 5.0.4 since it was redundant (the
* string length is stored before the string). The Q_CATALOG_CODE will never be
* written by a new master, but can still be understood by a new slave.
*
* See Q_CHARSET_DATABASE_CODE in the table above.
*
* Query_log_event_notes_on_previous_versions Notes on Previous Versions Status
* vars were introduced in version 5.0. To read earlier versions correctly,
* check the length of the Post-Header. The status variable Q_CATALOG_CODE == 2
* existed in MySQL 5.0.x, where 0<=x<=3. It was identical to Q_CATALOG_CODE,
* except that the string had a trailing '\0'. The '\0' was removed in 5.0.4
* since it was redundant (the string length is stored before the string). The
* Q_CATALOG_CODE will never be written by a new master, but can still be
* understood by a new slave. See Q_CHARSET_DATABASE_CODE in the table above.
* When adding new status vars, please don't forget to update the
* MAX_SIZE_LOG_EVENT_STATUS, and update function code_name
*
* @see mysql-5.1.6/sql/logevent.cc - Query_log_event
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public class QueryLogEvent extends LogEvent
{
public class QueryLogEvent extends LogEvent {
/**
The maximum number of updated databases that a status of
Query-log-event can carry. It can redefined within a range
[1.. OVER_MAX_DBS_IN_EVENT_MTS].
* The maximum number of updated databases that a status of Query-log-event
* can carry. It can redefined within a range [1..
* OVER_MAX_DBS_IN_EVENT_MTS].
*/
public static final int MAX_DBS_IN_EVENT_MTS = 16;
/**
When the actual number of databases exceeds MAX_DBS_IN_EVENT_MTS
the value of OVER_MAX_DBS_IN_EVENT_MTS is is put into the
mts_accessed_dbs status.
* When the actual number of databases exceeds MAX_DBS_IN_EVENT_MTS the
* value of OVER_MAX_DBS_IN_EVENT_MTS is is put into the mts_accessed_dbs
* status.
*/
public static final int OVER_MAX_DBS_IN_EVENT_MTS = 254;
public static final int SYSTEM_CHARSET_MBMAXLEN = 3;
public static final int NAME_CHAR_LEN = 64;
/* Field/table name length */
@@ -340,23 +281,68 @@ public class QueryLogEvent extends LogEvent
* log_event status vars estimation:
*/
public static final int MAX_SIZE_LOG_EVENT_STATUS = (1 + 4 /* type, flags2 */
+ 1 + 8 /* type, sql_mode */
+ 1 + 1 + 255 /* type, length, catalog */
+ 1 + 4 /* type, auto_increment */
+ 1 + 6 /* type, charset */
+ 1 + 1 + 255 /* type, length, time_zone */
+ 1 + 2 /* type, lc_time_names_number */
+ 1 + 2 /* type, charset_database_number */
+ 1 + 8 /* type, table_map_for_update */
+ 1 + 4 /* type, master_data_written */
/* type, db_1, db_2, ... */
/* type, microseconds */
/* MariaDb type, sec_part of NOW() */
+ 1 + (MAX_DBS_IN_EVENT_MTS * (1 + NAME_LEN)) + 3
+ 1 + 16 + 1 + 60/* type, user_len, user, host_len, host */);
+ 1 + 8 /*
* type,
* sql_mode
*/
+ 1 + 1 + 255 /*
* type,
* length
* ,
* catalog
*/
+ 1 + 4 /*
* type,
* auto_increment
*/
+ 1 + 6 /*
* type,
* charset
*/
+ 1 + 1 + 255 /*
* type,
* length
* ,
* time_zone
*/
+ 1 + 2 /*
* type,
* lc_time_names_number
*/
+ 1 + 2 /*
* type,
* charset_database_number
*/
+ 1 + 8 /*
* type,
* table_map_for_update
*/
+ 1 + 4 /*
* type,
* master_data_written
*/
/*
* type, db_1, db_2,
* ...
*/
/* type, microseconds */
/*
* MariaDb type,
* sec_part of NOW()
*/
+ 1 + (MAX_DBS_IN_EVENT_MTS * (1 + NAME_LEN)) + 3 + 1 + 16 + 1 + 60/*
* type
* ,
* user_len
* ,
* user
* ,
* host_len
* ,
* host
*/);
/**
* Fixed data part:
*
* <ul>
* <li>4 bytes. The ID of the thread that issued this statement. Needed for
* temporary tables. This is also useful for a DBA for knowing who did what
@@ -385,7 +371,6 @@ public class QueryLogEvent extends LogEvent
* <li>2 bytes (not present in v1, v3). The length of the status variable
* block.</li>
* </ul>
*
* Variable part:
* <ul>
* <li>Zero or more status variables (not present in v1, v3). Each status
@@ -397,7 +382,6 @@ public class QueryLogEvent extends LogEvent
* the variable part (the sizes are given in the fixed data part), so by
* subtraction it can know the size of the statement.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private String user;
@@ -409,10 +393,11 @@ public class QueryLogEvent extends LogEvent
protected final String dbname;
/** The number of seconds the query took to run on the master. */
// The time in seconds that the statement took to execute. Only useful for inspection by the DBA
// The time in seconds that the statement took to execute. Only useful for
// inspection by the DBA
private final long execTime;
private final int errorCode;
private final long sessionId; /* thread_id */
private final long sessionId; /* thread_id */
/**
* 'flags2' is a second set of flags (on top of those in Log_event), for
@@ -434,9 +419,8 @@ public class QueryLogEvent extends LogEvent
private String timezone;
public QueryLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent) throws IOException
{
public QueryLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent)
throws IOException{
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
@@ -447,8 +431,7 @@ public class QueryLogEvent extends LogEvent
* format-tolerant. We use QUERY_HEADER_MINIMAL_LEN which is the same
* for 3.23, 4.0 & 5.0.
*/
if (buffer.limit() < (commonHeaderLen + postHeaderLen))
{
if (buffer.limit() < (commonHeaderLen + postHeaderLen)) {
throw new IOException("Query event length is too short.");
}
int dataLen = buffer.limit() - (commonHeaderLen + postHeaderLen);
@@ -462,24 +445,21 @@ public class QueryLogEvent extends LogEvent
errorCode = buffer.getUint16(); // Q_ERR_CODE_OFFSET
/*
* 5.0 format starts here. Depending on the format, we may or not
* have affected/warnings etc The remaining post-header to be parsed
* has length:
* 5.0 format starts here. Depending on the format, we may or not have
* affected/warnings etc The remaining post-header to be parsed has
* length:
*/
int statusVarsLen = 0;
if (postHeaderLen > QUERY_HEADER_MINIMAL_LEN)
{
if (postHeaderLen > QUERY_HEADER_MINIMAL_LEN) {
statusVarsLen = buffer.getUint16(); // Q_STATUS_VARS_LEN_OFFSET
/*
Check if status variable length is corrupt and will lead to very
wrong data. We could be even more strict and require data_len to
be even bigger, but this will suffice to catch most corruption
errors that can lead to a crash.
*/
if (statusVarsLen > Math.min(dataLen, MAX_SIZE_LOG_EVENT_STATUS))
{
throw new IOException("status_vars_len (" + statusVarsLen
+ ") > data_len (" + dataLen + ")");
* Check if status variable length is corrupt and will lead to very
* wrong data. We could be even more strict and require data_len to
* be even bigger, but this will suffice to catch most corruption
* errors that can lead to a crash.
*/
if (statusVarsLen > Math.min(dataLen, MAX_SIZE_LOG_EVENT_STATUS)) {
throw new IOException("status_vars_len (" + statusVarsLen + ") > data_len (" + dataLen + ")");
}
dataLen -= statusVarsLen;
}
@@ -489,7 +469,7 @@ public class QueryLogEvent extends LogEvent
* dedicated to derived events (e.g. Execute_load_query...)
*/
/* variable-part: the status vars; only in MySQL 5.0 */
/* variable-part: the status vars; only in MySQL 5.0 */
final int start = commonHeaderLen + postHeaderLen;
final int limit = buffer.limit(); /* for restore */
final int end = start + statusVarsLen;
@@ -501,27 +481,19 @@ public class QueryLogEvent extends LogEvent
/* A 2nd variable part; this is common to all versions */
final int queryLen = dataLen - dbLen - 1;
dbname = buffer.getFixString(dbLen + 1);
if (clientCharset >= 0)
{
if (clientCharset >= 0) {
charsetName = CharsetConversion.getJavaCharset(clientCharset);
if ((charsetName != null) && (Charset.isSupported(charsetName)))
{
if ((charsetName != null) && (Charset.isSupported(charsetName))) {
query = buffer.getFixString(queryLen, charsetName);
}
else
{
logger.warn("unsupported character set in query log: "
+ "\n ID = " + clientCharset + ", Charset = "
+ CharsetConversion.getCharset(clientCharset)
+ ", Collation = "
+ CharsetConversion.getCollation(clientCharset));
} else {
logger.warn("unsupported character set in query log: " + "\n ID = " + clientCharset + ", Charset = "
+ CharsetConversion.getCharset(clientCharset) + ", Collation = "
+ CharsetConversion.getCollation(clientCharset));
query = buffer.getFixString(queryLen);
}
}
else
{
} else {
query = buffer.getFixString(queryLen);
}
}
@@ -567,215 +539,198 @@ public class QueryLogEvent extends LogEvent
public static final int Q_MASTER_DATA_WRITTEN_CODE = 10;
public static final int Q_INVOKER = 11;
/**
Q_UPDATED_DB_NAMES status variable collects of the updated databases
total number and their names to be propagated to the slave in order
to facilitate the parallel applying of the Query events.
* Q_UPDATED_DB_NAMES status variable collects of the updated databases
* total number and their names to be propagated to the slave in order to
* facilitate the parallel applying of the Query events.
*/
public static final int Q_UPDATED_DB_NAMES = 12;
public static final int Q_MICROSECONDS = 13;
/**
* FROM MariaDB 5.5.34
/**
* FROM MariaDB 5.5.34
*/
public static final int Q_HRNOW = 128;
private final void unpackVariables(LogBuffer buffer, final int end)
throws IOException
{
private final void unpackVariables(LogBuffer buffer, final int end) throws IOException {
int code = -1;
try
{
while (buffer.position() < end)
{
switch (code = buffer.getUint8())
{
case Q_FLAGS2_CODE:
flags2 = buffer.getUint32();
break;
case Q_SQL_MODE_CODE:
sql_mode = buffer.getLong64(); // QQ: Fix when sql_mode is ulonglong
break;
case Q_CATALOG_NZ_CODE:
catalog = buffer.getString();
break;
case Q_AUTO_INCREMENT:
autoIncrementIncrement = buffer.getUint16();
autoIncrementOffset = buffer.getUint16();
break;
case Q_CHARSET_CODE:
// Charset: 6 byte character set flag.
// 1-2 = character set client
// 3-4 = collation client
// 5-6 = collation server
clientCharset = buffer.getUint16();
clientCollation = buffer.getUint16();
serverCollation = buffer.getUint16();
break;
case Q_TIME_ZONE_CODE:
timezone = buffer.getString();
break;
case Q_CATALOG_CODE: /* for 5.0.x where 0<=x<=3 masters */
final int len = buffer.getUint8();
catalog = buffer.getFixString(len + 1);
break;
case Q_LC_TIME_NAMES_CODE:
// lc_time_names_number = buffer.getUint16();
buffer.forward(2);
break;
case Q_CHARSET_DATABASE_CODE:
// charset_database_number = buffer.getUint16();
buffer.forward(2);
break;
case Q_TABLE_MAP_FOR_UPDATE_CODE:
// table_map_for_update = buffer.getUlong64();
buffer.forward(8);
break;
case Q_MASTER_DATA_WRITTEN_CODE:
// data_written = master_data_written = buffer.getUint32();
buffer.forward(4);
break;
case Q_INVOKER:
user = buffer.getString();
host = buffer.getString();
break;
case Q_MICROSECONDS:
// when.tv_usec= uint3korr(pos);
buffer.forward(3);
break;
case Q_UPDATED_DB_NAMES:
int mtsAccessedDbs = buffer.getUint8();
/**
Notice, the following check is positive also in case of
the master's MAX_DBS_IN_EVENT_MTS > the slave's one and the event
contains e.g the master's MAX_DBS_IN_EVENT_MTS db:s.
*/
if (mtsAccessedDbs > MAX_DBS_IN_EVENT_MTS) {
mtsAccessedDbs = OVER_MAX_DBS_IN_EVENT_MTS;
try {
while (buffer.position() < end) {
switch (code = buffer.getUint8()) {
case Q_FLAGS2_CODE:
flags2 = buffer.getUint32();
break;
}
String mtsAccessedDbNames[] = new String[mtsAccessedDbs];
for (int i = 0; i < mtsAccessedDbs && buffer.position() < end; i++) {
int length = end - buffer.position();
mtsAccessedDbNames[i] = buffer.getFixString(length < NAME_LEN ? length : NAME_LEN);
}
break;
case Q_HRNOW:
// int when_sec_part = buffer.getUint24();
buffer.forward(3);
break;
default:
/* That's why you must write status vars in growing order of code */
if (logger.isDebugEnabled())
logger.debug("Query_log_event has unknown status vars (first has code: "
+ code + "), skipping the rest of them");
break; // Break loop
case Q_SQL_MODE_CODE:
sql_mode = buffer.getLong64(); // QQ: Fix when sql_mode
// is ulonglong
break;
case Q_CATALOG_NZ_CODE:
catalog = buffer.getString();
break;
case Q_AUTO_INCREMENT:
autoIncrementIncrement = buffer.getUint16();
autoIncrementOffset = buffer.getUint16();
break;
case Q_CHARSET_CODE:
// Charset: 6 byte character set flag.
// 1-2 = character set client
// 3-4 = collation client
// 5-6 = collation server
clientCharset = buffer.getUint16();
clientCollation = buffer.getUint16();
serverCollation = buffer.getUint16();
break;
case Q_TIME_ZONE_CODE:
timezone = buffer.getString();
break;
case Q_CATALOG_CODE: /* for 5.0.x where 0<=x<=3 masters */
final int len = buffer.getUint8();
catalog = buffer.getFixString(len + 1);
break;
case Q_LC_TIME_NAMES_CODE:
// lc_time_names_number = buffer.getUint16();
buffer.forward(2);
break;
case Q_CHARSET_DATABASE_CODE:
// charset_database_number = buffer.getUint16();
buffer.forward(2);
break;
case Q_TABLE_MAP_FOR_UPDATE_CODE:
// table_map_for_update = buffer.getUlong64();
buffer.forward(8);
break;
case Q_MASTER_DATA_WRITTEN_CODE:
// data_written = master_data_written =
// buffer.getUint32();
buffer.forward(4);
break;
case Q_INVOKER:
user = buffer.getString();
host = buffer.getString();
break;
case Q_MICROSECONDS:
// when.tv_usec= uint3korr(pos);
buffer.forward(3);
break;
case Q_UPDATED_DB_NAMES:
int mtsAccessedDbs = buffer.getUint8();
/**
* Notice, the following check is positive also in case
* of the master's MAX_DBS_IN_EVENT_MTS > the slave's
* one and the event contains e.g the master's
* MAX_DBS_IN_EVENT_MTS db:s.
*/
if (mtsAccessedDbs > MAX_DBS_IN_EVENT_MTS) {
mtsAccessedDbs = OVER_MAX_DBS_IN_EVENT_MTS;
break;
}
String mtsAccessedDbNames[] = new String[mtsAccessedDbs];
for (int i = 0; i < mtsAccessedDbs && buffer.position() < end; i++) {
int length = end - buffer.position();
mtsAccessedDbNames[i] = buffer.getFixString(length < NAME_LEN ? length : NAME_LEN);
}
break;
case Q_HRNOW:
// int when_sec_part = buffer.getUint24();
buffer.forward(3);
break;
default:
/*
* That's why you must write status vars in growing
* order of code
*/
if (logger.isDebugEnabled()) logger.debug("Query_log_event has unknown status vars (first has code: "
+ code + "), skipping the rest of them");
break; // Break loop
}
}
}
catch (RuntimeException e)
{
throw new IOException("Read " + findCodeName(code) + " error: "
+ e.getMessage(), e);
} catch (RuntimeException e) {
throw new IOException("Read " + findCodeName(code) + " error: " + e.getMessage(), e);
}
}
private static final String findCodeName(final int code)
{
switch (code)
{
case Q_FLAGS2_CODE:
return "Q_FLAGS2_CODE";
case Q_SQL_MODE_CODE:
return "Q_SQL_MODE_CODE";
case Q_CATALOG_CODE:
return "Q_CATALOG_CODE";
case Q_AUTO_INCREMENT:
return "Q_AUTO_INCREMENT";
case Q_CHARSET_CODE:
return "Q_CHARSET_CODE";
case Q_TIME_ZONE_CODE:
return "Q_TIME_ZONE_CODE";
case Q_CATALOG_NZ_CODE:
return "Q_CATALOG_NZ_CODE";
case Q_LC_TIME_NAMES_CODE:
return "Q_LC_TIME_NAMES_CODE";
case Q_CHARSET_DATABASE_CODE:
return "Q_CHARSET_DATABASE_CODE";
case Q_TABLE_MAP_FOR_UPDATE_CODE:
return "Q_TABLE_MAP_FOR_UPDATE_CODE";
case Q_MASTER_DATA_WRITTEN_CODE:
return "Q_MASTER_DATA_WRITTEN_CODE";
case Q_UPDATED_DB_NAMES:
return "Q_UPDATED_DB_NAMES";
case Q_MICROSECONDS:
return "Q_MICROSECONDS";
private static final String findCodeName(final int code) {
switch (code) {
case Q_FLAGS2_CODE:
return "Q_FLAGS2_CODE";
case Q_SQL_MODE_CODE:
return "Q_SQL_MODE_CODE";
case Q_CATALOG_CODE:
return "Q_CATALOG_CODE";
case Q_AUTO_INCREMENT:
return "Q_AUTO_INCREMENT";
case Q_CHARSET_CODE:
return "Q_CHARSET_CODE";
case Q_TIME_ZONE_CODE:
return "Q_TIME_ZONE_CODE";
case Q_CATALOG_NZ_CODE:
return "Q_CATALOG_NZ_CODE";
case Q_LC_TIME_NAMES_CODE:
return "Q_LC_TIME_NAMES_CODE";
case Q_CHARSET_DATABASE_CODE:
return "Q_CHARSET_DATABASE_CODE";
case Q_TABLE_MAP_FOR_UPDATE_CODE:
return "Q_TABLE_MAP_FOR_UPDATE_CODE";
case Q_MASTER_DATA_WRITTEN_CODE:
return "Q_MASTER_DATA_WRITTEN_CODE";
case Q_UPDATED_DB_NAMES:
return "Q_UPDATED_DB_NAMES";
case Q_MICROSECONDS:
return "Q_MICROSECONDS";
}
return "CODE#" + code;
}
public final String getUser()
{
public final String getUser() {
return user;
}
public final String getHost()
{
public final String getHost() {
return host;
}
public final String getQuery()
{
public final String getQuery() {
return query;
}
public final String getCatalog()
{
public final String getCatalog() {
return catalog;
}
public final String getDbName()
{
public final String getDbName() {
return dbname;
}
/**
* The number of seconds the query took to run on the master.
*/
public final long getExecTime()
{
public final long getExecTime() {
return execTime;
}
public final int getErrorCode()
{
public final int getErrorCode() {
return errorCode;
}
public final long getSessionId()
{
public final long getSessionId() {
return sessionId;
}
public final long getAutoIncrementIncrement()
{
public final long getAutoIncrementIncrement() {
return autoIncrementIncrement;
}
public final long getAutoIncrementOffset()
{
public final long getAutoIncrementOffset() {
return autoIncrementOffset;
}
public final String getCharsetName()
{
public final String getCharsetName() {
return charsetName;
}
public final String getTimezone()
{
public final String getTimezone() {
return timezone;
}
@@ -784,8 +739,7 @@ public class QueryLogEvent extends LogEvent
*
* @return Returns the charsetID.
*/
public final int getClientCharset()
{
public final int getClientCharset() {
return clientCharset;
}
@@ -794,8 +748,7 @@ public class QueryLogEvent extends LogEvent
*
* @return Returns the clientCollationId.
*/
public final int getClientCollation()
{
public final int getClientCollation() {
return clientCollation;
}
@@ -804,19 +757,16 @@ public class QueryLogEvent extends LogEvent
*
* @return Returns the serverCollationId.
*/
public final int getServerCollation()
{
public final int getServerCollation() {
return serverCollation;
}
/**
* Returns the sql_mode value.
*
* <p>
* The sql_mode variable. See the section "SQL Modes" in the MySQL manual,
* and see mysql_priv.h for a list of the possible flags. Currently
* (2007-10-04), the following flags are available:
*
* <ul>
* <li>MODE_REAL_AS_FLOAT==0x1</li>
* <li>MODE_PIPES_AS_CONCAT==0x2</li>
@@ -851,14 +801,12 @@ public class QueryLogEvent extends LogEvent
* <li>MODE_NO_ENGINE_SUBSTITUTION=0x40000000</li>
* <li>MODE_PAD_CHAR_TO_FULL_LENGTH==0x80000000</li>
* </ul>
*
* All these flags are replicated from the server. However, all flags except
* MODE_NO_DIR_IN_CREATE are honored by the slave; the slave always
* preserves its old value of MODE_NO_DIR_IN_CREATE. This field is always
* written to the binlog.
*/
public final long getSqlMode()
{
public final long getSqlMode() {
return sql_mode;
}
@@ -880,40 +828,35 @@ public class QueryLogEvent extends LogEvent
* Manual. This field is always written to the binlog in version >= 5.0, and
* never written in version < 5.0.
*/
public final long getFlags2()
{
public final long getFlags2() {
return flags2;
}
/**
* Returns the OPTION_AUTO_IS_NULL flag.
*/
public final boolean isAutoIsNull()
{
public final boolean isAutoIsNull() {
return ((flags2 & OPTION_AUTO_IS_NULL) == OPTION_AUTO_IS_NULL);
}
/**
* Returns the OPTION_NO_FOREIGN_KEY_CHECKS flag.
*/
public final boolean isForeignKeyChecks()
{
public final boolean isForeignKeyChecks() {
return ((flags2 & OPTION_NO_FOREIGN_KEY_CHECKS) != OPTION_NO_FOREIGN_KEY_CHECKS);
}
/**
* Returns the OPTION_NOT_AUTOCOMMIT flag.
*/
public final boolean isAutocommit()
{
public final boolean isAutocommit() {
return ((flags2 & OPTION_NOT_AUTOCOMMIT) != OPTION_NOT_AUTOCOMMIT);
}
/**
* Returns the OPTION_NO_FOREIGN_KEY_CHECKS flag.
*/
public final boolean isUniqueChecks()
{
public final boolean isUniqueChecks() {
return ((flags2 & OPTION_RELAXED_UNIQUE_CHECKS) != OPTION_RELAXED_UNIQUE_CHECKS);
}
@@ -7,30 +7,21 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* Logs random seed used by the next RAND(), and by PASSWORD() in 4.1.0. 4.1.1
* does not need it (it's repeatable again) so this event needn't be written in
* 4.1.1 for PASSWORD() (but the fact that it is written is just a waste, it
* does not cause bugs).
*
* The state of the random number generation consists of 128 bits, which are
* stored internally as two 64-bit numbers.
*
* Binary Format
*
* does not cause bugs). The state of the random number generation consists of
* 128 bits, which are stored internally as two 64-bit numbers. Binary Format
* The Post-Header for this event type is empty. The Body has two components:
*
* <table>
* <caption>Body for Rand_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>seed1</td>
* <td>8 byte unsigned integer</td>
* <td>64 bit random seed1.</td>
* </tr>
*
* <tr>
* <td>seed2</td>
* <td>8 byte unsigned integer</td>
@@ -41,19 +32,16 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class RandLogEvent extends LogEvent
{
public final class RandLogEvent extends LogEvent {
/**
* Fixed data part: Empty
*
* <p>
* Variable data part:
*
* <ul>
* <li>8 bytes. The value for the first seed.</li>
* <li>8 bytes. The value for the second seed.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private final long seed1;
@@ -63,21 +51,17 @@ public final class RandLogEvent extends LogEvent
public static final int RAND_SEED1_OFFSET = 0;
public static final int RAND_SEED2_OFFSET = 8;
public RandLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public RandLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
/* The Post-Header is empty. The Variable Data part begins immediately. */
buffer.position(descriptionEvent.commonHeaderLen
+ descriptionEvent.postHeaderLen[RAND_EVENT - 1]
+ RAND_SEED1_OFFSET);
buffer.position(descriptionEvent.commonHeaderLen + descriptionEvent.postHeaderLen[RAND_EVENT - 1]
+ RAND_SEED1_OFFSET);
seed1 = buffer.getLong64(); // !uint8korr(buf+RAND_SEED1_OFFSET);
seed2 = buffer.getLong64(); // !uint8korr(buf+RAND_SEED2_OFFSET);
}
public final String getQuery()
{
public final String getQuery() {
return "SET SESSION rand_seed1 = " + seed1 + " , rand_seed2 = " + seed2;
}
}
@@ -4,57 +4,44 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* This will be deprecated when we move to using sequence ids.
*
* Binary Format
*
* The Post-Header has one component:
*
* This will be deprecated when we move to using sequence ids. Binary Format The
* Post-Header has one component:
* <table>
* <caption>Post-Header for Rotate_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>position</td>
* <td>8 byte integer</td>
* <td>The position within the binlog to rotate to.</td>
* </tr>
*
* </table>
*
* The Body has one component:
*
* <table>
* <caption>Body for Rotate_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>new_log</td>
* <td>variable length string without trailing zero, extending to the end of the
* event (determined by the length field of the Common-Header)</td>
* <td>Name of the binlog to rotate to.</td>
* </tr>
*
* </table>
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class RotateLogEvent extends LogEvent
{
public final class RotateLogEvent extends LogEvent {
/**
* Fixed data part:
*
* <ul>
* <li>8 bytes. The position of the first event in the next log file. Always
* contains the number 4 (meaning the next event starts at position 4 in the
@@ -62,14 +49,11 @@ public final class RotateLogEvent extends LogEvent
* is assumed to be 4.</li>
* </ul>
* <p>
*
* Variable data part:
*
* <ul>
* <li>The name of the next binary log. The filename is not null-terminated.
* Its length is the event size minus the size of the fixed parts.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private final String filename;
@@ -91,22 +75,22 @@ public final class RotateLogEvent extends LogEvent
*
* @throws MySQLExtractException
*/
public RotateLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public RotateLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int headerSize = descriptionEvent.commonHeaderLen;
final int postHeaderLen = descriptionEvent.postHeaderLen[ROTATE_EVENT - 1];
buffer.position(headerSize + R_POS_OFFSET);
position = (postHeaderLen != 0) ? buffer.getLong64() : 4; // !uint8korr(buf + R_POS_OFFSET)
position = (postHeaderLen != 0) ? buffer.getLong64() : 4; // !uint8korr(buf
// +
// R_POS_OFFSET)
final int filenameOffset = headerSize + postHeaderLen;
int filenameLen = buffer.limit() - filenameOffset;
if (filenameLen > FN_REFLEN - 1)
filenameLen = FN_REFLEN - 1;
if (filenameLen > FN_REFLEN - 1) filenameLen = FN_REFLEN - 1;
buffer.position(filenameOffset);
filename = buffer.getFixString(filenameLen);
}
@@ -114,8 +98,7 @@ public final class RotateLogEvent extends LogEvent
* Creates a new <code>Rotate_log_event</code> without log information. This
* is used to generate missing log rotation events.
*/
public RotateLogEvent(String filename)
{
public RotateLogEvent(String filename){
super(ROTATE_HEADER);
this.filename = filename;
@@ -125,21 +108,18 @@ public final class RotateLogEvent extends LogEvent
/**
* Creates a new <code>Rotate_log_event</code> without log information.
*/
public RotateLogEvent(String filename, final long position)
{
public RotateLogEvent(String filename, final long position){
super(ROTATE_HEADER);
this.filename = filename;
this.position = position;
}
public final String getFilename()
{
public final String getFilename() {
return filename;
}
public final long getPosition()
{
public final long getPosition() {
return position;
}
}
@@ -8,6 +8,8 @@ import java.util.BitSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.taobao.tddl.dbsync.binlog.JsonConversion;
import com.taobao.tddl.dbsync.binlog.JsonConversion.Json_Value;
import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
@@ -24,6 +26,8 @@ public final class RowsLogBuffer {
public static final long DATETIMEF_INT_OFS = 0x8000000000L;
public static final long TIMEF_INT_OFS = 0x800000L;
public static final long TIMEF_OFS = 0x800000000000L;
private final LogBuffer buffer;
private final int columnLen;
private final String charsetName;
@@ -162,10 +166,13 @@ public final class RowsLogBuffer {
case LogEvent.MYSQL_TYPE_TIMESTAMP:
case LogEvent.MYSQL_TYPE_DATETIME:
case LogEvent.MYSQL_TYPE_TIMESTAMP2:
case LogEvent.MYSQL_TYPE_DATETIME2:
javaType = Types.TIMESTAMP;
break;
case LogEvent.MYSQL_TYPE_TIME:
case LogEvent.MYSQL_TYPE_TIME2:
javaType = Types.TIME;
break;
@@ -437,14 +444,23 @@ public final class RowsLogBuffer {
break;
}
String second = null;
if (tv_sec == 0) {
value = "0000-00-00 00:00:00";
second = "0000-00-00 00:00:00";
} else {
Timestamp time = new Timestamp(tv_sec * 1000);
time.setNanos(tv_usec * 1000);
String v = time.toString();
value = v.substring(0, v.length() - 2);
second = time.toString();
second = second.substring(0, second.length() - 2);// 去掉毫秒精度.0
}
if (meta >= 1) {
String microSecond = usecondsToStr(tv_usec, meta);
microSecond = microSecond.substring(0, meta);
value = second + '.' + microSecond;
} else {
value = second;
}
javaType = Types.TIMESTAMP;
length = 4 + (meta + 1) / 2;
break;
@@ -487,7 +503,6 @@ public final class RowsLogBuffer {
* .YYdddddh.hhhhmmmm.mmssssss.ffffffff.ffffffff.ffffffff
*/
long intpart = buffer.getBeUlong40() - DATETIMEF_INT_OFS; // big-endian
@SuppressWarnings("unused")
int frac = 0;
switch (meta) {
case 0:
@@ -510,8 +525,9 @@ public final class RowsLogBuffer {
break;
}
String second = null;
if (intpart == 0) {
value = "0000-00-00 00:00:00";
second = "0000-00-00 00:00:00";
} else {
// 构造TimeStamp只处理到秒
long ymd = intpart >> 17;
@@ -524,7 +540,7 @@ public final class RowsLogBuffer {
// % (1 << 5)), (int) (hms >> 12),
// (int) ((hms >> 6) % (1 << 6)), (int) (hms % (1 << 6)));
// value = new Timestamp(cal.getTimeInMillis());
value = String.format("%04d-%02d-%02d %02d:%02d:%02d",
second = String.format("%04d-%02d-%02d %02d:%02d:%02d",
(int) (ym / 13),
(int) (ym % 13),
(int) (ymd % (1 << 5)),
@@ -532,6 +548,15 @@ public final class RowsLogBuffer {
(int) ((hms >> 6) % (1 << 6)),
(int) (hms % (1 << 6)));
}
if (meta >= 1) {
String microSecond = usecondsToStr(frac, meta);
microSecond = microSecond.substring(0, meta);
value = second + '.' + microSecond;
} else {
value = second;
}
javaType = Types.TIMESTAMP;
length = 5 + (meta + 1) / 2;
break;
@@ -603,7 +628,8 @@ public final class RowsLogBuffer {
frac -= 0x100; /* -(0x100 - frac) */
// fraclong = frac * 10000;
}
ltime = intpart << 24 + frac * 10000;
frac = frac * 10000;
ltime = intpart << 24;
break;
case 3:
case 4:
@@ -619,12 +645,14 @@ public final class RowsLogBuffer {
frac -= 0x10000; /* -(0x10000-frac) */
// fraclong = frac * 100;
}
ltime = intpart << 24 + frac * 100;
frac = frac * 100;
ltime = intpart << 24;
break;
case 5:
case 6:
intpart = buffer.getBeUlong48() - TIMEF_INT_OFS;
intpart = buffer.getBeUlong48() - TIMEF_OFS;
ltime = intpart;
frac = (int) (intpart % (1L << 24));
break;
default:
intpart = buffer.getBeUint24() - TIMEF_INT_OFS;
@@ -632,8 +660,9 @@ public final class RowsLogBuffer {
break;
}
String second = null;
if (intpart == 0) {
value = "00:00:00";
second = "00:00:00";
} else {
// 目前只记录秒,不处理us frac
// if (cal == null) cal = Calendar.getInstance();
@@ -644,13 +673,21 @@ public final class RowsLogBuffer {
// value = new Time(cal.getTimeInMillis());
long ultime = Math.abs(ltime);
intpart = ultime >> 24;
value = String.format("%s%02d:%02d:%02d",
second = String.format("%s%02d:%02d:%02d",
ltime >= 0 ? "" : "-",
(int) ((intpart >> 12) % (1 << 10)),
(int) ((intpart >> 6) % (1 << 6)),
(int) (intpart % (1 << 6)));
}
if (meta >= 1) {
String microSecond = usecondsToStr(Math.abs(frac), meta);
microSecond = microSecond.substring(0, meta);
value = second + '.' + microSecond;
} else {
value = second;
}
javaType = Types.TIME;
length = 3 + (meta + 1) / 2;
break;
@@ -911,6 +948,22 @@ public final class RowsLogBuffer {
length = len;
break;
}
case LogEvent.MYSQL_TYPE_JSON: {
len = buffer.getUint16();
buffer.forward(meta - 2);
int position = buffer.position();
Json_Value jsonValue = JsonConversion.parse_value(buffer.getUint8(), buffer, len - 1);
StringBuilder builder = new StringBuilder();
jsonValue.toJsonString(builder);
value = builder.toString();
buffer.position(position + len);
// byte[] binary = new byte[len];
// buffer.fillBytes(binary, 0, len);
// value = binary;
javaType = Types.VARCHAR;
length = len;
break;
}
case LogEvent.MYSQL_TYPE_GEOMETRY: {
/*
* MYSQL_TYPE_GEOMETRY: copy from BLOB or TEXT
@@ -973,4 +1026,24 @@ public final class RowsLogBuffer {
public final int getLength() {
return length;
}
private String usecondsToStr(int frac, int meta) {
String sec = String.valueOf(frac);
if (meta > 6) {
throw new IllegalArgumentException("unknow useconds meta : " + meta);
}
if (sec.length() < 6) {
StringBuilder result = new StringBuilder(6);
int len = 6 - sec.length();
for (; len > 0; len--) {
result.append('0');
}
result.append(sec);
sec = result.toString();
}
return sec.substring(0, meta);
}
}
@@ -12,41 +12,33 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public abstract class RowsLogEvent extends LogEvent
{
public abstract class RowsLogEvent extends LogEvent {
/**
* Fixed data part:
*
* <ul>
* <li>6 bytes. The table ID.</li>
* <li>2 bytes. Reserved for future use.</li>
* </ul>
*
* <p>
* Variable data part:
*
* <ul>
* <li>Packed integer. The number of columns in the table.</li>
*
* <li>Variable-sized. Bit-field indicating whether each column is used, one
* bit per column. For this field, the amount of storage required for N
* columns is INT((N+7)/8) bytes.</li>
*
* <li>Variable-sized (for UPDATE_ROWS_LOG_EVENT only). Bit-field indicating
* whether each column is used in the UPDATE_ROWS_LOG_EVENT after-image; one
* bit per column. For this field, the amount of storage required for N
* columns is INT((N+7)/8) bytes.</li>
*
* <li>Variable-sized. A sequence of zero or more rows. The end is
* determined by the size of the event. Each row has the following format:
*
* <ul>
* <li>Variable-sized. Bit-field indicating whether each field in the row is
* NULL. Only columns that are "used" according to the second field in the
* variable data part are listed here. If the second field in the variable
* data part has N one-bits, the amount of storage required for this field
* is INT((N+7)/8) bytes.</li>
*
* <li>Variable-sized. The row-image, containing values of all table fields.
* This only lists table fields that are used (according to the second field
* of the variable data part) and non-NULL (according to the previous
@@ -54,16 +46,18 @@ public abstract class RowsLogEvent extends LogEvent
* number of zero bits in the previous field (not counting padding bits in
* the last byte). The format of each value is described in the
* log_event_print_value() function in log_event.cc.</li>
*
* <li>(for UPDATE_ROWS_EVENT only) the previous two fields are repeated,
* representing a second table row.</li>
* </ul>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private final long tableId; /* Table ID */
private TableMapLogEvent table; /* The table the rows belong to */
private TableMapLogEvent table; /*
* The table
* the rows
* belong to
*/
/** Bitmap denoting columns available */
protected final int columnLen;
@@ -78,15 +72,17 @@ public abstract class RowsLogEvent extends LogEvent
protected final BitSet changeColumns;
/** XXX: Don't handle buffer in another thread. */
private final LogBuffer rowsBuf; /* The rows in packed format */
private final LogBuffer rowsBuf; /*
* The rows in
* packed
* format
*/
/**
* enum enum_flag
*
* These definitions allow you to combine the flags into an appropriate flag
* set using the normal bitwise operators. The implicit conversion from an
* enum-constant to an integer is accepted by the compiler, which is then
* used to set the real set of flags.
* enum enum_flag These definitions allow you to combine the flags into an
* appropriate flag set using the normal bitwise operators. The implicit
* conversion from an enum-constant to an integer is accepted by the
* compiler, which is then used to set the real set of flags.
*/
private final int flags;
@@ -112,43 +108,40 @@ public abstract class RowsLogEvent extends LogEvent
public static final int RW_V_TAG_LEN = 1;
public static final int RW_V_EXTRAINFO_TAG = 0;
public RowsLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public RowsLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
final int postHeaderLen = descriptionEvent.postHeaderLen[header.type - 1];
int headerLen = 0;
buffer.position(commonHeaderLen + RW_MAPID_OFFSET);
if (postHeaderLen == 6)
{
/* Master is of an intermediate source tree before 5.1.4. Id is 4 bytes */
if (postHeaderLen == 6) {
/*
* Master is of an intermediate source tree before 5.1.4. Id is 4
* bytes
*/
tableId = buffer.getUint32();
}
else
{
} else {
tableId = buffer.getUlong48(); // RW_FLAGS_OFFSET
}
flags = buffer.getUint16();
if (postHeaderLen == FormatDescriptionLogEvent.ROWS_HEADER_LEN_V2)
{
if (postHeaderLen == FormatDescriptionLogEvent.ROWS_HEADER_LEN_V2) {
headerLen = buffer.getUint16();
headerLen -= 2;
int start = buffer.position();
int end = start + headerLen;
for(int i = start ;i < end; ){
for (int i = start; i < end;) {
switch (buffer.getUint8(i++)) {
case RW_V_EXTRAINFO_TAG:
// int infoLen = buffer.getUint8();
buffer.position(i + EXTRA_ROW_INFO_LEN_OFFSET);
int checkLen = buffer.getUint8(); // EXTRA_ROW_INFO_LEN_OFFSET
int val= checkLen - EXTRA_ROW_INFO_HDR_BYTES;
assert(buffer.getUint8() == val); //EXTRA_ROW_INFO_FORMAT_OFFSET
for (int j= 0; j < val; j++) {
assert(buffer.getUint8() == val); // EXTRA_ROW_INFO_HDR_BYTES + i
int val = checkLen - EXTRA_ROW_INFO_HDR_BYTES;
assert (buffer.getUint8() == val); // EXTRA_ROW_INFO_FORMAT_OFFSET
for (int j = 0; j < val; j++) {
assert (buffer.getUint8() == val); // EXTRA_ROW_INFO_HDR_BYTES
// + i
}
break;
default:
@@ -157,17 +150,14 @@ public abstract class RowsLogEvent extends LogEvent
}
}
}
buffer.position(commonHeaderLen + postHeaderLen + headerLen);
columnLen = (int) buffer.getPackedLong();
columns = buffer.getBitmap(columnLen);
if (header.type == UPDATE_ROWS_EVENT_V1 || header.type == UPDATE_ROWS_EVENT)
{
if (header.type == UPDATE_ROWS_EVENT_V1 || header.type == UPDATE_ROWS_EVENT) {
changeColumns = buffer.getBitmap(columnLen);
}
else
{
} else {
changeColumns = columns;
}
@@ -176,46 +166,38 @@ public abstract class RowsLogEvent extends LogEvent
rowsBuf = buffer.duplicate(dataSize);
}
public final void fillTable(LogContext context)
{
public final void fillTable(LogContext context) {
table = context.getTable(tableId);
// end of statement check:
if ((flags & RowsLogEvent.STMT_END_F) != 0)
{
if ((flags & RowsLogEvent.STMT_END_F) != 0) {
// Now is safe to clear ignored map (clear_tables will also
// delete original table map events stored in the map).
context.clearAllTables();
}
}
public final long getTableId()
{
public final long getTableId() {
return tableId;
}
public final TableMapLogEvent getTable()
{
public final TableMapLogEvent getTable() {
return table;
}
public final BitSet getColumns()
{
public final BitSet getColumns() {
return columns;
}
public final BitSet getChangeColumns()
{
public final BitSet getChangeColumns() {
return changeColumns;
}
public final RowsLogBuffer getRowsBuf(String charsetName)
{
public final RowsLogBuffer getRowsBuf(String charsetName) {
return new RowsLogBuffer(rowsBuf.duplicate(), columnLen, charsetName);
}
public final int getFlags(final int flags)
{
public final int getFlags(final int flags) {
return this.flags & flags;
}
}
@@ -5,21 +5,18 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* Start_log_event_v3 is the Start_log_event of binlog format 3 (MySQL 3.23 and
* 4.x).
*
* Format_description_log_event derives from Start_log_event_v3; it is the
* 4.x). Format_description_log_event derives from Start_log_event_v3; it is the
* Start_log_event of binlog format 4 (MySQL 5.0), that is, the event that
* describes the other events' Common-Header/Post-Header lengths. This event is
* sent by MySQL 5.0 whenever it starts sending a new binlog if the requested
* position is >4 (otherwise if ==4 the event will be sent naturally).
*
* @see mysql-5.1.60/sql/log_event.cc - Start_log_event_v3
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public class StartLogEventV3 extends LogEvent
{
public class StartLogEventV3 extends LogEvent {
/**
* We could have used SERVER_VERSION_LENGTH, but this introduces an obscure
* dependency - if somebody decided to change SERVER_VERSION_LENGTH this
@@ -34,9 +31,7 @@ public class StartLogEventV3 extends LogEvent
protected int binlogVersion;
protected String serverVersion;
public StartLogEventV3(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public StartLogEventV3(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
buffer.position(descriptionEvent.commonHeaderLen);
@@ -44,18 +39,15 @@ public class StartLogEventV3 extends LogEvent
serverVersion = buffer.getFixString(ST_SERVER_VER_LEN); // ST_SERVER_VER_OFFSET
}
public StartLogEventV3()
{
public StartLogEventV3(){
super(new LogHeader(START_EVENT_V3));
}
public final String getServerVersion()
{
public final String getServerVersion() {
return serverVersion;
}
public final int getBinlogVersion()
{
public final int getBinlogVersion() {
return binlogVersion;
}
}
@@ -4,19 +4,15 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* Stop_log_event.
*
* The Post-Header and Body for this event type are empty; it only has the
* Common-Header.
* Stop_log_event. The Post-Header and Body for this event type are empty; it
* only has the Common-Header.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class StopLogEvent extends LogEvent
{
public StopLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent description_event)
{
public final class StopLogEvent extends LogEvent {
public StopLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent description_event){
super(header);
}
}
@@ -8,44 +8,34 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* In row-based mode, every row operation event is preceded by a
* Table_map_log_event which maps a table definition to a number. The table
* definition consists of database name, table name, and column definitions.
*
* The Post-Header has the following components:
*
* definition consists of database name, table name, and column definitions. The
* Post-Header has the following components:
* <table>
* <caption>Post-Header for Table_map_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>table_id</td>
* <td>6 bytes unsigned integer</td>
* <td>The number that identifies the table.</td>
* </tr>
*
* <tr>
* <td>flags</td>
* <td>2 byte bitfield</td>
* <td>Reserved for future use; currently always 0.</td>
* </tr>
*
* </table>
*
* The Body has the following components:
*
* <table>
* <caption>Body for Table_map_log_event</caption>
*
* <tr>
* <th>Name</th>
* <th>Format</th>
* <th>Description</th>
* </tr>
*
* <tr>
* <td>database_name</td>
* <td>one byte string length, followed by null-terminated string</td>
@@ -55,20 +45,17 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* by a terminating 0 byte. (Note the redundancy in the representation of the
* length.)</td>
* </tr>
*
* <tr>
* <td>table_name</td>
* <td>one byte string length, followed by null-terminated string</td>
* <td>The name of the table, encoded the same way as the database name above.</td>
* </tr>
*
* <tr>
* <td>column_count</td>
* <td>packed_integer "Packed Integer"</td>
* <td>The number of columns in the table, represented as a packed
* variable-length integer.</td>
* </tr>
*
* <tr>
* <td>column_type</td>
* <td>List of column_count 1 byte enumeration values</td>
@@ -78,13 +65,11 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* listed in the table Table_table_map_log_event_column_types "below" (along
* with description of the associated metadata field).</td>
* </tr>
*
* <tr>
* <td>metadata_length</td>
* <td>packed_integer "Packed Integer"</td>
* <td>The length of the following metadata block</td>
* </tr>
*
* <tr>
* <td>metadata</td>
* <td>list of metadata for each column</td>
@@ -93,7 +78,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* metadata for each column are listed in the table
* Table_table_map_log_event_column_types "below".</td>
* </tr>
*
* <tr>
* <td>null_bits</td>
* <td>column_count bits, rounded up to nearest byte</td>
@@ -104,13 +88,10 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* first byte, the ninth is in the least significant bit of the second byte, and
* so on.</td>
* </tr>
*
* </table>
*
* The table below lists all column types, along with the numerical identifier
* for it and the size and interpretation of meta-data used to describe the
* type.
*
* <table>
* <caption>Table_map_log_event column types: numerical identifier and
* metadata</caption>
@@ -120,35 +101,30 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <th>Size of metadata in bytes</th>
* <th>Description of metadata</th>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_DECIMAL</td>
* <td>0</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_TINY</td>
* <td>1</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_SHORT</td>
* <td>2</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_LONG</td>
* <td>3</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_FLOAT</td>
* <td>4</td>
@@ -156,7 +132,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>1 byte unsigned integer, representing the "pack_length", which is equal
* to sizeof(float) on the server from which the event originates.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_DOUBLE</td>
* <td>5</td>
@@ -164,63 +139,54 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>1 byte unsigned integer, representing the "pack_length", which is equal
* to sizeof(double) on the server from which the event originates.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_NULL</td>
* <td>6</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_TIMESTAMP</td>
* <td>7</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_LONGLONG</td>
* <td>8</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_INT24</td>
* <td>9</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_DATE</td>
* <td>10</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_TIME</td>
* <td>11</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_DATETIME</td>
* <td>12</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_YEAR</td>
* <td>13</td>
* <td>0</td>
* <td>No column metadata.</td>
* </tr>
*
* <tr>
* <td><i>MYSQL_TYPE_NEWDATE</i></td>
* <td><i>14</i></td>
@@ -228,14 +194,12 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_VARCHAR</td>
* <td>15</td>
* <td>2 bytes</td>
* <td>2 byte unsigned integer representing the maximum length of the string.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_BIT</td>
* <td>16</td>
@@ -245,7 +209,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* occupied by the bitfield. The number of bytes is either int((length+7)/8) or
* int(length/8).</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_NEWDECIMAL</td>
* <td>246</td>
@@ -253,7 +216,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>A 1 byte unsigned int representing the precision, followed by a 1 byte
* unsigned int representing the number of decimals.</td>
* </tr>
*
* <tr>
* <td><i>MYSQL_TYPE_ENUM</i></td>
* <td><i>247</i></td>
@@ -261,7 +223,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td><i>MYSQL_TYPE_SET</i></td>
* <td><i>248</i></td>
@@ -269,7 +230,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_TINY_BLOB</td>
* <td>249</td>
@@ -277,7 +237,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td><i>MYSQL_TYPE_MEDIUM_BLOB</i></td>
* <td><i>250</i></td>
@@ -285,7 +244,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td><i>MYSQL_TYPE_LONG_BLOB</i></td>
* <td><i>251</i></td>
@@ -293,7 +251,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td><i>This enumeration value is only used internally and cannot exist in a
* binlog.</i></td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_BLOB</td>
* <td>252</td>
@@ -301,7 +258,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>The pack length, i.e., the number of bytes needed to represent the length
* of the blob: 1, 2, 3, or 4.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_VAR_STRING</td>
* <td>253</td>
@@ -312,7 +268,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* unsigned integer representing the field size, i.e., the number of bytes
* needed to store the length of the string.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_STRING</td>
* <td>254</td>
@@ -321,7 +276,6 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* byte is the field size, i.e., the number of bytes in the representation of
* size of the string: 3 or 4.</td>
* </tr>
*
* <tr>
* <td>MYSQL_TYPE_GEOMETRY</td>
* <td>255</td>
@@ -329,21 +283,19 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* <td>The pack length, i.e., the number of bytes needed to represent the length
* of the geometry: 1, 2, 3, or 4.</td>
* </tr>
*
* </table>
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class TableMapLogEvent extends LogEvent
{
public final class TableMapLogEvent extends LogEvent {
/**
* Fixed data part:
* <ul>
* <li>6 bytes. The table ID.</li>
* <li>2 bytes. Reserved for future use.</li>
* </ul>
*
* <p>
* Variable data part:
* <ul>
@@ -360,7 +312,6 @@ public final class TableMapLogEvent extends LogEvent
* one bit per column. For this field, the amount of storage required for N
* columns is INT((N+7)/8) bytes.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
protected final String dbname;
@@ -369,14 +320,15 @@ public final class TableMapLogEvent extends LogEvent
/**
* Holding mysql column information.
*/
public static final class ColumnInfo
{
public static final class ColumnInfo {
public int type;
public int meta;
}
protected final int columnCnt;
protected final ColumnInfo[] columnInfo; // buffer for field metadata
protected final ColumnInfo[] columnInfo; // buffer for field
// metadata
protected final long tableId;
protected BitSet nullBits;
@@ -388,22 +340,20 @@ public final class TableMapLogEvent extends LogEvent
/**
* Constructor used by slave to read the event from the binary log.
*/
public TableMapLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public TableMapLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
final int commonHeaderLen = descriptionEvent.commonHeaderLen;
final int postHeaderLen = descriptionEvent.postHeaderLen[header.type - 1];
/* Read the post-header */
buffer.position(commonHeaderLen + TM_MAPID_OFFSET);
if (postHeaderLen == 6)
{
/* Master is of an intermediate source tree before 5.1.4. Id is 4 bytes */
if (postHeaderLen == 6) {
/*
* Master is of an intermediate source tree before 5.1.4. Id is 4
* bytes
*/
tableId = buffer.getUint32();
}
else
{
} else {
// DBUG_ASSERT(post_header_len == TABLE_MAP_HEADER_LEN);
tableId = buffer.getUlong48();
}
@@ -419,15 +369,13 @@ public final class TableMapLogEvent extends LogEvent
// Read column information from buffer
columnCnt = (int) buffer.getPackedLong();
columnInfo = new ColumnInfo[columnCnt];
for (int i = 0; i < columnCnt; i++)
{
for (int i = 0; i < columnCnt; i++) {
ColumnInfo info = new ColumnInfo();
info.type = buffer.getUint8();
columnInfo[i] = info;
}
if (buffer.position() < buffer.limit())
{
if (buffer.position() < buffer.limit()) {
final int fieldSize = (int) buffer.getPackedLong();
decodeFields(buffer, fieldSize);
nullBits = buffer.getBitmap(columnCnt);
@@ -439,41 +387,38 @@ public final class TableMapLogEvent extends LogEvent
*
* @see mysql-5.1.60/sql/rpl_utility.h
*/
private final void decodeFields(LogBuffer buffer, final int len)
{
private final void decodeFields(LogBuffer buffer, final int len) {
final int limit = buffer.limit();
buffer.limit(len + buffer.position());
for (int i = 0; i < columnCnt; i++)
{
for (int i = 0; i < columnCnt; i++) {
ColumnInfo info = columnInfo[i];
switch (info.type)
{
case MYSQL_TYPE_TINY_BLOB:
case MYSQL_TYPE_BLOB:
case MYSQL_TYPE_MEDIUM_BLOB:
case MYSQL_TYPE_LONG_BLOB:
case MYSQL_TYPE_DOUBLE:
case MYSQL_TYPE_FLOAT:
case MYSQL_TYPE_GEOMETRY:
/*
These types store a single byte.
*/
info.meta = buffer.getUint8();
break;
case MYSQL_TYPE_SET:
case MYSQL_TYPE_ENUM:
/*
* log_event.h : MYSQL_TYPE_SET & MYSQL_TYPE_ENUM : This
* enumeration value is only used internally and cannot
* exist in a binlog.
*/
logger.warn("This enumeration value is only used internally "
+ "and cannot exist in a binlog: type=" + info.type);
break;
case MYSQL_TYPE_STRING:
{
switch (info.type) {
case MYSQL_TYPE_TINY_BLOB:
case MYSQL_TYPE_BLOB:
case MYSQL_TYPE_MEDIUM_BLOB:
case MYSQL_TYPE_LONG_BLOB:
case MYSQL_TYPE_DOUBLE:
case MYSQL_TYPE_FLOAT:
case MYSQL_TYPE_GEOMETRY:
case MYSQL_TYPE_JSON:
/*
* These types store a single byte.
*/
info.meta = buffer.getUint8();
break;
case MYSQL_TYPE_SET:
case MYSQL_TYPE_ENUM:
/*
* log_event.h : MYSQL_TYPE_SET & MYSQL_TYPE_ENUM : This
* enumeration value is only used internally and cannot
* exist in a binlog.
*/
logger.warn("This enumeration value is only used internally "
+ "and cannot exist in a binlog: type=" + info.type);
break;
case MYSQL_TYPE_STRING: {
/*
* log_event.h : The first byte is always
* MYSQL_TYPE_VAR_STRING (i.e., 253). The second byte is the
@@ -485,59 +430,52 @@ public final class TableMapLogEvent extends LogEvent
info.meta = x;
break;
}
case MYSQL_TYPE_BIT:
info.meta = buffer.getUint16();
break;
case MYSQL_TYPE_VARCHAR:
/*
* These types store two bytes.
*/
info.meta = buffer.getUint16();
break;
case MYSQL_TYPE_NEWDECIMAL:
{
case MYSQL_TYPE_BIT:
info.meta = buffer.getUint16();
break;
case MYSQL_TYPE_VARCHAR:
/*
* These types store two bytes.
*/
info.meta = buffer.getUint16();
break;
case MYSQL_TYPE_NEWDECIMAL: {
int x = buffer.getUint8() << 8; // precision
x += buffer.getUint8(); // decimals
info.meta = x;
break;
}
case MYSQL_TYPE_TIME2:
case MYSQL_TYPE_DATETIME2:
case MYSQL_TYPE_TIMESTAMP2:
{
case MYSQL_TYPE_TIME2:
case MYSQL_TYPE_DATETIME2:
case MYSQL_TYPE_TIMESTAMP2: {
info.meta = buffer.getUint8();
break;
}
default:
info.meta = 0;
break;
default:
info.meta = 0;
break;
}
}
buffer.limit(limit);
}
public final String getDbName()
{
public final String getDbName() {
return dbname;
}
public final String getTableName()
{
public final String getTableName() {
return tblname;
}
public final int getColumnCnt()
{
public final int getColumnCnt() {
return columnCnt;
}
public final ColumnInfo[] getColumnInfo()
{
public final ColumnInfo[] getColumnInfo() {
return columnInfo;
}
public final long getTableId()
{
public final long getTableId() {
return tableId;
}
}
@@ -8,10 +8,9 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class UnknownLogEvent extends LogEvent
{
public UnknownLogEvent(LogHeader header)
{
public final class UnknownLogEvent extends LogEvent {
public UnknownLogEvent(LogHeader header){
super(header);
}
}
@@ -4,19 +4,16 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
/**
* Log row updates with a before image. The event contain several update rows
* for a table. Note that each event contains only rows for one table.
*
* Also note that the row data consists of pairs of row data: one row for the
* old data and one row for the new data.
* for a table. Note that each event contains only rows for one table. Also note
* that the row data consists of pairs of row data: one row for the old data and
* one row for the new data.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class UpdateRowsLogEvent extends RowsLogEvent
{
public UpdateRowsLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public final class UpdateRowsLogEvent extends RowsLogEvent {
public UpdateRowsLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header, buffer, descriptionEvent);
}
}
@@ -8,22 +8,19 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
import com.taobao.tddl.dbsync.binlog.LogEvent;
/**
* User_var_log_event.
*
* Every time a query uses the value of a user variable, a User_var_log_event is
* written before the Query_log_event, to set the user variable.
* User_var_log_event. Every time a query uses the value of a user variable, a
* User_var_log_event is written before the Query_log_event, to set the user
* variable.
*
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class UserVarLogEvent extends LogEvent
{
public final class UserVarLogEvent extends LogEvent {
/**
* Fixed data part: Empty
*
* <p>
* Variable data part:
*
* <ul>
* <li>4 bytes. the size of the user variable name.</li>
* <li>The user variable name.</li>
@@ -39,7 +36,6 @@ public final class UserVarLogEvent extends LogEvent
* <li>Variable-sized. For a string variable, this is the string. For a
* float or integer variable, this is its value in 8 bytes.</li>
* </ul>
*
* Source : http://forge.mysql.com/wiki/MySQL_Internals_Binary_Log
*/
private final String name;
@@ -66,80 +62,65 @@ public final class UserVarLogEvent extends LogEvent
public static final int UV_NAME_LEN_SIZE = 4;
public static final int UV_CHARSET_NUMBER_SIZE = 4;
public UserVarLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent) throws IOException
{
public UserVarLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent)
throws IOException{
super(header);
/* The Post-Header is empty. The Variable Data part begins immediately. */
buffer.position(descriptionEvent.commonHeaderLen
+ descriptionEvent.postHeaderLen[USER_VAR_EVENT - 1]);
buffer.position(descriptionEvent.commonHeaderLen + descriptionEvent.postHeaderLen[USER_VAR_EVENT - 1]);
final int nameLen = (int) buffer.getUint32();
name = buffer.getFixString(nameLen); // UV_NAME_LEN_SIZE
isNull = (0 != buffer.getInt8());
if (isNull)
{
if (isNull) {
type = STRING_RESULT;
charsetNumber = 63; /* binary */
value = null;
}
else
{
} else {
type = buffer.getInt8(); // UV_VAL_IS_NULL
charsetNumber = (int) buffer.getUint32(); // buf + UV_VAL_TYPE_SIZE
final int valueLen = (int) buffer.getUint32(); // buf + UV_CHARSET_NUMBER_SIZE
final int valueLen = (int) buffer.getUint32(); // buf +
// UV_CHARSET_NUMBER_SIZE
final int limit = buffer.limit(); /* for restore */
buffer.limit(buffer.position() + valueLen);
/* @see User_var_log_event::print */
switch (type)
{
case REAL_RESULT:
value = Double.valueOf(buffer.getDouble64()); // float8get
break;
case INT_RESULT:
if (valueLen == 8)
value = Long.valueOf(buffer.getLong64()); // !uint8korr
else if (valueLen == 4)
value = Long.valueOf(buffer.getUint32());
else
throw new IOException("Error INT_RESULT length: "
+ valueLen);
break;
case DECIMAL_RESULT:
final int precision = buffer.getInt8();
final int scale = buffer.getInt8();
value = buffer.getDecimal(precision, scale); // bin2decimal
break;
case STRING_RESULT:
String charsetName = CharsetConversion.getJavaCharset(charsetNumber);
value = buffer.getFixString(valueLen, charsetName);
break;
case ROW_RESULT:
// this seems to be banned in MySQL altogether
throw new IOException("ROW_RESULT is unsupported");
default:
value = null;
break;
switch (type) {
case REAL_RESULT:
value = Double.valueOf(buffer.getDouble64()); // float8get
break;
case INT_RESULT:
if (valueLen == 8) value = Long.valueOf(buffer.getLong64()); // !uint8korr
else if (valueLen == 4) value = Long.valueOf(buffer.getUint32());
else throw new IOException("Error INT_RESULT length: " + valueLen);
break;
case DECIMAL_RESULT:
final int precision = buffer.getInt8();
final int scale = buffer.getInt8();
value = buffer.getDecimal(precision, scale); // bin2decimal
break;
case STRING_RESULT:
String charsetName = CharsetConversion.getJavaCharset(charsetNumber);
value = buffer.getFixString(valueLen, charsetName);
break;
case ROW_RESULT:
// this seems to be banned in MySQL altogether
throw new IOException("ROW_RESULT is unsupported");
default:
value = null;
break;
}
buffer.limit(limit);
}
}
public final String getQuery()
{
if (value == null)
{
public final String getQuery() {
if (value == null) {
return "SET @" + name + " := NULL";
}
else if (type == STRING_RESULT)
{
} else if (type == STRING_RESULT) {
// TODO: do escaping !?
return "SET @" + name + " := \'" + value + '\'';
}
else
{
} else {
return "SET @" + name + " := " + String.valueOf(value);
}
}
@@ -9,11 +9,9 @@ import com.taobao.tddl.dbsync.binlog.LogBuffer;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class WriteRowsLogEvent extends RowsLogEvent
{
public WriteRowsLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public final class WriteRowsLogEvent extends RowsLogEvent {
public WriteRowsLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header, buffer, descriptionEvent);
}
}
@@ -10,23 +10,19 @@ import com.taobao.tddl.dbsync.binlog.LogEvent;
* @author <a href="mailto:changyuan.lh@taobao.com">Changyuan.lh</a>
* @version 1.0
*/
public final class XidLogEvent extends LogEvent
{
public final class XidLogEvent extends LogEvent {
private final long xid;
public XidLogEvent(LogHeader header, LogBuffer buffer,
FormatDescriptionLogEvent descriptionEvent)
{
public XidLogEvent(LogHeader header, LogBuffer buffer, FormatDescriptionLogEvent descriptionEvent){
super(header);
/* The Post-Header is empty. The Variable Data part begins immediately. */
buffer.position(descriptionEvent.commonHeaderLen
+ descriptionEvent.postHeaderLen[XID_EVENT - 1]);
buffer.position(descriptionEvent.commonHeaderLen + descriptionEvent.postHeaderLen[XID_EVENT - 1]);
xid = buffer.getLong64(); // !uint8korr
}
public final long getXid()
{
public final long getXid() {
return xid;
}
}
@@ -10,8 +10,8 @@ import com.taobao.tddl.dbsync.binlog.event.RowsLogBuffer;
import com.taobao.tddl.dbsync.binlog.event.RowsLogEvent;
import com.taobao.tddl.dbsync.binlog.event.RowsQueryLogEvent;
import com.taobao.tddl.dbsync.binlog.event.TableMapLogEvent;
import com.taobao.tddl.dbsync.binlog.event.XidLogEvent;
import com.taobao.tddl.dbsync.binlog.event.TableMapLogEvent.ColumnInfo;
import com.taobao.tddl.dbsync.binlog.event.XidLogEvent;
import com.taobao.tddl.dbsync.binlog.event.mariadb.AnnotateRowsEvent;
public class BaseLogFetcherTest {
@@ -20,36 +20,38 @@ public class BaseLogFetcherTest {
protected Charset charset = Charset.forName("utf-8");
protected void parseQueryEvent(QueryLogEvent event) {
System.out.println(String.format("================> binlog[%s:%s] , name[%s]", binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen(),
event.getCatalog()));
System.out.println(String.format("================> binlog[%s:%s] , name[%s]",
binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen(),
event.getCatalog()));
System.out.println("sql : " + event.getQuery());
}
protected void parseRowsQueryEvent(RowsQueryLogEvent event) throws Exception {
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen()));
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName, event.getHeader()
.getLogPos() - event.getHeader().getEventLen()));
System.out.println("sql : " + new String(event.getRowsQuery().getBytes("ISO-8859-1"), charset.name()));
}
protected void parseAnnotateRowsEvent(AnnotateRowsEvent event) throws Exception {
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen()));
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName, event.getHeader()
.getLogPos() - event.getHeader().getEventLen()));
System.out.println("sql : " + new String(event.getRowsQuery().getBytes("ISO-8859-1"), charset.name()));
}
protected void parseXidEvent(XidLogEvent event) throws Exception {
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen()));
System.out.println(String.format("================> binlog[%s:%s]", binlogFileName, event.getHeader()
.getLogPos() - event.getHeader().getEventLen()));
System.out.println("xid : " + event.getXid());
}
protected void parseRowsEvent(RowsLogEvent event) {
try {
System.out.println(String.format("================> binlog[%s:%s] , name[%s,%s]", binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen(),
event.getTable().getDbName(), event.getTable().getTableName()));
System.out.println(String.format("================> binlog[%s:%s] , name[%s,%s]",
binlogFileName,
event.getHeader().getLogPos() - event.getHeader().getEventLen(),
event.getTable().getDbName(),
event.getTable().getTableName()));
RowsLogBuffer buffer = event.getRowsBuf(charset.name());
BitSet columns = event.getColumns();
BitSet changeColumns = event.getChangeColumns();
@@ -90,6 +92,10 @@ public class BaseLogFetcherTest {
final ColumnInfo[] columnInfo = map.getColumnInfo();
for (int i = 0; i < columnCnt; i++) {
if (!cols.get(i)) {
continue;
}
ColumnInfo info = columnInfo[i];
buffer.nextValue(info.type, info.meta);
@@ -5,8 +5,7 @@ import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Statement;
import junit.framework.Assert;
import org.junit.Assert;
import org.junit.Test;
import com.taobao.tddl.dbsync.binlog.event.DeleteRowsLogEvent;
@@ -25,12 +24,12 @@ public class DirectLogFetcherTest extends BaseLogFetcherTest {
DirectLogFetcher fecther = new DirectLogFetcher();
try {
Class.forName("com.mysql.jdbc.Driver");
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306", "xxxxx", "xxxxx");
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306", "root", "hello");
Statement statement = connection.createStatement();
statement.execute("SET @master_binlog_checksum='@@global.binlog_checksum'");
statement.execute("SET @mariadb_slave_capability='" + LogEvent.MARIA_SLAVE_CAPABILITY_MINE + "'");
fecther.open(connection, "mysql-bin.000003", 4L, 2);
fecther.open(connection, "mysql-bin.000001", 4L, 2);
LogDecoder decoder = new LogDecoder(LogEvent.UNKNOWN_EVENT, LogEvent.ENUM_END_EVENT);
LogContext context = new LogContext();
@@ -39,7 +38,8 @@ public class DirectLogFetcherTest extends BaseLogFetcherTest {
event = decoder.decode(fecther, context);
if (event == null) {
throw new RuntimeException("parse failed");
continue;
// throw new RuntimeException("parse failed");
}
int eventType = event.getHeader().getType();
@@ -4,8 +4,7 @@ import java.io.File;
import java.io.IOException;
import java.net.URL;
import junit.framework.Assert;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
@@ -27,7 +26,6 @@ public class FileLogFetcherTest extends BaseLogFetcherTest {
URL url = Thread.currentThread().getContextClassLoader().getResource("dummy.txt");
File dummyFile = new File(url.getFile());
directory = new File(dummyFile.getParent() + "/binlog").getPath();
// directory = "/home/jianghang/tmp/binlog";
}
@Test
@@ -37,8 +35,8 @@ public class FileLogFetcherTest extends BaseLogFetcherTest {
LogDecoder decoder = new LogDecoder(LogEvent.UNKNOWN_EVENT, LogEvent.ENUM_END_EVENT);
LogContext context = new LogContext();
File current = new File(directory, "mysql-bin.000006");
fetcher.open(current);
File current = new File(directory, "mysql-bin.000001");
fetcher.open(current, 2051L);
context.setLogPosition(new LogPosition(current.getName()));
while (fetcher.fetch()) {
@@ -48,23 +46,22 @@ public class FileLogFetcherTest extends BaseLogFetcherTest {
int eventType = event.getHeader().getType();
switch (eventType) {
case LogEvent.ROTATE_EVENT:
binlogFileName = ((RotateLogEvent)
event).getFilename();
binlogFileName = ((RotateLogEvent) event).getFilename();
break;
case LogEvent.WRITE_ROWS_EVENT_V1:
case LogEvent.WRITE_ROWS_EVENT:
parseRowsEvent((WriteRowsLogEvent) event);
parseRowsEvent((WriteRowsLogEvent) event);
break;
case LogEvent.UPDATE_ROWS_EVENT_V1:
case LogEvent.UPDATE_ROWS_EVENT:
parseRowsEvent((UpdateRowsLogEvent) event);
parseRowsEvent((UpdateRowsLogEvent) event);
break;
case LogEvent.DELETE_ROWS_EVENT_V1:
case LogEvent.DELETE_ROWS_EVENT:
parseRowsEvent((DeleteRowsLogEvent) event);
parseRowsEvent((DeleteRowsLogEvent) event);
break;
case LogEvent.QUERY_EVENT:
parseQueryEvent((QueryLogEvent) event);
parseQueryEvent((QueryLogEvent) event);
break;
case LogEvent.ROWS_QUERY_LOG_EVENT:
parseRowsQueryEvent((RowsQueryLogEvent) event);
@@ -5,14 +5,11 @@ import java.math.BigInteger;
import junit.framework.TestCase;
import com.taobao.tddl.dbsync.binlog.LogBuffer;
public class LogBufferTest extends TestCase {
public class LogBufferTest extends TestCase
{
public static final int LOOP = 10000;
public void testSigned()
{
public void testSigned() {
byte[] array = { 0, 0, 0, (byte) 0xff };
LogBuffer buffer = new LogBuffer(array, 0, array.length);
@@ -24,17 +21,15 @@ public class LogBufferTest extends TestCase
System.out.println(buffer.getUint24(1));
}
public void testBigInteger()
{
byte[] array = { (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff };
public void testBigInteger() {
byte[] array = { (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff, (byte) 0xff,
(byte) 0xff };
LogBuffer buffer = new LogBuffer(array, 0, array.length);
long tt1 = 0;
long l1 = 0;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t1 = System.nanoTime();
l1 = buffer.getLong64(0);
tt1 += System.nanoTime() - t1;
@@ -45,8 +40,7 @@ public class LogBufferTest extends TestCase
long tt2 = 0;
BigInteger l2 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t2 = System.nanoTime();
l2 = buffer.getUlong64(0);
tt2 += System.nanoTime() - t2;
@@ -57,18 +51,15 @@ public class LogBufferTest extends TestCase
}
/* Reads big-endian integer from no more than 4 bytes */
private static int convertNBytesToInt(byte[] buffer, int offset, int length)
{
private static int convertNBytesToInt(byte[] buffer, int offset, int length) {
int ret = 0;
for (int i = offset; i < (offset + length); i++)
{
for (int i = offset; i < (offset + length); i++) {
ret = (ret << 8) | (0xff & buffer[i]);
}
return ret;
}
private static int convert4BytesToInt(byte[] buffer, int offset)
{
private static int convert4BytesToInt(byte[] buffer, int offset) {
int value;
value = (0xff & buffer[offset + 3]);
value += (0xff & buffer[offset + 2]) << 8;
@@ -77,24 +68,20 @@ public class LogBufferTest extends TestCase
return value;
}
public static short convert1ByteToShort(byte[] buffer, int offset)
{
public static short convert1ByteToShort(byte[] buffer, int offset) {
short value;
value = (short) buffer[offset + 0];
return value;
}
public static short convert2bytesToShort(byte[] buffer, int offset)
{
public static short convert2bytesToShort(byte[] buffer, int offset) {
short value;
value = (short) (buffer[offset + 0] << 8);
value += (short) (buffer[offset + 1] & 0xff);
return value;
}
public static final BigDecimal extractDecimal(byte[] buffer, int precision,
int scale)
{
public static final BigDecimal extractDecimal(byte[] buffer, int precision, int scale) {
//
// Decimal representation in binlog seems to be as follows:
// 1 byte - 'precision'
@@ -113,7 +100,7 @@ public class LogBufferTest extends TestCase
// 0x00000005 0x1b38b060 0x00
// 5 456700000 0
// 54567000000 / 10^{10} = 5.4567
//
//
// int_size below shows how long is integer part
//
// offset = offset + 2; // offset of the number part
@@ -129,34 +116,28 @@ public class LogBufferTest extends TestCase
int sign = (buffer[offset] & 0x80) == 0x80 ? 1 : -1;
// how many bytes are used to represent given amount of digits?
int integerSize = intg0 * LogBuffer.SIZE_OF_INT32
+ LogBuffer.dig2bytes[intg0x];
int decimalSize = frac0 * LogBuffer.SIZE_OF_INT32
+ LogBuffer.dig2bytes[frac0x];
int integerSize = intg0 * LogBuffer.SIZE_OF_INT32 + LogBuffer.dig2bytes[intg0x];
int decimalSize = frac0 * LogBuffer.SIZE_OF_INT32 + LogBuffer.dig2bytes[frac0x];
int bin_size = integerSize + decimalSize; // total bytes
byte[] d_copy = new byte[bin_size];
if (bin_size > buffer.length)
{
throw new ArrayIndexOutOfBoundsException("Calculated bin_size: "
+ bin_size + ", available bytes: " + buffer.length);
if (bin_size > buffer.length) {
throw new ArrayIndexOutOfBoundsException("Calculated bin_size: " + bin_size + ", available bytes: "
+ buffer.length);
}
// Invert first bit
d_copy[0] = buffer[0];
d_copy[0] ^= 0x80;
if (sign == -1)
{
if (sign == -1) {
// Invert every byte
d_copy[0] ^= 0xFF;
}
for (int i = 1; i < bin_size; i++)
{
for (int i = 1; i < bin_size; i++) {
d_copy[i] = buffer[i];
if (sign == -1)
{
if (sign == -1) {
// Invert every byte
d_copy[i] ^= 0xFF;
}
@@ -167,62 +148,49 @@ public class LogBufferTest extends TestCase
BigDecimal intPart = new BigDecimal(0);
if (offset > 0)
intPart = BigDecimal.valueOf(convertNBytesToInt(d_copy, 0, offset));
if (offset > 0) intPart = BigDecimal.valueOf(convertNBytesToInt(d_copy, 0, offset));
while (offset < integerSize)
{
intPart = intPart.movePointRight(LogBuffer.DIG_PER_DEC1).add(
BigDecimal.valueOf(convert4BytesToInt(d_copy, offset)));
while (offset < integerSize) {
intPart = intPart.movePointRight(LogBuffer.DIG_PER_DEC1).add(BigDecimal.valueOf(convert4BytesToInt(d_copy,
offset)));
offset += 4;
}
// Decimal part
BigDecimal fracPart = new BigDecimal(0);
int shift = 0;
for (int i = 0; i < frac0; i++)
{
for (int i = 0; i < frac0; i++) {
shift += LogBuffer.DIG_PER_DEC1;
fracPart = fracPart.add(BigDecimal.valueOf(
convert4BytesToInt(d_copy, offset)).movePointLeft(shift));
fracPart = fracPart.add(BigDecimal.valueOf(convert4BytesToInt(d_copy, offset)).movePointLeft(shift));
offset += 4;
}
if (LogBuffer.dig2bytes[frac0x] > 0)
{
fracPart = fracPart.add(BigDecimal.valueOf(
convertNBytesToInt(d_copy, offset,
LogBuffer.dig2bytes[frac0x])).movePointLeft(
shift + frac0x));
if (LogBuffer.dig2bytes[frac0x] > 0) {
fracPart = fracPart.add(BigDecimal.valueOf(convertNBytesToInt(d_copy, offset, LogBuffer.dig2bytes[frac0x]))
.movePointLeft(shift + frac0x));
}
return BigDecimal.valueOf(sign).multiply(intPart.add(fracPart));
}
public static final byte[] array1 = { (byte) 0x80, 0x00, 0x00, 0x05, 0x1b,
0x38, (byte) 0xb0, 0x60, 0x00 };
public static final byte[] array1 = { (byte) 0x80, 0x00, 0x00, 0x05, 0x1b, 0x38, (byte) 0xb0, 0x60, 0x00 };
public static final byte[] array2 = { (byte) 0x7f, (byte) 0xff,
(byte) 0xff, (byte) 0xfb, (byte) 0xe4, (byte) 0xc7, (byte) 0x4f,
(byte) 0xa0, (byte) 0xff };
public static final byte[] array2 = { (byte) 0x7f, (byte) 0xff, (byte) 0xff, (byte) 0xfb, (byte) 0xe4, (byte) 0xc7,
(byte) 0x4f, (byte) 0xa0, (byte) 0xff };
public static final byte[] array3 = { -128, 0, 6, 20, 113, 56, 6, 26, -123 };
public static final byte[] array4 = { -128, 7, 0, 0, 0, 1, 0, 0, 3 };
public static final byte[] array5 = { -128, 0, 0, 0, 0, 1, 1, -122, -96,
-108 };
public static final byte[] array5 = { -128, 0, 0, 0, 0, 1, 1, -122, -96, -108 };
public void testBigDecimal() throws InterruptedException
{
do
{
public void testBigDecimal() throws InterruptedException {
do {
System.out.println("old extract decimal: ");
long tt1 = 0;
BigDecimal bd1 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t1 = System.nanoTime();
bd1 = extractDecimal(array2, 19, 10);
tt1 += System.nanoTime() - t1;
@@ -233,8 +201,7 @@ public class LogBufferTest extends TestCase
long tt2 = 0;
BigDecimal bd2 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t2 = System.nanoTime();
bd2 = extractDecimal(array1, 19, 10);
tt2 += System.nanoTime() - t2;
@@ -245,8 +212,7 @@ public class LogBufferTest extends TestCase
long tt3 = 0;
BigDecimal bd3 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t3 = System.nanoTime();
bd3 = extractDecimal(array3, 18, 6);
tt3 += System.nanoTime() - t3;
@@ -257,8 +223,7 @@ public class LogBufferTest extends TestCase
long tt4 = 0;
BigDecimal bd4 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t4 = System.nanoTime();
bd4 = extractDecimal(array4, 18, 6);
tt4 += System.nanoTime() - t4;
@@ -269,8 +234,7 @@ public class LogBufferTest extends TestCase
long tt5 = 0;
BigDecimal bd5 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t5 = System.nanoTime();
bd5 = extractDecimal(array5, 18, 6);
tt5 += System.nanoTime() - t5;
@@ -278,11 +242,9 @@ public class LogBufferTest extends TestCase
System.out.print(tt5 / LOOP);
System.out.print("ns >> ");
System.out.println(bd5);
}
while (false);
} while (false);
do
{
do {
System.out.println("new extract decimal: ");
LogBuffer buffer1 = new LogBuffer(array2, 0, array2.length);
@@ -293,8 +255,7 @@ public class LogBufferTest extends TestCase
long tt1 = 0;
BigDecimal bd1 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t1 = System.nanoTime();
bd1 = buffer1.getDecimal(0, 19, 10);
tt1 += System.nanoTime() - t1;
@@ -305,8 +266,7 @@ public class LogBufferTest extends TestCase
long tt2 = 0;
BigDecimal bd2 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t2 = System.nanoTime();
bd2 = buffer2.getDecimal(0, 19, 10);
tt2 += System.nanoTime() - t2;
@@ -317,8 +277,7 @@ public class LogBufferTest extends TestCase
long tt3 = 0;
BigDecimal bd3 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t3 = System.nanoTime();
bd3 = buffer3.getDecimal(0, 18, 6);
tt3 += System.nanoTime() - t3;
@@ -329,8 +288,7 @@ public class LogBufferTest extends TestCase
long tt4 = 0;
BigDecimal bd4 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t4 = System.nanoTime();
bd4 = buffer4.getDecimal(0, 18, 6);
tt4 += System.nanoTime() - t4;
@@ -341,8 +299,7 @@ public class LogBufferTest extends TestCase
long tt5 = 0;
BigDecimal bd5 = null;
for (int i = 0; i < LOOP; i++)
{
for (int i = 0; i < LOOP; i++) {
final long t5 = System.nanoTime();
bd5 = buffer5.getDecimal(0, 18, 6);
tt5 += System.nanoTime() - t5;
@@ -350,7 +307,6 @@ public class LogBufferTest extends TestCase
System.out.print(tt5 / LOOP);
System.out.print("ns >> ");
System.out.println(bd5);
}
while (false);
} while (false);
}
}
+1 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
+17 -5
View File
@@ -1,10 +1,14 @@
#!/bin/bash
cygwin=false;
linux=false;
case "`uname`" in
CYGWIN*)
cygwin=true
;;
Linux*)
linux=true
;;
esac
get_pid() {
@@ -15,11 +19,19 @@ get_pid() {
JAVA_CMD=`cygpath --path --unix $JAVA_CMD`
JAVA_PID=`ps |grep $JAVA_CMD |awk '{print $1}'`
else
if [ ! -z "$PID" ]; then
JAVA_PID=`ps -C java -f --width 1000|grep "$STR"|grep "$PID"|grep -v grep|awk '{print $2}'`
else
JAVA_PID=`ps -C java -f --width 1000|grep "$STR"|grep -v grep|awk '{print $2}'`
fi
if $linux; then
if [ ! -z "$PID" ]; then
JAVA_PID=`ps -C java -f --width 1000|grep "$STR"|grep "$PID"|grep -v grep|awk '{print $2}'`
else
JAVA_PID=`ps -C java -f --width 1000|grep "$STR"|grep -v grep|awk '{print $2}'`
fi
else
if [ ! -z "$PID" ]; then
JAVA_PID=`ps aux |grep "$STR"|grep "$PID"|grep -v grep|awk '{print $2}'`
else
JAVA_PID=`ps aux |grep "$STR"|grep -v grep|awk '{print $2}'`
fi
fi
fi
echo $JAVA_PID;
}
@@ -5,6 +5,7 @@ import java.util.Properties;
import org.I0Itec.zkclient.IZkStateListener;
import org.I0Itec.zkclient.exception.ZkNoNodeException;
import org.I0Itec.zkclient.exception.ZkNodeExistsException;
import org.apache.commons.lang.BooleanUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.zookeeper.Watcher.Event.KeeperState;
@@ -15,6 +16,7 @@ import org.springframework.beans.factory.BeanFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import com.alibaba.otter.canal.common.CanalException;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import com.alibaba.otter.canal.common.zookeeper.ZkClientx;
import com.alibaba.otter.canal.common.zookeeper.ZookeeperPathUtils;
@@ -32,11 +34,12 @@ import com.alibaba.otter.canal.instance.core.CanalInstanceGenerator;
import com.alibaba.otter.canal.instance.manager.CanalConfigClient;
import com.alibaba.otter.canal.instance.manager.ManagerCanalInstanceGenerator;
import com.alibaba.otter.canal.instance.spring.SpringCanalInstanceGenerator;
import com.alibaba.otter.canal.server.embeded.CanalServerWithEmbeded;
import com.alibaba.otter.canal.server.embedded.CanalServerWithEmbedded;
import com.alibaba.otter.canal.server.exception.CanalServerException;
import com.alibaba.otter.canal.server.netty.CanalServerWithNetty;
import com.google.common.base.Function;
import com.google.common.collect.MapMaker;
import com.google.common.collect.MigrateMap;
/**
* canal调度控制器
@@ -58,7 +61,7 @@ public class CanalController {
private boolean autoScan = true;
private InstanceAction defaultAction;
private Map<InstanceMode, InstanceConfigMonitor> instanceConfigMonitors;
private CanalServerWithEmbeded embededCanalServer;
private CanalServerWithEmbedded embededCanalServer;
private CanalServerWithNetty canalServer;
private CanalInstanceGenerator instanceGenerator;
@@ -69,7 +72,7 @@ public class CanalController {
}
public CanalController(final Properties properties){
managerClients = new MapMaker().makeComputingMap(new Function<String, CanalConfigClient>() {
managerClients = MigrateMap.makeComputingMap(new Function<String, CanalConfigClient>() {
public CanalConfigClient apply(String managerAddress) {
return getManagerClient(managerAddress);
@@ -86,9 +89,9 @@ public class CanalController {
cid = Long.valueOf(getProperty(properties, CanalConstants.CANAL_ID));
ip = getProperty(properties, CanalConstants.CANAL_IP);
port = Integer.valueOf(getProperty(properties, CanalConstants.CANAL_PORT));
embededCanalServer = new CanalServerWithEmbeded();
embededCanalServer = CanalServerWithEmbedded.instance();
embededCanalServer.setCanalInstanceGenerator(instanceGenerator);// 设置自定义的instanceGenerator
canalServer = new CanalServerWithNetty(embededCanalServer);
canalServer = CanalServerWithNetty.instance();
canalServer.setIp(ip);
canalServer.setPort(port);
@@ -106,7 +109,7 @@ public class CanalController {
final ServerRunningData serverData = new ServerRunningData(cid, ip + ":" + port);
ServerRunningMonitors.setServerData(serverData);
ServerRunningMonitors.setRunningMonitors(new MapMaker().makeComputingMap(new Function<String, ServerRunningMonitor>() {
ServerRunningMonitors.setRunningMonitors(MigrateMap.makeComputingMap(new Function<String, ServerRunningMonitor>() {
public ServerRunningMonitor apply(final String destination) {
ServerRunningMonitor runningMonitor = new ServerRunningMonitor(serverData);
@@ -170,6 +173,8 @@ public class CanalController {
if (zkclientx != null) {
runningMonitor.setZkClient(zkclientx);
}
// 触发创建一下cid节点
runningMonitor.init();
return runningMonitor;
}
}));
@@ -182,14 +187,15 @@ public class CanalController {
public void start(String destination) {
InstanceConfig config = instanceConfigs.get(destination);
if (config == null) {
config = new InstanceConfig(globalInstanceConfig);
// 重新读取一下instance config
config = parseInstanceConfig(properties, destination);
instanceConfigs.put(destination, config);
}
if (!config.getLazy() && !embededCanalServer.isStart(destination)) {
if (!embededCanalServer.isStart(destination)) {
// HA机制启动
ServerRunningMonitor runningMonitor = ServerRunningMonitors.getRunningMonitor(destination);
if (!runningMonitor.isStart()) {
if (!config.getLazy() && !runningMonitor.isStart()) {
runningMonitor.start();
}
}
@@ -214,7 +220,7 @@ public class CanalController {
}
};
instanceConfigMonitors = new MapMaker().makeComputingMap(new Function<InstanceMode, InstanceConfigMonitor>() {
instanceConfigMonitors = MigrateMap.makeComputingMap(new Function<InstanceMode, InstanceConfigMonitor>() {
public InstanceConfigMonitor apply(InstanceMode mode) {
int scanInterval = Integer.valueOf(getProperty(properties, CanalConstants.CANAL_AUTO_SCAN_INTERVAL));
@@ -228,7 +234,13 @@ public class CanalController {
if (StringUtils.isEmpty(rootDir)) {
rootDir = "../conf";
}
monitor.setRootConf(rootDir);
if (StringUtils.equals("otter-canal", System.getProperty("appName"))) {
monitor.setRootConf(rootDir);
} else {
// eclipse debug模式
monitor.setRootConf("src/main/resources/");
}
return monitor;
} else if (mode.isManager()) {
return new ManagerInstanceConfigMonitor();
@@ -283,6 +295,9 @@ public class CanalController {
System.setProperty(CanalConstants.CANAL_DESTINATION_PROPERTY, destination);
instanceGenerator.setBeanFactory(getBeanFactory(config.getSpringXml()));
return instanceGenerator.generate(destination);
} catch (Throwable e) {
logger.error("generator instance failed.", e);
throw new CanalException(e);
} finally {
System.setProperty(CanalConstants.CANAL_DESTINATION_PROPERTY, "");
}
@@ -377,16 +392,16 @@ public class CanalController {
final String destination = entry.getKey();
InstanceConfig config = entry.getValue();
// 创建destination的工作节点
if (!config.getLazy() && !embededCanalServer.isStart(destination)) {
if (!embededCanalServer.isStart(destination)) {
// HA机制启动
ServerRunningMonitor runningMonitor = ServerRunningMonitors.getRunningMonitor(destination);
if (!runningMonitor.isStart()) {
if (!config.getLazy() && !runningMonitor.isStart()) {
runningMonitor.start();
}
}
if (autoScan) {
instanceConfigMonitors.get(config.getMode()).regeister(destination, defaultAction);
instanceConfigMonitors.get(config.getMode()).register(destination, defaultAction);
}
}
@@ -436,6 +451,9 @@ public class CanalController {
String parentDir = path.substring(0, path.lastIndexOf('/'));
zkclientx.createPersistent(parentDir, true);
zkclientx.createEphemeral(path);
} catch (ZkNodeExistsException e) {
// ignore
// 因为第一次启动时创建了cid,但在stop/start的时可能会关闭和新建,允许出现NodeExists问题s
}
}
@@ -42,7 +42,7 @@ public class CanalLauncher {
controller.stop();
} catch (Throwable e) {
logger.warn("##something goes wrong when stopping canal Server:\n{}",
ExceptionUtils.getFullStackTrace(e));
ExceptionUtils.getFullStackTrace(e));
} finally {
logger.info("## canal server is down.");
}
@@ -51,7 +51,7 @@ public class CanalLauncher {
});
} catch (Throwable e) {
logger.error("## Something goes wrong when starting up the canal Server:\n{}",
ExceptionUtils.getFullStackTrace(e));
ExceptionUtils.getFullStackTrace(e));
System.exit(0);
}
}
@@ -11,15 +11,15 @@ public interface InstanceAction {
/**
* 启动destination
*/
public void start(String destination);
void start(String destination);
/**
* 停止destination
*/
public void stop(String destination);
void stop(String destination);
/**
* 重载destination,可能需要stop,start操作,或者只是更新下内存配置
*/
public void reload(String destination);
void reload(String destination);
}
@@ -10,7 +10,7 @@ import com.alibaba.otter.canal.common.CanalLifeCycle;
*/
public interface InstanceConfigMonitor extends CanalLifeCycle {
public void regeister(String destination, InstanceAction action);
void register(String destination, InstanceAction action);
public void unRegeister(String destination);
void unregister(String destination);
}
@@ -9,11 +9,11 @@ import com.alibaba.otter.canal.common.CanalLifeCycle;
*/
public class ManagerInstanceConfigMonitor extends AbstractCanalLifeCycle implements InstanceConfigMonitor, CanalLifeCycle {
public void regeister(String destination, InstanceAction action) {
public void register(String destination, InstanceAction action) {
}
public void unRegeister(String destination) {
public void unregister(String destination) {
}
@@ -24,6 +24,7 @@ import com.alibaba.otter.canal.common.CanalLifeCycle;
import com.alibaba.otter.canal.common.utils.NamedThreadFactory;
import com.google.common.base.Function;
import com.google.common.collect.MapMaker;
import com.google.common.collect.MigrateMap;
/**
* 监听基于spring配置的instance变化
@@ -39,7 +40,7 @@ public class SpringInstanceConfigMonitor extends AbstractCanalLifeCycle implemen
private long scanIntervalInSecond = 5;
private InstanceAction defaultAction = null;
private Map<String, InstanceAction> actions = new MapMaker().makeMap();
private Map<String, InstanceConfigFiles> lastFiles = new MapMaker().makeComputingMap(new Function<String, InstanceConfigFiles>() {
private Map<String, InstanceConfigFiles> lastFiles = MigrateMap.makeComputingMap(new Function<String, InstanceConfigFiles>() {
public InstanceConfigFiles apply(String destination) {
return new InstanceConfigFiles(destination);
@@ -72,7 +73,7 @@ public class SpringInstanceConfigMonitor extends AbstractCanalLifeCycle implemen
lastFiles.clear();
}
public void regeister(String destination, InstanceAction action) {
public void register(String destination, InstanceAction action) {
if (action != null) {
actions.put(destination, action);
} else {
@@ -80,7 +81,7 @@ public class SpringInstanceConfigMonitor extends AbstractCanalLifeCycle implemen
}
}
public void unRegeister(String destination) {
public void unregister(String destination) {
actions.remove(destination);
}
@@ -174,6 +175,7 @@ public class SpringInstanceConfigMonitor extends AbstractCanalLifeCycle implemen
InstanceAction action = actions.remove(destination);
try {
action.stop(destination);
lastFiles.remove(destination);
logger.info("auto notify stop {} successful.", destination);
} catch (Throwable e) {
logger.error("scan delete found[{}] but stop failed", destination, ExceptionUtils.getFullStackTrace(e));
+6 -1
View File
@@ -40,6 +40,11 @@ canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB
# binlog ddl isolation
canal.instance.get.ddl.isolation = false
@@ -59,4 +64,4 @@ canal.instance.global.lazy = false
#canal.instance.global.manager.address = 127.0.0.1:1099
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml
@@ -178,6 +178,9 @@
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="supportBinlogFormats" value="${canal.instance.binlog.format}" />
<property name="supportBinlogImages" value="${canal.instance.binlog.image}" />
</bean>
</beans>
</beans>
@@ -163,6 +163,9 @@
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="supportBinlogFormats" value="${canal.instance.binlog.format}" />
<property name="supportBinlogImages" value="${canal.instance.binlog.image}" />
</bean>
</beans>
</beans>
@@ -161,6 +161,8 @@
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="supportBinlogFormats" value="${canal.instance.binlog.format}" />
<property name="supportBinlogImages" value="${canal.instance.binlog.image}" />
</bean>
<bean id="eventParser2" class="com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser">
@@ -248,6 +250,9 @@
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="supportBinlogFormats" value="${canal.instance.binlog.format}" />
<property name="supportBinlogImages" value="${canal.instance.binlog.image}" />
</bean>
</beans>
</beans>
@@ -0,0 +1,163 @@
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tx="http://www.springframework.org/schema/tx"
xmlns:aop="http://www.springframework.org/schema/aop" xmlns:lang="http://www.springframework.org/schema/lang"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd
http://www.springframework.org/schema/lang http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd"
default-autowire="byName">
<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
<property name="ignoreResourceNotFound" value="true" />
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
<property name="locationNames">
<list>
<value>classpath:canal.properties</value>
<value>classpath:${canal.instance.destination:}/instance.properties</value>
</list>
</property>
</bean>
<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer">
<property name="propertyEditorRegistrars">
<list>
<ref bean="socketAddressEditor" />
</list>
</property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
<property name="destination" value="${canal.instance.destination}" />
<property name="eventParser">
<ref local="eventParser" />
</property>
<property name="eventSink">
<ref local="eventSink" />
</property>
<property name="eventStore">
<ref local="eventStore" />
</property>
<property name="metaManager">
<ref local="metaManager" />
</property>
<property name="alarmHandler">
<ref local="alarmHandler" />
</property>
<property name="eventFetcher">
<ref local="eventFetcher"/>
</property>
</bean>
<bean id="eventFetcher" class="com.alibaba.otter.canal.mq.fetch.CanalEventFetcherWithMq">
<property name="enable" value="${canal.instance.mq.enable}"/>
<property name="sendMode" value="${canal.instance.mq.sendMode}"/>
</bean>
<!-- 报警处理类 -->
<bean id="alarmHandler" class="com.alibaba.otter.canal.common.alarm.MsgAlarmHandler">
<property name="enabled" value="${canal.alarm.enable}" />
<property name="environment" value="${canal.alarm.environment}" />
<property name="intercenterApi" value="${canal.alarm.intercenter.api}" />
<property name="administrators" value="${canal.alarm.administrators}" />
</bean>
<bean id="zkClientx" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean" >
<property name="targetClass" value="com.alibaba.otter.canal.common.zookeeper.ZkClientx" />
<property name="targetMethod" value="getZkClient" />
<property name="arguments">
<list>
<value>${canal.zkServers:127.0.0.1:2181}</value>
</list>
</property>
</bean>
<bean id="metaManager" class="com.alibaba.otter.canal.meta.PeriodMixedMetaManager">
<property name="zooKeeperMetaManager">
<bean class="com.alibaba.otter.canal.meta.ZooKeeperMetaManager">
<property name="zkClientx" ref="zkClientx" />
</bean>
</property>
<property name="period" value="${canal.zookeeper.flush.period:1000}" />
</bean>
<bean id="eventStore" class="com.alibaba.otter.canal.store.memory.MemoryEventStoreWithBuffer">
<property name="bufferSize" value="${canal.instance.memory.buffer.size:16384}" />
<property name="bufferMemUnit" value="${canal.instance.memory.buffer.memunit:1024}" />
<property name="batchMode" value="${canal.instance.memory.batch.mode:MEMSIZE}" />
<property name="ddlIsolation" value="${canal.instance.get.ddl.isolation:false}" />
</bean>
<bean id="eventSink" class="com.alibaba.otter.canal.sink.entry.EntryEventSink">
<property name="eventStore" ref="eventStore" />
</bean>
<bean id="eventParser" class="com.alibaba.otter.canal.parse.inbound.mysql.LocalBinlogEventParser">
<property name="destination" value="${canal.instance.destination}" />
<property name="alarmHandler" ref="alarmHandler" />
<!-- 解析过滤处理 -->
<property name="eventFilter">
<bean class="com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter" >
<constructor-arg index="0" value="${canal.instance.filter.regex:.*\..*}" />
</bean>
</property>
<property name="eventBlackFilter">
<bean class="com.alibaba.otter.canal.filter.aviater.AviaterRegexFilter" >
<constructor-arg index="0" value="${canal.instance.filter.black.regex:}" />
<constructor-arg index="1" value="false" />
</bean>
</property>
<!-- 最大事务解析大小,超过该大小后事务将被切分为多个事务投递 -->
<property name="transactionSize" value="${canal.instance.transaction.size:1024}" />
<!-- 解析编码 -->
<property name="connectionCharset" value="${canal.instance.connectionCharset:UTF-8}" />
<!-- 解析位点记录 -->
<property name="logPositionManager">
<bean class="com.alibaba.otter.canal.parse.index.FailbackLogPositionManager">
<property name="primary">
<bean class="com.alibaba.otter.canal.parse.index.MemoryLogPositionManager" />
</property>
<property name="failback">
<bean class="com.alibaba.otter.canal.parse.index.MetaLogPositionManager">
<property name="metaManager" ref="metaManager" />
</bean>
</property>
</bean>
</property>
<!-- 解析数据库信息 -->
<property name="masterInfo">
<bean class="com.alibaba.otter.canal.parse.support.AuthenticationInfo">
<property name="address" value="${canal.instance.master.address}" />
<property name="username" value="${canal.instance.dbUsername:retl}" />
<property name="password" value="${canal.instance.dbPassword:retl}" />
<property name="defaultDatabaseName" value="${canal.instance.defaultDatabaseName:retl}" />
</bean>
</property>
<!-- 解析起始位点 -->
<property name="masterPosition">
<bean class="com.alibaba.otter.canal.protocol.position.EntryPosition">
<property name="journalName" value="${canal.instance.master.journal.name}" />
<property name="position" value="${canal.instance.master.position}" />
<property name="timestamp" value="${canal.instance.master.timestamp}" />
</bean>
</property>
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="needWait" value="${canal.instance.parser.needWait:true}"/>
<property name="directory" value="${canal.instance.parser.directory}"/>
</bean>
</beans>
@@ -151,6 +151,9 @@
<property name="filterQueryDml" value="${canal.instance.filter.query.dml:false}" />
<property name="filterQueryDcl" value="${canal.instance.filter.query.dcl:false}" />
<property name="filterQueryDdl" value="${canal.instance.filter.query.ddl:false}" />
<property name="filterRows" value="${canal.instance.filter.rows:false}" />
<property name="filterTableError" value="${canal.instance.filter.table.error:false}" />
<property name="supportBinlogFormats" value="${canal.instance.binlog.format}" />
<property name="supportBinlogImages" value="${canal.instance.binlog.image}" />
</bean>
</beans>
</beans>
+1 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -12,6 +12,7 @@ import org.slf4j.LoggerFactory;
import com.alibaba.otter.canal.parse.driver.mysql.packets.HeaderPacket;
import com.alibaba.otter.canal.parse.driver.mysql.packets.client.ClientAuthenticationPacket;
import com.alibaba.otter.canal.parse.driver.mysql.packets.client.QuitCommandPacket;
import com.alibaba.otter.canal.parse.driver.mysql.packets.server.ErrorPacket;
import com.alibaba.otter.canal.parse.driver.mysql.packets.server.HandshakeInitializationPacket;
import com.alibaba.otter.canal.parse.driver.mysql.packets.server.Reply323Packet;
@@ -26,25 +27,27 @@ import com.alibaba.otter.canal.parse.driver.mysql.utils.PacketManager;
*/
public class MysqlConnector {
private static final Logger logger = LoggerFactory.getLogger(MysqlConnector.class);
private InetSocketAddress address;
private String username;
private String password;
private static final Logger logger = LoggerFactory.getLogger(MysqlConnector.class);
private InetSocketAddress address;
private String username;
private String password;
private byte charsetNumber = 33;
private String defaultSchema = "retl";
private int soTimeout = 30 * 1000;
private int receiveBufferSize = 16 * 1024;
private int sendBufferSize = 16 * 1024;
private byte charsetNumber = 33;
private String defaultSchema = "retl";
private int soTimeout = 30 * 1000;
private int receiveBufferSize = 16 * 1024;
private int sendBufferSize = 16 * 1024;
private SocketChannel channel;
private AtomicBoolean connected = new AtomicBoolean(false);
private SocketChannel channel;
private volatile boolean dumping = false;
// mysql connectinnId
private long connectionId = -1;
private AtomicBoolean connected = new AtomicBoolean(false);
public MysqlConnector(){
}
public MysqlConnector(InetSocketAddress address, String username, String password){
this.address = address;
this.username = username;
this.password = password;
@@ -86,11 +89,30 @@ public class MysqlConnector {
if (channel != null) {
channel.close();
}
logger.info("disConnect MysqlConnection to {}...", address);
} catch (Exception e) {
throw new IOException("disconnect " + this.address + " failure:" + ExceptionUtils.getStackTrace(e));
}
// 执行一次quit
if (dumping && connectionId >= 0) {
MysqlConnector connector = null;
try {
connector = this.fork();
connector.connect();
MysqlUpdateExecutor executor = new MysqlUpdateExecutor(connector);
executor.update("KILL CONNECTION " + connectionId);
} catch (Exception e) {
// 忽略具体异常
logger.info("KILL DUMP " + connectionId + " failure:" + ExceptionUtils.getStackTrace(e));
} finally {
if (connector != null) {
connector.disconnect();
}
}
dumping = false;
}
} else {
logger.info("the channel {} is not connected", this.address);
}
@@ -113,6 +135,17 @@ public class MysqlConnector {
return connector;
}
public void quit() throws IOException {
QuitCommandPacket quit = new QuitCommandPacket();
byte[] cmdBody = quit.toBytes();
HeaderPacket quitHeader = new HeaderPacket();
quitHeader.setPacketBodyLength(cmdBody.length);
quitHeader.setPacketSequenceNumber((byte) 0x00);
PacketManager.write(channel,
new ByteBuffer[] { ByteBuffer.wrap(quitHeader.toBytes()), ByteBuffer.wrap(cmdBody) });
}
// ====================== help method ====================
private void configChannel(SocketChannel channel) throws IOException {
@@ -140,6 +173,7 @@ public class MysqlConnector {
}
HandshakeInitializationPacket handshakePacket = new HandshakeInitializationPacket();
handshakePacket.fromBytes(body);
connectionId = handshakePacket.threadId; // 记录一下connection
logger.info("handshake initialization packet received, prepare the client authentication packet to send");
@@ -291,4 +325,20 @@ public class MysqlConnector {
this.password = password;
}
public long getConnectionId() {
return connectionId;
}
public void setConnectionId(long connectionId) {
this.connectionId = connectionId;
}
public boolean isDumping() {
return dumping;
}
public void setDumping(boolean dumping) {
this.dumping = dumping;
}
}
@@ -14,15 +14,17 @@ import com.alibaba.otter.canal.common.utils.CanalToStringStyle;
* </pre>
*
* <br>
* The Packet Header will not be shown in the descriptions of packets that follow this section. Think of it as always
* there. But logically, it "precedes the packet" rather than "is included in the packet".<br>
* The Packet Header will not be shown in the descriptions of packets that
* follow this section. Think of it as always there. But logically, it
* "precedes the packet" rather than "is included in the packet".<br>
*
* @author fujohnwang
*/
public class HeaderPacket implements IPacket {
/**
* this field indicates the packet length that follows the header, with header packet's 4 bytes excluded.
* this field indicates the packet length that follows the header, with
* header packet's 4 bytes excluded.
*/
private int packetBodyLength;
private byte packetSequenceNumber;
@@ -12,6 +12,7 @@ import java.io.IOException;
* @see 1.0
*/
public interface IPacket {
/**
* un-marshal raw bytes into {@link IPacket} state for application usage.<br>
*
@@ -0,0 +1,35 @@
package com.alibaba.otter.canal.parse.driver.mysql.packets.client;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import com.alibaba.otter.canal.parse.driver.mysql.packets.CommandPacket;
/**
* quit cmd
*
* @author agapple 2016年3月1日 下午8:33:02
* @since 1.0.22
*/
public class QuitCommandPacket extends CommandPacket {
public static final byte[] QUIT = new byte[] { 1, 0, 0, 0, 1 };
public QuitCommandPacket(){
setCommand((byte) 0x01);
}
@Override
public void fromBytes(byte[] data) throws IOException {
}
@Override
public byte[] toBytes() throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(getCommand());
out.write(QUIT);
return out.toByteArray();
}
}
@@ -56,24 +56,26 @@ public class FieldPacket extends PacketWithHeaderPacket {
this.name = reader.readLengthCodedString(data);
this.originalName = reader.readLengthCodedString(data);
index = reader.getIndex();
//
//
index++;
//
//
this.character = ByteHelper.readUnsignedShortLittleEndian(data, index);
index += 2;
//
this.length = ByteHelper.readUnsignedIntLittleEndian(data, index);
index += 4;
//
//
this.type = data[index];
index++;
//
//
this.flags = ByteHelper.readUnsignedShortLittleEndian(data, index);
index += 2;
//
//
this.decimals = data[index];
index++;
//
//
index += 2;//skip filter
//
if (index < data.length) {
reader.setIndex(index);
this.definition = reader.readLengthCodedString(data);
@@ -75,7 +75,8 @@ public class HandshakeInitializationPacket extends PacketWithHeaderPacket {
// 8. bypass filtered bytes
index += 13;
// 9. read rest of scramble_buff
this.restOfScrambleBuff = ByteHelper.readFixedLengthBytes(data, index, 12); // 虽然Handshake Initialization
this.restOfScrambleBuff = ByteHelper.readFixedLengthBytes(data, index, 12); // 虽然Handshake
// Initialization
// Packet规定最后13个byte是剩下的scrumble,
// 但实际上最后一个字节是0, 不应该包含在scrumble中.
// end read
@@ -23,7 +23,7 @@ public abstract class ByteHelper {
out.write(str.getBytes());
out.write(MSC.NULL_TERMINATED_STRING_DELIMITER);
}
public static void writeNullTerminated(byte[] data, ByteArrayOutputStream out) throws IOException {
out.write(data);
out.write(MSC.NULL_TERMINATED_STRING_DELIMITER);
@@ -26,7 +26,7 @@ public class LengthCodedStringReader {
try {
return new String(ArrayUtils.subarray(data, getIndex(), (int) (getIndex() + length)),
encoding == null ? CODE_PAGE_1252 : encoding);
encoding == null ? CODE_PAGE_1252 : encoding);
} finally {
setIndex((int) (getIndex() + length));
}
@@ -30,8 +30,10 @@ public abstract class PacketManager {
}
/**
* Since We r using blocking IO, so we will just write once and assert the length to simplify the read operation.<br>
* If the block write doesn't work as we expected, we will change this implementation as per the result.
* Since We r using blocking IO, so we will just write once and assert the
* length to simplify the read operation.<br>
* If the block write doesn't work as we expected, we will change this
* implementation as per the result.
*
* @param ch
* @param len
@@ -3,7 +3,7 @@ package com.alibaba.otter.canal.parse.driver.mysql;
import java.io.IOException;
import java.net.InetSocketAddress;
import junit.framework.Assert;
import org.junit.Assert;
import org.junit.Test;
+3 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -33,7 +33,9 @@
<plugins>
<!-- deploy模块的packaging通常是jar,如果项目中没有java 源代码或资源文件,加上这一段配置使项目能通过构建 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<archive>
<addMavenDescriptor>true</addMavenDescriptor>
+90
View File
@@ -0,0 +1,90 @@
#! /bin/bash
current_path=`pwd`
case "`uname`" in
Linux)
bin_abs_path=$(readlink -f $(dirname $0))
;;
*)
bin_abs_path=`cd $(dirname $0); pwd`
;;
esac
base=${bin_abs_path}/..
# default values
PORT=3306
MYSQL_IP="0"
# parse arguments
while getopts "h:p:" opt; do
case $opt in
p)
if [[ $OPTARG -gt 0 ]]; then
PORT=$OPTARG
else
echo "Invalid -p option: -$OPTARG, should be positive integer >&2"
exit 1
fi
;;
h)
MYSQL_IP=$OPTARG
;;
\?)
printUsage
exit 1
;;
esac
done
function printUsage
{
cat << __Usage
Initialise the instance config property file automatically. This tool will connect to Mysql master server and get the binlog file and position then update the instance config property file for you.
Usage: $0 -p [port] -h [hostname]
Options:
-p: mysql port, default to 3306
-h: mysql host ip.
For example:
$0 -p 3306 -h 172.17.0.2
Enjoy!!!
__Usage
}
if [[ $PORT -lt 0 ]]; then
echo "invalid port argument $PORT"
printUsage
exit 1
fi
# check if mysql ip is specified
if [[ $MYSQL_IP = "0" ]]; then
echo '-h mysql host ip argument is required.'
printUsage
exit 1
fi
MYSQL_ADDR="$MYSQL_IP:$PORT"
MASTER_STATUS=`mysql -ucanal -pcanal -h$MYSQL_IP -P$PORT -e 'show master status' 2>/dev/null | head -n 2 | tail -n 1`
LOG_FILE=`echo "$MASTER_STATUS" | cut -f 1`
LOG_POS=`echo "$MASTER_STATUS" | cut -f 2`
INSTANCE_CONF_FILE=$base/conf/example/instance.properties
if [ ! -f $INSTANCE_CONF_FILE ]; then
echo "Can not find instance config file: $INSTANCE_CONF_FILE"
exit 1
fi
# replace the binlog file and position
sed -i.bak -e "s/canal\.instance\.master\.address = .*/canal\.instance\.master\.address = $MYSQL_ADDR/" -e "s/canal\.instance\.master\.journal\.name = .*/canal\.instance\.master\.journal\.name = $LOG_FILE/" -e "s/canal\.instance\.master\.position = .*/canal\.instance\.master\.position = $LOG_POS/" $INSTANCE_CONF_FILE
cat $INSTANCE_CONF_FILE
echo
echo 'Above is your instance config file, check if the settings are correct'
echo
@@ -24,7 +24,7 @@ public class SimpleCanalClientTest extends AbstractCanalClientTest {
// 根据ip,直接创建链接,无HA的功能
String destination = "example";
CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(AddressUtils.getHostIp(),
11111), destination, "", "");
11111), destination, "", "");
final SimpleCanalClientTest clientTest = new SimpleCanalClientTest(destination);
clientTest.setConnector(connector);
+1 -1
View File
@@ -3,7 +3,7 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
@@ -10,33 +10,32 @@ import org.apache.oro.text.regex.Perl5Compiler;
import com.alibaba.otter.canal.filter.exception.CanalFilterException;
import com.google.common.base.Function;
import com.google.common.collect.MapMaker;
import com.google.common.collect.MigrateMap;
/**
* 提供{@linkplain Pattern}的lazy get处理
*
*
* @author jianghang 2013-1-22 下午09:36:44
* @version 1.0.0
*/
public class PatternUtils {
private static Map<String, Pattern> patterns = new MapMaker().softValues().makeComputingMap(
new Function<String, Pattern>() {
@SuppressWarnings("deprecation")
private static Map<String, Pattern> patterns = MigrateMap.makeComputingMap(new MapMaker().softValues(),
new Function<String, Pattern>() {
public Pattern apply(
String pattern) {
try {
PatternCompiler pc = new Perl5Compiler();
return pc.compile(
pattern,
Perl5Compiler.CASE_INSENSITIVE_MASK
| Perl5Compiler.READ_ONLY_MASK
| Perl5Compiler.SINGLELINE_MASK);
} catch (MalformedPatternException e) {
throw new CanalFilterException(
e);
}
}
});
public Pattern apply(String pattern) {
try {
PatternCompiler pc = new Perl5Compiler();
return pc.compile(pattern,
Perl5Compiler.CASE_INSENSITIVE_MASK
| Perl5Compiler.READ_ONLY_MASK
| Perl5Compiler.SINGLELINE_MASK);
} catch (MalformedPatternException e) {
throw new CanalFilterException(e);
}
}
});
public static Pattern getPattern(String pattern) {
return patterns.get(pattern);
@@ -1,6 +1,6 @@
package com.alibaba.otter.canal.filter;
import junit.framework.Assert;
import org.junit.Assert;
import org.junit.Test;
@@ -67,16 +67,14 @@ public class AviaterFilterTest {
result = filter3.filter("fooooot");
Assert.assertEquals(true, result);
AviaterRegexFilter filter4 = new AviaterRegexFilter(
"otter2.otter_stability1|otter1.otter_stability1|retl.retl_mark|retl.retl_buffer|retl.xdual");
AviaterRegexFilter filter4 = new AviaterRegexFilter("otter2.otter_stability1|otter1.otter_stability1|retl.retl_mark|retl.retl_buffer|retl.xdual");
result = filter4.filter("otter1.otter_stability1");
Assert.assertEquals(true, result);
}
@Test
public void testDisordered() {
AviaterRegexFilter filter = new AviaterRegexFilter(
"u\\..*,uvw\\..*,uv\\..*,a\\.x,a\\.xyz,a\\.xy,abc\\.x,abc\\.xyz,abc\\.xy,ab\\.x,ab\\.xyz,ab\\.xy");
AviaterRegexFilter filter = new AviaterRegexFilter("u\\..*,uvw\\..*,uv\\..*,a\\.x,a\\.xyz,a\\.xy,abc\\.x,abc\\.xyz,abc\\.xy,ab\\.x,ab\\.xyz,ab\\.xy");
boolean result = filter.filter("u.abc");
Assert.assertEquals(true, result);
@@ -5,7 +5,7 @@ import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;
import junit.framework.Assert;
import org.junit.Assert;
import org.apache.commons.lang.RandomStringUtils;
import org.junit.Test;
+1 -2
View File
@@ -3,10 +3,9 @@
<parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal</artifactId>
<version>1.0.19</version>
<version>1.0.24-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<groupId>com.alibaba.otter</groupId>
<artifactId>canal.instance.core</artifactId>
<packaging>jar</packaging>
<name>canal instance core module for otter ${project.version}</name>

Some files were not shown because too many files have changed in this diff Show More