wyl/DataX
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

v0.0

Browse Source
This commit is contained in:
shf
2018-01-31 16:24:49 +08:00
commit 7ec3fb68e5
651 changed files with 69696 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ftpreader/doc/ftpreader.md
+329
View File
@@ -0,0 +1,329 @@
# DataX FtpReader 说明
------------
## 1 快速介绍
FtpReader提供了读取远程FTP文件系统数据存储的能力。在底层实现上，FtpReader获取远程FTP文件数据，并转换为DataX传输协议传递给Writer。
**本地文件内容存放的是一张逻辑意义上的二维表，例如CSV格式的文本信息。**
## 2 功能与限制
FtpReader实现了从远程FTP文件读取数据并转为DataX协议的功能，远程FTP文件本身是无结构化数据存储，对于DataX而言，FtpReader实现上类比TxtFileReader，有诸多相似之处。目前FtpReader支持功能如下：
1. 支持且仅支持读取TXT的文件，且要求TXT中shema为一张二维表。
2. 支持类CSV格式文件，自定义分隔符。
3. 支持多种类型数据读取(使用String表示)，支持列裁剪，支持列常量
4. 支持递归读取、支持文件名过滤。
5. 支持文本压缩，现有压缩格式为zip、gzip、bzip2。
6. 多个File可以支持并发读取。
我们暂时不能做到：
1. 单个File支持多线程并发读取，这里涉及到单个File内部切分算法。二期考虑支持。
2. 单个File在压缩情况下，从技术上无法支持多线程并发读取。
## 3 功能说明
### 3.1 配置样例
```json
{
"setting": {},
"job": {
"setting": {
"speed": {
"channel": 2
}
},
"content": [
{
"reader": {
"name": "ftpreader",
"parameter": {
"protocol": "sftp",
"host": "127.0.0.1",
"port": 22,
"username": "xx",
"password": "xxx",
"path": [
"/home/hanfa.shf/ftpReaderTest/data"
],
"column": [
{
"index": 0,
"type": "long"
},
{
"index": 1,
"type": "boolean"
},
{
"index": 2,
"type": "double"
},
{
"index": 3,
"type": "string"
},
{
"index": 4,
"type": "date",
"format": "yyyy.MM.dd"
}
],
"encoding": "UTF-8",
"fieldDelimiter": ","
}
},
"writer": {
"name": "ftpWriter",
"parameter": {
"path": "/home/hanfa.shf/ftpReaderTest/result",
"fileName": "shihf",
"writeMode": "truncate",
"format": "yyyy-MM-dd"
}
}
}
]
}
}
```
### 3.2 参数说明
* **protocol**
* 描述：ftp服务器协议，目前支持传输协议有ftp和sftp。 <br />
* 必选：是 <br />
* 默认值：无 <br />
* **host**
* 描述：ftp服务器地址。 <br />
* 必选：是 <br />
* 默认值：无 <br />
* **port**
* 描述：ftp服务器端口。 <br />
* 必选：否 <br />
* 默认值：若传输协议是sftp协议，默认值是22；若传输协议是标准ftp协议，默认值是21 <br />
* **timeout**
* 描述：连接ftp服务器连接超时时间，单位毫秒。 <br />
* 必选：否 <br />
* 默认值：60000（1分钟）<br />
* **connectPattern**
* 描述：连接模式（主动模式或者被动模式）。该参数只在传输协议是标准ftp协议时使用，值只能为：PORT (主动)，PASV（被动）。两种模式主要的不同是数据连接建立的不同。对于Port模式，是客户端在本地打开一个端口等服务器去连接建立数据连接，而Pasv模式就是服务器打开一个端口等待客户端去建立一个数据连接。<br />
* 必选：否 <br />
* 默认值：PASV<br />
* **username**
* 描述：ftp服务器访问用户名。 <br />
* 必选：是 <br />
* 默认值：无 <br />
* **password**
* 描述：ftp服务器访问密码。 <br />
* 必选：是 <br />
* 默认值：无 <br />
* **path**
* 描述：远程FTP文件系统的路径信息，注意这里可以支持填写多个路径。 <br />
当指定单个远程FTP文件，FtpReader暂时只能使用单线程进行数据抽取。二期考虑在非压缩文件情况下针对单个File可以进行多线程并发读取。
当指定多个远程FTP文件，FtpReader支持使用多线程进行数据抽取。线程并发数通过通道数指定。
当指定通配符，FtpReader尝试遍历出多个文件信息。例如: 指定/*代表读取/目录下所有的文件，指定/bazhen/\*代表读取bazhen目录下游所有的文件。**FtpReader目前只支持\*作为文件通配符。**
**特别需要注意的是，DataX会将一个作业下同步的所有Text File视作同一张数据表。用户必须自己保证所有的File能够适配同一套schema信息。读取文件用户必须保证为类CSV格式，并且提供给DataX权限可读。**
**特别需要注意的是，如果Path指定的路径下没有符合匹配的文件抽取，DataX将报错。**
* 必选：是 <br />
* 默认值：无 <br />
* **column**
* 描述：读取字段列表，type指定源数据的类型，index指定当前列来自于文本第几列(以0开始)，value指定当前类型为常量，不从源头文件读取数据，而是根据value值自动生成对应的列。 <br />
默认情况下，用户可以全部按照String类型读取数据，配置如下：
```json
"column": ["*"]
```
用户可以指定Column字段信息，配置如下：
```json
{
"type": "long",
"index": 0 //从远程FTP文件文本第一列获取int字段
},
{
"type": "string",
"value": "alibaba" //从FtpReader内部生成alibaba的字符串字段作为当前字段
}
```
对于用户指定Column信息，type必须填写，index/value必须选择其一。
* 必选：是 <br />
* 默认值：全部按照string类型读取 <br />
* **fieldDelimiter**
* 描述：读取的字段分隔符 <br />
* 必选：是 <br />
* 默认值：, <br />
* **compress**
* 描述：文本压缩类型，默认不填写意味着没有压缩。支持压缩类型为zip、gzip、bzip2。 <br />
* 必选：否 <br />
* 默认值：没有压缩 <br />
* **encoding**
* 描述：读取文件的编码配置。<br />
* 必选：否 <br />
* 默认值：utf-8 <br />
* **skipHeader**
* 描述：类CSV格式文件可能存在表头为标题情况，需要跳过。默认不跳过。<br />
* 必选：否 <br />
* 默认值：false <br />
* **nullFormat**
* 描述：文本文件中无法使用标准字符串定义null(空指针)，DataX提供nullFormat定义哪些字符串可以表示为null。<br />
例如如果用户配置: nullFormat:"\N"，那么如果源头数据是"\N"，DataX视作null字段。
* 必选：否 <br />
* 默认值：\N <br />
* **maxTraversalLevel**
* 描述：允许遍历文件夹的最大层数。<br />
* 必选：否 <br />
* 默认值：100 <br />
* **csvReaderConfig**
* 描述：读取CSV类型文件参数配置，Map类型。读取CSV类型文件使用的CsvReader进行读取，会有很多配置，不配置则使用默认值。<br />
* 必选：否 <br />
* 默认值：无 <br />
常见配置：
```json
"csvReaderConfig":{
"safetySwitch": false,
"skipEmptyRecords": false,
"useTextQualifier": false
}
```
所有配置项及默认值,配置时 csvReaderConfig 的map中请**严格按照以下字段名字进行配置**：
```
boolean caseSensitive = true;
char textQualifier = 34;
boolean trimWhitespace = true;
boolean useTextQualifier = true;//是否使用csv转义字符
char delimiter = 44;//分隔符
char recordDelimiter = 0;
char comment = 35;
boolean useComments = false;
int escapeMode = 1;
boolean safetySwitch = true;//单列长度是否限制100000字符
boolean skipEmptyRecords = true;//是否跳过空行
boolean captureRawRecord = true;
```
### 3.3 类型转换
远程FTP文件本身不提供数据类型，该类型是DataX FtpReader定义：
| DataX 内部类型| 远程FTP文件 数据类型 |
| -------- | ----- |
|
| Long |Long |
| Double |Double|
| String |String|
| Boolean |Boolean |
| Date |Date |
其中：
* 远程FTP文件 Long是指远程FTP文件文本中使用整形的字符串表示形式，例如"19901219"。
* 远程FTP文件 Double是指远程FTP文件文本中使用Double的字符串表示形式，例如"3.1415"。
* 远程FTP文件 Boolean是指远程FTP文件文本中使用Boolean的字符串表示形式，例如"true"、"false"。不区分大小写。
* 远程FTP文件 Date是指远程FTP文件文本中使用Date的字符串表示形式，例如"2014-12-31"，Date可以指定format格式。
## 4 性能报告
## 5 约束限制
略
## 6 FAQ
略
ftpreader/pom.xml
Executable
+92
View File
@@ -0,0 +1,92 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.alibaba.datax</groupId>
<artifactId>datax-all</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>ftpreader</artifactId>
<name>ftpreader</name>
<description>FtpReader提供了读取指定ftp服务器文件功能，并可以根据用户配置的类型进行类型转换，建议开发、测试环境使用。</description>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>com.alibaba.datax</groupId>
<artifactId>datax-common</artifactId>
<version>${datax-project-version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.alibaba.datax</groupId>
<artifactId>plugin-unstructured-storage-util</artifactId>
<version>${datax-project-version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
<version>0.1.51</version>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>3.3</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- compiler plugin -->
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
<encoding>${project-sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptors>
<descriptor>src/main/assembly/package.xml</descriptor>
</descriptors>
<finalName>datax</finalName>
</configuration>
<executions>
<execution>
<id>dwzip</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
ftpreader/src/main/assembly/package.xml
Executable
+35
View File
@@ -0,0 +1,35 @@
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id></id>
<formats>
<format>dir</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>src/main/resources</directory>
<includes>
<include>plugin.json</include>
<include>plugin_job_template.json</include>
</includes>
<outputDirectory>plugin/reader/ftpreader</outputDirectory>
</fileSet>
<fileSet>
<directory>target/</directory>
<includes>
<include>ftpreader-0.0.1-SNAPSHOT.jar</include>
</includes>
<outputDirectory>plugin/reader/ftpreader</outputDirectory>
</fileSet>
</fileSets>
<dependencySets>
<dependencySet>
<useProjectArtifact>false</useProjectArtifact>
<outputDirectory>plugin/reader/ftpreader/libs</outputDirectory>
<scope>runtime</scope>
</dependencySet>
</dependencySets>
</assembly>
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/Constant.java
Executable
+14
View File
@@ -0,0 +1,14 @@
package com.alibaba.datax.plugin.reader.ftpreader;
public class Constant {
public static final String SOURCE_FILES = "sourceFiles";
public static final int DEFAULT_FTP_PORT = 21;
public static final int DEFAULT_SFTP_PORT = 22;
public static final int DEFAULT_TIMEOUT = 60000;
public static final int DEFAULT_MAX_TRAVERSAL_LEVEL = 100;
public static final String DEFAULT_FTP_CONNECT_PATTERN = "PASV";
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/FtpHelper.java
+107
View File
@@ -0,0 +1,107 @@
package com.alibaba.datax.plugin.reader.ftpreader;
import java.io.InputStream;
import java.util.HashSet;
import java.util.List;
public abstract class FtpHelper {
/**
*
* @Title: LoginFtpServer
* @Description: 与ftp服务器建立连接
* @param @param host
* @param @param username
* @param @param password
* @param @param port
* @param @param timeout
* @param @param connectMode
* @return void
* @throws
*/
public abstract void loginFtpServer(String host, String username, String password, int port, int timeout,String connectMode) ;
/**
*
* @Title: LogoutFtpServer
* todo 方法名首字母
* @Description: 断开与ftp服务器的连接
* @param
* @return void
* @throws
*/
public abstract void logoutFtpServer();
/**
*
* @Title: isDirExist
* @Description: 判断指定路径是否是目录
* @param @param directoryPath
* @param @return
* @return boolean
* @throws
*/
public abstract boolean isDirExist(String directoryPath);
/**
*
* @Title: isFileExist
* @Description: 判断指定路径是否是文件
* @param @param filePath
* @param @return
* @return boolean
* @throws
*/
public abstract boolean isFileExist(String filePath);
/**
*
* @Title: isSymbolicLink
* @Description: 判断指定路径是否是软链接
* @param @param filePath
* @param @return
* @return boolean
* @throws
*/
public abstract boolean isSymbolicLink(String filePath);
/**
*
* @Title: getListFiles
* @Description: 递归获取指定路径下符合条件的所有文件绝对路径
* @param @param directoryPath
* @param @param parentLevel 父目录的递归层数（首次为0）
* @param @param maxTraversalLevel 允许的最大递归层数
* @param @return
* @return HashSet<String>
* @throws
*/
public abstract HashSet<String> getListFiles(String directoryPath, int parentLevel, int maxTraversalLevel);
/**
*
* @Title: getInputStream
* @Description: 获取指定路径的输入流
* @param @param filePath
* @param @return
* @return InputStream
* @throws
*/
public abstract InputStream getInputStream(String filePath);
/**
*
* @Title: getAllFiles
* @Description: 获取指定路径列表下符合条件的所有文件的绝对路径
* @param @param srcPaths 路径列表
* @param @param parentLevel 父目录的递归层数（首次为0）
* @param @param maxTraversalLevel 允许的最大递归层数
* @param @return
* @return HashSet<String>
* @throws
*/
public HashSet<String> getAllFiles(List<String> srcPaths, int parentLevel, int maxTraversalLevel){
HashSet<String> sourceAllFiles = new HashSet<String>();
if (!srcPaths.isEmpty()) {
for (String eachPath : srcPaths) {
sourceAllFiles.addAll(getListFiles(eachPath, parentLevel, maxTraversalLevel));
}
}
return sourceAllFiles;
}
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/FtpReader.java
+253
View File
@@ -0,0 +1,253 @@
package com.alibaba.datax.plugin.reader.ftpreader;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.plugin.RecordSender;
import com.alibaba.datax.common.spi.Reader;
import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil;
public class FtpReader extends Reader {
public static class Job extends Reader.Job {
private static final Logger LOG = LoggerFactory.getLogger(Job.class);
private Configuration originConfig = null;
private List<String> path = null;
private HashSet<String> sourceFiles;
// ftp链接参数
private String protocol;
private String host;
private int port;
private String username;
private String password;
private int timeout;
private String connectPattern;
private int maxTraversalLevel;
private FtpHelper ftpHelper = null;
@Override
public void init() {
this.originConfig = this.getPluginJobConf();
this.sourceFiles = new HashSet<String>();
this.validateParameter();
UnstructuredStorageReaderUtil.validateParameter(this.originConfig);
if ("sftp".equals(protocol)) {
//sftp协议
this.port = originConfig.getInt(Key.PORT, Constant.DEFAULT_SFTP_PORT);
this.ftpHelper = new SftpHelper();
} else if ("ftp".equals(protocol)) {
// ftp 协议
this.port = originConfig.getInt(Key.PORT, Constant.DEFAULT_FTP_PORT);
this.ftpHelper = new StandardFtpHelper();
}
ftpHelper.loginFtpServer(host, username, password, port, timeout, connectPattern);
}
private void validateParameter() {
//todo 常量
this.protocol = this.originConfig.getNecessaryValue(Key.PROTOCOL, FtpReaderErrorCode.REQUIRED_VALUE);
boolean ptrotocolTag = "ftp".equals(this.protocol) || "sftp".equals(this.protocol);
if (!ptrotocolTag) {
throw DataXException.asDataXException(FtpReaderErrorCode.ILLEGAL_VALUE,
String.format("仅支持 ftp和sftp 传输协议 , 不支持您配置的传输协议: [%s]", protocol));
}
this.host = this.originConfig.getNecessaryValue(Key.HOST, FtpReaderErrorCode.REQUIRED_VALUE);
this.username = this.originConfig.getNecessaryValue(Key.USERNAME, FtpReaderErrorCode.REQUIRED_VALUE);
this.password = this.originConfig.getNecessaryValue(Key.PASSWORD, FtpReaderErrorCode.REQUIRED_VALUE);
this.timeout = originConfig.getInt(Key.TIMEOUT, Constant.DEFAULT_TIMEOUT);
this.maxTraversalLevel = originConfig.getInt(Key.MAXTRAVERSALLEVEL, Constant.DEFAULT_MAX_TRAVERSAL_LEVEL);
// only support connect pattern
this.connectPattern = this.originConfig.getUnnecessaryValue(Key.CONNECTPATTERN, Constant.DEFAULT_FTP_CONNECT_PATTERN, null);
boolean connectPatternTag = "PORT".equals(connectPattern) || "PASV".equals(connectPattern);
if (!connectPatternTag) {
throw DataXException.asDataXException(FtpReaderErrorCode.ILLEGAL_VALUE,
String.format("不支持您配置的ftp传输模式: [%s]", connectPattern));
}else{
this.originConfig.set(Key.CONNECTPATTERN, connectPattern);
}
//path check
String pathInString = this.originConfig.getNecessaryValue(Key.PATH, FtpReaderErrorCode.REQUIRED_VALUE);
if (!pathInString.startsWith("[") && !pathInString.endsWith("]")) {
path = new ArrayList<String>();
path.add(pathInString);
} else {
path = this.originConfig.getList(Key.PATH, String.class);
if (null == path || path.size() == 0) {
throw DataXException.asDataXException(FtpReaderErrorCode.REQUIRED_VALUE, "您需要指定待读取的源目录或文件");
}
for (String eachPath : path) {
if(!eachPath.startsWith("/")){
String message = String.format("请检查参数path:[%s],需要配置为绝对路径", eachPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.ILLEGAL_VALUE, message);
}
}
}
}
@Override
public void prepare() {
LOG.debug("prepare() begin...");
this.sourceFiles = ftpHelper.getAllFiles(path, 0, maxTraversalLevel);
LOG.info(String.format("您即将读取的文件数为: [%s]", this.sourceFiles.size()));
}
@Override
public void post() {
}
@Override
public void destroy() {
try {
this.ftpHelper.logoutFtpServer();
} catch (Exception e) {
String message = String.format(
"关闭与ftp服务器连接失败: [%s] host=%s, username=%s, port=%s",
e.getMessage(), host, username, port);
LOG.error(message, e);
}
}
// warn: 如果源目录为空会报错，拖空目录意图=>空文件显示指定此意图
@Override
public List<Configuration> split(int adviceNumber) {
LOG.debug("split() begin...");
List<Configuration> readerSplitConfigs = new ArrayList<Configuration>();
// warn:每个slice拖且仅拖一个文件,
// int splitNumber = adviceNumber;
int splitNumber = this.sourceFiles.size();
if (0 == splitNumber) {
throw DataXException.asDataXException(FtpReaderErrorCode.EMPTY_DIR_EXCEPTION,
String.format("未能找到待读取的文件,请确认您的配置项path: %s", this.originConfig.getString(Key.PATH)));
}
List<List<String>> splitedSourceFiles = this.splitSourceFiles(new ArrayList(this.sourceFiles), splitNumber);
for (List<String> files : splitedSourceFiles) {
Configuration splitedConfig = this.originConfig.clone();
splitedConfig.set(Constant.SOURCE_FILES, files);
readerSplitConfigs.add(splitedConfig);
}
LOG.debug("split() ok and end...");
return readerSplitConfigs;
}
private <T> List<List<T>> splitSourceFiles(final List<T> sourceList, int adviceNumber) {
List<List<T>> splitedList = new ArrayList<List<T>>();
int averageLength = sourceList.size() / adviceNumber;
averageLength = averageLength == 0 ? 1 : averageLength;
for (int begin = 0, end = 0; begin < sourceList.size(); begin = end) {
end = begin + averageLength;
if (end > sourceList.size()) {
end = sourceList.size();
}
splitedList.add(sourceList.subList(begin, end));
}
return splitedList;
}
}
public static class Task extends Reader.Task {
private static Logger LOG = LoggerFactory.getLogger(Task.class);
private String host;
private int port;
private String username;
private String password;
private String protocol;
private int timeout;
private String connectPattern;
private Configuration readerSliceConfig;
private List<String> sourceFiles;
private FtpHelper ftpHelper = null;
@Override
public void init() {//连接重试
/* for ftp connection */
this.readerSliceConfig = this.getPluginJobConf();
this.host = readerSliceConfig.getString(Key.HOST);
this.protocol = readerSliceConfig.getString(Key.PROTOCOL);
this.username = readerSliceConfig.getString(Key.USERNAME);
this.password = readerSliceConfig.getString(Key.PASSWORD);
this.timeout = readerSliceConfig.getInt(Key.TIMEOUT, Constant.DEFAULT_TIMEOUT);
this.sourceFiles = this.readerSliceConfig.getList(Constant.SOURCE_FILES, String.class);
if ("sftp".equals(protocol)) {
//sftp协议
this.port = readerSliceConfig.getInt(Key.PORT, Constant.DEFAULT_SFTP_PORT);
this.ftpHelper = new SftpHelper();
} else if ("ftp".equals(protocol)) {
// ftp 协议
this.port = readerSliceConfig.getInt(Key.PORT, Constant.DEFAULT_FTP_PORT);
this.connectPattern = readerSliceConfig.getString(Key.CONNECTPATTERN, Constant.DEFAULT_FTP_CONNECT_PATTERN);// 默认为被动模式
this.ftpHelper = new StandardFtpHelper();
}
ftpHelper.loginFtpServer(host, username, password, port, timeout, connectPattern);
}
@Override
public void prepare() {
}
@Override
public void post() {
}
@Override
public void destroy() {
try {
this.ftpHelper.logoutFtpServer();
} catch (Exception e) {
String message = String.format(
"关闭与ftp服务器连接失败: [%s] host=%s, username=%s, port=%s",
e.getMessage(), host, username, port);
LOG.error(message, e);
}
}
@Override
public void startRead(RecordSender recordSender) {
LOG.debug("start read source files...");
for (String fileName : this.sourceFiles) {
LOG.info(String.format("reading file : [%s]", fileName));
InputStream inputStream = null;
inputStream = ftpHelper.getInputStream(fileName);
UnstructuredStorageReaderUtil.readFromStream(inputStream, fileName, this.readerSliceConfig,
recordSender, this.getTaskPluginCollector());
recordSender.flush();
}
LOG.debug("end read source files...");
}
}
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/FtpReaderErrorCode.java
Executable
+52
View File
@@ -0,0 +1,52 @@
package com.alibaba.datax.plugin.reader.ftpreader;
import com.alibaba.datax.common.spi.ErrorCode;
/**
* Created by haiwei.luo on 14-9-20.
*/
public enum FtpReaderErrorCode implements ErrorCode {
REQUIRED_VALUE("FtpReader-00", "您缺失了必须填写的参数值."),
ILLEGAL_VALUE("FtpReader-01", "您填写的参数值不合法."),
MIXED_INDEX_VALUE("FtpReader-02", "您的列信息配置同时包含了index,value."),
NO_INDEX_VALUE("FtpReader-03","您明确的配置列信息,但未填写相应的index,value."),
FILE_NOT_EXISTS("FtpReader-04", "您配置的目录文件路径不存在或者没有权限读取."),
OPEN_FILE_WITH_CHARSET_ERROR("FtpReader-05", "您配置的文件编码和实际文件编码不符合."),
OPEN_FILE_ERROR("FtpReader-06", "您配置的文件在打开时异常."),
READ_FILE_IO_ERROR("FtpReader-07", "您配置的文件在读取时出现IO异常."),
SECURITY_NOT_ENOUGH("FtpReader-08", "您缺少权限执行相应的文件操作."),
CONFIG_INVALID_EXCEPTION("FtpReader-09", "您的参数配置错误."),
RUNTIME_EXCEPTION("FtpReader-10", "出现运行时异常, 请联系我们"),
EMPTY_DIR_EXCEPTION("FtpReader-11", "您尝试读取的文件目录为空."),
FAIL_LOGIN("FtpReader-12", "登录失败,无法与ftp服务器建立连接."),
FAIL_DISCONNECT("FtpReader-13", "关闭ftp连接失败,无法与ftp服务器断开连接."),
COMMAND_FTP_IO_EXCEPTION("FtpReader-14", "与ftp服务器连接异常."),
OUT_MAX_DIRECTORY_LEVEL("FtpReader-15", "超出允许的最大目录层数."),
LINK_FILE("FtpReader-16", "您尝试读取的文件为链接文件."),;
private final String code;
private final String description;
private FtpReaderErrorCode(String code, String description) {
this.code = code;
this.description = description;
}
@Override
public String getCode() {
return this.code;
}
@Override
public String getDescription() {
return this.description;
}
@Override
public String toString() {
return String.format("Code:[%s], Description:[%s].", this.code,
this.description);
}
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/Key.java
Executable
+13
View File
@@ -0,0 +1,13 @@
package com.alibaba.datax.plugin.reader.ftpreader;
public class Key {
public static final String PROTOCOL = "protocol";
public static final String HOST = "host";
public static final String USERNAME = "username";
public static final String PASSWORD = "password";
public static final String PORT = "port";
public static final String TIMEOUT = "timeout";
public static final String CONNECTPATTERN = "connectPattern";
public static final String PATH = "path";
public static final String MAXTRAVERSALLEVEL = "maxTraversalLevel";
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/SftpHelper.java
+246
View File
@@ -0,0 +1,246 @@
package com.alibaba.datax.plugin.reader.ftpreader;
import java.io.InputStream;
import java.util.HashSet;
import java.util.Properties;
import java.util.Vector;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.SftpATTRS;
import com.jcraft.jsch.SftpException;
import com.jcraft.jsch.ChannelSftp.LsEntry;
public class SftpHelper extends FtpHelper {
private static final Logger LOG = LoggerFactory.getLogger(SftpHelper.class);
Session session = null;
ChannelSftp channelSftp = null;
@Override
public void loginFtpServer(String host, String username, String password, int port, int timeout,
String connectMode) {
JSch jsch = new JSch(); // 创建JSch对象
try {
session = jsch.getSession(username, host, port);
// 根据用户名，主机ip，端口获取一个Session对象
// 如果服务器连接不上，则抛出异常
if (session == null) {
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN,
"session is null,无法通过sftp与服务器建立链接，请检查主机名和用户名是否正确.");
}
session.setPassword(password); // 设置密码
Properties config = new Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config); // 为Session对象设置properties
session.setTimeout(timeout); // 设置timeout时间
session.connect(); // 通过Session建立链接
channelSftp = (ChannelSftp) session.openChannel("sftp"); // 打开SFTP通道
channelSftp.connect(); // 建立SFTP通道的连接
//设置命令传输编码
//String fileEncoding = System.getProperty("file.encoding");
//channelSftp.setFilenameEncoding(fileEncoding);
} catch (JSchException e) {
if(null != e.getCause()){
String cause = e.getCause().toString();
String unknownHostException = "java.net.UnknownHostException: " + host;
String illegalArgumentException = "java.lang.IllegalArgumentException: port out of range:" + port;
String wrongPort = "java.net.ConnectException: Connection refused";
if (unknownHostException.equals(cause)) {
String message = String.format("请确认ftp服务器地址是否正确，无法连接到地址为: [%s] 的ftp服务器", host);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
} else if (illegalArgumentException.equals(cause) || wrongPort.equals(cause) ) {
String message = String.format("请确认连接ftp服务器端口是否正确，错误的端口: [%s] ", port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
}
}else {
if("Auth fail".equals(e.getMessage())){
String message = String.format("与ftp服务器建立连接失败,请检查用户名和密码是否正确: [%s]",
"message:host =" + host + ",username = " + username + ",port =" + port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message);
}else{
String message = String.format("与ftp服务器建立连接失败 : [%s]",
"message:host =" + host + ",username = " + username + ",port =" + port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
}
}
}
}
@Override
public void logoutFtpServer() {
if (channelSftp != null) {
channelSftp.disconnect();
}
if (session != null) {
session.disconnect();
}
}
@Override
public boolean isDirExist(String directoryPath) {
try {
SftpATTRS sftpATTRS = channelSftp.lstat(directoryPath);
return sftpATTRS.isDir();
} catch (SftpException e) {
if (e.getMessage().toLowerCase().equals("no such file")) {
String message = String.format("请确认您的配置项path:[%s]存在，且配置的用户有权限读取", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
String message = String.format("进入目录：[%s]时发生I/O异常,请确认与ftp服务器的连接正常", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
}
@Override
public boolean isFileExist(String filePath) {
boolean isExitFlag = false;
try {
SftpATTRS sftpATTRS = channelSftp.lstat(filePath);
if(sftpATTRS.getSize() >= 0){
isExitFlag = true;
}
} catch (SftpException e) {
if (e.getMessage().toLowerCase().equals("no such file")) {
String message = String.format("请确认您的配置项path:[%s]存在，且配置的用户有权限读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
} else {
String message = String.format("获取文件：[%s] 属性时发生I/O异常,请确认与ftp服务器的连接正常", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
}
return isExitFlag;
}
@Override
public boolean isSymbolicLink(String filePath) {
try {
SftpATTRS sftpATTRS = channelSftp.lstat(filePath);
return sftpATTRS.isLink();
} catch (SftpException e) {
if (e.getMessage().toLowerCase().equals("no such file")) {
String message = String.format("请确认您的配置项path:[%s]存在，且配置的用户有权限读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
} else {
String message = String.format("获取文件：[%s] 属性时发生I/O异常,请确认与ftp服务器的连接正常", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
}
}
HashSet<String> sourceFiles = new HashSet<String>();
@Override
public HashSet<String> getListFiles(String directoryPath, int parentLevel, int maxTraversalLevel) {
if(parentLevel < maxTraversalLevel){
String parentPath = null;// 父级目录,以'/'结尾
int pathLen = directoryPath.length();
if (directoryPath.contains("*") || directoryPath.contains("?")) {//*和？的限制
// path是正则表达式
String subPath = UnstructuredStorageReaderUtil.getRegexPathParentPath(directoryPath);
if (isDirExist(subPath)) {
parentPath = subPath;
} else {
String message = String.format("不能进入目录：[%s]," + "请确认您的配置项path:[%s]存在，且配置的用户有权限进入", subPath,
directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
} else if (isDirExist(directoryPath)) {
// path是目录
if (directoryPath.charAt(pathLen - 1) == IOUtils.DIR_SEPARATOR) {
parentPath = directoryPath;
} else {
parentPath = directoryPath + IOUtils.DIR_SEPARATOR;
}
} else if(isSymbolicLink(directoryPath)){
//path是链接文件
String message = String.format("文件:[%s]是链接文件，当前不支持链接文件的读取", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.LINK_FILE, message);
}else if (isFileExist(directoryPath)) {
// path指向具体文件
sourceFiles.add(directoryPath);
return sourceFiles;
} else {
String message = String.format("请确认您的配置项path:[%s]存在，且配置的用户有权限读取", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
try {
Vector vector = channelSftp.ls(directoryPath);
for (int i = 0; i < vector.size(); i++) {
LsEntry le = (LsEntry) vector.get(i);
String strName = le.getFilename();
String filePath = parentPath + strName;
if (isDirExist(filePath)) {
// 是子目录
if (!(strName.equals(".") || strName.equals(".."))) {
//递归处理
getListFiles(filePath, parentLevel+1, maxTraversalLevel);
}
} else if(isSymbolicLink(filePath)){
//是链接文件
String message = String.format("文件:[%s]是链接文件，当前不支持链接文件的读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.LINK_FILE, message);
}else if (isFileExist(filePath)) {
// 是文件
sourceFiles.add(filePath);
} else {
String message = String.format("请确认path:[%s]存在，且配置的用户有权限读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
} // end for vector
} catch (SftpException e) {
String message = String.format("获取path：[%s] 下文件列表时发生I/O异常,请确认与ftp服务器的连接正常", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
return sourceFiles;
}else{
//超出最大递归层数
String message = String.format("获取path：[%s] 下文件列表时超出最大层数,请确认路径[%s]下不存在软连接文件", directoryPath, directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.OUT_MAX_DIRECTORY_LEVEL, message);
}
}
@Override
public InputStream getInputStream(String filePath) {
try {
return channelSftp.get(filePath);
} catch (SftpException e) {
String message = String.format("读取文件 : [%s] 时出错,请确认文件：[%s]存在且配置的用户有权限读取", filePath, filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.OPEN_FILE_ERROR, message);
}
}
}
ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/StandardFtpHelper.java
+229
View File
@@ -0,0 +1,229 @@
package com.alibaba.datax.plugin.reader.ftpreader;
import java.io.IOException;
import java.io.InputStream;
import java.net.UnknownHostException;
import java.util.HashSet;
import org.apache.commons.io.IOUtils;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPClientConfig;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil;
public class StandardFtpHelper extends FtpHelper {
private static final Logger LOG = LoggerFactory.getLogger(StandardFtpHelper.class);
FTPClient ftpClient = null;
@Override
public void loginFtpServer(String host, String username, String password, int port, int timeout,
String connectMode) {
ftpClient = new FTPClient();
try {
// 连接
ftpClient.connect(host, port);
// 登录
ftpClient.login(username, password);
// 不需要写死ftp server的OS TYPE,FTPClient getSystemType()方法会自动识别
// ftpClient.configure(new FTPClientConfig(FTPClientConfig.SYST_UNIX));
ftpClient.setConnectTimeout(timeout);
ftpClient.setDataTimeout(timeout);
if ("PASV".equals(connectMode)) {
ftpClient.enterRemotePassiveMode();
ftpClient.enterLocalPassiveMode();
} else if ("PORT".equals(connectMode)) {
ftpClient.enterLocalActiveMode();
// ftpClient.enterRemoteActiveMode(host, port);
}
int reply = ftpClient.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftpClient.disconnect();
String message = String.format("与ftp服务器建立连接失败,请检查用户名和密码是否正确: [%s]",
"message:host =" + host + ",username = " + username + ",port =" + port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message);
}
//设置命令传输编码
String fileEncoding = System.getProperty("file.encoding");
ftpClient.setControlEncoding(fileEncoding);
} catch (UnknownHostException e) {
String message = String.format("请确认ftp服务器地址是否正确，无法连接到地址为: [%s] 的ftp服务器", host);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
} catch (IllegalArgumentException e) {
String message = String.format("请确认连接ftp服务器端口是否正确，错误的端口: [%s] ", port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
} catch (Exception e) {
String message = String.format("与ftp服务器建立连接失败 : [%s]",
"message:host =" + host + ",username = " + username + ",port =" + port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
}
}
@Override
public void logoutFtpServer() {
if (ftpClient.isConnected()) {
try {
//todo ftpClient.completePendingCommand();//打开流操作之后必须，原因还需要深究
ftpClient.logout();
} catch (IOException e) {
String message = "与ftp服务器断开连接失败";
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_DISCONNECT, message, e);
}finally {
if(ftpClient.isConnected()){
try {
ftpClient.disconnect();
} catch (IOException e) {
String message = "与ftp服务器断开连接失败";
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_DISCONNECT, message, e);
}
}
}
}
}
@Override
public boolean isDirExist(String directoryPath) {
try {
return ftpClient.changeWorkingDirectory(new String(directoryPath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING));
} catch (IOException e) {
String message = String.format("进入目录：[%s]时发生I/O异常,请确认与ftp服务器的连接正常", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
}
@Override
public boolean isFileExist(String filePath) {
boolean isExitFlag = false;
try {
FTPFile[] ftpFiles = ftpClient.listFiles(new String(filePath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING));
if (ftpFiles.length == 1 && ftpFiles[0].isFile()) {
isExitFlag = true;
}
} catch (IOException e) {
String message = String.format("获取文件：[%s] 属性时发生I/O异常,请确认与ftp服务器的连接正常", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
return isExitFlag;
}
@Override
public boolean isSymbolicLink(String filePath) {
boolean isExitFlag = false;
try {
FTPFile[] ftpFiles = ftpClient.listFiles(new String(filePath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING));
if (ftpFiles.length == 1 && ftpFiles[0].isSymbolicLink()) {
isExitFlag = true;
}
} catch (IOException e) {
String message = String.format("获取文件：[%s] 属性时发生I/O异常,请确认与ftp服务器的连接正常", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
return isExitFlag;
}
HashSet<String> sourceFiles = new HashSet<String>();
@Override
public HashSet<String> getListFiles(String directoryPath, int parentLevel, int maxTraversalLevel) {
if(parentLevel < maxTraversalLevel){
String parentPath = null;// 父级目录,以'/'结尾
int pathLen = directoryPath.length();
if (directoryPath.contains("*") || directoryPath.contains("?")) {
// path是正则表达式
String subPath = UnstructuredStorageReaderUtil.getRegexPathParentPath(directoryPath);
if (isDirExist(subPath)) {
parentPath = subPath;
} else {
String message = String.format("不能进入目录：[%s]," + "请确认您的配置项path:[%s]存在，且配置的用户有权限进入", subPath,
directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
} else if (isDirExist(directoryPath)) {
// path是目录
if (directoryPath.charAt(pathLen - 1) == IOUtils.DIR_SEPARATOR) {
parentPath = directoryPath;
} else {
parentPath = directoryPath + IOUtils.DIR_SEPARATOR;
}
} else if (isFileExist(directoryPath)) {
// path指向具体文件
sourceFiles.add(directoryPath);
return sourceFiles;
} else if(isSymbolicLink(directoryPath)){
//path是链接文件
String message = String.format("文件:[%s]是链接文件，当前不支持链接文件的读取", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.LINK_FILE, message);
}else {
String message = String.format("请确认您的配置项path:[%s]存在，且配置的用户有权限读取", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
try {
FTPFile[] fs = ftpClient.listFiles(new String(directoryPath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING));
for (FTPFile ff : fs) {
String strName = ff.getName();
String filePath = parentPath + strName;
if (ff.isDirectory()) {
if (!(strName.equals(".") || strName.equals(".."))) {
//递归处理
getListFiles(filePath, parentLevel+1, maxTraversalLevel);
}
} else if (ff.isFile()) {
// 是文件
sourceFiles.add(filePath);
} else if(ff.isSymbolicLink()){
//是链接文件
String message = String.format("文件:[%s]是链接文件，当前不支持链接文件的读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.LINK_FILE, message);
}else {
String message = String.format("请确认path:[%s]存在，且配置的用户有权限读取", filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FILE_NOT_EXISTS, message);
}
} // end for FTPFile
} catch (IOException e) {
String message = String.format("获取path：[%s] 下文件列表时发生I/O异常,请确认与ftp服务器的连接正常", directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, message, e);
}
return sourceFiles;
} else{
//超出最大递归层数
String message = String.format("获取path：[%s] 下文件列表时超出最大层数,请确认路径[%s]下不存在软连接文件", directoryPath, directoryPath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.OUT_MAX_DIRECTORY_LEVEL, message);
}
}
@Override
public InputStream getInputStream(String filePath) {
try {
return ftpClient.retrieveFileStream(new String(filePath.getBytes(),FTP.DEFAULT_CONTROL_ENCODING));
} catch (IOException e) {
String message = String.format("读取文件 : [%s] 时出错,请确认文件：[%s]存在且配置的用户有权限读取", filePath, filePath);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.OPEN_FILE_ERROR, message);
}
}
}
ftpreader/src/main/resources/plugin-template.json
Executable
+38
View File
@@ -0,0 +1,38 @@
{
"name": "ftpreader",
"parameter": {
"host": "",
"port": "",
"username": "",
"password": "",
"protocol": "",
"path": [
""
],
"encoding": "UTF-8",
"column": [
{
"index": 0,
"type": "long"
},
{
"index": 1,
"type": "boolean"
},
{
"index": 2,
"type": "double"
},
{
"index": 3,
"type": "string"
},
{
"index": 4,
"type": "date",
"format": "yyyy.MM.dd"
}
],
"fieldDelimiter": ","
}
}
ftpreader/src/main/resources/plugin.json
Executable
+7
View File
@@ -0,0 +1,7 @@
{
"name": "ftpreader",
"class": "com.alibaba.datax.plugin.reader.ftpreader.FtpReader",
"description": "useScene: test. mechanism: use datax framework to transport data from txt file. warn: The more you know about the data, the less problems you encounter.",
"developer": "alibaba"
}
ftpreader/src/main/resources/plugin_job_template.json
+19
View File
@@ -0,0 +1,19 @@
{
"name": "ftpreader",
"parameter": {
"host": "",
"protocol": "sftp",
"port":"",
"username": "",
"password": "",
"path": [],
"column": [
{
"index": 0,
"type": ""
}
],
"fieldDelimiter": ",",
"encoding": "UTF-8"
}
}