都9102年,还用properties配置文件吗?非也非也。
Azkaban flow 2.0使用yaml进行作业配置:
上传的文件夹中,可以包含多个project 的yml配置文件。
Flow YAML File
关于Flow的文件有如下定义:
- 每个Flow对应一个yml文件.
- Flow的名称以yml文件的名称, 如: my-flow-name.flow.
- 包含所有的执行DAG节点.
- 每个执行节点可以是job或者是flow.
- 每个执行节点可以有一下属性:name, type, config, dependsOn 和节点.
- Node dependencies are specified by listing the parent nodes in dependsOn list.
- 包含其他的flow配置、.
- 所有的properties配置文件,将会迁移到YAML文件里面
Demo:acceptance-test.flow
---config: user.to.proxy: azktest param.hadoopOutData: /tmp/wordcounthadoopout param.inData: /tmp/wordcountpigin param.outData: /tmp/wordcountpigout# This section defines the list of jobs# A node can be a job or a flow# In this example, all nodes are jobsnodes: # Job definition # The job definition is like a YAMLified version of properties file # with one major difference. All custom properties are now clubbed together # in a config section in the definition. # The first line describes the name of the job - name: AZTest type: noop # The dependsOn section contains the list of parent nodes the current # node depends on dependsOn: - hadoopWC1 - NoOpTest1 - hive2 - java1 - jobCommand2 - name: pigWordCount1 type: pig # The config section contains custom arguments or parameters which are # required by the job config: pig.script: src/main/pig/wordCountText.pig - name: hadoopWC1 type: hadoopJava dependsOn: - pigWordCount1 config: classpath: ./* force.output.overwrite: true input.path: ${param.inData} job.class: com.linkedin.wordcount.WordCount main.args: ${param.inData} ${param.hadoopOutData} output.path: ${param.hadoopOutData} - name: hive1 type: hive config: hive.script: src/main/hive/showdb.q - name: NoOpTest1 type: noop - name: hive2 type: hive dependsOn: - hive1 config: hive.script: src/main/hive/showTables.sql - name: java1 type: javaprocess config: Xms: 96M java.class: com.linkedin.foo.HelloJavaProcessJob - name: jobCommand1 type: command config: command: echo "hello world from job_command_1" - name: jobCommand2 type: command dependsOn: - jobCommand1 config: command: echo "hello world from job_command_2"
然后打包的zip文件的架构如下:
project_root├── sample_project.project├── flow1.flow├── flow2.flow├── ...├── flown.flow├── lib│ ├── ...│ └── paranamer-2.4.1.jar└── src └── main ├── hive │ └── query.q └── pig └── pig1.pig
在此之上,还可以做条件flow:
https://github.com/azkaban/azkaban/blob/e9a02f4fff4b06476a8d3bbebbce7de1b7a6d864/docs/conditionalFlow.rst