Introduction
1. Introducing Spring Cloud Data Flow for Nomad
This project provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices to Hashicorp Nomad.
2. Spring Cloud Data Flow
Spring Cloud Data Flow is a cloud-native orchestration service for composable data microservices on modern runtimes. With Spring Cloud Data Flow, developers can create and orchestrate data pipelines for common use cases such as data ingest, real-time analytics, and data import/export.
The Spring Cloud Data Flow architecture consists of a server that deploys Streams and Tasks. Streams are defined using a DSL or visually through the browser based designer UI. Streams are based on the Spring Cloud Stream programming model while Tasks are based on the Spring Cloud Task programming model. The sections below describe more information about creating your own custom Streams and Tasks
For more details about the core architecture components and the supported features, please review Spring Cloud Data Flow’s core reference guide. There’re several samples available for reference.
3. Spring Cloud Stream
Spring Cloud Stream is a framework for building message-driven microservice applications. Spring Cloud Stream builds upon Spring Boot to create standalone, production-grade Spring applications, and uses Spring Integration to provide connectivity to message brokers. It provides opinionated configuration of middleware from several vendors, introducing the concepts of persistent publish-subscribe semantics, consumer groups, and partitions.
For more details about the core framework components and the supported features, please review Spring Cloud Stream’s reference guide.
There’s a rich ecosystem of Spring Cloud Stream Application-Starters that can be used either as standalone data microservice applications or in Spring Cloud Data Flow. For convenience, we have generated RabbitMQ and Apache Kafka variants of these application-starters that are available for use from Maven Repo and Docker Hub as maven artifacts and docker images, respectively.
Do you have a requirement to develop custom applications? No problem. Refer to this guide to create custom stream applications. There’re several samples available for reference.
4. Spring Cloud Task
Spring Cloud Task makes it easy to create short-lived microservices. We provide capabilities that allow short-lived JVM processes to be executed on demand in a production environment.
For more details about the core framework components and the supported features, please review Spring Cloud Task’s reference guide.
There’s a rich ecosystem of Spring Cloud Task Application-Starters that can be used either as standalone data microservice applications or in Spring Cloud Data Flow. For convenience, the generated application-starters are available for use from Maven Repo. There are several samples available for reference.
Features
The Data Flow Server for Nomad includes the following highlighted features.
5. Support for Maven and Docker resources
Nomad supports both Java and Docker drivers that the Data Flow server can utilise to support apps registered as Maven and Docker resources.
For example, both the below app registrations (via the Data Flow Shell) are valid and supported:
dataflow:>app register --name http-mvn --type source --uri maven://org.springframework.cloud.stream.app:http-source-rabbit:1.1.0.RELEASE
dataflow:>app import --name http-docker --type source --uri app register --name http --type source --uri docker:springcloudstream/http-source-rabbit:1.1.0.RELEASE
See the Getting Started section for examples of deploying both Docker and Maven resource types.
6. Docker volume support
Docker volume support was added in Nomad 0.5. which allows the Data Flow Server for Nomad to support defining Docker volumes, both as deployer and app deployment properties.
Volumes defined at deployer level will be added to all deployed apps. This is handy for common shared folders that should be available to all apps. |
Below is an example of volumes defined as a server deployer property:
spring.cloud.deployer.nomad:
volumes: /opt/data:/data,/opt/config:/config
where volumes are defined as a comma separated list in the form of host_path:container_path
.
Examples of the deployment property (via the Data Flow Shell) variation of defining volumes below:
dataflow:>stream create --name test --definition "time | file"
Created new stream 'timezoney'
dataflow:>stream deploy test --properties "app.file.spring.cloud.deployer.nomad.volumes=/opt/data:/data,/opt/config:/config"
See the Nomad Docker volume documentation for more details.
7. Ephemeral disks
The new ephemeral_disk stanza added in Nomad 0.5 is supported, allowing you to configure an ephemeral disk with app deployment properties.
8. App status derived from Consul
The default states for a Nomad job
are not really indicative of a healthy application.
For instance an app may appear to be running based on the Nomad Job/Allocation state
but is in fact not healthy as determined by invoking the app’s
health endpoint (e.g. /health
). For a more accurate app status, the Data Flow Server for Nomad can now
include the Consul (if available) health check status fo the registered app’s Service.
This feature is implemented using Spring Cloud Consul.
Getting Started
9. Deploying Streams on Nomad
The following guide assumes that you have a Nomad 0.5+ cluster available. If you do not have a Nomad cluster available, see the next section which describes running a local Nomad for development/testing otherwise continue to Installing the Data Flow Server for Nomad.
9.1. A local Nomad cluster with Vagrant Hashistack
There are a few ways to stand up a local Nomad cluster on your machine for testing. For the purpose of this guide, the hashistack-vagrant project will be used.
The Please see the Resource Allocations
section in the |
9.1.1. Installation and Getting Started
Follow the Quickstart section of the hashistack-vagrant.
Once you have successfully started the Vagrant VM and (with tmuxp load full-hashistack.yml
) "hashistack",
make sure you can use the nomad
client to query the local instance:
vagrant@hashistack:~$ nomad status
No running jobs
You could also install the
|
9.2. Installing the Data Flow Server for Nomad
To install a Data Flow Server and supporting infrastructure components to Nomad we will use the provided Job specification provided
in the src/etc/nomad/
directory of the project’s GitHub repository.
This Job requires the Docker driver. |
This job specification includes the following tasks:
-
Spring Cloud Data Flow Server for Nomad
-
Spring Cloud Data Flow Metrics Collector
-
MySQL - as the datasource for the Data Flow Server
-
Redis - for analytics support
-
Kafka - as the Spring Cloud Stream default binder implementation
If you are not using the hashistack-vagrant VM, please adjust the region and datacenters
values in the scdf.nomad job specification file accordingly.
|
Next, using the SSH session to the VM, run the job with:
vagrant@hashistack:~$ nomad run https://raw.githubusercontent.com/donovanmuller/spring-cloud-dataflow-server-nomad/v1.2.2.RELEASE/src/etc/nomad/nexus.nomad
==> Monitoring evaluation "491168de"
Evaluation triggered by job "scdf"
Allocation "63c64eaf" created: node "774539c7", group "scdf"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "491168de" finished with status "complete"
Allow some time for the Docker images to be pulled and all containers to be started. You can verify that all tasks have been successfully started by checking the corresponding allocation status:
vagrant@hashistack:~$ nomad alloc-status 63c64eaf
ID = 63c64eaf
Eval ID = 491168de
Name = scdf.scdf[0]
Node ID = 774539c7
Job ID = scdf
Client Status = running
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created At = 08/01/17 20:58:44 UTC
Task "kafka" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
25/500 MHz 507 MiB/512 MiB 500 MiB 0 kafka: 172.16.0.2:9092
Recent Events:
Time Type Description
08/01/17 21:02:16 UTC Started Task started by client
08/01/17 21:01:59 UTC Restarting Task restarting in 16.424762972s
08/01/17 21:01:59 UTC Terminated Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
08/01/17 21:01:58 UTC Started Task started by client
08/01/17 21:01:40 UTC Restarting Task restarting in 16.967973166s
08/01/17 21:01:40 UTC Terminated Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
08/01/17 21:01:38 UTC Started Task started by client
08/01/17 20:58:44 UTC Driver Downloading image wurstmeister/kafka:0.10.1.0
08/01/17 20:58:44 UTC Task Setup Building Task Directory
08/01/17 20:58:44 UTC Received Task received by client
Task "mysql" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
2/500 MHz 117 MiB/128 MiB 500 MiB 0 db: 172.16.0.2:3306
Recent Events:
Time Type Description
08/01/17 21:00:50 UTC Started Task started by client
08/01/17 20:58:44 UTC Driver Downloading image mysql:5.6
08/01/17 20:58:44 UTC Task Setup Building Task Directory
08/01/17 20:58:44 UTC Received Task received by client
Task "redis" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
3/256 MHz 6.2 MiB/64 MiB 500 MiB 0 redis: 172.16.0.2:6379
Recent Events:
Time Type Description
08/01/17 20:58:57 UTC Started Task started by client
08/01/17 20:58:44 UTC Driver Downloading image redis:3-alpine
08/01/17 20:58:44 UTC Task Setup Building Task Directory
08/01/17 20:58:44 UTC Received Task received by client
Task "scdf-metrics-collector" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
518/500 MHz 507 MiB/512 MiB 500 MiB 0 http: 172.16.0.2:8080
Recent Events:
Time Type Description
08/01/17 21:02:10 UTC Started Task started by client
08/01/17 21:01:54 UTC Restarting Task restarting in 15.187972956s
08/01/17 21:01:54 UTC Terminated Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
08/01/17 21:01:34 UTC Started Task started by client
08/01/17 21:01:17 UTC Restarting Task restarting in 16.424762972s
08/01/17 21:01:17 UTC Terminated Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
08/01/17 21:00:57 UTC Started Task started by client
08/01/17 21:00:39 UTC Restarting Task restarting in 16.967973166s
08/01/17 21:00:39 UTC Terminated Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
08/01/17 21:00:16 UTC Started Task started by client
Task "scdf-server" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
5/500 MHz 323 MiB/384 MiB 500 MiB 0 http: 172.16.0.2:9393
Recent Events:
Time Type Description
08/01/17 21:11:26 UTC Started Task started by client
08/01/17 21:11:08 UTC Restarting Task restarting in 17.905628447s
08/01/17 21:11:08 UTC Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
08/01/17 21:05:27 UTC Started Task started by client
08/01/17 21:05:09 UTC Restarting Task restarting in 18.119676483s
08/01/17 21:05:09 UTC Driver Failure failed to initialize task "scdf-server" for alloc "63c64eaf-5a43-3e18-8e55-0134f8c18f93": Failed to pull `donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE`: API error (404): {"message":"manifest for donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE not found"}
08/01/17 21:05:05 UTC Driver Downloading image donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE
08/01/17 21:04:47 UTC Restarting Task restarting in 17.088216893s
08/01/17 21:04:47 UTC Driver Failure failed to initialize task "scdf-server" for alloc "63c64eaf-5a43-3e18-8e55-0134f8c18f93": Failed to pull `donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE`: API error (404): {"message":"manifest for donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE not found"}
08/01/17 21:04:42 UTC Driver Downloading image donovanmuller/spring-cloud-dataflow-server-nomad:1.2.3.RELEASE
Task "zookeeper" is "running"
Task Resources
CPU Memory Disk IOPS Addresses
2/500 MHz 45 MiB/128 MiB 500 MiB 0 zookeeper: 172.16.0.2:2181
follower: 172.16.0.2:2888
leader: 172.16.0.2:3888
Recent Events:
Time Type Description
08/01/17 21:02:12 UTC Started Task started by client
08/01/17 20:58:44 UTC Driver Downloading image digitalwonderland/zookeeper:latest
08/01/17 20:58:44 UTC Task Setup Building Task Directory
08/01/17 20:58:44 UTC Received Task received by client
...
or alternatively check the health status of all services using the Consul UI:
If you are using a local
|
9.3. Download and run the Spring Cloud Data Flow Shell
Download and run the Shell, targeting the Data Flow Server exposed via a Fabio route.
$ wget http://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/1.2.3.RELEASE/spring-cloud-dataflow-shell-1.2.3.RELEASE.jar
$ java -jar spring-cloud-dataflow-shell-1.2.3.RELEASE.jar \
--dataflow.uri=http://scdf-server.hashistack.vagrant/ \
--dataflow.username=user \
--dataflow.password=password
____ ____ _ __
/ ___| _ __ _ __(_)_ __ __ _ / ___| | ___ _ _ __| |
\___ \| '_ \| '__| | '_ \ / _` | | | | |/ _ \| | | |/ _` |
___) | |_) | | | | | | | (_| | | |___| | (_) | |_| | (_| |
|____/| .__/|_| |_|_| |_|\__, | \____|_|\___/ \__,_|\__,_|
____ |_| _ __|___/ __________
| _ \ __ _| |_ __ _ | ___| | _____ __ \ \ \ \ \ \
| | | |/ _` | __/ _` | | |_ | |/ _ \ \ /\ / / \ \ \ \ \ \
| |_| | (_| | || (_| | | _| | | (_) \ V V / / / / / / /
|____/ \__,_|\__\__,_| |_| |_|\___/ \_/\_/ /_/_/_/_/_/
1.2.3.RELEASE
Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
dataflow:>
Security is enabled by default. See the Security configuration section for more information. There are two default users created:
|
9.4. Registering Stream applications with Docker resource
Now register all out-of-the-box stream applications using the Docker resource type, built with the Kafka binder in bulk with the following command.
For more details, review how to register applications. |
dataflow:>app import --uri http://bit.ly/Bacon-RELEASE-stream-applications-kafka-10-docker
Successfully registered applications: [source.tcp, sink.jdbc, source.http, sink.rabbit, source.rabbit, source.ftp, sink.gpfdist, processor.transform, source.loggregator, source.sftp, processor.filter, sink.cassandra, processor.groovy-filter, sink.router, source.trigger, sink.hdfs-dataset, processor.splitter, source.load-generator, processor.tcp-client, source.time, source.gemfire, source.twitterstream, sink.tcp, source.jdbc, sink.field-value-counter, sink.redis-pubsub, sink.hdfs, processor.bridge, processor.pmml, processor.httpclient, source.s3, sink.ftp, sink.log, sink.gemfire, sink.aggregate-counter, sink.throughput, source.triggertask, sink.s3, source.gemfire-cq, source.jms, source.tcp-client, processor.scriptable-transform, sink.counter, sink.websocket, source.mongodb, source.mail, processor.groovy-transform, source.syslog]
9.5. Deploy a simple stream in the shell
Create a simple ticktock
stream definition and deploy it immediately using the following command:
dataflow:>stream create --name ticktock --definition "time | log" --deploy
Created new stream 'ticktock'
Deployment request has been sent
Verify the deployed apps using the by checking the status of the apps using the Shell:
To verify that the stream is working as expected, tail the logs of the ticktock-log
using nomad
:
vagrant@hashistack:~$ nomad logs 71f7aba1
...
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:49:59
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:01
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:02
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:03
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:04
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:05
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 14:50:06
...
9.6. Registering Stream applications with Maven resource
The Data Flow Server for Nomad also supports apps registered with a Maven resource URI
in addition to the Docker resource type. Using the ticktock
stream example above, we will create a similar stream definition
but using the Maven resource versions of the apps.
For this example we will register the apps individually using the following command:
dataflow:>app register --type source --name time-mvn --uri maven://org.springframework.cloud.stream.app:time-source-kafka:1.2.3.RELEASE
Successfully registered application 'source:time-mvn'
dataflow:>app register --type sink --name log-mvn --uri maven://org.springframework.cloud.stream.app:log-sink-kafka:1.2.3.RELEASE
Successfully registered application 'sink:log-mvn'
We couldn’t bulk import the Maven version of the apps as we did for the Docker versions because the app names
would conflict, as the names defined in the bulk import files are the same across resource types. Hence we register the
Maven apps with a -mvn suffix.
|
9.7. Deploy a simple stream in the shell
Create a simple ticktock-mvn
stream definition and deploy it immediately using the following command:
dataflow:>stream create --name ticktock-mvn --definition "time-mvn | log-mvn" --deploy
Created new stream 'ticktock-mvn'
Deployment request has been sent
There could be a slight delay once the above command is issued. This is due to the Maven artifacts being resolved and cached locally. Depending on the size of the artifacts, this could take some time. |
To verify that the stream is working as expected, tail the logs of the ticktock-mvn-log-mvn
using nomad
:
$ nomad logs -f 3f474cc7
...
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 18:34:23
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 18:34:25
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 18:34:26
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 18:34:27
10. Deploying Tasks on Nomad
Deploying Task applications using the Data Flow Server for Nomad is a similar affair to deploying Stream apps. Therefore, for brevity, only the Maven resource version of the task will be shown as an example.
10.1. Registering Task application with Maven resource
This time we will bulk import the Task application, as we do not have any Docker resource versions imported which would cause conflicts in naming. Import all Maven task applications with the following command:
dataflow:>app import --uri http://bit.ly/Belmont-GA-task-applications-maven
10.2. Launch a simple task in the shell
Let’s create a simple task definition and launch it.
dataflow:>task create task1 --definition "timestamp"
dataflow:>task launch task1
Verify that the task executed successfully by executing these commands:
dataflow:>task list
╔═════════╤═══════════════╤═══════════╗
║Task Name│Task Definition│Task Status║
╠═════════╪═══════════════╪═══════════╣
║task1 │timestamp │unknown ║
╚═════════╧═══════════════╧═══════════╝
dataflow:>task execution list
╔═════════╤══╤═════════════════════════════╤═════════════════════════════╤═════════╗
║Task Name│ID│ Start Time │ End Time │Exit Code║
╠═════════╪══╪═════════════════════════════╪═════════════════════════════╪═════════╣
║task1 │1 │Wed Aug 02 15:34:01 SAST 2017│Wed Aug 02 15:34:01 SAST 2017│0 ║
╚═════════╧══╧═════════════════════════════╧═════════════════════════════╧═════════╝
You can also view the task execution status by using the Data Flow Server UI.
10.2.1. Cleanup completed tasks
If you want to delete the Build and Pod created by this task execution, execute the following:
dataflow:>task destroy --name task1
Configuration
The Data Flow Server for Nomad supports all the common configuration options. See NomadDeployerProperties for the supported deployer configuration items.
11. Maven Configuration
The Maven configuration is important for resolving Maven app artifacts.
The following example configures a spring
remote Maven repository:
maven:
remote-repositories.spring:
url: http://repo.spring.io/libs-snapshot
auth:
username:
password:
More configuration options can be seen in the Configure Maven Properties section in the Data Flow reference documentation. |
Server Implementation
12. Server Properties
The following properties can be used to configure the Data Flow Nomad Server.
Name | Usage Example | Description |
---|---|---|
API Hostname/IP |
|
The hostname/IP address where a Nomad client is listening. Default is |
API Port |
|
The port where a Nomad client is listening. Default is |
Region |
|
The region to deploy apps into. Default to |
Datacenters |
|
A comma separated list of datacenters that should be targeted for deployment. Default value is |
Priority |
|
The default job priority. Default value is 50. See here |
Environment Variables |
|
Common environment variables to set for any deployed app. |
Expose app via Fabio |
|
Flag to indicate whether an app should be exposed via Fabio |
HTTP Health Check - Path |
|
The path of the HTTP endpoint which Consul (if available) will query to query the health |
HTTP Health Check - Interval |
|
This indicates the frequency of the health checks that Consul will perform. Specified in milliseconds. See here |
HTTP Health Check - Timeout |
|
This indicates how long Consul will wait for a health check query to succeed. Specified in milliseconds. See here |
Resource - CPU |
|
The CPU required in MHz. Default is 1000MHz |
Resource - Memory |
|
The memory required in MB. Default is 512MB |
Resource - Network |
|
The number of MBits in bandwidth required |
Ephemeral Disk - Sticky |
|
Specifies that Nomad should make a best-effort attempt to place the updated allocation on the same machine. See here |
Ephemeral Disk - Migrate |
|
Specifies that the Nomad client should make a best-effort |
Ephemeral Disk - Size |
|
Specifies the size of the ephemeral disk in MB |
Logging - Max Files |
|
The maximum number of rotated files Nomad will retain. The default is 1 log file retention size |
Logging - Max File Size |
|
The size of each rotated file. The size is specified in MB. The default is 10MB max log file size |
Restart Policy - Delay |
|
A duration to wait before restarting a task. Specified in milliseconds. Default is 30000 milliseconds (30 seconds). See here |
Restart Policy - Interval |
|
The Interval begins when the first task starts and ensures that only X number of attempts number of. Specified in milliseconds. Default is 120000 milliseconds (120 seconds / 3 minutes). See here |
Restart Policy - Attempts |
|
Attempts is the number of restarts allowed in an Interval. Default is 3 attempts within 3 minutes. See here |
Restart Policy - Mode |
|
Mode is given as a string and controls the behavior when the task fails more than X number of attempts times in an Interval. Default value is "delay". See here |
Docker - Entrypoint Style |
|
Entry point style used for the Docker image. To be used to determine how to pass in properties |
Docker - Volumes |
|
A comma separated list of |
Maven - Artifact Destination |
|
The destination (path) where artifacts will be downloaded by default. Default value is |
Maven - Java Options |
|
A comma separated list of default Java options to pass to the JVM. Only applicable to the Maven resource deployer implementation. See for reference |
Maven - Deployer Scheme |
|
The URI scheme that the deployer server is running on. When deploying Maven resource based apps the artifact source URL includes the servers host and port. This property value is used when constructing the source URL. See here |
Maven - Deployer Host |
|
The resolvable hostname of IP address that the deployer server is running on. When deploying Maven resource based apps the artifact source URL includes the servers host and port. This property value is used when constructing the source URL. See here |
Maven - Deployer Port |
|
The port that the deployer server is listening on. When deploying Maven resource based apps the artifact source URL includes the servers host and port. This property value is used when constructing the source URL. See here |
Maven - Minimum Java Version |
|
If set, the allocated node must support at least this version of a Java runtime environment. E.g. '1.8' for a minimum of a Java 8 JRE/JDK. See here |
Deployment Properties
The following deployment properties are supported by the Data Flow Server for Nomad. These properties are passed as deployment properties when deploying streams or tasks. Below is an example of deploying a stream definition:
dataflow:>stream create --name test --definition "time | custom | log"
Created new stream 'test'
dataflow:>stream deploy test --properties "app.custom.spring.cloud.deployer.nomad.job.priority=75"
Deployment request has been sent for stream 'test'
Note the deployment property app.custom.spring.cloud.deployer.nomad.job.priority=75
.
13. Supported Deployment Properties
Name | Usage Example | Description |
---|---|---|
Job Priority |
|
Job priority. See here |
Fabio - Expose flag |
|
A flag to indicate whether the tags/labels that enable Fabio to configure routing are added or not. Specify a value of |
Fabio - Hostname |
|
The hostname that will be exposed via Fabio. If no hostname is provided, the deploymentId will be used. See here |
Resources - CPU |
|
The CPU required in MHz |
Resources - Memory |
|
The memory required in MB |
Environment Variables |
|
Environment variables passed at deployment time. This is to cater for adding variables like JAVA_OPTS to supported deployer types |
Docker - Entrypoint Style |
|
Entry point style used for the Docker image. To be used to determine how to pass in properties |
Docker - Volumes |
|
A comma separated list of |
Meta |
|
An optional comma separated list of meta stanzas to add at the Job level |
Ephemeral Disk - Sticky |
|
Specifies that Nomad should make a best-effort attempt to place the updated allocation on the same machine. See here |
Ephemeral Disk - Migrate |
|
Specifies that the Nomad client should make a best-effort |
Ephemeral Disk - Size |
|
Specifies the size of the ephemeral disk in MB |
Maven - Java Options |
|
A comma separated list of default Java options to pass to the JVM. Only applicable to the Maven resource deployer implementation. See for reference |
‘How-to’ guides
This section provides answers to some common ‘how do I do that…’ type of questions that often arise when using Spring Cloud Data Flow.
14. Deploying Custom Stream App as a Maven Resource
This section walks you through deploying a simple Spring Cloud Stream based application, packaged as a Maven artifact, with Nomad. The source code for this app is available in the following GitHub repository.
This guide assumes that you have gone through the Getting Started section and are using a local hashistack-vagrant environment. Adjust the steps accordingly if you are using an existing Nomad cluster.
14.1. Deploy a Nexus Repository
For Nomad to deploy the Maven artifact, it must be able to resolve and download the custom app’s Jar artifact. This means that the custom app must be deployed to an accessible Maven repository.
We will deploy a Nexus job specification to which we can deploy our custom application. Deploying the Nexus image is trivial thanks to a provided Nexus job specification available here.
Using nomad
, run the Nexus job with:
vagrant@hashistack:~$ nomad run https://raw.githubusercontent.com/donovanmuller/spring-cloud-dataflow-server-nomad/v1.2.2.RELEASE/src/etc/nomad/nexus.nomad
...
If you are using a local
|
Wait for the Nexus image to be pulled and the deployment to be successful. Once the task is successfully running you should be able to access the Nexus UI at nexus.hashistack.vagrant.
The default credential for Nexus is admin /admin123 or deployment /deployment123
|
14.2. Configuring the Data Flow Server for Nomad
We need to configure the Data Flow Server to use this new Nexus instance as a remote Maven repository. If you have an existing deployment from the Getting Started section you will have to change it’s configuration.
You could edit the scdf.nomad
job specification and rerun the job but possibly the easiest way to
include the Nexus configuration is to remove the existing scdf
job and run the scdf-with-nexus.nomad
job specification.
vagrant@hashistack:~$ nomad stop scdf
...
vagrant@hashistack:~$ nomad run https://raw.githubusercontent.com/donovanmuller/spring-cloud-dataflow-server-nomad/v1.2.2.RELEASE/src/etc/nomad/scdf-with-nexus.nomad
...
Wait for the job and all tasks to be in a running state. You can verify that everything including Nexus has been started by checking Consul:
14.3. Cloning and Deploying the App
Next step is to deploy our custom app into the Nexus instance. First though, we need to clone the custom app source.
$ git clone https://github.com/donovanmuller/timezone-processor-kafka.git
$ cd timezone-processor-kafka
Next we deploy the application into our Nexus repository with
$ ./mvnw -s .settings.xml deploy -Dnexus.url=http://nexus.hashistack.vagrant/content/repositories/snapshots
...
Uploading: http://nexus.hashistack.vagrant/content/repositories/snapshots/io/switchbit/timezone-processor-kafka/maven-metadata.xml
Uploaded: http://nexus.hashistack.vagrant/content/repositories/snapshots/io/switchbit/timezone-processor-kafka/maven-metadata.xml (294 B at 9.3 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.171 s
[INFO] Finished at: 2017-08-02T15:39:43+02:00
[INFO] Final Memory: 25M/326M
[INFO] ------------------------------------------------------------------------
Substitute the value for -Dnexus.url with the URL matching your Nexus instance.
|
14.4. Deploying the Stream
Now that our custom app is ready, let’s register it with the Data Flow Server.
Using the Data Flow Shell, targeted to our Nomad instance, register the timezone
app with:
dataflow:>app register --name timezone --type processor --uri maven://io.switchbit:timezone-processor-kafka:1.0-SNAPSHOT
Successfully registered application 'processor:timezone'
The assumption is that the out-of-the-box apps have been imported previously as part of the Getting Started section. If the apps are not imported, import them now with:
It does not really matter whether the Docker or Maven out-of-the-box apps are registered. |
Now we can define a stream using our timezone processor with:
dataflow:>stream create --name timezoney --definition "time | timezone | log"
Created new stream 'timezoney'
and deploy it with:
dataflow:>stream deploy timezoney --properties "app.timezone.timezone=Africa/Johannesburg"
Deployment request has been sent for stream 'timezoney'
The provided deployment property (app.timezone.timezone=Africa/Johannesburg ) is the required timezone to convert the input times too.
|
Check the jobs created with:
vagrant@hashistack:~$ nomad status
ID Type Priority Status
nexus service 50 running
scdf-nexus service 50 running
timezoney-log service 50 running
timezoney-time service 50 running
timezoney-timezone service 50 running
or check the Consul health checks for the timezoney-*
stream apps:
View both the timezoney-timezone-0
and timezoney-log-0
apps for the expected log outputs.
vagrant@hashistack:~$ nomad logs -f 35c70aec
...
... INFO 17275 --- [afka-consumer-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 : partitions revoked:[timezoney.time-0]
... INFO 17275 --- [afka-consumer-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 : partitions assigned:[timezoney.time-0]
... INFO 17275 --- [afka-listener-2] io.switchbit.TimezoneProcessor : Converting time '02/08/17 20:01:29' to timezone: 'Africa/Johannesburg'
... INFO 17275 --- [afka-listener-2] io.switchbit.TimezoneProcessor : Converting time '02/08/17 20:01:30' to timezone: 'Africa/Johannesburg'
... INFO 17275 --- [afka-listener-2] io.switchbit.TimezoneProcessor : Converting time '02/08/17 20:01:31' to timezone: 'Africa/Johannesburg'
... INFO 17275 --- [afka-listener-2] io.switchbit.TimezoneProcessor : Converting time '02/08/17 20:01:32' to timezone: 'Africa/Johannesburg'
... INFO 17275 --- [afka-listener-2] io.switchbit.TimezoneProcessor : Converting time '02/08/17 20:01:33' to timezone: 'Africa/Johannesburg'
vagrant@hashistack:~$ nomad logs -f 047ef240
...
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 22:03:17
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 22:03:18
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 22:03:19
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 22:03:20
... INFO 1 --- [afka-listener-1] log-sink : 02/08/17 22:03:21
Once you’re done, destroy the stream with:
dataflow:>stream destroy timezoney
Destroyed stream 'timezoney'