Usage

NebulaStream comes with two executables: nesCoordinator for the NebulaStream Coordinator and nesWorker for NebulaStream Workers. Both can be either started:

  1. without any parameter (a default configuration is taken)
  2. using an explicit configuration, --configPath=$CONFIG_PATH
  3. by providing command line parameter as --key=value pair Note that, if command line parameter, as well as a config file, are supplied, the configuration file parameter will be overwritten by the command line parameters.

NebulaStream Coordinator

Each NebulaStream instance allows only one NebulaStream Coordinator.

The NebulaStream Coordinator is started by:

./nesCoordinator --configPath=$CONFIG_PATH
or
./nesCoordinator --key=value ...

For example:

  • ./nesCoordinator # start using default config
  • ./nesCoordinator --configPath=/home/user/config.yaml # start using user provided config
  • ./nesCoordinator --restPort=1234 --rpcPort=1235 --dataPort=1236 # start and overwrite ports with user defined values

An overview of the available parameter is listed configuration. For networking/port forwarding, the NebulaStream Coordinator will open three ports:

  • restPort: Users can interact with the NebulaStream Coordinator over this port, e.g., using the UI, clients, or Rest API directly.
  • rpcPort: This port is used to connect the NebulaStream Coordinator with all workers and is required to sent control messages, e.g., start/stop query.
  • dataPort: This port is used to send data to the NebulaStream Coordinator from the NebulaStream Worker.

NebulaStream Worker

Each NebulaStream instance allows multiple NebulaStream Worker.

The NebulaStream Worker is started by:

./nesWorker --configPath=$CONFIG_PATH
or
./nesWorker --key=value ...

For example:

  • ./nesWorker # start with default config
  • ./nesWorker --configPath=/home/user/config.yaml # start using user provided config
  • ./nesCoordinator --coordinatorIp=127.0.0.1 --coordinatorPort=1235 #start and connect to coordinator with ip and port

An overview of the available parameter is listed configuration. In addition to general parameters, the worker config can specify multiple sources that are attached to the worker node.

Connection

Two mandatory parameters have to be set for the worker: the address (ip and port) on which it connects to a NebulaStream Coordinator.

  1. --coordinatorIp=IP_ADDRESS specifies the IP address of the coordinator
  2. --coordinatorPort=PORT specifies the port on which the coordinator listen to request

These values can either be set in the coordinator config, of if left default, can be seen in the coordinator log.

Note that, in the current setup, each nesWorker has to connect to the coordinator on startup, if the connection fails, the nesWorker will shut down. In addition, a worker will automatically shut down if it loses the connection to the coordinator.

Contributing Data

In addition to contribute processing resources to NebulaStream, each nesWorker can provide data, e.g., from its attached sensors. To this end, a nesWorker has to specify two properties:

  1. a logical source
  2. a physical source

A logical source specifies the overall stream to which this source wants to contribute. It has to be created before usage, e.g, via the Rest Interface or at startup of the coordinator.

The physical source defines all parameters of this source, e.g, the type (MQTT, CSV file), as well as source specific parameter like the file path or delimiter.

Setup Prebuild Executables

NebulaStream provides prebuild executables in the form of docker images and DEB packages for Ubuntu.

Docker

The Executable image starts a standalone-cluster of NebulaStream with pre-configured parameters. This image is automatically updated, whenever a new version of NebulaStream is released. See Dockerhub for details (https://hub.docker.com/r/nebulastream/nes-executable-image). To start the NES execution engine, please execute:

docker run -d -p 8081:8081 nebulastream/nes-executable-image:latest

To explore the logs of the docker container, please execute:

docker logs <container_name>

💡 port 8081 is exposed so that you can submit the query using NebulaStream Java Client or via a REST client.

Ubuntu

We provided prebuild Debian packages for ubuntu 18.04, ubuntu 20.04, and ubuntu 22.04 on x64 and for ubuntu 20.04 on arm64.

https://github.com/nebulastream/nebulastream-dev-builds

As prerequisites you have to install the following dependencies:

gcc-9
libnuma-dev
libomp-12-dev
libdw-dev
iproute2
ifupdown
Edit this page on GitHub
← Tutorial
Configuration options →