NebulaStream comes with two executables: nesCoordinator for the NebulaStream Coordinator and nesWorker for NebulaStream Workers. Both can be either started:
- without any parameter (a default configuration is taken)
- using an explicit configuration,
- by providing command line parameter as
--key=valuepair Note that, if command line parameter, as well as a config file, are supplied, the configuration file parameter will be overwritten by the command line parameters.
Each NebulaStream instance allows only one NebulaStream Coordinator.
The NebulaStream Coordinator is started by:
./nesCoordinator --configPath=$CONFIG_PATH or ./nesCoordinator --key=value ...
./nesCoordinator # start using default config
./nesCoordinator --configPath=/home/user/config.yaml # start using user provided config
./nesCoordinator --restPort=1234 --rpcPort=1235 --dataPort=1236 # start and overwrite ports with user defined values
An overview of the available parameter is listed configuration. For networking/port forwarding, the NebulaStream Coordinator will open three ports:
- restPort: Users can interact with the NebulaStream Coordinator over this port, e.g., using the UI, clients, or Rest API directly.
- rpcPort: This port is used to connect the NebulaStream Coordinator with all workers and is required to sent control messages, e.g., start/stop query.
- dataPort: This port is used to send data to the NebulaStream Coordinator from the NebulaStream Worker.
Each NebulaStream instance allows multiple NebulaStream Worker.
The NebulaStream Worker is started by:
./nesWorker --configPath=$CONFIG_PATH or ./nesWorker --key=value ...
./nesWorker # start with default config
./nesWorker --configPath=/home/user/config.yaml # start using user provided config
./nesCoordinator --coordinatorIp=127.0.0.1 --coordinatorPort=1235 #start and connect to coordinator with ip and port
An overview of the available parameter is listed configuration. In addition to general parameters, the worker config can specify multiple sources that are attached to the worker node.
Two mandatory parameters have to be set for the worker: the address (ip and port) on which it connects to a NebulaStream Coordinator.
--coordinatorIp=IP_ADDRESSspecifies the IP address of the coordinator
--coordinatorPort=PORTspecifies the port on which the coordinator listen to request
These values can either be set in the coordinator config, of if left default, can be seen in the coordinator log.
Note that, in the current setup, each nesWorker has to connect to the coordinator on startup, if the connection fails, the nesWorker will shut down. In addition, a worker will automatically shut down if it loses the connection to the coordinator.
In addition to contribute processing resources to NebulaStream, each nesWorker can provide data, e.g., from its attached sensors. To this end, a nesWorker has to specify two properties:
A logical source specifies the overall stream to which this source wants to contribute. It has to be created before usage, e.g, via the Rest Interface or at startup of the coordinator.
The physical source defines all parameters of this source, e.g, the type (MQTT, CSV file), as well as source specific parameter like the file path or delimiter.
Setup Prebuild Executables
NebulaStream provides prebuild executables in the form of docker images and
DEB packages for Ubuntu.
The Executable image starts a standalone-cluster of NebulaStream with pre-configured parameters. This image is automatically updated, whenever a new version of NebulaStream is released. See Dockerhub for details (https://hub.docker.com/r/nebulastream/nes-executable-image). To start the NES execution engine, please execute:
docker run -d -p 8081:8081 nebulastream/nes-executable-image:latest
To explore the logs of the docker container, please execute:
docker logs <container_name>
💡 port 8081 is exposed so that you can submit the query using NebulaStream Java Client or via a REST client.
We provided prebuild Debian packages for
ubuntu 20.04, and
ubuntu 22.04 on x64 and for
ubuntu 20.04 on arm64.
As prerequisites you have to install the following dependencies:
gcc-9 libnuma-dev libomp-12-dev libdw-dev iproute2 ifupdown