Configuration options

This page describes how to configure the runtime behavior of the NebulaStream Coordinator and Workers.

The NebulaStream Coordinator and Worker provide two ways to configure their runtime behavior.

1. Setting configuration options on the command line.
2. Setting configuration options in a YAML configuration file.

❗ For the NebulaStream Coordinator, configuration setting is specified on the command line take precedence over the same option in the YAML configuration file.

❗ For the NebulaStream Worker, command line options do not override the settings in the YAML configuration file.

❗ For most configuration options, the key is always the same, regardless whether it is specified on the command line or in a YAML configuration file. The exception is the configuration of logical sources in the NebulaStream coordinator and physical sources in the NebulaStream worker.

In this page, we describe the general way how to set configuration options on the command line or in a YAML configuration file. We also describe every configuration option and provide their default values.

Configuration on the Command Line #

If the NebulaStream Coordinator or Worker are started without any command line parameters, all configuration options are initialized to their default values.

To set the configuration option key to the value value, use the syntax --key=value.

💡 In the following example, the configuration option logLevel is set to LOG_INFO:

nesCoordinator --logLevel=LOG_INFO

Configuration in a YAML file #

It is also possible to create a YAML configuration file. To set configuration options set from this file, use the command line parameter configPath.

💡 In the following example, configuration options are set using the contents of the file coordinator.yml:

nesCoordinator --configPath=coordinator.yml

We provide examples of the YAML configuration files in the NebulaStream Tutorial repository:

• coordinator.yml
• worker.yml

Coordinator Configuration Options #

The configuration options of the Coordinator configure network settings, the NebulaStream optimizer, logical sources, and enable experimental features.

General Coordinator Configuration #

The following table lists general configuration options of the NebulaStream Coordinator in alphabetical order. As the Coordinator encapsulates a worker internally, it supports all worker configurations via the “worker:” configuration option.

Optimizer configuration #

The following table lists configuration options of the NebulaStream optimizer in alphabetical order. These configuration options begin with the prefix optimizer..

Internal worker configuration #

At the moment, the coordinator starts an internal worker. This internal worker can be configured in the coordinator configuration by prefixing a worker configuration option with the prefix worker.. It is also possible to extract the configuration options for the internal worker in a separate YAML configuration file, using the configuration option --workerConfigPath. The evaluation order of options which configure the internal worker are as follows:

1. A configuration option prefixed with worker. in the YAML configuration file for the coordinator.
2. The contents of a YAML configuration file that is specified by the key workerConfigPath inside the YAML configuration file of the coordinator.
3. The contents of a YAML configuration file that is specified by the key workerConfigPath on the command line when starting the coordinator.
4. A configuration option prefixed with worker. on the command line when starting the coordinator.

This order follows the expectation that options on the command line (including workerConfigPath) overwrite options in the configuration file.

There are three exceptions, which are the worker options coordinatorHost, coordinatorPort, and localWorkerHost. These are always set to the respective values of the coordinator.

Logical Sources Configuration #

Logical sources can only be configured in the YAML configuration file and not on the command line. That is because it is not possible to define multiple logical sources on the command line.

A logical source is defined by a name (logicalSourceName) and a schema. The schema consists of a number of fields that also have a name and a type. Valid types are: INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, FLOAT32, UINT64, FLOAT64, BOOLEAN, TEXT and CHAR. The TEXT type represents a variable-length string. The CHAR type represents a fixed-length string and also requires a length value. The type FLOAT32 is represented as a single precision, and the type FLOAT64 is double precision.

💡 The example below shows how to define a logical source with the name default_logical and a schema consisting of the fields id, value, and char_value.

logicalSources:
  - logicalSourceName: "default_logical"
    fields:
      - name: "id"
        type: "UINT32"
      - name: "sensor_value"
        type: "UINT64"
      - name: "char_value"
        type: "CHAR"
        length: 5

Worker Configuration Options #

The configuration options of the Worker configure network settings, the NebulaStream query compiler, physical sources, and enable experimental features.

General Worker Configuration #

The following table lists general configuration options of the NebulaStream Worker in alphabetical order.

Query Compiler Configuration #

The following table lists configuration options of the NebulaStream query compiler in alphabetical order. These configuration options begin with the prefix queryCompiler..

Physical Sources Configuration #

Physical sources can be defined both on the command line and also in the YAML configuration file.

❗ On the command line, we can only define a single physical source. In contrast, in the YAML configuration file, we can define multiple physical sources.

The following table lists the configuration options that have to be specified for every physical source. The configuration options for physical sources begin with the prefix physicalSources..

NebulaStream supports the following physical sources types:

• BinarySource: Reads data from a binary file.
• CSVSource: Reads data from a CSV file and repeats the data multiple times.
• KafkaSource: Reads data from a Kafka broker.
• MQTTSource: Reads data from a MQTT broker.
• MaterializedViewSource: Read from a materialized view.
• OPCSource: Reads data from an OPC server.

These source types require additional configuration options which we describe below.

BinarySource #

A BinarySource can be configured with the following configuration options.

CSVSource #

A CSVSource can be configured with the following configuration options.

KafkaSource #

A KafkaSource can be configured with the following configuration options.

MQTTSource #

A MQTTSource can be configured with the following configuration options.

MaterializedViewSource #

❗ This source type is experimental.

A MaterializedViewSource can be configured with the following configuration options.

OPCSource #

An OPCSource can be configured with the following configuration options.