NES::Runtime Namespace Reference

Turn this on to have Thread::current_num_threads_ keep a count of currently-active threads. More...

Namespaces
	detail
	Execution
	MemoryLayouts
	Profiler

Classes
class	AbstractBufferProvider
class	AbstractPoolProvider
class	AbstractBufferStorage
	The Abstract buffer storage class to backup tuple buffers that are passing through node engine. More...
class	AbstractLineageManager
	The Abstract Lineage Manager class is used to map of all tuples that got their sequence number changed by stateful operators. More...
class	FixedPagesAllocator
class	NesDefaultMemoryAllocator
	The default memory resource of nes that use posix_memalign. More...
class	AsyncTaskExecutor
	This is a multi-threaded task executor that execute asynchronously tasks. Submitted task are paired to a future value that can be retrieved when the async task is executed. More...
class	BloomFilter
	A bloom filter that works with 64-bit keys. More...
class	BufferManager
	The BufferManager is responsible for: More...
class	BufferRecycler
	Interface for buffer recycling mechanism. More...
struct	BufferSorter
class	BufferStorage
	The Buffer Storage class stores tuples inside a queue and trims it when the right acknowledgement is received. More...
class	BaseEvent
	This is the base event type. All events supported in NES shall inherit from this class. More...
class	CustomEventWrapper
	This class shall be used to define custom events with user-supplied data. More...
class	StartSourceEvent
	This class represents a start method for static data sources. More...
class	FixedSizeBufferPool
	A local buffer pool that uses N exclusive buffers and then falls back to the global buffer manager. More...
class	HardwareManager
	This class is responsible to look up OS/HW specs of the underlying hardware, e.g., numa. More...
class	InMemoryLineageManager
	The Lineage Manager class stores map of all tuples that got their sequence number changed by stateful operators. More...
class	LocalBufferPool
	A local buffer pool that uses N exclusive buffers and then falls back to the global buffer manager. More...
class	NesThread
class	NodeEngine
	this class represents the interface and entrance point into the query processing part of NES. It provides basic functionality such as deploying, undeploying, starting, and stopping. More...
struct	WorkerOpenCLDeviceInfo
class	OpenCLManager
class	AbstractQueryManager
class	DynamicQueryManager
class	MultiQueueQueryManager
class	QueryStatistics
class	Reconfigurable
	Nes components that require to be reconfigured at Runtime need to inherit from this class. It provides a reconfigure callback that will be called per thread and a postReconfigurationCallback that will be called on the last thread the executes the reconfiguration. More...
class	ReconfigurationMessage
	this class contains the description of the reconfiguration that must be carried out More...
class	RuntimeEventListener
	This is the listener for runtime events. More...
class	BasicPhysicalField
class	Task
	Task abstraction to bind processing (compiled binary) and data (incoming buffers @Limitations: -. More...
class	ThreadPool
	the tread pool handles the dynamic scheduling of tasks during Runtime @Limitations More...
class	TupleBuffer
	The TupleBuffer allows Runtime components to access memory to store records in a reference-counted and thread-safe manner. More...
class	WorkerContext
	A WorkerContext represents the current state of a worker thread Note that it is not thread-safe per se but it is meant to be used in a thread-safe manner by the ThreadPool. More...
class	FormatIteratorTestUtil
	Provides utility function to write multiple tuples to a test tuple buffer and to create expected output. Currently, the functionality is limited to a single buffer containing all test tuples. More...
struct	JsonKVPair
	In this test, the expected output is consists of JSON key value (KV) pairs. More...
class	JsonFormatTest
	Testing the functionality of the iterating over the json format. Since the created json objects may store the key-value-pairs in a different order, compared to our schema, we just check if the results contain the expected strings. More...
struct	OpenCLDeviceInfo

Typedefs
using	QueryStatisticsPtr = std::shared_ptr< QueryStatistics >
using	NodeEnginePtr = std::shared_ptr< NodeEngine >
using	BufferManagerPtr = std::shared_ptr< BufferManager >
using	NesDefaultMemoryAllocatorPtr = std::shared_ptr< NesDefaultMemoryAllocator >
using	AsyncTaskExecutorPtr = std::shared_ptr< AsyncTaskExecutor >
using	BufferStoragePtr = std::shared_ptr< Runtime::BufferStorage >
using	LineageManagerPtr = std::shared_ptr< Runtime::InMemoryLineageManager >
using	ThreadPoolPtr = std::shared_ptr< ThreadPool >
using	QueryManagerPtr = std::shared_ptr< AbstractQueryManager >
using	DynamicQueryManagerPtr = std::shared_ptr< DynamicQueryManager >
using	MultiQueueQueryManagerPtr = std::shared_ptr< MultiQueueQueryManager >
using	ReconfigurablePtr = std::shared_ptr< Reconfigurable >
using	RuntimeEventListenerPtr = std::shared_ptr< RuntimeEventListener >
using	PhysicalFieldPtr = std::shared_ptr< PhysicalField >
using	PhysicalSchemaPtr = std::shared_ptr< PhysicalSchema >
using	HardwareManagerPtr = std::shared_ptr< HardwareManager >
using	OpenCLManagerPtr = std::shared_ptr< OpenCLManager >
using	LocalBufferPoolPtr = std::shared_ptr< LocalBufferPool >
using	FixedSizeBufferPoolPtr = std::shared_ptr< FixedSizeBufferPool >
using	WorkerContextRef = WorkerContext &
using	WorkerContextPtr = std::shared_ptr< WorkerContext >

Enumerations
enum class	BufferManagerType : uint8_t { GLOBAL , LOCAL , FIXED }
enum class	EventType : uint8_t { kInvalidEvent , kCustomEvent , kStartSourceEvent }
	this enum defines the event that can occur in the system runtime More...
enum class	QueryTerminationType : uint8_t {   Graceful = 0 , HardStop , Failure , Reconfiguration ,   Invalid }
enum class	ReconfigurationType : uint8_t {   Initialize , Destroy , SoftEndOfStream , HardEndOfStream ,   FailEndOfStream , ConnectToNewReceiver , ConnectionEstablished , UpdateVersion ,   ReconfigurationMarker }
enum class	NumaAwarenessFlag : int8_t { ENABLED , DISABLED }

Functions
TupleBuffer	allocateVariableLengthField (std::shared_ptr< AbstractBufferProvider > provider, uint32_t size)
	This function allocates a TupleBuffer of a desired size via a provider. More...
constexpr uint32_t	alignBufferSize (uint32_t bufferSize, uint32_t withAlignment)
	Computes aligned buffer size based on original buffer size and alignment. More...
ReconfigurationType	terminationTypeToReconfigurationType (QueryTerminationType terminationType)
QueryTerminationType	reconfigurationTypeToTerminationType (ReconfigurationType reconfigurationType)
bool	recycleTupleBuffer (void *bufferPointer)
	This method determines the control block based on the ptr to the data region and decrements the reference counter. More...
template<typename T >
T *	allocateWithin (TupleBuffer &buffer)
	Allocates an object of T in the tuple buffer. Set the number of tuples to one. More...
	TEST_F (JsonFormatTest, createJsonIterator)
	Tests that we can construct a json iterator. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithASingleInteger)
	Tests that we can convert a tuple buffer with a single integer to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithUnsignedIntegers)
	Tests that we can convert a tuple buffer with unsigned integers to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithSignedIntegers)
	Tests that we can convert a tuple buffer with signed integers to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithSignedBoolAndChar)
	Tests that we can convert a tuple buffer with a lower and uppercase char and a true and a false bool to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithFloatingPoints)
	Tests that we can convert a tuple buffer with single and a double precision to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithText)
	Tests that we can convert a tuple buffer containing Text to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithNumberAndText)
	Tests that we can convert a tuple buffer with a number and text to json. More...
	TEST_F (JsonFormatTest, useJsonIteratorWithMixedDataTypes)
	Tests that we can convert a tuple buffer with many different basic types, and multiple Text types to json. More...
	NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE (OpenCLDeviceInfo, platformVendor, platformName, deviceName, doubleFPSupport, maxWorkItems, deviceAddressBits, deviceType, deviceExtensions, availableProcessors)

Variables
template<typename T >
concept	IsNesEvent = requires(T t) { t.getEventType(); }
const std::chrono::seconds	DEFAULT_BUFFER_TIMEOUT = std::chrono::seconds(10)

Detailed Description

Turn this on to have Thread::current_num_threads_ keep a count of currently-active threads.

This enum reflects the different types of buffer managers in the system global: overall buffer manager local: buffer manager that we give to the processing fixed: buffer manager that we use for sources

Typedef Documentation

AsyncTaskExecutorPtr

typedef std::shared_ptr< AsyncTaskExecutor > NES::Runtime::AsyncTaskExecutorPtr

BufferManagerPtr

typedef std::shared_ptr< BufferManager > NES::Runtime::BufferManagerPtr

BufferStoragePtr

typedef std::shared_ptr< Runtime::BufferStorage > NES::Runtime::BufferStoragePtr

DynamicQueryManagerPtr

using NES::Runtime::DynamicQueryManagerPtr = typedef std::shared_ptr<DynamicQueryManager>

FixedSizeBufferPoolPtr

using NES::Runtime::FixedSizeBufferPoolPtr = typedef std::shared_ptr<FixedSizeBufferPool>

HardwareManagerPtr

using NES::Runtime::HardwareManagerPtr = typedef std::shared_ptr<HardwareManager>

LineageManagerPtr

using NES::Runtime::LineageManagerPtr = typedef std::shared_ptr<Runtime::InMemoryLineageManager>

LocalBufferPoolPtr

using NES::Runtime::LocalBufferPoolPtr = typedef std::shared_ptr<LocalBufferPool>

MultiQueueQueryManagerPtr

using NES::Runtime::MultiQueueQueryManagerPtr = typedef std::shared_ptr<MultiQueueQueryManager>

NesDefaultMemoryAllocatorPtr

using NES::Runtime::NesDefaultMemoryAllocatorPtr = typedef std::shared_ptr<NesDefaultMemoryAllocator>

NodeEnginePtr

typedef std::shared_ptr< NodeEngine > NES::Runtime::NodeEnginePtr

OpenCLManagerPtr

using NES::Runtime::OpenCLManagerPtr = typedef std::shared_ptr<OpenCLManager>

PhysicalFieldPtr

using NES::Runtime::PhysicalFieldPtr = typedef std::shared_ptr<PhysicalField>

PhysicalSchemaPtr

using NES::Runtime::PhysicalSchemaPtr = typedef std::shared_ptr<PhysicalSchema>

QueryManagerPtr

typedef std::shared_ptr< AbstractQueryManager > NES::Runtime::QueryManagerPtr

QueryStatisticsPtr

typedef std::shared_ptr< QueryStatistics > NES::Runtime::QueryStatisticsPtr

ReconfigurablePtr

using NES::Runtime::ReconfigurablePtr = typedef std::shared_ptr<Reconfigurable>

RuntimeEventListenerPtr

typedef std::shared_ptr< RuntimeEventListener > NES::Runtime::RuntimeEventListenerPtr

ThreadPoolPtr

typedef std::shared_ptr< ThreadPool > NES::Runtime::ThreadPoolPtr

WorkerContextPtr

using NES::Runtime::WorkerContextPtr = typedef std::shared_ptr<WorkerContext>

WorkerContextRef

using NES::Runtime::WorkerContextRef = typedef WorkerContext&

Enumeration Type Documentation

BufferManagerType

enum NES::Runtime::BufferManagerType : uint8_t

strong

Enumerator
GLOBAL
LOCAL
FIXED

EventType

enum NES::Runtime::EventType : uint8_t

strong

this enum defines the event that can occur in the system runtime

Enumerator
kInvalidEvent
kCustomEvent
kStartSourceEvent

NumaAwarenessFlag

enum NES::Runtime::NumaAwarenessFlag : int8_t

strong

Enumerator
ENABLED
DISABLED

QueryTerminationType

enum NES::Runtime::QueryTerminationType : uint8_t

strong

Enumerator
Graceful
HardStop
Failure
Reconfiguration
Invalid

ReconfigurationType

enum NES::Runtime::ReconfigurationType : uint8_t

strong

Enumerator
Initialize	use Initialize for reconfiguration tasks that initialize a reconfigurable instance
Destroy	use Destroy for reconfiguration tasks that cleans up a reconfigurable instance
SoftEndOfStream	graceful stop of a query
HardEndOfStream	forceful stop of a query without a failure
FailEndOfStream	forceful stop of a query with a failure
ConnectToNewReceiver	start the process of connecting to a new network source
ConnectionEstablished	indicate successful establishment of a network connection
UpdateVersion	Indicates that the version of the operator is updated due to reconfiguration as part of query redeployment.
ReconfigurationMarker	reconfiguration marker

Function Documentation

alignBufferSize()

constexpr uint32_t NES::Runtime::alignBufferSize ( uint32_t  bufferSize, uint32_t  withAlignment  )

constexpr

Computes aligned buffer size based on original buffer size and alignment.

References NES::bufferSize.

Referenced by NES::Runtime::BufferManager::getUnpooledBuffer(), and NES::Runtime::TupleBuffer::reinterpretAsTupleBuffer().

Here is the caller graph for this function:

allocateVariableLengthField()

TupleBuffer NES::Runtime::allocateVariableLengthField	(	std::shared_ptr< AbstractBufferProvider >	provider,
		uint32_t	size
	)

This function allocates a TupleBuffer of a desired size via a provider.

Parameters

provider	the buffer allocator
size	the desired size

Returns

a tuplebuffer of size size allocated via the given provider

References NES_ASSERT2_FMT, and size().

Referenced by NES::TEST_F().

Here is the call graph for this function:

Here is the caller graph for this function:

allocateWithin()

template<typename T >

T* NES::Runtime::allocateWithin

(

TupleBuffer &

buffer

)

Allocates an object of T in the tuple buffer. Set the number of tuples to one.

Template Parameters

T

Parameters

buffer

Returns

T+

References NES::Runtime::TupleBuffer::getBuffer(), and NES::Runtime::TupleBuffer::setNumberOfTuples().

Here is the call graph for this function:

NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE()

NES::Runtime::NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE	(	OpenCLDeviceInfo	,
		platformVendor	,
		platformName	,
		deviceName	,
		doubleFPSupport	,
		maxWorkItems	,
		deviceAddressBits	,
		deviceType	,
		deviceExtensions	,
		availableProcessors
	)

reconfigurationTypeToTerminationType()

QueryTerminationType NES::Runtime::reconfigurationTypeToTerminationType

(

ReconfigurationType

reconfigurationType

)

References FailEndOfStream, Failure, Graceful, HardEndOfStream, HardStop, NES_ASSERT, Reconfiguration, ReconfigurationMarker, and SoftEndOfStream.

Referenced by NES::SinkMedium::postReconfigurationCallback(), and NES::Network::NetworkSink::reconfigure().

Here is the caller graph for this function:

recycleTupleBuffer()

bool NES::Runtime::recycleTupleBuffer

(

void *

bufferPointer

)

This method determines the control block based on the ptr to the data region and decrements the reference counter.

Parameters

bufferPointer

pointer to the data region of an buffer.

References NES_ASSERT2_FMT, and NES::Runtime::detail::BufferControlBlock::release().

Here is the call graph for this function:

terminationTypeToReconfigurationType()

ReconfigurationType NES::Runtime::terminationTypeToReconfigurationType

(

QueryTerminationType

terminationType

)

References FailEndOfStream, Failure, Graceful, HardEndOfStream, HardStop, NES_ASSERT, Reconfiguration, ReconfigurationMarker, and SoftEndOfStream.

Referenced by NES::Runtime::Execution::ExecutableQueryPlan::notifyPipelineCompletion(), NES::Runtime::Execution::ExecutableQueryPlan::notifySinkCompletion(), and NES::Runtime::Execution::ExecutableQueryPlan::notifySourceCompletion().

Here is the caller graph for this function:

TEST_F() [1/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		createJsonIterator
	)

Tests that we can construct a json iterator.

References NES::Schema::create(), and NES::UINT8.

Here is the call graph for this function:

TEST_F() [2/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithASingleInteger
	)

Tests that we can convert a tuple buffer with a single integer to json.

References NES::Schema::create(), and NES::UINT8.

Here is the call graph for this function:

TEST_F() [3/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithFloatingPoints
	)

Tests that we can convert a tuple buffer with single and a double precision to json.

References NES::Schema::create(), NES::FLOAT32, and NES::FLOAT64.

Here is the call graph for this function:

TEST_F() [4/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithMixedDataTypes
	)

Tests that we can convert a tuple buffer with many different basic types, and multiple Text types to json.

References NES::BOOLEAN, NES::CHAR, NES::Schema::create(), NES::DataTypeFactory::createText(), NES::FLOAT32, NES::FLOAT64, NES::INT16, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [5/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithNumberAndText
	)

Tests that we can convert a tuple buffer with a number and text to json.

References NES::Schema::create(), NES::DataTypeFactory::createText(), and NES::UINT8.

Here is the call graph for this function:

TEST_F() [6/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithSignedBoolAndChar
	)

Tests that we can convert a tuple buffer with a lower and uppercase char and a true and a false bool to json.

References NES::BOOLEAN, NES::CHAR, and NES::Schema::create().

Here is the call graph for this function:

TEST_F() [7/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithSignedIntegers
	)

Tests that we can convert a tuple buffer with signed integers to json.

References NES::Schema::create(), NES::INT16, NES::INT32, NES::INT64, and NES::INT8.

Here is the call graph for this function:

TEST_F() [8/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithText
	)

Tests that we can convert a tuple buffer containing Text to json.

References NES::Schema::create(), and NES::DataTypeFactory::createText().

Here is the call graph for this function:

TEST_F() [9/9]

NES::Runtime::TEST_F	(	JsonFormatTest	,
		useJsonIteratorWithUnsignedIntegers
	)

Tests that we can convert a tuple buffer with unsigned integers to json.

References NES::Schema::create(), NES::UINT16, NES::UINT32, NES::UINT64, and NES::UINT8.

Here is the call graph for this function:

Variable Documentation

DEFAULT_BUFFER_TIMEOUT

const std::chrono::seconds NES::Runtime::DEFAULT_BUFFER_TIMEOUT = std::chrono::seconds(10)

IsNesEvent

template<typename T >

concept NES::Runtime::IsNesEvent = requires(T t) { t.getEventType(); }