NES::Util Namespace Reference

Namespaces
	detail

Classes
class	FileMutex
	This is a mutex that uses files to perform locking. It implements the Mutex named requirements. More...
class	LoadedPlugin
class	PluginLoader
class	PluginRegistry
	The plugin registry allows the dynamic registration of plugins at runtime. A plugin is a provider of a specific type T, which defines the plugin interface. Plugins use [[maybe_unused]] static T::Add<PluginXType> pluginX; to register them self to the plugin. More...
class	NamedPluginRegistry
	The plugin registry allows the dynamic registration of plugins at runtime. A plugin is a provider of a specific type T, which defines the plugin interface. Plugins use [[maybe_unused]] static T::Add<PluginXType> pluginX; to register them self to the registry. More...
class	PluginFactory
	The plugin factory allows the dynamic registration of plugins at that dynamically create instances of the plugin at runtime. A plugin is a provider of a specific type T, which defines the plugin interface. Plugins use [[maybe_unused]] static T::Add<PluginXType> pluginX; to register them self to the registry. More...
class	Subprocess
	This class spawns a new subprocess and executes a specific command. As soon as the subprocess object goes out of scope the subprocess is terminated. More...

Enumerations
enum class	ends_of_pipe : uint8_t { READ = 0 , WRITE = 1 }

Functions
std::string	printTupleBufferAsText (Runtime::TupleBuffer &buffer)
	Outputs a tuple buffer in text format. More...
std::string	printTupleBufferAsCSV (Runtime::TupleBuffer tbuffer, const SchemaPtr &schema, const std::string &lineSuffix="")
	create CSV lines from the tuples More...
std::vector< PhysicalTypePtr >	getPhysicalTypes (SchemaPtr schema)
	Returns the physical types of all fields of the schema. More...
std::string	toCSVString (const SchemaPtr &schema)
	method to get the schema as a csv string More...
Runtime::MemoryLayouts::MemoryLayoutPtr	createMemoryLayout (SchemaPtr schema, uint64_t bufferSize)
	Creates a memory layout from the schema and the buffer Size. More...
bool	assignPropertiesToQueryOperators (const QueryPlanPtr &queryPlan, std::vector< std::map< std::string, std::any >> properties)
std::vector< Runtime::TupleBuffer >	createBuffersFromCSVFile (const std::string &csvFile, const SchemaPtr &schema, Runtime::BufferManagerPtr bufferManager, const std::string &timeStampFieldName, uint64_t lastTimeStamp)
	Creates multiple TupleBuffers from the csv file until the lastTimeStamp has been read. More...
std::string	escapeJson (const std::string &str)
	escapes all non text characters in a input string, such that the string could be processed as json. More...
std::string_view	trimWhiteSpaces (std::string_view in)
	removes leading and trailing whitespaces More...
std::string_view	trimChar (std::string_view in, char trimFor)
	removes leading and trailing occurences of trimFor More...
bool	endsWith (const std::string &fullString, const std::string &ending)
	Checks if a string ends with a given string. More...
uint64_t	numberOfUniqueValues (std::vector< uint64_t > &values)
	Checks if a string starts with a given string. More...
bool	startsWith (const std::string &fullString, const std::string &ending)
	Get number of unique elements. More...
std::string	toUpperCase (std::string string)
	transforms the string to an upper case version More...
template<typename T >
std::vector< T >	splitWithStringDelimiter (std::string_view inputString, std::string_view delim, std::function< T(std::string_view)> fromStringToT=detail::SplitFunctionHelper< T >::FUNCTION)
	splits a string given a delimiter into multiple substrings stored in a T vector the delimiter is allowed to be a string rather than a char only. More...
template<typename T >
std::shared_ptr< T >	checkNonNull (std::shared_ptr< T > ptr, const std::string &errorMessage)
	this method checks if the object is null More...
void	findAndReplaceAll (std::string &data, const std::string &toSearch, const std::string &replaceStr)
	function to replace all string occurrences More...
std::string	replaceFirst (std::string origin, const std::string &search, const std::string &replace)
	This function replaces the first occurrence of search term in a string with the replace term. More...
std::string	updateSourceName (std::string queryPlanSourceConsumed, std::string subQueryPlanSourceConsumed)
	: Update the source names by sorting and then concatenating the source names from the sub- and query plan More...
void	writeHeaderToCsvFile (const std::string &csvFileName, const std::string &header)
	Truncates the file and then writes the header string as is to the file. More...
void	writeRowToCsvFile (const std::string &csvFileName, const std::string &row)
	Appends the row as is to the csv file. More...
template<typename T >
std::vector< std::vector< T > >	partition (const std::vector< T > &vec, size_t n)
template<typename T >
void	padVectorToSize (std::vector< T > &vector, size_t newSize, T newValue)
	appends newValue until the vector contains a minimum of newSize elements More...
uint64_t	murmurHash (uint64_t key)
	hashes the key with murmur hash More...
uint64_t	countLines (const std::string &str)
	Counts the number of lines of a string. More...
uint64_t	countLines (std::istream &stream)
	Counts the number of lines of a stream, e.g., a file. More...
template<typename T >
void	updateAtomicMax (std::atomic< T > &curVal, const T &newVal)
	Tries to update curVal until it succeeds or curVal is larger then newVal. More...

Enumeration Type Documentation

ends_of_pipe

enum NES::Util::ends_of_pipe : uint8_t

strong

Enumerator
READ
WRITE

Function Documentation

assignPropertiesToQueryOperators()

bool NES::Util::assignPropertiesToQueryOperators	(	const QueryPlanPtr &	queryPlan,
		std::vector< std::map< std::string, std::any >>	properties
	)

Parameters

queryPlan	queryIdAndCatalogEntryMapping to which the properties are assigned
properties	properties to assign

Returns

true if the assignment success, and false otherwise

References NES_ERROR.

Referenced by NES::TEST_F().

Here is the caller graph for this function:

checkNonNull()

template<typename T >

std::shared_ptr<T> NES::Util::checkNonNull	(	std::shared_ptr< T >	ptr,
		const std::string &	errorMessage
	)

this method checks if the object is null

Returns

pointer to the object

References NES_ASSERT.

countLines() [1/2]

uint64_t NES::Util::countLines

(

const std::string &

str

)

Counts the number of lines of a string.

Parameters

str

Returns

number of lines

Referenced by NES::Runtime::Execution::JoinDeploymentTest::runAndValidateJoinQueryTwoLogicalStreams(), NES::Runtime::Execution::MultipleJoinsTest::runJoinQuery(), NES::FileSinkIntegrationTest::runQueryAndVerifyExpectedResults(), and NES::TEST_F().

Here is the caller graph for this function:

countLines() [2/2]

uint64_t NES::Util::countLines

(

std::istream &

stream

)

Counts the number of lines of a stream, e.g., a file.

Parameters

stream

Returns

number of lines

createBuffersFromCSVFile()

std::vector< Runtime::TupleBuffer > NES::Util::createBuffersFromCSVFile	(	const std::string &	csvFile,
		const SchemaPtr &	schema,
		Runtime::BufferManagerPtr	bufferManager,
		const std::string &	timeStampFieldName,
		uint64_t	lastTimeStamp
	)

Creates multiple TupleBuffers from the csv file until the lastTimeStamp has been read.

Parameters

csvFile
schema
timeStampFieldName
lastTimeStamp
bufferManager

Returns

Vector of TupleBuffers

References NES::Runtime::MemoryLayouts::TestTupleBuffer::createTestTupleBuffer(), NES::TestUtils::getPhysicalTypes(), and NES_ASSERT2_FMT.

Here is the call graph for this function:

createMemoryLayout()

Runtime::MemoryLayouts::MemoryLayoutPtr NES::Util::createMemoryLayout	(	SchemaPtr	schema,
		uint64_t	bufferSize
	)

Creates a memory layout from the schema and the buffer Size.

Parameters

schema
bufferSize

Returns

MemoryLayoutPtr

References NES::bufferSize, NES::Schema::COLUMNAR_LAYOUT, NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), and NES::Schema::ROW_LAYOUT.

Here is the call graph for this function:

endsWith()

bool NES::Util::endsWith	(	const std::string &	fullString,
		const std::string &	ending
	)

Checks if a string ends with a given string.

Parameters

fullString
ending

Returns

true if it ends with the given string, else false

Referenced by NES::Monitoring::MetricUtils::validateFieldsInSchema().

Here is the caller graph for this function:

escapeJson()

std::string NES::Util::escapeJson

(

const std::string &

str

)

escapes all non text characters in a input string, such that the string could be processed as json.

Parameters

s	input string.

Returns

result sing.

findAndReplaceAll()

void NES::Util::findAndReplaceAll	(	std::string &	data,
		const std::string &	toSearch,
		const std::string &	replaceStr
	)

function to replace all string occurrences

Parameters

data	input string will be replaced in-place
toSearch	search string
replaceStr	replace string

References data.

Referenced by NES::Parser::writeFieldValueToTupleBuffer().

Here is the caller graph for this function:

getPhysicalTypes()

std::vector< PhysicalTypePtr > NES::Util::getPhysicalTypes

(

SchemaPtr

schema

)

Returns the physical types of all fields of the schema.

Parameters

schema

Returns

PhysicalTypes of the schema's field

References NES::DefaultPhysicalTypeFactory::getPhysicalType().

Here is the call graph for this function:

murmurHash()

uint64_t NES::Util::murmurHash

(

uint64_t

key

)

hashes the key with murmur hash

Parameters

key

Returns

calculated hash

numberOfUniqueValues()

uint64_t NES::Util::numberOfUniqueValues

(

std::vector< uint64_t > &

values

)

Checks if a string starts with a given string.

Parameters

fullString
start

Returns

true if it ends with the given string, else false

References magic_enum::detail::values().

Here is the call graph for this function:

padVectorToSize()

template<typename T >

void NES::Util::padVectorToSize	(	std::vector< T > &	vector,
		size_t	newSize,
		T	newValue
	)

appends newValue until the vector contains a minimum of newSize elements

Template Parameters

T

Parameters

vector	the vector
newSize	the size of the padded vector
newValue	the value that should be added

partition()

template<typename T >

std::vector<std::vector<T> > NES::Util::partition	(	const std::vector< T > &	vec,
		size_t	n
	)

Partition a vector in n chunks, e.g., ([1, 2, 3, 4, 5], 3) -> [[1, 2], [3, 4], [5]]

Parameters

input	the vector
n	the chunks

Returns

the chunked vector

References magic_enum::detail::n().

Referenced by NES::Network::PartitionManager::addSubpartitionEventListener(), NES::Network::PartitionManager::clear(), NES::Network::PartitionManager::getConsumerRegistrationStatus(), NES::Network::PartitionManager::getDataEmitter(), NES::Network::PartitionManager::getEventListener(), NES::Network::PartitionManager::getProducerRegistrationStatus(), NES::Network::PartitionManager::getSubpartitionConsumerCounter(), NES::Network::PartitionManager::getSubpartitionProducerCounter(), NES::Network::PartitionManager::getVersion(), NES::Network::PartitionManager::pinSubpartitionConsumer(), NES::Network::PartitionManager::pinSubpartitionProducer(), NES::Network::PartitionManager::registerSubpartitionConsumer(), NES::Network::PartitionManager::registerSubpartitionProducer(), NES::Network::PartitionManager::unregisterSubpartitionConsumer(), and NES::Network::PartitionManager::unregisterSubpartitionProducer().

Here is the call graph for this function:

Here is the caller graph for this function:

printTupleBufferAsCSV()

std::string NES::Util::printTupleBufferAsCSV	(	Runtime::TupleBuffer	tbuffer,
		const SchemaPtr &	schema,
		const std::string &	lineSuffix = ""
	)

create CSV lines from the tuples

Parameters

tbuffer	the tuple buffer
schema	how to read the tuples from the buffer
lineSuffix	a string that will be appended at the end of each line

Returns

a full string stream as string

References NES::Runtime::TupleBuffer::getBuffer(), NES::Runtime::TupleBuffer::getNumberOfTuples(), NES_DEBUG, and NES::Runtime::MemoryLayouts::readVarSizedData().

Referenced by NES::CSVSource::fillBuffer(), NES::CsvFormat::getFormattedBuffer(), NES::Runtime::Execution::JoinDeploymentTest::runAndValidateJoinQueryTwoLogicalStreams(), and TEST_P().

Here is the call graph for this function:

Here is the caller graph for this function:

printTupleBufferAsText()

std::string NES::Util::printTupleBufferAsText

(

Runtime::TupleBuffer &

buffer

)

Outputs a tuple buffer in text format.

Parameters

buffer

the tuple buffer

Returns

string of tuple buffer

References NES::Runtime::TupleBuffer::getBuffer(), and NES::Runtime::TupleBuffer::getNumberOfTuples().

Here is the call graph for this function:

replaceFirst()

std::string NES::Util::replaceFirst	(	std::string	origin,
		const std::string &	search,
		const std::string &	replace
	)

This function replaces the first occurrence of search term in a string with the replace term.

Parameters

origin	- The original string that is to be manipulated
search	- The substring/term which we want to have replaced
replace	- The string that is replacing the search term.

Returns

Referenced by NES::QueryParsingService::createQueryFromCodeString(), and NES::TEST().

Here is the caller graph for this function:

splitWithStringDelimiter()

template<typename T >

std::vector<T> NES::Util::splitWithStringDelimiter	(	std::string_view	inputString,
		std::string_view	delim,
		std::function< T(std::string_view)>	fromStringToT = detail::SplitFunctionHelper<T>::FUNCTION
	)

splits a string given a delimiter into multiple substrings stored in a T vector the delimiter is allowed to be a string rather than a char only.

Parameters

data	- the string that is to be split
delimiter	- the string that is to be split upon e.g. / or -
fromStringtoT	- the function that converts a string to an arbitrary type T

Returns

startsWith()

bool NES::Util::startsWith	(	const std::string &	fullString,
		const std::string &	ending
	)

Get number of unique elements.

Parameters

fullString
start

Returns

true if it ends with the given string, else false

toCSVString()

std::string NES::Util::toCSVString

(

const SchemaPtr &

schema

)

method to get the schema as a csv string

Parameters

schema

Returns

schema as csv string

Referenced by NES::CsvFormat::getFormattedSchema().

Here is the caller graph for this function:

toUpperCase()

std::string NES::Util::toUpperCase

(

std::string

string

)

transforms the string to an upper case version

Parameters

string

Returns

string

trimChar()

std::string_view NES::Util::trimChar	(	std::string_view	in,
		char	trimFor
	)

removes leading and trailing occurences of trimFor

Referenced by NES::TEST(), and NES::JSONParser::writeInputTupleToTupleBuffer().

Here is the caller graph for this function:

trimWhiteSpaces()

std::string_view NES::Util::trimWhiteSpaces

(

std::string_view

in

)

removes leading and trailing whitespaces

Referenced by NES::CsvFormat::getFormattedSchema(), and NES::TEST().

Here is the caller graph for this function:

updateAtomicMax()

template<typename T >

void NES::Util::updateAtomicMax	(	std::atomic< T > &	curVal,
		const T &	newVal
	)

Tries to update curVal until it succeeds or curVal is larger then newVal.

Template Parameters

T

Parameters

curVal
newVal

updateSourceName()

std::string NES::Util::updateSourceName	(	std::string	queryPlanSourceConsumed,
		std::string	subQueryPlanSourceConsumed
	)

: Update the source names by sorting and then concatenating the source names from the sub- and query plan

Parameters

string	consumed sources of the current queryPlan
string	consumed sources of the subQueryPlan

Returns

string with new source name

writeHeaderToCsvFile()

void NES::Util::writeHeaderToCsvFile	(	const std::string &	csvFileName,
		const std::string &	header
	)

Truncates the file and then writes the header string as is to the file.

Parameters

csvFileName
header

writeRowToCsvFile()

void NES::Util::writeRowToCsvFile	(	const std::string &	csvFileName,
		const std::string &	row
	)

Appends the row as is to the csv file.

Parameters

csvFileName
row