NES::Runtime::MemoryLayouts Namespace Reference

Classes
class	ColumnLayout
	Implements a columnar layout, that maps all tuples in a tuple buffer to a column-wise layout. For a schema with 3 fields (F1, F2, and F3) we retrieve the following layout. More...
class	ColumnLayoutField
	The ColumnLayoutField enables assesses to a specific field in a columnar layout. It overrides the operator[] for a more user friendly access of tuples for a predefined field. As this required direct knowledge of a particular memory layout at compile-time, consider to use the TestTupleBuffer. More...
class	MemoryLayout
	A MemoryLayout defines a strategy in which a specific schema / a individual tuple is mapped to a tuple buffer. To this end, it requires the definition of an schema and a specific buffer size. Currently. we support a RowLayout and a ColumnLayout. More...
class	MemoryLayoutTupleBuffer
	This abstract class is the base class for DynamicRowLayoutBuffer and DynamicColumnLayoutBuffer. As the base class, it has multiple methods or members that are useful for both derived classes. @caution This class is non-thread safe. More...
class	RowLayout
	Implements a row layout, that maps all tuples in a tuple buffer to a row-wise layout. For a schema with 3 fields (F1, F2, and F3) we retrieve the following layout. More...
class	RowLayoutField
	The RowLayoutField enables assesses to a specific field in a row layout. It overrides the operator[] for a more user friendly access of tuples for a predefined field. As this required direct knowledge of a particular memory layout at compile-time, consider to use the TestTupleBuffer. More...
class	DynamicField
	The DynamicField allows to read and write a field at a specific address and a specific data type. For all field accesses we check that the template type is the same as the selected physical field type. If the type is not compatible accesses result in a BufferAccessException. More...
class	DynamicTuple
	The DynamicRecords allows to read individual fields of a tuple. Field accesses are safe in the sense that if is checked the field exists. More...
class	TestTupleBuffer
	The TestTupleBuffers allows to read records and individual fields from an tuple buffer. To this end, it assumes a specific data layout, i.e., RowLayout or ColumnLayout. This allows for dynamic accesses to a tuple buffer in the sense that at compile-time a user has not to specify a specific memory layout. Therefore, the memory layout can be a runtime option, whereby the code that operates on the tuple buffer stays the same. Furthermore, the TestTupleBuffers trades-off performance for safety. To this end, it checks field bounds and field types and throws BufferAccessException if the passed parameters would lead to invalid buffer accesses. The TestTupleBuffers supports different access methods: More...
class	ColumnarMemoryLayoutTest
class	DynamicMemoryLayoutTestParameterized
class	RowMemoryLayoutTest
class	TestTupleBufferTest

Typedefs
using	FIELD_SIZE = uint64_t
using	MemoryLayoutPtr = std::shared_ptr< MemoryLayout >
using	ColumnLayoutPtr = std::shared_ptr< ColumnLayout >
using	RowLayoutPtr = std::shared_ptr< RowLayout >
using	MemoryLayoutBufferPtr = std::shared_ptr< MemoryLayoutTupleBuffer >
using	VarSizeDataTuple = std::tuple< VAR_SIZED_DATA_TYPES >
using	FixedSizedDataTuple = std::tuple< FIXED_SIZED_DATA_TYPES >

Functions
std::string	readVarSizedData (const TupleBuffer &buffer, uint64_t childBufferIdx)
	Reads the variable sized data from the child buffer at the provided index. More...
std::optional< uint32_t >	writeVarSizedData (const TupleBuffer &buffer, const std::string_view value, BufferManager &bufferManager)
	Writes the variable sized data to the buffer. More...
std::ostream &	operator<< (std::ostream &os, const TestTupleBuffer &buffer)
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutCreateTest)
	Tests that we can construct a column layout. More...
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutMapCalcOffsetTest)
	Tests that the field offsets are are calculated correctly using a TestTupleBuffer. More...
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutPushRecordAndReadRecordTestOneRecord)
	Tests that we can write a single record to and read from a TestTupleBuffer correctly. More...
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutPushRecordAndReadRecordTestMultipleRecord)
	Tests that we can write many records to and read from a TestTupleBuffer correctly. More...
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutLayoutFieldSimple)
	Tests that we can access fields of a TupleBuffer that is used in a TestTupleBuffer correctly. More...
	TEST_F (ColumnarMemoryLayoutTest, columnLayoutLayoutFieldBoundaryCheck)
	Tests whether whether an error is thrown if we try to access non-existing fields of a TupleBuffer. More...
	TEST_F (ColumnarMemoryLayoutTest, getFieldViaFieldNameColumnLayout)
	Tests whether we can only access the correct fields. More...
	TEST_F (ColumnarMemoryLayoutTest, accessDynamicColumnBufferTest)
	Tests whether reading from the TestTupleBuffer works correctly. More...
	TEST_F (ColumnarMemoryLayoutTest, pushRecordTooManyRecordsColumnLayout)
	Tests if an error is thrown if more tuples are added to a TupleBuffer than the TupleBuffer can store. More...
	TEST_P (DynamicMemoryLayoutTestParameterized, readWriteColumnartestBufferTest)
	TEST_P (DynamicMemoryLayoutTestParameterized, iteratetestBufferTest)
	TEST_P (DynamicMemoryLayoutTestParameterized, toStringTestRowLayout)
	INSTANTIATE_TEST_CASE_P (TestInputs, DynamicMemoryLayoutTestParameterized, ::testing::Values(Schema::MemoryLayoutType::COLUMNAR_LAYOUT, Schema::MemoryLayoutType::ROW_LAYOUT), [](const testing::TestParamInfo< DynamicMemoryLayoutTestParameterized::ParamType > &info) { return std::string(magic_enum::enum_name(info.param));})
	TEST_F (RowMemoryLayoutTest, rowLayoutCreateTest)
	Tests that we can construct a column layout. More...
	TEST_F (RowMemoryLayoutTest, rowLayoutMapCalcOffsetTest)
	Tests that the field offsets are are calculated correctly using a TestTupleBuffer. More...
	TEST_F (RowMemoryLayoutTest, rowLayoutPushRecordAndReadRecordTestOneRecord)
	Tests that we can write a single record to and read from a TestTupleBuffer correctly. More...
	TEST_F (RowMemoryLayoutTest, rowLayoutPushRecordAndReadRecordTestMultipleRecord)
	Tests that we can write many records to and read from a TestTupleBuffer correctly. More...
	TEST_F (RowMemoryLayoutTest, rowLayoutLayoutFieldSimple)
	Tests that we can access fields of a TupleBuffer that is used in a TestTupleBuffer correctly. More...
	TEST_F (RowMemoryLayoutTest, rowLayoutLayoutFieldBoundaryCheck)
	Tests whether whether an error is thrown if we try to access non-existing fields of a TupleBuffer. More...
	TEST_F (RowMemoryLayoutTest, getFieldViaFieldNameRowLayout)
	Tests whether we can only access the correct fields. More...
	TEST_F (RowMemoryLayoutTest, pushRecordTooManyRecordsRowLayout)
	Tests if an error is thrown if more tuples are added to a TupleBuffer than the TupleBuffer can store. More...
	TEST_P (TestTupleBufferTest, readWritetestBufferTest)
	TEST_P (TestTupleBufferTest, readWritetestBufferTestVarSizeData)
	TEST_P (TestTupleBufferTest, readWritetestBufferTestFullBuffer)
	TEST_P (TestTupleBufferTest, readWritetestBufferTestFullBufferVarSizeData)
	TEST_P (TestTupleBufferTest, countOccurrencesTest)
	TEST_P (TestTupleBufferTest, countOccurrencesTestVarSizeData)
	TEST_P (TestTupleBufferTest, DynamicTupleCompare)
	TEST_P (TestTupleBufferTest, DynamicTupleCompareVarSizeData)
	INSTANTIATE_TEST_CASE_P (TestInputs, TestTupleBufferTest, ::testing::Values(Schema::MemoryLayoutType::COLUMNAR_LAYOUT, Schema::MemoryLayoutType::ROW_LAYOUT), [](const testing::TestParamInfo< TestTupleBufferTest::ParamType > &info) { return std::string(magic_enum::enum_name(info.param));})

Typedef Documentation

ColumnLayoutPtr

using NES::Runtime::MemoryLayouts::ColumnLayoutPtr = typedef std::shared_ptr<ColumnLayout>

FIELD_SIZE

using NES::Runtime::MemoryLayouts::FIELD_SIZE = typedef uint64_t

FixedSizedDataTuple

using NES::Runtime::MemoryLayouts::FixedSizedDataTuple = typedef std::tuple<FIXED_SIZED_DATA_TYPES>

MemoryLayoutBufferPtr

using NES::Runtime::MemoryLayouts::MemoryLayoutBufferPtr = typedef std::shared_ptr<MemoryLayoutTupleBuffer>

MemoryLayoutPtr

using NES::Runtime::MemoryLayouts::MemoryLayoutPtr = typedef std::shared_ptr<MemoryLayout>

RowLayoutPtr

using NES::Runtime::MemoryLayouts::RowLayoutPtr = typedef std::shared_ptr<RowLayout>

VarSizeDataTuple

using NES::Runtime::MemoryLayouts::VarSizeDataTuple = typedef std::tuple<VAR_SIZED_DATA_TYPES>

Function Documentation

INSTANTIATE_TEST_CASE_P() [1/2]

NES::Runtime::MemoryLayouts::INSTANTIATE_TEST_CASE_P	(	TestInputs	,
		DynamicMemoryLayoutTestParameterized	,
		::testing::Values(Schema::MemoryLayoutType::COLUMNAR_LAYOUT, Schema::MemoryLayoutType::ROW_LAYOUT)	,
		[] (const testing::TestParamInfo< DynamicMemoryLayoutTestParameterized::ParamType > &info) { return std::string(magic_enum::enum_name(info.param));}
	)

References magic_enum::enum_name().

Here is the call graph for this function:

INSTANTIATE_TEST_CASE_P() [2/2]

NES::Runtime::MemoryLayouts::INSTANTIATE_TEST_CASE_P	(	TestInputs	,
		TestTupleBufferTest	,
		::testing::Values(Schema::MemoryLayoutType::COLUMNAR_LAYOUT, Schema::MemoryLayoutType::ROW_LAYOUT)	,
		[] (const testing::TestParamInfo< TestTupleBufferTest::ParamType > &info) { return std::string(magic_enum::enum_name(info.param));}
	)

References magic_enum::enum_name().

Here is the call graph for this function:

operator<<()

std::ostream& NES::Runtime::MemoryLayouts::operator<<	(	std::ostream &	os,
		const TestTupleBuffer &	buffer
	)

Parameters

os	output stream
buffer	dynamic tupleBuffer

Returns

result stream

readVarSizedData()

std::string NES::Runtime::MemoryLayouts::readVarSizedData	(	const TupleBuffer &	buffer,
		uint64_t	childBufferIdx
	)

Reads the variable sized data from the child buffer at the provided index.

Parameters

buffer
childBufferIdx

Returns

Variable sized data as a string

References NES::Runtime::TupleBuffer::getBuffer(), NES::Runtime::TupleBuffer::loadChildBuffer(), and apex::memcpy().

Referenced by NES::Runtime::MemoryLayouts::DynamicTuple::operator==(), NES::Util::printTupleBufferAsCSV(), NES::Statistic::CountMinStatisticFormat::readStatisticsFromBuffer(), NES::Statistic::HyperLogLogStatisticFormat::readStatisticsFromBuffer(), NES::Runtime::MemoryLayouts::DynamicTuple::readVarSized(), and NES::Runtime::MemoryLayouts::DynamicTuple::toString().

Here is the call graph for this function:

Here is the caller graph for this function:

TEST_F() [1/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		accessDynamicColumnBufferTest
	)

Tests whether reading from the TestTupleBuffer works correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [2/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutCreateTest
	)

Tests that we can construct a column layout.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), and NES::UINT8.

Here is the call graph for this function:

TEST_F() [3/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutLayoutFieldBoundaryCheck
	)

Tests whether whether an error is thrown if we try to access non-existing fields of a TupleBuffer.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::Runtime::MemoryLayouts::ColumnLayoutField< T, boundaryChecks >::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [4/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutLayoutFieldSimple
	)

Tests that we can access fields of a TupleBuffer that is used in a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::Runtime::MemoryLayouts::ColumnLayoutField< T, boundaryChecks >::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [5/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutMapCalcOffsetTest
	)

Tests that the field offsets are are calculated correctly using a TestTupleBuffer.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [6/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutPushRecordAndReadRecordTestMultipleRecord
	)

Tests that we can write many records to and read from a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [7/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		columnLayoutPushRecordAndReadRecordTestOneRecord
	)

Tests that we can write a single record to and read from a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [8/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		getFieldViaFieldNameColumnLayout
	)

Tests whether we can only access the correct fields.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [9/17]

NES::Runtime::MemoryLayouts::TEST_F	(	ColumnarMemoryLayoutTest	,
		pushRecordTooManyRecordsColumnLayout
	)

Tests if an error is thrown if more tuples are added to a TupleBuffer than the TupleBuffer can store.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::ColumnLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [10/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		getFieldViaFieldNameRowLayout
	)

Tests whether we can only access the correct fields.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [11/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		pushRecordTooManyRecordsRowLayout
	)

Tests if an error is thrown if more tuples are added to a TupleBuffer than the TupleBuffer can store.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [12/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutCreateTest
	)

Tests that we can construct a column layout.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), and NES::UINT8.

Here is the call graph for this function:

TEST_F() [13/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutLayoutFieldBoundaryCheck
	)

Tests whether whether an error is thrown if we try to access non-existing fields of a TupleBuffer.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::Runtime::MemoryLayouts::RowLayoutField< T, boundaryChecks >::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [14/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutLayoutFieldSimple
	)

Tests that we can access fields of a TupleBuffer that is used in a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::Runtime::MemoryLayouts::RowLayoutField< T, boundaryChecks >::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [15/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutMapCalcOffsetTest
	)

Tests that the field offsets are are calculated correctly using a TestTupleBuffer.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [16/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutPushRecordAndReadRecordTestMultipleRecord
	)

Tests that we can write many records to and read from a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_F() [17/17]

NES::Runtime::MemoryLayouts::TEST_F	(	RowMemoryLayoutTest	,
		rowLayoutPushRecordAndReadRecordTestOneRecord
	)

Tests that we can write a single record to and read from a TestTupleBuffer correctly.

References NES::Schema::create(), NES::Runtime::MemoryLayouts::RowLayout::create(), NES::UINT16, NES::UINT32, and NES::UINT8.

Here is the call graph for this function:

TEST_P() [1/11]

NES::Runtime::MemoryLayouts::TEST_P	(	DynamicMemoryLayoutTestParameterized	,
		iteratetestBufferTest
	)

TEST_P() [2/11]

NES::Runtime::MemoryLayouts::TEST_P	(	DynamicMemoryLayoutTestParameterized	,
		readWriteColumnartestBufferTest
	)

TEST_P() [3/11]

NES::Runtime::MemoryLayouts::TEST_P	(	DynamicMemoryLayoutTestParameterized	,
		toStringTestRowLayout
	)

References NES::expectedOutput.

TEST_P() [4/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		countOccurrencesTest
	)

References FIXED_SIZED_DATA_TYPES.

TEST_P() [5/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		countOccurrencesTestVarSizeData
	)

References VAR_SIZED_DATA_TYPES.

TEST_P() [6/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		DynamicTupleCompare
	)

References FIXED_SIZED_DATA_TYPES.

TEST_P() [7/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		DynamicTupleCompareVarSizeData
	)

References VAR_SIZED_DATA_TYPES.

TEST_P() [8/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		readWritetestBufferTest
	)

References FIXED_SIZED_DATA_TYPES.

TEST_P() [9/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		readWritetestBufferTestFullBuffer
	)

References FIXED_SIZED_DATA_TYPES.

TEST_P() [10/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		readWritetestBufferTestFullBufferVarSizeData
	)

References VAR_SIZED_DATA_TYPES.

TEST_P() [11/11]

NES::Runtime::MemoryLayouts::TEST_P	(	TestTupleBufferTest	,
		readWritetestBufferTestVarSizeData
	)

References VAR_SIZED_DATA_TYPES.

writeVarSizedData()

std::optional< uint32_t > NES::Runtime::MemoryLayouts::writeVarSizedData	(	const TupleBuffer &	buffer,
		const std::string_view	value,
		BufferManager &	bufferManager
	)

Writes the variable sized data to the buffer.

Parameters

buffer
value
bufferManager

Returns

Index of the child buffer

References NES::Runtime::BufferManager::getUnpooledBuffer(), apex::memcpy(), NES::Runtime::TupleBuffer::storeChildBuffer(), and magic_enum::detail::value().

Referenced by NES::Statistic::CountMinStatisticFormat::writeStatisticsIntoBuffers(), and NES::Statistic::HyperLogLogStatisticFormat::writeStatisticsIntoBuffers().

Here is the call graph for this function:

Here is the caller graph for this function: