Skip to content

Streams

datastreams

Streams, also known as datastreams, are one of the primary ways of moving around and manipulating data in nvgt. With their convenient function calls, low memory footprint when dealing with large datasets and with their ability to connect to one another to create a chain of mutations on any data, datastreams are probably the most convenient method for data manipulation in NVGT that exist.

A datastream could be a file on disk, a file downloading from/uploading to the internet, a simple incapsulated string, or some sort of a manipulator such as an encoder, decryptor or compression stream.

Datastreams can roughly be split into 3 categories, or if you want to be really specific 2.5.

Occasionally, you may see a reader referred to as a decoder, and a writer referred to as an encoder.

Particularly when considering reader and writer streams that manipulate data, you can typically chain any number of streams of the same type together to cause a sequence of data mutations to be performed in one function call. For example you could connect an inflating_reader to a hex_decoder that is in turn connected to a file object in read mode. From that point, calling any of the read functions associated with the inflating stream would automatically first read from the file, hex decode it, and decompress/inflate it as needed. Inversely, you could connect a deflating_writer to a hex encoder which is connected to a file object in write mode, causing any data written to the inflating stream to be compressed, hex encoded, and finally written to the file.

There are a few readers and writers that, instead of manipulating data in any way as they pass through, give you details about that data. You can see counting_reader and counting_writer as examples of this, these streams count the number of lines and characters that are read or written through them.

Lets put this together with a little demonstration. Say you have a compressed and hex encoded file with many lines in it, and you'd like to read the file while determining the line count as the file is read. You could execute the following, for example:

counting_reader f(inflating_reader(hex_decoder(file("test.txt", "rb"))));
string result;
while(f.good()) {
	result += f.read(100);
	alert("test", "currently on line " + f.lines + " on character position " + f.pos);
}
f.close();
alert("test", "The file contains a total of " + f.lines + " and after decoding, contains the following text: \n" + result);

More datastreams could be added to the engine at any time.

Note that most datastreams do not support seeking as this would result in needing to store more data in memory than we are comfortable with just as a start, however file objects and raw datastreams that read from a string do support seeking, E source streams. Other than these stream types, you should assume that seek operations will not work unless noted otherwise for each type of stream.

The available datastreams are listed below in this subsection of the documentation.

datastream

The base class for all datastreams, this stream can read and write to an internal string buffer maintained by the class if constructed directly. Any other datastream can also be cast to the "datastream" type to facilitate passing datastreams of any type throughout the application, and thus any child datastreams such as encoders, file sources or any others will contain the functions listed here as they are derived from the datastream class.

  1. datastream();

  2. datastream(string initial_data = "", string encoding = "", datastream_byte_order byteorder = STREAM_BYTE_ORDER_NATIVE);

Arguments (2):

Remarks:

If this class is directly instantiated, the effect is basically that the string class gets wrapped with streaming functions. The default datastream class will create an internal string object, and any data passed into the initial_data parameter will be copied to that internal string. Then, the read/write functions on the stream will do their work on the internal string object held by the datastream class, which can be read or retrieved in full at any time. Internally, this wraps the std::stringstream class in c++.

It must be noted, however, that this is the parent class for all other datastreams in nvgt. This means that any child datastream such as the file class can be cast to a generic datastream handle. In this case, the read/write functions for the casted handle will perform the function of the child stream which has been casted instead of on an internal string. This is the same for any other parent/child class relationship in programming, but it was mentioned here to avoid any confusion between the default datastream implementation and a datastream handle casted from a different stream.

The encoding argument, present in nearly all child datastream constructors, controls what encoding if any strings should be converted to from UTF8 as they are written to the stream with the write_string() function or << operator while the binary property on the stream is set to true, as well as what encoding to convert from when reading them with read_string() or the >> operator. If set to an empty string (the default), strings are left in UTF8 when writing, and already expected to be in UTF8 in a stream when reading from it.

The byteorder argument, again present in nearly all child datastream constructors, controls what endianness is used when reading/writing binary data from/to a stream, that is the read_int/write_float/similar functions when the binary property on the stream is set to true. When a value takes more than one byte, the endianness or byte order controls whether the bytes of that value are read/written from left to right or right to left, or in proper terms whether the most significant byte of the value should be written first. The values that can be accepted here are:

Usually, you can leave the byteorder value at the default for most streams. However in some situations where you are transmitting binary data between systems running on different architectures, it may be better to set the transmitting and receiving streams of such an application to a common byte order that is not system native.

Though this was mentioned above, it's worth reiterating once more that all other stream types contain all of the functions listed in this base datastream class unless otherwise noted, and thus are not documented multiple times in child classes.

If an initial_data argument is provided when constructing a datastream, the stream will be set at the beginning, ready to read the initial data rather than at the end. If you wish to append more data to the datastream after it is constructed, you should call the seek_end() method on it first.

Example:

void main() {
	datastream ds1("This is a demonstration.");
	alert("example", ds1.read()); // Will display "This is a demonstration."
	datastream ds2;
	ds2.write("Hello there, ");
	ds2.write("good bye.");
	ds2.seek(0);
	alert("example", ds2.read()); // Will display "Hello there, good bye."
	// The following shows how this datastream can be used as an area to store encoded data.
	datastream encoded;
	hex_encoder h(encoded); // We attach the encoded datastream to a hex encoder.
	h.write("I am a hex string"); // "I am a hex string" in hex is written to the datastream object called encoded.
	h.close();
	encoded.seek(0);
	alert("example", hex_decoder(encoded).read()); // We attach a hex_decoder to the encoded datastream and read from it, thus this will display "I am a hex string".
}

Methods

close

Close a datastream, freeing any associated resources and leaving the datastream in an inactive state.

bool close(bool close_connected = false);

arguments:
Returns:

bool: true on success, false on failure such as if the stream is already closed.

Remarks:

Generally it is best to call this function when you are done working with any stream, particularly when dealing with files, encoders, or any stream that is opened in write mode. In reality there are a few streams, such as the default datastream class, where it is OK to not call the close method.

The reason that it is sometimes OK to not call the close function on a stream when you are done with it is because this function is automatically called in the internal destructor for any datastream, meaning when any datastream object is destroyed, it's close method will be called. For some streams like the default datastream class which is just wrapping a string, this is fine (you don't close a string, after all), but in many cases you will want to close a stream at a certain time before it is destroyed. For example closing an encoding stream may write a few final characters to a connected file stream as it is closing, meaning that the close method on the encoder must be called prior to the close function on the stream connected to it, something which you may not have control of when objects are getting destructed.

The close_connected argument controls whether to also close any streams that are connected to the stream that close is being called on. For the default datastream class that wraps a string or for any other datastream that doesn't connect to another one, this argument has no effect.

Example:
void main() {
	datastream ds;
	ds.close();
	alert("example", ds.active); // Will display false, as the stream is no longer active.
	alert("example", ds.write("this is a test")); // Will return 0 instead of a positive number of bytes written as the stream is not opened.
	ds.open();
	alert("example", ds.write("this is a test")); // Now returns 14, as expected.
	// Calling the close method a final time is not required for this example because it uses an instance of the default datastream class. It will be taken care of when the script exits.
}

close_all

Close a datastream as well as any that are connected to it.

bool close_all();

Returns:

bool: true if the stream and any connected to it could be closed, false otherwise.

Remarks:

This is exactly the same thing as calling the close method with the close_connected boolean argument set to true. On the default datastream class it will have no effect besides freeing the internal string, however you can look at the example below to see a case where there is an effect by calling close_all().

Example:
void main() {
	datastream ds;
	hex_encoder h(ds);
	h.write("hi");
	h.close_all(); // Will also close the datastream called ds because it is connected to the hex encoder.
	alert("example", ds.active); // Will display false, indicating that calling h.close_all() also caused ds.close() to implicitly be called.
}

open

Open a datastream, connecting it to a source or another stream depending on type.

bool open(...);

Arguments:

Same as in the constructor for the datastream you wish to open (see remarks).

Returns:

bool: Returns true if the stream could be opened, false otherwise.

Remarks:

Most datastreams can be closed and then reopened, or can be created in an uninitialized state meaning they must be opened to begin with.

All datastreams can be constructed in an already initialized/opened state, and the constructors that allow this for each stream will contain the exact same arguments as that streams associated open() function, and each streams constructor topics is where such arguments are documented.

For example, the true signature for the open function on the default datastream class is bool open(string initial_data = "", string encoding = "", datastream_byteorder byteorder = STREAM_BYTE_ORDER_NATIVE) while the signature of the open function for the file datastream is bool open(string filename, string mode, string encoding = "", datastream_byte_order byteorder = STREAM_BYTE_ORDER_NATIVE).

As mentioned in the top level datastreams topic, the encoding and byteorder arguments are present in each stream and thus will not be redocumented for each one.

For all streams, this function will call the associated close() function prior to opening the requested resource if the stream is already active at this time. If it turns out that users do not desire this behaviour, it may be made optional or could get removed in the future.

Example:
void main() {
	datastream ds;
	if (!ds.open("this is a demonstration")) {
		alert("Oh no", "Maybe this could happen for a file but it should never happen for a basic datastream!");
		exit();
	}
	alert("example", ds.read()); // Will display "this is a demonstration".
}

read

Read raw bytes from a stream.

string read(uint amount = 0);

Arguments:
Returns:

string: The data that was read from the stream or an empty string on failure.

Remarks:

If the length of the string returned by this function is less than the number of bytes requested in the amount argument, either the end of the stream was reached or there was an error. You can check what happened by evaluating the fail or eof properties on the stream.

This is the lowest level method of reading from a stream. All other reading functions either read a certain datatype or come with some condition, for example this should not be confused with the read_string function which does not take a byte amount argument, but which instead either reads a binary number denoting the length of a string before reading that number of bytes or reads a single word from the stream based on the state of the stream's binary flag. If you want direct control over the read operation, use this function.

Typically it is only useful to call this function if the good property on the stream is true, therefor you can query the stream's good property in a loop to see if you should continue reading data if you are trying to perform some sort of buffered reading of a stream.

Example:
void main() {
	datastream ds("Hello there, I am a string wrapped in a sstream!");
	alert("example", ds.read(6)); // Will display Hello followed by a space.
	alert("example", ds.read()); // Will display there, I am a string wrapped in a sstream!
}

Properties

active

This property will be true if a stream is opened and ready for use, false otherwise.

const bool active;

available

This property returns the number of bytes immediately available to be read from a stream.

const int available;

Remarks:

This property may not be implemented in all streams, for example decoders. If it is unavailable, it will return 0.

Example:
void main() {
	datastream ds("example");
	alert("example", ds.available); // Displays 7.
	ds.read(2);
	alert("example", ds.available); // Now shows 5, as 2 of the 7 bytes have been read.
}

eof

This property will be true on a stream when it has no more data to read.

const bool eof;

Example:
void main() {
	datastream ds("hello");
	alert("example", ds.eof); // Will show false because we always start at the beginning of a default datastream, thus there is data to read.
	ds.read();
	alert("example", ds.eof); // Will now show true because any further .read calls will fail as we have reached the end of the stream.
	ds.seek(3);
	alert("example", ds.eof); // Will again show false because we are no longer at the end of the stream.
}

good

This property will be true when a datastream is ready to read or write, such as when the active property is true and the end of file has not been reached.

const bool good;

Remarks:

This property is specifically true so long as the stream is opened and the eof, fail, and bad properties all return false.

Example:
void main() {
	datastream ds("hello");
	alert("example", ds.good); // Will display true because there is data to read.
	ds.read();
	alert("example", ds.good); // Will now display false because the end of file has been reached, ds.eof is true now.
}