Please reach us at andy3474@msn.com if you cannot find an answer to your question.
Didgets is a general-purpose data management system. It is currently composed of a data engine called the 'Didget Manager' and a GUI application called the 'Didget Browser'.
The manager is a cross-platform engine written entirely in C++. The code uses multi-threading extensively which makes it very performant on modern CPUs with many cores. It is currently built as a static library that has an API for managing 'Didgets' (Data Widgets) which are intelligent, persistent data objects. Using the API, an application can create or load a 'Pod', which is a logical container for the Didgets; perform CRUD operations on individual Didgets within the Pod; or perform searches for subsets of existing Didgets.
The browser is also cross-platform using Qt as its windowing interface. The browser calls the manager API to perform various data operations. The browser is a useful administration tool; a testing tool; and a useful data analysis tool. It is primarily designed to showcase the power and speed of the underlying engine code.
The system is currently in beta with a Windows version. With enough resources, a Linux version and a MacOS version should be available soon. The manager code could also be made into a shared library or run as a separate service. With proper development and testing, it could also run on a separate server and allow multiple clients to remotely connect to it.
A Didget consists of three parts: a fixed-sized meta-data record; an optional data stream; and an optional set of contextual meta-data tags. There are a variety of different Didget types, but all of them conform to this basic structure.
Like with a file system that has an inode, FRS record, or another structure for every file; every Didget within a Pod has a Didget record within a system table. This Didget record is extremely small - just 64 bytes. This means that the entire table for a Pod with 200 million Didgets is less than 13 GB in size. Such a table can be read from a fast SSD in just a couple of seconds and stored entirely in memory on most computers. Searching for Didgets with certain characteristics is extremely fast even when there are hundreds of millions of them.
Like a file, each Didget can have a data stream consisting of a variable number of bytes (from just a single byte to many terabytes). The Didget Manager is responsible for allocating and managing space for each stream using the pool of blocks (each 4 KiB) in the Pod container. Unlike a file system, there are Didgets that have their data streams directly managed by the manager code. Traditional file systems treat file contents like a 'black box'. 'File Didgets' do not have managed streams, but all other types of Didgets have them.
Each Didget can also have a set of tags attached (up to 255 of them). A tag may be the name of someone in a picture; the author of a document; or typical file system meta-data like a name, a full path, or timestamps. Each tag is defined with a data type (just like how each column in a relational database table is defined). Tags can be strings, numbers, dateTime values, booleans, etc. Searching for Didgets that have certain tags attached is also extremely fast.
Didgets that are not 'File Didgets' can be powerful building blocks for creating more complex systems. The bitmap used to track free/used space within a Pod is stored within a 'Bitmap Didget'. The tag information is stored within 'Tag Didgets'. Lists, indexes, dictionaries, schema, and other kinds of information are likewise stored within specialized Didgets.
The variety of Didgets make it possible to built complex data systems using them. Just like an erector set consists of a small number of different types of pieces, but many different types of things (e.g. cars, bridges, cranes, etc.) can be built with them; Didgets can be used in different configurations to build data systems that handle unstructured data, highly structured data, and semi-structured data using a variety of data models.
Each type of Didget is designed to do one thing and do it extremely well. A 'Set Didget' for example is designed to manage large numbers of the IDs of other Didgets. This type of Didget can be used in a variety of ways. It can be used as a folder within a file system. It can be a list of photos or songs that form an album. It can tie together the Didgets that form a relational table. Likewise, 'Tag Didgets' can be used for tagging files or can be used to hold column data within a table.
This makes it possible to build things like hierarchical file systems, relational databases, key-value stores, graph databases, and other kinds of generalized data systems using a set of Didgets. The code that manages each kind of Didget is optimized to allow large quantities of data to be effectively managed within each one.
So far the file system and database table features are the most developed for use within the Didget Browser application. Logging frameworks, dictionaries, and indexes of file contents are other features that are partially implemented. Other features like configuration management, policies, and content management are still on the drawing board, but are certainly possible.