Dolt's SQL engine is built as a layer on top of its storage engine. This allows us to implement non-SQL operations, such as the CLI commands, directly against the storage layer. It also leaves the door open to build additional database dialects on top of Dolt in the future.
The SQL engine is implemented by three different open source projects, listed below.
vitess is a MySQL sharding management system. It implements the MySQL parser and the MySQL server, including the wire protocol.
Read more about the vitess project and how it's used in Dolt.
is a storage agnostic SQL query engine written in pure Go. It implements a query analyzer and execution engine, and defines interfaces for storage backend integrators (like Dolt) to implement.
The Dolt project itself implements the interfaces defined in to make the Dolt storage engine available for reads and writes in a SQL context. It also implements many System variables and custom SQL functions.
Dolt is responsible for the various interfaces to start, stop, configure and manage SQL servers.
Dolt's SQL server runs on a fork of Vitess. Vitess is a MySQL database sharding solution that was created to scale YouTube:
Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. Vitess has been a core component of YouTube's database infrastructure since 2011, and has grown to encompass tens of thousands of MySQL nodes.
In addition to implementing the MySQL server and wire protocol, Vitess also provides the SQL parser for the database.
Dolt's fork of Vitess has headed in a different direction from the main project. In addition to adding support for DDL statements, stored procedures, and triggers (never a priority of the Vitess project), Dolt's fork prunes away the 90% of Vitess that isn't vital to Dolt's current roadmap. You can read details about this work in .
is the query engine for Dolt. It's a MySQL compatible parser, server, and query execution engine written in pure Go. As with Dolt, its goal is to be a 100% compatible drop-in replacement for MySQL.
is storage engine agnostic, which means other projects can write their own storage engine plugins to query them via a MySQL connection. There are only two notable backend implementations so far:
In-memory database. This ships with go-mysql-server and is useful for testing the engine, or for using in tests for golang projects that want a fast, local MySQL database.
Dolt. In addition to the novel git-like storage engine, Dolt also adds a number of ,
go-mysql-server defines a number of interfaces to allow integrators to communicate about their database's capabilities, what tables exist, what the schemas of tables are, and so on. Then another set of interfaces let integrators iterate over their table rows, update them with new values, delete them, and so on. For more information, see the project docs.
Broadly, the system has three main components:
The parser and server, which are mostly provided by Vitess. This component receives queries on the wire and parses them into an AST.
The query analyzer, which repeatedly transforms the AST to determine an optimal execution strategy, doing things like applying indexes or ordering tables for a join.
The execution engine, which iterates over table rows provided by integrators, evaluates expressions and functions, and produces the ultimate result set.
For much more technical details on how these pieces function, refer to the following blog articles: