LogoLogo
DoltHubBlogDiscordGitHubDolt
  • Introduction
    • What Is Dolt?
    • Installation
      • Linux
      • Windows
      • Mac
      • Build from Source
      • Application Server
      • Docker
      • Upgrading
    • Getting Started
      • Version Controlled Database
      • Git For Data
      • Versioned MySQL Replica
    • Use Cases
      • Data Sharing
      • Data and Model Quality Control
      • Manual Data Curation
      • Version Control for your Application
      • Versioned MySQL Replica
      • Audit
      • Configuration Management
      • Offline First
  • Concepts
    • Dolt
      • Git
        • Commits
        • Log
        • Diff
        • Branch
        • Merge
        • Conflicts
        • Remotes
        • Working Set
      • SQL
        • Databases
        • Schema
        • Tables
        • Primary Keys
        • Types
        • Indexes
        • Views
        • Constraints
        • Triggers
        • Procedures
        • Users/Grants
        • Transactions
        • System Variables
      • RDBMS
        • Server
        • Backups
        • Replication
    • DoltHub/DoltLab
      • Permissions
      • Pull Requests
      • Issues
      • Forks
  • SQL Reference
    • Running the Server
      • Configuration
      • Access Management
      • Branch Permissions
      • Backups
      • Garbage Collection
      • Metrics
      • Replication
      • Troubleshooting
    • Version Control Features
      • Using Branches
      • Merges
      • Querying History
      • Using Remotes
      • Procedures
      • Functions
      • System Tables
      • System Variables
      • Saved Queries
    • SQL Language Support
      • Data Description
      • Expressions, Functions, Operators
      • Supported Statements
      • MySQL Information Schema
      • Collations and Character Sets
      • System Variables
      • Miscellaneous
    • Supported Clients
      • Programmatic
      • SQL Editors
    • Benchmarks and Metrics
      • Correctness
      • Latency
      • Import
  • CLI Reference
    • Commands
    • Git Comparison
  • Architecture
    • Overview
    • Storage Engine
      • Commit Graph
      • Prolly Trees
      • Block Store
    • SQL
      • Go MySQL Server
      • Vitess
  • Guides
    • Cheat Sheet
    • Contributing
      • dolt
      • go-mysql-server
    • MySQL to Dolt Replication
    • Importing Data
    • Integrations
  • Other
    • FAQ
    • Roadmap
    • Versioning
  • Products
    • Hosted Dolt
      • Getting Started
      • Notable Features
      • SQL Workbench
      • Cloning a Hosted Database
      • Using DoltHub as a Remote
      • Infrastructure
    • DoltHub
      • Data Sharing
      • API
        • Authentication
        • SQL
        • CSV
        • Database
        • Hooks
      • Continuous Integration
        • Getting Started
        • Workflow Reference
      • Transform File Uploads
      • Workspaces
    • DoltLab
    • Dolt Workbench
    • DoltgreSQL
Powered by GitBook
On this page
  • Architecture
  • Using Dolt

Was this helpful?

Edit on GitHub
Export as PDF
  1. Products
  2. Hosted Dolt

Infrastructure

PreviousUsing DoltHub as a RemoteNextDoltHub

Last updated 1 year ago

Was this helpful?

Before Hosted Dolt, if you wanted to run as your OLTP database you had to stand up your own instances deployed with the Dolt binary, and then handle all your own monitoring, alerting, and upgrades. This page will go through the infrastructure behind Hosted Dolt and the decision to use Dolt as our production OLTP server.

Architecture

The diagram below shows the key pieces of infrastructure used to run .

Once a new deployment is complete, the Deployment Manager will update the Hosted API with information such as the DNS to reach the host, the instance and volume ids, and other information about the deployment. After that the Deployment Manager continuously sends heartbeat requests, and monitors the state of running deployments.

Using Dolt

Resolving conflicts is not something MySQL supports so requiring conflict resolution in order to continue after a failed transaction would cause problems when integrating with standard MySQL tools and libraries. In order to support a workflow that involves merging and conflict resolution, Dolt allows you to create branches, work transactionally against your branch, then merge your changes with changes made by other users. If merging your changes results in a conflict, you have the ability to see the cells in conflict, and resolve the conflicts using standard SQL syntax.

In building we used many of the same technologies used on . Our website runs on React. It communicates with GraphQL servers which provide a translation layer for the Hosted API. The Hosted API is a Golang service providing GRPC endpoints which drive the website, and the hosted control plane. All state is managed through a Dolt sql-server instance.

The Deployment Manager is a Golang service which polls the Hosted API to understand what should be deployed. It also polls the EC2 API to understand what instances are currently running. It communicates with the running instances to get the health of hosted instances. With the full picture of what should be deployed, and what is currently running, it will use the EC2 API to create and destroy instances. New hosted instances are created using the user's spec and a prebuilt . The AMI has the , and our Instance Manager. After an instance is created we generate certificates, and create an using . Next it will generate either a self-signed cert, or a WebPKI certificate using 's API based on the options selected by the user. Once that is complete the Deployment Manager will make a deploy request to the Instance Manager.

The Instance Manager is another Golang application which runs on hosted instances and provides GRPC endpoints which are only open to the Deployment Manager. Once an instance is up the Deployment Manager sends a deploy request to the Instance Manager. The request includes all the information needed to start including IAM credentials and certificates. Once Dolt is running the Instance Manager will respond to heartbeat requests from the Deployment Manager, and poll the Prometheus endpoint to get application level metrics which it sends to Cloudwatch.

Also running on the instance, handles pushing logs to Cloudwatch as well as getting host level metrics such as cpu, memory, and disk usage.

When we started building the backend for we thought we would use an identical tech stack to what is used on . This means using Postgres as our OLTP DB. But the more we thought about it, the more we wanted to use instead. Dolt has some killer features that make it great choice for our application DB. First, the ability to clone the database in order to debug and develop locally is incredibly powerful. Also, in the case of an operational event, the ability to look at the diffs in the data and can significantly help with our ability to . Dolt also allows for the integration of CI/CD systems that can verify the quality of the data and look at how it's changed. These systems can look at things like the statistical variations of fields throughout the history of the commit graph.

Additionally, how can we tell others to use it for their applications if we aren't using it ourselves? This has many other benefits. We are learning a lot about operating our own software, and hitting errors before our customers do. As an example, failed Dolt transactions left the working set of a connection in a state with conflicts and did not automatically roll back changes. Dolt required you to resolve the conflicts, or rollback the transaction explicitly. The standard MySQL behavior automatically rolls back when a transaction fails, and the to interact with our database relies on this behavior (Explicitly calling rollback after a failed transaction is a NOP). So we changed the behavior of Dolt to match that of MySQL.

https://hosted.doltdb.com
DoltHub
AMI
latest Dolt
Amazon's Cloudwatch Agent
A record
Route 53
Let's Encrypt
Dolt
Amazon's Cloudwatch Agent
https://hosted.doltdb.com
DoltHub
Dolt
rollback changes
recover from outages
library we use
Dolt
https://hosted.doltdb.com
Hosted DoltDB Infrastructure