Troubleshooting
Debugging a running Dolt server can be challenging. This document covers the debugging basics and how to diagnose what is happening from common symptoms.
Basics
Make sure you are running the latest Dolt version
To upgrade the server, download the latest Dolt binary for your platform and replace the Dolt binary on your PATH
with the downloaded one. Running the install process on most platforms again will do this for you. Restart the Dolt server using dolt sql-server
to have your running server start using the latest binary.
Examine your CPU, Memory, and Disk usage
Set your log level to DEBUG or TRACE
To see queries being run against the server, query results, and query latency set your Dolt log level to DEBUG
or TRACE
. This can be done by starting the server like so dolt sql-server --loglevel=debug
or by setting log_level: debug
in your config.yaml
. Your logs should be visible in the shell you started dolt sql-server
in.
EXPLAIN for complex queries
Dolt supports the SQL EXPLAIN
operation in order for you to see the plan for complex queries. Rearranging your query to perform fewer JOIN
s or make better use of indexes can help speed up complex queries.
Compare to MySQL
Submitting Issues
Problems
Dolt operational issues usually manifest as slow SQL queries. In rare occasions, Dolt may consume more of your system's resources than you expect. In these cases, this document has some recommendations.
Server Consuming Disk
Dolt creates disk garbage on write. This can sometimes become a substantial portion of the disk Dolt is consuming. Dolt ships with a garbage collection function. Running the garbage collection function can free disk.
To run garbage collection offline, stop your dolt sql-server
, navigate to the Dolt directory where your database is stored and run dolt gc
. Once the operation is complete, restart your server using dolt sql-server
.
Disk garbage is especially pronounced after imports. We recommend concluding imports with a dolt gc
call.
Using primary keys with random values. Inserts into indexes with random values guarantees that edits will occur all throughout the index instead of being clustered around the same key space. This results in a rewrite of the prolly tree thereby increasing storage disproportionately to the delta of the changes.
Adding a column to a table. A new column forks the storage of the table resulting in a loss of structural sharing. Dolt is row major and builds chunks for each primary key, row values pair. The row values encodes the schema length so every row now requires a new chunk.
Server Consuming Memory
Serving Dolt databases requires a fair amount of memory. As a general rule, we recommend a minimum of 2GB available RAM for any production use case. Larger databases or heavier workloads should start at 4GB of RAM, and 8GB is common for our production customers. Your server's RAM requirements grow with the size of the database being served, the number of concurrent connections, the size / complexity of queries being executed, and other factors. These numbers can vary dramatically and are only intended as first-step guidance on resource requirements. Your use case may require more or less memory to run well, and you should load test to determine the correct ceiling.
Server Consuming CPU
Too much write concurrency
Currently, Dolt is not a high-throughput database for writes. The current transaction model serializes all writes, which means that after a certain threshold of writer concurrency, you'll observe increasing latency for write operations which becomes worse as more writers pile up. As of this writing Dolt can handle approximately 300 writes per second, but this number can be lower depending on the size of the database, the size of the updates, replication settings, and other factors.
Improving maximum write concurrency is an ongoing project.
Last updated