Administrator Guide

In DoltLab's current version, there is no Administrator (Admin) web-based UI or dashboard as it is still in development. In the meantime, the following information can help DoltLab Admins manually perform some common administration tasks, see below for details.
1.​File Issues and View Release Notes​
2.​Backup DoltLab Data​
3.​Upgrade DoltLab Versions Without Data Loss​
4.​Send Service Logs To DoltLab Team​
5.​Connect with the DoltLab Team​
6.​Authenticate a Dolt Client to use DoltLab Account​
7.​Monitor DoltLab with cAdvisor and Prometheus​
8.​Connect to an SMTP Server with Implicit TLS​
9.​Troubleshoot SMTP Server Connection Problems​
10.​Prevent Unauthorized User Account Creation​
File Issues and View Release Notes
DoltLab's source code is currently closed, but you can file DoltLab issues or view DoltLab's release notes in our issues repository.
Backup and Restore Volumes
DoltLab currently persists all data to local disk using Docker volumes. To backup or restore DoltLab's data, we recommend the following steps which follow Docker's official volume backup and restore documentation, with the exception of DoltLab's postgres server. To backup the postgres server we recommend dumping the database with pg_dump and restoring the database from the dump using psql.
Backing Up and Restoring Remote Data and User Uploaded Data
To backup DoltLab's remote data, the database data for all database on a given DoltLab instance, leave DoltLab's services up and run:
1
# backup remote data stored in DoltLab RemoteAPI's volume and save to a tar file
2
docker run --rm --volumes-from doltlab_doltlabremoteapi_1 -v $(pwd):/backup ubuntu tar cvf /backup/remote-data.tar /doltlab-remote-storage
Copied!
This will create a tar file called remote-data.tar in your working directory.
To backup user uploaded files, run:
1
# backup remote data stored in DoltLab RemoteAPI's volume and save to a tar file
2
docker run --rm --volumes-from doltlab_doltlabfileserviceapi_1 -v $(pwd):/backup ubuntu tar cvf /backup/user-uploaded-data.tar /doltlab-user-uploads
Copied!
This will create a tar file called user-uploaded-data.tar in your working directory.
Before restoring DoltLab's volumes from a backup, first, stop the running DoltLab services, prune the Docker containers, and remove the old volume(s):
1
cd doltlab
2
​
3
# stop the DoltLab services
4
docker-compose stop
5
​
6
# prune containers
7
docker container prune
8
​
9
# remove the remote data volume
10
docker volume rm doltlab_doltlab-remote-storage
11
​
12
# remove the user uploaded data
13
docker volume rm doltlab_doltlab-user-uploads
Copied!
Next, start DoltLab's services using the start-doltlab.sh script. After the script completes, cd into the directory containing the remote-data.tar backup file and run:
1
# restore remote data from tar
2
docker run --rm --volumes-from doltlab_doltlabremoteapi_1 -v $(pwd):/backup ubuntu bash -c "cd /doltlab-remote-storage && tar xvf /backup/remote-data.tar --strip 1"
Copied!
To restore user uploaded data, cd into the directory containing user-uploaded-data.tar and run:
1
# restore remote data from tar
2
docker run --rm --volumes-from doltlab_doltlabfileserviceapi_1 -v $(pwd):/backup ubuntu bash -c "cd /doltlab-user-uploads && tar xvf /backup/user-uploaded-data.tar --strip 1"
Copied!
Backing Up and Restoring PostgreSQL Data
For backing up data from DoltLab's postgres server, we recommend executing a data dump with pg_dump. To do so, keep DoltLab's services up and run:
1
# dump postgres to postgres-dump.sql
2
docker run --rm --network doltlab_doltlab -e PGPASSWORD=<POSTGRES_PASSWORD> -v $(pwd):/doltlab-db-dumps postgres:13-bullseye bash -c "pg_dump --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi > /doltlab-db-dumps/postgres-dump.sql"
Copied!
The value of PGPASSWORD should be the POSTGRES_PASSWORD set when DoltLab was first deployed.
To restore a postgres server from postgres-dump.sql, first stop the running DoltLab services, remove the stopped containers, and remove the old postgres server volume:
1
cd doltlab
2
​
3
# stop DoltLab
4
docker-compose stop
5
​
6
# remove containers
7
docker container prune
8
​
9
# remove old postgres data volume
10
docker volume rm doltlab_doltlabdb-data
Copied!
Next, edit the postgres-dump.sql file by adding the line SET session_replication_role = replica; near the top of the file:
1
SET statement_timeout = 0;
2
SET lock_timeout = 0;
3
SET idle_in_transaction_session_timeout = 0;
4
SET client_encoding = 'UTF8';
5
SET standard_conforming_strings = on;
6
SELECT pg_catalog.set_config('search_path', '', false);
7
SET check_function_bodies = false;
8
SET xmloption = content;
9
SET client_min_messages = warning;
10
SET row_security = off;
11
​
12
/* We add this to disable triggers and rules during import */
13
SET session_replication_role = replica;
14
...
Copied!
​Start DoltLab's services again using the start-doltlab.sh script. After the script completes, cd into the directory containing the postgres-dump.sql file and run:
1
# import the postgres dump into the running server
2
docker run --rm --network doltlab_doltlab -e PGPASSWORD=<POSTGRES_PASSWORD> -v $(pwd):/doltlab-db-dumps postgres:13-bullseye bash -c "psql --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi < /doltlab-db-dumps/postgres-dump.sql"
Copied!
Send Service Logs to DoltLab Team
DoltLab is comprised of multiple services running in a single Docker network via Docker compose. Logs for a particular service can be viewed using the docker logs <container name> command. For example, to view to logs of doltlabapi service, run:
1
docker logs doltlab_doltlabapi_1
Copied!
If you need to send service logs to the DoltLab team, first locate the logs on the host using the docker inspect command, then cp the logs to your working directory:
1
DOLTLABAPI_LOGS=$(docker inspect --format='{{.LogPath}}' doltlab_doltlabapi_1)
2
cp "$DOLTLABAPI_LOGS" ./doltlab-api-logs.json
Copied!
Next change permissions on the copied file to enable reads by running:
1
chmod 0644 ./doltlab-api-logs.json
Copied!
Finally, download the copied log file from your DoltLab host using scp. You can then send this and any other log files to the DoltLab team member you're working with via email.
Upgrading DoltLab
Upgrading to newer versions of DoltLab requires downtime. This means that DoltLab Admins will need to kill the old version of DoltLab, and install the newer one.
In addition, some early versions have different database schemas than newer ones. If the docker volumes of an old version of DoltLab contain non-precious or test-only data, then DoltLab Admins can simply remove these Docker volumes and run the start-doltlab.sh script from the newer DoltLab version. This script will simply create new Docker volumes with the appropriate schema for that DoltLab version.
If you want to upgrade your DoltLab version without losing any data, please follow the upgrade guidelines below.
Upgrade from DoltLab v0.3.0+
DoltLab versions >= v0.3.0 support schema migrations without data loss. To upgrade to a DoltLab version after v0.3.0, simply stop your old version of DoltLab, then download and unzip the newer DoltLab version to the same location as your previous version. This will ensure that when you start the new version of DoltLab using the start-doltlab.sh script, the old DoltLab version's Docker volumes get attached to the new version's containers.
1
# stop old DoltLab
2
cd doltlab
3
docker-compose stop
4
cd ../
5
rm -rf doltlab
6
​
7
# download and unzip newer DoltLab
8
curl -OL https://doltlab-releases.s3.amazonaws.com/linux/amd64/doltlab-latest.zip
9
unzip doltlab-latest.zip -d doltlab
10
​
11
# start DoltLab
12
cd doltlab
13
./start-doltlab.sh
Copied!
Upgrade from DoltLab v0.2.0 to v0.3.0
To upgrade without data loss, follow the same instructions for upgrading found in the Upgrade from DoltLab v0.1.0 v0.2.0 section.
Upgrade from DoltLab v0.1.0 to v0.2.0 Without Data Loss
To upgrade DoltLab v0.1.0 to v0.2.0, leave DoltLab v0.1.0's services running and connect a PostgreSQL client from inside the doltlab_doltlab Docker network to the running doltlab_doltlabdb_1 server. On the DoltLab host machine, run:
1
# dump data from DoltLab v0.1.0's postgres server to doltlab-db-dumps
2
docker run --rm --network doltlab_doltlab -e PGPASSWORD=<POSTGRES_PASSWORD> -v $(pwd):/doltlab-db-dumps postgres:13-bullseye bash -c "pg_dump --data-only --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi > /doltlab-db-dumps/doltlab-v0.1.0-dump-data-only.sql"
Copied!
The value of PGPASSWORD should be the POSTGRES_PASSWORD set when DoltLab v0.1.0 was first deployed. You should now see the SQL dump file called doltlab-v0.1.0-dump-data-only.sql.
Next, edit the doltlab-v0.1.0-dump-data-only.sql file by adding the line SET session_replication_role = replica; near the top of the file:
1
SET statement_timeout = 0;
2
SET lock_timeout = 0;
3
SET idle_in_transaction_session_timeout = 0;
4
SET client_encoding = 'UTF8';
5
SET standard_conforming_strings = on;
6
SELECT pg_catalog.set_config('search_path', '', false);
7
SET check_function_bodies = false;
8
SET xmloption = content;
9
SET client_min_messages = warning;
10
SET row_security = off;
11
​
12
/* We add this to disable triggers and rules during import */
13
SET session_replication_role = replica;
14
...
Copied!
You can now stop the DoltLab v0.1.0 services and delete the Docker caches and stopped containers on the host by running:
1
cd doltlab
2
​
3
# stop DoltLab
4
docker-compose stop
5
​
6
# remove containers
7
docker container prune
Copied!
Next, remove the Docker volume used with DoltLab v0.1.0's doltlab_doltlabdb_1 postgres server by running:
1
docker volume rm doltlab_doltlabdb-data
Copied!
​Download DoltLab v0.2.0, unzip it's contents, and start DoltLab v0.2.0's services by running the start-doltlab.sh script.
After the script completes, confirm DoltLab v0.2.0's services are running with docker ps:
1
docker ps
2
CONTAINER ID   IMAGE                                                             COMMAND                  CREATED       STATUS       PORTS                                                                                     NAMES
3
3f47c64db5af   public.ecr.aws/dolthub/doltlab/dolthub-server:v0.2.0              "docker-entrypoint.s…"   2 hours ago   Up 2 hours   3000/tcp                                                                                  doltlab_doltlabui_1
4
444819d72e64   public.ecr.aws/dolthub/doltlab/dolthubapi-graphql-server:v0.2.0   "docker-entrypoint.s…"   2 hours ago   Up 2 hours   9000/tcp                                                                                  doltlab_doltlabgraphql_1
5
d2f5583a19fb   public.ecr.aws/dolthub/doltlab/dolthubapi-server:v0.2.0           "/app/go/services/do…"   2 hours ago   Up 2 hours                                                                                             doltlab_doltlabapi_1
6
defdd789c7c3   public.ecr.aws/dolthub/doltlab/doltremoteapi-server:v0.2.0        "/app/go/services/do…"   2 hours ago   Up 2 hours   0.0.0.0:100->100/tcp, :::100->100/tcp, 0.0.0.0:50051->50051/tcp, :::50051->50051/tcp      doltlab_doltlabremoteapi_1
7
93d8c28b9762   public.ecr.aws/dolthub/doltlab/fileserviceapi-server:v0.2.0       "/app/go/services/fi…"   2 hours ago   Up 2 hours                                                                                             doltlab_doltlabfileserviceapi_1
8
e3adba4e26ce   envoyproxy/envoy-alpine:v1.18-latest                              "/docker-entrypoint.…"   2 hours ago   Up 2 hours   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:4321->4321/tcp, :::4321->4321/tcp, 10000/tcp   doltlab_doltlabenvoy_1
9
20dbd21a94ff   public.ecr.aws/dolthub/doltlab/postgres-server:v0.2.0             "docker-entrypoint.s…"   2 hours ago   Up 2 hours   5432/tcp                                                                                  doltlab_doltlabdb_1
Copied!
Now, connect a PostgreSQL client from inside the doltlab_doltlab Docker network to the running doltlab_doltlabdb_1 server to perform the data dump import. Then, update the has_dolt column in the users table.
1
# import the data dump into the running DoltLab v0.2.0 postgres server
2
docker run --rm --network doltlab_doltlab -e PGPASSWORD=<POSTGRES_PASSWORD> -v $(pwd)/doltlab-db-dumps:/doltlab-db-dumps postgres:13-bullseye bash -c "psql --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi < /doltlab-db-dumps/doltlab-v0.1.0-dump-data-only.sql"
3
​
4
# update the users table to prevent DoltLab v0.2.0 error
5
docker run --rm --network doltlab_doltlab -e PGPASSWORD=<POSTGRES_PASSWORD> -v $(pwd)/doltlab-db-dumps:/doltlab-db-dumps postgres:13-bullseye bash -c "psql --host=doltlab_doltlabdb_1 --port=5432 --username=dolthubadmin dolthubapi -c 'update users set has_dolt = 'false';'"
Copied!
You have now completed the upgrade, and should no be running DoltLab v0.2.0 with your postgres data from DoltLab v0.1.0.
Connect with the DoltLab Team
If you need to connect to a DoltLab team member, the best way to do so is on Discord, in the #doltlab server.
Authenticate a Dolt Client to use a DoltLab Account
As of Dolt v0.39.0, the dolt login command can be used to authenticate against DoltLab instances.
To authenticate a client against DoltLab with this command, use the --auth-endpoint, --login-url, and --insecure arguments to point your Dolt client at the DoltLab instance you want to authenticate against.
--auth-endpoint should point at the DoltLab RemoteAPI Server running on port 50051. --login-url should point at the DoltLab instance's credentials page. --insecure a boolean flag, should be used since DoltLab does not currently support TLS gRPC connections.
1
dolt login --insecure --auth-endpoint doltlab.dolthub.com:50051 --login-url http://doltlab.dolthub.com/settings/credentials
Copied!
Running the command will open your browser window to the --login-url with credentials populated in the "Public Key" field. Simply add a "Description" and click "Create", then return to your terminal to see your Dolt client successfully authenticated.
1
Credentials created successfully.
2
pub key: 9pg8rrkaqouuno4rihdlkb6pvaf16t134dscb68dlg0u0261rmtg
3
/.dolt/creds/cepi47m5sojotm8s2e8rkqm9fdjocidsc42canggvb8e0.jwk
4
Opening a browser to:
5
	http://doltlab.dolthub.com/settings/credentials#9pg8rrkaqouuno4rihdlkb6pvaf16t134dscb68dlg0u0261rmtg
6
Please associate your key with your account.
7
Checking remote server looking for key association.
8
requesting update
9
requesting update
10
​
11
​
12
​
13
Key successfully associated with user: <user> email <email>
Copied!
For Dolt clients < v0.39.0, or to authenticate without using the dolt login command, first run the dolt creds new command, which will output a new public key:
1
dolt creds new
2
Credentials created successfully.
3
pub key: fef0kj7ia389i5atv8mcb31ksg9h3i6cji7aunm4jea9tccdl2cg
Copied!
Copy the generated public key and run the dolt creds use command:
1
dolt creds use fef0kj7ia389i5atv8mcb31ksg9h3i6cji7aunm4jea9tccdl2cg
Copied!
Next, login to your DoltLab account, click your profile image, then click "Settings" and then "Credentials".
Paste the public key into the "Public Key" field, write a description in the "Description" field, then click "Create".
Your Dolt client is now authenticated for this DoltLab account.
Monitor DoltLab with cAdvisor and Prometheus
As of DoltLab v0.3.0, Prometheus gRPC service metrics for DoltLab's Remote API Server, doltlabremoteapi, and DoltLab's Main API server, doltlabapi, are published on port 7770.
The metrics endpoints for these services are available at endpoints corresponding to their container name. For DoltLab's Remote API, thats :7770/doltlabremoteapi, and for DoltLab's Main API that's :7770/doltlabapi.
You can view the doltlabremoteapi service metrics for our DoltLab demo instance here, http://doltlab.dolthub.com:7770/doltlabremoteapi and you can view the doltlabapi service metrics here http://doltlab.dolthub.com:7770/doltlabapi.
To make these endpoints available to Prometheus, open port 7770 on your DoltLab host.
We recommend monitoring DoltLab with cAdvisor, which will expose container resource and performance metrics to Prometheus. Before running cAdvisor, open port 8080 on your DoltLab host as well. cAdvisor will display DoltLab's running containers via a web UI on :8080/docker and will publish Prometheus metrics for DoltLab's container at :8080/metrics by default.
Run cAdvisor as a Docker container in daemon mode with:
1
docker run -d -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro -v /dev/disk/:/dev/disk:ro -p 8080:8080 --name=cadvisor --privileged gcr.io/cadvisor/cadvisor:v0.39.3
Copied!
To run a Prometheus server on your DoltLab host machine, first open port 9090 on the DoltLab host. Then, write the following prometheus.yml file on the host:
1
global:
2
  scrape_interval:     5s
3
  evaluation_interval: 10s
4
scrape_configs:
5
  - job_name: cadivsor
6
    static_configs:
7
      - targets: ['host.docker.internal:8080']
8
  - job_name: prometheus
9
    static_configs:
10
      - targets: ['localhost:9090']
11
  - job_name: doltlabremoteapi
12
    metrics_path: '/doltlabremoteapi'
13
    static_configs:
14
      - targets: ['host.docker.internal:7770']
15
  - job_name: doltlabapi
16
    metrics_path: '/doltlabapi'
17
    static_configs:
18
      - targets: ['host.docker.internal:7770']
Copied!
Then, start the Prometheus server as a Docker container running in daemon mode:
1
docker run -d --add-host host.docker.internal:host-gateway --name=prometheus -p 9090:9090 -v "$(pwd)"/prometheus.yml:/etc/prometheus/prometheus.yml:ro prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml
Copied!
--add-host host.docker.internal:host-gateway is only required if running the Prometheus server on the DoltLab host. If running it elsewhere, this argument may be omitted, and the host.docker.internal hostname in prometheus.yml can be changed to the hostname of your DoltLab host.
Connect to an SMTP Server with Implicit TLS
Starting with DoltLab v0.4.2, connections to existing SMTP servers using implicit TLS (on port 465, for example) are supported. To connect using implicit TLS, edit the docker-compose.yaml included in the DoltLab zip. Under the doltlabapi section, in the command block, add the following argument:
1
...
2
doltlabapi:
3
  ...
4
  command:
5
    ...
6
    -emailImplicitTLS
7
    ...
Copied!
After adding the argument, restart DoltLab for it to take effect. Additionally, TLS verification can be skipped by adding the additional argument -emailInsecureTLS.
Troubleshoot SMTP Server Connection Problems
DoltLab requires a connection to an existing SMTP server in order for users to create accounts, verify email addresses, reset forgotten passwords, and collaborate on databases.
Starting with DoltLab v0.4.1, the default user admin is created when DoltLab starts up, which allows admins to sign in to their DoltLab instance even if they are experiencing SMTP server connection issues.
To help troubleshoot and resolve SMTP server connection issues, we've published the following go tool to help diagnose the SMTP connection issues on the host running DoltLab.
Starting with DoltLab v0.4.2, this tool is now included as an executable binary in DoltLab's zip, called smtp_connection_helper.
For usage run ./smtp_connection_helper --help which will output:
1
'smtp_connection_helper' is a simple tool used to ensure you can successfully connect to an smtp server.
2
If the connection is successful, this tool will send a test email to a single recipient from a single sender.
3
By default 'smtp_connection_helper' will attempt to connect to the SMTP server with STARTTLS. To use implicit TLS, use --implicit-tls
4
​
5
Usage:
6
​
7
./smtp_connection_helper \
8
--host <smtp hostname> \
9
--port <smtp port> \
10
--from <email address> \
11
--to <email address> \
12
--message {This is a test email message sent with smtp_connection_helper!} \
13
--subject {Testing SMTP Server Connection} \
14
--client-hostname {localhost} \
15
--auth <plain|external|anonymous|oauthbearer|disable> \
16
[--username smtp username] \
17
[--password smtp password] \
18
[--token smtp oauth token] \
19
[--identity smtp identity] \
20
[--trace anonymous trace] \
21
[--implicit-tls] \
22
[--insecure]
23
​
Copied!
To send a test email using plain authentication, run:
1
./smtp_connection_helper \
2
--host existing.smtp.server.hostname \
3
--port 587 \ #STARTTLS port
4
--auth plain \
5
--username XXXXXXXX \
6
--password YYYYYYY \
7
--from [email protected] \
8
--to [email protected]
9
Sending email with auth method: plain
10
Successfully sent email!
Copied!
To send a test email using plain authentication with implicit TLS, run:
1
./smtp_connection_helper \
2
--host existing.smtp.server.hostname \
3
--port 465 \ #TLS Wrapper Port
4
--implicit-tls \
5
--auth plain \
6
--username XXXXXXXX \
7
--password YYYYYYY \
8
--from [email protected] \
9
--to [email protected]
10
Sending email with auth method: plain
11
Successfully sent email!
Copied!
Prevent Unauthorized User Account Creation
DoltLab for non-enterprise use currently supports explicit email whitelisting to prevent account creation by unauthorized users.
To enable DoltLab's email whitelisting feature, edit the docker-compose.yaml file included in DoltLab's zip.
Under the doltlabapi section, in the command block, remove the argument -dolthubWhitelistAllowAll. Restart your DoltLab instance for this to take effect.
Once DoltLab is restarted, your DoltLab instance will now check a PostgreSQL table called email_whitelist_elements before permitting account creation. Only user's with email addresses present in this table will be able to create accounts on your DoltLab instance.
To whitelist an email for account creation in your instance, you will need to insert their email address into the email_whitelist_elements table.
As of DoltLab v0.4.2, a script to easily connect to your DoltLab instance's running PostgreSQL server is included in the zip, called shell-db.sh.
Use this script by supplying the POSTGRES_PASSWORD you used to start your DoltLab instance, as PGPASSWORD here. Run:
1
PGPASSWORD=<your postgres password> ./shell-db.sh
Copied!
You will see a dolthubapi=# PostgresSQL prompt connected to your DoltLab instance.
You can now execute the following INSERT to allow a specific user with [email protected] to create an account on your DolLab instance:
1
INSERT INTO email_whitelist_elements (email_address, updated_at, created_at) VALUES ('[email protected]', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);
Copied!