# Aug 23 Monday

## Continued from last week logs
I haven't got visor to log to DB, so I posted a github issue:
https://github.com/filecoin-project/lily/issues/673

I also asked Forrest about this. But Ian got back to me on the issue and
I was able to verify that visor did not print the crucial DEBUG OUTPUT:

DEBUG visor/storage storage/catalog.go:27 registering storage {"name": "Database1", "type": "postgresql"}

HERE'S WHAT I RAN TO VERIFY THAT
**Note: I tried to get docker to output logs with some sort of *--head* option, even though it's not implemented. I read documents on `--until` and `--since` and printed the docker's start unix time and used it for `--until`. None of it worked. With or without the `-f` option.**
The behavior was that the grep is always stuck.

Finally followed this: https://github.com/moby/moby/issues/13096#issuecomment-543299275
And ran
`docker logs Sentinel-Plus | head -n 30000 | grep -w "registering storage"
`
And I saw that no such line is printed.
However, when I stopped and rerun daemon with this option now
`visor daemon --config=/root/.lotus/config.toml 2>&1 >/dev/null | grep -w "registering storage"`

It printed the line I wanted. Therefore I have proven that I ran daemon wrong. It did not realize there is a database1 connected.

Nobody told me to run with the --config tag before now BTW.
## Re-run visor daemon with the `--config` specified.
I ran:
1. visor daemon --config /root/.lotus/config.toml
2. visor sync wait
3. "Wait for visor to catch up"
4. visor watch --storage=Database1 --tasks=messages,blocks,chaineconomics --window=60s --confidence 10
GO back to the host
`docker logs -f timescaledb`
I got a bunch of ERORROS SAYING NO SPACE IS AVAILABLE. BUT `df -h` shows plenty of space.

[centos@ip-172-31-25-159 ~]$ docker logs -f timescaledb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2021-08-21 00:47:51.669 UTC [30] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2021-08-21 00:47:52.066 UTC [35] LOG:  starting PostgreSQL 12.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
2021-08-21 00:47:52.068 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 00:47:52.085 UTC [36] LOG:  database system was shut down at 2021-08-21 00:47:51 UTC
2021-08-21 00:47:52.088 UTC [35] LOG:  database system is ready to accept connections
2021-08-21 00:47:52.089 UTC [42] LOG:  TimescaleDB background worker launcher connected to shared catalogs
 done
server started

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/000_install_timescaledb.sh
2021-08-21 00:47:52.330 UTC [48] WARNING:
        WELCOME TO
         _____ _                               _     ____________
        |_   _(_)                             | |    |  _  \ ___ \
          | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
          | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
          | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
          |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
                       Running version 1.7.4
psql:/tmp/tmp.FabNcc:1: WARNING:
WELCOME TO
 _____ _                               _     ____________
|_   _(_)                             | |    |  _  \ ___ \
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 1.7.4
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

        For more information on TimescaleDB, please visit the following links:

         1. Getting started: https://docs.timescale.com/getting-started
         2. API reference documentation: https://docs.timescale.com/api
         3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

        Note: TimescaleDB collects anonymous reports to better understand and assist our users.
        For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

2021-08-21 00:47:52.330 UTC [48] CONTEXT:  PL/pgSQL function inline_code_block line 23 at RAISE
CREATE EXTENSION
2021-08-21 00:47:52.498 UTC [51] WARNING:
        WELCOME TO
         _____ _                               _     ____________
        |_   _(_)                             | |    |  _  \ ___ \
          | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
          | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
          | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
          |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
                       Running version 1.7.4
        For more information on TimescaleDB, please visit the following links:

         1. Getting started: https://docs.timescale.com/getting-started
         2. API reference documentation: https://docs.timescale.com/api
         3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

        Note: TimescaleDB collects anonymous reports to better understand and assist our users.
        For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

2021-08-21 00:47:52.498 UTC [51] CONTEXT:  PL/pgSQL function inline_code_block line 23 at RAISE
psql:/tmp/tmp.FabNcc:1: WARNING:
WELCOME TO
 _____ _                               _     ____________
|_   _(_)                             | |    |  _  \ ___ \
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ /
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 1.7.4
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

2021-08-21 00:47:52.501 UTC [53] ERROR:  TimescaleDB background worker connected to template database, exiting
2021-08-21 00:47:52.501 UTC [52] WARNING:  telemetry error: HTTP connection read error
CREATE EXTENSION

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/001_timescaledb_tune.sh
2021-08-21 00:47:52.502 UTC [35] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 53) exited with exit code 1
Using postgresql.conf at this path:
/var/lib/postgresql/data/postgresql.conf

Writing backup to:
/tmp/timescaledb_tune.backup202108210047

Recommendations based on 248.99 GB of available memory and 32 CPUs for PostgreSQL 12
shared_buffers = 63742MB
effective_cache_size = 191227MB
maintenance_work_mem = 2047MB
work_mem = 20397kB
timescaledb.max_background_workers = 8
max_worker_processes = 43
max_parallel_workers_per_gather = 16
max_parallel_workers = 32
wal_buffers = 16MB
min_wal_size = 512MB
default_statistics_target = 500
random_page_cost = 1.1
checkpoint_completion_target = 0.9
max_locks_per_transaction = 512
autovacuum_max_workers = 10
autovacuum_naptime = 10
effective_io_concurrency = 200
timescaledb.last_tuned = '2021-08-21T00:47:52Z'
timescaledb.last_tuned_version = '0.9.0'
Saving changes to: /var/lib/postgresql/data/postgresql.conf

waiting for server to shut down...2021-08-21 00:47:52.527 UTC [35] LOG:  received fast shutdown request
.2021-08-21 00:47:52.528 UTC [35] LOG:  aborting any active transactions
2021-08-21 00:47:52.528 UTC [49] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 00:47:52.528 UTC [42] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-08-21 00:47:52.528 UTC [49] FATAL:  terminating connection due to administrator command
2021-08-21 00:47:52.528 UTC [42] FATAL:  terminating connection due to administrator command
2021-08-21 00:47:52.529 UTC [49] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 00:47:52.529 UTC [35] LOG:  background worker "logical replication launcher" (PID 43) exited with exit code 1
2021-08-21 00:47:52.529 UTC [35] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 42) exited with exit code 1
2021-08-21 00:47:52.529 UTC [35] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 49) exited with exit code 1
2021-08-21 00:47:52.530 UTC [37] LOG:  shutting down
2021-08-21 00:47:52.561 UTC [35] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2021-08-21 00:47:52.636 UTC [1] LOG:  starting PostgreSQL 12.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
2021-08-21 00:47:52.636 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 00:47:52.636 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 00:47:52.639 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 00:47:53.529 UTC [71] LOG:  database system was shut down at 2021-08-21 00:47:52 UTC
2021-08-21 00:47:53.536 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 00:47:53.537 UTC [77] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2021-08-21 00:48:15.478 UTC [98] FATAL:  database "visor_test" does not exist
2021-08-21 01:47:52.590 UTC [3338] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 01:47:52.590 UTC [3338] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 02:47:52.637 UTC [6592] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 02:47:52.637 UTC [6592] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 03:10:11.625 UTC [7800] LOG:  invalid length of startup packet
2021-08-21 03:10:11.625 UTC [7801] LOG:  invalid length of startup packet
2021-08-21 03:47:52.650 UTC [9846] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 03:47:52.650 UTC [9846] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 04:47:52.553 UTC [13104] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 04:47:52.553 UTC [13104] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 05:30:15.665 UTC [15403] LOG:  invalid length of startup packet
2021-08-21 05:30:15.665 UTC [15404] LOG:  invalid length of startup packet
2021-08-21 05:47:52.535 UTC [16354] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 05:47:52.535 UTC [16354] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 06:47:52.620 UTC [19622] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 06:47:52.620 UTC [19622] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 07:30:47.442 UTC [21936] LOG:  invalid length of startup packet
2021-08-21 07:30:47.443 UTC [21937] LOG:  invalid length of startup packet
2021-08-21 07:47:52.553 UTC [22860] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 07:47:52.553 UTC [22860] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 08:47:52.876 UTC [26097] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 08:47:52.876 UTC [26097] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 09:47:52.690 UTC [29332] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 09:47:52.690 UTC [29332] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 10:47:52.552 UTC [32565] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 10:47:52.552 UTC [32565] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 11:47:52.594 UTC [3328] LOG:  the "timescaledb" extension is not up-to-date
2021-08-21 11:47:52.594 UTC [3328] HINT:  The most up-to-date version is 2.4.0, the installed version is 1.7.4
2021-08-21 12:40:48.884 UTC [6173] LOG:  invalid length of startup packet
2021-08-21 12:40:48.884 UTC [6174] LOG:  invalid length of startup packet
2021-08-21 17:29:58.072 UTC [1] LOG:  received fast shutdown request
2021-08-21 17:29:58.077 UTC [1] LOG:  aborting any active transactions
2021-08-21 17:29:58.078 UTC [77] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-08-21 17:29:58.078 UTC [79] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 17:29:58.078 UTC [21432] FATAL:  terminating connection due to administrator command
2021-08-21 17:29:58.078 UTC [77] FATAL:  terminating connection due to administrator command
2021-08-21 17:29:58.078 UTC [79] FATAL:  terminating connection due to administrator command
2021-08-21 17:29:58.087 UTC [1] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 77) exited with exit code 1
2021-08-21 17:29:58.088 UTC [1] LOG:  background worker "logical replication launcher" (PID 78) exited with exit code 1
2021-08-21 17:29:58.088 UTC [1] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 79) exited with exit code 1
2021-08-21 17:29:58.100 UTC [72] LOG:  shutting down
2021-08-21 17:29:59.491 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-08-21 17:30:14.080 UTC [1] LOG:  starting PostgreSQL 12.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
2021-08-21 17:30:14.080 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 17:30:14.080 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 17:30:14.090 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 17:30:15.202 UTC [21] LOG:  database system was shut down at 2021-08-21 17:29:58 UTC
2021-08-21 17:30:15.206 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 17:30:15.209 UTC [27] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2021-08-21 17:51:28.252 UTC [1] LOG:  received fast shutdown request
2021-08-21 17:51:28.253 UTC [1] LOG:  aborting any active transactions
2021-08-21 17:51:28.253 UTC [29] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 17:51:28.253 UTC [27] LOG:  terminating TimescaleDB background worker launcher due to administrator command
2021-08-21 17:51:28.253 UTC [221] FATAL:  terminating connection due to administrator command
2021-08-21 17:51:28.253 UTC [27] FATAL:  terminating connection due to administrator command
2021-08-21 17:51:28.253 UTC [29] FATAL:  terminating connection due to administrator command
2021-08-21 17:51:28.253 UTC [29] LOG:  terminating TimescaleDB job scheduler due to administrator command
2021-08-21 17:51:28.254 UTC [1] LOG:  background worker "TimescaleDB Background Worker Launcher" (PID 27) exited with exit code 1
2021-08-21 17:51:28.254 UTC [1] LOG:  background worker "logical replication launcher" (PID 28) exited with exit code 1
2021-08-21 17:51:28.254 UTC [1] LOG:  background worker "TimescaleDB Background Worker Scheduler" (PID 29) exited with exit code 1
2021-08-21 17:51:28.368 UTC [22] LOG:  shutting down
2021-08-21 17:51:28.816 UTC [1] LOG:  database system is shut down

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-08-21 17:51:38.422 UTC [1] LOG:  starting PostgreSQL 12.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.3.0) 9.3.0, 64-bit
2021-08-21 17:51:38.422 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-08-21 17:51:38.422 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-08-21 17:51:38.426 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-21 17:51:39.328 UTC [21] LOG:  database system was shut down at 2021-08-21 17:51:28 UTC
2021-08-21 17:51:39.332 UTC [1] LOG:  database system is ready to accept connections
2021-08-21 17:51:39.333 UTC [27] LOG:  TimescaleDB background worker launcher connected to shared catalogs
2021-08-21 18:01:52.270 UTC [575] FATAL:  no PostgreSQL user name specified in startup packet
2021-08-21 18:01:52.344 UTC [576] FATAL:  no PostgreSQL user name specified in startup packet
2021-08-21 18:02:05.964 UTC [594] FATAL:  no PostgreSQL user name specified in startup packet
2021-08-21 18:02:06.037 UTC [595] FATAL:  no PostgreSQL user name specified in startup packet
2021-08-21 19:36:53.536 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.178 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.178 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.630 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.630 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.822 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:54.822 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.274 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.274 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.467 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.467 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.918 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:55.918 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.111 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.111 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.563 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.563 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.755 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:56.755 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.207 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.207 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.399 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.399 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.851 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:57.852 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.043 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.043 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.496 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.496 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.687 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:58.687 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.140 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.140 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.332 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.332 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.784 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.784 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.976 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:36:59.976 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:00.428 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:00.428 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:00.620 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:00.620 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.073 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.073 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.265 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.265 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.717 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.717 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.725 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.725 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.726 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.727 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.909 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:01.909 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:02.362 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:02.362 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:02.553 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:02.553 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.006 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.006 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.197 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.197 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.599 UTC [5667] LOG:  using stale statistics instead of current ones because stats collector is not responding
2021-08-21 19:37:03.601 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.601 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.601 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.602 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.641 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:03.641 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.285 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.285 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.687 UTC [25] LOG:  using stale statistics instead of current ones because stats collector is not responding
2021-08-21 19:37:04.688 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.688 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.703 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.703 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.704 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:04.704 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.326 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.326 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.348 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.348 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.970 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.971 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.992 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:05.992 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:06.615 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:06.615 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:06.636 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:06.636 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.259 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.259 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.281 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.281 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.904 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.904 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.924 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:07.924 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:08.548 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:08.548 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:08.568 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:08.569 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.192 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.192 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.213 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.213 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.837 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.837 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.857 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:09.857 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:10.482 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:10.482 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device
2021-08-21 19:37:10.501 UTC [26] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": No space left on device

Oh I'm not reading the latest logs!
Ran this instead
`docker logs -f --tail 100 timescaledb`
And it seems to be working.

## Next steps, discussed with Forrest
My overall next step is to learn how to 1.export everything blockstore had into db. 2. maybe clean/delete the blockstore’s contents and 3. rerun daemon (i have been doing that already) safely. This step’s purpose is to help me conserve disk space.

> Forrests suggests that we just try and error.