Skip to content

Failed clone cleanup

Goal

Fix the error in DLE logs:

2021/01/25 04:16:40 [ERROR] Failed to check the idleness of clone: failed to get the last session activity: failed to init file selector: failed to read a log directory: open /var/lib/dblab/clones/dblab_clone_6002/data/log: no such file or directory.

In dblab_server:


bash-5.0# ls -la /var/lib/dblab/clones/dblab_clone_6002
total 9
drwxr-xr-x    2 root     root             2 Dec 28 12:01 .
drwxr-xr-x  276 root     root           276 Jan 25 00:00 ..

In clone container:

ubuntu@db-clones:~/.dblab$ sudo docker exec -it dblab_clone_6002 bash
root@99c8113376eb:/# ls -la /var/lib/dblab/clones/dblab_clone_6002/data/log | head
total 21546
drwx------  2 postgres postgres   3092 Jan 25 04:00 .
drwx------ 21 postgres postgres     36 Jan 25 04:00 ..
-rwx------  1 postgres postgres   2256 Dec 24 11:17 postgresql-2020-12-24_111738.csv
-rwx------  1 postgres postgres    144 Dec 24 11:17 postgresql-2020-12-24_111738.log
-rwx------  1 postgres postgres   2226 Dec 24 11:17 postgresql-2020-12-24_111751.csv
-rwx------  1 postgres postgres    144 Dec 24 11:17 postgresql-2020-12-24_111751.log
-rwx------  1 postgres postgres   2218 Dec 24 11:18 postgresql-2020-12-24_111759.csv
-rwx------  1 postgres postgres    143 Dec 24 11:18 postgresql-2020-12-24_111759.log
-rwx------  1 postgres postgres   2226 Dec 24 11:18 postgresql-2020-12-24_111813.csv

It seems the reason is that the mount points have been removed while snapshots cleaned up

2021/01/03 18:00:00 [ERROR]  failed to clean up snapshots automatically: failed to clean up snapshots: failed to clean up snapshots: RunnerError(cmd="zfs list -t snapshot -H -o name -s dblab:datastateat 
-s creation -r dblab_pool | grep -v clone | head -n -4 | xargs -n1 --no-run-if-empty zfs destroy -R", inerr="exit status 123", stderr="umount: can't unmount /var/lib/dblab/clones/clone_pre_20201224121041
: Resource busy
cannot unmount '/var/lib/dblab/clones/clone_pre_20201224121041': umount failed
cannot destroy snapshot dblab_pool@snapshot_20201224121041_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6000': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20201226180000@snapshot_20201226175931: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20201226180000_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6001': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20201227120000@snapshot_20201227114009: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20201227120000_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6002': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20210102180000@snapshot_20210102175434: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20210102180000_pre: snapshot is cloned
" exit="123")

TODO / How to implement

  • avoid forced deletion
  • remove temporary created _pre snapshots and clones

Acceptance criteria