bayayaqd
Reputation: 59
when i run crawler on EC2, why didn't it work well?
i setted docker on ec2 and upload crawler scrapy code. since yesterday when i run ec2, it could collect data well. but suddenly now it doesn't work well. i guessed it is EC2 problem, because it could work well till today lunch time.
then i try to check EC2 condition, typed df -h, it showed
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 19M 778M 3% /run
/dev/xvda1 7.7G 7.7G 0 100% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 33M 33M 0 100% /snap/snapd/11107
/dev/loop1 33M 33M 0 100% /snap/snapd/11402
/dev/loop2 56M 56M 0 100% /snap/core18/1988
/dev/loop3 132M 132M 0 100% /snap/docker/796
/dev/loop4 34M 34M 0 100% /snap/amazon-ssm-agent/3552
tmpfs 796M 0 796M 0% /run/user/1000
surely yesterday /dev/xvda1 this part('used') wasn't 100%.
so i tried to increase size of /dev/xvda1 volume on EBS.
i hope this trying can solve the issue.
after EBS, when i type lsblk on EC2 terminal it showed this
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 32.3M 1 loop /snap/snapd/11107
loop1 7:1 0 33.3M 1 loop /snap/amazon-ssm-agent/3552
loop2 7:2 0 32.3M 1 loop /snap/snapd/11402
loop3 7:3 0 55.5M 1 loop /snap/core18/1988
loop4 7:4 0 131.6M 1 loop /snap/docker/796
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
anyway i wonder how to delete /dev/xvda1size. because i have plan that my crawler will work periodically openly, but always if i have to increase volume size, it will make me messy.
please help me,
how can my crawler can work without /dev/xvda1 worry.
when i typedsudo find / -type f -size +10M -exec ls -lh {} \;
it showed this
sudo: unable to resolve host ip-172-31-79-98: Resource temporarily unavailable
-rwxr-xr-x 1 root root 19M Mar 8 10:47 /snap/snapd/11402/usr/bin/snap
-rwxr-xr-x 1 root root 13M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Mar 8 10:47 /snap/snapd/11402/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 19M Feb 10 10:44 /snap/snapd/11107/usr/bin/snap
-rwxr-xr-x 1 root root 13M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 12M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snap-preseed
-rwxr-xr-x 1 root root 22M Feb 10 10:44 /snap/snapd/11107/usr/lib/snapd/snapd
-rwxr-xr-x 1 root root 51M Feb 5 15:56 /snap/docker/796/bin/containerd
-rwxr-xr-x 1 root root 26M Feb 5 15:56 /snap/docker/796/bin/ctr
-rwxr-xr-x 1 root root 29M Feb 5 16:02 /snap/docker/796/bin/dnet
-rwxr-xr-x 1 root root 82M Feb 5 15:58 /snap/docker/796/bin/docker
-rwxr-xr-x 1 root root 31M Feb 5 16:02 /snap/docker/796/bin/docker-machine
-rwxr-xr-x 1 root root 98M Feb 5 16:00 /snap/docker/796/bin/dockerd
-rwxr-xr-x 1 root root 13M Feb 5 16:03 /snap/docker/796/bin/runc
-rw-r--r-- 1 root root 26M Mar 16 2020 /snap/docker/796/usr/lib/x86_64-linux-gnu/libicudat a.so.60.2
-rwxr-xr-x 1 root root 24M Dec 21 16:37 /snap/amazon-ssm-agent/3552/amazon-ssm-agent
-rwxr-xr-x 1 root root 39M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-agent-worker
-rwxr-xr-x 1 root root 27M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-cli
-rwxr-xr-x 1 root root 36M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-document-worker
-rwxr-xr-x 1 root root 13M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-logger
-rwxr-xr-x 1 root root 34M Dec 21 16:37 /snap/amazon-ssm-agent/3552/ssm-session-worker
-rw-r--r-- 1 root root 21M Mar 22 07:02 /boot/initrd.img-5.4.0-1039-aws
-rw-r--r-- 1 root root 21M Mar 22 07:01 /boot/initrd.img-5.4.0-1038-aws
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1
-rw------- 1 root root 16M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:03.0/resource1_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0_wc
-rw------- 1 root root 32M Mar 23 07:14 /sys/devices/pci0000:00/0000:00:02.0/resource0
-r-------- 1 root root 128T Mar 23 07:14 /proc/kcore
find: ‘/proc/2374/task/2374/fdinfo/6’: No such file or directory
find: ‘/proc/2374/fdinfo/5’: No such file or directory
-rwxr-xr-x 1 root root 15M Mar 8 14:14 /usr/bin/runc
-rwxr-xr-x 1 root root 16M Mar 2 20:16 /usr/bin/rootlesskit
-rwxr-xr-x 1 root root 19M Feb 2 08:21 /usr/bin/snap
-rwxr-xr-x 1 root root 111M Mar 2 20:16 /usr/bin/dockerd
-rwxr-xr-x 1 root root 30M Mar 8 14:14 /usr/bin/ctr
-rwxr-xr-x 1 root root 55M Mar 8 14:14 /usr/bin/containerd
-rwxr-xr-x 1 root root 11M Nov 23 2018 /usr/bin/lxc
-rwxr-xr-x 1 root root 69M Mar 2 20:16 /usr/bin/docker
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7.a
-rw-r--r-- 1 root root 20M Feb 27 15:10 /usr/lib/python2.7/config-x86_64-linux-gnu/libpytho n2.7-pic.a
-rw-r--r-- 1 root root 26M Mar 16 2020 /usr/lib/x86_64-linux-gnu/libicudata.so.60.2
-rwxr-xr-x 1 root root 18M Nov 23 2018 /usr/lib/lxd/lxd
-rwxr-xr-x 1 root root 13M Feb 2 08:21 /usr/lib/snapd/snap-bootstrap
-rwxr-xr-x 1 root root 22M Feb 2 08:21 /usr/lib/snapd/snapd
-rw-r--r-- 1 root root 36M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m-pic.a
-rw-r--r-- 1 root root 37M Jan 26 15:33 /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/lib python3.6m.a
-rwxr-xr-x 1 root root 25M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus
-rwxr-xr-x 1 root root 22M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/lto1
-rwxr-xr-x 1 root root 23M Dec 4 2019 /usr/lib/gcc/x86_64-linux-gnu/7/cc1
-rwxr-xr-x 1 root root 62M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-buildx
-rwxr-xr-x 1 root root 36M Mar 2 20:16 /usr/libexec/docker/cli-plugins/docker-app
-r--r--r-- 1 root root 21M Mar 22 07:05 /home/ubuntu/crawlab-1/.git/objects/pack/pack-6c0a8218eeb814173c677e3df54850423986d710.pack
-rw-r--r-- 1 root root 26M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_i18n_Translati on-en
-rw-r--r-- 1 root root 11M Mar 22 15:48 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd 64_Packages
-rw-r--r-- 1 root root 49M Apr 26 2018 /var/lib/apt/lists/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_P ackages
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/cache/aa3d5fe426c66baa3261f1641b9b5fe284cd08a16761eb765438cb5396b1c0ba96abe1e29 6135543871dbe68423be6b1
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/seed/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/seed/snaps/amazon-ssm-agent_3552.snap
-rw------- 1 root root 32M Feb 24 17:36 /var/lib/snapd/seed/snaps/snapd_11036.snap
-rw------- 1 root root 33M Mar 23 04:18 /var/lib/snapd/snaps/snapd_11402.snap
-rw------- 2 root root 56M Feb 24 17:36 /var/lib/snapd/snaps/core18_1988.snap
-rw------- 2 root root 34M Feb 24 17:36 /var/lib/snapd/snaps/amazon-ssm-agent_3552.snap
-rw------- 2 root root 132M Mar 22 07:14 /var/lib/snapd/snaps/docker_796.snap
-rw------- 1 root root 33M Mar 22 09:39 /var/lib/snapd/snaps/snapd_11107.snap
-rw-r----- 1 root root 14M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/containers/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a11237 8c1b6123a12d2c16c27/5f951fb4ee5e008ad09cf3e5a86bbd5c25061a0a112378c1b6123a12d2c16c27-json.log
-rw------- 1 999 docker 14M Mar 23 06:41 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a6 f609dda8aae79d462/_data/collection-10-5521340255836326071.wt
-rw------- 1 999 docker 100M Mar 23 06:45 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerLog.0000000034
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000001
-rw------- 1 999 docker 100M Mar 23 06:35 /var/snap/docker/common/var-lib-docker/volumes/b5276a9f65fc8a8995ce9a76c50bcd27ddd90043201d4a 6f609dda8aae79d462/_data/journal/WiredTigerPreplog.0000000002
-rwxr-xr-x 1 root root 12M Mar 13 04:01 /var/snap/docker/common/var-lib-docker/overlay2/b1991d45bb2288ac16dcab9c1860d3eb821543b2e66a22f e261d9a159dd2019e/diff/usr/local/bin/redis-server
-rw-r--r-- 1 root root 27M Mar 17 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/x86_64-linux-gnu/libicudata.so.66.1
-rwxr-xr-x 1 root root 26M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1plus
-rwxr-xr-x 1 root root 24M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/lto1
-rwxr-xr-x 1 root root 25M Aug 8 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/usr/lib/gcc/x86_64-linux-gnu/9/cc1
-rw-r--r-- 1 root root 18M Apr 23 2020 /var/snap/docker/common/var-lib-docker/overlay2/7a9d391c4792c7234e369212d9e4300b3279bb7a805a187 36a83f30428033492/diff/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_universe_binary-amd64_Packages.lz4
-rw------- 1 root root 15M Mar 22 08:07 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.HNlE5E/Default/Cache/b5f47fbc4d23c5f6_0
-rw------- 1 root root 15M Mar 22 08:05 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/.com.google.Chrome.diyKvW/Default/Cache/b5f47fbc4d23c5f6_0
-rw-r--r-- 1 root root 73M Mar 23 06:42 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:27 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rw------- 1 root root 19M Mar 22 07:26 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef7411 d7a1a1c98f3e88add/diff/root/.cache/pip/http/e/f/1/7/2/ef17289f5b2b98078e36fd05289a14ec9579924ee0366839cdf7611a
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/60c66677cf21304fe0dd7cd0143564d909529e65fef741 1d7a1a1c98f3e88add/diff/opt/google/chrome/chrome
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoexport
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongorestore
-rwxr-xr-x 1 root root 51M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongo
-rwxr-xr-x 1 root root 62M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongos
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongoimport
-rwxr-xr-x 1 ubuntu ubuntu 16M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/bsondump
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongotop
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:14 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongofiles
-rwxr-xr-x 1 ubuntu ubuntu 19M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongostat
-rwxr-xr-x 1 ubuntu ubuntu 20M Jan 22 17:13 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e515 9971c54372cccfc5004f9/diff/usr/bin/mongodump
-rwxr-xr-x 1 root root 79M Dec 19 2013 /var/snap/docker/common/var-lib-docker/overlay2/c0c98c129cd60c3d3e895eda9c5417178012f18e5159971 c54372cccfc5004f9/diff/usr/bin/mongod
-rwxr-xr-x 1 root root 48M Dec 11 02:49 /var/snap/docker/common/var-lib-docker/overlay2/d06370e7f6ed90afffd39f640723116f2818ff7b96619d4 499e497870be4df5f/diff/opt/bin/crawlab
-rwxr-xr-x 1 root root 48M Dec 11 02:55 /var/snap/docker/common/var-lib-docker/overlay2/0d641b4b0ae1855a87a93c6f8d275be503912cf266ac69b 03be851e6a42b881b/diff/usr/local/bin/crawlab-server
-rw-r--r-- 1 root root 73M Mar 22 07:22 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/tmp/chrome.deb
-rwxr-xr-x 1 root root 12M Jan 27 03:32 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/usr/local/bin/chromedriver
-rwxr-xr-x 1 root staff 11M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_linux64
-rwxr-xr-x 1 root staff 16M Mar 22 07:19 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/usr/local/lib/python3.8/dist-packages/chromedriver_py/chromedriver_mac64
-rw------- 1 root root 19M Mar 22 07:18 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9dc acd8427c4a9e475a2/diff/root/.cache/pip/http/6/0/4/2/9/60429761adcdcb005ac1b1419b8c7b286bf36316aa97d5f62b7ccb34
-rwxr-xr-x 1 root root 165M Mar 11 23:57 /var/snap/docker/common/var-lib-docker/overlay2/a033a3ea16c7b174830b9872fb6dea5a504363b42b9b9d cacd8427c4a9e475a2/diff/opt/google/chrome/chrome
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/srcpkgcache.bin
-rw-r--r-- 1 root root 39M Mar 23 04:18 /var/cache/apt/pkgcache.bin
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.5~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 27M Mar 9 00:39 /var/cache/apt/archives/containerd.io_1.4.4-1_amd64.deb
-rw-r--r-- 1 root root 40M Mar 8 23:12 /var/cache/apt/archives/docker-ce-cli_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r--r-- 1 root root 11M Mar 1 06:20 /var/cache/apt/archives/linux-aws-5.4-headers-5.4.0-1039_5.4.0-1039.41~18.04.1_all.deb
-rw-r--r-- 1 root root 14M Mar 1 06:20 /var/cache/apt/archives/linux-modules-5.4.0-1039-aws_5.4.0-1039.41~18.04.1_amd64.deb
-rw-r--r-- 1 root root 24M Mar 8 23:12 /var/cache/apt/archives/docker-ce_5%3a20.10.4~3-0~ubuntu-bionic_amd64.deb
-rw-r-----+ 1 root systemd-journal 16M Mar 22 11:18 /var/log/journal/ea09dcfc1f4741e1b257d2463c341f29/system@9cf46084d404438fb64d33dbd9 5ecc2a-0000000000000001-0005be1a8eeb62f4.journal
i made new instance then i setted docker and scrapy blah blah blah...
anyway, after i crawl 1 time. when i typed df -Th it showed this.
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 5.7G 2.1G 74% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 5.7G 2.1G 74% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
and after i crawl 9 times it showed this
Filesystem Type Size Used Avail Use% Mounted on
udev devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs tmpfs 796M 1.1M 795M 1% /run
/dev/xvda1 ext4 7.7G 7.0G 732M 91% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/loop0 squashfs 34M 34M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop1 squashfs 32M 32M 0 100% /snap/snapd/11036
/dev/loop2 squashfs 56M 56M 0 100% /snap/core18/1988
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/e7965025d305ee6d51b55dee17fe547b70b51ebea9750d22e1ff55337d54b3ea/merged
tmpfs tmpfs 796M 0 796M 0% /run/user/1000
/dev/loop3 squashfs 33M 33M 0 100% /snap/snapd/11107
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e/merged
overlay overlay 7.7G 7.0G 732M 91% /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359/merged
after i crawl 1 time, i wrote du -h --max-depth=3 /var/lib | sort -h on terminal, it showed this.
....
304M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
306M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.1G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.5G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.3G /var/lib/docker/overlay2
6.6G /var/lib/docker
6.9G /var/lib
after 9 times.
306M /var/lib/docker/volumes/b8a9c7ac497041f99b629fdb14e9dfc283f4108d73512d3f87a79e9a5af3d0fd
308M /var/lib/docker/volumes
350M /var/lib/docker/overlay2/daf820fd04371808cf9213038ab552fdec5fc659d1fbf3b1643529e8ba1049e9
430M /var/lib/docker/overlay2/3569c54068321f2973fcde4f80695f482840915df635105f6bdb68226a113741
437M /var/lib/docker/overlay2/414d298f3057de6461ca2d5b9ceeb290e35215500cbfdaa3d000ad7afda46359
2.3G /var/lib/docker/overlay2/b03949ee74691b6ef3f1471c37ebf58d5cd6947b8e139430b4f1c0776c06016c
2.7G /var/lib/docker/overlay2/02159a834b6c0c51d81686f024aab575da916d001d91a7027d2c3868f7b2696e
6.7G /var/lib/docker/overlay2
7.0G /var/lib/docker
7.4G /var/lib
Upvotes: 0
Views: 240
Answers (1)
Tasos P.
Reputation: 4114
It is clear that your EC2 instance has disk usage issue (disk gets full).
You successfully increased the volume size (as shown in lsblk output) but you also need to extend the file system, as described here.
Essentially, you should run sudo growpart /dev/xvda 1 and sudo resize2fs /dev/xvda1. Use df -h again to verify.
In order to optimize performance, you should consider:
- Using a dedicated volume for your workload (i.e. mount a new, larger EBS volume on
/mnt/dataand configure your application to use this path for data storage) - Resize this volume and/or change type (i.e. use provisioned IOPS SSD)
Scrapy also supports S3 for storage.
You can configure scrapy logging to use a specific log file (--logfile FILE which you can later delete) or disabled it altogether (--nolog, not recommended).
Upvotes: 1
Related Questions
- Running Scrapy in a docker container
- Scrapy-splash no results at "Server listening on http://0.0.0.0:8050"
- How to install Scrapy Docker Container on Amazon Linux 2 AMI EC2
- scrapy getting stuck after some time
- Running Scrapy in Docker Container
- Scrapy spiders drastically slows down while running on AWS EC2
- Scrapy Splash server not starting via docker run
- Getting TCP connection timed out: 110: Connection timed out. on AWS while using scrapy?
- unable to run scrapy in ec2
- Scrapy deploy no longer working