Cannot install grab-site on WSL2 (Ubuntu)

[jacob-willden]

After following the steps in the README to install grab-site on Windows Subsystem for Linux (Ubuntu) on Windows 10, I get stuck on the last command in step 2 for Linux installation:

$ ~/gs-venv/bin/pip install --no-binary lxml --upgrade git+https://github.com/ArchiveTeam/grab-site
Collecting git+https://github.com/ArchiveTeam/grab-site
  Cloning https://github.com/ArchiveTeam/grab-site to /tmp/pip-req-build-g0_z8bxl
  Running command git clone --filter=blob:none --quiet https://github.com/ArchiveTeam/grab-site /tmp/pip-req-build-g0_z8bxl
  Resolved https://github.com/ArchiveTeam/grab-site to commit f4027f57fcb6c7f19421e348fa8c8d7acf725267
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip (from grab-site==2.2.7)
  Using cached https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting click>=6.3 (from grab-site==2.2.7)
  Using cached click-8.1.8-py3-none-any.whl.metadata (2.3 kB)
Collecting manhole>=1.0.0 (from grab-site==2.2.7)
  Using cached manhole-1.8.1-py3-none-any.whl.metadata (13 kB)
Collecting lmdb>=0.89 (from grab-site==2.2.7)
  Downloading lmdb-1.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.1 kB)
Collecting autobahn>=0.12.1 (from grab-site==2.2.7)
  Downloading autobahn-23.1.2.tar.gz (480 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting google-re2>=1.0.6 (from grab-site==2.2.7)
  Downloading google_re2-1.1.20240702-1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (2.2 kB)
Collecting websockets>=6.0 (from grab-site==2.2.7)
  Downloading websockets-13.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting cchardet>=1.0.0 (from grab-site==2.2.7)
  Downloading cchardet-2.1.7-cp38-cp38-manylinux2010_x86_64.whl.metadata (7.7 kB)
Collecting txaio>=21.2.1 (from autobahn>=0.12.1->grab-site==2.2.7)
  Using cached txaio-23.1.1-py2.py3-none-any.whl.metadata (5.4 kB)
Collecting cryptography>=3.4.6 (from autobahn>=0.12.1->grab-site==2.2.7)
  Downloading cryptography-44.0.2-cp37-abi3-manylinux_2_34_x86_64.whl.metadata (5.7 kB)
Collecting hyperlink>=21.0.0 (from autobahn>=0.12.1->grab-site==2.2.7)
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl.metadata (1.5 kB)
Requirement already satisfied: setuptools in ./gs-venv/lib/python3.8/site-packages (from autobahn>=0.12.1->grab-site==2.2.7) (56.0.0)
Collecting chardet (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached chardet-5.2.0-py3-none-any.whl.metadata (3.4 kB)
Collecting dnspython (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Downloading dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB)
Collecting html5-parser (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached html5-parser-0.4.12.tar.gz (270 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting lxml (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached lxml-5.4.0.tar.gz (3.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting namedlist (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached namedlist-1.8-py2.py3-none-any.whl.metadata (14 kB)
Collecting sqlalchemy==1.3.24 (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Downloading SQLAlchemy-1.3.24-cp38-cp38-manylinux2010_x86_64.whl.metadata (7.9 kB)
Collecting tornado==4.5.3 (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached tornado-4.5.3.tar.gz (484 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting yapsy (from wpull@ https://github.com/ArchiveTeam/ludios_wpull/archive/refs/tags/3.0.9.zip->grab-site==2.2.7)
  Using cached Yapsy-1.12.2.tar.gz (83 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.12 (from cryptography>=3.4.6->autobahn>=0.12.1->grab-site==2.2.7)
  Downloading cffi-1.17.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting idna>=2.5 (from hyperlink>=21.0.0->autobahn>=0.12.1->grab-site==2.2.7)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting pycparser (from cffi>=1.12->cryptography>=3.4.6->autobahn>=0.12.1->grab-site==2.2.7)
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading cchardet-2.1.7-cp38-cp38-manylinux2010_x86_64.whl (265 kB)
Using cached click-8.1.8-py3-none-any.whl (98 kB)
Downloading google_re2-1.1.20240702-1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (546 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 546.5/546.5 kB 9.1 MB/s eta 0:00:00
Downloading lmdb-1.6.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (294 kB)
Using cached manhole-1.8.1-py3-none-any.whl (16 kB)
Downloading websockets-13.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (164 kB)
Downloading SQLAlchemy-1.3.24-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 10.6 MB/s eta 0:00:00
Downloading cryptography-44.0.2-cp37-abi3-manylinux_2_34_x86_64.whl (4.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 11.2 MB/s eta 0:00:00
Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Using cached txaio-23.1.1-py2.py3-none-any.whl (30 kB)
Using cached chardet-5.2.0-py3-none-any.whl (199 kB)
Downloading dnspython-2.6.1-py3-none-any.whl (307 kB)
Using cached namedlist-1.8-py2.py3-none-any.whl (15 kB)
Downloading cffi-1.17.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: grab-site, autobahn, wpull, tornado, html5-parser, lxml, yapsy
  Building wheel for grab-site (pyproject.toml) ... done
  Created wheel for grab-site: filename=grab_site-2.2.7-py3-none-any.whl size=47959 sha256=5b192fbbba215628c6e224306cf8fc808fe56d4a05dd4d00349ad4260cb7579e
  Stored in directory: /tmp/pip-ephem-wheel-cache-vb8u8y2k/wheels/e5/6d/cd/9797e1e7f685e22a0ead92e0c9bc1851617ad441fcf2ad36b8
  Building wheel for autobahn (pyproject.toml) ... done
  Created wheel for autobahn: filename=autobahn-23.1.2-py2.py3-none-any.whl size=666848 sha256=fc68f88ad64b73035f2808ef7ddcd31d5cdbbf16b8d4c5a7cf46f344f9596759
  Stored in directory: /home/somebody/.cache/pip/wheels/f1/a4/a2/92e99f91e89b278683501744f82533a888326e65adffee831a
  Building wheel for wpull (pyproject.toml) ... done
  Created wheel for wpull: filename=wpull-3.0.9-py3-none-any.whl size=681787 sha256=46d24cb87ccde840105fca3956265f86f075793029a48dd20f7f2886a42cbe33
  Stored in directory: /tmp/pip-ephem-wheel-cache-vb8u8y2k/wheels/f7/bc/04/560151d39f165759bf4cad45339f4a45a74b1282a30a8a338f
  Building wheel for tornado (pyproject.toml) ... done
  Created wheel for tornado: filename=tornado-4.5.3-cp38-cp38-linux_x86_64.whl size=421258 sha256=a9f341da111d53d6cd44aab4aeae07e97ff251a4f7013168f06c9ae86934fd72
  Stored in directory: /home/somebody/.cache/pip/wheels/20/fe/26/0c03adccd0cec2744261ec6b8380fbe5e726d1ea4cf3a9370e
  Building wheel for html5-parser (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for html5-parser (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/dom.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/encoding_names.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/encoding_parser.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/soup.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/stdlib_etree.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      copying src/html5_parser/__init__.py -> build/lib.linux-x86_64-cpython-38/html5_parser
      running build_ext
      building 'html5_parser.html_parser' extension
      creating build/temp.linux-x86_64-cpython-38/gumbo
      creating build/temp.linux-x86_64-cpython-38/src
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/attribute.c -o build/temp.linux-x86_64-cpython-38/gumbo/attribute.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/char_ref.c -o build/temp.linux-x86_64-cpython-38/gumbo/char_ref.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/error.c -o build/temp.linux-x86_64-cpython-38/gumbo/error.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/gumbo_edit.c -o build/temp.linux-x86_64-cpython-38/gumbo/gumbo_edit.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/parser.c -o build/temp.linux-x86_64-cpython-38/gumbo/parser.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/string_buffer.c -o build/temp.linux-x86_64-cpython-38/gumbo/string_buffer.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/string_piece.c -o build/temp.linux-x86_64-cpython-38/gumbo/string_piece.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/svg_attrs.c -o build/temp.linux-x86_64-cpython-38/gumbo/svg_attrs.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/svg_tags.c -o build/temp.linux-x86_64-cpython-38/gumbo/svg_tags.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/tag.c -o build/temp.linux-x86_64-cpython-38/gumbo/tag.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/tokenizer.c -o build/temp.linux-x86_64-cpython-38/gumbo/tokenizer.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/utf8.c -o build/temp.linux-x86_64-cpython-38/gumbo/utf8.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/util.c -o build/temp.linux-x86_64-cpython-38/gumbo/util.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c gumbo/vector.c -o build/temp.linux-x86_64-cpython-38/gumbo/vector.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c src/as-libxml.c -o build/temp.linux-x86_64-cpython-38/src/as-libxml.o -O3 -std=c99 -fvisibility=hidden
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DMAJOR=0 -DMINOR=4 -DPATCH=12 -I/usr/include/libxml2 -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c src/as-python-tree.c -o build/temp.linux-x86_64-cpython-38/src/as-python-tree.o -O3 -std=c99 -fvisibility=hidden
      In file included from src/as-python-tree.c:7:
      src/as-python-tree.h:10:10: fatal error: Python.h: No such file or directory
         10 | #include <Python.h>
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for html5-parser
  Building wheel for lxml (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for lxml (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [110 lines of output]
      Building lxml version 5.4.0.
      Building without Cython.
      Building against libxml2 2.9.14 and libxslt 1.1.39
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/builder.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/cssselect.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/sax.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/__init__.py -> build/lib.linux-x86_64-cpython-38/lxml
      creating build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-cpython-38/lxml/includes
      creating build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/builder.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/clean.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/defs.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/diff.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-cpython-38/lxml/html
      creating build/lib.linux-x86_64-cpython-38/lxml/isoschematron
      copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron
      copying src/lxml/etree.h -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/etree_api.h -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/etree.pyx -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/objectify.pyx -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/apihelpers.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/classlookup.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/cleanup.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/debug.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/docloader.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/dtd.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/extensions.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/iterparse.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/nsclasses.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/objectpath.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/parser.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/parsertarget.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/proxy.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/public-api.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/readonlytree.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/relaxng.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/saxparser.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/schematron.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/serializer.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xinclude.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xmlerror.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xmlid.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xmlschema.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xpath.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xslt.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/xsltext.pxi -> build/lib.linux-x86_64-cpython-38/lxml
      copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/__init__.pxd -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-cpython-38/lxml/includes
      copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-cpython-38/lxml/includes
      creating build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/rng
      copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/rng
      creating build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl
      copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl
      creating build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-cpython-38/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
      running build_ext
      building 'lxml.etree' extension
      creating build/temp.linux-x86_64-cpython-38/src/lxml
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/home/somebody/gs-venv/include -I/usr/include/python3.8 -c src/lxml/etree.c -o build/temp.linux-x86_64-cpython-38/src/lxml/etree.o -w
      src/lxml/etree.c:105:10: fatal error: Python.h: No such file or directory
        105 | #include "Python.h"
            |          ^~~~~~~~~~
      compilation terminated.
      Compile failed: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      creating tmp
      cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitm7irsdwa.c -o tmp/xmlXPathInitm7irsdwa.o
      cc tmp/xmlXPathInitm7irsdwa.o -lxml2 -o a.out
      <string>:67: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lxml
  Building wheel for yapsy (pyproject.toml) ... done
  Created wheel for yapsy: filename=yapsy-1.12.2-py3-none-any.whl size=39009 sha256=978efd2c4bf71a4a7fe44f1f19c4c954c22d2e540478cc2d5bdc763056f78de2
  Stored in directory: /home/somebody/.cache/pip/wheels/da/b4/b7/500e17e7611250b358a234e4fa4010d08228ef0e46015f61ea
Successfully built grab-site autobahn wpull tornado yapsy
Failed to build html5-parser lxml
ERROR: Failed to build installable wheels for some pyproject.toml based projects (html5-parser, lxml)

Are there any ways I can troubleshoot this? Thanks in advance!

[Execrit]

Have you ever found a solution?

[HeliosLHC]

Try installing the Python dev package: apt-get install python3-dev

[Execrit]

I have, it still does not work. also I have had trouble with other methods due to the site's cloudflare. despite copying cookies it does not work..,i kinda gave up

[jacob-willden]

Installing python3-dev actually allowed me to install grab-site! Though now actually running grab-site with any URL results in the following trace-back, so I'll need to do some more troubleshooting:

Traceback (most recent call last):
  File "/home/debian/gs-venv/bin/grab-site", line 4, in <module>
    main.main()
  File "/home/debian/gs-venv/lib/python3.8/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/debian/gs-venv/lib/python3.8/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/debian/gs-venv/lib/python3.8/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/debian/gs-venv/lib/python3.8/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/debian/gs-venv/lib/python3.8/site-packages/libgrabsite/main.py", line 213, in main
    no_proto_no_trailing = claim_start_url.split('://', 1)[1].rstrip('/')[:100]
IndexError: list index out of range

Thanks for the guidance, @HeliosLHC. @SomeoneFighter, with Cloudflare getting in the way, you might be able to get a crawl working with software like Selenium, if you're still interested in crawling those sites. Also, out of curiosity, what kind of error message(s) did you get after installing python3-dev and trying to install grab-site?