You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci(util): use requests for the scripts' HTTP fetches and psutil for vendor_test_gate's memory polling #1894
On #1885 the owner asked for the repo's scripts to use requests and bs4[html5lib], and pexpect or psutil where they read a subprocess's output. #1885 moves util/conda_vendor.py --refresh-pins to requests; this covers the other scripts.
requests, replacing urllib.request:
util/ci_timing.py: urlopen at :437-446, Link-header pagination at :412. ci-timing.yml runs it on a bare interpreter and installs nothing (:57), so that job needs setup-python and an install step.
util/update_python_versions.py: fetch() at :179-191. The detect job in cron-python-versions.yml installs only PyYAML (:61-62).
examples/generators/pcapng.py: _download and its Retry-After backoff at :832-917. Every leg that runs it already installs plain requests through the test extra, but unit-tests.yml:473-476 says it needs only the standard library, so update that comment. Keep the offline fallback and the retry behaviour.
markdown-it-py, replacing the pipe-splitting in util/update_python_versions.pyparse_images() (:245-268), which reads the runner-images README's image table. An escaped pipe or a pipe inside a code span would shift its columns; read the table through a GFM-table parser instead (added after the owner's follow-up on #1885). It runs in the same detect job as fetch().
lxml, replacing xml.etree.ElementTree in util/ci_timing.pyparse_junit() and util/pyshark_encap_map.py's PDML read, for its strict parsing (the owner's preference on #1885). Use a parser that resolves no entities and has no network access. lxml has wheels for CPython 3.8–3.15 (free-threaded included) and PyPy 3.9–3.11, none for GraalPy: add it to the test extra with a marker excluding GraalPy, gate its tests there, and install it in ci-timing.yml.
psutil, replacing the ps -A -o pgid= -o rss= polling in util/vendor_test_gate.py:157-171. psutil is in no extra, and tests/project runs this script's tests on every leg, including KTLO and experimental. So either confirm that psutil installs on every one of those legs, or keep ps as the fallback.
The tests that stub urllib move with their scripts: tests/project/test_ci_timing.py and tests/project/test_pcapng_download_retry_1658_unit.py.
Out of scope:
util/project_status.py stays on gh. Its job runs on pull_request_target, holds PROJECT_TOKEN, and installs nothing by design.
bs4 and pexpect have nothing to replace: no script outside pcapkit/vendor parses HTML, and none drives an interactive program.
They are in the test extra already. #1896 adds all three there:
psutil, on CPython only;
markdown-it-py;
lxml, everywhere except GraalPy.
The Pipfile's [dev-packages] and the dev extra don't list any of them yet, so #1896 will add all three to both, with the same markers. requests is already in both.
Correction to my reply above: #1896 adds them to the Pipfile's [dev-packages] and the test extra, but not to the dev extra.
process.rst defines dev as what pylint, mypy and autodoc must resolve for pcapkit/. That is why requests and bs4 are there: pcapkit/vendor imports them. Only the scripts under util/ and examples/ import these three, and the toolchain does not analyse those.
Activity
ci(util): fetch with requests and poll memory with psutil in the scri…
JarryShaw commentedon Oct 17, 2026
I suppose these should also go into the test extras and/or Pipfile dev deps?
JarryShaw commentedon Oct 17, 2026
They are in the
testextra already. #1896 adds all three there:psutil, on CPython only;markdown-it-py;lxml, everywhere except GraalPy.The Pipfile's
[dev-packages]and thedevextra don't list any of them yet, so #1896 will add all three to both, with the same markers.requestsis already in both.ci(util): fetch with requests and poll memory with psutil in the scri…
JarryShaw commentedon Oct 17, 2026
Correction to my reply above: #1896 adds them to the Pipfile's
[dev-packages]and thetestextra, but not to thedevextra.process.rstdefinesdevas what pylint, mypy and autodoc must resolve forpcapkit/. That is whyrequestsandbs4are there:pcapkit/vendorimports them. Only the scripts underutil/andexamples/import these three, and the toolchain does not analyse those.ci(util): fetch with requests and poll memory with psutil in the scri…
ci(util): fetch with requests and poll memory with psutil in the scri…