Rendering $\LaTeX$ is generally handled in a special way when compared with most other renderers in JupyterLab. For this reason, it is not presently covered by a pre-built extension, but rather by adding MathJax 2 directly to the page. As it changes very slowly, and is relatively benign if missing for most use cases, this use of a CDN is the default for JupyterLite.

Configuring fullMathjaxUrl and mathjaxConfig in jupyter-lite.json allows you to specify a relative or remote location, replacing (or avoiding) the CDN. If jupyter-server-mathjax is installed, the default configuration TeX-AMS-MML_HTMLorMML-full,Safe will be copied into the output folder.