JORTS(7) Miscellaneous Information Manual JORTS(7) NAME Introducing Jorts – june's ports DESCRIPTION Alright so I've gone off the deep end, maybe. After continual frustration with MacPorts culminating in not being able to install nvi(1) on my work MacBook, I have just gone ahead and started my own personal ports tree for macOS. After a couple of weeks, I have 32 ports in my tree and only two remaining requested ports installed from MacPorts. I set out with a couple ideas in mind: • This will be my own personal ports tree. It only has to work for me. Since I'm using it on both my personal Intel MacBook Pro still running Catalina and my work M1 MacBook Pro running Monterey, it is at least that portable. • It's ok to rely on system libraries and tools provided by macOS. I'm not creating a distro, so it doesn't need to be totally isolated. This lets me skip really annoying things like compiler toolchains. • Sources get vendored, either from release tarballs or with git-subtree(1). This allows totally pain-free local patching, and boy has this paid off. I can just do what I need to do to get the thing to build how I want and commit it in git like anything else. It also means that the tree itself is entirely self-contained and doesn't rely on any external sources or network access. Honestly with some old and obscure software it feels like upstream could disappear at any moment, so this gives me peace of mind too. Another advantage of vendoring upstream sources is that all of the code installed on my system (in /usr/local anyway) is easily inspected, much like /usr/src on a BSD. This can be super useful for debugging or just for reference. • Produce simple package tarballs. They're just the contents of DESTDIR after a staged install. They get installed for real by untarring them in /. They can then be uninstalled (or upgraded) by removing the paths contained in the tarball from the system. • Track installed packages with symbolic links to specific package tarballs. Keep old tarballs around for rollbacks. This means I can see what's installed with plain old ls(1)! $ ls */Installed ... libretls/Installed toilet/Installed mandoc/Installed tree/Installed $ ls -l toilet/Installed lrwxr-xr-x 1 root staff 19 17 Jan 21:45 toilet/Installed -> toilet-0.3~1.tar.gz • Use bmake(1). It's scrutable. It also knows how to bootstrap itself pretty well. Since bmake(1) is itself a port in my tree that would require bmake(1) to build and install, I wrote a small Bootstrap shell script to install bmake(1) “manually” then use that bmake(1) to build and install its own port. It also requires a bit of care when upgrading the bmake(1) port since macOS rather doesn't like a binary deleting itself while it's running. • No GNU software. I simply refuse to do it. To that end, prefer configuring/building with cmake(1) where at all possible. I fell into this early on since I originally just wanted to install nvi(1) and lichray/nvi2 is a better upstream source these days that uses cmake(1). With a port and support for cmake(1) in Port.mk, I can make changes to CMakeLists.txt files without issue. I can also vendor upstreams directly from git rather than having to find release tarballs with generated configure scripts and so on. When I need to make changes to the build systems of projects using autotools, I either have to have autotools installed (from outside my tree) or painstakingly reflect my edits by hand in the generated files, both of which suck hard. Ok so that's actually quite a number of ideas. But they have come together into something surprisingly usable surprisingly quickly! Like I said, this is only intended to be my own personal ports tree, but I hope that some of these ideas are interesting and maybe inspire others to explore similar approaches. But wait, I'm not done yet! There are some other interesting things that I came up with along the way, and also some complaints about some upstreams, but I'll try to keep those to a minimum. So it turns out that dependencies are hard. Who knew? It's easy enough to enforce direct dependencies at build time by just checking for the required Installed symlinks. It's less straightforward to do this recursively, which you need if you want to be able to say, “Install nvi for me!” and get ncurses(3), cmake(1) and pkgconf(1) installed first if they aren't already. Rather than trying to do all that in bmake(1), I wrote a shell script called Plan, which itself produces a shell script. Given a list of ports to install or upgrade, it recursively gathers their dependencies and feeds them to tsort(1), which is a neat utility which topologically sorts a graph. In other words, it determines the order in which the graph of dependencies should be installed. The Plan script produces a list of bmake(1) commands to make that happen on standard output, which can be piped to sh(1). So, the way to say the above is: $ ./Plan -j4 nvi | sh -e Now, what's missing from this approach is the ability to automatically uninstall no-longer-needed dependencies. It's something I've criticized Homebrew for lacking and one of the reasons I started using MacPorts, so it's somewhat ironic that my own system lacks it as well. However, I don't think it's much of a problem, since I'm only packaging what I actually want installed in the first place. On my personal computer, I have all 32 of my ports installed, and I expect that to continue. I can always keep using MacPorts to install things I only intend to use temporarily. Another thing I was slightly concerned about from the beginning was disk usage. I think the benefits of vendoring sources far outweigh the cost in storage, but it would be nice to at least minimize that cost. Previously, I wrote about git-sparse-checkout(1), which allows you to only have certain paths checked out in your git working tree. Since port sources aren't always interesting and only required while actually building the port, it makes sense to not have them always checked out. Rather than manipulate git-sparse-checkout(1) myself, I added support for it directly into Port.mk. If sparse checkout is enabled, building a port will automatically add its source tree to the checkout list, and cleaning that port will remove it from the list. At rest, only the port system itself and the package tarballs need to be present on the file system. It turns out that upstream build system behaviour is super inconsistent, even among projects using the same tools. I started collecting a list of checks to perform on the output of my port builds to make sure they didn't do anything weird. They live in Check.sh, which gets run when a package tarball is created. The current list of checks is: • Check for directories not included by PACKAGE_DIRS. In other words, make sure the port isn't trying to install anything outside of /usr/local. Sometimes this makes sense, though, which is what PACKAGE_DIRS is for. • Check for references to PWD, i.e. the build directory. This can mean the build didn't understand PREFIX and DESTDIR correctly, or that it built with debug info. • Check for binaries without manuals. If your software installs an executable in bin but not a manual page, your software is incomplete! Sometimes this just means I missed an extra documentation install target. • Check for dynamic linking to outside objects. In other words, if something ended up linking to a library installed by MacPorts rather than the one from jorts or macOS. • Check for dynamic linking to system libraries jorts provides instead. Similar to the last one, if both macOS and jorts provide a library, check that ports link with the latter. • Check for scripts with outside interpreters. This is analogous to the linking checks but for scripts, checking that their shebang lines refer to interpreters installed by macOS or jorts. A number of my ports still fail some of these checks, but I have fixed a lot of problems the script called out. Speaking of problem ports... git's build system is truly awful. I'm sorry, it's just really disappointing. On the upside though, I did manage to patch it to use asciidoctor(1) directly to generate manual pages from asciidoc source, rather than generating docbook or whatever then converting that. One less build dependency! I also fixed up curl's CMakeLists.txt (which I guess are normally only used on Windows) to build and install documentation properly. And I got libcaca's Cocoa driver working again! Very important to be able to run cacafire(1) in a Cocoa window. Shout out to SDL2, which didn't require any patching or extra options beyond USE_CMAKE=yes. Model upstream. Some other odds and ends: I like being able to name ports how I want (for example, ag) and use my own port version convention, using ‘+’ to append VCS revisions and ‘~’ to append port revisions. I don't think those are likely to ever clash with upstream versioning schemes. Not that I even need to follow upstream versioning. There is no reason the version number of dash(1) should start with a zero. Speaking of versions, a big downside of maintaining your own ports tree is that you actually need to update it. Thankfully, once I packaged curl(1) and jq(1) (which needs a new release dammit, it's been 4 years and the build is broken on macOS), I could use the Repology API to check if I'm behind everyone else. Far more reliable than trying to automate checking upstreams for new versions. That lives in the Outdated shell script. Phew! I wrote a lot about this. It feels a little self-indulgent, but I've had fun working on this and want to share. If anyone else tries anything similar, or is weird enough to give jorts a try themselves, I'd love to hear about it! SEE ALSO https://git.causal.agency/jorts/ https://youtu.be/Sx3ORAO1Y6s AUTHORS june Listening to Arcade Fire — Arcade Fire (EP), Arcade Fire — The Suburbs. Typed on a brand new Leopold FC660M with Cherry MX Red switches. Lovely keyboard. Causal Agency February 2, 2022 Causal Agency