Despite a successful compile of the hotplug enabled kernel, the first boot led
to a panic. Cherry (cherry@) was quick to point out that kmem(9) is not
available until uvm_page_init() has done with all the initialization. This led
to a kind of chicken and egg scenario where in the rbtree(3) nodes which now
hold the memory segment information can only be created via the kmem_zalloc()
call, and this call is not yet available in the pre-boot situation.

In our previous discussions related to this specific “chicken-egg problem”,
Cherry was in favor of the idea of maintaining a minimal “static array” whose
size is VM_PHYSSEG_MAX and once the init process is over, switch over to the
kmem(9) allocator.

In order to achieve this couple of wrapper functions that exist only within the
realm of uvm_physseg.c called uvm_physseg_alloc() and uvm_physseg_free()
were introduced.

This is the original declaration of the uvm_physseg[] static array that is now
being used during the pre-boot process.

static size_t nseg = 0;
static struct uvm_physseg uvm_physseg[VM_PHYSSEG_MAX];

So now allocations in uvm_physseg.c now utilize uvm_physseg_alloc()

static void *
uvm_physseg_alloc(size_t sz)
{
        /*
         * During boot time, we only support allocating vm_physseg
         * entries from the static array.
         * We need to assert for this.
         */

        if (__predict_false(uvm.page_init_done == false)) {
                if (sz % sizeof(struct uvm_physseg))
                        panic("%s: tried to alloc size other than multiple"
                            "of struct uvm_physseg at boot\n", __func__);

                size_t n = sz / sizeof(struct uvm_physseg);
                nseg += n;

                KASSERT(nseg > 0 && nseg <= VM_PHYSSEG_MAX);

                return &uvm_physseg[nseg - n];
        }

        return kmem_zalloc(sz, KM_NOSLEEP);
}

The pre-boot part is quite clever in the fact that, it returns a pointer to an
entry in the static array, to the caller of uvm_physseg_alloc() it looks like
a fresh memory location and the rbtree(3) api is quite happy to take this in
without any errors.

The equivalent free() now utilize uvm_physseg_free()

static void
uvm_physseg_free(void *p, size_t sz)
{
        /*
         * This is a bit tricky. We do allow simulation of free()
         * during boot (for eg: when MD code is "steal"ing memory,
         * and the segment has been exhausted (and thus needs to be
         * free() - ed.
         * free() also complicates things because we leak the
         * free(). Therefore calling code can't assume that free()-ed
         * memory is available for alloc() again, at boot time.
         *
         * Thus we can't explicitly disallow free()s during
         * boot time. However, the same restriction for alloc()
         * applies to free(). We only allow uvm_physseg related free()s
         * via this function during boot time.
         */

        if (__predict_false(uvm.page_init_done == false)) {
                if (sz % sizeof(struct uvm_physseg))
                        panic("%s: tried to free size other than struct uvm_physseg"
                            "at boot\n", __func__);

        }

        /*
         * Could have been in a single if(){} block - split for
         * clarity
         */

        if ((struct uvm_physseg *)p >= uvm_physseg &&
            (struct uvm_physseg *)p < (uvm_physseg + VM_PHYSSEG_MAX)) {
                if (sz % sizeof(struct uvm_physseg))
                        panic("%s: tried to free() other than struct uvm_physseg"
                            "from static array\n", __func__);

                if ((sz / sizeof(struct uvm_physseg)) >= VM_PHYSSEG_MAX)
                        panic("%s: tried to free() the entire static array!", __func__);
                return; /* Nothing to free */
        }

        kmem_free(p, sz);
}

As mentioned in the above comment the free() has a caveat that it actually
does not free anything in case of a static array, but it does have some sanity
checks in place to make sure that something that is not within the limits of
uvm_physseg[] array is not free()ed and that we do not end up free()ing
the whole uvm_physseg[] array.

As you can see once post-boot is reached the kmem(9) allocators come into action
and things work out as expected.