In order to understand how hot-plug was implemented in balloon(4) one needs to
understand what balloon(4) does.
Here is an excerpt from the man page[1].
The balloon driver supports the memory ballooning operations offered in Xen
environments. It allows shrinking or extending a domain’s available memory by
passing pages between different domains. At any time, the total memory
available to a domain is called the ``reservation’’.
In the Xen environment the xl(1) tool is used for managing the various guest
domUs within Xen. xl(1) has an option called mem-max[2] which sets the maximum
usable memory to a domain. The balloon(4) driver is responsible for managing the
memory consumption in various domains, the balloon expands as it collects the
unused pages from a domain and contracts as it gives out pages to a domain. Here
is another excerpt explaining how balloon(4) works.
Any domain is free to request memory from balloon up to the maximum value set
by the host’s administrator through the mem-max command of xm(1).
Alternatively, the host’s administrator is free to request to a particular
domain to give some memory back. This command requires the targeted domain’s
cooperation and requires balloon support within it. This can be done through
the mem-set command of xm(1).
However once a domain starts up, the mem-max value cannot be changed on the
fly and such an increase in memory would require the domain to be shutdown
changes be made and then restarted, and this is mentioned in the man page as one
of the unsupported features at the time balloon(4) driver was written.
memory ‘hot-plug’ unsupported - clipping reservation %zu => %zu pages.
An attempt was made by domain to get more memory than initially obtained
during boot. As physical memory pages cannot be added to memory manage-
ment sub-system dynamically, balloon will limit reservation up to the
maximum value it can handle.
Cherry (cherry@) being one of the co-authors of balloon(4) saw this as an
opportunity to test out uvm_hotplug(9). So after some hours of work fiddling
around with his balloon driver in sys/arch/xen/xen/balloon.c he was able to
implement hot-plug via the new uvm_physseg_plug() call into the existing call.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
/* reclaim pages from balloon */
@@ -492,16 +495,27 @@
/* plug pages back into memory through bpge entries */
for (rpages = 0; rpages < ret; rpages++) {
-
-#ifdef noyet
- if (sc->balloon_num_page_entries == 0) {
- /*
- * XXX This is the case where extra "hot-plug"
- * mem w.r.t boot comes in
- */
- device_printf(sc->sc_dev,
- "List empty. Cannot be collapsed further!\n");
- break;
+ extern paddr_t pmap_pa_end;
+ if (sc->balloon_num_page_entries == 0) { /*XXX: consolidate */
+ /* "hot-plug": Stick it at the end of memory */
+ pa = pmap_pa_end;
+
+ /* P2M update */
+ s = splvm();
+ pmap_pa_end += PAGE_SIZE; /* XXX: TLB flush ?*/
+ xpmap_ptom_map(pa, ptoa(mfn_list[rpages]));
+ xpq_queue_machphys_update(ptoa(mfn_list[rpages]), pa);
+ splx(s);
+
+ if (uvm_physmem_plug(atop(pa), 1, NULL) == false) {
+ /* Undo P2M */
+ s = splvm();
+ xpmap_ptom_unmap(pa);
+ xpq_queue_machphys_update(ptoa(mfn_list[rpages]), 0);
+ pmap_pa_end -= PAGE_SIZE; /* XXX: TLB flush ?*/
+ splx(s);
+ break;
+ }
+ continue;
}
-#endif
Even though I am not well versed in Xen or the code related to it, I was quite
happy to see it in action for the first time. And balloon(4) being the first
consumer of this new API.
NOTE: The balloon(4) man page references xm(1) which is a deprecated tool for
managing Xen, xl(1) is the updated tool.
References
- http://netbsd.gw.com/cgi-bin/man-cgi?balloon+4+NetBSD-current
- https://xenbits.xen.org/docs/unstable/man/xl.1.html
P.S: One thing I should have done is, asked Cherry is to give me a screenshot of
the logs of the hot-plug in action. It is not too late, keep an eye out for
updates.