We present the behavior of multiple solid solutions within ternary (BaxCa1–x)B6 and (BaxSr1–x)B6 compounds and demonstrate that nanodomain formation is preferred over uniform solid solutions under certain processing conditions. Instead of the expected single solid solution of M1 and/or M2 atoms within the MB6 phase, we note separation into nanodomain regions rich in either M1 or M2. This phase separation has been observed from detailed analyses of the shapes of the peaks in X-ray diffraction data, where peak splitting and asymmetry are the result of multiple solid solutions with lattice parameters differing by up to 1.4%. High-resolution transmission electron microscopy confirms the presence of these nanodomains, which are about 2–3 nm in size, and reveals varying degrees of lattice misalignment. We also present X-ray diffraction analysis of (BaxCa1–x)B6 powders calcined from 1273 to 1973 K and document the enhancement in sample homogeneity as the separated phases merge into a uniform solid solution. As subsequent calcinations at lower temperatures do not result in a re-separation of phases, the nanodomains are deemed metastable. The greatest degree of phase separation is observed in the (BaxCa1–x)B6 system, which corresponds to the largest difference in cation radii (0.161 vs. 0.134 nm for Ba2+ and Ca2+, respectively). Analysis of the chemical reactions that occur during synthesis suggests that the decomposition of the metal precursors (nitrates and carbonates) to metal oxides may cause selective MB6 phase formation in mixed-cation hexaborides.