Xbox Linux Issues

Here is a list of issues that are blocking general functionality on Xbox Linux.

Native Boot Partition Filesystem Compatibility with Cromwell

When booting a "native" install, Cromwell scans the MBR and looks for partitions that are flagged as bootable. If the partition is bootable, Cromwell attempts to determine the filesystem, and load linuxboot.cfg. If the user launches that instance, the kernel, and the initrd will then be loaded from that partition. (The presence of initrd and kernel is not checked until launching)

When setting up a native install, a boot partition formatted with ext2 worked fine as formatted by historical X-DSL. The linuxboot.cfg was detected, and Cromwell presented a hard drive icon to load the native install.

When formatting the same partition using a modern distro, Cromwell does not detect the linuxboot.cfg, but it does detect the ext2 filesystem and attempts to read it without errors. (This has been tested by formatting on Gentoo, Debian, and from a Gentoo liveCD on the Xbox.)

Something about the way these ext2 partitions are being formatted by modern tools is preventing the Xbox from reading files on them. Turning off the new ext2 features using mkfs.ext2 /dev/sdb1 -O ^resize_inode,^dir_index,^large_file,^ext_attr did not make a difference.

Formatting this partition from modern Linux with vfat and no options also did not work.

Formatting as Fat32 in Windows with EaseUs Partition Master did work.

Note that Cromwell is able to read the root partition and get linuxboot.cfg from there even for larger root partitions (tested up to 80GB); a reserved boot partition is not strictly needed, but Cromwell still needs to be able to load the kernel and initrd.

So the main question here is, what kind of partitions does Cromwell support? How can at least one kind of these partitions be reliably formatted using a terminal command in Linux?

Or, can Cromwell be easily updated to support new filesystem developments? Since a boot partition is only needed for Cromwell to load the kernel and initrd, it would be acceptable for a more archaic filesystem configuration to be used as the boot partition, which would still give users the freedom to format the root partition with the kernel-supported filesystem of their choice (which is basically the point of a boot partition.) Ext2 is not necessarily the only option here either.

XBPartitioner Table and MBR Incompatibility

This issue is relevant to native Linux installs. In a native Linux install, the user formats empty regions of their hard drive and adds an MBR to the hard drive, which is used by Cromwell to access the Linux boot partition and subsequently load linuxboot.cfg, the kernel, and the initrd. The way that Cromwell is currently set up, it can't find Linux partitions without the MBR.

The MBR partitioning/boot scheme uses a 512 byte record at the beginning of the drive. Since the actual boot data isn't needed on Xbox, the first 440 bytes are filled with zeroes when using fdisk, and can be considered unused here. [layout information]

The FATX filesystem has a config area of 1024 bytes starting at the beginning of the drive. Only 48 bytes are officially used; the rest are zeroes. On a stock-formatted Xbox hard drive, the contents of this config area don't seem to matter much to ind-BIOS (tested), although Insignia may use it in the future. If a stock FATX config area has its first 512 bytes overwritten by an MBR, it is not known to cause problems with any other Xbox functionality. (based on the fact that native Linux installs have been run in this way for years)

The problem arises with the XBPartitioner table, a custom partitioning scheme for Xbox hard drives that was never officially supported by Xbox Linux. It is common on Xboxes with larger hard drives. The XBPartitioner table provides 14 partition "slots" for FATX partitions, and it allows users to format their Xbox hard drive with additional FATX partitions. The partition table is written into unused space in the FATX config area, immediately after the stock data. XBPartitioner wipes the remaining bytes of the config area when writing a partition table including any MBR contents; other tools are untested so far. And fdisk overwrites the first 512 bytes of the FATX config area with an MBR, replacing the first 440 bytes with zeroes.

Another issue arises in that the two partitioning schemes overlap, even if only the necessary data were written for both XBPartitioner table and MBR; that is if they didn't write the zeroes.

This struct shows the layout of the config area, followed by the partition entry struct. From the unofficial XBPartitioner patches for the old Xbox kernel:

// Temporary struct used to load XBPartitioner data directly from disk.
typedef struct {
	char                 magic[16];
	char                 reserved[32];
	XBPartitionerTableEntry partitions[FATX_XBPARTITIONER_PARTITIONS_MAX];
} XBPartitionerTable;

// Temporary struct used to load XBPartitioner data directly from disk.
typedef struct
{
	char name[16];
	u32  flags;
	u32  start;
	u32  size;
	u32  reserved;
} XBPartitionerTableEntry;

The XBPartitioner table entries start at 48 bytes, and each partition's entry is 32 bytes in size. With 14 partitions, the XBPartitioner table ends at 496 bytes, and overwrites a good chunk of the relevant MBR content, which starts at 440 bytes. So tool modification alone can't make these schemes compatible.

The XBPartitioner table has no metadata, only partition entries, and has no known way of redefining the number of available partitions. However, if the XBPartitioner table only had 12 partitions, the partition table would end at 432 bytes and the two schemes could coexist.

Possible Solutions

F Takes All

This was the typical scheme used by historical Xbox Linux. The area after E would be used for Linux partitions. With larger hard drives available, there is a need for Xbox Linux to coexist with other content on the user's drive. This is okay when the user doesn't want to later modify the partition layout of their hard drive using XBPartitioner, and they want to reserve that hard drive primarily for Linux.

A 12 partition XBPartitioner Table

This would involve attempting to get general acceptance of changing the XBPartitioner spec to 12 partitions, so it doesn't overlap with MBR. Or providing a custom set of tools and bioses that support this. Pros

• Cromwell would not need to be modified significantly
• The 512 byte boundary could be considered the starting point for future config area modifications
• Very few users have more than 7, much less 14 partitions on their Xbox which provide little practical use
• Any Linux PC can easily mount the Linux partitions

Cons

• Lots of software would need to be updated, including BIOSes, partitioning tools, and disk formatting tools in Xbox Linux and on PC. There might be a more elegant solution.

Using the XBPartitioner Table To Define Linux Partitions

Pros

• No partitioning scheme incompatiblity management
• No dual partitioning scheme; both Linux and Xbox tools would have awareness of the full drive partition table, to avoid accidentally creating overlapping partitions
• Unused per-partition flags in the XBPartitioner table could be used to mark partitions as Linux as a future standard
• Scripts utilizing dd to read the XBPartitioner table could be used as a stopgap before full kernel support is developed

Cons

• The Cromwell Grub code would need to be modified to use the XBPartitioner table in place of MBR (this may not be overly complicated)
• Scripts, tools, or kernel support will be needed in order to mount these partitions in Linux, both on the Xbox and on other PCs
• Any tools used to mount Linux partitions and edit their characteristics would have to be updated to support it, primarily apps that mount non-FATX partitions. This could be simple
• FATX kernel support would need to be ported for a long term performant solution. (fatxfs performance still needs to be evaluated on Xbox, as an alterative option)
• The user is limited to 14 total partitions; with F and G they have 7 partitions which is enough for 3 Linux installs (3 boot partitions, 3 data partitions, and shared swap). It would be workable though for most use cases.

Using Contiguous Files Allocated on FATX as a Filesystem

This was proposed by mborgerson, as an option to avoid dealing with partition tables at all. Since coexistence of Linux filesystems with FATX is desired, and there are incompatibilities in the current partition table schemes, and correcting this would take a great deal of effort, it was suggested to contain the raw partition data on FATX partitions. A FATX Linux utility could scan a FATX partition for contiguous available space. It could then allocate several 4GB files (the FATX file size limit) across that range of space. Then, the entire range of sectors could be formatted a Linux partition or group of partitions by sector range, and a distribution installed there.

Pros

• This avoids modifying partition tables at all, so we don't have to worry about changing existing standards.
• The tool fatxfs provides a starting point for interacting with the FATX filesystem without kernel FATX support. FATX partitions only need to be temporarily accessed as such, but once the contiguous files are created and the range of the files is known, Linux can work with that area of the disk directly. After starting up and mounting any partitions, Linux does not need any knowledge of FATX.
• Contiguous files can be checked for contiguity before mounting to avoid overwriting other data
• Users could reserve a FATX partition for Linux in XBPartitioner and then use Linux to fill it with contiguous files, the end result should be equivalent to formatting a partition and using MBR to access it, aside from the mounting process itself.
• Swap could be shared between Linux installs as long as some kind of common file/directory name or other method is used to detect shared swap on a FATX partition.
• Users can place the kernel and intitrd on FATX partitions and launch using existing Cromwell functionality for loading from FATX, so no boot partitions are needed here

Cons

• A custom tool would be needed to access and mount these partitions, both on the Xbox and outside of it. This is still much simpler to address than the issues in the previous proposals.
• The user's filesystem may be fragmented in about the same manner as a FAT filesystem. In order to create space for contiguous files, defragmentation may be needed, especially on partitions which have seen a lot of use.
• If a user downloads these contiguous files and FTPs them to their Xbox, these files may become fragmented. (It is not fully understood in what situations the BIOS decides to fragment a file.) In order for FTP to work for sharing these files, a utility would have to be able to "defragment" an image and place the data in the right order on the hard drive, without gaps. This could take some time for larger images. Maybe a Linux tool could be built to defragment the drive first and then handle the FTP part to make sure it gets transferred correctly.

Lack of FATX Kernel Support

Loop devices are a common method of packaging Xbox Linux installations, in order to avoid modifying the partition table on the Xbox hard drive. A loop device is a filesystem image that can be mounted from within another filesystem. When put on a FATX partition, they allow Xbox Linux to be run from the hard drive without modifying the user's partition table at all. These setups can be used to bundle Linux builds and other apps for easy distribution. (XMugen, for instance.)

The current Xbox Linux patches don't port over the support for mounting the FATX filesystem in Linux. This is because the underlying code has changed significantly in a few places since then. User:VannevarKush is working on porting the old FATX code to the kernel for native support, but it is a work in progress.

The lack of native FATX support means that mounting loop devices can't be done. Right now, native install and live CDs are the current available boot options for Xbox Linux.

The FUSE based fatxfs code by mborgerson may be usable as a stopgap, although its performance hasn't been evaluated yet on an Xbox.

Advanced Video Support

The Xbox Linux kernel currently only uses the most basic video support available (defined by CONFIG_FRAMEBUFFER_SIMPLE in the kernel config). More advanced support may be needed to run desktop environments and to run graphical applications with better performance. Does code need to be merged over from the original sources for xbox video?