Thanks to

In the end we were able to fix ...
and that "we" includes:

• Matt Thomas
• Nick Hudson
• Jared McNeill
• Jörg Sonnenberger
• FUKAUMI Naoki

Special thanks to everyone involved (even if I happened to forgot to mention you in above list)!

Missing drivers

Image from cubieboard.org under creative commons license.

Missing network

• Based on a Synopsis Designware IP core
• Used in slightly different variations in other Allwinner based devices (BanaPI, Hummingbird A31 and A80)
• Later also found on Odroid-C1
• No usable documentation available from Synopsis or Sunxi (the Allwinner manufacturer)
• But luckily we found some documentation from other SoC vendors using a later version of the same core.

With the help of others, I wrote the "awge" driver for network part.

Now the only important missing parts are:

• NAND (work in progress, Sunxi just released a source code for a Linux driver; before only a binary blob was available)
• WiFi (no docs available).

Otherwise the CUBIETRUCK support is pretty complete.

New Object Format?

Marketing required compatibility to old object files.

But the new cores required all instructions to be encoded in little endian byte order!

Luckily all ARM provided tools already marked code sections with special symbols.

So the "compatibility magic" was put into the linker, and the ABI extended by a subset of the special symbols always used by ARM tools.

Special Symbols

Simple idea: find all 32bit code parts and swap the instructions accordingly, find all 16bit thumb instruction parts and swap those.

• $a, $a.1 .. $a.N
  marks the start of a sequence of 32bit "arm" instructions
• $t, $t.1 .. $t.N
  marks the start of a sequence of 16bit "thumb" instructions
• $d, $d.1 .. $d.N
  marks the start of other data (like static strings).

Early console did not work

After noticing that the early console worked well for little endian kernels, the bug was easy to spot:

The three minimalistic functions used for polled console do direct hardware access (no bus_space abstraction involved), and those accesses did not provide byte swapping if needed.

Early console did not work

After fixing, the code to wait for the com device to become ready to transmit a character looks like this:

 while ((le32toh(uart_base[com_lsr])&& LSR_TXRDY) == 0 && --timo > 0)
     ;

Adding a few le32toh() and htole32() calls, like in the example above, fixed the issue.

Swapping Instruction Byte Order

 int
 kobj_machdep(kobj_t ko, void *base,
 size_t size, bool load)
 {
      if (load) {
 #if __ARMEB__
          if (CPU_IS_ARMV7_P())
               kobj_be8_fixup(ko);
 #endif

Then we need a simple function to categorize symbols:

Categorizing Special Symbols

 static enum be8_magic_sym_type
 be8_sym_type(const char *name, int info)
 {
     if (ELF_ST_BIND(info) != STB_LOCAL)
         return Other;
     if (ELF_ST_TYPE(info) != STT_NOTYPE)
         return Other;
     if (name[0] != '$' || name[1] == '\0' ||
        (name[2] != '\0' && name[2] != '.'))
         return Other;
     switch (name[1]) {
     case 'a':
         return ArmStart;
     case 'd':
         return DataStart;
     case 't':
         return ThumbStart;
     default:
         return Other;
     }
 }

Iterating Symbols

The following code iterates all symbols in the new loaded module:

 /*
  * Count all special relocations symbols
  */
 ksyms_mod_foreach(ko->ko_name, be8_ksym_count, &relsym_cnt);

where relsym_cnt is:

 long relsym_cnt = 0;

and the callback function be8_ksym_count looks like:

Count Special Symbols

 static int
 be8_ksym_count(const char *name, int symindex, void *value,
     uint32_t size, int info, void *cookie)
 {
     size_t *res = cookie;
     enum be8_magic_sym_type t = be8_sym_type(name, info);

     if (t != Other)
         (*res)++;
     return 0;
 }

Swapping the Instructions

After counting we allocate storage for all the relevant symbols, and run another iteration where each symbol, together with type and address is stored.

This array then is sorted by address (calling kheapsort()).

Finally we run through the array in ascending address order, and for all sections, depending on type of the symbol describing it, swap 32 bits, swap 16 bits, or do nothing.

kobj_machdep() will flush caches after we are done.

libgcc

But: when finally linking executables, the linker is invoked with -be8 option and tries to do the magic byte swapping for thumb and arm instructions.

If during this links it pulls in some function from libgcc, the swapping will not work for that function, as we stripped the "unused" local symbols in the libgcc pre-linking step.

The first program affected during a multi-user boot was fc-cache (after installation or X/font related updates), it uses __popcountsi2 on armv7.

Trivial Fix

The obvious fix: do not strip.

But other symbols had to be removed, as we would get duplicated symbols (it is a mess, maybe better do not ask).

So strip got replaced by a slightly more magic objcopy invocation that removed all local symbols but left the special ones in place.

ld.elf_so and gdb interaction

This breakpoint is hit whenever a new shared object is loaded by ld.elf_so (and the main program binary is the first one to trigger). Gdb then extracts all necessary information about the new shared library (or main module) and continues from the breakpoint.

On ARM, a gdb breakpoint is done by replacing the instruction temporarily with a special illegal instruction and then trap the SIGILL via ptrace. The replacement instruction is coded as an array of bytes in gdb - and there is a little endian and a big endian variant.

Out of sync

Code inspection showed that Gdb upstream had added a new member "byte_order_for_code" in the struct "gdbarch_info", but the NetBSD specific breakpoint instruction was still selected by the old "byte_order" member. Upstream had fixed it for all other ARM targets, but somehow the NetBSD specific code is not in sync and did not get updated.

This should not happen at various levels, but...

What happened?

So gdb selected the different endian encoding for the break point illegal instruction, and instead of causing a trap, this just did some random arithmetic and continued in whatever function happened to live in ld.elf_so next to it:

 0x21e8 <_rtld_debug_state>: smlattne r0, r6, r0, r0
 0x21ec <_rtld_objlist_clear>: mov r12, sp
 0x21f0 <_rtld_objlist_clear+4>: push {r3, r4, r11, r12, lr, pc} 

(smlattne = signed multiply long accumulate, top half * top half, conditional if "not equal")

This corrupted internal ld.elf_so data structures immediately before returning to ld.elf_so internal fixup work.

Obvious fix: select breakpoint according to new member (and better sync with upstream)

CFI and unwinding

ATF is written in C++ and heavily relies on exceptions, e.g. to abort a failing test case.

One of the steps in exception unwinding is to identify the call frame from the current %pc value and then use the corresponding Call Frame Information stored by the compiler to unwind the stack properly and find the parent call frame.

Code details were just slightly different on BE8 to trigger a bug in the binary search to identify the relevant CFI entry.

Exceptionless FPUs

So tests grew code like this:

 #elif defined(__arm__) && !__SOFTFP__
     /*
      * Some NEON fpus do not implement IEEE exception handling,
      * skip these tests if running on them and compiled for
      * hard float.
      */
     if (0 == fpsetmask(fpsetmask(FP_X_INV)))
          atf_tc_skip("FPU does not implement exception handling");
 #endif