AmigaOS Documentation Wiki - User contributions [en]

The Hacking Way: Part 1 - First Steps

2022-02-09T10:21:19Z

Roman Kargin: /* Genuine ELF executables */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities, and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects (which is not really shared unlike classic amiga libraries). All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType, etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS, there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in any version of AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialization for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to being executed at an absolute address of 0x01000000. But this is a placeholder, i.e. only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes, as, without relocations, each new process would be loaded at 0x01000000 and overwrites the previous one which will cause all sorts of weird crashes and issues. The ELF loader just ignores the load address of 0x1000000+size_of_headers from the executable completely, and just allocates some free memory and loads the program segment there.

At the beginning of the AmigaOS4 era from 2004 and till 2008, this placeholder was 0x0000000, but then in 2008, in the SDK 53.3, it was changed to 0x01000000. That switch was necessary for the shared objects support since these did add a .plt section, potentially in front of the .text (which means 0 as the text base address was no longer working). Of course, this placeholder can be any different value, but we can assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. The "-q" switch is the one that says to the linker to relocate all the absolute addresses within the program code and data.

But to perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at the linking stage, which contains all the OS-dependent initializations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course, it is possible to make a working binary without relocation if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course, you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[https://www.nxp.com/files-static/product/doc/MPCFPE32B.pdf?&srch=1 Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-09T09:55:11Z

Roman Kargin: /* Myth #1: AmigaOS behaves like UNIX */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities, and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects (which is not really shared unlike classic amiga libraries). All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType, etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS, there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in any version of AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialization for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[https://www.nxp.com/files-static/product/doc/MPCFPE32B.pdf?&srch=1 Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-09T09:54:50Z

Roman Kargin: /* Myth #1: AmigaOS behaves like UNIX */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities, and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects (which is not really shared unlike classic amiga libraries). All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType, etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS, there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in any AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialization for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[https://www.nxp.com/files-static/product/doc/MPCFPE32B.pdf?&srch=1 Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-09T09:53:36Z

Roman Kargin: /* Myth #1: AmigaOS behaves like UNIX */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities, and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects (which is not really shared unlike classic amiga libraries). All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType, etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS, there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialization for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[https://www.nxp.com/files-static/product/doc/MPCFPE32B.pdf?&srch=1 Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-08T17:53:47Z

Roman Kargin: /* Links */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects. All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux, or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialisation for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[https://www.nxp.com/files-static/product/doc/MPCFPE32B.pdf?&srch=1 Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-08T17:51:26Z

Roman Kargin: /* Links */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects. All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux, or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialisation for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf PPC SYSV4 ABI]<br/>
[http://www.freescale.com/files/product/doc/MPCFPE32B.pdf Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

The Hacking Way: Part 1 - First Steps

2022-02-08T17:49:45Z

Roman Kargin: /* Assembler programming using libc */

= Author =

Roman Kargin<br/>
Copyright (c) 2012 Roman Kargin<br/>
Proofread and grammar corrections by Daniel jedlicka.<br/>
Used by permission.

= Introduction =

Back in the past, I wanted to make the smallest possible executables on UNIX-ish operating systems (SunOS, Tru64, OS9, OpenVMS and others). As a result of my research I wrote a couple of small tutorials for various hacking-related magazines (like Phrack or x25zine). Doing the same on AmigaOS naturally became a topic of interest for me - even more so when I started seeing, in Amiga forums, questions like "Why are AmigaOS binaries bigger than they should be?" Therefore I believe that producing small AmigaOS executables could make an interesting topic for an article. Further in the text I'll explain how ldscripts can help the linker make non-aligned binaries, and cover various other aspects associated with the topic. I hope that at least for programmers the article will be an interesting and thought-provoking read.

Before you go on, please note that it is assumed here that you have basic programming skills and understanding of C and assembler, that you are familiar with BSD syntax, know how UNIX and AmigaOS work, and that you have the PPC V.4-ABI and ELF specification at hand. But if you don't, there's no need to stop reading as I'll try to cover the basics where necessary.

= The Basics =

To begin with, let's present and discuss some basic terms and concepts. We'll also dispel some popular myths.

== The C standard library (libc) ==

Thirty years ago, when the C language developed so much that its different implementations started to pose a practical problem, the American National Institute of Standards (ANSI) formed a committee for the standardization of the language. The standard, generally referred to as ANSI C, was finally adopted in 1989 (this is why it is sometimes called C89). Part of this standard was a library including common functions, called the "C standard library", or "C library", or "libc". The library has been an inherent part of all subsequently adopted C standards.

Libc is platform-independent in the sense that it provides the same functionality regardless of operating system - be it UNIX, Linux, AmigaOS, OpenVMS, whatever. The actual implementation may vary from OS to OS. For example in UNIX, the most popular implementation of the C standard library is glibc (GNU Library C). But there are others: uClibc (for embedded Linux systems, without MMU), dietlibc (as the name suggests, it is meant to compile/link programs to the smallest possible size) or Newlib. Originally developed for a wide range of embedded systems, Newlib is the preferred C standard library in AmigaOS and is now part of the kernel.

On AmigaOS, three implementations of libc are used: clib2, newlib and vclib. The GCC compiler supports clib2 and newlib, the VBCC compiler supports newlib and vclib.

=== clib2 ===

This is an Amiga-specific implementation originally written from scratch by Olaf Barthel, with some ideas borrowed from the BSD libc implementation, libnix, etc. Under AmigaOS, clib2 is most often used when maximum compatibility with POSIX is required. The GCC compiler distributed as part of the AmigaOS SDK uses Newlib by default (as if you used the -mcrt=newlib switch). An important note: clib2 is only available for static linking, while Newlib is opened at runtime (thus making your executables smaller). Clib2 is open source, the latest version can be found at http://sourceforge.net/projects/clib2/

=== Newlib ===

A better and more modern libc implementation. While the AmigaOS version is closed source (all adaptations and additional work is done by the OS development team), it's based on the open source version of Newlib. The original version is maintained by RedHat developer Jeff Johnston, and is used in most commercial and non-commercical GCC ports for non-Linux embedded systems: http://www.sourceware.org/newlib/

Newlib does not cover the ANSI C99 standard only: it's an expanded library that also includes common POSIX functions (clib2 implements them as well). But certain POSIX functions - such as glob(), globfree(), or fork() - are missing; and while some of them are easy to implement, others are not - fork() being an example of the latter.

Newlib is also available as a shared object.

=== vclib ===

This library was made for the vbcc compiler. Like clib2 it is linked statically, but only provides ANSI C/C99 functions (i.e. no POSIX).

= Myth #1: AmigaOS behaves like UNIX =

From time to time you can hear voices saying that AmigaOS is becoming UNIX. This popular myth stems from three main sources. First, many games, utilities and libraries are ported over from the UNIX world. Second, AmigaOS uses genuine ELF, the standard binary file format used in UNIX and UNIX-like systems. Third, the OS supports, as of version 4.1, shared objects. All of this enables AmigaOS to provide more stuff for both programmers and users, and to complement native applications made for it. Today, it is quite normal that an operating system provides all the popular third-party libraries like SDL, OpenGL, Cairo, Boost, OpenAL, FreeType etc. Not only they make software development faster but they also allow platform-independent programming.

Yet getting close to UNIX or Linux in terms of software or programming tools does not mean that AmigaOS behaves in the same way as regards, for example, library initialization, passing arguments or system calls. On AmigaOS there are no "system calls" as they are on UNIXes, where you can simply pass arguments to registers and then use an instruction (like "int 0x80h" on x86 Linux, "trap 0" on M68 Linux, or "sc" on some PPC/POWER CPU based OSes), which will cause a software interrupt and enter the kernel in supervisor mode. The concept of AmigaOS is completely different. There is no kernel as such; Amiga's Kickstart is actually a collection of libraries (of which "kernel.kmod" is just one module - a new incarnation of the original exec.library). Also, an AmigaOS program, when calling a library function, won’t enter supervisor mode but rather stays in user mode when the function is executed.

[[File:HackingWayPart1-1.png|center]]

Since the very first version of the OS that came with the Amigas in 1985, you must open a library and use its vector table to execute a library function, so there’s no "system call" involved. The pointer to the first library (exec.library) is always at address 4 and that hasn’t changed in AmigaOS.

When you program in assembler under AmigaOS, you cannot do much until you initialize and open all the needed libraries (unlike, for example, on UNIX where the kernel does all the necessary initialisation for you).

= Myth #2: AmigaOS binaries are fat =

This misunderstanding stems from the fact that the latest AmigaOS SDK uses a newer version of binutils, which now aligns ELF segments to 64K so that they can be easily loaded with mmap(). Binutils are, naturally, developed with regard to UNIX-like OSes where the mmap() function actually exists so the modifications make sense - but since mmap() isn’t a genuine AmigaOS function (it’s just a wrapper using AllocVec() etc.), this kind of alignment is not needed for AmigaOS.

Luckily, the size difference is only noticeable in small programs, like Hello World, where the resulting executable grows to 65KB. Which of course is unbelievable and looks like something is wrong. But once you start programming for real and produce bigger programs, the code fills up the ELF segments as required, there’s little need for padding, and so there’s little size difference in the end. The worst-case scenario is ~64KB of extra padding, which only happens (as we said) in very small programs, or when you’re out of luck and your code only just exceeds a boundary between two segments.

It is likely that a newer SDK will adapt binutils for AmigaOS and the padding will no longer be needed. Currently, to avoid alignment you can use the "-N" switch, which tells the linker to use an ldscript that builds non-aligned binaries. Check the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory; all the files ending with an "n" (like “AmigaOS.xn” or “ELF32ppc.xbn”) are linker scripts that ensure non-aligned builds. Such a script will be used when the GCC compiler receives the “-N” switch. See the following:

<pre>
7/0.RAM Disk:> type hello.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
6/1.Work:> gcc hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
65k
6/1.Work:> hello
aaaa
</pre>

<pre>
6/1.Work:> gcc -N hello.c -o hello
6/1.Work:> strip hello
6/1.Work:> filesize format=%s hello
5480
6/1.work:> hello
aaaa
</pre>

= Genuine ELF executables =

Just like libc, the Executable and Linkable Format (ELF) is a common standard. It is a file format used for executables, objects and shared libraries. It gets the most attention in connection with UNIX but it is really used on numerous other operating systems: all UNIX derivatives (Solaris, Irix, Linux, BSD, etc.), OpenVMS, several OSes used in mobile phones/devices, game consoles such as the PlayStation, the Wii and others. PowerUP, the PPC Amiga kernel made by Phase5 back in the 1990s used the ELF format as well.

A more detailed description of the ELF internals will be given later; all you need to know for now is that the executable ELF file contains headers (the main header, and headers for the various sections) and sections/segments. The ELF file layout looks like this:

[[File:HackingWayPart1-2.png|center]]

AmigaOS uses genuine ELF executables versus relocatable objects.

The advantage of objects is that they are smaller and that relocations are always included. But there is a drawback as well: the linker will not tell you automatically whether all symbols have been resolved because an object is allowed to have unresolved references. (On the other hand, vlink could always detect unresolved references when linking PowerUP objects because it sees them as a new format.) This is why ELF shared objects cannot be used easily (though it’s still kind of possible using some hacks), and it explains why the AmigaOS team decided to go for real executables.

By specification, ELF files are meant to be executed from a fixed absolute address, and so AmigaOS programs need to be relocated (because all processes share the same address space). To do that, the compiler is passed the -q switch ("keep relocations"). Relocations are handled by the MMU, which will create a new virtual address space for each new process.

If you look at the linker scripts provided to build AmigaOS executables (in the SDK:gcc/ppc-AmigaOS/lib/ldscripts directory), you’ll find the following piece of code:

<pre>
ENTRY(_start)
....
SECTIONS
{
PROVIDE (__executable_start = 0x01000000); . = 0x01000000 + SIZEOF_HEADERS;
[...]
</pre>

As you can see, AmigaOS executables look like they are linked to be executed at an absolute address of 0x01000000. But this is only faked; the ELF loader and relocations will recalculate all absolute addresses in the program before it executes. Without relocations, each new process would be loaded at 0x01000000, where it would crash happily due to overwriting certain important areas, and because of other reasons. You may ask why 0x01000000 is used at all, considering that it’s just a placeholder and any number (be it 0x00000000, 0x99999999, 0xDEADBEEF or 0xFEEDFACE) can be used instead. We can speculate and assume that 0x01000000 was chosen because it is the beginning of the memory map accessible for instruction execution. But anyway, the value is currently not important.

To perform a test, let’s see what happens if we build our binary without the "-q" switch (that is, without making the binary relocatable):

<pre>
7/0.RAM Disk:> type test.c
</pre>

<syntaxhighlight>
#include <stdio.h>
main()
{
printf("aaaa");
}
</syntaxhighlight>

<pre>
shell:> gcc test.c -S -o test.s
shell:> as test.s -o test
shell:> ld test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
</pre>

When you run the executable, you get a DSI with the 80000003 error, on the 0x1c offset in _start (i.e. the code from the crtbegin.o). Ignoring the error will produce a yellow recoverable alert. The crash occurs because we have compiled an ELF file to be executed at the 0x01000000 address, and as no "-q" switch was used, the remapping did not take place. To better understand why it happens you can check the crtbegin.o code, i.e. the code added to the binary at linking stage, which contains all the OS-dependent initialisations. If you know nothing about PPC assembler you can skip the following part for now and return when you’ve read the entire article:

<pre>
6/0.RAM Disk:> objdump -D --no-show-raw-insn --stop-address=0x10000d0 test | grep -A8 "_start"
</pre>

<pre>
010000b0 <_start>:

10000b0: stwu r1,-64(r1) #
10000b4: mflr r0 # prologue (reserve 64 byte stack frame)
10000b8: stw r0,68(r1) #

10000bc: lis r9,257 # 257 is loaded into the higher half-word (msw) of r9 (257 << 16)
10000c0: stmw r25,36(r1) # offset into the stack frame
10000c4: mr r25,r3 # save command line stack pointer
10000c8: mr r27,r13 # r13 can be used as small data pointer in the V.4-ABI, and it also saved here
10000cc: stw r5,20(r9) # Write value (257 << 16) + 20 = 0x01010014 to the r5 (DOSBase pointer)
</pre>

The address in the last instruction points to a data segment starting at 0x010100000. But the address is invalid because, without any relocation, there is no data there and the MMU produces a data storage interrupt (DSI) error.

Of course it is possible to make a working binary without relocation, if the program doesn’t need to relocate and you are lucky enough to have the 0x1000000 address free of important contents. And of course you can use a different address for the entry point, by hex-editing the binary or at build-time using self-made ldscripts. Making a non-relocatable binary will be discussed further in the text.

= PowerPC assembly =

In case you are not familiar and have no experience with PowerPC assembly, the following section will explain some basic terms and concepts.

== Registers ==

The PowerPC processor architecture provides 32 general-purpose registers and 32 floating-point registers. We’ll only be interested in certain general-purpose registers and a couple of special ones. The following overview describes the registers as they are used under AmigaOS:

=== General-purpose registers ===

{| class="wikitable"
! Register
! AmigaOS usage
|-
| r0 || volatile register that may be modified during function linkage
|-
| r1 || stack-frame pointer, always valid
|-
| r2 || system reserved register
|-
| r3 || command-line pointer
|-
| r4 || command-line length
|-
| r5 || DOSBase pointer
|-
| colspan="2" | The contents of registers r3-r5 is only valid when the program starts)
|-
| r6 - r10 || volatile registers used for parameter passing
|-
| r11 - r12 || volatile registers that may be modified during function linkage
|-
| r13 || small data area pointer register
|-
| r14 - r30 || registers used for local variables; they are non-volatile; functions have to save and restore them
|-
| r31 || preferred by GCC in position-independent code (e.g. in shared objects) as a base pointer into the GOT section; however, the pointer can also be stored in another register
|}

Important note: This general-purpose register description shows that arguments can only be passed in registers r3 and above (that is, not in r0, r1 or r2). You need to keep that in mind when assembling/disassembling under AmigaOS.

=== Some special registers ===

{| class="wikitable"
| lr || link register; stores the "ret address" (i.e. the address to which a called function normally returns)
|-
| cr || condition register
|}

== Instructions ==

There are many different PowerPC instructions that serve many different purposes: there are branch instructions, condition register instructions, instructions for storage access, integer arithmetic, comparison, logic, rotation, cache control, processor management, and so on. In fact there are so many instructions that it would make no sense to cover them all here. You can download Freescale’s Green Book (see the Links section at the end of the article) if you are interested in a more detailed description but we’ll just stick to a number of instructions that are interesting and useful for our purposes.

; b
: Relative branch on address (example: "b 0x7fcc7244"). Note that there are both relative and absolute branches (ba). Relative branches can branch to a distance of -32 to +32MB. Absolute branches can jump to 0x00000000 - 0x01fffffc and 0xfe000000 - 0xfffffffc. However, absolute branches will not be used in AmigaOS programs.

; bctr
: Branch with count register. It uses the count register as a target address, so that the link register with, say, our return address remains unmodified.

; lis
: Stands for "load immediate shifted". The PowerPC instruction set doesn’t allow loading a 32-bit constant with a single instruction. You will always need two instructions that load the upper and the lower 16-bit half, respectively. For example, if you want to load 0x12345678 into register r3, you need to do the following:

<pre>
lis %r3,0x1234
ori %r3,%r3,0x5678
</pre>

Later in the article you’ll notice that this kind of construction is used all the time.

; mtlr
: "move to link register". In reality this is just a mnemonic for "mtspr 8,r". The instruction is typically used for transferring an address from register r0 to the link register (lr), but you can of course move contents to lr from other registers, not just r0.

; stwu
: "store word and update" (all instructions starting with “st” are for storing). For example, stwu %r1, -16(%r1) stores the contents of register r1 into a memory location whose effective address is calculated by taking the value of 16 from r1. At the same time, r1 is updated to contain the effective address. As we already know, register r1 contains the stack-frame pointer so our instruction stores the contents of the register to a position at offset -16 from the current top of stack and then decrements the stack pointer by 16.

The PowerPC processor has many more instructions and various kinds of mnemonics, all of which are well covered in numerous PPC-related tutorials, so to avoid copying-and-pasting (and wasting space here) we have described a few that happen to be used very often. You’ll need to refer to the relevant documentation if you want to read more about the PowerPC instruction set (see Links below).

== Function prologue and epilogue ==

When a C function executes, its code – seen from the assembler perspective – will contain two parts called the prologue (at the beginning of the function) and the epilogue (at the end of the function). The purpose of these parts is to save the return address so that the function knows where to jump after the subroutine is finished.

<pre>
stwu %r1,-16(%r1)
mflr %r0 # prologue, reserve 16 byte stack frame
stw %r0,20(%r1)

...

lwz %r0,20(%r1)
addi %r1,%r1,16 # epilogue, restore back
mtlr %r0
blr
</pre>

The prologue code generally opens a stack frame with a stwu instruction that increments register r1 and stores the old value at the first address of the new frame. The epilogue code just loads r1 with the old stack value.

C programmers needn’t worry at all about the prologue and epilogue because the compiler will add them to their functions automatically. When you write your programs in pure assembler you can skip the prologue and the epilogue if you don’t need to keep the return address.

Plus, a new stack frame doesn’t need to be allocated for functions that do not call any subroutine. By the way, the V.4-ABI (application binary interface) defines a specific layout of the stack frame and stipulates that it should be aligned to 16 bytes.

= Writing programs in assembler =

There are two ways to write assembler programs under AmigaOS:

; using libc
: all initializations are done by crtbegin.o/crtend.o and libc is attached to the binary

; the old way
: all initializations - opening libraries, interfaces etc. - have to be done manually in the code

The advantage of using libc is that you can run your code "out of the box" and that all you need to know is the correct offsets to the function pointers. On the minus side, the full library is attached to the binary, making it bigger. Sure, a size difference of ten or even a hundred kilobytes doesn’t play a big role these days – but here in this article we’re going down the old hacking way (that’s why we’re fiddling with assembler at all) so let’s call it a drawback.

The advantage of not using libc is that you gain full control of your program, you can only use the functions you need, and the resulting binary will be as small as possible (a fully working binary can have as little as 100 bytes in size). The drawback is that you have to initialize everything manually.

We’ll first discuss assembler programming with the use of libc.

== Assembler programming using libc ==

To illustrate how this works we’ll compile a Newlib-based binary (the default GCC setting) using the –gstabs switch (“include debugging information”) and then put the [[GDB_for_Beginners|GDB]] debugger on the job:

<syntaxhighlight>
#include <stdio.h>

main()
{
printf("aaaa");
exit(0);
}
</syntaxhighlight>

<pre>
6/0.RAM Disk:> gcc -gstabs -O2 2.c -o 2
2.c: In function 'main':
2.c:6: warning: incompatible implicit declaration of built-in function 'exit'

6/0.RAM Disk:> GDB -q 2
</pre>

<pre>
(GDB) break main
Breakpoint 1 at 0x7fcc7208: file 2.c, line 4.
(GDB) r
Starting program: /RAM Disk/2
BS 656d6ed8
Current action: 2

Breakpoint 1, main () at 2.c:4
4 {
(GDB) disas
Dump of assembler code for function main:
0x7fcc7208 <main+0>: stwu r1,-16(r1)
0x7fcc720c <main+4>: mflr r0
0x7fcc7210 <main+8>: lis r3,25875 ; that addr
0x7fcc7214 <main+12>: addi r3,r3,-16328 ; on our string
0x7fcc7218 <main+16>: stw r0,20(r1)
0x7fcc721c <main+20>: crclr 4*cr1+eq
0x7fcc7220 <main+24>: bl 0x7fcc7234 <printf>
0x7fcc7224 <main+28>: li r3,0
0x7fcc7228 <main+32>: bl 0x7fcc722c <exit>
End of assembler dump.
(GDB)
</pre>

Now we’ll use [[GDB_for_Beginners|GDB]] to disassemble the printf() and exit() functions from Newlib’s LibC.a. As mentioned above, Newlib is used by default, there’s no need to use the –mcrt switch unless we want clib2 instead (in which case we’d compile the source with “-mcrt=clib2”).

<pre>
(GDB) disas printf
Dump of assembler code for function printf:
0x7fcc723c <printf+0>: li r12,1200
0x7fcc7240 <printf+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)

(GDB) disas exit
Dump of assembler code for function exit:
0x7fcc7234 <exit+0>: li r12,1620
0x7fcc7238 <exit+4>: b 0x7fcc7244 <__NewLibCall>
End of assembler dump.
(GDB)
</pre>

You can see that register r12 contains some values depending on the function - they are function pointer offsets in Newlib’s interface structure (INewLib). Then there’s the actual jump to __NewLibCall, so let’s have a look at it:

<pre>
(GDB) disas __NewLibCall
Dump of assembler code for function __NewLibCall:
0x7fcc7244 <__NewLibCall+0>: lis r11,26006
0x7fcc7248 <__NewLibCall+4>: lwz r0,-25500(r11)
0x7fcc724c <__NewLibCall+8>: lwzx r11,r12,r0
0x7fcc7250 <__NewLibCall+12>: mtctr r11
0x7fcc7254 <__NewLibCall+16>: bctr
End of assembler dump.
(GDB)
</pre>

Of course you can use "objdump":

<pre>
6/0.RAM Disk:> objdump -d 1 | grep -A5 "<__NewLibCall>:"
</pre>

<pre>
01000280 <__NewLibCall>:
1000280: 3d 60 01 01 lis r11,257
1000284: 80 0b 00 24 lwz r0,36(r11)
1000288: 7d 6c 00 2e lwzx r11,r12,r0
100028c: 7d 69 03 a6 mtctr r11
1000290: 4e 80 04 20 bctr
</pre>

But using [[GDB_for_Beginners|GDB]] is more comfortable: you don’t need to scroll through the full objdump output, or search in it with grep, etc. You can, too, obtain assembler output by compiling the source with the –S switch but [[GDB_for_Beginners|GDB]] makes it possible to get as deep into the code as you wish (in fact down to the kernel level).

We will now remove the prologue (because we don’t need it in this case) and reorganize the code a bit:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
bl printf #

li %r3,0 # exit(0);
bl exit #

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5360
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

When we compile our Hello World program in C (with the -N switch and stripping, of course) it is 5504 bytes in size; our assembler code gives 5360 bytes. Nice, but let’s try to reduce it some more (even if we’ll still keep libc attached). Instead of branching to the functions themselves (“bl function”) we’ll use function pointer offsets and branch to __NewLibCall:

<pre>
.globl main
main:
#printf("aaaa")

lis %r3,.msg@ha # arg1 part1
la %r3,.msg@l(%r3) # arg1 part2
li %r12, 1200 # 1200 - pointer offset to function
b __NewLibCall

#exit(0)

li %r3, 0 # arg1
li %r12, 1620 # 1620 - pointer offset to function
b __NewLibCall

.msg:
.string "aaaa"
</pre>

<pre>
6/0.RAM Disk:> as test.s -o test.o
6/0.RAM Disk:> ld -N -q test.o -o test /SDK/newlib/lib/crtbegin.o /SDK/newlib/lib/LibC.a /SDK/newlib/lib/crtend.o
6/0.RAM Disk:> strip test
6/0.RAM Disk:> filesize format=%s test
5336
6/0.RAM Disk:> test
aaaa
6/0.RAM Disk:>
</pre>

The size is now 5336. We’ve saved 24 bytes, no big deal! Now let’s get real heavy and try to mimic __NewLibCall using our own code:

<pre>
.globl main
main:
lis %r3,.msg@ha #
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

li %r3, 0
li %r12, 1620 # exit

lis %r11,26006
lwz %r0,-25500(%r11)
lwzx %r11,%r12,%r0 # __NewLibCall
mtctr %r11
bctr

.msg:
.string "aaaa"
</pre>

It crashes but why? Because lis %r11,26006 and lwz %r0,-25500(%r11) load a pointer from 0x010100024. In the original __NewLibCall code this is a read access to the NewLib interface pointer. But as we already know, we cannot read from the absolute address 0x01010024 because it’s illegal, and the ELF loader must relocate this address to point to the real NewLib interface pointer (INewlib). We didn’t see that before because we used objdump without the "-r" switch (which shows relocations), so let’s use it now:

<pre>
7/0.RAM Disk:> objdump -dr 1 | grep -A7 "<__NewLibCall>:"
</pre>

<pre>
01000298 <__NewLibCall>:
1000298: 3d 60 01 01 lis r11,257
100029a: R_PPC_ADDR16_HA INewlib
100029c: 80 0b 00 24 lwz r0,36(r11)
100029e: R_PPC_ADDR16_LO INewlib
10002a0: 7d 6c 00 2e lwzx r11,r12,r0
10002a4: 7d 69 03 a6 mtctr r11
10002a8: 4e 80 04 20 bctr
</pre>

So we’ll rewrite our code using the normal interface pointer, and turn the __NewLibCall code into a macro:

<pre>
.macro OUR_NEWLibCALL
lis %r11,INewlib@ha
lwz %r0,INewlib@l(%r11)
lwzx %r11,%r12,%r0
mtctr %r11
bctr
.endm

.globl main
main:
lis %r3,.msg@ha
la %r3,.msg@l(%r3) # printf("aaaa");
li %r12, 1200

OUR_NEWLibCALL

li %r3, 0
li %r12, 1620 # exit(0);

OUR_NEWLibCALL

.msg:
.string "aaaa"
</pre>

Works now! Still, after stripping, the size is 5336 bytes but at least the code is fully in our hands and we can play with instructions. It’s time to read some good stuff like the Green Book (see Links below) if you want to do real beefy hacking.

By the way, when we debug our binary, you’ll notice that GCC has put a strangely-looking instruction right before the call to a libc function: crxor 6,6,6 (crclr 4*cr1+eq). This is done in compliance with the ABI specification, which says that before a variadic function is called, an extra instruction (crxor 6,6,6 or creqv 6,6,6) must be executed that sets Condition Register 6 (CR6) to either 1 or 0. The value depends on whether one or more arguments need to go to a floating-point register. If no arguments are being passed in floating-point registers, crxor 6,6,6 is added in order to set the Condition Register to 0. If you call a variadic function with floating-point arguments, the call will be preceded by a creqv 6,6,6 that sets Condition Register 6 to the value of 1.

You may ask where on Earth we got the numerical values (offsets) for the libc functions, i.e. “1200” representing printf() and “1620” representing exit(). For newlib.library, there is no documentation, header files or an interface description in the official AmigaOS SDK so you have to find it all out yourself. There are a couple of ways to do it:

# Write the program in C and obtain the numbers by disassembling the code (using [[GDB_for_Beginners|GDB]] or objdump). Not much fun but at least you can inspect what arguments are used and in which registers they are stored.
# If you only need the list of function offsets you can disassemble the LibC.a file using objdump:

<pre>
shell:> objdump -dr SDK:newlib/lib/LibC.a
</pre>

The library only contains stub functions, and output will look like the following:

<pre>
---- SNIP ----

Disassembly of section .text:

00000000 <realloc>:
0: 39 80 01 64 li r12,356
4: 48 00 00 00 b 4 <realloc+0x4>
4: R_PPC_REL24 __NewLibCall

stub_realpath.o: file format ELF32-AmigAOS

Disassembly of section .text:

00000000 <realpath>:
0: 39 80 0c 00 li r12,3072
4: 48 00 00 00 b 4 <realpath+0x4>
4: R_PPC_REL24 __NewLibCall

stub_recv.o: file format ELF32-AmigaOS

---- SNIP ----
</pre>

You can write a simple script that will parse the disassembly and give you the list in any form you wish.

== Assembler programming without libc ==

If you want to write programs without using the C standard library, your code should do what runtime objects would normally take care of: that is, initialize all the necessary system-related stuff. It is almost the same as on AmigaOS 3.x, only with some AmigaOS 4.x-specific parts. This is what you should do:

* obtain SysBase (pointer to exec.library)
* obtain the exec.library interface
* IExec->Obtain()
* open dos.library and its interface (if you want to use dos.library functions)
* IExec->GetInterface()
... your code ...
* IExec->DropInterface()
* IExec->CloseLibrary()
* IExec->Release()
* exit(0)

As of now, we can no longer use printf() because it’s a libc function - if we want to produce a really small binary, we cannot afford the luxury of attaching the entire libc to be able to use printf() only! Instead, we need to use the AmigaOS API: in this particular case, the Write() function from dos.library.

There is a Hello World example written by Frank Wille for his assembler 'vasm'; I’ll adapt it for the GNU assembler ('as') in order to make the article related to one compiler. (Both the original and the adapted version can be found in the archive that comes with the article):

<pre>
# ExecBase
.set ExecBase,4
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:

mflr %r0
stwu %r1,-32(%r1)
stmw %r28,8(%r1)
mr %r31,%r0

# get SysBase
li %r11,ExecBase
lwz %r3,0(%r11)

# get Exec-Interface
lwz %r30,MainInterface(%r3) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
lis %r4,dos_name@ha
addi %r4,%r4,dos_name@l
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
lis %r5,main_name@ha
addi %r5,%r5,main_name@l
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
lis %r5,hello_world@ha
addi %r5,%r5,hello_world@l
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
mtlr %r31
lmw %r28,8(%r1)
addi %r1,%r1,32
blr

.rodata

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

If you did assembler programming under AmigaOS 3.x, you can see that the logic is the same, just the assembler is different and some AmigaOS 4.x-specific bits and pieces (the interface-related stuff) have been added. Now let’s compile and link the source and then strip the binary to see how our “slimming diet” works:

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -q hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> filesize format=%s hello
4624
</pre>

Right, so we got down to 4624 bytes. A little better when compared with the libc version (which was 5336 in size), but still too much for a Hello World program.

To obtain the numerical values that identify system functions, you need to study the interface description XML files that are provided in the AmigaOS SDK. For example, for exec.library functions you need to read the file “SDK:include/interfaces/exec.xml”. All interfaces contain a jump table. The offset for the first interface "method" is 60, the next one is 64 and so on. So you just open the appropriate interface description XML file, start counting from 60, and add +4 for any method that follows.

= Hacking it for real =

== Linker scripts (ldscripts) ==

Every time you perform linking to produce an executable, the linker uses a special script called ldscript (pass the “-verbose” argument to see which one is used by default). The script is written in the linker’s command language. The main purpose of the linker script is to describe how the sections in the input file(s) should be mapped into the output file, and to control the memory layout of the output file. Most linker scripts do nothing more that that, but – should you have the need – the script can also direct the linker to perform other operations, using the available set of commands in the command language. To provide your own, custom script to the linker, the "-T" switch is used. (By the way, the "-N" switch, mentioned earlier and used to make non-aligned executables, also affects the choice of the default linker script.)

What does all of that mean for us and how is it related to this article? Well, when you read the ldscripts documentation (see Links below), you can build your own ldscript that will only create the necessary sections. That is: we can produce a minimum working executable and thus get rid of parts that even 'strip' wouldn’t be able to remove.

So following the first-test example from the ldscript documentation, we’ll write our own script now:

<pre>
SECTIONS
{
. = 0x00000000;
.text : { *(.text) }
}
</pre>

But why did we put 0x00000000 here as the entry point of the code? Well as we discussed earlier, the address is just a placeholder so it has no real meaning under AmigaOS (the ELF loader will perform relocation and calculate the proper address). Nevertheless, the address value is used when the ELF binary is created, and it can make a difference as regards the executable size because of paging. So, let’s compile our non-libc assembler code and provide our custom linker script:

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=66713
</pre>

OMG! 66 kilobytes! But that was quite expected, considering the entry point address we have provided. You can now play with the address value to see what difference in the executable size it makes. For example, if you try 0x11111111, the size of the binary is 5120 bytes; 0xAAAAAAAA will result in 44440 bytes. Apparently, this generally meaningless address does make a difference because it affects paging. So all we need to do is choose a value that will, hopefully, avoid any kind of paging. We can consult the ldscripts manual again and we’ll find this:

SIZEOF_HEADERS: Returns the size in bytes of the output file’s headers. You can use this number as the start address of the first section, to facilate paging.

This looks like the thing we need, so:

<pre>
SECTIONS
{
. = SIZEOF_HEADERS;
.text : { *(.text) }
}
</pre>

<pre>
shell:> as hello.s -o hello.o
shell:> ld -Tldscript -q -o hello hello.o
shell:> stat -c=%s hello
=1261

shell:> strip hello
shell:> stat -c=%s hello
=832

shell:> hello
Hello World!
shell:>
</pre>

832 bytes of size and works!

== Getting rid of relocation ==

Now, lets see what kind of sections our 832 bytes binary has:

<pre>
7/0.Work:> readelf -S hello
There are 7 section headers, starting at offset 0x198:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00000054 000054 0000f8 00 AX 0 0 1
[ 2] .rela.text RELA 00000000 0002f8 000048 0c 5 1 4
[ 3] .rodata PROGBITS 0000014c 00014c 00001e 00 A 0 0 1
[ 4] .shstrtab STRTAB 00000000 00016a 00002e 00 0 0 1
[ 5] .symtab SYMTAB 00000000 0002b0 000040 10 6 3 4
[ 6] .strtab STRTAB 00000000 0002f0 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

7/0.Work:>
</pre>

As you can see there are some sections that should be relocated:

# .rela.text - relocations for .text.
# .rodata - data (our strings like "helloworld", "dos.library", etc)

And the next three sections (.shstrtab, .symtab and .strtab) are stanadard in the AmigaOS implementation of ELF, as the AmigaOS ELF loader requires them. Usually the linker ('ld' or 'vlink', does not matter) would remove .symtab and .strtab, when the "-s" option is used at linking stage, but whilst that is true for UNIX, it's not true not for AmigaOS because the AmigaOS ELF loader needs the _start symbol to find the program entry point, so we can't delete those two sections. As for .shstrtab, we can't delete it either because we still need the sections (we will discuss why later).

So what about .rela.text and .rodata? Well, they can be removed by modifing our code a bit, to avoid any relocations (thanks to Frank again). We place the data to the .text section, together with the code. So the distance between the strings and the code is constant (kind of like base-relative addressing). With "bl initbase" we jump to the following instruction while the CPU places the address of this instruction into LR. This is the base address which we can use:

<pre>
# non-relocated Hello World
# by Frank Wille, janury 2012
# adapted for "as" by kas1e

# ExecBase
.set MainInterface,632

# Exec Interface
.set Obtain,60
.set Release,64
.set OpenLibrary,424
.set CloseLibrary,428
.set GetInterface,448
.set DropInterface,456

# DOS Interface
.set Write,88
.set Output,96

.macro CALLOS reg,val # Interface register, function offset
lwz %r0,\val(\reg)
mr %r3,\reg
mtctr %r0
bctrl
.endm

.text

.global _start
_start:
mflr %r0
stw %r0,4(%r1)
stwu %r1,-32(%r1)
stmw %r28,8(%r1)

# initialize data pointer
bl initbase
initbase:
mflr %r31 # r31 initbase

# get Exec-Interface
lwz %r30,MainInterface(%r5) # r30 IExec

# IExec->Obtain()
CALLOS %r30,Obtain

# open dos.library and get DOS-Interface
# IExec->OpenLibrary("dos.library",50)
addi %r4,%r31,dos_name-initbase
li %r5,50
CALLOS %r30,OpenLibrary
mr. %r28,%r3 # r28 DOSBase
beq release_exec

# IExec->GetInterface(DOSBase,"main",1,0)
mr %r4,%r28
addi %r5,%r31,main_name-initbase
li %r6,1
li %r7,0
CALLOS %r30,GetInterface
mr. %r29,%r3 # r29 IDOS
beq close_dos

# IDOS->Output()
CALLOS %r29,Output

# IDOS->Write(stdout,"Hello World!\n",13)
mr %r4,%r3
addi %r5,%r31,hello_world-initbase
li %r6,hello_world_end-hello_world
CALLOS %r29,Write

# IExec->DropInterface(IDOS)
mr %r4,%r29
CALLOS %r30,DropInterface

close_dos:
# IExec->CloseLibrary(DOSBase)
mr %r4,%r28
CALLOS %r30,CloseLibrary

release_exec:
# IExec->Release()
CALLOS %r30,Release

# exit(0)
li %r3,0
lmw %r28,8(%r1)
addi %r1,%r1,32
lwz %r0,4(%r1)
mtlr %r0
blr

dos_name:
.string "dos.library"
main_name:
.string "main"
hello_world:
.string "Hello World!"
hello_world_end:
</pre>

<pre>
6/0.Work:> as hello.s -o hello.o
6/0.Work:> ld -Tldscript hello.o -o hello
6/0.Work:> strip hello
6/0.Work:> stat -c=%s hello
=644

6/0.Work:> hello
Hello World!
6/0.Work:>
</pre>

644 bytes of size, and still works. If we check the sections in the binary now, we'll see that currently it only contains the .text section and the three symbol-related sections that are required in AmigaOS binaries:

<pre>
6/0.Work:> readelf -S hello
There are 5 section headers, starting at offset 0x184:

Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 10000054 000054 00010e 00 AX 0 0 1
[ 2] .shstrtab STRTAB 00000000 000162 000021 00 0 0 1
[ 3] .symtab SYMTAB 00000000 00024c 000030 10 4 2 4
[ 4] .strtab STRTAB 00000000 00027c 000008 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings)
I (info), L (link order), G (group), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)

6/0.Work:>
</pre>

== The ELF loader ==

If you want to understand the internals of the ELF format, the best book of reference is the ELF specification (see Links), where you can find everything about headers, sections, segments, section headers and so on. But of course it is only a specification and so it does not cover ELF loaders and parsers, which are implemented differenty on different operating systems. While the implementation does not vary too much among UNIXes, the ELF loader in AmigaOS is rather specific.

Let's briefly cover the parts an ELF executable contains:

* ELF Header
* Program (segments) header table
* Segments
* Sections header table
* optional sections (certain sections can sometimes come before the sections header table, like for example .shstrtab)

Although it may seem that sections and segments are the same thing, this is not the case. Sections are elements of the ELF file. When you load the file into memory, sections are joined to form segments. Segments are file elements too but they are loaded to memory and can be directly handled by the loader. So you can think of sections as segments, just you should know that segments are something that executes in memory, while sections is the material from which segments are built in memory.

This is what our 644-byte Hello World example looks like, with the various parts defined by the ELF specification highlighted in different colours:

[[File:HackingWayPart1-3.png|center]]

Every part of an ELF file (be it the ELF header, segments header, or any other part) has a different structure, described in depth in the ELF specification. For a better understanding, let‘s describe the ELF header (the first part in the image above, highlighted in dark green):

<pre>
db 0x7f, "ELF" ; magic
db 1,2,1 ; 32 bits, big endian, version 1
db 0,0,0,0,0,0,0,0,0 ; os info

db 0,2 ; e_type (for executable=2)
db 0,0x14 ; 14h = powerpc.
db 0,0,0,1 ; version (always must be set to 1)
dd 0x10000054 ; entry point (on AmigaOS it makes no sense)
dd 0x00000034 ; program header table file offset in bytes
dd 0x00000184 ; section header table file offset in bytes
db 0,0,0,0 ; e_flag - processor specific flags
db 0,0x34 ; e_ehsize - size of ELF header in bytes

db 0,0x20 ; e_phentsize - size of one entry in bytes, of program header table (all the entries are the same size)
db 0,2 ; e_phnum - number of entires in the program header table.

db 0,0x28 ; e_shentsize - section headers size in bytes
db 0,5 ; e_shnum - number of entires in the section header table
db 0,2 ; e_eshstrndx - section header table index of the entry assosiated with the section name string table
</pre>

When you try to execute a program, the ELF loader first checks if it's a genuine ELF binary or not. Depending on the result, the loading of the executable is either allowed or denied. Once loaded in memory, code from the respective segments is executed. As I said before, the necessary fields are parsed differently on different operating systems. For example under Linux, the loader parses the ELF structure going into greater depth compared to the AmigaOS loader. Still there is some common ground; on both OSes you can, for instance, write anything you want to the "os info" field. On AmigaOS you can fully reuse more fields, and here is how the AmigaOS ELF loader parses the ELF headers:

<pre>
* magic (first 7 bytes): db 0x7f,"ELF", 0x01,0x02,0x01 (100% required)
* all the subsequent fields are not parsed at all and can contain any data, until the loader reaches the section header tables' file offset in bytes field (required)
* then again there can be any data, until e_phnum (the number of entires in the program header table, which is required as well)
* and then the next 8 bytes of info (4 fields) about section headers/sections are required
</pre>

Take a look at the image below, which shows an ELF header in which all unparsed bytes are marked by "A" letters. You can use these bytes for anything you want.

[[File:HackingWayPart1-4.png|center]]

But please bear in mind that doing so would breach the specification. The fact that it works now doesn't mean it will also work with the next version of the ELF loader, as the AmigaOS developers could use the currently unparsed fields for something meaningful in the future.

The ELF header is not the only place where you can insert (at least with the current version of the loader) your own data. After the ELF header there come program headers (i.e. headers that describe segments). In our particular case we have one program section header for the .text segment. And here comes the suprise: the AmigaOS ELF loader does not parse the program headers at all! Instead, the parsing is done in sections and section headers only. Apparently, the AmigaOS loader does something that on UNIXes is normally put in the ELF executable and the loader just gets data from it. But under AmigaOS this is not the case. Although the ELF binary produced by GCC is built correctly and according to specification, half of the sections and many fields are not used under AmigaOS.

So the programs section headers can fully be used for your own needs. We can remove section names completely (and give them, for example, an "empty" name by writing 0 string-offset in the sh_name field of each section header entry). But .shstrtab must still be kept, with a size of 1 byte. A NULL section header can be reused too (you can see that a NULL section header comes after the .shrstab section, so we have plenty of space). Check the file "bonus/unused_fields/hello" to see which areas can be reused (these are indicated by 0xAA bytes).

Now it‘s clear that we can manipulate sections (i.e. delete empty ones and those ignored by the ELF loader) and recalculate all the addresses in the necessary fields. To do that you will really need to dig into the ELF specification. For example, you can put the _start label to any suitable place (such as the ELF header, or right at the begining of an ignored field) and then just put the adjusted address in the .strtab section offset field. This way you can save 8 bytes, so the size of our binary is now 636 bytes. Then there is the .symtab section at the end of the file, which is 48 bytes long. We can put it right in the place of .shstrtab (34 bytes in our case) and in the following part of the NULL section header (so as to squeeze the remaining 14 bytes in). Just like this:

[[File:HackingWayPart1-5.png|center]]

As a result, the size of our binary becomes mere 588 bytes, and the executable still works of course. Tools like 'readelf' will surely be puzzled by such custom-hacked ELF files, but we only need to worry about what the ELF loader thinks about them. If the loader is happy, the binary is working and the code is executed in memory.

In the bonus directory that comes with this article, you can try out an example binary the altered structure of which is depicted by the image above. In the binary, .strtab (the _start symbol) is moved to the program section header, and .symtab is moved on top of .shstrtab + the NULL section header (see directory "bonus/shift_sections").

= Final Words =

The article, of course, aims at encouraging learning. If you are an application programmer, you'll probably never need to use assembler directly or construct ELFs from scratch byte per byte. But the knowledge of how things work at low level can help you understand and resolve many problems that may turn up from time to time and that are related to compilers, linkers and assembler-code parts. Also, it can give you a better overview of the AmigaOS internals so when you start a project, it will be much easier for you to get rid of problems: without asking questions in the forums and losing hours fiddling with the basics.

= Links =

[http://flint.cs.yale.edu/cs422/doc/ELF_Format.pdf ELF specification]<br/>
[http://refspecs.linuxbase.org/ELF/ELFspec_ppc.pdf PPC SYSV4 ABI]<br/>
[http://www.freescale.com/files/product/doc/MPCFPE32B.pdf Green Book (MPCFPE32B)]<br/>
[https://www.gnu.org/software/gdb/documentation/ GDB]<br/>
[http://sourceware.org/binutils/docs/ld/Scripts.html#Scripts Linker Scripts] or SDK:Documentation/CompilerTools/ld.pdf , chapter 3.0 "Linker Scripts"

Advanced Serial Debugging Guide

2020-12-26T18:52:44Z

Roman Kargin: /* Firmware method */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline "serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

On X5000 it's the same U-Boot as with Sam/AmigaOne, so as example set serial, munge and debuglevel to 5:

setenv os4_commandline serial munge debuglevel=5
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debuglevel 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

To note ,that in firmware method we use '=' for debuglevel, i.e. 'debuglevel=5'. But when we use 'kdebug", then instead of '=' we put space, i.e. 'debuglevel 5'.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2019-09-26T15:28:18Z

Roman Kargin: /* KDebug method */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline "serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

On X5000 it's the same U-Boot as with Sam/AmigaOne, so as example set serial, munge and debuglevel to 5:

setenv os4_commandline "serial munge debuglevel=5"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debuglevel 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

To note ,that in firmware method we use '=' for debuglevel, i.e. 'debuglevel=5'. But when we use 'kdebug", then instead of '=' we put space, i.e. 'debuglevel 5'.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2019-09-26T15:16:12Z

Roman Kargin: /* KDebug method */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline "serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

On X5000 it's the same U-Boot as with Sam/AmigaOne, so as example set serial, munge and debuglevel to 5:

setenv os4_commandline "serial munge debuglevel=5"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debuglevel 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

{{Note|As can be seen, in firmware method we use "=" for debuglevel, i.e. "debuglevel=5". But when we use "munge", then instead of "=" we use space, i.e. "debuglevel 5".}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2019-09-26T15:13:57Z

Roman Kargin: /* Firmware method */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline "serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

On X5000 it's the same U-Boot as with Sam/AmigaOne, so as example set serial, munge and debuglevel to 5:

setenv os4_commandline "serial munge debuglevel=5"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debug level 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2019-09-26T08:23:33Z

Roman Kargin: /* Firmware method */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline "serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

On X5000 U-Boot same with serial, munge and debuglevel to 5:

setenv os4_commandline "serial munge debuglevel 5"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debug level 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2015-04-06T09:09:34Z

Roman Kargin: /* Introduction */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. Because of that it should come as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline="serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debug level 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2015-04-06T09:08:27Z

Roman Kargin: /* Introduction */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. It comes as no surprise that the old AmigaOS 3 which works on classic Amigas throw all the debug information via serial port as well. Today, we have different kinds of hardware, where for example USB is pervasive, but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline="serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debug level 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

Advanced Serial Debugging Guide

2015-04-06T09:04:41Z

Roman Kargin: /* Introduction */

= Author =

Roman Kargin and Lyle Hazelwood<br/>
Copyright © 2013 Roman Kargin and Lyle Hazelwood<br/>
Proofreading and grammar corrections by the AmigaOS Wiki team.<br/>

= Introduction =

While AmigaOS has "hi-level" capabilities to help debug problems (e.g. redirecting debug outputs to memory and displaying the log with DumpDebugBuffer or using serial redirection tools like Sashimi) it is sometimes still not enough. For example, there is no protection for areas like input and Intuition so we can get into a situation where the GUI is not operational but the OS is still working. Intuition, input and any other necessary components can crash and then the GrimReaper utility has no way to spawn itself. Meanwhile, the kernel, still can output debug information about the crash on serial.

To begin with, you should know that AmigaOS has 2 ''reapers''. One is a user friendly utility from SYS:System called GrimReaper. The GrimReaper can be called a "hi-level" one. Another ''reaper'' is called Reaper which is a part of the kernel itself. The kernel's Reaper actually launches the user-friendly GrimReaper when a required. So, you can imagine that when things get nasty, the kernel's reaper not all the time can spawn a GrimReaper, but still can catch some information and output it to the serial. Visually, when such a nasty things happens and input and/or intuition die, you just see a "freeze" of the OS and may think that everything is freezes, while it not: you just can't operate with OS anymore as necessary parts of OS crashes, but OS itself still working and kernel's reaper already throw all the info into the serial.

That means that when you want to do some serious work and you need any way to catch your bugs and/or debug any of your problematic code, then serial debugging is the best and only one way. Of course it's still possible to hope that the system friendly GrimReaper will popup for you and that there will be nothing so hardcore which it will crash Intuition and "freeze" the OS but that's just not enough when it come to doing real work.

You may ask now why the serial port is used at all? Why not parallel or something else? Why does everyone keep telling you to use serial debugging as if there is nothing else you can connect a cable to? The answer is in simplicity and reliability. Serial was chosen because it is simple to transfer data, bit by bit, not like for example done with a parallel port. There are different interfaces which also can transfer information bit by bit (like Ethernet, Firewire and USB) but the standard serial port is common to all platforms and far less complex. It comes as no surprise that the old AmigaOS 3 which works on classic Amigas, all output debug information via serial ports. Now, we have different kinds of hardware where USB is pervasive but USB is more complex and still not as reliable for debugging purposes.

= Serial and Null Modem =

== Serial ==

Serial ports (often called RS-232 but it's really RS-232C) have disappeared as a standard option on many user level computers and have been replaced by USB. You can still find many business level x86 motherboards with serial ports. There are solutions for PC hardware without a serial port like USB to serial converters but we will talk about that later. Right now, all we need to know is that all our PowerPC based Amigas, classic Amigas and even the AmigaOne-X1000 have a serial port and the AmigaOS kernel's reaper outputs debug information over that serial port.

Serial ports most often come in two pin outs: 9 pins and 25 pins.

[[File:02-serial_db9.png|frame|center|9 Pin Connector]]

[[File:01-serial_db25.png|frame|center|25 Pin Connector]]

The 25 pin one, which is usually called DB25, was the first one and is used on the classic Amigas. After some time, software was simplified to use only half of the pins and then 9 pin connector (usually called DB9 but in reality it is DE9) was created. While classic Amigas have the older serial connectors with 25 pins, the Pegasos2, X1000 and Sams all have 9 pin connectors. In general, it makes no differences for our purposes. This is just information in case you decide to build your cable from scratch.

While the information given there is enough for our article, you still can check out Wikipedia for more information about serial ports and the RS232C standard itself at http://en.wikipedia.org/wiki/Serial_port.

== Null Modem ==

In the good old days, when Modems/Teletypes were used to connect over the serial port, it was expected that one side is the ''Data Terminal Equipment'' (DTE) (usually a computer/terminal) and the other side is ''Data Circuit-terminating Equipment'' (DCE) (usually a modem). One side which is an output on a DTE is an input on a DCE and vice versa. Because of that DCE can be connected to a DTE with a straight wired cable.

Today things are different. Terminals and modems have mostly disappeared and computers have the same kind of serial port that terminals used to have: DTE. As a result, we can't connect them with a straight, pin to pin cable because it will connect the data transfer pins (TX to TX and RX to RX) like it was when we connect a serial port with a modem and that will not work. The solution is to swap the data transfer wires: TX to RX. That's what is simply called a ''Null Modem Cable'' (though without ground connected which is always better to do).

Usually when you go into a shop and find a modem cable it is most often not a null modem cable. You need to ask for a null modem cable where the data pins have been crossed over. Visually, the cables look exactly the same. Only the wires inside are not cross over and thus it will not transfer any serial debug output.

A null modem cable can one of 4 types:

#No hardware handshaking<br/>This cable has only the data and signal ground wires connected. I.e. RX, TX and ground.
#Loop back handshaking<br/>Just some "cheat" cable to create fake hardware flow control.
#Partial handshaking<br/>Advanced "cheat" cable.
#Full handshaking<br/>No cheats, just everything that can be in null modem cable.

In general, to create a basic null modem cable (that one, with "no hardware handshaking") it is enough to cross over pins 2 and 3 (RxD with TxD) and connect the ground pin. Even with such a simple null modem cable of just 3 wires, you will be able to output debug information and everything will work. You can of course build a cable with full handshaking and it will also work but the hardware may or may support it. For example, the X1000 does not support hardware handshaking and even if you have such a cable, the terminal software where you will capture your debug output you still need to choose the "no handshaking" option. Classic Amiga serial ports support hardware h handshaking. For example, AmigaExplorer from AmigaForever transfers data between Amiga and Windows computers via serial and handshaking helps in that case. So it's better to have a full-handshaking cable and to test it with your hardware. But if you need a null modem cable just for capturing debug output, a simple 3 wire null modem cable will allow up to 115200 and it's more than enough.

{{Note|text=If you have 2 computers which you want to connect via serial ports you need a null modem cable and '''not''' a ''straight'' serial cable.}}

Since serial ports can be only of 2 types (DB25 and DB9) there are 3 possible pin outs for null modem cables with full handshaking: DB25 to DB25, DB9 to DB9 and DB9 to DB25.

[[File:03-db25-db25.png|center|800px]]
[[File:05-db9-db9.png|center|800px]]
[[File:04-db25-db9.png|center|800px]]

DB9 to DB9 serial cables are fairly easy to find in stores today but they are not likely to be null modem cables and you will need to work on them. DB25 are more rare and you may need to build a cable yourself. Just buy connectors and connect the necessary wires as shown above.

You can also build a cable which has 2 connectors on either side (i.e. DB9 and DB25 on one side and DB9 and DB25 on the other) so you can use either connector type as needed. This can be handy when you have a classic Amiga and modern PowerPC hardware and connected to a PC for example. To build such a cable just follow the same pin logic as shown below and just double them where needed.

Another way to solve the problem would be to buy a straight serial cable and add a small connector-adapter which will change the wiring from straight to null modem like in the screenshot below:

[[File:06-null_modem_adapter.png|frame|center]]

You can find even more information about null modem cables on [http://en.wikipedia.org/wiki/Null_modem Wikipedia].

= Connect them now! =

== Serial<->Serial ==

There is nothing else you need except a working null modem cable. It must be a real null modem cable or a ''straight'' serial cable with a null modem adapter; doesn't matter which one.

== Serial<->USB ==

This section is important and interesting for anyone who works with AmigaOS. The reason is that it is more difficult to find x86 hardware (which most of us have use as a necessary part of life) with built-in serial ports. It is even more difficult to find a notebook with a serial port. The solution is to use commonly available USB to serial adapters. There are plenty of them from all sorts of different companies. Despite the fact that they are common, some adapters can be better than others. Although most will work without problems and in practice all of the tested ones works fine, you still may found some adapters which can give you some problems. The reason is that USB does not directly translate to serial and data must be buffered between the two while at the same time keeping accurate timing. There is software inside the adapters and more than one way to implement the necessary hardware buffering thus not all adapters are equal and some are better than others. For serious serial work, nothing beats an actual UART which is why they are still an available option on industrial, laboratory and business PCs.

If you in interested in building a Serial to USB adapter yourself, you can find plans on the Internet. One example is this [http://pinouts.ru/Converters/usb_serial_adapter_pinout.shtml USB to serial adapter].

[[File:07-usb2serial.png|frame|center]]

The look of the adapter can be different of course since it depends on what connectors are used when building a cable. I use one from Gembird based on the PS2303 chip on my notebook with Windows XP, where after installing of drivers it shows ups as COM17:

[[File:08-settings_win_show.png|frame|center]]

After you have the USB to serial adapter connected in your system, everything will be the same as if you connected Serial with Serial. A null modem cable is still required or a "straight" cable with a null modem adapter.

== Serial<->Anything ==

You can connect serial to any port in a computer. You just need the correct adapter and cable. There are serial to USB (either purchased or made yourself), serial to Firewire adapters, serial to Ethernet, etc.

What I am trying to point here is that it does not matter how but you still need to make a real serial port in your system. USB to serial adapters are most popular because they are readily available, easy to find, inexpensive and well tested.

= Terminal Software and port settings =

Now, after you have your cable connected between 2 machines, you need software which will catch the data from one side. The "receiver" side can be an x86 system with Windows, Linux, Mac OS or whatever else. You could also use an Amiga with AmigaOS, pOS, Unix or whatever else runs on classic Amiga hardware or any kind of another hardware. The software is called "Terminal" software and since many of us use x86 systems to capture debug output from our Amigas, we will start with x86.

One of the best terminal software packages for x86 is PuTTY. You can get PuTTY from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. Be sure that you get latest/full version because there are some stripped versions around with no support of serial connections. The normal version of PuTTY with working serial support will looks like this:

[[File:09-putty_serial.png|frame|center]]

On Amiga (if you connect 2 Amigas by serial) you can use the old 68k Term 4.8:

[[File:10-term48_serial.png|frame|center]]

On Linux you can also use PuTTY or the well-known minicom.

Whatever software is used, it is all about getting the debug data from your Amiga. Debug output is ASCII text data, so anything which can just grab information from your serial port and redirect it where you need it (console or file) will work. You can even write your own simple code which will receive data without the need for any terminal programs. But that's in theory, in practice it is better to use something which is already done.

The settings which you need to use and which always works on any Amiga hardware from which you want to get your serial output are:

baud: 115200
data bits: 8
stop bits: 1
parity: none
flow control: none

It may work with other settings (like different parity and flow control), but as it was pointed out before, on some hardware it may not work at all, on another it will not work as intended, so just these setting to do your first test. After all, you can always change your settings later and experiment after everything works.

= How to redirect debug output to serial =

By default, AmigaOS debug output is sent to a memory debug buffer and not to serial. This is done for simplicity. If something happens, especially bugs, then just type "DumpDebugBuffer" in a shell and it shows you what the debug output was. This is similar to using the Sashimi tool which redirects serial output to the console.

For developers, it is usually best to output everything to the serial port so you need to switch the output to serial. There are 2 ways to do this: boot option in the firmware (U-Boot for Sams/Eyetech AmigaOnes, OpenFirmware for Pegasos2 and CFE for X1000) and the "KDebug" utility.

== Firmware method ==

Given there are so many different hardware platforms on which to run AmigaOS today, it should come as no surprise that they use different boot firmwares. AmigaOne-XE/MicroA1-C/Sam uses U-Boot, Pegasos2 uses OpenFirmware and the AmigaOne X1000 uses CFE. For all of them the boot menu options can be a bit different and you need to refer to the documentation which comes with the hardware. But in all cases you have a common variable called '''os4_commandline''' which you need to set. The name of that variable is the same for all the hardware platforms which support AmigaOS. Just the way the variable is setup varies.

For example on Pegasos2 OpenFirmware, to redirect output to serial with the "munge" debug kernel option and debug level 7 in OF you do:

Pegasos BIOS Extensions Copyright 2001-2004 by bplan GmbH.
All rights Reserved.
Auto-boot in 1 seconds - press ESC to abort, ENTER to boot: aborted
ok#
ok# setenv os4_commandline serial munge debuglevel=7

and then boot AmigaOS.

On AmigaOne X1000/CFE set munge, debuglevel to 4 and output to serial:

setenv -p os4_commandline="serial munge debuglevel=4"

and then boot AmigaOS. For CFE if you are not using -p, then the variable is only saved for that session, not permanently.

On Sam/AmigaOne U-Boot with the same munge, debuglevel=7 and serial redirect:

setenv os4_commandline "serial munge debuglevel=7"
saveenv

In other words, use SetEnv to set the os4_commandline variable and then boot the OS. You can also verify what the variable is using the "printenv" command.

Just so you are not confused, "amigaboot.of" (the code which first boots) is the component which parses the os4_commandline '''firmware variable''' and ''didn't'' parse any command line arguments. It only parses firmware variables and then passes them on to the kernel. If you try something like ''amigaboot.of os4_commandline = "serial"'', it will not work and will be ignored, the same as any other extra parameters you provide. So be sure that you use "setenv" and check via "printenv" that you set the variable correctly.

== KDebug method ==

KDebug is command line utility found in SYS:C which communicates with the AmigaOS kernel. With this utility you can redirect your output to serial or memory. It will override whatever you do with the os4_commandline firmware variable. This is very handy for setting different debug levels at the time you need it. For example, you may also put a command in S:User-Startup
run >NIL: kdebug "debug level 5"
run >NIL: kdebug "console serial"

This will set the debug level to 5 and output via the serial port.

{{Note|Quotation marks are mandatory when using the KDebug tool. What KDebug does is send whatever string you enter directly to the AmigaOS kernel for interpretation. The KDebug command itself does not parse the string.}}

{{Note|KDebug method have higher priority. I.e. if you set your environment variable to some values, and then after os is boots use KDebug to change the settings, then they ''will'' be taken in account and previous one will be forgotten. The only thing that you can't enable/disable by KDebug is 'munge'.}}

= Getting the Most out of Serial Debugging =

To start with, you need to read the following articles:
# [[Debug_Kernel|The Debug Kernel]]
# [[Exec_Debug|Exec Debug]]

To get the most out of serial debugging, developers should use what is called the debug kernel (named '''kernel.debug'''). The debug kernel provides you with more functionality compared with the release kernel at the cost of some execution speed. One thing provided is the ability to play with debug levels where you can choose how much "system" debug output you want to see. It's pretty easy to miss the fact that debug levels only work on debug kernels. You can play with levels, see no change, and you then wonder "Why doesn't it work?". It works, it is just that the kernel should be the debug one.

And while the debug kernel does give you some debug functionality, it still may not cover everything. For example, tools like the old MungWall tool can do a bit more: debug kernel has only the "munge" feature but not the "wall" feature. Another example is MemGuard (which can be found on [http://www.os4depot.net OS4Depot]) adds additional debug capabilities on top of what the debug kernel provides and catches more difficult to find bugs.

So, to get most of serial debugging, you need to use a combination: debug kernel with "munge" option enabled, MemGuard running in the background and set debug level to the value you find most informative. Usually 5 or 7 is enough but sometimes, when a bug is behaving very strangely, you can raise the value in the hope that you see something interesting.

= If nothing works =

Now on to troubleshooting. If you build/buy a cable, adapters and all the stuff, connect everything, fire up the terminal software, set up everything correctly and still nothing works, then:

# Check the cable
# Check the ports.
# Check the cable again :)
# Check program's settings and settings of the port if it USB to serial one.

Be sure that for you have right settings and in the terminal software (especially the baud, flow control and parity).

If you have a few '''spare''' connectors that you can use for troubleshooting, just solder a jumper between TX and RX (pins 2 and 3 on DB9).
Then open a terminal program on the computer, set it up as described above, and turn "Local Echo" OFF. When Local Echo is OFF it does
NOT show what you are typing, it only shows what comes in over the serial port.

The "loopback" connector has TX and RX connected, so it will loop whatever you send back to you. Plug a female loopback directly into the serial port connector and you should see whatever you type, because it is being "looped back" through your test plug. Now take the loopback off, connect your cable, and attach a male loopback at the other end of the cable. If you can see your typing, then you are successfully going out the serial port, down your cable, then looping back up the cable and back into your port.. You get the idea. Having one male and one female "loopback" will let you test ALMOST every possible cable, connection, and software configuration.

Desperate times: If you don't have any "spare" connectors or a soldering iron, a small paper clip can be used instead of a male loopback connector.. Just bend the clip and insert the ends into pins 2 and 3 of the cable end, and connect the other end to the serial port being tested. If the clip does not fit into the cable end, don't force it, find a smaller paper clip.

The one thing that a loopback can NOT test is whether your cable is a "Null Modem" cable or not.
The best way to check a cable is with a multimeter. Set it so that when you touch wires on different sides you will heard a "beep" if it connected. That setting usually looks like a little speaker. Check that all of the connections are as they should be. If you use null modem adapter on top of "straight" serial cable, then check it anyway, just in case, with the same meter. The meter lead probably won't fit inside the female connector pins. Another small paper clip helps with that.

To work properly, (describe a null modem cable, assuming 9 pin to 9 pin):
* Pin 2 on end A should connect to pin 3 on end B.
* Pin 3 on end A should connect to pin 2 on end B.
* Pin 5 on end A should connect to pin 5 on end B.

If these three connections work, it should be enough to get you going.

= Final words =

While this topic is more or less an easy one, there wasn't a guide, until now, which describes everything from the beginning. What kind of cable you can use, how to build it, how to use USB to serial adapters, what settings should be used and what to do with it all. We hope that this guide will at least cover all the gaps and any developers told to "use serial debugging" will just read that article and will have no questions. Anyway, if you have any more info to add, or just found a wrong part somewhere, just send an email to kas1e@yandex.ru or use the [http://www.amigaos.net/Contact AmigaOS Contact Form] and we can update article.

= Links =

[1.] http://en.wikipedia.org/wiki/Serial_port
[2.] http://en.wikipedia.org/wiki/Null_modem
[3.] [[Debug_Kernel|Debug Kernel]]
[4.] [[Exec_Debug|Exec Debug]]

AmiDock and Dockies

2014-05-01T09:27:42Z

Roman Kargin: /* Introduction */

= Introduction =

AmiDock is a tool to maintain a graphical menu bars (Docks) at the Workbench screen that can be used or to execute other programs from a selection of icons (either in the main Dock or in sub-Docks as specified by the AmiDock preferences), or to handle special kind of application called "Dockies", which can provide different functionality: render different kind of data right into the dock, provide user with a menu related to the Docky of choice and so on.

AmiDock implemented as commodity, and together with being responsible for creating and controlling Docks and Dockies it provide different functionality such as ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is a Dock and a Docky? =

== Dock ==

A Dock is an area or window where the user can put a program icon or an interactive program.

[[File:Laucher_dock.png|center]]

== Docky ==

A Docky is a little program to be placed in a dock. Dockies are able to control most of AmiDock's features and provides a great way to expand AmiDock beyond its default functionality. Dockies may be invisible to the user or show static or dynamic (animated) content. You may change their behavior, size and look according to requirements. Dockies are one of two main types:

=== Standalone Dockies ===

A standalone Docky is a special type of program which is made to show an icon in a Dock, delivering some functionality to the user. It is the most common type of Docky. This Docky type uses the standard shared library feature of AmigaOS as a common interface.

=== Application Dockies ===

The second type of Docky is an application Docky which may sometimes be known as an 'AppDocky' or 'AppDockIcon Docky'. An AppDocky is a Docky which is introduced to the system at run time by an application using the application.library registration mechanism for applications. The biggest difference between the two types of Dockies types is that AppDockies belong to a running application and usually are used to represent the current state of the owning application.

To sum up in "standalone dockies" the docky itself is the application (the only reason of existence of the application is the docky) while for "AppDockIcon dockies" the docky is just a graphical representation of a bigger application (the docky is just here to add user friendliness or visual feedback of the surrounding application)

= How to create and manipulate a simple docky =

You write a docky just like you'd write a standard Exec library. In your code you must implement certain specifically-named functions like DockyGet(), DockyProcess(), DockySet() etc. The docky manager (AmiDock) calls these functions when it needs to.

See the [http://os4depot.net/share/utility/docky/datetime_docky.lha datetime.docky source code] to get started.

= FAQ =

== Is it possible to create an application docky and then update its content on the fly and how? ==

Set the icon type to APPICONT_Docky; but you'll need to create a real docky as well then. If it doesn't need to be truly active it is possible just to change the icon's imagery (I think you have to un-register/re-register to get it to change though, so no good for frequent changes)

== I have successfully drawn a datatype (a picture) in a Docky and the picture file has more than 256 colors but it is rendered in 256 colors only. Why? ==

Just add PDTA_DestMode, PMODE_V43 to the NewDTObject call. You might also want to add DTA_GroupID, GID_PICTURE in order to restrict the file type to pictures.

== How are context menus created? ==

Context menus are dynamically built in response to DOCKYGET_ContextMenu attribute:

<syntaxhighlight>
BOOL DockyGet (struct DockyIFace *Self, uint32 msgType, uint32 *msgData)
{
switch (msgType)
{
/* ... */

case DOCKYGET_ContextMenu:
{
Object *contextMenu = (Object *)msgData;

Object *item1 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_PREFS),
PMIA_ID, PMID_PREFS,
PopupMenuItemEnd;

Object *item2 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_SAVEASDEFAULT),
PMIA_ID, PMID_SAVEASDEFAULT,
PopupMenuItemEnd;

Object *item3 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_USEASDEFAULT),
PMIA_ID, PMID_USEASDEFAULT,
PopupMenuItemEnd;

if (item1 && item2 && item3)
{
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item1);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item2);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item3);
}

}
break;

/* ... */
}
</syntaxhighlight>

== How do you use an alpha layer for a Docky? ==

On non-composited screens, AmiDock uses a "fake" transparency effect (i.e. the bitmap is filled with the contents of the window behind the dock), meaning that you can only add things over the background but you cannot render half or totally transparent contents without doing the blending at the same time (or you'll lose background information).

So if you want to support this configuration you need to follow this rule or provide an alternate rendering if you want to do fancy stuff when compositing is enabled.

Now, what you are trying to do should work on composited screens, but you have to tell AmiDock that you are using composited mode (see DOCKYGET_SupportsComposite and DOCKYGET_CompositeMode) so that it does not try to de-multiply the docky bitmap. Also I'm not sure you can use legacy pens to fill alpha channel, you're better off with direct ARGB painting (see SetRPAttrs() with RPTAG_APenColor).

== Is there a way to trigger a redraw from outside the docky? ==

Use DOCKYGET_NeedsAttention for this. This is how a docky can be notified from an external task by signaling AmiDock process. Quite straightforward, this is how e.g. [http://os4depot.net/share/utility/docky/winbar-docky.lha winbar.docky] ([http://os4depot.net/share/utility/docky/winbar_docky_src.zip source code is here], prefsobjects_macro.h include file which is required [http://openamiga.org/attachment/project/29/prefsobjects_macros.h here]) is told to rethink its layout in response to a change in preferences, or Intuition windows list. A shared structure and [[Exec_Mutexes|Mutex]] to protect the data. By the way, winbar.docky source code can be helpful not only for that, but for all other things such as adding context menus, firing requests to AmiDock or other of the many still undocumented features of the API.

You are notified about the task/bit to signal through DOCKYSET_DockyAttention (you would store it in the library base for an easy access by the main program). When your program signals AmiDock, every docky is sent the DOCKYGET_NeedsAttention message, so it is a good idea to use a flag to know if the notification really comes from your program.

== How does DOCKYGET_SupportsComposite work? ==

When compositing is enabled, a docky receives a DOCKYGET_SupportsComposite query. If it returns TRUE, it will then receive a DOCKYGET_CompositeMode query to which it can reply either DOCKYCOMPMODE_PreblendedRGB (0) or DOCKYCOMPMODE_RawRGB (1). The former has no advantages over the previous method, but was included for backward compatibility. The latter, however, allows you to directly *copy* the source ARGB data to the destination bitmap (or buffer) rather than blend it yourself. This way, it will be AmiDock itself that does the blending (once) without the need to undo the
previous blending to get the raw RGB data.

One thing to take into account when using this method, though, is that on a 16-bit screen you will get an actual 16-bit destination bitmap, not 32-bit as it happens when returning FALSE to DOCKYGET_SupportsComposite. In this case, you will need to write the alpha channel separately to the
(8-bit) bitmap provided in the DockyRenderDestination's alpha.RP field.

= See Also =

1. SDK:Documentation/AutoDocs/docky.doc

2. SYS:Documentation/Commodities/AmiDock_Arexx.doc

3. SDK:Examples/AmiDock/

AmiDock and Dockies

2014-05-01T05:20:00Z

Roman Kargin: /* What is AmiDock */

= Introduction =

AmiDock is a tool to maintain a graphical menu bars (Docks) at the Workbench screen that can be used or to execute other programs from a selection of icons (either in the main Dock or in sub-Docks as specified by the AmiDock preferences), or to handle special kind of application called "Dockies", which can provide different functionality: render different kind of data right into the dock, provide user with a menu related to the docky of choice and so on.

Amidock implemented as commodity, and together with being responsible for creating and controlling docks and dockies it provide different functionality such as ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is a Dock and a Docky? =

== Dock ==

A Dock is an area or window where the user can put a program icon or an interactive program.

[[File:Laucher_dock.png|center]]

== Docky ==

A Docky is a little program to be placed in a dock. Dockies are able to control most of AmiDock's features and provides a great way to expand AmiDock beyond its default functionality. Dockies may be invisible to the user or show static or dynamic (animated) content. You may change their behavior, size and look according to requirements. Dockies are one of two main types:

=== Standalone Dockies ===

A standalone Docky is a special type of program which is made to show an icon in a Dock, delivering some functionality to the user. It is the most common type of Docky. This Docky type uses the standard shared library feature of AmigaOS as a common interface.

=== Application Dockies ===

The second type of Docky is an application Docky which may sometimes be known as an 'AppDocky' or 'AppDockIcon Docky'. An AppDocky is a Docky which is introduced to the system at run time by an application using the application.library registration mechanism for applications. The biggest difference between the two types of Dockies types is that AppDockies belong to a running application and usually are used to represent the current state of the owning application.

To sum up in "standalone dockies" the docky itself is the application (the only reason of existence of the application is the docky) while for "AppDockIcon dockies" the docky is just a graphical representation of a bigger application (the docky is just here to add user friendliness or visual feedback of the surrounding application)

= How to create and manipulate a simple docky =

You write a docky just like you'd write a standard Exec library. In your code you must implement certain specifically-named functions like DockyGet(), DockyProcess(), DockySet() etc. The docky manager (AmiDock) calls these functions when it needs to.

See the [http://os4depot.net/share/utility/docky/datetime_docky.lha datetime.docky source code] to get started.

= FAQ =

== Is it possible to create an application docky and then update its content on the fly and how? ==

Set the icon type to APPICONT_Docky; but you'll need to create a real docky as well then. If it doesn't need to be truly active it is possible just to change the icon's imagery (I think you have to un-register/re-register to get it to change though, so no good for frequent changes)

== I have successfully drawn a datatype (a picture) in a Docky and the picture file has more than 256 colors but it is rendered in 256 colors only. Why? ==

Just add PDTA_DestMode, PMODE_V43 to the NewDTObject call. You might also want to add DTA_GroupID, GID_PICTURE in order to restrict the file type to pictures.

== How are context menus created? ==

Context menus are dynamically built in response to DOCKYGET_ContextMenu attribute:

<syntaxhighlight>
BOOL DockyGet (struct DockyIFace *Self, uint32 msgType, uint32 *msgData)
{
switch (msgType)
{
/* ... */

case DOCKYGET_ContextMenu:
{
Object *contextMenu = (Object *)msgData;

Object *item1 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_PREFS),
PMIA_ID, PMID_PREFS,
PopupMenuItemEnd;

Object *item2 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_SAVEASDEFAULT),
PMIA_ID, PMID_SAVEASDEFAULT,
PopupMenuItemEnd;

Object *item3 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_USEASDEFAULT),
PMIA_ID, PMID_USEASDEFAULT,
PopupMenuItemEnd;

if (item1 && item2 && item3)
{
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item1);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item2);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item3);
}

}
break;

/* ... */
}
</syntaxhighlight>

== How do you use an alpha layer for a Docky? ==

On non-composited screens, AmiDock uses a "fake" transparency effect (i.e. the bitmap is filled with the contents of the window behind the dock), meaning that you can only add things over the background but you cannot render half or totally transparent contents without doing the blending at the same time (or you'll lose background information).

So if you want to support this configuration you need to follow this rule or provide an alternate rendering if you want to do fancy stuff when compositing is enabled.

Now, what you are trying to do should work on composited screens, but you have to tell AmiDock that you are using composited mode (see DOCKYGET_SupportsComposite and DOCKYGET_CompositeMode) so that it does not try to de-multiply the docky bitmap. Also I'm not sure you can use legacy pens to fill alpha channel, you're better off with direct ARGB painting (see SetRPAttrs() with RPTAG_APenColor).

== Is there a way to trigger a redraw from outside the docky? ==

Use DOCKYGET_NeedsAttention for this. This is how a docky can be notified from an external task by signaling AmiDock process. Quite straightforward, this is how e.g. [http://os4depot.net/share/utility/docky/winbar-docky.lha winbar.docky] ([http://os4depot.net/share/utility/docky/winbar_docky_src.zip source code is here], prefsobjects_macro.h include file which is required [http://openamiga.org/attachment/project/29/prefsobjects_macros.h here]) is told to rethink its layout in response to a change in preferences, or Intuition windows list. A shared structure and [[Exec_Mutexes|Mutex]] to protect the data. By the way, winbar.docky source code can be helpful not only for that, but for all other things such as adding context menus, firing requests to AmiDock or other of the many still undocumented features of the API.

You are notified about the task/bit to signal through DOCKYSET_DockyAttention (you would store it in the library base for an easy access by the main program). When your program signals AmiDock, every docky is sent the DOCKYGET_NeedsAttention message, so it is a good idea to use a flag to know if the notification really comes from your program.

== How does DOCKYGET_SupportsComposite work? ==

When compositing is enabled, a docky receives a DOCKYGET_SupportsComposite query. If it returns TRUE, it will then receive a DOCKYGET_CompositeMode query to which it can reply either DOCKYCOMPMODE_PreblendedRGB (0) or DOCKYCOMPMODE_RawRGB (1). The former has no advantages over the previous method, but was included for backward compatibility. The latter, however, allows you to directly *copy* the source ARGB data to the destination bitmap (or buffer) rather than blend it yourself. This way, it will be AmiDock itself that does the blending (once) without the need to undo the
previous blending to get the raw RGB data.

One thing to take into account when using this method, though, is that on a 16-bit screen you will get an actual 16-bit destination bitmap, not 32-bit as it happens when returning FALSE to DOCKYGET_SupportsComposite. In this case, you will need to write the alpha channel separately to the
(8-bit) bitmap provided in the DockyRenderDestination's alpha.RP field.

= See Also =

1. SDK:Documentation/AutoDocs/docky.doc

2. SYS:Documentation/Commodities/AmiDock_Arexx.doc

3. SDK:Examples/AmiDock/

AmiDock and Dockies

2014-05-01T05:19:25Z

Roman Kargin: /* Introduction */

= Introduction =

AmiDock is a tool to maintain a graphical menu bars (Docks) at the Workbench screen that can be used or to execute other programs from a selection of icons (either in the main Dock or in sub-Docks as specified by the AmiDock preferences), or to handle special kind of application called "Dockies", which can provide different functionality: render different kind of data right into the dock, provide user with a menu related to the docky of choice and so on.

Amidock implemented as commodity, and together with being responsible for creating and controlling docks and dockies it provide different functionality such as ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is AmiDock =

AmiDock is the program responsible for creating and controlling docks and dockies. AmiDock is implemented as a commodity but it's more than just that and provides many features like ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is a Dock and a Docky? =

== Dock ==

A Dock is an area or window where the user can put a program icon or an interactive program.

[[File:Laucher_dock.png|center]]

== Docky ==

A Docky is a little program to be placed in a dock. Dockies are able to control most of AmiDock's features and provides a great way to expand AmiDock beyond its default functionality. Dockies may be invisible to the user or show static or dynamic (animated) content. You may change their behavior, size and look according to requirements. Dockies are one of two main types:

=== Standalone Dockies ===

A standalone Docky is a special type of program which is made to show an icon in a Dock, delivering some functionality to the user. It is the most common type of Docky. This Docky type uses the standard shared library feature of AmigaOS as a common interface.

=== Application Dockies ===

The second type of Docky is an application Docky which may sometimes be known as an 'AppDocky' or 'AppDockIcon Docky'. An AppDocky is a Docky which is introduced to the system at run time by an application using the application.library registration mechanism for applications. The biggest difference between the two types of Dockies types is that AppDockies belong to a running application and usually are used to represent the current state of the owning application.

To sum up in "standalone dockies" the docky itself is the application (the only reason of existence of the application is the docky) while for "AppDockIcon dockies" the docky is just a graphical representation of a bigger application (the docky is just here to add user friendliness or visual feedback of the surrounding application)

= How to create and manipulate a simple docky =

You write a docky just like you'd write a standard Exec library. In your code you must implement certain specifically-named functions like DockyGet(), DockyProcess(), DockySet() etc. The docky manager (AmiDock) calls these functions when it needs to.

See the [http://os4depot.net/share/utility/docky/datetime_docky.lha datetime.docky source code] to get started.

= FAQ =

== Is it possible to create an application docky and then update its content on the fly and how? ==

Set the icon type to APPICONT_Docky; but you'll need to create a real docky as well then. If it doesn't need to be truly active it is possible just to change the icon's imagery (I think you have to un-register/re-register to get it to change though, so no good for frequent changes)

== I have successfully drawn a datatype (a picture) in a Docky and the picture file has more than 256 colors but it is rendered in 256 colors only. Why? ==

Just add PDTA_DestMode, PMODE_V43 to the NewDTObject call. You might also want to add DTA_GroupID, GID_PICTURE in order to restrict the file type to pictures.

== How are context menus created? ==

Context menus are dynamically built in response to DOCKYGET_ContextMenu attribute:

<syntaxhighlight>
BOOL DockyGet (struct DockyIFace *Self, uint32 msgType, uint32 *msgData)
{
switch (msgType)
{
/* ... */

case DOCKYGET_ContextMenu:
{
Object *contextMenu = (Object *)msgData;

Object *item1 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_PREFS),
PMIA_ID, PMID_PREFS,
PopupMenuItemEnd;

Object *item2 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_SAVEASDEFAULT),
PMIA_ID, PMID_SAVEASDEFAULT,
PopupMenuItemEnd;

Object *item3 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_USEASDEFAULT),
PMIA_ID, PMID_USEASDEFAULT,
PopupMenuItemEnd;

if (item1 && item2 && item3)
{
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item1);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item2);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item3);
}

}
break;

/* ... */
}
</syntaxhighlight>

== How do you use an alpha layer for a Docky? ==

On non-composited screens, AmiDock uses a "fake" transparency effect (i.e. the bitmap is filled with the contents of the window behind the dock), meaning that you can only add things over the background but you cannot render half or totally transparent contents without doing the blending at the same time (or you'll lose background information).

So if you want to support this configuration you need to follow this rule or provide an alternate rendering if you want to do fancy stuff when compositing is enabled.

Now, what you are trying to do should work on composited screens, but you have to tell AmiDock that you are using composited mode (see DOCKYGET_SupportsComposite and DOCKYGET_CompositeMode) so that it does not try to de-multiply the docky bitmap. Also I'm not sure you can use legacy pens to fill alpha channel, you're better off with direct ARGB painting (see SetRPAttrs() with RPTAG_APenColor).

== Is there a way to trigger a redraw from outside the docky? ==

Use DOCKYGET_NeedsAttention for this. This is how a docky can be notified from an external task by signaling AmiDock process. Quite straightforward, this is how e.g. [http://os4depot.net/share/utility/docky/winbar-docky.lha winbar.docky] ([http://os4depot.net/share/utility/docky/winbar_docky_src.zip source code is here], prefsobjects_macro.h include file which is required [http://openamiga.org/attachment/project/29/prefsobjects_macros.h here]) is told to rethink its layout in response to a change in preferences, or Intuition windows list. A shared structure and [[Exec_Mutexes|Mutex]] to protect the data. By the way, winbar.docky source code can be helpful not only for that, but for all other things such as adding context menus, firing requests to AmiDock or other of the many still undocumented features of the API.

You are notified about the task/bit to signal through DOCKYSET_DockyAttention (you would store it in the library base for an easy access by the main program). When your program signals AmiDock, every docky is sent the DOCKYGET_NeedsAttention message, so it is a good idea to use a flag to know if the notification really comes from your program.

== How does DOCKYGET_SupportsComposite work? ==

When compositing is enabled, a docky receives a DOCKYGET_SupportsComposite query. If it returns TRUE, it will then receive a DOCKYGET_CompositeMode query to which it can reply either DOCKYCOMPMODE_PreblendedRGB (0) or DOCKYCOMPMODE_RawRGB (1). The former has no advantages over the previous method, but was included for backward compatibility. The latter, however, allows you to directly *copy* the source ARGB data to the destination bitmap (or buffer) rather than blend it yourself. This way, it will be AmiDock itself that does the blending (once) without the need to undo the
previous blending to get the raw RGB data.

One thing to take into account when using this method, though, is that on a 16-bit screen you will get an actual 16-bit destination bitmap, not 32-bit as it happens when returning FALSE to DOCKYGET_SupportsComposite. In this case, you will need to write the alpha channel separately to the
(8-bit) bitmap provided in the DockyRenderDestination's alpha.RP field.

= See Also =

1. SDK:Documentation/AutoDocs/docky.doc

2. SYS:Documentation/Commodities/AmiDock_Arexx.doc

3. SDK:Examples/AmiDock/

AmiDock and Dockies

2014-04-29T05:30:50Z

Roman Kargin: /* Introduction */

= Introduction =

AmiDock is a tool to maintain a graphical menu bars (Docks) at the Workbench screen that can be used or to execute other programs from a selection of icons (either in the main Dock or in sub-Docks as specified by the AmiDock preferences), or to handle special kind of application called "Dockies", which can provide different functionality, render different kind of data right into the dock, provide user with a menu related to the docky of choice and so on.

= What is AmiDock =

AmiDock is the program responsible for creating and controlling docks and dockies. AmiDock is implemented as a commodity but it's more than just that and provides many features like ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is a Dock and a Docky? =

== Dock ==

A Dock is an area or window where the user can put a program icon or an interactive program.

[[File:Laucher_dock.png|center]]

== Docky ==

A Docky is a little program to be placed in a dock. Dockies are able to control most of AmiDock's features and provides a great way to expand AmiDock beyond its default functionality. Dockies may be invisible to the user or show static or dynamic (animated) content. You may change their behavior, size and look according to requirements. Dockies are one of two main types:

=== Standalone Dockies ===

A standalone Docky is a special type of program which is made to show an icon in a Dock, delivering some functionality to the user. It is the most common type of Docky. This Docky type uses the standard shared library feature of AmigaOS as a common interface.

=== Application Dockies ===

The second type of Docky is an application Docky which may sometimes be known as an 'AppDocky' or 'AppDockIcon Docky'. An AppDocky is a Docky which is introduced to the system at run time by an application using the application.library registration mechanism for applications. The biggest difference between the two types of Dockies types is that AppDockies belong to a running application and usually are used to represent the current state of the owning application.

To sum up in "standalone dockies" the docky itself is the application (the only reason of existence of the application is the docky) while for "AppDockIcon dockies" the docky is just a graphical representation of a bigger application (the docky is just here to add user friendliness or visual feedback of the surrounding application)

= How to create and manipulate a simple docky =

You write a docky just like you'd write a standard Exec library. In your code you must implement certain specifically-named functions like DockyGet(), DockyProcess(), DockySet() etc. The docky manager (AmiDock) calls these functions when it needs to.

See the [http://os4depot.net/share/utility/docky/datetime_docky.lha datetime.docky source code] to get started.

= FAQ =

== Is it possible to create an application docky and then update its content on the fly and how? ==

Set the icon type to APPICONT_Docky; but you'll need to create a real docky as well then. If it doesn't need to be truly active it is possible just to change the icon's imagery (I think you have to un-register/re-register to get it to change though, so no good for frequent changes)

== I have successfully drawn a datatype (a picture) in a Docky and the picture file has more than 256 colors but it is rendered in 256 colors only. Why? ==

Just add PDTA_DestMode, PMODE_V43 to the NewDTObject call. You might also want to add DTA_GroupID, GID_PICTURE in order to restrict the file type to pictures.

== How are context menus created? ==

Context menus are dynamically built in response to DOCKYGET_ContextMenu attribute:

<syntaxhighlight>
BOOL DockyGet (struct DockyIFace *Self, uint32 msgType, uint32 *msgData)
{
switch (msgType)
{
/* ... */

case DOCKYGET_ContextMenu:
{
Object *contextMenu = (Object *)msgData;

Object *item1 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_PREFS),
PMIA_ID, PMID_PREFS,
PopupMenuItemEnd;

Object *item2 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_SAVEASDEFAULT),
PMIA_ID, PMID_SAVEASDEFAULT,
PopupMenuItemEnd;

Object *item3 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_USEASDEFAULT),
PMIA_ID, PMID_USEASDEFAULT,
PopupMenuItemEnd;

if (item1 && item2 && item3)
{
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item1);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item2);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item3);
}

}
break;

/* ... */
}
</syntaxhighlight>

== How do you use an alpha layer for a Docky? ==

On non-composited screens, AmiDock uses a "fake" transparency effect (i.e. the bitmap is filled with the contents of the window behind the dock), meaning that you can only add things over the background but you cannot render half or totally transparent contents without doing the blending at the same time (or you'll lose background information).

So if you want to support this configuration you need to follow this rule or provide an alternate rendering if you want to do fancy stuff when compositing is enabled.

Now, what you are trying to do should work on composited screens, but you have to tell AmiDock that you are using composited mode (see DOCKYGET_SupportsComposite and DOCKYGET_CompositeMode) so that it does not try to de-multiply the docky bitmap. Also I'm not sure you can use legacy pens to fill alpha channel, you're better off with direct ARGB painting (see SetRPAttrs() with RPTAG_APenColor).

== Is there a way to trigger a redraw from outside the docky? ==

Use DOCKYGET_NeedsAttention for this. This is how a docky can be notified from an external task by signaling AmiDock process. Quite straightforward, this is how e.g. [http://os4depot.net/share/utility/docky/winbar-docky.lha winbar.docky] ([http://os4depot.net/share/utility/docky/winbar_docky_src.zip source code is here], prefsobjects_macro.h include file which is required [http://openamiga.org/attachment/project/29/prefsobjects_macros.h here]) is told to rethink its layout in response to a change in preferences, or Intuition windows list. A shared structure and [[Exec_Mutexes|Mutex]] to protect the data. By the way, winbar.docky source code can be helpful not only for that, but for all other things such as adding context menus, firing requests to AmiDock or other of the many still undocumented features of the API.

You are notified about the task/bit to signal through DOCKYSET_DockyAttention (you would store it in the library base for an easy access by the main program). When your program signals AmiDock, every docky is sent the DOCKYGET_NeedsAttention message, so it is a good idea to use a flag to know if the notification really comes from your program.

== How does DOCKYGET_SupportsComposite work? ==

When compositing is enabled, a docky receives a DOCKYGET_SupportsComposite query. If it returns TRUE, it will then receive a DOCKYGET_CompositeMode query to which it can reply either DOCKYCOMPMODE_PreblendedRGB (0) or DOCKYCOMPMODE_RawRGB (1). The former has no advantages over the previous method, but was included for backward compatibility. The latter, however, allows you to directly *copy* the source ARGB data to the destination bitmap (or buffer) rather than blend it yourself. This way, it will be AmiDock itself that does the blending (once) without the need to undo the
previous blending to get the raw RGB data.

One thing to take into account when using this method, though, is that on a 16-bit screen you will get an actual 16-bit destination bitmap, not 32-bit as it happens when returning FALSE to DOCKYGET_SupportsComposite. In this case, you will need to write the alpha channel separately to the
(8-bit) bitmap provided in the DockyRenderDestination's alpha.RP field.

= See Also =

1. SDK:Documentation/AutoDocs/docky.doc

2. SYS:Documentation/Commodities/AmiDock_Arexx.doc

3. SDK:Examples/AmiDock/

Hard and Soft Links

2014-04-28T05:07:41Z

Roman Kargin:

Hard and soft links allow an AmigaDOS user to refer to a single file or directory by more than one name. A '''hard link''' associates a new name with a file or directory by linking to its physical location on disk. A '''soft link''' associates a new name with a file or directory by linking to its path name. Hard and soft links are implemented in the filesystem as modified file header blocks. A new file header block is added for each link
created.

The header block for a hard link has a type of T.SHORT (2) and a secondary type of ST_LINKFILE (-4) or ST_LINKDIR (4) depending on whether it is linked to a file or directory. Hard links point to their object via
a block number pointer stored at size - 11.

Any AmigaDOS object which has a hard link pointing at it will get a new field at size - 10 that is a pointer back to the hard link. In addition, the hard link header block has this new file at size - 10 which is used to
chain together multiple hard links pointing at the same object.

If a hard link is deleted, it is first removed from the chain of hard links and then its file header block is freed. If the object a hard link points to is deleted, then the first hard link in the chain is altered so
that it becomes the new file header block. The original file header block is then freed.

Soft links have type T.SHORT and secondary type of ST_SOFTLINK (3). In this kind of link, the hash table area is used to store a BCPL string representing the path and name of the object being linked to, for example,
work:foo/bar/cap. The filesystem does not attempt to access work:foo/bar/cap but tells the caller that the file they are trying to access is a soft link. The caller must then execute the correct DOS call, ReadLink(), to find out what file should really be opened.

If a soft link is deleted then its file header block is free. If the object a soft link points to is deleted then the soft link is left pointing at a nonexistent file. Subsequent references to the soft link will return the "object not found" error from AmigaDOS.

Note that since Makelink 53.3, "Soft" option is the default and you need to specify the "Hard" option to make a hard link.

AmiDock and Dockies

2013-09-07T10:59:50Z

Roman Kargin: /* FAQ */

{{WIP}}

= Introduction =

AmiDock is a tool to maintain a graphical menu bar at the bottom of the Workbench screen that can be used to execute other programs from a selection of icons, either in the main Dock or in sub-Docks as specified by the AmiDock preferences.

= What is AmiDock =

AmiDock for AmigaOS 4 is a rewritten version of the former AmiDock from AmigaOS 3.9. While AmiDock is a single commodity, it's more than just that and provides many features like ARexx support and a rich API by which your Dockies can control most of AmiDock's features.

= What is a Dock and a Docky? =

== Dock ==

A Dock is an area or window where the user can put a program icon or an interactive program.

[[File:Laucher_dock.png|center]]

== Docky ==

A Docky is a little program to be placed in a dock. Dockies are able to control most of AmiDock's features and provides a great way to expand AmiDock beyond its default functionality. Dockies may be invisible to the user or show static or dynamic (animated) content. You may change their behavior, size and look according to requirements. Dockies are one of two main types:

=== Standalone Dockies ===

A standalone Docky is a special type of program which is made to show an icon in a Dock, delivering some functionality to the user. It is the most common type of Docky. This Docky type uses the standard shared library feature of AmigaOS as a common interface.

=== Application Dockies ===

The second type of Docky is an application Docky which may sometimes be known as an 'AppDocky' or 'AppDockIcon Docky'. An AppDocky is a Docky which is introduced to the system at run time by an application using the application.library registration mechanism for applications. The biggest difference between the two types of Dockies types is that AppDockies belong to a running application and usually are used to represent the current state of the owning application.

To sum up in "standalone dockies" the docky itself is the application (the only reason of existence of the application is the docky) while for "AppDockIcon dockies" the docky is just a graphical representation of a bigger application (the docky is just here to add user friendliness or visual feedback of the surrounding application)

= AmiDock's API =

= Different types of Dockies =

== Standalone Dockies ==

== Application Dockies ==

= How to create and manipulate a simple docky =

You write a docky just like you'd write a standard Exec library. In your code you must implement certain specifically-named functions like DockyGet(), DockyProcess(), DockySet() etc. The docky manager (AmiDock) calls these functions when it needs to.

See the [http://os4depot.net/share/utility/docky/datetime_docky.lha datetime.docky source code] to get started.

= FAQ =

''Q: Is it possible to create an application docky and then update its content on the fly and how ?

A: Set the icon type to APPICONT_Docky; but you'll need to create a real docky as well then. If it doesn't need to be truly active it is possible just to change the icon's imagery (I think you have to un-register/re-register to get it to change though, so no good for frequent changes)

''Q: I have successfully drawn a datatype (a picture) in a Docky and the picture file has more than 256 colors, but it is rendered in 256 colors only. Why ?

A: Just add PDTA_DestMode, PMODE_V43 to the NewDTObject call. You might also want to add DTA_GroupID, GID_PICTURE in order to restrict the file type to pictures.

''Q: How are context menus created? I don't see any contextmenu example in the SDK examples, but DOCKYGET_ContextMenu takes en Object * as data. What kind of Object are we talking about?

A: Context menus are dynamically built in response to DOCKYGET_ContextMenu attribute:

<syntaxhighlight>
BOOL DockyGet (struct DockyIFace *Self, uint32 msgType, uint32 *msgData)
{
switch (msgType)
{
/* ... */

case DOCKYGET_ContextMenu:
{
Object *contextMenu = (Object *)msgData;

Object *item1 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_PREFS),
PMIA_ID, PMID_PREFS,
PopupMenuItemEnd;

Object *item2 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_SAVEASDEFAULT),
PMIA_ID, PMID_SAVEASDEFAULT,
PopupMenuItemEnd;

Object *item3 = PopupMenuItemObject,
PMIA_Title, GetString(&li, LOCALE_ITEM_USEASDEFAULT),
PMIA_ID, PMID_USEASDEFAULT,
PopupMenuItemEnd;

if (item1 && item2 && item3)
{
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item1);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item2);
IIntuition->IDoMethod(contextMenu, OM_ADDMEMBER, item3);
}

}
break;

/* ... */
}
</syntaxhighlight>

''Q: How to use Alpha layer for Docky ?

A: On non-composited screens, AmiDock uses a "fake" transparency effect (i.e. the bitmap is filled with the contents of the window behind the dock), meaning that you can only add things over the background but you cannot render half or totally transparent contents without doing the blending at the same time (or you'll lose background information).

So if you want to support this configuration you need to follow this rule or provide an alternate rendering if you want to do fancy stuff when compositing is enabled.

Now, what you are trying to do should work on composited screens, but you have to tell AmiDock that you are using composited mode (see DOCKYGET_SupportsComposite and DOCKYGET_CompositeMode) so that it does not try to de-multiply the docky bitmap. Also I'm not sure you can use legacy pens to fill alpha channel, you're better off with direct ARGB painting (see SetRPAttrs() with RPTAG_APenColor).

''Q: Is there a way to trigger a redraw from outside the docky ?

A: Use DOCKYGET_NeedsAttention for this. This is how a docky can be notified from an external task by signaling AmiDock process. Quite straightforward, this is how e.g. [http://os4depot.net/share/utility/docky/winbar-docky.lha winbar.docky] ([http://os4depot.net/share/utility/docky/winbar_docky_src.zip source code is here], prefsobjects_macro.h include file which is required [http://openamiga.org/attachment/project/29/prefsobjects_macros.h here]) is told to rethink its layout in response to a change in preferences, or Intuition windows list. A shared structure and [[Exec_Mutexes|Mutex]] to protect the data. By the way, winbar.docky source code can be helpful not only for that, but for all other things such as adding context menus, firing requests to AmiDock or other of the many still undocumented features of the API.

I.e. you are notified about the task/bit to signal through DOCKYSET_DockyAttention (you would store it in the library base for an easy access by the main program). When your program signals AmiDock, every docky is sent the DOCKYGET_NeedsAttention message, so it is a good idea to use a flag to know if the notification really comes from your program.

= Final Words =

= Links =

1. SDK:Documentation/AutoDocs/docky.doc

2. Sys:Documentation/Commodities/AmiDock_Arexx.doc

3. SDK:Examples/AmiDock/

Tutorials:Main

2013-08-07T05:10:16Z

Roman Kargin: deleted initial attempt of programming article which not fits into rulz.

= Tutorials =

Tutorials have been provided by various authors.

== General ==

[[The Hacking Way: Part 1 - First Steps]]

[[The Right Tool for the Job (Shared Objects)]]

[[How to Build Stubs for 68k Libraries]]

== Debugging ==

[[How to install a hardware interrupt]]

[[How to open and use the exec debug interface]]

[[GDB for Beginners]]

[[Using Crash-Logs for Debugging]]

[[Debug Logging on AmigaOS]]

[[Redirecting Debug Output to the Serial Port on Startup]]

[[Advanced Serial Debugging Guide]]

== GUI ==

[[BOOPSI Gadget Help Strings]]

[[BOOPSI Popup Menus - Part 1]]

[[BOOPSI Popup Menus - Part 2]]

== AmiWest 2013 ==

[[AmiWest 2013 Programming Conference Synopsis]]

{{Note|This section is under development and will be changing.}}

== AmiWest 2012 ==

[[AmiWest Setup]]

[[AmiWest Lesson 1|AmiWest Lesson 1: Coding Basics]]

[[AmiWest Lesson 2|AmiWest Lesson 2: AmigaOS Fundamentals]]

[[AmiWest Lesson 3|AmiWest Lesson 3: Input and Output]]

[[AmiWest Lesson 4|AmiWest Lesson 4: ProcTree]]

[[AmiWest Lesson 5|AmiWest Lesson 5: MIDI]]

[[AmiWest Lesson 6|AmiWest Lesson 6: Application Library]]

[[AmiWest Lesson 7|AmiWest Lesson 7: Screen Blanker]]

[[AmiWest Lesson 8|AmiWest Lesson 8: ARexx Ports]]

[[AmiWest Support]]