Libraries and Devices

Introduction

Overview

This guide is most specifically directed at programmers wanting to convert library code to AmigaOS 4.x. Finally, a (hopefully) comprehensive explanation of the process of converting a library will be presented.

For an introduction to some basic concepts see Exec Libraries.

Anatomy of an Interface

The first four function pointers are fixed and need to be present in every interface. After that, any number of function pointers may follow.

Each interface has a version number. Any change in the interface will keep the version number if and only if function pointers are added to the interface. That is, if any prototype or location of an interface function changes, the version number needs to be incremented. Programs using an interface can request a specific version number; if that version is available, it will be returned, otherwise an error is generated.

The include file <exec/interfaces.h> introduces the overall structure of an interface:

struct InterfaceData
{
 struct Node Link; /* Node for linking several interfaces */
 struct Library *LibBase; /* Library this interface belongs to */
 ULONG RefCount; /* Reference count */
 ULONG Version; /* Version number of the interface */
 ULONG Flags; /* Various flags (see below) */
 ULONG CheckSum; /* Checksum of the interface */
 ULONG PositiveSize; /* Size of the function pointer part */
 ULONG NegativeSize; /* Size of the data area */
 // ...
};
 
struct Interface
{
 struct InterfaceData Data; /* Interface data area */
 /* Increment reference count */
 ULONG (*Obtain)(struct Interface *Self) APICALL;
 /* Decrement reference count */
 ULONG (*Release)(struct Interface *Self) APICALL;
 /* Destroy interface. May be NULL */
 void (*Expunge)(struct Interface *Self) APICALL;
 /* Clone interface. May be NULL */
 struct Interface * (*Clone)(struct Interface *Self) APICALL;
};

Let's first have a look at the embedded InterfaceData structure. RefCount is a counter that works like the OpenCnt in the library base. Only Interfaces with a RefCount of zero can be deleted. The Version number is stored in the Version field. Flags contains a number of bitdefs that specify certain behaviour principles of the interface. A number of flags have been currently defined:

• IFLF_PROTECTED indicates that this is a protected interface, that is, an interface that must not be modified by exec/SetMethod() (exec/SetMethod is to interfaces what SetFunction() is to AmigaOS 3.9 libraries). An attempt to patch a protected interface fails. This protection may or may not be enforced by hardware. Potential usage for this is an SSL interface, Password/Login interface, or any interface that is security-sensitive.

• IFLF_NOT_NATIVE indicates that this is an interface that contains emulator stubs for calling a classic 68k jump table. This flag doesn't serve any particular purpose other than to mark this interface as potentially slow.

• IFLF_PRIVATE indicates that this is a private interface. Private interfaces are non-sharable, and usually carry contextual information that are local to a process or task.

• IFLF_CHANGED and IFLF_UNMODIFIED are used by the system. The Set-Method method sets the IFLF_CHANGED flag, and the SumInterface method recalculates the checksum and sets IFLF_UNMODIFIED again.

• IFLF_CLONED marks a cloned interface. See below in the description of the Clone method.

The CheckSum field contains a checksum over the interface, and is used for sanity checking and for security, much the same way as the library checksum.

The PositiveSize and NegativeSize fields specify, as the name implies the positive and negative size of an interface. The positive size is the size from the interface start to the end, stretching across the fields at the start of the interface and all function pointers. The negative size is usually zero; if it is non-zero, it specifies the size of the interface's private data area, which is located NegativeSize bytes before the Interface data. Contrary to the AmigaOS 3.9 library, data is always stored in front of the interface. Most interfaces don’t contain their private data, and therefore the NegativeSize is zero.

The interface itself start with the embedded InterfaceData structure, followed by the four standard interface functions (we will call interface functions "methods" from now on). The four standard methods are:

• ULONG Obtain() This method adds a reference to the interface. It increments the RefCount field by one and returns the current RefCount. In order to use an interface, the Obtain method must be called first. The exec library method "GetInterface" does this implicitly.

• ULONG Release() This method removes a reference from the interface. After releasing the interface it must not be used again.

• void Expunge() This method deletes the interface and frees all associated resources. This function is usually not called directly.

• struct Interface *Clone() This method clones an interface. Usually it is sufficient to use Obtain() to be able to use the interface. However, some interfaces might contain data that need to be cloned. For example, consider an interface that can load an image from disk, and store the image data within the interface. That interface would have methods for reading/writing pixels within the picture. Now your paint package want to apply a filter and offer a preview function. For this, the picture would need to be modified, but the original needs to stay around in case the user pressed cancel. For this reason, the interface would be cloned (and hence the picture with it), and the preview could be done on the clone.

Using Libraries and Interfaces

In order to use an interface, the appropriate library needs to be opened first. This is done with the usual call to OpenLibrary(), which returns a library base pointer like under AmigaOS 3.x. It is not possible, however, to directly call function from that library (except for "classic" libraries).

To call library functions, you need to retrieve an interface pointer first. This is done using the exec.library GetInterface function (see the autodoc for GetInterface). The method returns an already Obtained() interface that must be disposed with exec.library/DropInterface after usage. Interfaces are retrieved by name; most libraries export at least the "main" interface, which is the representation of their basic functionality, especially for classic libraries.

If an interface pointer is passed along to another entity, for example another thread, then that entity must call Obtain() before using the interface. Likewise, when the entity is done, it needs to call Release().

To sum up: New interfaces must always be created using GetInterface/DropInterface; existing interfaces must be referenced/dereferenced with Obtain/Release.

"Classic" libraries pose a special case in this equation, since any source code from the classic OS expects to open e.g. intuition.library and directly call OpenWindow on it. These libraries (which are supposedly opened during startup code) have special inline functions that call the appropriate method in the "main" interface.

A simple example illustrates this. We assume that the global variable "IExec" contains the main interface pointer of exec.library. To open "test.library", we would call

struct Library *testLib = IExec->OpenLibrary("test.library", 0);

Note that we could also write

struct Library *testLib = OpenLibrary("test.library", 0);

because there is a stub function OpenLibrary() for backwards compatibility with classic programs. The OpenLibrary stub function assumes that the global variable IExec contains the "main" interface pointer of exec. The first version is the preferred version for AmigaOS 4.

In any case the above will yield a struct Library pointer in testLib that we can use to retrieve interfaces from test.library. Since this library isn't a classic library (it has no parallel version under AmigaOS 3.9) it doesn't automatically export any methods. To get the "main" interface of this library we would write:

struct TestMainIF *IMain = (struct TestMainIF *)
 IExec->GetInterface(testLib, "main", 1, NULL);

The struct TestMainIF is a structure that contains an embedded struct Interface along with additional function pointers - the methods that the interface offers. After this code sequence we can use the IMain interface. Note that by convention, all interface pointers are written as a capital 'I', followed by a capitalized name. IExec and IIntuition would be examples. A test for a NULL pointer should always be performed before using any interface.

While this at first glance looks a bit intimidating, keep in mind that you usually only do this once and for libraries like Intuition, most of this stuff is already handled by startup code.

Calling a function from IMain would look something like

int32 test = IMain->;DoSomething(10, 100);
IDOS->Printf("IMain->DoSomething(10, 100) = %ld\n", test);

To dispose of the interface, one would call

IExec->DropInterface((struct Interface *)IMain);

Finally, to close the library call

IExec->CloseLibrary((struct Library*)testLib);

Using Devices

Using devices is almost exactly the same as under AmigaOS 3.9, only that you call the methods of IExec. Since you usually don't call any of the device vectors directly, there is no apparent change to using a device. You still create an IORequest and message port, open the Device with the IExec->OpenDevice() call and call IExec->DoIO() to perform tasks.

The Self Pointer and the APICALL Tag

If you look at the standard vector declarations above, you will notice that their first parameter is always a "struct Interface *Self", but we do not specify that Self parameter in the calls. Instead, all method function pointers have a tag APICALL added. That tag identifies them as interface function pointers.

Through this special function attribute, the gcc compiler for AmigaOS 4 will recognize a construct of the form

IExec->CloseLibrary(libBase);

and implicitly (not by preprocessor magic, but through the code generator) transform this into

IExec->CloseLibrary(IExec, libBase);

This saves the user the hassle of writing this, but of course this only works on the AmigaOS 4 compiler. Obviously, any compiler adapted for AmigaOS 4 must implement this mechanism.

Library Internals

Internally, libraries and devices look very similar. They both can export any number of interfaces, making it possible for e.g. timer.device to export its time arithmetic functions in its main interface (or under any other name). The only difference between libraries and devices is that a library must export at least one interface called "__library" of type LibraryManagerInterface, and a device must export an interface called "__device" of type DeviceManagerInterface.

Both interfaces are defined in <exec/interfaces.h>:

struct LibraryManagerInterface
{
 struct InterfaceData Data;
 
 ULONG APICALL (*Obtain) (struct LibraryManagerInterface *Self);
 ULONG APICALL (*Release)(struct LibraryManagerInterface *Self);
 VOID APICALL (*Expunge)(struct LibraryManagerInterface *Self);
 struct Interface * APICALL (*Clone)(struct LibraryManagerInterface *Self);
 struct Library * APICALL (*Open)(struct LibraryManagerInterface *Self, ULONG version);
 APTR APICALL (*Close)(struct LibraryManagerInterface *Self);
 APTR APICALL (*LibExpunge)(struct LibraryManagerInterface *Self);
 struct Interface * APICALL (*GetInterface)(struct LibraryManagerInterface *Self, STRPTR name, ULONG version,
   struct TagItem *taglist);
};
 
struct DeviceManagerInterface
{
 struct InterfaceData Data;
 ULONG APICALL (*Obtain) (struct DeviceManagerInterface *Self);
 ULONG APICALL (*Release)(struct DeviceManagerInterface *Self);
 VOID APICALL (*Expunge)(struct DeviceManagerInterface *Self);
 struct Interface * APICALL (*Clone)(struct DeviceManagerInterface *Self);
 VOID APICALL (*Open)(struct DeviceManagerInterface *Self, struct IORequest *ior, ULONG unit, ULONG flags);
 APTR APICALL (*Close)(struct DeviceManagerInterface *Self, struct IORequest *ior);
 APTR APICALL (*LibExpunge)(struct DeviceManagerInterface *Self);
 struct Interface * APICALL (*GetInterface)(struct DeviceManagerInterface *Self, STRPTR name, ULONG version,
   struct TagItem *taglist);
 VOID APICALL (*BeginIO)(struct DeviceManagerInterface *Self, struct IORequest *ior);
 VOID APICALL (*AbortIO)(struct DeviceManagerInterface *Self, struct IORequest *ior);
};

As you can see, a library has four and a device six additional function pointers. The four library function pointers are also represented in the device manager interface. The Open/Close/Expunge vectors serve the exact same purpose as under AmigaOS 3.9, and their implementation can more or less follow their AmigaOS 3.9 counterpart. The fourth vector, GetInterface, is usually empty, because exec.library already implements that functionality. However, some libraries might want to override the functionality and implement their own GetInterface function.

A device manager interface has two additional vectors, BeginIO and AbortIO whose functionality mirrors that of the AmigaOS 3.9 device.

Migrating a Library from AmigaOS 3.x

Introduction

Migrating a library from AmigaOS 3.x may or may not be trivial. It all depends on the internals of the library. For a library function itself, the most obvious change is that the library base will not be passed along, but the interface pointer. The interface pointer, however, contains the library base pointer, so if that is required it can still be gotten from the interface. Also, the interface itself may carry along private or instance data.

Interface Definition Files

The first thing to do when starting a new library or migrating an existing one is to write the IDL file . The IDL file is what used to be the FD file in the old system, with a few substantial differences. First of all, the file format is now XML, using a special "library" DTD. The file should have the same base name as the library with the suffix ".xml" added, e.g. exec.xml, expansion.xml and so on. Each IDL file should start like this:

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE library SYSTEM "library.dtd">
<library name="expansion" basename="ExpansionBase">
...
</library>

This declares an XML file and includes the dtd. The third line (<library...>) defines the document's contents, in this case a single library. The "name" attribute specifies the name of the library. A ".library" is implicitly added to this name. The second attribute (basename) specifies the global name of the library base and the name of the base structure (i.e. struct ExpansionBase * in this case). Note that the library base pointer need not be global at all for a function to be called - only the interface pointer is used for that.

The <library> element may contain any number of <include> elements. <include> elements specify include files that need to be present to define all types that are used within the library. The files <exec/interfaces.h> and <exec/types.h> are implicitly added. The name of the include file must be given as contents to the <include> element, and only one such file is allowed per element:

<include>expansion/pci.h</include>

After the <include> element, you should start listing all library interfaces. There can be any number of interfaces. Interfaces are specified using the <interface> element:

<interface name="pci" version="1" flags="protected" struct="PCIIFace" prefix="_pci_">

The following attributes exist:

• name (required) The name of the interface. This name will be used by an application to request the interface.

• version (default: 1) The version of the interface. Interface versions start at 1. Multiple interfaces with the same name may be defined if and only if they have different version numbers. Once an interface is published, it must not be changed unless the interface version number is incremented.

• flags (default: none) The flags to set for this interface. This may be "protected", "private", or "none", with obvious mappings to the interface flags.

• struct (required) The name of the structure that holds the interface, automatically prefixed by "struct". The example above specifies the struct name to be "PCIIFace", that means that any interface acquired from expansion library by the name of "pci" will have the layout of "struct PCIIFace".

• prefix (default: "_impl_") The common prefix for all interface functions. It is highly recommended that all interface functions get a command prefix, since some function names might be ambiguous, like Obtain, Release, Expunge and Clone. This prefix is used for the automatic library creation process.

The <interface> element may contain any number of <method> elements. <method> elements describe the methods that an interface provides. Contrary to FD files the method description also contains the return types and parameter types. The following lists the four standard interface methods, which must be first in every interface:

<method name="Obtain" result="uint32"></method>
<method name="Release" result="uint32"></method>
<method name="Expunge" result="void" status="unimplemented"></method>
<method name="Clone" result="struct Interface *" status="unimplemented"></method>

These methods don't have any other parameter but the Self pointer, which is implicit and need not be named. The "name" attribute defines the name of that method, the name under which the method is called when used. Together with the prefix specified in the interface, it forms the name of the function (see the testlib example). The "result" attribute describes what the method returns, which may be "void" if the function does not return anything, or a type like "uint32", or even a complex type like "struct Interface *". Finally, the "status" attribute used by some methods (which defaults to "implemented") specifies if the method actually exists or not. If it doesn't exist, set this to "unimplemented". This will "reserve" a slot in the interface's jump table, but fill it with NULL. Obviously, calling such a function results in Violent Program Termination.

After these initial methods, you need to add a <method> element for each method in the interface. Methods without parameters can be written as above, or in a single tag, as per XML specifications:

<method name="Obtain" result="uint32"/>

If a method has arguments (the most common case), the arguments must be listed as <arg> elements:

<method name="AllocResource" result="struct PCIResourceRange *">
 <arg name="ResType" type="uint32"/>
 <arg name="NumBytes" type="uint32"/>
</method>
 
<method name="FreeResource" result="void">
 <arg name="Resource" type="struct PCIResourceRange *"/>
</method>

These two declarations would result in the following two method prototypes:

struct PCIResourceRange *AllocResource(struct PCIIFace *Self,
 uint32 ResType, uint32 NumBytes);
void FreeResource(struct PCIIFace *Self, struct PCIResource *Resource);

<arg> elements must not have any content, and hence should always be written in the one-tag way.

Remember to close the <interface> element by a trailing </interface>. As was said before, any number of <interface> elements may follow.

Once you have written the interface description, the "idltool" program must be used to generate the required header files. Running idltool with the "--all" option generates all available targets.

Migrating the Source Code

As has been said before, library calls in AmigaOS 4 are arranged into interfaces. There are four interface methods that every interface has. Usually you can use generic versions of these. The idltool will output a complete skeleton for you which makes it much easier to get going.

All functions need to be modified in such a way that the LibBase parameter (if present) is removed, and the new Self parameter needs to be added. For example, consider this function (AmigaOS 3.9 style) from a fictitious library "fubar.library":

ULONG FubarFunc(ULONG param1, ULONG param2, struct FubarBase *libBase)
{
 libBase->FubarCalled++;
 ULONG x = param1 * libBase->FubarFactor / param2;
 return x;
}

The modifications required mostly depend on a few design choices we have to make. The "FubarCalled" variable is part of the library base in the example above but that need not be the case for AmigaOS 4 - we can move this field either to the interface private data or leave it in the library base.

We assume that our interface for the library is called "FubarMainIFace". You may chose any name but it is recommended (though not enforced) that they start with the library name and end with "IFace".

uint32 FubarFunc(struct FubarMainIFace *Self, uint32 param1, uint32 param2)
{

So far, things that have changed is the types are modernized and the removal of the libBase parameter and the addition of the interface pointer. What follows depends on the choice of placement of the FubarCalled and FoobarFactor fields. Let's first assume that we left them in the library base, so we need a pointer to the library. That pointer is always stored in the interface:

struct FubarBase *libBase = (struct FubarBase *)Self->Data.LibBase;
 libBase->FubarCalled++;
 uint32 x = param1 * libBase->FubarFactor / param2;
 return x;
}

Now, if we decided to move these fields into the interface itself, things look a bit different. Generally if you think that data is specific to an interface, and hence specific to a task context, put the fields into the interface. Otherwise leave them in the base. Migrated libraries will probably want to keep them in the base structure, but just in case this is the way to access the data from the interface:

struct FubarMainData *ifData =
 (struct FubarMainData *)((uint32)Self - Self->Data.NegativeSize);
 ifData->FubarCalled++;
 uint32 x = param1 * ifData->FubarFactor / param2;
 return x;
}

This example assumes that your data layout for the interface looks like the struct FubarMainData structure.

Library Initialization

Initialization via init tables is no longer supported starting with AmigaOS 4. To initialize a library you need to provide a ROMTAG structure like with previous versions of the OS, although the interpretation of some of the fields are a bit different. Most notably, the rt_Flags field must have the RTF_NATIVE flag set to indicate that the code in this library is native PowerPC and not 68k. If the RTF_AUTOINIT flag is not set, the rt_Init field is assumed to point to a function that is called up on initialization (depending on the RTF_NATIVE flag, it might be called under emulator control).

If the RTF_AUTOINIT flag is set, then the rt_Init must point to a taglist that will be passed to a function - the function again depends on RTF_NATIVE ROMTAGs without the RTF_NATIVE flag will be handled identically to the old way - their rt_Init field points to a init table and is passed to MakeLibrary.

Native libraries expect to find a taglist at rt_Init that is passed to exec.library/CreateLibrary. This creates a new-style PowerPC-native library with an optional 68k interface part (We'll cover the 68k part later).

A typical taglist for this looks somewhat like this:

struct TagItem fubar_libCreateTags[] =
{
 {CLT_DataSize, sizeof(struct FubarBase)},
 {CLT_InitFunc, (uint32)fubar_libInit},
 {CLT_Interfaces, (uint32)fubar_libInterfaces},
 {TAG_END, 0}
};

In this case, the first tag specifies the size of the base structure, the second tag specifies the libinit function (we'll look at that in a minute) and the last tag specifies the interfaces array. The interfaces array is a null-terminated array of pointers to tag lists for the exec.library/MakeInterface call. Typically you will have at least two interfaces here, the library manager interface and an exported interface (which will most likely be "main"). The layout of the library manager interface has already been discussed, we are going to cover the methods in a few moments.

The interfaces array might look something like this:

uint32 fubar_libInterfaces[] =
{
 (uint32)fubar_managerTags,
 (uint32)fubar_mainTags,
 0
};

The first would be the taglist for creating the manager interface, the second for the main interface. We will only cover the manager interface here, since the main interface looks exactly the same, with different functions only. Note that both start with the four default vectors.

For out library manager interface, the tag list looks like this:

struct TagItem foobar_managerTags[] =
{
 {MIT_Name, (uint32)"__library"},
 {MIT_VectorTable, (uint32)foobar_manager_vectors},
 {MIT_Version, 1},
 {TAG_END, 0}
};

Here, the name is fixed (in any other interface the name could be chosen freely) and the version number is also fixed to "1". you may use different version numbers in "normal" interfaces, but an interface of this type must be present.

The foobar_manager_vectors variable is the vector table. This is simply a null-terminated array of methods, in the order in which they appear in the interface. For the library manager interface, we need to have the methods as outlined earlier in this document. The table should look somewhat like this:

void *fubar_manager_vectors[] =
{
 (void *)generic_Obtain,
 (void *)generic_Release,
 (void *)0,
 (void *)0,
 // In general, start the "normal" methods here
 (void *)fubar_libOpen,
 (void *)fubar_libClose,
 (void *)fubar_libExpunge,
 (void *)0,
 (void *)-1,
};

As per usual the interface starts with the default methods. We use the generic obtain and release methods, and leave both expunge and clone unimplemented (if you leave expunge unimplemented, the interface will simply be freed).

The following three vectors are the same as with classic libraries, only that they are assumed to be native code as opposed to 68k code. We'll list sample implementations a bit further down.

The last (in this case unimplemented) vector is the GetInterface call. With all probability you will leave this empty for just about everything. Exec already offers a GetInterface function that should be sufficient for just about every purpose. Only very special cases require this vector, so we will skip its discussion for the purpose of this document.

The idltool provides a good sample implementation of the three above-mentioned library manager methods and should be referenced for more details.

We now need to look at the libInit function. This was specified using the CLT_InitFunc tag in the ROMTAG-anchored taglist for CreateLibrary. The function is responsible for setting up one-shot resource stuff and things that are not handled in the open code. Generally, you will almost always be able to use the generic implementation provided by the idltool.

Handling 68K

Overview

There are two possible scenarios from where a library might be called: From native code, or from 68k code. The native code calls will invariably go via an interface; the 68k calls will always be base-relative.

This means that for a library to be native-callable, at least one interface must exist. Since this is only some sort of switchboard, we will call these "proxy" interfaces: the interface just redirects calls through the emulator.

This also means that if a native library needs to be callable from 68k, it must have a jump table. Of course, for a 68k program to call a native library, this program must know the library; it is highly unlikely that a 68k program will call the PCI interface of expansion.library directly (i.e. not through an emulation layer) because it will most likely not know that expansion.library offers anything but the "normal" functions.

Native Interfaces to 68K Libraries

A 68k library that needs to be native-callable needs an interface. That interface may be created through two different mechanisms. For a disk-based library, the loader will scan the directory for a file that contains this interface, to be loaded with the library. This file may or may not contain additional code that replaces some of the library vectors.

For resident libraries, this looks a bit different. Resident libraries must have the interface already created in memory. Since these libraries are most likely part of the system itself, these libraries should be treated very much like a native library.

Assume for example that graphics.library would remain 68k. In order to call it from native code, it needs to have an interface (preferably "main"). This is achieved by inserting a ROMTAG with a priority of one lower than the actual library, which constructs the interface and stores it as an interface of Exec, under the name "graphics.interface". When a program attempts to obtain an interface from a non-AmigaOS 4 library, Exec's GetInterface function constructs this name from the library name and returns the pointer to this "proxy" interface.

Using idltool

The utility program idltool can be used to generate libraries as well, in a very convenient manner. All you have to do is provide the XML interface description file and invoke idltool with the command line

idltool -k <interfacefile>.xml

This will create a directory <interfacefile>_files in your current directory that contains a complete skeleton library with a Makefile and a README with additional instructions. Simply typing "make" should build the library. For more information, refer to the documentation that comes with idltool.