How to Build Stubs for 68k Libraries

Author

Roman Kargin
Copyright (c) 2013 Roman Kargin
Proofreading and grammar corrections by the AmigaOS Wiki team.

Introduction

The idea for this tutorial came from when we was playing with Dopus5 source code and wanted to make one single function from the 68k version of dopus5.library to work from native AmigaOS 4 code. While we have all kind of auto generation tools to build all kind of protos/interfaces/inlines/stubs/etc, it still wasn't clear to me how to do what I needed.

Since the beginning a very helpful article was Thomas Rapp's How to create PPC stubs for M68K libraries where he shows how to use fd2pragma and IDLTool to build a jpeg.l.main stub from a 68k jpeg.library, as well as helpful answers from Fredrik Wikstrom. In the end, all this information was important to write down so I put everything into this tutorial. I hope other developers will find it useful.

PowerPC->68k Glue Stubs

While 68k libraries are old and almost all libraries on AmigaOS today are PowerPC native ones, there are still some old AmigaOS 3.x libraries which are necessary and have no native PowerPC replacement. In that case we need to create for them necessary PowerPC->68k glue stub files, so we can call those 68k libraries's routines from native, PowerPC code, or, we can use EmulateTags(). The latter makes it easy for your code to use both PowerPC and 68k native libraries transparent with only a small bit of small overhead; just a few instructions. But this only works when the PowerPC native code is available and when there are just a few functions in the library. Quite often, you just don't want to deal with all this because of the large number of functions in a 68k library and the different ways of using them from the code. In this case, creating of PowerPC->68k stubs is a simpler and faster way.

For a better explanation of what a PowerPC->68k stub is, refer to the Migration Guide. In short, if you have, let's say, the 68k version of dopus5.library and want to use it from native AmigaOS 4 binaries then you need to create dopus5.l.main. When code in your native binary needs to interface to a old AmigaOS3 library, the system (ramlib to be more precise) will automatically scan its library search path for the file "dopus5.l.main" and if found, it will open it and obtain the interface from there. That l.main file is pure non-startup code which contains the necessary stubs. You do not need to write the stubs from scratch manually. This is a job for different set of tools (fd2pragma, fdtrans and IDLTool). The library.l.main code consists of 2 files if you auto generate them: PowerPC->68k crosscall stubs and vectors. Of course you can write them from scratch manually, but its a pain and there are already tools which do it for you.

The name of the file in general does not matter. A library does not necessarily have to be named ".library" and could be named ".module" or whatever. Ramlib will search for the first "." in the library name, starting from the end, the first part up to and including the dot plus one more character, usually the 'l' in library, is used, another dot is appended, then the interface name is added. It then calls the exec function.

For example, if the interface name is 'main' and the library name is 'foobar' then this is what it looks for:

foobar.library  -> foobar.l.main
foobar.blabla   -> foobar.b.main
foobar.module   -> foobar.m.main

Note

Because of the way RamLib parses glue-stubs names, you can't call your glue stubs say "foobar.library" as "foobar.library.main". It is understandable from the explanation above, but it can be misleading when you run for example "snoopy" and can see how it tries to search for "foobar.library.main", but it didn't tries to search for a file, just for "resident" code in memory. The real file ramlib will look for is only foobar.l.main when the main library is called as foobar.library.

Note
You can't put your glue stub files in directories other than PROGDIR:, PROGDIR:Libs or LIBS: as RamLib will not find them. If you have let's say a directory "modules" where you store some of your plugins (which in reality are libraries), then you can't put your glue stub files in the same "modules" directory, but to the place mention above.

pragmas/fd/sfd/protos

Often, old 68k libraries were not done with GCC. On old m68k machines, SAS/C was the popular C compiler and pure assembler was even more popular for everyday needs. So many of those old libraries were provided with includes for SAS/C only or for assembler (like .i files) or a mix of the two. You may have such library with which you have only a pragma file and a bunch of proto files for functions which are present in more than one proto file (i.e. no .fd file at all, no single include with all protos). The first thing to do is to create a single proto file from those different proto files you have in different places, manually make your .fd file from pragma (check all offsets and order how they are placed in pragma file, and in the same order put them to your new .fd). Then when you have only one proto file and one .fd file, you can generate a .sfd file and an .xml one with fd2pragma. Then generate from it all includes/vectors/stubs and only after that you can generate lib.l.main :) Usually its not that hard, and most old libraries have .fd files and a single proto include file, but sometimes you will have to do it for difficult libs, so you will know what to do.

All we need to know now is that libraries have a jump table. Jump tables are a list of library functions at specific offsets. Offsets can't be changed because they are function offsets in the library that your code will call. Be it with the old SAS/C, an assembler or the new GCC on AmigaOS 4, if we work with a 68k library and create any fd/proto files from scratch then the offsets should always be the same.

For example you have dopus_pragmas.h file and dopus.fd files (fd files are a standard way to describe the interface for libraries) and dopus_pragmas entries look like this:

#pragma libcall DOpusBase RemovedFunc0 1e 0
#pragma libcall DOpusBase Random 24 001
#pragma libcall DOpusBase Atoh 2a 0802
....

It means that the first "user" library function in a jump table starts at offset 0x1e (30) and then multiply by 6 for every function. So first function is at 30 (0x1e), second one is at 36 (0x24), the next one is at 42 (0x2a) and so on for all the functions. That means that if for example function Atoh() in the jump table is placed at offset 0x2a, then every single code and includes file which we will generate for AmigaOS 4 should respect that jump-table offset.

If we look at the dopus_lib.fd file we see:

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
RemovedFunc0()()
Random(limit)(d0)
*
Atoh(str,len)(a0,d0)
....

This means that public functions start from offset 30 (bias 30), and they are placed in the same order as they are in the pragma file. So if you change the order of function names in the FD file you will end calling a function instead of another one, which of course means crashes, bugs and problems.

So respect offsets in the jump table.

Note that .sfd files work like .fd files but with more possibilities. You won't find them in old 68k dev archives of libraries as its something new. They contain a lot more infomation than .fd and are easier to use. But more or less they follow the same logic, the same respect for offsets, etc. To see a full description of .fd and .sfd formats check fd2pragma guide.

Now What?

We have 2 methods to build lib.l.main and required includes. But for both of them you need .fd file as a minimum (if you have .sfd its even better), and a proto file which will describe all (or not all) your functions. While .fd/.sfd should be done by the original developer, you still can make them from scratch if you have a pragma file with lib calls (you make this file with the same order and the same offsets). If you don't have one proto file, but many in different includes, then you can make your own one, just copy one by one all the protos from all the include files, while all your FD entries will not have necessary proto descriptions in proto file (btw, if you don't have proto for some functions, then entries from the fd/sfd file will be reserved when you will auto generate AmigaOS code/includes, so respect offsets).

Let's say we have dopus_lib.fd and only protos in different includes in different places. We then create proto file from all those includes (in referring to the dopus_lib.fd), and then auto-tools time coming. As I say we have 2 ways:

First method: fdtrans + sfd

1. generate a .sfd file:

ram:> fd2pragma dopus5_lib.fd clib our_proto.h special 112
SourceFile: dopus5_lib.fd
ResultFile: dopus5_lib.sfd

2. generate xml and ppc->68k crosscall stubs via fdtrans:

ram:> fdtrans dopus5_lib.sfd -x -s

Change manually in the crosscall-stubs file (dopus5.c) "main_vectors" in "main_v1_vectors", as it seems some typo between different tools. No other changes is needed, fdtrans automatically insert including of interface (while fd2pragma will skip it, see later).

3. generate proto/interface/inline (i.e. includes) and vectors:

ram:> idltool -p -i -n -c dopus5.xml 

4. build dopus5.l.main

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build a test program to use one function from dopus5.library, here StrConcat()

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct DOpusIFace *IDOpus;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDOpus=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDOpus->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}

ram:> gcc -Iinclude testcase-native-fdtrans.c 
ram:> a.out
ram:> Result=First part-Second part

It works!

Second method: fd2pragma + fd

1. Create an XML file:

ram:> fd2pragma dopus5_lib.fd clib proto.h special 140
SourceFile: dopus5_lib.fd
Resultfile: dopus5.xml

2. generate ppc->68k crosscall stubs:

ram:> fd2pragma dopus5_lib.fd clib proto.h special 141
SourceFile: dopus5_lib.fd
Resultfile: dopus5.c

In the result file (dopus5.c) manually add including of interfaces/dopus5.h file and replace value "main_vectors" with "main_v1_vectors"

3. generate proto/interface/inline (i.e. includes) and vectors:

ram:> idltool -p -i -n -c dopus5.xml 

4. build dopus5.l.main

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build a test program (its a bit different compared to the previous test program because the name of the interface is a bit different (big/small letters, 5 at end, etc) :

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct Dopus5IFace *IDopus5;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDopus5=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDopus5->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}

Then hard-reboot is needed if you did the previous test with dopus5.l.main done from fd2trans as it is still in memory (and it can freeze the OS because of different code of the same stub libs), and then:

ram:> gcc -Iinclude testcase-native-fd2pragma.c 
ram:> a.out
ram:> Result=First part-Second part

It works as well!

Conclusion

Which method is better? In my opinion, the fdtrans one because:

1. You need less changes (no need to manually include interfaces/dopus5.h file).
2. The code of stubs from fdtrans look better and cleaner in comparison with the code of stubs from fd2pragma. For example even small things like comments saying what values in hex mean in your offset jump table is pretty nice (especially when you work manually with pragmas, where hex offsets are present).
3. Size of dopus.l.main done from fd2pragma output is larger in comparison with the size of dopus.l.main created from fdtrans output.

I want one single function to work

Now something interesting: you may want to check only if it all works from AmigaOS 4 and with your new lib.l.main and native AmigaOS includes, code and co. You then choose any "simple" function from library and want to make a simple AmigaOS 4 native test program for it. Let's say it will be the StrConcat() function from dopus5.library, which is described in pragma file as:

#pragma libcall DOpusBase StrConcat 47a 09803

and in dopus_lib.fd like:

StrConcat(s1,s2,len)(a0/a1,d0)

We then simply calculate:

1. 0x47a = 1146
2. 1146 / 6 = 191
3. 191 - 4 (i.e. 30/6 from beginning 5, but 30 is our first one, so -4) = 187

i.e. entry in jump table for StrConcat is 187. The entry in the dopus_lib.fd file for Strconcat is also 187 in the list of functions (+ header of .fd)

In other words you can build your .fd file from scratch, which will look like this:

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
aa()()
aa()()
aa()()
aa()()
.... 186 entries to fill the gap in jump table ....
StrConcat(s1,s2,len)(a0/a1,d0)
....
##end

Obviously, you can go the more elegant way and use .sfd instead, which have "reserved" keyword, so no need to put all those empty-functions like:

==id $Id: dopus5_lib.sfd,v 1.0 2013/03/29 09:24:08 noname Exp $ 
* "dopus5.library" 
==base _DopusBase 
==basetype struct Library * 
==libname dopus5.library 
==bias 30 
==public 
==include <exec/types.h> 
==reserve 186 
BOOL StrConcat(char * s1, char * s2, int len) (a0,a1,d0) 
==end

And when you will generate all necessary stuff via fd2grama/fdtrans/idltool from .fd or from .sfd, offsets will be respected. When you will call IDopus->StrConcat(blablab) from your code it will work because StrConcat will be found in the jump-table at the right offset.

Final words

As usual, there is nothing hardcore in here. This is just a short article about the problems which I encountered. I hope that it can be of any help for new developers. Thanks to Fredrik Wikstrom for hints about offsets calculation, Colin Wenzel for explaining how RamLib works when it scans paths for stubs, and many others for proofreading and grammar corrections.

Links

[1] Libraries and devices: http://wiki.amigaos.net/index.php/Libraries_and_Devices

[2] RamLib autodoc.

[3] fd2pragma guide from: http://aminet.net/dev/misc/fd2pragma.lha

[4] How to create PPC stubs for M68K libraries: http://thomas-rapp.homepage.t-online.de/ppclib.html