How to Build Stubs for 68k Libraries

Author

Roman Kargin
Copyright (c) 2013 Roman Kargin
Proofreading and grammar corrections by the AmigaOS Wiki team.

Introduction

The idea for this tutorial came from when I was playing with Dopus5 source code and wanted to make one single function from the 68k version of dopus5.library to work from native AmigaOS 4 code. While we have all kind of auto generation tools to build all kind of protos/interfaces/inlines/stubs/etc, it still wasn't clear to me how to do what I needed.

Since the beginning a very helpful article was Thomas Rapp's How to create PPC stubs for M68K libraries where he shows how to use fd2pragma and IDLTool to build a jpeg.l.main stub from a 68k jpeg.library, as well as helpful answers from Fredrik Wikstrom. In the end, all this information was important to write down so I put everything into this tutorial. I hope other developers will find it useful.

68k->PowerPC Glue Stubs

While 68k libraries are old and almost all libraries on AmigaOS today are PowerPC native ones, there are still some old 68k libraries which are necessary and have no native PowerPC replacement. 68k->PowerPC glue stubs are necessary to be able to call 68k routines from PowerPC code in AmigaOS.

If you created native code which needs to call some old 68k library (i.e native aos4-ppc binary with available source code which will use 68k library), you can either use 68k-PowerPC glue stubs or use EmulateTags() and call your library functions directly. The latter makes it easy for your code to call both PowerPC and 68k native libraries with only a small bit of small overhead; just a few instructions. This greatly reduces the amount of work and will support both PowerPC and 68k modules transparently with no stub-files needed. But this only works when the PowerPC native code is available and when there are just a few functions in the library. Quite often, you just don't want to deal with all this because of the large number of functions in a library and the different ways of using them from the code. In this case, using 68k-PowerPC stubs is a simpler way.

For a better explanation of what a 68k-PowerPC stub is, refer to the Migration Guide. In short, if you have let's say the 68k version of dopus5.library and want to use it from native AmigaOS 4 binaries then you need to create dopus5.l.main. When code in your native binary needs to interface to a 68k library, the system (ramlib to be more precise) will automatically scan its library search path for the file "dopus5.l.main" and if found, it will open it and obtain the interface from there. That l.main file is pure non-startup code which contains the necessary stubs. You do not need to write the stubs from scratch manually. This is a job for different set of tools (fd2pragma, fdtrans and IDLTool). The library.l.main code consists of 2 files if you auto generate them: PowerPC->68k crosscall stubs and vectors. Of course you can write them from scratch manually, but its a pain and there are already tools which do it for you.

The name of the file in general does not matter. A library does not necessarily have to be named ".library" and could be named ".module" or whatever. Ramlib will search for the first "." in the library name, starting from the end, the first part up to and including the dot plus one more character, usually the 'l' in library, is used, another dot is appended, then the interface name is added. It then calls the exec function.

For example, if the interface name is 'main' and the library name is 'foobar' then this is what it looks for:

foobar.library  -> foobar.l.main
foobar.dummy    -> foobar.d.main
foobar.module   -> foobar.m.main

Things to note:

1. Because of the way RamLib parses glue-stubs names, you can't call your glue stubs say "foobar.library" as "foobar.library.main". It is understandable from the explanation above, but it can be misleading when you run for example "snoopy" and can see how it tries to search for "foobar.library.main", but it tries to search for a file, but for "resident". The file ramlib will look for is only foobar.l.main when the main library is called as foobar.library.

2. You can't put your glue stub files in directories other than PROGDIR:, PROGDIR:Libs or LIBS: as RamLib will not find them. If you have let's say a directory "modules" where you store some of your plugins which in reality are libraries then you can't put your glue stub files in the same "modules" directory.

pragmas/fd/sfd/protos

Old 68k libraries are usually not something which were done with GCC. On old 68k machines, SAS/C was the popular one for C and pure assembler was even more popular for everyday needs. And so, many of those old 68k libraries provide developers only includes for SAS/C or only for assembler (like .i files) or a mix, but mix of those oldies. You may have such a library where you have just a pragma file and bunch of protos for function which are present in more than one proto file (i.e. no .fd file at all, no single include with all protos). Then you need to first, create a single proto file from those different proto files in all the places, make manually your .fd file from pragma (check all offsets and way how they placed in pragma file, and in the same order put them to your new .fd), and then , when you have only single proto file and .fd file, you can generate .sfd by fd2pragma, from that do .xml, then generate from it all includes/vectors/stubs and only after you can generate lib.l.main :) Usually its not that hard, and most of even old libraries have and .fd files , and single proto include file, but some time you can force with "hard" libs, so you will know what to do.

So.. All that we need to know now is that libraries have a jump table. Jump tables have a list of library functions at specific offsets. Offsets can't be changed because they are offsets in a jump table that your code needs to call the functions in a library. Be it old SAS/C or assembler or new GCC on AmigaOS 4, if we work with a 68k library and create any fd/proto files from scratch then the offsets should be always be the same.

For example you have dopus_pragmas.h file and dopus.fd files (fd files are a standard way to describe the interface for libraries) and dopus_pragmas entries look like this:

#pragma libcall DOpusBase RemovedFunc0 1e 0
#pragma libcall DOpusBase Random 24 001
#pragma libcall DOpusBase Atoh 2a 0802
....

It mean that first "user" library function in a jump table starts at offset 0x1e (30) and then multiply by 6 for every function. So first function is at 30 (0x1e), second one is at 36 (0x24), the next one is at 42 (0x2a) and so on for all the functions. That means that if for example function Atoh() in the jump table is placed at offset 0x2a, then every single code and includes file which we will generate for AmigaOS 4 should respect that jump-table offset.

If we look at the dopus_lib.fd file we will see:

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
RemovedFunc0()()
Random(limit)(d0)
*
Atoh(str,len)(a0,d0)
....

What mean that public functions starts from offset 30 (bias 30), and they placed in the same necessary order as they placed in the pragma file. I.e. if you for example will just put Atoh() from FD file at first place, and RemovedFunc0() at place of Atoh(), generate all your files for os4, and build native binary to use library: then instead of Atoh() you will have RemovedFunc0() and instead of RemovedFunc0() you will have Atoh(), what of course mean crashes, bugs and problems.

So, respect offsets in jump table is must.

Need to add, that also .sfd kind of files present (you will not find them usually in old 68k dev archives of libraryes as its something fresh in compare with .fd), and they hold a lot more infomation in compare with .fd and easy to operate. To see full description of .fd and .sfd formats check fd2pragma guide. But in general they more or less the same by logic, the same respect for offsets, etc.

Now What?

We have 2 ways to build lib.l.main and necessary includes. But for both of them you need .fd file as minimum (if you have .sfd its even better), and proto file which will describe all (or not all) your functions. While .fd/.sfd its something which should be done by developers initally, you still can make them from scratch just if you have pragma file with lib calls (you make that files in the same order by the same offsets). If you didn't have one proto file, but many in different includes, then you just can make your own one, just string by string grab from all the include files all the protos, while all your FD entrys will not have necessary proto descriptions in proto file (btw, if you will not have proto for some function, then entry from fd/sfd file will be reserved when you will auto generate os4 code/includes, so respect offsets).

Let's say we have dopus_lib.fd and nothing more nowhere. Only protos in different includes in diffrent places. We then string by string create proto file from all those includes (in refering to the dopus_lib.fd), and then auto-tools time coming. As i say we have 2 ways:

First way: fdtrans + sfd

1. generate .sfd file:

ram:> fd2pragma dopus5_lib.fd clib our_proto.h special 112
SourceFile: dopus5_lib.fd
ResultFile: dopus5_lib.sfd

2. generate xml and ppc->68k crosscall stubs via fdtrans:

ram:> fdtrans dopus5_lib.sfd -x -s

Change manually in the crosscall-stubs file (dopus5.c) "main_vectors" on "main_v1_vectors", as it seems some typo beetwen different tools. No other changes need it, fdtrans automatically insert including of interface (while fd2pragma will skip it, see later).

3. generate proto/interface/inline (i.e. includes) and vectors:

ram:> idltool -p -i -n -c dopus5.xml 

4. build dopus5.l.main

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build test case which will use one of functions from dopus5.library called StrConcat()

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct DOpusIFace *IDOpus;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDOpus=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDOpus->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}

ram:> gcc -Iinclude testcase-native-fdtrans.c 
ram:> a.out
ram:> Result=First part-Second part

It works !

Second way: fd2pragma + fd

1. Create xml:

ram:> fd2pragma dopus5_lib.fd clib proto.h special 140
SourceFile: dopus5_lib.fd
Resultfile: dopus5.xml

2. generate ppc->68k crosscall stubs:

ram:> fd2pragma dopus5_lib.fd clib proto.h special 141
SourceFile: dopus5_lib.fd
Resultfile: dopus5.c

In result file (dopus5.c) manually add including of interfaces/dopus5.h and replace value "main_vectors" on "main_v1_vectors"

3. generate proto/interface/inline (i.e. includes) and vectors:

ram:> idltool -p -i -n -c dopus5.xml 

4. build dopus5.l.main

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build test case (its a bit different in compare with one from fdtrans, because names of iface a bit different (big/small letters, 5 at end, etc) :

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct Dopus5IFace *IDopus5;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDopus5=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDopus5->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}

Then hard-reboot is need it in case you already run previous test case on dopus5.l.main done from fd2trans and which still in memory (as it can freeze the OS hard because of different code of the same stub libs), and then:

ram:> gcc -Iinclude testcase-native-fd2pragma.c 
ram:> a.out
ram:> Result=First part-Second part

It works as well !

Conclusion

What way is better? In my opinion, the fdtrans one because:

1. You need to change less (no need to manually include interfaces/dopus5.h file).
2. The code of stubs from fdtrans look better and cleaner in comparison with the code of stubs from fd2pragma. For example even small things like comments saying what value in hex means in your offset jump table is pretty nice (expecially when you work manually with pragmas, where hex offsets are present).
3. Size of dopus.l.main done from fd2pragma output is larger in comparison with the size of dopus.l.main created from fdtrans output.

I want one single function to work

Now something intersting : you for example want to just check if it all will works at all for you from os4 and with your new lib.l.main and native os4 includes, code and co. You then choice any "simply" function from library and want to make a simple os4 native test case for it. Let's say it will be StrConcat() function from dopus5.library, which is described in pragma file as:

#pragma libcall DOpusBase StrConcat 47a 09803

and in dopus_lib.fd like:

StrConcat(s1,s2,len)(a0/a1,d0)

We then simply calc:

1. 0x47a = 1146 2. 1146 / 6 = 191 3. 191 - 4 (i.e. 30/6 from begining 5, but 30 is our first one, so -4) = 187

I.e. entry in jump table for StrConcat is 187. And entry in the dopus_lib.fd file for Strconcat also 187 in the list of functions (+ header of .fd)

In other words you can build your .fd file from scratch, which will looks like this:

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
aa()()
aa()()
aa()()
aa()()
.... 186 entrys to fill the gap in jump table ....
StrConcat(s1,s2,len)(a0/a1,d0)
....
##end

You of coures can go more elegant way and use .sfd instead, which have "reserved" keyword, so no need to put all those empty-funct. I.e.:

==id $Id: dopus5_lib.sfd,v 1.0 2013/03/29 09:24:08 noname Exp $ 
* "dopus5.library" 
==base _DopusBase 
==basetype struct Library * 
==libname dopus5.library 
==bias 30 
==public 
==include <exec/types.h> 
==reserve 186 
BOOL StrConcat(char * s1, char * s2, int len) (a0,a1,d0) 
==end

And when you will generate all necessary stuff via fd2grama/fdtrans/idltool from .fr or from .sfd, offsets will be respected, and when you will call from your code IDopus->StrConcat(blablab);, it will works because code will find out StrConcat in jump-table by right offset and will works.

Final words

As usual, there is nothing hardcore in here. This is just a short article about the problems which I encountered. I hope that it can be any help for new developers. In end thanks to Fredrik Wikstrom for hints about offsets calculation, Colin Wenzel for explaining of how RamLib works when scan paths for stubs, and many others for proofreading and grammar corrections.

Links

[1] OS4 libraries and devices: http://wiki.amigaos.net/index.php/Libraries_and_Devices

[2] RamLib autodoc.

[3] fd2pragma guide from: http://aminet.net/dev/misc/fd2pragma.lha

[4] How to create PPC stubs for M68K libraries: http://thomas-rapp.homepage.t-online.de/ppclib.html