Copyright (c) Hyperion Entertainment and contributors.

How to Build Stubs for 68k Libraries

From AmigaOS Documentation Wiki
Jump to navigation Jump to search

Author

Roman Kargin
Copyright (c) 2013 Roman Kargin
Proofreading and grammar corrections by the AmigaOS Wiki team.

Introduction

The idea for this tutorial came from when I was playing with Dopus5 source code and wanted to make one single function from the 68k version of dopus5.library to work from native AmigaOS 4 code. While we have all kind of auto generation tools to build all kind of protos/interfaces/inlines/stubs/etc, it still wasn't clear to me how to do what I needed.

Since the beginning a very helpful article was Thomas Rapp's How to create PPC stubs for M68K libraries where he shows how to use fd2pragma and IDLTool to build a jpeg.l.main stub from a 68k jpeg.library, as well as helpful answers from Fredrik Wikstrom. In the end, all this information was important to write down so I put everything into this tutorial. I hope other developers will find it useful.

68k->PowerPC Glue Stubs

While 68k libraries are old and almost all libraries on AmigaOS today are PowerPC native ones, there are still some old 68k libraries which are necessary and have no native PowerPC replacement. 68k->PowerPC glue stubs are necessary to be able to call 68k routines from PowerPC code in AmigaOS.

If you created native code which needs to call some old 68k library (i.e native aos4-ppc binary with available source code which will use 68k library), you can either use 68k-PowerPC glue stubs or use EmulateTags() and call your library functions directly. The latter makes it easy for your code to call both PowerPC and 68k native libraries with only a small bit of small overhead; just a few instructions. This greatly reduces the amount of work and will support both PowerPC and 68k modules transparently with no stub-files needed. But this only works when the PowerPC native code is available and when there are just a few functions in the library. Quite often, you just don't want to deal with all this because of the large number of functions in a library and the different ways of using them from the code. In this case, using 68k-PowerPC stubs is a simpler way.

For a better explanation of what a 68k-PowerPC stub is, refer to the Migration Guide. In short, if you have let's say the 68k version of dopus5.library and want to use it from native AmigaOS 4 binaries then you need to create dopus5.l.main. When code in your native binary needs to interface to a 68k library, the system (ramlib to be more precise) will automatically scan its library search path for the file "dopus5.l.main" and if found, it will open it and obtain the interface from there. That l.main file is pure non-startup code which contains the necessary stubs. You do not need to write the stubs from scratch manually. This is a job for different set of tools (fd2pragma, fdtrans and IDLTool). The library.l.main code consists of 2 files if you auto generate them: PowerPC->68k crosscall stubs and vectors. Of course you can write them from scratch manually, but its a pain and there are already tools which do it for you.

The name of the file in general does not matter. A library does not necessarily have to be named ".library" and could be named ".module" or whatever. Ramlib will search for the first "." in the library name, starting from the end, the first part up to and including the dot plus one more character, usually the 'l' in library, is used, another dot is appended, then the interface name is added. It then calls the exec function.

For example, if the interface name is 'main' and the library name is 'foobar' then this is what it looks for: 

foobar.library  -> foobar.l.main
foobar.dummy    -> foobar.d.main
foobar.module   -> foobar.m.main

Things to note:

1. Because of the way RamLib parses glue-stubs names, you can't call your glue stubs say "foobar.library" as "foobar.library.main". It is understandable from the explanation above, but it can be misleading when you run for example "snoopy" and can see how it tries to search for "foobar.library.main", but it tries to search for a file, but for "resident". The file ramlib will look for is only foobar.l.main when the main library is called as foobar.library.

2. You can't put your glue stub files in directories other than PROGDIR:, PROGDIR:Libs or LIBS: as RamLib will not find them. If you have let's say a directory "modules" where you store some of your plugins which in reality are libraries then you can't put your glue stub files in the same "modules" directory.

pragmas/fd/sfd/protos

Often, old 68k libraries were not done with GCC. On old 68k machines, SAS/C was the popular C compiler and pure assembler was even more popular for everyday needs. So many of those old 68k libraries were provided with includes for SAS/C only or for assembler (like .i files) or a mix of the two. You may have such library with which you have only a pragma file and a bunch of proto files for functions which are present in more than one proto file (i.e. no .fd file at all, no single include with all protos). The first thing to do is to create a single proto file from those different proto files you have in different places, manually make your .fd file from pragma (check all offsets and way (???) how they are placed in pragma file, and in the same order put them to your new .fd). Then when you have only one proto file and one .fd file, you can generate a .sfd file and an .xml one with fd2pragma. Then generate from it all includes/vectors/stubs and only after that you can generate lib.l.main :) Usually its not that hard, and most old libraries have .fd files and a single proto include file, but sometimes you will have to do it for difficult libs, so you will know what to do.

So all we need to know now is that libraries have a jump table. Jump tables are a list of library functions at specific offsets. Offsets can't be changed because they are function offsets in the library that your code will call. Be it with the old SAS/C, an assembler or the new GCC on AmigaOS 4, if we work with a 68k library and create any fd/proto files from scratch then the offsets should always be the same.

For example you have dopus_pragmas.h file and dopus.fd files (fd files are a standard way to describe the interface for libraries) and dopus_pragmas entries look like this: 

#pragma libcall DOpusBase RemovedFunc0 1e 0
#pragma libcall DOpusBase Random 24 001
#pragma libcall DOpusBase Atoh 2a 0802
....

It means that the first "user" library function in a jump table starts at offset 0x1e (30) and then multiply by 6 for every function. So first function is at 30 (0x1e), second one is at 36 (0x24), the next one is at 42 (0x2a) and so on for all the functions. That means that if for example function Atoh() in the jump table is placed at offset 0x2a, then every single code and includes file which we will generate for AmigaOS 4 should respect that jump-table offset.

If we look at the dopus_lib.fd file we see: 

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
RemovedFunc0()()
Random(limit)(d0)
*
Atoh(str,len)(a0,d0)
....

This means that public functions start from offset 30 (bias 30), and they are placed in the same order as they are in the pragma file. So if you change the order of function names in the FD file you will end calling a function instead of another one, which of course means crashes, bugs and problems.

So respect offsets in the jump table.

Note that .sfd files work like .fd files but with more possibilities. You won't find them in old 68k dev archives of libraries as its something new. They contain a lot more infomation than .fd and are easier to use. But more or less they follow the same logic, the same respect for offsets, etc. To see a full description of .fd and .sfd formats check fd2pragma guide.

Now What?

We have 2 methods to build lib.l.main and required includes. But for both of them you need .fd file as a minimum (if you have .sfd its even better), and a proto file which will describe all (or not all) your functions. While .fd/.sfd should be done by the original developer, you still can make them from scratch if you have a pragma file with lib calls (you make this file with the same order and the same offsets). If you don't have one proto file, but many in different includes, then you can make your own one, just copy one by one all the protos from all the include files, while all your FD entries will not have necessary proto descriptions in proto file (btw, if you don't have proto for some functions, then entries from the fd/sfd file will be reserved when you will auto generate OS4 code/includes, so respect offsets).

Let's say we have dopus_lib.fd and only protos in different includes in different places. We then create proto file from all those includes (in referring to the dopus_lib.fd), and then auto-tools time coming. As I say we have 2 ways:

First method: fdtrans + sfd

1. generate a .sfd file: 

ram:> fd2pragma dopus5_lib.fd clib our_proto.h special 112
SourceFile: dopus5_lib.fd
ResultFile: dopus5_lib.sfd

2. generate xml and ppc->68k crosscall stubs via fdtrans: 

ram:> fdtrans dopus5_lib.sfd -x -s

Change manually in the crosscall-stubs file (dopus5.c) "main_vectors" in "main_v1_vectors", as it seems some typo between different tools. No other changes is needed, fdtrans automatically insert including of interface (while fd2pragma will skip it, see later).

3. generate proto/interface/inline (i.e. includes) and vectors: 

ram:> idltool -p -i -n -c dopus5.xml

4. build dopus5.l.main 

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build a test program to use one function from dopus5.library, here StrConcat() 

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct DOpusIFace *IDOpus;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDOpus=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDOpus->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}
ram:> gcc -Iinclude testcase-native-fdtrans.c 
ram:> a.out
ram:> Result=First part-Second part

It works!

Second method: fd2pragma + fd

1. Create an XML file: 

ram:> fd2pragma dopus5_lib.fd clib proto.h special 140
SourceFile: dopus5_lib.fd
Resultfile: dopus5.xml

2. generate ppc->68k crosscall stubs: 

ram:> fd2pragma dopus5_lib.fd clib proto.h special 141
SourceFile: dopus5_lib.fd
Resultfile: dopus5.c

In the result file (dopus5.c) manually add including of interfaces/dopus5.h (???) and replace value "main_vectors" with "main_v1_vectors"

3. generate proto/interface/inline (i.e. includes) and vectors: 

ram:> idltool -p -i -n -c dopus5.xml

4. build dopus5.l.main 

ram:> gcc -Iinclude -nostartfiles dopus5.c -o dopus5.l.main

5. build a test program (its a bit different compared to the previous test program because the name of the interface is a bit different (big/small letters, 5 at end, etc) : 

#include <proto/exec.h>
#include <proto/dopus5.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
 
struct Library *DOpusBase;
struct Dopus5IFace *IDopus5;
 
void main(int argc,char **argv)
{
	char buffer[256] = "First part-";
	char *concat = "Second part";
 
	// Need dopus library
	if (!(DOpusBase=IExec->OpenLibrary("PROGDIR:dopus5.library",1L))) {
		printf("cant library");
		exit(10);
	};
	if (!(IDopus5=IExec->GetInterface((struct Library *)DOpusBase,"main",1L,NULL))) {
		printf("cant interface");
	   exit(10);
   };
 
	IDopus5->StrConcat(buffer, concat, 36);
	printf("Result=%s\n", buffer);
 
	IExec->CloseLibrary(DOpusBase);
}

Then hard-reboot is needed if you did the previous test with dopus5.l.main done from fd2trans as it is still in memory (and it can freeze the OS because of different code of the same stub libs), and then: 

ram:> gcc -Iinclude testcase-native-fd2pragma.c 
ram:> a.out
ram:> Result=First part-Second part

It works as well!

Conclusion

Which method is better? In my opinion, the fdtrans one because:

  1. You need less changes (no need to manually include interfaces/dopus5.h file).
  2. The code of stubs from fdtrans look better and cleaner in comparison with the code of stubs from fd2pragma. For example even small things like comments saying what values in hex mean in your offset jump table is pretty nice (especially when you work manually with pragmas, where hex offsets are present).
  3. Size of dopus.l.main done from fd2pragma output is larger in comparison with the size of dopus.l.main created from fdtrans output.

I want one single function to work

Now something intersting : you for example want to just check if it all will works at all for you from os4 and with your new lib.l.main and native os4 includes, code and co. You then choice any "simply" function from library and want to make a simple os4 native test case for it. Let's say it will be StrConcat() function from dopus5.library, which is described in pragma file as: 

#pragma libcall DOpusBase StrConcat 47a 09803

and in dopus_lib.fd like: 

StrConcat(s1,s2,len)(a0/a1,d0)

We then simply calc:

1. 0x47a = 1146 2. 1146 / 6 = 191 3. 191 - 4 (i.e. 30/6 from begining 5, but 30 is our first one, so -4) = 187

I.e. entry in jump table for StrConcat is 187. And entry in the dopus_lib.fd file for Strconcat also 187 in the list of functions (+ header of .fd)

In other words you can build your .fd file from scratch, which will looks like this: 

* "dopus.library"                                             
##base _DOpusBase
##bias 30
*                                                             
* Support routines for Directory Opus and associated programs 
* (c) Copyright 1995 Jonathan Potter                          
*                                                             
##public
aa()()
aa()()
aa()()
aa()()
.... 186 entrys to fill the gap in jump table ....
StrConcat(s1,s2,len)(a0/a1,d0)
....
##end

You of coures can go more elegant way and use .sfd instead, which have "reserved" keyword, so no need to put all those empty-funct. I.e.: 

==id $Id: dopus5_lib.sfd,v 1.0 2013/03/29 09:24:08 noname Exp $ 
* "dopus5.library" 
==base _DopusBase 
==basetype struct Library * 
==libname dopus5.library 
==bias 30 
==public 
==include <exec/types.h> 
==reserve 186 
BOOL StrConcat(char * s1, char * s2, int len) (a0,a1,d0) 
==end

And when you will generate all necessary stuff via fd2grama/fdtrans/idltool from .fr or from .sfd, offsets will be respected, and when you will call from your code IDopus->StrConcat(blablab);, it will works because code will find out StrConcat in jump-table by right offset and will works.

Final words

As usual, there is nothing hardcore in here. This is just a short article about the problems which I encountered. I hope that it can be any help for new developers. In end thanks to Fredrik Wikstrom for hints about offsets calculation, Colin Wenzel for explaining of how RamLib works when scan paths for stubs, and many others for proofreading and grammar corrections.

Links

[1] OS4 libraries and devices: http://wiki.amigaos.net/index.php/Libraries_and_Devices

[2] RamLib autodoc.

[3] fd2pragma guide from: http://aminet.net/dev/misc/fd2pragma.lha

[4] How to create PPC stubs for M68K libraries: http://thomas-rapp.homepage.t-online.de/ppclib.html

Retrieved from "http://wiki.amigaos.net/w/index.php?title=How_to_Build_Stubs_for_68k_Libraries&oldid=5557"