Anatomy of a SATA Device Driver

Introduction

This article describes the p5020sata.device and how it works. It is intended to be used in conjunction with the actual source code. The author will be going through this article at the AmiWest 2016 DevCon on October 8 and 9, 2016.

Initialization

Everything begins with a resident kickstart module. You can see a list of kickstart modules in the SYS:Kickstart/Kicklayout file. All of the modules listed in there will be loaded and linked when the system boots up. They become the system kernel and they are no loaded again until you do a cold reboot.

Creating a resident kickstart module is pretty simple. Here is the resident structure used by the p5020sata.device driver:

// Priority of resident kernel module
//
// Chosen to be lower than mounter.library which is at -45
// which means this device is initialized after mounter.library.
#define KMOD_PRIORITY -46
 
static struct Resident launch_restag USED =
{
	RTC_MATCHWORD,
	&launch_restag,
	(APTR)(&launch_restag + 1),
	RTF_NATIVE | RTF_COLDSTART,
	54,
	NT_UNKNOWN,
	KMOD_PRIORITY,
	DEVICE_NAME " launch",
	"$VER: " DEVICE_NAME " launch",
	launch_dev
};

The most important item to note is the choice of kickstart module priority from 127 to -128 in that order. Each kickstart module will be executed in the order of its priority. So if you have a kickstart module which dependent on another kickstart module you need to ensure the other module is executed first. In this case we need mounter.library to be available.

You can see the complete list of resident modules using Ranger in the Exec/Residents tab.

The entry point is named launch_dev and it looks like this:

APTR launch_dev(struct Library *lib, APTR seglist, struct ExecBase *exec)
{
	struct ExecIFace *IExec = (struct ExecIFace*)exec->MainInterface;
 
	IExec->Obtain();
 
	IExec->DebugPrintF("launch_dev() enter\n");
 
	struct SataDevice *device = create_sata_device(IExec);
 
	IExec->DebugPrintF("launch_dev() exit\n");
 
	IExec->Release();
 
	if (device == NULL)
	{
		return 0;
	}
	else
	{
		return (APTR)1;
	}
}

The create_sata_device() function is described below. The return value from the launch routine is either 0 for failure or 1 for success.

Testing Conundrum

Before we go any further we have to stop and think about testing. Performing a cold reboot each time we want to test our driver is going to become tedious very quickly.

The trick is to remove our kickstart module from the Kicklayout file and launch it manually instead. The downside is that you can't use your driver to boot your system or you are back into conundrum territory. Booting using another device driver or USB are two ways to make it work.

This is one way to load the kickstart module manually via a shell command:

int main()
{
	BPTR seg = IDOS->LoadSeg("p5020sata.device.kmod");
	if (seg == ZERO)
	{
		IExec->DebugPrintF("LoadSeg() failed\n");
		return RETURN_FAIL;
	}
 
	struct Resident *res = NULL;
 
	int32 count = IDOS->GetSegListInfoTags(seg,
		GSLI_ResidentStruct, &res,
		TAG_END);
 
	if (count != 1 || res == NULL)
	{
		IExec->DebugPrintF("GetSegListInfoTags() failed\n");
		IDOS->UnLoadSeg(seg);
		return RETURN_FAIL;
	}
 
	APTR obj = IExec->InitResident(res, seg);
 
	if (obj == NULL)
	{
		IExec->DebugPrintF("InitResident() failed\n");
		IDOS->UnLoadSeg(seg);
		return RETURN_FAIL;
	}
 
	struct MsgPort *port = IExec->AllocSysObjectTags(ASOT_PORT, TAG_END);
 
	struct IOStdReq *req = IExec->AllocSysObjectTags(ASOT_IOREQUEST,
		ASOIOR_Size, sizeof(struct IOStdReq),
		ASOIOR_ReplyPort, port,
		TAG_END);
 
	// Test SATA port 0.
	test_sata_port(req, 0);
 
	// Test SATA port 1.
	test_sata_port(req, 1);
 
	IExec->FreeSysObject(ASOT_IOREQUEST, req);
	IExec->FreeSysObject(ASOT_PORT, port);
 
	IExec->DebugPrintF("*** Test Completed ***\n");
	IExec->DebugPrintF("Reboot to test again\n");
	IExec->Wait(0);
 
	return 0;
}

The most important call above is IExec->InitResident() which will end up triggering our launch routine we wrote above. That initializes the device and gets it ready. We then use the usual Exec device interface (MsgPort and IOStdReq) to talk to our device and exercise it.

Creating the Exec Device

Once inside create_sata_device() we have to do some trickery to synchronize with mounter.library and friends. In order to avoid a race condition, we need to create ourselves and add our device manually. Only after that can we inform mounter.library that we ready to go.

struct SataDevice *create_sata_device(struct ExecIFace *IExec)
{
	struct SataDevice *sd = (struct SataDevice*)
		IExec->CreateLibrary(dev_createtags);
 
	if (sd == NULL)
	{
		IExec->DebugPrintF("CreateLibrary() failed\n");
		return NULL;
	}
 
	BOOL port0_ready = port_is_ready(sd->ports[0]);
	BOOL port1_ready = port_is_ready(sd->ports[1]);
 
	if (!port0_ready && !port1_ready)
	{
		IExec->DebugPrintF("no supported ATA devices detected\n");
		return NULL;
	}
 
	// From this point on the device is operational.
	IExec->AddDevice(&sd->device);
	...
}

After we have our device added to the system then mounter.library needs to be informed that there is something attached to each physical port. This is accomplished using the IMounter->AnnounceDeviceTags() which occurs on the SataPort Task context.

There isn't anything special about our device and we support all the usual vectors as described in Exec Device I/O.

Exec Device Unit Design Decision

Every Exec device driver has a design decision to make regarding what a Unit is. For a SATA driver it is fairly obvious if we ignore port multipliers so we'll do just that for now.

The next problem is removable media such as CDs and DVDs. Reusing the same unit number on AmigaOS has always been problematic. It was a great feature back in the days of floppy disks but the system has to somehow know when you replace the same disk.

The p5020sata.device currently supports dynamic unit numbers. The idea is that each time you change media you get a new (unused) unit number. If DOS locks are open on a media when it is ejected then that unit number remains used. However, this confuses many of the tools which assume unit numbers are not dynamic. I plan to redesign the interface to use fixed unit numbers again but that work has been deferred until after the DevCon has completed.

So here is how the driver works today:

1. There are two physical SATA ports and each port has a Task assigned to manage it.
2. Each port can have multiple unit numbers with each unit representing a mounted volume.

The software has been organized so that a SataPort is the physical SATA port and each SataUnit is assigned to one SataPort. A SataPort may have zero to many SataUnits attached to it.

The libata Connection

Disk device drivers are notoriously difficult to implement and test. There are hundreds, maybe even thousands of different devices which a user can plug in. Writing a driver from scratch is almost always the wrong decision. It will seem like a good idea at the time and the first version might even work. But then the bug reports will start flooding in as users try every cheap (i.e. not standards conforming) piece of hardware they can find on Amazon with free shipping.

Luckily, others have already suffered through the pain and decided to share their source code. In this case, I chose the libata package from the Linux world to do all the heavy lifting. This package is well proven, very compatible and has all the features I want from a disk device driver.

All you have to do then is figure out how libata works, add the necessary AmigaOS interface and all the rest of the operating system won't know the difference. Easy, right?

Where to Begin

There is no formula or how to for porting software from other operating systems. It is a skill that must be learned through hard work and experimentation. Some are much better than others at holding the myriad of esoteric facts about each OS in their heads and making sense of it all.

For a disk device driver (and many other device drivers), it all boils down to reading and writing registers and handling interrupts. So the first thing you want to do is peruse the datasheet and pick out the critical bits of information that you know will be in the driver you are going to port: register names, hardware limits and interrupt numbers are a great place to start.

With libata, you will quickly discover there are templates for each piece of hardware. For the P5020 that template lives in the file sata_fsl.c and you probably would have figured that out yourself if you searched using those register names, hardware limits and interrupt numbers from the datasheet.

There is also documentation on how libata works. The documentation is not bad but it is written by Linux programmers for Linux programmers. If you don't understand the internals of Linux it might hurt to read. So having the source code to scan while you are going through the docs is essential to understanding.

What to Work on First?

So now you have your device driver skeleton and you have libata and you are ready to steal some code.

Disk drivers need to know if a device is connected and then they need to query the device. That is where I started. I found the point of entry and that function is sata_fsl_probe() in the sata_fsl.c file mentioned earlier.

From there I basically implemented the API functions one at a time. Copy the libata source code from the Linux source tree and paste into the AmigaOS source tree.

The trick is to stay focused on the task at hand. Anything not related to probing a device and querying it was ignored for the first pass through. You want to discover the structure of the libata software. Reading about int he docs is one thing. Getting their source code to compile and run is completely another. You start small and slowly build up in complexity.

Others with more experience might just try the shotgun approach instead. That is, try to compile the whole libata source tree and hope you configure all the options you need to make it go. I decided against that kind of approach because a) you won't understand what you are porting and b) you will have a heck of time debugging such a large code base.

Both approaches are valid of course and there may be others. But I prefer taking baby steps and having working software along the way. That way, the people that are interested in the progress of my driver know exactly what it can do because they can just run it. I find the other approach leads to 90% done progress reports at each status report meeting and nobody has any working software to prove that number is right or wrong.

The BeginIO Vector

Zooming ahead in time we have our devices being recognized and initialized by libata during the AmigaOS device initialization sequence. The create_sata_device() function is executing which in turn triggers our dev_init() function (via IExec->CreateLibrary()) where the SataPort tasks are launched.

It wasn't that easy but let's pretend it was.

Now you have AmigaOS making all sorts of demands on your driver. The demands come in the form of I/O requests which is the way AmigaOS talks to device drivers. They all enter the driver via the dev_begin_io() function.

It is possible that any Task in the system is calling your dev_begin_io() function so be careful how it is implemented. There must not be any global data for example. Anything you do need is packaged up in the SataDevice' structure.

I started with the NSCMD_DEVICEQUERY command. This is usually the first thing your driver will be asked and it lists all the AmigaOS commands your driver supports. To test it, use your test suite you have been building along with the driver or a tool like Ranger which can query your device at this most basic level.

Then you are left with the decision on what to do next. Do you want to read and write data? Do you want to support disk changes? How about disk geometry queries?

What I decided was that Media Toolbox would be the first AmigaOS application I supported. This necessitated knowing what commands Medai Toolbox was sending to my driver so I added some debug code:

void dev_begin_io(struct DeviceManagerInterface *self, struct IOStdReq *req)
{
	struct SataDevice *sd = (struct SataDevice*)req->io_Device;
	struct SataUnit *su = (struct SataUnit*)req->io_Unit;
 
	struct SataPort *sp = su->port;  // May be NULL if disk removed.
 
	struct ExecIFace *IExec = sd->IExec;
 
	if (sp != NULL)
	{
		IExec->DebugPrintF("%lu:dev_begin_io() unit_num=%lu [%s]\n",
		  sp->port_num, su->unit_num, IExec->FindTask(0)->tc_Node.ln_Name);
		IExec->DebugPrintF("%lu:req=%p io_Command=0x%04lx\n",
		  sp->port_num, req, req->io_Command);
	}
	else
	{
		IExec->DebugPrintF("?:dev_begin_io() unit_num=%lu [%s]\n",
		  su->unit_num, IExec->FindTask(0)->tc_Node.ln_Name);
		IExec->DebugPrintF("?:req=%p io_Command=0x%04lx\n",
		  req, req->io_Command);
	}
 
	switch (req->io_Command)
	{
	...

To test, what I do is boot the system via some other method. Then I open a shell and run my test program which loads my driver and opens the Exec device. Then I run Media Toolbox and collect the debug output for analysis.

Over time, I slowly discovered which components and applications want which commands in order to function. Here is an incomplete table summarizing my discoveries:

Command	Component
CMD_READ (0x0002)	mounter.library CrossDOSFileSystem
CMD_WRITE (0x0003)	Media Toolbox save RDB
CMD_UPDATE (0x0004)	FastFileSystem needs this to write
CMD_CLEAR (0x0005)	Format
TD_MOTOR (0x0009)	FastFileSystem PartitionWizard
TD_FORMAT (0x000B)	Format
TD_CHANGESTATE (0x000E)	CDFileSystem
TD_GETDRIVETYPE (0x0012)	Media Toolbox
TD_ADDCLIENTINT (0x0014)	CDFileSystem
TD_REMCHANGEINT (0x00015)	CDFileSystem
TD_GETGEOMETRY (0x0015)	mounter.library
HD_SCSICMD (0x001C)	diskboot Media Toolbox CDFileSystem
NSCMD_DEVICEQUERY (0x4000)	All programs
NSCMD_TD_READ64 (0xC000)	NGFS FFS2 Media Toolbox mounter.library
NSCMD_TD_FORMAT64 (0xC003)	Format

Over time, each command was implemented one by one using libata to do the actual work of talking to the device.

Why Include SCSI Support?

For reasons lost to history, SCSI commands are still the preferred way of talking to CD, DVD, etc. devices. At first, I concentrated on hard drives only which meant I could ignore the SCSI support. However, libata is implemented in such a way that all commands are SCSI commands even if they are for hard drives only.

Here is a simplified summary of how a read command is executed:

1. Some program calls the IDOS->Read() function.
2. The Read() function is translated into a port vector call on the underlying file system. In the old days, it would have become a DOS packet.
3. The file system may (NGFS) or may not (FFS2) divide the read into file system sector-sized chunks and then issues each command to our driver.
4. Our driver takes each command and divides it into device sector-sized chunks and issues SCSI commands to the libata framework.
5. Each SCSI command is translated by libata into an ATA device command.
6. The ATA device command is finally sent to the attached device.
7. After the command completes it raises an interrupt.
8. The interrupt is handled and the various commands are unwound in reverse order.
9. Complete commands as required to satisfy the original IDOS->Read() call.

The way the driver is implemented now, the SCSI layer is still not there for read and write requests to hard drives but that is likely to change.

Why? Because libata works a lot better if everything is considered a SCSI command even if it doesn't need to be. Error handling is the key to understanding this complexity. As long as there are no errors, a device driver is pretty simple and it just chugs along happily. Add in errors and you have to decide whether to retry or just give up. Just giving up could be very expensive in terms of time lost. So libata is implemented to handle every device in every scenario and it is the SCSI layer that adds that functionality.

The reason I avoided SCSI in the first place was to get the driver working for hard drives as quickly as possible. When I added ATAPI support, then SCSI became absolutely necessary.

That's It?

At this point I suspect we will have likely ran out of time at the Dev Con. There are still dozens of topics to explore and, depending on the room, we can explore a few of them at the Dev Con directly.

For those reason this article, it will be changed into a full tutorial at some point. Please be patient.