Exec Extended Memory

Overview

AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like "sizeof(struct Message)" the 32 bit nature is fused into his code.

This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.

Physical versus Virtual

A physical address of a memory block is implicitly defined by its position within the memory chips and the order in which the modules are inserted into the main board’s memory slots. They start at zero and go up to a specific maximum.

A virtual address, on the other hand, is what the CPU and hence the application program sees. They might be the same but as a general rule they are different. Virtual addresses are given on the fly but there is a rule that every memory cell must have a unique virtual address because all references to that cell are stored as the virtual address the application sees.

Physical Memory vs Virtual Address Space

Modern systems like the AmigaOne X1000 and upcoming models can take more than 4 gigabytes of memory, but so far, the extra memory will never be used. Even in a 4 gigabyte system, there is memory that will never be touched because there is just no free address. Unfortunately, every byte needs to have its own virtual address and no two bytes can have the same.

This is where extended memory objects enter the picture.

Extended Memory Object

Extended memory objects (ExtMem) are a means to access memory beyond the 2 gigabyte barrier by applications that are written to make use of them. In a nutshell, an extended memory object is a chunk of physical memory that exists in a nirvana state somewhere in the memory of the computer without a virtual address of its own. The memory cannot be accessed by anyone or anything in this state. In order to access it, an application must map part of the object into its own virtual address space. This mapping does make a part of the memory represented by the ExtMem object accessible in a memory window in the application’s own address space.

There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each mapping opens up a view into a part of the memory object and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.

ExtMem Object Mapping

A mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.

There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.

As an example, consider the following situation. We want to access byte 3000 of the ExtMem object. We created a mapping that has length 4000 and starts at offset 2000. The resulting address for our byte would be the base address of the mapping plus 1000, since the offset of the beginning is already at 2000.

Downsides of the ExtMem system

If you think now that this all sounds suspiciously like bank switching then you are correct. The method has been used way back in the Home computer age and even earlier. The Sinclair ZX Spectrum 128K was equipped with twice as much memory as the Z80 CPU could address; the upper 16k of the machine could be swapped between different chunks of the rest of the memory. Similarly, the Commodore 64 used bank switching to address a larger memory than its 6502 CPU could handle. It was the only possibility at the time to add more memory.

This method we employ now is basically the same but with a bit more added comfort.

Obviously, the method is a compromise. A real 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.

Limitations

Extended memory should only be used in the context of an application and should not be passed around. For example, you must never pass extended memory pointers to device drivers.

Pegasos 2 Warning
The extended memory feature currently does not work on the Pegasos II platform. Programmers are still encouraged to use ExtMem but may need to add an exception for the Pegasos II platform. Use IExpansion->GetMachineInfo() to verify which platform your code is executing on.

Who can benefit from ExtMem objects?

Every application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline or display a single frame that the user is working with.

Another example is RAM disk. The V54 RAM disk makes use of the ExtMem object interface allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated on-demand instead of ahead of time. This will make RAM disk have an even lower footprint on top of making it possible to store larger amounts of data than ever before.

It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.

So, as you can see, a good number of applications have a natural tendency to only access a very small subset of their memory at a given time. All of these are good candidate for using ExtMem objects to break the memory barrier.

Example

    /****************************************************************************/
    /*
    **  This example allocates blocks from the extmem space.
    **  The map() and unmap() allocate and free address space for the datablock->data.
    **
    **  Note that memory returned in extmem space can have addresses mapped
    **  above the first 2 gig.  (ie: High bit is always set).
    **  Be careful passing these addresses to other system and user functions,
    **  they may not use unsigned pointers and addresses may appear negative.
    **
    **  This example is written without test compilation or decent formatting,
    **  be careful of typos.  cjw.
    */
    /****************************************************************************/
    /*
    ** Example of a data block that can go into a list.
    */
    struct DataBlock   
    {
         struct MinNode      node;        /* minlist node */
         struct ExtMemIFace *iextmem;     /* extmem interface pointer */
         uint32              allocsize;   /* size of data allocated */
         APTR                data;        /* pointer to the mapped memory or 0 if unmapped */
    };
 
    /****************************************************************************/
    /*
    **  Block free function.
    */
    void free_block(struct DataBlock *block)
    {
       if( block )
       {
          if( block->iextmem )
          {
             IExec->FreeSysObject(ASOT_EXTMEM,block->iextmem);
          }
          IExec->FreeVec(block);
       }
 
       return;
    }
 
    /****************************************************************************/
    /*
    **  Deallocate all blocks in a list.
    */
    void free_block_list( struct List *list )
    {
       struct Node *n;
 
       /* free all blocks in a list */
       while(( n = IExec->RemTail(list) ))
       {
          free_block((APTR)n);
       }
 
       return;
    }
 
    /****************************************************************************/
    /*
    **  Allocate a single block.
    */
    struct DataBlock * allocate_block(uint32 size)
    {
       struct DataBlock *block, *result = NULL;
       uint64 size64;
 
       if((block = IExec->AllocVecTags(sizeof(*block),AVT_Type,MEMF_SHARED,
                                                      AVT_Lock,FALSE,
                                                      AVT_ClearWithValue,0,
                                                      TAG_END)))
       {
          size64 = size;  /* must be in a uint64 variable.*/
          block->iextmem = IExec->AllocSysObjectTags(ASOT_EXTMEM,
                                                     ASOEXTMEM_Size, &size64,
                                                     ASOEXTMEM_AllocationPolicy, EXTMEMPOLICY_IMMEDIATE,
                                                     TAG_END);
          if( block->iextmem )
          {
             block->allocsize = size;
             result = block;
          }
          else
          {
             free_block(block); /* free datablock struct on failure */
          }
       }
 
       return(result);
    }
 
    /****************************************************************************/
    /*
    **  Map address space for block data.
    */
    int32 map_block_data(struct DataBlock *block)
    {
       int32 result = FALSE;   /* default for failure */
 
       if( block )
       {
          if( block->iextmem )
          {
             block->data = block->iextmem->Map(0,block->allocsize, 0LL, 0);
             if( block->data )
             {
                result = TRUE;
             }
          }
       }
       return(result);
    }
 
    /****************************************************************************/
    /*
    **  Release address space for block data.
    */
    void unmap_block_data(struct DataBlock *block)
    {
       if( block )
       {
          if( block->iextmem )
          {
             if( block->data )
             {
                block->iextmem->Unmap(block->data, block->allocsize);
                block->data = NULL;   /* clear pointer */
             }
          }
       }
    }
    /****************************************************************************/