Exec Extended Memory - Revision history

Costel Mincea: Text replacement - "" to ""

2025-01-26T19:09:20Z

Text replacement - "<syntaxhighlight>" to "<syntaxhighlight lang="C" line>"

← Older revision		Revision as of 19:09, 26 January 2025
Line 57:		Line 57:
	= Example =		= Example =

	<syntaxhighlight>		<syntaxhighlight lang="C" line>
	/****************************************************************************/		/****************************************************************************/
	/*		/*

Steven Solie at 01:46, 20 October 2016

2016-10-20T01:46:53Z

← Older revision		Revision as of 01:46, 20 October 2016
Line 63:		Line 63:
	** The map() and unmap() allocate and free address space for the datablock->data.		** The map() and unmap() allocate and free address space for the datablock->data.
	**		**
	** Note that memory returned in extmem space ~~has~~ addresses mapped ~~above the~~		** Note that memory returned in extmem space can have addresses mapped
	** first 2 gig. (ie: High bit is always set).		** above the first 2 gig. (ie: High bit is always set).
	** Be ~~carefull~~ passing these addresses to other system and user functions,		** Be careful passing these addresses to other system and user functions,
	** they may not use unsigned pointers and addresses may appear negative.		** they may not use unsigned pointers and addresses may appear negative.
	**		**
	** This example is written without test compilation or decent formatting,		** This example is written without test compilation or decent formatting,
	** be ~~carefull~~ of typos. cjw.		** be careful of typos. cjw.
	*/		*/
	/****************************************************************************/		/****************************************************************************/

Steven Solie: /* Downsides of the ExtMem system */

2015-11-12T19:53:43Z

Downsides of the ExtMem system

← Older revision		Revision as of 19:53, 12 November 2015
Line 38:		Line 38:

	Obviously, the method is a compromise. A ''real'' 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.		Obviously, the method is a compromise. A ''real'' 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.

			= Limitations =

			Extended memory should only be used in the context of an application and should not be passed around. For example, you must never pass extended memory pointers to device drivers.

	{{Note\|title=Pegasos 2 Warning\|text=The extended memory feature currently does not work on the Pegasos II platform. Programmers are still encouraged to use ExtMem but may need to add an exception for the Pegasos II platform. Use IExpansion->GetMachineInfo() to verify which platform your code is executing on.}}		{{Note\|title=Pegasos 2 Warning\|text=The extended memory feature currently does not work on the Pegasos II platform. Programmers are still encouraged to use ExtMem but may need to add an exception for the Pegasos II platform. Use IExpansion->GetMachineInfo() to verify which platform your code is executing on.}}

Steven Solie: /* Downsides of the ExtMem system */

2015-09-02T15:19:00Z

Downsides of the ExtMem system

← Older revision		Revision as of 15:19, 2 September 2015
Line 38:		Line 38:

	Obviously, the method is a compromise. A ''real'' 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.		Obviously, the method is a compromise. A ''real'' 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.

			{{Note\|title=Pegasos 2 Warning\|text=The extended memory feature currently does not work on the Pegasos II platform. Programmers are still encouraged to use ExtMem but may need to add an exception for the Pegasos II platform. Use IExpansion->GetMachineInfo() to verify which platform your code is executing on.}}

	= Who can benefit from ExtMem objects? =		= Who can benefit from ExtMem objects? =

Steven Solie: /* Who can benefit from ExtMem objects? */

2015-09-02T15:15:55Z

Who can benefit from ExtMem objects?

← Older revision		Revision as of 15:15, 2 September 2015
Line 43:		Line 43:
	Every application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline or display a single frame that the user is working with.		Every application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline or display a single frame that the user is working with.

	Another example is RAM disk. ~~Plans~~ ~~are currently underway to update the~~ RAM disk ~~to make~~ use of the ExtMem object interface, allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated ''on-demand'' instead of ahead of time. This will make RAM disk have an even lower footprint on top of making it possible to store larger amounts of data than ever before.		Another example is RAM disk. The V54 RAM disk makes use of the ExtMem object interface allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated ''on-demand'' instead of ahead of time. This will make RAM disk have an even lower footprint on top of making it possible to store larger amounts of data than ever before.

	It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.		It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.

Eldee Stephens: removed extra white space in code example

2015-09-02T13:22:42Z

removed extra white space in code example

← Older revision		Revision as of 13:22, 2 September 2015
Line 52:		Line 52:

	<syntaxhighlight>		<syntaxhighlight>

	/****************************************************************************/		/****************************************************************************/
	/*		/*
Line 66:		Line 65:
	** be carefull of typos. cjw.		** be carefull of typos. cjw.
	*/		*/




	/****************************************************************************/		/****************************************************************************/



	/*		/*
	** Example of a data block that can go into a list.		** Example of a data block that can go into a list.
Line 79:		Line 71:
	struct DataBlock		struct DataBlock
	{		{
	struct MinNode node; /* minlist node */		struct MinNode node; /* minlist node */
	struct ExtMemIFace iextmem; / extmem interface pointer */		struct ExtMemIFace iextmem; / extmem interface pointer */
	uint32 allocsize; /* size of data allocated */		uint32 allocsize; /* size of data allocated */
	APTR data; /* pointer to the mapped memory or 0 if unmapped */		APTR data; /* pointer to the mapped memory or 0 if unmapped */
	};		};




	/****************************************************************************/		/****************************************************************************/



	/*		/*
	** Block free function.		** Block free function.
Line 108:		Line 94:
	return;		return;
	}		}





	/****************************************************************************/		/****************************************************************************/




	/*		/*
	** Deallocate all blocks in a list.		** Deallocate all blocks in a list.
Line 125:		Line 103:
	struct Node *n;		struct Node *n;

	/*		/* free all blocks in a list */
	** free all blocks in a list.
	*/
	while(( n = IExec->RemTail(list) ))		while(( n = IExec->RemTail(list) ))
	{		{
Line 135:		Line 111:
	return;		return;
	}		}





	/****************************************************************************/		/****************************************************************************/





	/*		/*
	** Allocate a single block.		** Allocate a single block.
Line 154:		Line 121:
	uint64 size64;		uint64 size64;

	if((block = IExec->AllocVecTags(sizeof(*block),AVT_Type,MEMF_SHARED,~~AVT_Lock,FALSE,AVT_ClearWithValue,0,TAG_END)))~~		if((block = IExec->AllocVecTags(sizeof(*block),AVT_Type,MEMF_SHARED,
			AVT_Lock,FALSE,
			AVT_ClearWithValue,0,
			TAG_END)))
	{		{
	size64 = size; /* must be in a uint64 variable.*/		size64 = size; /* must be in a uint64 variable.*/

	block->iextmem = IExec->AllocSysObjectTags(ASOT_EXTMEM,		block->iextmem = IExec->AllocSysObjectTags(ASOT_EXTMEM,
	ASOEXTMEM_Size, &size64,		ASOEXTMEM_Size, &size64,
	ASOEXTMEM_AllocationPolicy, EXTMEMPOLICY_IMMEDIATE,		ASOEXTMEM_AllocationPolicy, EXTMEMPOLICY_IMMEDIATE,
	TAG_END);		TAG_END);
	if( block->iextmem )		if( block->iextmem )
	{		{
Line 175:		Line 144:
	return(result);		return(result);
	}		}





	/****************************************************************************/		/****************************************************************************/




	/*		/*
	** Map address space for block data.		** Map address space for block data.
Line 197:		Line 158:
	{		{
	block->data = block->iextmem->Map(0,block->allocsize, 0LL, 0);		block->data = block->iextmem->Map(0,block->allocsize, 0LL, 0);

	if( block->data )		if( block->data )
	{		{
Line 204:		Line 164:
	}		}
	}		}

	return(result);		return(result);
	}		}




	/****************************************************************************/		/****************************************************************************/




	/*		/*
	** Release address space for block data.		** Release address space for block data.
Line 233:		Line 185:
	}		}
	}		}


	/****************************************************************************/		/****************************************************************************/
	/* EOF */
	</syntaxhighlight>		</syntaxhighlight>

Steven Solie at 03:02, 2 September 2015

2015-09-02T03:02:53Z

← Older revision		Revision as of 03:02, 2 September 2015
Line 1:		Line 1:
	= Overview =		= Overview =

	AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like ~~<syntaxhighlight>~~sizeof(struct Message)~~</syntaxhighlight>~~ the 32 bit nature is fused into his code.		AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like "sizeof(struct Message)" the 32 bit nature is fused into his code.

	This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.		This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.

Steven Solie at 03:00, 2 September 2015

2015-09-02T03:00:37Z

Steven Solie at 02:18, 2 September 2015

2015-09-02T02:18:29Z

Show changes

Steven Solie at 02:16, 2 September 2015

2015-09-02T02:16:25Z

← Older revision		Revision as of 02:16, 2 September 2015
Line 23:		Line 23:
	There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time, and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each mappings opens up a view into a part of the memory object, and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.		There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time, and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each mappings opens up a view into a part of the memory object, and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.

			[[File:fig2-ext-mem.jpg\|frame\|center\|ExtMem Object Mapping]]
⚫	~~fig2A~~ mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.

		⚫	A mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.

	There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset, just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.		There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset, just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.

← Older revision		Revision as of 03:00, 2 September 2015
Line 1:		Line 1:
	= Overview =		= Overview =

	AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like ~~“sizeof~~(struct Message)”, the 32 bit nature is fused into his code.		AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like <syntaxhighlight>sizeof(struct Message)</syntaxhighlight> the 32 bit nature is fused into his code.

	This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.		This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.

	= Physical ~~Vs.~~ Virtual =		= Physical versus Virtual =

	A physical address of a memory block is implicitly defined by its position within the memory chips, and the order in which the modules are inserted into the ~~mainboard’s~~ memory slots. They start at zero and go up to a specific maximum.		A physical address of a memory block is implicitly defined by its position within the memory chips and the order in which the modules are inserted into the main board’s memory slots. They start at zero and go up to a specific maximum.

	A virtual address, on the other hand, is what the CPU and hence the application program sees. They might be the same, but as a general rule, they are different. Virtual addresses are given on the fly, but there is a rule that every memory cell must have a unique virtual address, because all references to that cell are stored as the virtual address the application sees.		A virtual address, on the other hand, is what the CPU and hence the application program sees. They might be the same but as a general rule they are different. Virtual addresses are given on the fly but there is a rule that every memory cell must have a unique virtual address because all references to that cell are stored as the virtual address the application sees.

	[[File:fig1-ext-mem.jpg\|frame\|center\|Physical Memory vs Virtual Address Space]]		[[File:fig1-ext-mem.jpg\|frame\|center\|Physical Memory vs Virtual Address Space]]

	Modern systems like the X1000 or upcoming models can take more than 4 gigabytes of memory, but so far, the extra memory will never be used. Even in a 4 gigabyte system, there is memory that will never be touched because there is just no free address~~; and~~ ~~unfortunately~~, every byte needs to have its own virtual address, and no two bytes can have the same.		Modern systems like the AmigaOne X1000 and upcoming models can take more than 4 gigabytes of memory, but so far, the extra memory will never be used. Even in a 4 gigabyte system, there is memory that will never be touched because there is just no free address. Unfortunately, every byte needs to have its own virtual address and no two bytes can have the same.

			This is where extended memory objects enter the picture.
	Unless…

	= Extended Memory Object =		= Extended Memory Object =

	Extended memory objects (ExtMem) are a means to access memory beyond the 2 gigabyte barrier by applications that are written to make use of them. In a nutshell, an extended memory object is a chunk of physical memory that exists in a ~~“nirvana”~~ state somewhere in the memory of the computer without a virtual address of its own. The memory cannot be accessed by anyone or anything in this state. In order to access it, an application must map part of the object into its own virtual address space. This mapping does make a part of the memory represented by the ExtMem object accessible in a memory window in the application’s own address space.		Extended memory objects (ExtMem) are a means to access memory beyond the 2 gigabyte barrier by applications that are written to make use of them. In a nutshell, an extended memory object is a chunk of physical memory that exists in a ''nirvana'' state somewhere in the memory of the computer without a virtual address of its own. The memory cannot be accessed by anyone or anything in this state. In order to access it, an application must map part of the object into its own virtual address space. This mapping does make a part of the memory represented by the ExtMem object accessible in a memory window in the application’s own address space.

	There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time, and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each ~~mappings~~ opens up a view into a part of the memory object, and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.		There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each mapping opens up a view into a part of the memory object and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.

	[[File:fig2-ext-mem.jpg\|frame\|center\|ExtMem Object Mapping]]		[[File:fig2-ext-mem.jpg\|frame\|center\|ExtMem Object Mapping]]
Line 27:		Line 27:
	A mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.		A mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.

	There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset, just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.		There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.

	As an example, consider the following situation. We want to access byte 3000 of the ExtMem object. We created a mapping that has length 4000 and starts at offset 2000. The resulting address for our byte would be the base address of the mapping plus 1000, since the offset of the beginning is already at 2000.		As an example, consider the following situation. We want to access byte 3000 of the ExtMem object. We created a mapping that has length 4000 and starts at offset 2000. The resulting address for our byte would be the base address of the mapping plus 1000, since the offset of the beginning is already at 2000.
Line 33:		Line 33:
	= Downsides of the ExtMem system =		= Downsides of the ExtMem system =

	If you think now that this all sounds suspiciously like bank switching, then you are ~~right~~. The method has been used way back in the Home computer age, and even earlier. The Sinclair ZX Spectrum 128K was equipped with twice as much memory as the Z80 CPU could address; the upper 16k of the machine could be swapped between different chunks of the rest of the memory. Similarly, the Commodore 64 used bank switching to address a larger memory than its 6502 CPU could handle. It was the only possibility at the time to add more memory.		If you think now that this all sounds suspiciously like bank switching then you are correct. The method has been used way back in the Home computer age and even earlier. The Sinclair ZX Spectrum 128K was equipped with twice as much memory as the Z80 CPU could address; the upper 16k of the machine could be swapped between different chunks of the rest of the memory. Similarly, the Commodore 64 used bank switching to address a larger memory than its 6502 CPU could handle. It was the only possibility at the time to add more memory.

	This method we employ now is basically the same (with a bit more added comfort).		This method we employ now is basically the same but with a bit more added comfort.

	Obviously, the method is a compromise. A ~~“real”~~ 64 bit system would be better, and much more transparent to use. However, as I ~~already outlined in the beginning~~, there is a lot of work involved to make AmigaOS 64 bit compatible~~, and~~ ~~with~~ the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.		Obviously, the method is a compromise. A ''real'' 64 bit system would be better and much more transparent to use. However, as stated earlier, there is a lot of work involved to make AmigaOS 64 bit compatible. With the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.

	= Who can benefit from ExtMem objects? =		= Who can benefit from ExtMem objects? =

	~~Well, every~~ application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline, or display a single frame that the user is working with.		Every application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline or display a single frame that the user is working with.

	Another example is RAM disk. Plans are currently underway to update the RAM disk to make use of the ExtMem object interface, allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated ~~“on~~-~~demand”~~ instead of ahead of time, ~~this~~ will make RAM disk have an even lower footprint, on top of making it possible to store larger amounts of data than ever before ~~in it~~.		Another example is RAM disk. Plans are currently underway to update the RAM disk to make use of the ExtMem object interface, allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated ''on-demand'' instead of ahead of time. This will make RAM disk have an even lower footprint on top of making it possible to store larger amounts of data than ever before.

	It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.		It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.