Copyright (c) Hyperion Entertainment and contributors.

Difference between revisions of "AmiWest 2013 Lesson 4"

From AmigaOS Documentation Wiki
Jump to navigation Jump to search
 
(16 intermediate revisions by 2 users not shown)
Line 19: Line 19:
   
 
First, we will look at creating a small client to retrieve some simple
 
First, we will look at creating a small client to retrieve some simple
information from an Internet website. Similar applications could be made
+
information from an Internet website. Such applications could be made
 
to retrieve basic information like stock quotes, email contents,
 
to retrieve basic information like stock quotes, email contents,
server conditions or other basic interactions. Aminet has a number
+
server conditions or other basic interactions.
examples of such clients written in ARexx that use the TCP: device
 
that can be examined.
 
   
 
The Roadshow shortcut we will use is its built-in TCP: handler. This is
 
The Roadshow shortcut we will use is its built-in TCP: handler. This is
mechanism that creates a virtual filesystem device ("TCP:") whenever
+
mechanism that creates a virtual file system device ("TCP:") whenever
Roadhsow gets online. To interact with a remote server, one just opens a
+
Roadhsow gets online. To interact with a remote server, one3 opens a
 
file with the name being the URL of the server and port to be accessed.
 
file with the name being the URL of the server and port to be accessed.
   
 
===Connect to the Server===
 
===Connect to the Server===
   
  +
When Roadshow gets online, it creates a virtual TCP: device. To open
To begin with, one opens a connection to the internet server as if one
 
  +
a connection, one opens a virtual file on the TCP: device to the URL and
were opening a file. As described above, when Roadshow gets online, it
 
  +
port in question using this format:
creates a virtual TCP: device. To open a connection, one opens a file
 
to the URL and port in question using this format:
 
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 59: Line 56:
 
that server and any further interaction with the "fileh" file handle
 
that server and any further interaction with the "fileh" file handle
 
communicates directly with that port on that server. In this case,
 
communicates directly with that port on that server. In this case,
we would have connected to a websserver.
+
we would have connected to a web server of the "Weather Underground"
  +
website.
   
 
===Talk to the Server===
 
===Talk to the Server===
Line 76: Line 74:
 
goals.
 
goals.
   
In the case of "HTTP" (or a "Hypertext Transfer Protocol") connection to
+
In the case of an "HTTP" (or a "Hypertext Transfer Protocol") connection to
 
a web server, we are usually expected to submit a "GET" request. Here are
 
a web server, we are usually expected to submit a "GET" request. Here are
 
a couple more complete references on the HTTP protocol:
 
a couple more complete references on the HTTP protocol:
Line 84: Line 82:
 
http://tools.ietf.org/html/rfc2616
 
http://tools.ietf.org/html/rfc2616
   
To keep things simple, lets just look at submitting a "GET" request, such
+
To keep things simple, lets mimic submitting a "GET" request (as if from IBrowse).
as a web browser, like IBrowse, would do. A simple HTTP version 1.1 GET
+
A simple HTTP version 1.1 GET request would include the following lines:
request would include the following lines:
 
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 100: Line 97:
 
</syntaxhighlight>
 
</syntaxhighlight>
   
The <path> would be the rest of the URL (if any) on that system and
+
The '''<path>''' would be the rest of the URL (if any) on that system and
<domain> is the name of the server.
+
'''<domain>''' is the name of the server.
   
 
Using the "WeatherUnderground" website URL from above and their path name for
 
Using the "WeatherUnderground" website URL from above and their path name for
Line 112: Line 109:
 
</syntaxhighlight>
 
</syntaxhighlight>
   
The GET request syntax can be transcribed into C code which builds the entire
+
Such a GET request syntax can be transcribed into C code which builds the entire
request in a string and then sends it to the server with a simple "fprintf" to our
+
request in a string and then sends it to the server with a simple "fprintf" to our open
open file handle, as follows:
+
TCP: file handle, as follows:
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 137: Line 134:
   
 
As you can see, the variables "path" and "domain" are used to fill in some
 
As you can see, the variables "path" and "domain" are used to fill in some
blanks in the request. The blank line at the end completes the GET request,
+
blanks in the request. The empty line at the end (with just a "\r\n") completes the
after which the server will reply using the same file handle.
+
GET request, after which the server will reply to your program.
   
 
===Listen to the Server===
 
===Listen to the Server===
Line 144: Line 141:
 
Once the GET request has been sent to the web server, the server will start
 
Once the GET request has been sent to the web server, the server will start
 
sending back either the requested webpage (or other content) or an error page.
 
sending back either the requested webpage (or other content) or an error page.
To see what sort of content is returned by the server, you can combine the
+
To see what sort of content is returned by the server in this example, you can
domain and path values above and enter them in your web browser:
+
combine the domain and path values above and enter them in your web
  +
browser's URL string gadget, like this:
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 151: Line 149:
 
</syntaxhighlight>
 
</syntaxhighlight>
   
Once the page is loaded, viewing the page source (f.e., IBrowse menu item
+
Once the page is loaded in your browser, viewing the page source (f.e., IBrowse
"Page/Display Source...") will show you the same text that your program
+
menu item "Page/Display Source...") will show you the same HTML text that your
will receive after sending the get request.
+
program will receive after sending the GET request.
   
As such, your program needs to start reading from the same file handle
+
As such, your program needs to start reading from the file handle
 
the request was sent with. This loop will print out the first hundred
 
the request was sent with. This loop will print out the first hundred
 
lines returned:
 
lines returned:
Line 173: Line 171:
 
Within this loop your application can parse the received lines for whatever
 
Within this loop your application can parse the received lines for whatever
 
data your application is trying to obtain. In the case of our example program below,
 
data your application is trying to obtain. In the case of our example program below,
the program searches for a string precedes the weather information we want.
+
the program searches for a string precedes the weather information we want to find.
   
 
After the returned content has been read, output or otherwise processed,
 
After the returned content has been read, output or otherwise processed,
Line 185: Line 183:
 
===EXAMPLE ONE: IPClient.c===
 
===EXAMPLE ONE: IPClient.c===
   
All of these elements are combined in the following example program
+
All of the above elements are combined in the following example program
"IPClient.c" that asks you for a IATA airport code (f.e., IAD = Washington
+
"IPClient.c" that asks you for an IATA airport code (f.e., IAD = Washington
 
Dulles, FRA=Frankfurt, SYD=Sydney), then parses & prints out the weather
 
Dulles, FRA=Frankfurt, SYD=Sydney), then parses & prints out the weather
conditions from the returned page.
+
conditions from the returned WeatherUnderground.com web page.
   
 
Please Note: this example program worked with the wunderground.com
 
Please Note: this example program worked with the wunderground.com
 
website in 2013-2014, as the served webpages change over time, it is
 
website in 2013-2014, as the served webpages change over time, it is
 
likely the parsing in this example is likely to fail and the program
 
likely the parsing in this example is likely to fail and the program
no longer return useful information.
+
no longer return useful information. Should this happen, you get to use what
  +
you learned above to upgrade or replace this example.
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 330: Line 329:
 
of connecting to a internet server and obtaining data are relatively
 
of connecting to a internet server and obtaining data are relatively
 
trivial. The real challenge for the developer lies in understanding
 
trivial. The real challenge for the developer lies in understanding
the protocol used and in processing the data sent and received.
+
the protocol used, in formatting the request and processing
  +
received data.
   
 
Public web servers make for fairly rich and easy targets to interact
 
Public web servers make for fairly rich and easy targets to interact
Line 340: Line 340:
 
Naturally, the more complicated the interactions, like logging into
 
Naturally, the more complicated the interactions, like logging into
 
a website, would require much more knowledge of the protocols used
 
a website, would require much more knowledge of the protocols used
more complicated code that is beyond the scope of this tutorial.
+
and more complicated code that is beyond the scope of this tutorial.
   
 
But there are a number of basic topics you should consider when
 
But there are a number of basic topics you should consider when
Line 347: Line 347:
 
====Server Suitability:====
 
====Server Suitability:====
   
Unless you are connecting to your own server or it invites such
+
Unless you are connecting to your own server or one which invites such
connections (some publish API's for you to make such connections),
+
connections (some even publish API's and example code for you to use),
 
your application's "visit" may not be welcome or even legal.
 
your application's "visit" may not be welcome or even legal.
   
Line 358: Line 358:
 
====Server Changes:====
 
====Server Changes:====
   
As with any programming that interacts with external inputs, your
+
As with any programming that interacts with external elements, your
code should provide for the possibility the connection may not
+
code should provide for the possibility things may not
succeed or that the returned results may not be what was expected.
+
succeed or the returned results may not be what was expected.
   
 
As servers and websites change over time, the processing and
 
As servers and websites change over time, the processing and
 
parsing of your program may often have to change accordingly.
 
parsing of your program may often have to change accordingly.
 
To maintain your program and deal with such changes, you may want
 
To maintain your program and deal with such changes, you may want
to consider how to make changing those things easy (by the user?).
+
to consider how to make modifying those things easy as possible (even
  +
by the user?).
The parsing strings could be kept in program tooltypes or config
 
  +
text file, you could provide a GUI for managing the parsing, employ
 
  +
For example, the parsing strings could be kept in the program's tooltype
an external, editable ARexx macro.
 
  +
or a config text file, you could provide a GUI for managing the strings, or
  +
employ an external, editable ARexx macro for processing.
   
 
====Protocol Details & Vagaries:====
 
====Protocol Details & Vagaries:====
Line 388: Line 390:
 
interactions with all sorts internet services, such as:
 
interactions with all sorts internet services, such as:
   
· Check stock quotes from a financial site.
+
* Check stock quotes from a financial site.
· Get package tracking information from a shipping site.
+
* Get package tracking information from a shipping site.
· Check for new emails on a POP email server.
+
* Check for new emails on a POP email server.
· Convert currencies with a financial or travel web site.
+
* Convert currencies with a financial or travel web site.
· Check for files on an FTP server.
+
* Check for files on an FTP server.
· Send messages with a SMTP email server.
+
* Send messages with a SMTP email server.
· Look up words on a dictionary website.
+
* Look up words on a dictionary website.
   
 
You could also write your own server for another Amiga and interact
 
You could also write your own server for another Amiga and interact
with your own client. A remote controlled media player?
+
with your own client. A remote controlled media player? The AmiNet
  +
also has many examples of simple clients (with C source code or in
 
  +
readable ARexx) that can be studied for means and methods.
 
   
 
==An Internet Server==
 
==An Internet Server==
Line 443: Line 445:
 
line with two or fewer characters (essentially an empty line with
 
line with two or fewer characters (essentially an empty line with
 
what should be a CR-LF). There is also a counter that kicks us out
 
what should be a CR-LF). There is also a counter that kicks us out
if the loop if we read 100 lines - that's a sign something else is
+
of the loop if we read 100 lines - that's a sign something else is
wrong, we should never see that with the HTTP protocol.
+
wrong, we should never see so long a request with the HTTP protocol.
   
 
With the HTTP protocol, the first line is where we would see what is
 
With the HTTP protocol, the first line is where we would see what is
being requested. Were our server to use other protocols, we may have to
+
being requested. Were our server to use another protocol, the processing of
add our server's "intelligence" elsewhere.
+
the received data would likely happe3n elsewhere.
   
We also do all our diagnostics output using the "DebugPrintf". Which
+
As you see, we do all our diagnostics output using the "DebugPrintf". Which
 
means we either need to run Sashimi or watch output on the serial port
 
means we either need to run Sashimi or watch output on the serial port
(with your second Amiga, naturally). In the case of this example, we
+
(with your second Amiga, naturally). With the above code, the diagnostics
will see the entire incoming request printed out on the serial port.
+
output will show the entire incoming request on the serial port. Of course,
Of course, there's a reason why we have to use the serial for our debug output...
+
there's a reason why we have to use "DebugPrintf" for diagnostics output...
   
 
===Respond to the Visitor===
 
===Respond to the Visitor===
Line 467: Line 469:
 
our web visitor.
 
our web visitor.
   
In this example, we create a basic loop that feeds a predefined series of lines
+
In this example, we create a basic loop that feeds a predefined series of strings (in the '''lines''' array)
of a very simple webpage to Roadshow and our web visitor:
+
of a very simple webpage through Roadshow, to our web visitor:
   
 
<syntaxhighlight>
 
<syntaxhighlight>
Line 497: Line 499:
 
stream and quit our program.
 
stream and quit our program.
   
  +
While our two samples of code simply receive and respond to an incoming web request,
The request was received, our little webpage sent, we quit and Roadshow takes
 
  +
one can imagine where this code can be expanded provide useful service and more
care of all the housekeeping.
 
  +
complex interactions.
   
 
===EXAMPLE TWO: IPserver.c===
 
===EXAMPLE TWO: IPserver.c===
Line 626: Line 629:
 
serving your Amiga's internet visitors.
 
serving your Amiga's internet visitors.
   
But how do we test this all out? First we have to let Roadshow know that
+
But how do we test this server out? First we have to let Roadshow know that
 
we've created this server and where visitors will find it.
 
we've created this server and where visitors will find it.
''Italic text''
 
   
 
===Roadshow Configuration===
 
===Roadshow Configuration===
   
 
There are two areas where Roadshow needs to be told about your server
 
There are two areas where Roadshow needs to be told about your server
and the service it is going to provide. Both of these can be configured
+
and the service it is going to provide, as described below. Both of these can
using Internet Prefs. You can also make this configuration by editing
+
be configured using Internet Prefs. You can also make this configuration by
the "servers" and "services" files in the "DEVS:internet/" directory.
+
manually editing the "servers" and "services" files in the "DEVS:internet/" directory.
 
Configuration of these files is described in the file:
 
Configuration of these files is described in the file:
   
 
<syntaxhighlight>
 
<syntaxhighlight>
SYS:Documentation?Roadshow/README
+
SYS:Documentation/Roadshow/README
 
</syntaxhighlight>
 
</syntaxhighlight>
   
Line 668: Line 670:
 
"Save" in Internet Prefs, this service will be recognized by Roadshow.
 
"Save" in Internet Prefs, this service will be recognized by Roadshow.
   
The same configuration could be added to Roadshow by adding this line
+
The same configuration could manually be added to Roadshow by adding this line
 
(in port number location) into the file "DEVS:Internet/Services":
 
(in port number location) into the file "DEVS:Internet/Services":
   
Line 706: Line 708:
 
Prefs will adjust Roadshow.
 
Prefs will adjust Roadshow.
   
We can also make this adjustment to Roadshow's server list by adding the
+
We can otherwise make this adjustment to Roadshow's server list by manually adding the
 
following line to the "DEVS:Internet/servers" file:
 
following line to the "DEVS:Internet/servers" file:
   
Line 736: Line 738:
 
</syntaxhighlight>
 
</syntaxhighlight>
   
In any case, the browser will connect to your Amiga at the port number
+
In any case, the browser will connect to your Amiga (at the port number
following the colon (":") and Roadshow will call up your server program
+
following the colon - ":") and Roadshow will call up your server program
 
and let it talk to your browser and send it the preset webpage.
 
and let it talk to your browser and send it the preset webpage.
   
===Where to go? More Webpages?===
+
===Where to go? Webpages? More?===
   
 
Clearly the above example does little more than say "Hello World" to a
 
Clearly the above example does little more than say "Hello World" to a
Line 756: Line 758:
 
</syntaxhighlight>
 
</syntaxhighlight>
   
In that case, the browser is requesting your server send an icon
+
In that case, the browser is requesting your server send an image file
 
that it can display on the browser tab. Just like that, anything in the URL
 
that it can display on the browser tab. Just like that, anything in the URL
 
after the machine's network name (or IP address) and the port number
 
after the machine's network name (or IP address) and the port number
Line 763: Line 765:
   
 
This just scratches the surface of requests your server may receive and
 
This just scratches the surface of requests your server may receive and
how it can respond to web visitors. To dig deeper you can search the
+
how it can process them and respond to web visitors. To dig deeper you can
many sources online or in your local bookstore on HTTP communications and
+
search the many sources online or in your local bookstore on HTTP communications
the formatting of HTML webpages. Of course, you should make sure your
+
and the formatting of HTML webpages. Of course, you should make sure your
server continues outputting the request contents so you know what your server
+
server can optionally send diagnostics output so you know what requests are
is getting and should respond to.
+
being received and how they are responded to.
   
 
===Just Webpages?===
 
===Just Webpages?===
   
There's also ne reason why your server needs to be limited to dealing
+
There's also no reason why your server needs to be limited to dealing
 
with browsers and sending webpages. The internet is full of other
 
with browsers and sending webpages. The internet is full of other
protocols and services that your could write a server for. Or you could
+
protocols and services that you could write a server for. Or you could
create your own online client and server pair of applications that just
+
create your own online client and server pair of applications that
 
talked with each other.
 
talked with each other.
   
Line 784: Line 786:
   
 
Obviously, opening your Amiga with an internet server represents more of a
 
Obviously, opening your Amiga with an internet server represents more of a
risk than if you had done nothing. So it is criticially important one
+
risk than if you had done nothing. So it is critically important one
 
considers security when creating a server. While it's beyond the scope
 
considers security when creating a server. While it's beyond the scope
 
of this article to inclusively discuss all the issues involved, here are some
 
of this article to inclusively discuss all the issues involved, here are some
Line 791: Line 793:
 
Don't blindly serve sensitive information to visitors. How can you be
 
Don't blindly serve sensitive information to visitors. How can you be
 
sure who's at the other end of the connection and who might be watching
 
sure who's at the other end of the connection and who might be watching
along the way? Even if randomized pathnames and request are used by
+
along the way? Even if randomized path names and requests are used by
 
your server, there's no guarantee the connection is 100% secure.
 
your server, there's no guarantee the connection is 100% secure.
   
Line 804: Line 806:
   
 
One has to keep in mind the "SuperServer" method where Roadshow runs
 
One has to keep in mind the "SuperServer" method where Roadshow runs
your server also means your server will be executed with each and every
+
your server also means your server program will be executed with each and every
 
request Roadshow receives for that port. In each case, your server
 
request Roadshow receives for that port. In each case, your server
 
will do its duty and then be expected to quit. That should be fine for
 
will do its duty and then be expected to quit. That should be fine for
Line 814: Line 816:
 
slow reaction time for each incoming request to an unacceptable level.
 
slow reaction time for each incoming request to an unacceptable level.
   
To avoid time consuming chores, one might considering using interprocess
+
To avoid time consuming chores, one might considering using inter-process
communciations (such as ARexx messages) to interact with a separate
+
communications (such as ARexx messages) to interact with a separate
 
master program that handles all the housekeeping and remains continuously running
 
master program that handles all the housekeeping and remains continuously running
while each of internet server instance handles an incoming request, gets the information it
+
while each instance of the internet server handles each incoming request, quickly
needs from the master program and quits.
+
relays the necessary information from the master program and quits.
   
 
Naturally, at some point usage requirements may require one bypasses the
 
Naturally, at some point usage requirements may require one bypasses the
 
Roadshow SuperServer method of running servers and goes to using
 
Roadshow SuperServer method of running servers and goes to using
 
longhand socket programming that is better suited to heavy duty use
 
longhand socket programming that is better suited to heavy duty use
- like running a serious public server. That's beyond the scope of this article.
+
- like running a serious public server. That's beyond the scope of this tutorial.
   
 
Furthermore, there are whole areas of study and technology involved with the
 
Furthermore, there are whole areas of study and technology involved with the
Line 834: Line 836:
 
a wider group of users (even those on lesser platforms!), one needs
 
a wider group of users (even those on lesser platforms!), one needs
 
to pay much closer attention to the protocols involved. With each new client
 
to pay much closer attention to the protocols involved. With each new client
and platform encountered, the more important implrementation details of a
+
and platform encountered, the more important implementation details of a
 
protocol are likely to be.
 
protocol are likely to be.
   
As mentioned in elsewhere in this text, there RFC's that describe the
+
As mentioned in elsewhere in this text, there are RFC's that describe the
 
formal protocols and there are likely countless webpages that address
 
formal protocols and there are likely countless webpages that address
idiosynacracies of implementing the protocols.
+
idiosyncrasies of implementing the protocols.
   
 
Beyond all those, a developer will need to do serious testing with as
 
Beyond all those, a developer will need to do serious testing with as
Line 866: Line 868:
 
Since all these programs are dependent on communications over many links,
 
Since all these programs are dependent on communications over many links,
 
these programs need to have a robustness for failed links. Transfers can
 
these programs need to have a robustness for failed links. Transfers can
be interupted mid-stream. Received data can be incomplete or corrupted.
+
be interrupted mid-stream. Received data can be incomplete or corrupted.
 
These applications need to have the error trapping to deal with such real
 
These applications need to have the error trapping to deal with such real
possibilities.
+
possibilities - that isn't reflected in the simple examples above.
   
 
====Statelessness====
 
====Statelessness====

Latest revision as of 19:52, 2 October 2015

Simple IP Clients & Servers

Simple network access is easy!

Developing applications that use TCP/IP "sockets" for network communications have a reputation of being a challenging task. But AmigaOS and its Roadshow TCP/IP stack provides a couple of built-in shorthand mechanisms for easily creating simple network clients and servers: the TCP: handler and Roadshow's Superserver.

Using these Roadshow shortcuts, creating simple network client and server applications for single user or light-duty uses is no more complicated than reading or writing a file. The real challenge is understanding the protocol to be used (f.e. HTTP communications with web browsers) and being careful not to create a security risk on your or another user's Amiga.


An Internet Client

First, we will look at creating a small client to retrieve some simple information from an Internet website. Such applications could be made to retrieve basic information like stock quotes, email contents, server conditions or other basic interactions.

The Roadshow shortcut we will use is its built-in TCP: handler. This is mechanism that creates a virtual file system device ("TCP:") whenever Roadhsow gets online. To interact with a remote server, one3 opens a file with the name being the URL of the server and port to be accessed.

Connect to the Server

When Roadshow gets online, it creates a virtual TCP: device. To open a connection, one opens a virtual file on the TCP: device to the URL and port in question using this format:

TCP:<server URL>/<port number>

For example, to connect to the website "www.wunderground.com" one would use:

	fileh = fopen("TCP:www.wunderground.com/80","r+");

The file name starts with TCP:, then the website URL and finally the port number - here "80" is site's HTTP port (used by almost all websites).

Typically, the port number reflects the type of port or protocol that is to be used. Such protocols could be FTP (file transfer), POP/SMTP (email) or many others. Port 80 is usually used for HTTP communications with websites.

If the file open request is successful, your Amiga is then connected to that server and any further interaction with the "fileh" file handle communicates directly with that port on that server. In this case, we would have connected to a web server of the "Weather Underground" website.

Talk to the Server

Once one has connected to a server, knowledge of the communications protocol for that type of connection is required.

For each of those, a server expects a certain format of interaction to proceed and respond. Fortunately, common protocols are publicly documented.

http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

For each protocol there is usually an "RFC" document that describes how one uses the protocol, interacts with the port and accomplishes that protocol's goals.

In the case of an "HTTP" (or a "Hypertext Transfer Protocol") connection to a web server, we are usually expected to submit a "GET" request. Here are a couple more complete references on the HTTP protocol:

http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

http://tools.ietf.org/html/rfc2616

To keep things simple, lets mimic submitting a "GET" request (as if from IBrowse). A simple HTTP version 1.1 GET request would include the following lines:

	GET /<path> HTTP/1.1
	host: <domain>
	user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build)
	Pragma: no-cache
	Accept-Language: en, *
	Accept: text/html;level=3
	Accept: text/html;version=3.0
	Accept: */*
	<blank line>

The <path> would be the rest of the URL (if any) on that system and <domain> is the name of the server.

Using the "WeatherUnderground" website URL from above and their path name for searching for airport conditions, we can find the current conditions at Washington's Dulles International Airport with these domain & path values:

   URL = www.wunderground.com
   PATH = cgi-bin/findweather/getForecast?query=IAD

Such a GET request syntax can be transcribed into C code which builds the entire request in a string and then sends it to the server with a simple "fprintf" to our open TCP: file handle, as follows:

	// build web req text
	strcpy(line,"GET /");
	strcat(line,path);
	strcat(line," HTTP/1.1\r\n");
	strcat(line,"host: ");
	strcat(line,domain);
	strcat(line,"\r\n");
	strcat(line,"user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build)\r\n");
	strcat(line,"Pragma: no-cache\r\n");
	strcat(line,"Accept-Language: en, *\r\n");
	strcat(line,"Accept: text/html;level=3\r\n");
	strcat(line,"Accept: text/html;version=3.0\r\n");
	strcat(line,"Accept: */*\r\n");
	strcat(line,"\r\n");
 
	// send web req line to server
	fprintf(fileh,"%s",line);

As you can see, the variables "path" and "domain" are used to fill in some blanks in the request. The empty line at the end (with just a "\r\n") completes the GET request, after which the server will reply to your program.

Listen to the Server

Once the GET request has been sent to the web server, the server will start sending back either the requested webpage (or other content) or an error page. To see what sort of content is returned by the server in this example, you can combine the domain and path values above and enter them in your web browser's URL string gadget, like this:

   www.wunderground.com/cgi-bin/findweather/getForecast?query=IAD

Once the page is loaded in your browser, viewing the page source (f.e., IBrowse menu item "Page/Display Source...") will show you the same HTML text that your program will receive after sending the GET request.

As such, your program needs to start reading from the file handle the request was sent with. This loop will print out the first hundred lines returned:

	// read in the response
	count = 0;
	while( (fgets(inStr,MAX_STR,fileh) != NULL) && (count<100) )
	{
		++ count;
		printf("line %ld = %s\n",count,inStr);
 
		// process lines received here !
	}

Within this loop your application can parse the received lines for whatever data your application is trying to obtain. In the case of our example program below, the program searches for a string precedes the weather information we want to find.

After the returned content has been read, output or otherwise processed, simply closing the file handle will close the connection to the server:

	fclose(fileh);


EXAMPLE ONE: IPClient.c

All of the above elements are combined in the following example program "IPClient.c" that asks you for an IATA airport code (f.e., IAD = Washington Dulles, FRA=Frankfurt, SYD=Sydney), then parses & prints out the weather conditions from the returned WeatherUnderground.com web page.

Please Note: this example program worked with the wunderground.com website in 2013-2014, as the served webpages change over time, it is likely the parsing in this example is likely to fail and the program no longer return useful information. Should this happen, you get to use what you learned above to upgrade or replace this example.

/*
************************************************************
**
** Created by: CodeBench 0.41 (12.10.2013)
** Project: IPclient
** Date: 12-10-2013 18:51:10
**
************************************************************
*/
 
#include <stdio.h>
#include <string.h>
 
#include <proto/exec.h>
#include <proto/dos.h>
 
/*****************************************************************************
 *		GLOBAL VARIABLES
 *****************************************************************************/
 
STATIC CONST_STRPTR version USED = "$VER: IPclient v.02 (12.10.2013)";
 
char URL[200] = "www.wunderground.com/cgi-bin/findweather/getForecast?query=";
char domain[200];
char path[200];
char fname[256] = "";
#define MAX_STR 2056
char line[MAX_STR] = "";
char inStr[MAX_STR];
uint16 uLen;
uint16 dLen;
uint16 count = 0;
char *found;
FILE *fileh;
 
/*****************************************************************************
 *		PROGRAM START
 *****************************************************************************/
 
int main(int argc,char **argv)
{
	// get URL from user
	printf("IPclient example\n");
	printf("======================\n");
 
	printf("   enter airport code >");
	fgets(line,sizeof(line),stdin);
	line[strlen(line)-1] = '\0';		// strip \n off end of string
	printf("======================\n");
 
	// append airport code on URLprefix
	strcat(URL,line);
 
	// parse domain from path
	uLen = strlen(URL) - 2;
	strcpy(domain,strtok(URL,"/\n"));
	dLen = strlen(domain);
	if (uLen>dLen)
		strcpy(path,strtok(NULL,"\n"));
	else
		printf("   No chars remain for path\n");
	printf("   URL domain >%s<\n",domain);
	printf("   URL path >%s<\n",path);
 
	// build URL filename from domain name
	strcpy(fname,"TCP:");
	strcat(fname,domain);
	strcat(fname,"/80");
 
	// open file access to URL
	printf("      Opening URL >%s<\n",fname);
	fileh = fopen(fname,"r+");
	if (fileh == NULL)
	{
		printf("Couldn't open connection domain server via \"%s\"\n",fname);
		return RETURN_ERROR;
	};
	printf("      TCP: file opened\n");
 
	// build web req text
	strcpy(line,"GET /");
	strcat(line,path);
	strcat(line," HTTP/1.1\r\n");
	strcat(line,"host: ");
	strcat(line,domain);
	strcat(line,"\r\n");
	strcat(line,"user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build)\r\n");
	strcat(line,"Pragma: no-cache\r\n");
	strcat(line,"Accept-Language: en, *\r\n");
	strcat(line,"Accept: text/html;level=3\r\n");
	strcat(line,"Accept: text/html;version=3.0\r\n");
	strcat(line,"Accept: */*\r\n");
	strcat(line,"\r\n");
 
	// send web req line to server
	fprintf(fileh,"%s",line);
	strcpy(line,"");
	printf("      Web req sent\n");
	printf("======================\n");
 
	// read in the response
	while( (fgets(inStr,MAX_STR,fileh) != NULL) && (count<100) )
	{
		++ count;
		//printf("line %d = %s\n",count,inStr);
 
		// look for name of airport
		found = strstr(inStr,"og:title");
		if (found != NULL)
		{
			found = strtok(inStr,"=");
			found = strtok(NULL,"=");
			printf("WeatherUnderground.com reports:\n");
			printf("   Airport   = %s\n",strtok(NULL,"|")+1);
			printf("   Temp      =%s F\n",strtok(NULL,"&"));
			found = strtok(NULL,"|");
			printf("   Condition =%s\n",strtok(NULL,"\""));			
		}
	}
	printf("======================\n");
 
	// close file
	fclose(fileh);
 
	return RETURN_OK;
}


Designing Your Network Client

As you see, the above code and the example program, the mechanics of connecting to a internet server and obtaining data are relatively trivial. The real challenge for the developer lies in understanding the protocol used, in formatting the request and processing received data.

Public web servers make for fairly rich and easy targets to interact with. You can explore simple interaction with any web browser and mimic that interaction with your code. In many cases, you can control the feedback just by crafting the web request as we did above and then parse the results.

Naturally, the more complicated the interactions, like logging into a website, would require much more knowledge of the protocols used and more complicated code that is beyond the scope of this tutorial.

But there are a number of basic topics you should consider when designing a web client!

Server Suitability:

Unless you are connecting to your own server or one which invites such connections (some even publish API's and example code for you to use), your application's "visit" may not be welcome or even legal.

At the very least, your program should "tread lightly" - do not abuse the server with unnecessary or intrusive requests. Furthermore, if your client uses the resources of someone else's server, you should credit the server within your application and documentation.

Server Changes:

As with any programming that interacts with external elements, your code should provide for the possibility things may not succeed or the returned results may not be what was expected.

As servers and websites change over time, the processing and parsing of your program may often have to change accordingly. To maintain your program and deal with such changes, you may want to consider how to make modifying those things easy as possible (even by the user?).

For example, the parsing strings could be kept in the program's tooltype or a config text file, you could provide a GUI for managing the strings, or employ an external, editable ARexx macro for processing.

Protocol Details & Vagaries:

Our example code presented a very simple interaction with a web server using the a widely supported version of the HTTP protocol. As one develops an application and uses a protocol, one should get familiar with the details of that protocol. Typically a "RFC" document (as linked above) will describe the details of how things are to work and how they might fail. Your application should be careful to comply with the details of the protocol ("Are we sending a CR-LF or LF-CR?!").

Ideas

The above example was just a quick, simple exercise in demonstrating the use of the AmigaOS Roadshow TCP: handler to access a web server. There are many ways applications could be created to have simple interactions with all sorts internet services, such as:

  • Check stock quotes from a financial site.
  • Get package tracking information from a shipping site.
  • Check for new emails on a POP email server.
  •  Convert currencies with a financial or travel web site.
  • Check for files on an FTP server.
  • Send messages with a SMTP email server.
  • Look up words on a dictionary website.

You could also write your own server for another Amiga and interact with your own client. A remote controlled media player? The AmiNet also has many examples of simple clients (with C source code or in readable ARexx) that can be studied for means and methods.

An Internet Server

Next we will look at creating a simple internet server program that relies on AmigaOS Roadshow's "SuperServer" to receive the incoming internet connection and share it with our application.

Simply speaking, once Roadshow is configured to recognize the incoming request and that your application is there to handle it; our program just has to deal with another case of simple file-like interaction.

For this example, we will create a simple server that accepts an HTTP protocol request (like from IBrowse) and then responds to it with a simple webpage. Of course, you could write a server to serve an interactive webpage, some machine information, media files, etc. Just be careful.

Receive the Request

When Roadshow receives an internet request on the designated port that corresponds to our server, it starts our server program and routes the network connection to our program using the "standard" input stream (otherwise known as "stdin").

Our application then reads the stream almost like reading from any other file or user input, like this:

	// read in the response
	while( (fgets(inStr,MAX_STR,(FILE *)stdin) != NULL) && (count<100) )
	{
		if (strlen(inStr)<3)
		{
			IExec->DebugPrintF("   received nearly empty line, ending read loop.\n",NULL,NULL);
			break;
		}
		++ count;
		IExec->DebugPrintF("   %s\n",inStr,NULL,NULL,NULL);
	}

As you can see, we loop reading incoming lines until we encounter a line with two or fewer characters (essentially an empty line with what should be a CR-LF). There is also a counter that kicks us out of the loop if we read 100 lines - that's a sign something else is wrong, we should never see so long a request with the HTTP protocol.

With the HTTP protocol, the first line is where we would see what is being requested. Were our server to use another protocol, the processing of the received data would likely happe3n elsewhere.

As you see, we do all our diagnostics output using the "DebugPrintf". Which means we either need to run Sashimi or watch output on the serial port (with your second Amiga, naturally). With the above code, the diagnostics output will show the entire incoming request on the serial port. Of course, there's a reason why we have to use "DebugPrintf" for diagnostics output...

Respond to the Visitor

Once we've read the (almost) empty line at the end of the incoming web request, we will use another "standard" stream (know as "stdout") to respond to our internet visitor.

Anything we output to "stdout" will be sent by Roadshow back to our internet visitor. This is the reason why we used the serial port for diagnostic print outs, since a simple "printf" would have gone to our web visitor.

In this example, we create a basic loop that feeds a predefined series of strings (in the lines array) of a very simple webpage through Roadshow, to our web visitor:

	//loop thru text writing to stdout
	do
	{
		if (strlen(lines[l]) > 0)
		{
			IExec->DebugPrintF("   printing line = %s\n",lines[l],NULL,NULL);			
			if (fprintf(stdout,"%s\n",lines[l]) < 0)
			{
				IExec->DebugPrintF("ERROR printing line\n",NULL,NULL);
				break;
			}
			fflush(stdout);
		}
		else
		{
			IExec->DebugPrintF("   empty line, loop end\n",NULL,NULL);
			break;
		}
		++l;
	} while (l < 25);		// emergency loop escape to stop endlessness

As you can see, our loop ends when we hit an empty line in our webpage definition or a maximum of 25 lines (a fail-safe). Then we "flush" the output stream and quit our program.

While our two samples of code simply receive and respond to an incoming web request, one can imagine where this code can be expanded provide useful service and more complex interactions.

EXAMPLE TWO: IPserver.c

We can see all these pieces and the definition of our simple webpage in the full program here:

/*
************************************************************
**
** Created by: CodeBench 0.41 (12.10.2013)
** Project: IPserver
** Date: 13-10-2013 23:41:17
**
************************************************************
*/
 
#include <string.h>
#include <stdio.h>
#include <proto/exec.h>
#include <proto/dos.h>
 
/*****************************************************************************
 *		GLOBAL VARIABLES
 *****************************************************************************/
 
STATIC CONST_STRPTR version USED = "$VER: IPserver 0.14 (13.10.2013)";
 
// define maximum web request lines & length
#define MAX_STR 2048
 
// ARS path & file name
uint16 count = 0;
char inStr[MAX_STR];
 
CONST_STRPTR lines[] =
{
	"HTTP/1.0\015",
	"Content-Type: text/html\015",
	"\015",
	"<HTML><HEAD><TITLE>IPserver</TITLE></HEAD><BODY BGCOLOR=\"cccccc\">\015",
	"<TABLE WIDTH=100%><TD ALIGN=\"center\" BGCOLOR=\"cccccc\">\015",
	"<BR><TABLE WIDTH=90%><TR><TD><B><H1>WELCOME</H1></B><BR></TD></TR></TABLE>\015",
	"<TR><TD BGCOLOR=\"6688ee\"><H3><P><P><FONT COLOR=\"white\"><B>AmigaOS IPserver example</B></FONT></TD></TR>\015",
	"<TR><TD ALIGN=\"center\" BGCOLOR=\"cccccc\">\015",
	"<BR><TABLE CELLSPACING=0 WIDTH=90%><TR BGCOLOR=\"ffffff\"><TD WIDTH=100><H4><B>And so it begins...</B></TD><TD> Your Amiga internet server! </TD></TR></TABLE>\015",
	"</TR></TD></TABLE></BODY></HTML>\015",
	""
};
uint16 l = 0;
 
/*****************************************************************************
 *		PROGRAM START
 *****************************************************************************/
 
// Starting program
int main(int argc,char **argv)
{
 
	IExec->DebugPrintF("======================================\n");
	IExec->DebugPrintF("IPserver starting\n");
	IExec->DebugPrintF("======================================\n");
 
	//		Was program started from shell or WB ?
	if (argc > 0)
	{
 
		// read in the response
		while( (fgets(inStr,MAX_STR,(FILE *)stdin) != NULL) && (count<100) )
		{
			//  ####   Test for CR-LR only line
			if (strlen(inStr)<3)
			{
				IExec->DebugPrintF("   received nearly empty line, ending read loop.\n",NULL,NULL);
				break;
			}
 
			++ count;
			IExec->DebugPrintF("   %s\n",inStr,NULL,NULL,NULL);
		}
 
		IExec->DebugPrintF("======================================\n");
		IExec->DebugPrintF("FINISHED READING WEB REQUEST\n");
		IExec->DebugPrintF("SERVING WEB PAGE...\n");
		IExec->DebugPrintF("======================================\n");
 
		//loop thru text writing to stdout
		do
		{
			if (strlen(lines[l]) > 0)
			{
				IExec->DebugPrintF("   printing line = %s\n",lines[l],NULL,NULL);
 
				if (fprintf(stdout,"%s\n",lines[l]) < 0)
				{
					IExec->DebugPrintF("ERROR printing line\n",NULL,NULL);
					break;
				}
 
				fflush(stdout);
			}
			else
			{
				IExec->DebugPrintF("   empty line, loop end\n",NULL,NULL);
				break;
			}
			++l;
		} while (l < 25);		// emergency loop escape to stop endlessness
 
		IExec->DebugPrintF("======================================\n");
		IExec->DebugPrintF("Web service finished... \n");
	}
	else
		printf("IPserver started from Workbench - Don't do it again, this is a Roadshow app!\n");
 
	IExec->DebugPrintF("IPserver Quitting!\n");
	IExec->DebugPrintF("======================================\n");
 
	return RETURN_OK;
}

As you can see there's also a bit of housekeeping code at the beginning to check if the program was run from the Workbench and to tell the user not to. As it stands, this is program is only meant to be used by Roadshow for serving your Amiga's internet visitors.

But how do we test this server out? First we have to let Roadshow know that we've created this server and where visitors will find it.

Roadshow Configuration

There are two areas where Roadshow needs to be told about your server and the service it is going to provide, as described below. Both of these can be configured using Internet Prefs. You can also make this configuration by manually editing the "servers" and "services" files in the "DEVS:internet/" directory. Configuration of these files is described in the file:

   SYS:Documentation/Roadshow/README

If you would like to spare your user from having to do either of those chores, your application can even modify those files. As soon as Roadshow sees those files have been modified, it will reconfigure itself accordingly.

Services

First, we need to tell Roadshow about the "service" we are providing. If one opens Internet Prefs and clicks on the "Services" page, one will see a list of standard internet services with their TCPIP port numbers, types and aliases.

Click the "New..." button to define a new Service. An "Add service" window will open where we can define how our server will be accessed. Critically, we need to pick a port number that is not already in use. Such settings could be:

   Name = IPserver
   Port = 7600
   Type = tcp

Then we click "Use" to accept our new values. As soon as one clicks "Save" in Internet Prefs, this service will be recognized by Roadshow.

The same configuration could manually be added to Roadshow by adding this line (in port number location) into the file "DEVS:Internet/Services":

   IPServer         7600/tcp

As soon as one saves the file, Roadshow will be notified and adjust itself for the service.

Servers

Before we can use the service above, we also need to tell Roadshow about our server app and where to find it. Again, this can be done while in Internet Prefs. Click on the "Servers" page and one will see any servers configured with the services they provide (as set above), their type, wait method and program path.

Click on the "New..." button to define a new Server. In the "Add server" window we can define what sort of service our server is for, the characteristics of how it will be called and where it is located on our Amiga system. Such settings could be:

   Service = IPserver
   Type = Stream
   Stack = 65536
   Program = data:Projects/C/OS4ex-IPserver/IPserver.debug
   Arguments = <empty>
   Active = <checked>
   Wait for completion = <not checked>
   Use socket I/O streams = <checked>

Naturally, you should enter the server program path & name for your own system. Again we click "Use" to accept our new Server entry and saving Internet Prefs will adjust Roadshow.

We can otherwise make this adjustment to Roadshow's server list by manually adding the following line to the "DEVS:Internet/servers" file:

   IPServer        stream     dos stack=65536 data:Projects/C/OS4ex-IPserver/IPserver.debug

Again, saving the file is automatically noticed by Roadhsow.

Once both the service and server are correctly configured, Roadshow is ready to use our new server.

Let's See Our Server!

To test our new server example, you can access it by connecting to your Amiga and the designated IP port number. On the same machine, you can enter this in your browser URL line:

   http://localhost:7600/

From another machine, you will need to have the IP address or defined host name for your Amiga running the test server program. The URL should look something like this:

   http://192.168.1.07:7600/

In any case, the browser will connect to your Amiga (at the port number following the colon - ":") and Roadshow will call up your server program and let it talk to your browser and send it the preset webpage.

Where to go? Webpages? More?

Clearly the above example does little more than say "Hello World" to a visiting web browser user. It offers no interactivity, performs no task and serves no media.

When this web server receives the incoming web request, the first line (starting with "GET") tells the server what the browser is looking for. For example, if you watch the serial output and connect with a newer browser than IBrowse, you can see most browsers actually make two connections to the server, with one of them saying:

   GET /favicon.ico HTTP/1.1

In that case, the browser is requesting your server send an image file that it can display on the browser tab. Just like that, anything in the URL after the machine's network name (or IP address) and the port number is taken as a requested path and your server will receive it after "GET" in the first line of the request.

This just scratches the surface of requests your server may receive and how it can process them and respond to web visitors. To dig deeper you can search the many sources online or in your local bookstore on HTTP communications and the formatting of HTML webpages. Of course, you should make sure your server can optionally send diagnostics output so you know what requests are being received and how they are responded to.

Just Webpages?

There's also no reason why your server needs to be limited to dealing with browsers and sending webpages. The internet is full of other protocols and services that you could write a server for. Or you could create your own online client and server pair of applications that talked with each other.

Just as we discussed with creating an Internet client, there are many topics you should consider when designing an internet server program. It's not just about the coding!

Security

Obviously, opening your Amiga with an internet server represents more of a risk than if you had done nothing. So it is critically important one considers security when creating a server. While it's beyond the scope of this article to inclusively discuss all the issues involved, here are some basics to consider.

Don't blindly serve sensitive information to visitors. How can you be sure who's at the other end of the connection and who might be watching along the way? Even if randomized path names and requests are used by your server, there's no guarantee the connection is 100% secure.

Limit commands and functions served to visitors. Obviously, creating a means of offering anything like command line access or unfettered file editing/deleting powers to a web visitor would represent dangerous hole in any system's security. But one also has to make sure no whatsoever holes or bugs exist in the server design that would let such access slip through.

Server Instances & Load

One has to keep in mind the "SuperServer" method where Roadshow runs your server also means your server program will be executed with each and every request Roadshow receives for that port. In each case, your server will do its duty and then be expected to quit. That should be fine for light duty, personal server tasks.

But one should focus server development on keeping things as "lightweight" and simple as possible. If the server needs to read data files or perform other complicated initialization, this may slow reaction time for each incoming request to an unacceptable level.

To avoid time consuming chores, one might considering using inter-process communications (such as ARexx messages) to interact with a separate master program that handles all the housekeeping and remains continuously running while each instance of the internet server handles each incoming request, quickly relays the necessary information from the master program and quits.

Naturally, at some point usage requirements may require one bypasses the Roadshow SuperServer method of running servers and goes to using longhand socket programming that is better suited to heavy duty use - like running a serious public server. That's beyond the scope of this tutorial.

Furthermore, there are whole areas of study and technology involved with the handling of network user load that should be explored in implementing a serious server system.

Protocols and Compatibility

As one develops a more a complex or interactive server meant to handle a wider group of users (even those on lesser platforms!), one needs to pay much closer attention to the protocols involved. With each new client and platform encountered, the more important implementation details of a protocol are likely to be.

As mentioned in elsewhere in this text, there are RFC's that describe the formal protocols and there are likely countless webpages that address idiosyncrasies of implementing the protocols.

Beyond all those, a developer will need to do serious testing with as wide a pool of possible clients to refine a server's operation. "You aren't in Kansas anymore!"

General Issues

Whether creating a server or a client program, there are many general issues that one should consider in developing such programs.

Complexity and Speed

These days it seems like few developers on lesser platforms care terribly much about optimization and speed. Those platforms typically just throw more horsepower & memory at their chores and try to go about their business.

In the Amiga world, without the glut of idle horsepower, development of any applications and internet clients & servers in particular should be mindful of overhead. For every client or server transaction, there is likely some user that clicked a button and is waiting for an answer. Keep things simple and optimize!

Incomplete transactions

Since all these programs are dependent on communications over many links, these programs need to have a robustness for failed links. Transfers can be interrupted mid-stream. Received data can be incomplete or corrupted. These applications need to have the error trapping to deal with such real possibilities - that isn't reflected in the simple examples above.

Statelessness

Given the nature of internet communications and web browsing in particular, servers and clients need to be "stateless" as possible. While a visitor may have just "logged in" to your server, it's not guaranteed the next transaction your server receives is that logged-in user, that the user hasn't left or hit the back or reload button on their browser (reloading the log-in page again). So a server should avoid making assumptions about the state of the connection and status of a visitor whenever possible or provide explicit means for addressing such things, if possible.