Copyright (c) Hyperion Entertainment and contributors.
AmiWest 2013 Lesson 4
Contents
Simple IP Clients & Servers
Simple network access is easy!
Developing applications that use TCP/IP "sockets" for network communications have a reputation of being a challenging task. But AmigaOS and its Roadshow TCP/IP stack provides a couple of built-in shorthand mechanisms for easily creating simple network clients and servers: the TCP: handler and Roadshow's Superserver.
Using these Roadshow shortcuts, creating simple network client and server applications for single user or light-duty uses is no more complicated than reading or writing a file. The real challenge is understanding the protocol to be used (f.e. HTTP communications with web browsers) and being careful not to create a security risk on your or another user's Amiga.
An Internet Client
First, we will look at creating a small client to retrieve some simple information from an Internet website. Similar applications could be made to retrieve basic information like stock quotes, email contents, server conditions or other basic interactions. Aminet has a number examples of such clients written in ARexx that use the TCP: device that can be examined.
The Roadshow shortcut we will use is its built-in TCP: handler. This is mechanism that creates a virtual filesystem device ("TCP:") whenever Roadhsow gets online. To interact with a remote server, one just opens a file with the name being the URL of the server and port to be accessed.
Connect to the Server
To begin with, one opens a connection to the internet server as if one were opening a file. As described above, when Roadshow gets online, it creates a virtual TCP: device. To open a connection, one opens a file to the URL and port in question using this format:
TCP:<server URL>/<port number>
For example, to connect to the website "www.wunderground.com" one would use:
fileh = fopen("TCP:www.wunderground.com/80","r+");
The file name starts with TCP:, then the website URL and finally the port number - here "80" is site's HTTP port (used by almost all websites).
Typically, the port number reflects the type of port or protocol that is to be used. Such protocols could be FTP (file transfer), POP/SMTP (email) or many others. Port 80 is usually used for HTTP communications with websites.
If the file open request is successful, your Amiga is then connected to that server and any further interaction with the "fileh" file handle communicates directly with that port on that server. In this case, we would have connected to a websserver.
Talk to the Server
Once one has connected to a server, knowledge of the communications protoccol for that type of connection is required.
For each of those, a server expects a certain format of interaction to proceed and respond. Fortunately, common protocols are publicly documented.
http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers
For each protocol there is usually an "RFC" document that describes how one uses the protocol, interacts with the port and accomplishes that protocol's goals.
In the case of "HTTP" (or a "Hypertext Transfer Protocol") connection to a webserver, we are usually expected to submit a "GET" request. Here are a couple more complete references on the HTTP protocol:
http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
http://tools.ietf.org/html/rfc2616
To keep things simple, lets just look at submitting a "GET" request, such as a web browser, like IBrowse, would do. A simple HTTP version 1.1 GET request would include the following lines:
GET /<path> HTTP/1.1 host: <domain> user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build) Pragma: no-cache Accept-Language: en, * Accept: text/html;level=3 Accept: text/html;version=3.0 Accept: */* <blank line>
The <path> would be the rest of the URL (if any) on that system and <domain> is the name of the server.
Using the "WeatherUnderground" website URL from above and their pathname for searching for airport conditions, we can find the current conditions at Washington's Dulles International Airport with these domain & path values:
URL = www.wunderground.com PATH = cgi-bin/findweather/getForecast?query=IAD
The GET request syntax can be transcribed into C code which builds the entire request in a string and then sends it to the server with a simple "fprintf" to our open file handle, as follows:
// build web req text strcpy(line,"GET /"); strcat(line,path); strcat(line," HTTP/1.1\r\n"); strcat(line,"host: "); strcat(line,domain); strcat(line,"\r\n"); strcat(line,"user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build)\r\n"); strcat(line,"Pragma: no-cache\r\n"); strcat(line,"Accept-Language: en, *\r\n"); strcat(line,"Accept: text/html;level=3\r\n"); strcat(line,"Accept: text/html;version=3.0\r\n"); strcat(line,"Accept: */*\r\n"); strcat(line,"\r\n"); // send web req line to server fprintf(fileh,"%s",line);
As you can see, the variables "path" and "domain" are used to fill in some blanks in the request. The blank line at the end completes the GET request, after which the server will reply using the same file handle.
Listen to the Server
Once the GET request has been sent to the webserver, the server will start sending back either the requested webpage (or other content) or an error page. To see what sort of content is returned by the server, you can combine the domain and path values above and enter them in your web browser:
www.wunderground.com/cgi-bin/findweather/getForecast?query=IAD
Once the page is loaded, viewing the page source (f.e., IBrowse menu item "Page/Display Source...") will show you the same text that your program will receive after sending the get request.
As such, your program needs to start reading from the same file handle the request was sent with. This loop will print out the first hundred lines returned:
// read in the response count = 0; while( (fgets(inStr,MAX_STR,fileh) != NULL) && (count<100) ) { ++ count; printf("line %ld = %s\n",count,inStr); // process lines received here ! }
Within this loop your applcation can parse the received lines for whatever data your application is tryng to obtain. In the case of our example program below, the program searches for a string precedes the weather information we want.
After the returned content has been read, output or otherwise processed, simply closing the file handle will close the connection to the server:
fclose(fileh);
EXAMPLE ONE: IPClient.c
All of these elements are combined in the following example program "IPClient.c" that asks you for a IATA airport code (f.e., IAD = Washington Dulles, FRA=Frankfurt, SYD=Sydney), then parses & prints out the weather conditions from the returned page.
Please Note: this example program worked with the wunderground.com website in 2013-2014, as the served webpages change over time, it is likely the parsing in this example is likely to fail and the program no longer return useful information.
/* ************************************************************ ** ** Created by: CodeBench 0.41 (12.10.2013) ** Project: IPclient ** Date: 12-10-2013 18:51:10 ** ************************************************************ */ #include <stdio.h> #include <string.h> #include <proto/exec.h> #include <proto/dos.h> /***************************************************************************** * GLOBAL VARIABLES *****************************************************************************/ STATIC CONST_STRPTR version USED = "$VER: IPclient v.02 (12.10.2013)"; char URL[200] = "www.wunderground.com/cgi-bin/findweather/getForecast?query="; char domain[200]; char path[200]; char fname[256] = ""; #define MAX_STR 2056 char line[MAX_STR] = ""; char inStr[MAX_STR]; uint16 uLen; uint16 dLen; uint16 count = 0; char *found; FILE *fileh; /***************************************************************************** * PROGRAM START *****************************************************************************/ int main(int argc,char **argv) { // get URL from user printf("IPclient example\n"); printf("======================\n"); printf(" enter airport code >"); fgets(line,sizeof(line),stdin); line[strlen(line)-1] = '\0'; // strip \n off end of string printf("======================\n"); // append airport code on URLprefix strcat(URL,line); // parse domain from path uLen = strlen(URL) - 2; strcpy(domain,strtok(URL,"/\n")); dLen = strlen(domain); if (uLen>dLen) strcpy(path,strtok(NULL,"\n")); else printf(" No chars remain for path\n"); printf(" URL domain >%s<\n",domain); printf(" URL path >%s<\n",path); // build URL filename from domain name strcpy(fname,"TCP:"); strcat(fname,domain); strcat(fname,"/80"); // open file access to URL printf(" Opening URL >%s<\n",fname); fileh = fopen(fname,"r+"); if (fileh == NULL) { printf("Couldn't open connection domain server via \"%s\"\n",fname); return RETURN_ERROR; }; printf(" TCP: file opened\n"); // build web req text strcpy(line,"GET /"); strcat(line,path); strcat(line," HTTP/1.1\r\n"); strcat(line,"host: "); strcat(line,domain); strcat(line,"\r\n"); strcat(line,"user-agent: IBrowse/2.4 (AmigaOS 4.1; PPC; 68K build)\r\n"); strcat(line,"Pragma: no-cache\r\n"); strcat(line,"Accept-Language: en, *\r\n"); strcat(line,"Accept: text/html;level=3\r\n"); strcat(line,"Accept: text/html;version=3.0\r\n"); strcat(line,"Accept: */*\r\n"); strcat(line,"\r\n"); // send web req line to server fprintf(fileh,"%s",line); strcpy(line,""); printf(" Web req sent\n"); printf("======================\n"); // read in the response while( (fgets(inStr,MAX_STR,fileh) != NULL) && (count<100) ) { ++ count; //printf("line %d = %s\n",count,inStr); // look for name of airport found = strstr(inStr,"og:title"); if (found != NULL) { found = strtok(inStr,"="); found = strtok(NULL,"="); printf("WeatherUnderground.com reports:\n"); printf(" Airport = %s\n",strtok(NULL,"|")+1); printf(" Temp =%s F\n",strtok(NULL,"&")); found = strtok(NULL,"|"); printf(" Condition =%s\n",strtok(NULL,"\"")); } } printf("======================\n"); // close file fclose(fileh); return RETURN_OK; }
Designing Your Network Client
As you see, the above code and the example program, the mechanics of connecting to a internet server and obtaining data are relatively trivial. The real challenge for the developer lies in understanding the protocol used and in processing the data sent and received.
Public webservers make for fairly rich and easy targets to interact with. You can explore simple interaction with any webbrowser and mimic that interaction with your code. In many cases, you can control the feedback just by crafting the web request as we did above and then parse the results.
Naturally, the more complicated the interactions, like logging into a website, would require much more knowledge of the protocols used more complicated code that is beyond the scope of this tutorial.
But there are a number of basic topics you should consider when designing a web client!
Server Suitability:
Unless you are connecting to your own server or it invites such connections (some publish API's for you to make such connections), your application's "visit" may not be welcome or even legal.
At the very least, your program should "tread lightly" - do not abuse the server with unnecesary or intrusive requests. Furthermore, if your client uses the resources of someone else's server, you should credit the server within your application and documentation.
Server Changes:
As with any programming that interacts with external inputs, your code should provide for the possibility the connection may not succeed or that the returned results may not be what was expected.
As servers and websites change over time, the processing and parsing of your program may often have to change accordingly. To maintain your program and deal with such changes, you may want to consider how to make changing those things easy (by the user?). The parsing strings could be kept in program tooltypes or config text file, you could provide a GUI for managing the parsing, employ an external, editable ARexx macro.
Protocol Details & Vagaries:
Our example code presented a very simple interaction with a web server using the a widely supported version of the HTTP protocol. As one develops an application and uses a protocol, one should get familiar with the details of that protocol. Typically a "RFC" document (as linked above) will describe the details of how things are to work and how they might fail. Your application should be careful to comply with the details of the protocol ("Are we sending a CR-LF or LF-CR?!").
Ideas
The above example was just a quick, simple exercise in demosntrating the use of the AmigaOS Roadshow TCP: handler to access a web server. There are many ways applications could be created to have simple interactions with all sorts internet services, such as:
· Check stock quotes from a financial site. · Get package tracking information from a shipping site. · Check for new emails on a POP email server. · Convert currencies with a financial or travel web site. · Check for files on an FTP server. · Send messages with a SMTP email server. · Look up words on a dictionary website.
You could also write your own server for another Amiga and interact with your own client. A remote controlled media player?
An Internet Server
Next we will look at creating a simple internet server program that relies on AmigaOS Roadshow's "SuperServer" to receive the incoming internet connection and share it with our application.
Simply speaking, once Roadshow is configured to recognize the incoming request and that your application is there to handle it; our program just has to deal with another case of simple file-like interaction.
For this example, we will create a simple server that accepts an HTTP protocol request (like from IBrowse) and then respponds to it with a simple webpage. Of course, you could write a server to serve an interactive webpage, some machine information, media files, etc. Just be careful.
Receive the Request
When Roadshow receives an internet request on the designated port that corresponds to our server, it starts our server program and routes the network connection to our program using the "standard" input stream (otherwise known as "stdin").
Our application then reads the stream almost like reading from any other file or user input, like this:
// read in the response while( (fgets(inStr,MAX_STR,(FILE *)stdin) != NULL) && (count<100) ) { if (strlen(inStr)<3) { IExec->DebugPrintF(" received nearly empty line, ending read loop.\n",NULL,NULL); break; } ++ count; IExec->DebugPrintF(" %s\n",inStr,NULL,NULL,NULL); }
As you can see, we loop reading incoming lines until we encounter a line with two or fewer characters (essentially an empty line with what should be a CR-LF). There is also a counter that kicks us out if the loop if we read 100 lines - that's a sign something else is wrong, we should never see that with the HTTP protocol.
With the HTTP protocol, the first line is where we would see what is being requested. Were our server to use other protocols, we may have to add our server's "intelligence" elsewhere.
We also do all our diagnostics output using the "DebugPrintf". Which means we either need to run Sashimi or watch output on the serial port (with your second Amiga, naturally). In the case of this example, we will see the entire incoming request printed out on the serial port. Of course, there's a reason why we have to use the serial for our debug output...
Respond to the Visitor
Once we've read the (almost) empty line at the end of the incoming web request, we will use another "standard" stream (know as "stdout") to respond to our internet visitor.
Anything we output to "stdout" will be sent by Roadshow back to our internet visitor. This is the reason why we used the serial port for diagnostic print outs, since a simple "printf" would have gone to our web visitor.
In this example, we create a basic loop that feeds a predefined series of lines of a very simple webpage to Roadshow and our web visitor:
//loop thru text writing to stdout do { if (strlen(lines[l]) > 0) { IExec->DebugPrintF(" printing line = %s\n",lines[l],NULL,NULL); if (fprintf(stdout,"%s\n",lines[l]) < 0) { IExec->DebugPrintF("ERROR printing line\n",NULL,NULL); break; } fflush(stdout); } else { IExec->DebugPrintF(" empty line, loop end\n",NULL,NULL); break; } ++l; } while (l < 25); // emergency loop escape to stop endlessness
As you can see, our loop ends when we hit an empty line in our webpage definition or a maximum of 25 lines (a failsafe). At then end we "flush" the output and quit our program. The request was received, our webpage sent and Roadshow takes care of all the housekeeping.
EXMAPLE TWO: IPserver.c
We can see all these pieces and the definition of our simple webpage in the full program here:
/* ************************************************************ ** ** Created by: CodeBench 0.41 (12.10.2013) ** Project: IPserver ** Date: 13-10-2013 23:41:17 ** ************************************************************ */ #include <string.h> #include <stdio.h> #include <proto/exec.h> #include <proto/dos.h> /***************************************************************************** * GLOBAL VARIABLES *****************************************************************************/ STATIC CONST_STRPTR version USED = "$VER: IPserver 0.14 (13.10.2013)"; // define maximum web request lines & length #define MAX_STR 2048 // ARS path & file name uint16 count = 0; char inStr[MAX_STR]; CONST_STRPTR lines[] = { "HTTP/1.0\015", "Content-Type: text/html\015", "\015", "<HTML><HEAD><TITLE>IPserver</TITLE></HEAD><BODY BGCOLOR=\"cccccc\">\015", "<TABLE WIDTH=100%><TD ALIGN=\"center\" BGCOLOR=\"cccccc\">\015", "<BR><TABLE WIDTH=90%><TR><TD><B><H1>WELCOME</H1></B><BR></TD></TR></TABLE>\015", "<TR><TD BGCOLOR=\"6688ee\"><H3><P><P><FONT COLOR=\"white\"><B>AmigaOS IPserver example</B></FONT></TD></TR>\015", "<TR><TD ALIGN=\"center\" BGCOLOR=\"cccccc\">\015", "<BR><TABLE CELLSPACING=0 WIDTH=90%><TR BGCOLOR=\"ffffff\"><TD WIDTH=100><H4><B>And so it begins...</B></TD><TD> Your Amiga internet server! </TD></TR></TABLE>\015", "</TR></TD></TABLE></BODY></HTML>\015", "" }; uint16 l = 0; /***************************************************************************** * PROGRAM START *****************************************************************************/ // Starting program int main(int argc,char **argv) { IExec->DebugPrintF("======================================\n"); IExec->DebugPrintF("IPserver starting\n"); IExec->DebugPrintF("======================================\n"); // Was program started from shell or WB ? if (argc > 0) { // read in the response while( (fgets(inStr,MAX_STR,(FILE *)stdin) != NULL) && (count<100) ) { // #### Test for CR-LR only line if (strlen(inStr)<3) { IExec->DebugPrintF(" received nearly empty line, ending read loop.\n",NULL,NULL); break; } ++ count; IExec->DebugPrintF(" %s\n",inStr,NULL,NULL,NULL); } IExec->DebugPrintF("======================================\n"); IExec->DebugPrintF("FINISHED READING WEB REQUEST\n"); IExec->DebugPrintF("SERVING WEB PAGE...\n"); IExec->DebugPrintF("======================================\n"); //loop thru text writing to stdout do { if (strlen(lines[l]) > 0) { IExec->DebugPrintF(" printing line = %s\n",lines[l],NULL,NULL); if (fprintf(stdout,"%s\n",lines[l]) < 0) { IExec->DebugPrintF("ERROR printing line\n",NULL,NULL); break; } fflush(stdout); } else { IExec->DebugPrintF(" empty line, loop end\n",NULL,NULL); break; } ++l; } while (l < 25); // emergency loop escape to stop endlessness IExec->DebugPrintF("======================================\n"); IExec->DebugPrintF("Web service finished... \n"); } else printf("IPserver started from Workbench - Don't do it again, this is a Roadshow app!\n"); IExec->DebugPrintF("IPserver Quitting!\n"); IExec->DebugPrintF("======================================\n"); return RETURN_OK; }
As you can see there's also a bit of housekeeping code at the beginning that tells a user this program wasn't meant to be run from the Workbench. As it stands, this is program is only meant to be used by Roadshow for serving internet visitors to your Amiga.
But how do we test this all out? First we have to let Roadshow know that we've created this
Roadshow Configuration
There are two areas where Roadshow needs to be told about your server and the service it is going to provide. Both of these can be configured using Internet Prefs. You can also make this configuration by editing the "servers" and "services" files in the "DEVS:internet/" directory. Configuration of these files is described in the file:
SYS:Documentation?Roadshow/README
If you would like to spare your user from having to do either of those chores, your application can even modify those files. As soon as Roadshow sees those files have been modified, it will reconfigure itself accordingly.
Services
First, we need to tell Roadshow about the "service" we are providing. If one opens Internet Prefs and clicks on the "Services" page, one will see a list of standard internet services with their TCPIP port numbers, type and aliases.
Click the "New..." button to define a new Service. An "Add service" window will open where we can define how our server will be accessed. Criticially, we need to pick a port number that is not already in use. Such settings could be:
Name = IPserver Port = 7600 Type = tcp
Then we click "Use" to accept our new values. As soon as one clicks "Save" in Internet Prefs, this service will be recognized by Roadshow.
The same configuration could be added to Roadshow by adding this line (in port number location) into the file "DEVS:Internet/Services":
IPServer 7600/tcp
As soon as one saves the file, Roadshow will be notified and adjust itself for the service.
Servers
Before we can use the service above, we also need to tell Roadshow about our server app and where to find it. Again, this can be done while in Internet Prefs. Click on the "Servers" page and one will see any servers configured with the services they provide (as set above), type, wait and program path.
Click on the "New..." button to define a new Server. In the "Add server" window we can define what sort of service our server is for, the characteristics of how it will be called and where it is located on our Amiga system. Such settings could be:
Service = IPserver Type = Stream Stack = 65536 Program = <where are server program is> Arguments = <empty> Active = <checked> Wait for completion = <not checked> Use socket I/O streams = <checked>
Again we click "Use" to accept our new Server entry and saving Internet Prefs will adjust Roadshow.
We can also make this adjustment to Roadshow's server list by adding the following line to the "DEVS:Internet/servers" file:
IPServer stream dos stack=65536 data:Projects/C/OS4ex-IPserver/IPserver.debug
Again, saving the file is automatically noticed by Roadhsow.
Once both the service and server are configured, Roadshow is ready to use our new server.
Let's See Our Server!
To test our new server example, you can access it by connecting to your Amiga and the designated IP port number. On the same machine, you can enter this in your browser URL line:
http://localhost:7600/
From another machine, you will need to have your test server Amiga's IP address or a host name configured on your network. The URL would look something like this:
http://192.168.1.07:7600/
In any case, the browser will connect to your Amiga at the port number following the colon (":") and Roadshow will call up your server program and let it talk to your browser.
Where to go? More Webpages?
Clearly the above example does little more than say "Hello World" to a visiting web browser user. It offers no interactivity, preforms no task and serves no media.
When this web server receives the incoming web request, the first line (starting with "GET" tells the server what the browser is looking for. If you watch the serial output and connect with a newer browser than IBrowse, you can see most browsers make two connections with one of them looking saying:
GET /favicon.ico HTTP/1.1
In that case, the browser is looking to see if your server has an icon to send for the displaying on the browser tab. Just like that, anything written in URL after the machine's name or IP address and the port number is taken a requested path and your server will receive it after "GET" in the first line of the request.
This just scratches the surface of requests your server may receive and how it can respond to web visitors. To dig deeper you can search the many sources online or in your local bookstore on HTTP communications and the formatting of HTML webpages.
Just Webpages?
There's also ne reason why your server needs to be limited to dealing with browsers and sending webpages. The internet is full of other protocols and services that your could write a server for. Or you could create your own online client and server functionality.
Just as we discussed with creating an Internet client, there are many topics you should consider when designing an internet server program. It's not just about the coding!
Security
Obviously, opening your Amiga with an internet server represents more of a risk than if you had done nothing. So it is criticially important one considers security when creating a server. While it's beyond the scope of this article to inclusively discuss all the issues involved, there are basics to consider.
Don't blindly serve sensitive information to visitors. How can you be sure who's at the other end of the connection and who might be watching along the way? Even if randomized pathnames and request are used by your server, there's no guarantee the connection is 100% secure.
Limit commands and functions served to visitors. Obviously, creating a means of offering anything like command line access or full file editing/deleting powers to a web visitor would represent dangerous hole in any system's security. But one has to make sure no whatsoever holes or bugs exist in the server design that would let such access slip through.
Server Instances & Load
One has to keep in mind the "SuperServer" method where Roadshow runs your server also means your server will be executed with each and every request Roadshow receives for that port. In each case, your server will do its duty and then be expected to quit. That should be fine light duty, personal server tasks.
But one should focus server development on keeping things as "lightweight" and simple as possible. If the server needs to do read data files or perform other complicated initialization, this may slow reaction time for each incoming request an unacceptable level.
To avoid time consuming chores, one might considering using interprocess communciations (such as ARexx messages) to interact with a separate master program that handles all the housekeeping and remains running while each of internet server instance handles an incoming request and quits.
Naturally, at some point use requirements may require one bypasses the Roadshow SuperServer method of running servers and goes to using longhand socket programming that is better suited to heavy duty use - like running a serious public server.
Furthermore, there are whole areas of study and technology involved with the handling of network user load that should be applied to any serious server system.
Protocols and Compatibility
As one develops a more complex or interactive server meant to handle a wider group of users (even those on lesser platforms!), one needs to pay much closer attention to the protocols involved.
As mentioned in elsewhere in this text, there RFC's that describe the formal protocols and there are likely countless webpages that address idiosynacracies of implementing the protocols.
Beyond all those, a developer will need to do serious testing with as wide a pool of possible clients to refine a server's operation. "You aren't in Kansas anymore!"
General Issues
Whether creating a server or a client program, there are many general issues that one should considerin developing such programs.
Complexity and Speed
These days it seems like few developers on lesser platforms care terribly much about optimization and speed. Those platforms typically just throw more horsepower & memory at their chores and try to go about their business.
In the Amiga world, without the glut of idle horsepower, development of any applications and internet clients & servers in particular should be mindful of overhead. For every client or server transaction, there is likely some user that clicked a button and is waiting for an answer. Keep things simple and optimize!
Incomplete transactions
Since all these programs are dependent on communications over many links, these programs need to have a robustness for failed links. Transfers can be interupted mid-stream. Received data can be incomplete or corrupted. These applications need to have the error trapping to deal with such real possibilities.
Statelessness
Given the nature of internet communications and web browsing in particular, servers and clients need to be "stateless" as possible. While a visitor may have just "logged in" to your server, there's no automatic way to know the next transaction your server receives is that logged in user, that the user hasn't left or hit the back or reload button on their browser (reloading the log-in page again). So a servers should avoid making assumptions about the state of the connection and interaction with a visitor whenever possible or provide explicit means for addressing such things.
