Wednesday, September 4, 2013

Back to basics: A demo http server and the TCP/IP stack on Windows

Let us build a web server

At work, I spend a lot of time on trivial things like 'how to fix html/css to make this page work for IE7'. I realized it has been a long time since I had to look at how the web server which powers our product really works under the hood. The best way to refresh my memory was - implement a web server and learn how it works bottom up. I used the language and platform I am currently using at work: C# and Windows.

Webserver implementation

Source code in GitHub
I updated code someone left on pastebin. The link is gone now, so I cant reference it. This is a toy implementation, error handling or performance are not considered. It uses the .NET TCP library and implements a small subset of the http server side protocol. Features:
- Simple threaded handling of requests
- Logging
- URL re-write
- Implement GET and POST
- CGI meta-variables
- Powershell as a CGI extension
- Process Cgi and File GET
- Implement some error codes - 400, 403 and 404


If I had to implement any of the protocol layers, the RFCs and standards corresponding to that layer would be a good starting point.
Diagram modified from:

Windows TCP/IP stack

OSI Layers 1 and 2

The driver interface is generally implemented by a Network Interface Card (NIC) vendor. To program a driver, you need to get the Windows Dev Kit (WDK), ndis.lib (Network Driver Interface Specification library). Good places to start would be the WDK 'passthru' sample and source code for the WinPcap (Windows Packet Capture) library. C or C++ are the languages to program with the NDIS.

OSI Layers 3 and 4

TCP/IP is implemented in the protocol driver TCPIP.sys, as part of the Windows kernel. At this layer AFAIK there is limited programmability. Windows does provide several configuration settings in the registry: TcpNumConnections, TcpWindowsSize and others. There are command line programs which operate at this level: arp, route, netstat, tracert, netsh, nsloopup, ipconfig. I have read about people who actually patch the TCPIP.sys, but I dont know how this is done.

OSI Layers 5-7

At this layer we are above the kernel in the user space. Various dlls implement different services:
TCP and UDP sockets - ws2_32.dll, winsock.dll
WinInet (FTP, HTTP 1.0, HTTP 1.1) - wininet.dll
RPC - rpc*.dll
NetBIOS - netapi.dll
WNet - mpr.dll
Diagram modified from:


Notes on protocol specs for the web server implementation.

HTTP 1.1 – RFC 2616

- The HTTP protocol is a request/response protocol.
- A client sends a request to the server in the form of a request method, URI, and protocol version, followed by a MIME-like message containing request modifiers, client information, and possible body content over a connection with a server.
- The server responds with a status line, including the message's protocol version and a success or error code, followed by a MIME-like message containing server information, entity meta-information, and possible entity-body content.

MIME – RFC 2045 etc.

- Multipurpose Internet Mail Extension
- Defines how to send data over the internet
- Content-types: text, multipart, message, application, image, audio, video

Diagram from:

HTTP Communication

request chain ------------------------>
<---------------------- response chain
UserAgent------------- Origin server
Browser---------------- Webserver

request chain -------------------------------------->
<------------------------------------- response chain
UserAgent --------A-------B------- Origin server
Browser----Firewall----Router----- Webserver

Http Request:

HTTP-message = Request | Response
Request = Request-Line
*(( general-header
| request-header
| entity-header ) CRLF)
[ message-body ]
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Method = "OPTIONS" | "GET" | "HEAD" | "POST" | "PUT" | "DELETE" | "TRACE" | "CONNECT" | extension-method

Http Response:

Response = Status-Line
*(( general-header
| response-header
| entity-header ) CRLF)
[ message-body ]
Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF
Status-Code = "100"| "101"| "200"| ...
Reason-Phrase = *

CGI – RFC 3875

Common Gateway Interface
How a HTTP server and script/program share responsibility to respond to client request.
Request meta-variables.
Contain data about the request passed from the server to the script.
Can be accessed by the script in a system defined manner.

No comments:

Post a Comment

Boston, MA, United States