Memory Lane Computing

Following the rush of work on the Ser-USB 2.0 drivers and the subsequent withdrawal of that product due to component supply problems, there has been a period of consolidation and planning which has meant that I haven’t had the time to post to the blog or update our web site. Now it’s time to get things moving again.

So what’s been happening?

Last week I attended the Quanta Workshop in Manchester where I gave a talk on what we’ve been doing here at Memory Lane Computing, and was also able to demonstrate the early QL-SD prototype. It was interesting to see an unexpanded QL boot from SD Card and run the MasterSpy Editor, albeit with very little free RAM left to edit anything larger than the boot file!

This week the first of the initial batch of QL-SDs was completed and installed in a test QL. The intention is to make these units available to early adopters and beta testers at a reduced price. It is hoped this will identify (and allow us to resolve) any remaining problems that may exist with the design. There will be an official announcement on the main web site within the next few weeks about the process for obtaining one of these units, so keep checking for news.

QL-SD consists of two small boards that fit inside the QL: QLROMEXT and the SD Card socket holder which fits into a vacated microdrive slot. Let’s take a look at the QLROMEXT, shown here in various stages of construction:

(Click on the thumbnails to see the full-sized images).

The board contains a CPLD (Complex Programmable Logic Device) which implements the SPI interface for the QL, a clock oscillator, Schmitt Trigger for signal cleaning and some passive components. This board fits into one of the QL’s ROM sockets, then a single 64KByte ROM is plugged into the QLROMEXT containing the operating system and the device driver. In order to ensure a tight fit, with reliable connections, some self-adhesive plastic bumpers are affixed to the underside of the keyboard’s aluminium backplate. When the case is closed these bumpers keep the QLROMEXT firmly seated in the socket.

A ribbon cable connects the QLROMEXT to the small board that holds the SD Card socket. This board can be mounted in either of the two microdrive positions, but it is recommended to use MDV2 because the rightmost position is further away from the video circuitry which can generate a lot of noise that affects the interface.

Here are some pictures of the SD Card slot sitting right where MDV2 used to be:

Fitting the QL-SD is relatively straightforward, but it does require you to open the QL and remove the existing ROM chips.

We will be testing this board with a Super Gold Card and a Gold Card (the latter kindly loaned to us by John Southern of Quanta) to see if recent changes made to the firmware and hardware have resolved the noise issues that we reported here in an earlier post.

Progress will be reported here, so stay tuned!

The constant barrage of spam has finally taken its toll and we no longer have the time to waste reviewing the garbage that gets posted here every day by spammers and hackers. So, in order to post comments here in future you will now, unfortunately, need to be registered.

Anyone who is interested in seeing the scale of the problem need only look at the Known Spammers page, which contains a list of the (alleged) e-mail addresses of the posters. Most of those people were incapable of formulating a single grammatically correct sentence; the remainder were generated by unsophisticated spambots that produced such clearly out of context comments that it is hard to understand what the perpetrators hoped to achieve.

Ah, well. C’est la vie!

With the Ser-USB project now finally closed, work has moved on to completing the QL-SD. Enough PCBs have been ordered to complete an initial run of six units, though these are earmarked for testing environments so it is unlikely that any production units will be available to purchase until late April at the very earliest.

However, after all of the problems that we experienced with the Ser-USB, the testing will take as long as it takes and there will no official release unless that process ends 100% satisfactorily.

We have dropped plans to offer several versions of the QL-SD as the cost of development and the investment in parts cannot be justified. There will only be one version that is designed to be fitted (by the user) inside the case of a standard Black Box QL. At present, QLs fitted with Gold Card or Super Gold Card extensions cannot be supported due to timing and signal noise issues. A decision on whether to invest in finding a solution to this problem will be made when we have a better idea of the likely demand.

Followers of this blog may not appreciate that the market for new QL hardware is actually very small (and diminishing) yet the cost of developing that hardware continues to rise due to the economic situation and the inability to buy parts in bulk (because there is insufficent demand for the end product). That said, we are committed to bringing the QL-SD to market at a price well below that of the Ser-USB.

Keep checking back here for any news on progress.

The Ser-USB 2.0 Drivers were finally released today, marking the end of over a year’s development work and finally drawing a line under the project. This has unfortunately coincided with our decision to withdraw the Ser-USB from sale; the USBWiz module at the heart of it is no longer being produced and no replacement has been announced.

The device was never popular, and always too expensive (both to build and for anybody to buy) so this was an easy decision for us to make. We will now concentrate our energies on future hardware such as QL-SD and Q-BUS. More details of these will soon be published here.

On a side note I am so fed up with the constant spamming of this blog that I have now decided to publish the e-mail addresses of everyone who spams us. Just click the link Known Spammers above. OK, 90% of them are obviously false, but the information may be useful if you operate a spam filter on your own systems.

The 2.0 Ser-USB drivers are now at Release Candidate 2 with the final release planned for early in February. This makes it approximately one year since the very first Ser-USB prototype driver displayed a directory from the S*BASIC command line! Here is one of the earliest screen shots of that driver running in debug mode under QPC-II on Windows:

If I had known then just how much work would be needed to produce a driver that would work on standard QL hardware and serial ports it is very likely that I would have abandoned the project. When I look back at the work records and project logs for Ser-USB over this last year I see literally thousands of hours expended on getting this system to work reliably – the vast majority of it dedicated to solving the issues of those damn serial ports!

Anyway, what is done is done. The 2.0 drivers are nearing completion and I believe that they are now as good as they can be, given the restrictions of the underlying hardware. Certainly, they are a big step forward from the current 1.x drivers.

Still, I hope that followers of this blog will understand when I say that I will be very glad to bring this particular journey to its conclusion.

- Adrian

Beta testing of the new Ser-USB 2.0 Legacy Driver continues with the latest release finally implementing a workaround for the QDOS restriction on trap #2 io.open/io.close that effectively requires a device driver to complete those operations atomically.

As has been discussed before, with a device driver that sits on top of the serial driver, and only calls it using the documented methods, it is simply not possible to guarantee that trap #2 calls into our driver will complete in a single call.

To recap: QDOS allows a timeout value to be passed to trap #3 calls; specifying that the IOSS (I/O Sub System) automatically retry an operation until the driver signals that it has either completed or failed. This is the primary reason for the existence of error code err.nc (Not Complete).

This can’t happen with Trap #2 so we had to do the next best thing: Cheat!

Therefore, the latest beta does two things differently:

io.open hi-jacks the user mode return address and inserts a retry handler; effectively allowing io.open to support the equivalent of the IOSS retry facility. This change should make it possible for commands like Toolkit II’s WCOPY to work without locking up, but it introduces an important restriction: a trap #2 io.open call into the driver, made from code running in supervisor mode, will probably lock up the system. This should not normally be happening in application programs, but some virtual device drivers and path extensions do this (because, as device drivers, they are also running in supervisor mode). There is no easy fix for this but it is considered that the pros far outweigh the cons.

io.close no longer releases the memory allocated for the Channel Definition Block until after the filing system has flushed; the close operation continues as a background process (hooked into the polling interrupt service) until all operations are completed.

Yesterday, a JS QL running the latest driver built, faultlessly WCOPY’d the entire contents of a floppy disk (30+ files) to an SD Card over a standard QL serial port at 4800 baud without falling over. This has to be seen as a major achievement. Whether this proves to be the final final solution for Ser-USB over legacy hardware remains to be seen.

Testing continues … because here at Memory Lane Computing we know that there’s always something else to go wrong.  Always.

We’re taking a step back from the earlier decision to restrict the new Ser-USB 2.0 Legacy Driver to users with Minerva 1.98. Not all users have, or want, any version of Minerva, let alone v1.98; therefore, we have decided that it if it is not possible to have a 2.0 driver that works for all ROM versions, then it would be better to have no new driver.

Consequently, we are pressing ahead with the beta testing of a legacy driver that incorporates some “unconventional” processing to allow it to work with QDOS ROMs JM, JS, MG and Minerva. It must be proven reliable on all ROM versions or it will be withdrawn.

The signs so far are positive; the new version is running well on a test QL here with a JS ROM and standard serial ports; considerably more reliably than the previous 1.x driver – even if it is restricted to 4800 baud.

We will report back here as the beta testing of the Ser-USB Final Solution for legacy QL hardware continues.

It’s been a while since the last post here, so belated New Year greetings to anyone who genuinely follows this blog … and a plague on all your houses to the spammers, hackers and porn merchants who cause me to waste so much of my time clicking the SPAM THIS POST button!

This first post of the new year finds us reporting a mixture of good news and some not so good news.

In 2012 we have two new projects moving forwards: the Q-BUS interface which is ready to move to a production prototype and an updated driver for the Ser-USB.

The Q-BUS has been covered extensively in previous blog entries. Currently, a PLD design is in the process of being finalised. This will replace most of the discrete logic with a single chip, resulting in a much smaller and more efficent device. It is expected that this work will be completed by the end of February.

The Ser-USB 2.0 driver is a significant update that introduces a number of changes, the most important one being that the driver is now split into two distinct versions: “Legacy” for Black Box QLs running Minerva, and “Destiny” for QL Emulators on PCs/Macs/Linux running JM to SMSQ.

The Legacy driver is the first version of Ser-USB that can run from a 16K ROM without the need for any overlays or the presence of a Queue Manager. It is a significant enhancement with many improvements, including the new Write As You Go map handling that massively reduces the amount of serial I/O when writing files. However, that has come at a price: we have had to take the decision that Ser-USB 2.0 Legacy will not be supported on any operating system prior to the current release of Minerva (v1.98). Reproduced below is a (slightly edited) letter to our beta testers explaining the problems and the reason for the decision:

After a lot of investigation on an issue that has plagued this driver in all its incarnations, I have sadly reached the conclusion that it is not possible to make the version 2 driver run on any operating system other than the most recent release of Minerva (v1.98).

This is why:

Ser-USB is a device driver sitting on top of another device driver; it needs to use the serial ports to communicate with the Ser-USB. The only reliable and supported way of doing this is via QDOS trap #2 and trap #3 calls to the serial driver. Thus, when the Ser-USB driver handles a trap #3 request for the SRU device it must ultimately pass the request down to the serial driver using its own trap #3 call.

Serial I/O calls from supervisor mode have to happen with zero timeouts or the scheduler will be re-entered whilst inside the driver code. This ultimately brings the system down. Not always immediately, but inevitably at some subsequent point.  Unfortunately, under most QDOS versions it is not possible to use zero timeouts in supervisor mode with trap #3 operations on the serial port because the serial I/O never completes. The exact reason for this is not known but is believed to be related to the way that interrupts are handled.

Ser-USB 1.x got around this by spawning a user mode process to do the I/O using a queue of transactions. Although it worked, I would be the first to admit that it was never sufficienty reliable or robust, despite hundreds of man hours having been expended on fixing it. That is why we introduced the FAT driver for “Black Box” QLs.  Whatever happens now, the Queue Manager’s day is over and it _will_ be deprecated.

Ser-USB 2.x takes a different approach. It uses zero timeouts for the trap #3 calls and immediately returns err.nc to QDOS; meaning that the QDOS IOSS is handling retries to the serial driver by routing them through the Ser-USB driver. This actually works very well, except in one scenario:

The designers of QDOS decided that trap #2 operations should not participate in the IOSS retry mechanism (you can’t pass a timeout in d3). This effectively means that an io.open call must complete atomically; it cannot return to QDOS until it is finished. It should therefore be obvious that zero timeouts for the trap #3 calls to the serial driver that are needed to open a file on the Ser-USB cannot happen – because they would never complete! Therefore, in that one scenario, a rule is broken and a fixed timeout is used to call the serial driver. I thought that we had found a fix, but it seems that it only appeared to work … as has so often been the case in the history of this issue

This does not cause any problems for Minerva, but ALL prior versions of QDOS will eventually crash as a result. One tester has experienced this problem first-hand with an MGE ROM. For him, the 2.0 driver cannot be used.

I have therefore decided that I will not support OS versions earlier than Minerva 1.98 in the 2.0 driver. If you do not have Minerva, please accept my apologies, but this effectively ends any beta testing that you will be able to do. It also means that Ser-USB Driver 1.04 will be the last official driver for pre-Minerva systems.

I am personally very disappointed to have had to take this decision, as I feel that the 2.0 driver is in all other respects superior to what has gone before.

Please note that the “Legacy” driver may appear to work under SMSQ/E but this is an illusion! It will crash.  The Ser-USB Legacy driver is not currently supported under SMSQ on any hardware platform.

btw The USBWiz over Q-BUS driver does not have any of these problems because it uses the Q-BUS hardware and has no need to use the serial driver.

So there we have it. The Good, the Bad and the ugly QDOS supervisor mode serial I/O issue that, to this day, remains stubbornly insoluble.

Happy New Year!

The EEPROM discussed in the last blog entry was intended to be a mechanism for storing additional drivers on the Q-BUS itself, without the need for any other functioning storage device. However, when looking at the code needed to achieve this on the QL side it became obvious that with just a little more effort it could be made into something slightly more sophisticated.

The latest version of the Q-BUS Driver (that’s the driver that will live in the ROM on-board the Q-BUS, as opposed to the previously mentioned USBWiz over Q-BUS driver which is loaded separately) implements our in-house filing system for very small storage devices: ATFS.

ATFS (A Tiny Filing System) was designed to allow small ROMs, EPROMs, EEPROMs and flash devices to hold small amounts of data in a managed way. It does this without the massive overhead of a File Allocation Table or large sector sizes, by storing everything as an object in a single linked list that starts from device address 0.

The Q-BUS driver provides a handful of S*BASIC commands that allow users to store and retrieve data from the EEPROM:

QE_PREP: Prepare the EEPROM for use as an ATFS volume.

QE_STORE, QE_BSTORE, QE_PSTORE, QE_XSTORE: Store a data file, S*BASIC program, executable program or resident extension to the EEPROM.

QE_LOAD: Load (view, execute or install) a file from the EEPROM.

QE_LIST: List the objects on the EEPROM.

For example, if you had the USBWiz over Q-BUS driver stored as an Extension Object on the EEPROM, you would be able to load the driver with the command:

QE_LOAD "qbd_bin"

Here are some screen dumps showing the output of QE_LIST and the result of QE_LOADing a DATA object.

There’s more on ATFS in the Q-BUS preliminary information document.

ATFS is not, and never will be, a substitute for a “real” filing system device but it does add a powerful feature that increases the usefulness of the Q-BUS design and greatly simplifies the loading of supplementary drivers and other extensions.

The alpha version of the USBWiz over Q-BUS driver now has all of the functionality of the current (v1.04) Ser-USB driver and appears to be working well. The most recent changes have been to implement the full API layer so that the existing utility programs (such as the Partition Manager) will work without changes.

Today saw the incorporation of the last hardware component of the Q-BUS design that had been lower down on the priority list. There is now a 64Kbyte EEPROM on-board which can be used to hold configuration data, driver extensions etc. These pictures show how it has been built onto a mini-breadboard on the top deck of the Q-BUS prototype enclosure, just between the Arduino MCU and the USBWiz. A production design would have this chip situated on the Q-BUS controller.

The device can be written to and read from through the remote management interface using a terminal program on a PC or Mac connected to the MCU’s USB port. Two new commands have been added to the MCU firmware to permit the contents of the EEPROM to be downloaded on the QL side. A typical application of this would be for the standard QL ROM on the Q-BUS controller to hold the base Q-BUS driver which, in turn, is able to load the USBWiz driver (or any other driver) from the EEPROM, thus sidestepping the QL plug-in ROM size limit.

The fact that the EEPROM can be updated from a PC makes the possibility of online firmware upgrades a very real possibility, and there is no real reason why the device could not also be written to from the QL side.