muFORTH

Rumours of muFORTH's death are exaggerated

2010-07-12T23:07:00.000-07:00

While it might appear that work on muFORTH has ceased entirely, such appearances are mere illusion. A look at the github repo will show a couple of bursts of activity. I spent most of September & October of 2009 working feverishly on the HC08 and HCS08 support, developing a nifty simple bootloader for the chips, and mastering eldritch art of programming the flash memory (which is different on the two families!).

Then, this past February, I returned to the ARM support and hacked feverishly on that for a month or so – creating a completely new target compiler, and re-writing and augmenting the ARM kernel to be direct-threaded (rather than subroutine-threaded), and self-hosting. That was the first time I had written a target compiler that compiles names (ie, a dictionary) on both the host and target. The definitions of defining words in such a system really become mind-bending...

I'm currently experimenting with some novel (for muFORTH) features, such as 32-way hashing of the dictionary, robust & portable double number support (in C – God help me), and a refactoring of the "platform" support code (so muFORTH could, in principle, run on non-*nix-alike platforms).

I look forward to someday returning to the HC08 work. I was having a great time last fall working on it, and still have high hopes for using the chips in practice.

Stay tuned!

Windows 7 impressions, post-install

2009-01-25T14:22:00.000-08:00

Well, that was interesting.

I had to install it twice, which is about par for installing any operating system, in my experience. Since I was installing and running it under VirtualBox (version 2.1.0), it would have been smart to check for any new Vbox updates before doing the install. As it was, I told Vbox I was installing "Windows 2008" and it initially worked - mostly.

But the post-install update failed mysteriously, and Help didn't work properly, and things seemed a bit off-kilter in general. In the middle of all this, I got a message that there was a new version of Vbox available - 2.1.2. Thinking this might have some support for Windows 7, I shut down the VM, downloaded and installed the new Vbox... and, sure enough, it has added support for Windows 7!

So I deleted the existing Windows 7 VM and started over, re-installing it under Vbox 2.1.2. This worked a lot better. Updates worked, things seemed pretty good.

Until I tried installing CodeWarrior.

Of course, once again, if I had done my due diligence I would have learned that CW is supported under Vista 32-bit, but not 64-bit! What was weird, though, was that even the installer wouldn't run. The arrow cursor would get its "spinning circle" side-kick, and it would never stop. And in fact, from this point on Windows wouldn't shut down properly.

Windows 7 has a "program compatibility" application that can create some kind of "shell" to runs applications in. I pointed this at the CW installer and got it to start up, but even so I couldn't complete an install. And why isn't this compatibility stuff simply built-in to the OS?

Even after re-installing under Vbox 2.1.2, Help doesn't work. If you dig down to an actual help page and select it, you jump back to the top of the tree, without having seen any useful text (not the Windows Help is often useful... ;-)

Windows 7 has potential. It's easy to install, the interface is attractive (I'm not sure if I was experiencing the "Aero" interface or the generic Vista one - I didn't have OpenGL support turned on in Vbox, so I assume the latter), it's easy to navigate, and the dialogs and help messages are informative and helpful. It would be nice if the Help worked, but I'm sure that will get fixed.

My two main beefs with it are that sitting idle with no applications running it uses 40% of the 768MB I allotted to it, and it keeps hitting the disk. It's a bit annoying. It would be really cool if there was a way to configure Windows so that services that you truly don't care about can be stopped and pulled from memory. There was a bunch of stuff in the Task Manager that looked dubious - several processes each consuming ~25MB of memory.

If I can fire up a bare Linux 2.6 kernel in ~30MB, why can't I do the same with Windows? Ok, granted, that's an unfair comparison, since Linux is running in console mode. So add another 30MB for X - again, before we've started any apps. That seems generous enough. So why doesn't Windows fit into 60MB? That's an enormous amount of memory.

I'm not holding my breath that Windows 7 will save Microsoft, but with more some work on their part it could be made into a workable & attractive system. Other people are more willing than I to ignore its (piggish) resource use - but not everyone! Witness the lack of adoption of Vista... people were unwilling to buy new machines (with 4x the memory of their old ones) simply to run that then-new OS.

That may end up still being true.

I'm downloading a beta of Windows 7

2009-01-24T16:59:00.001-08:00

God help me. ;-)

After reading piles of dire news about Microsoft's share price, and wondering what they were up to in general, I stumbled across some upbeat news about how Windows 7 has the potential to erase the mistakes (and bad press) of Vista. There is some hope that people who never upgraded (from XP to Vista) will upgrade to 7. Of course others claim that Microsoft's arrogance about dictating new UIs with every new version of Windows is going to be their downfall - and maybe we're seeing that already.

But there is initial excitement about this beta. When its availability was first announced the number of people downloading it caused Microsoft's servers to fall over, and they've since lifted the 2.5 million cap on downloads. That all bodes well, as long as people's initial impressions are good. The press that I read about has been almost uniformly positive.

Why the heck am I, a Windows detractor and well-known complainer, downloading a beta of Windows 7?

Good question. I guess I'm curious, and now that I know how to run VirtualBox on my Mac, I can pretty painlessly try it out. We'll see how it works out.

Sadly most hardware development-related software only runs under Windows. I initially installed VirtualBox (and an old copy of Windows 2000) to try out CodeWarrior - Freescale's (big and IMHO bloated) IDE for their microcontrollers. I was very impressed with Vbox, and CW was about what I expected. I tried to figure out how to use the tools inside it from the command line, with some success. I may write more about that if it ends up seeming useful.

I'm downloading the 64-bit version of Windows 7. (One joy of owning a newish Intel Mac is that I can run 64-bit versions of guest OSes in Vbox.) It's a 3.2GB DVD iso image, and at roughly 500KB/s it has taken almost 2hrs to download.

It just finished downloading!

Freescale forums save me again

2009-01-23T17:34:00.000-08:00

When I started the long bootstrapping process (that I'm still deeply mired in) I initially had trouble getting my breadboarded 908QB8 to talk to the serial port. (These parts have a bulit-in ROM monitor that bitbangs a serial protocol on a port pin.) After a day or so of going crazy, I found this thread on the 8-bit forum about the 908 parts requiring a really clean power-on reset (POR). Vdd needs to go to within a few tens of millivolts of ground before it comes back up.

I measured Vdd with the power disconnected from my breadboard, and, sure enough, it was stuck at 2v or so. The (cheap) voltmeter I used to measure this quickly discharged the on-board capacitors, and I got a good POR that way, and was able to get into the ROM monitor and talk to the chip.

By adding, between Vdd and Vss, a reversed-biased diode in series with a small resistance, I was able to get reliable resets. I would never had figured this out without the forum.

I've run into trouble in the second stage of my bootstrap as well: getting the 908QB8 to act as a background debug mode (BDM) host, talking to an S08QG8. I can measure BDM SYNC pulses with some measure of reliability and repeatability, but I can't seem to get anything out of the background debug controller (BDC) on the S08.

Last night, while perusing the 8-bit forum for more posts about "trouble" with the DEMO boards' built-in BDM interfaces, I stumbled across a link to my answer. It turns out that blank S08QG chips have to be powered up into active background mode, by holding the BKGD/MS (MS means "mode select") pin LOW during POR. I'm doing nothing of the kind. I plug a cable into a 4-pin header on the breadboard, and everything powers up at the same time. The 908 comes up fine every time, but apparently the S08QG (and other low-pin-count devices that lack a dedicated RESET pin) are a bit hard to get reliably into BDM when they are blank. They power up and start executing random code - unprogrammed Flash, peripheral registers - and continually either experience COP (computer operating properly - ie, watchdog) resets or illegal instruction resets. There isn't a long enough window between resets to get the device into active BDM mode - the protocol takes long enough that it's basically not possible.

So I need to fashion a way to power cycle the S08 from the 908. Initially I'm going to jumper it by hand, just to make sure this works.

Another side effect of powering up an S08QG in active BDM, is that its BDC operates from a 4Mhz clock (the bus clock) rather than the dedicated "alternate BDC clock", which is 8MHz out of reset. My 908 is running at 3.2MHz, and I figured out that bitbanging BKGD would be too slow, if the target is running at 8Mhz. So I dug up a 25Mhz crystal and fashioned an oscillator, so I could run my 908 at 6.25Mhz (bus clock).

Now I can go back to a simple setup, since my 3.2Mhz 908 is going to be plenty fast to talk to a 4Mhz S08.

Sorry about all the acronyms!

Is the Freescale ecosystem healthy?

2009-01-18T20:11:00.001-08:00

I'm a bit worried about it.

It seems that Freescale or P&E Micro are intentionally hobbling the USB-BDM interfaces on Freescale's DEMO boards, forcing people to buy development hardware that they don't need - like an external USB-Multilink.

One of Freescale's avowed core values is "to be the most ethical company in the business". I think they have some self-reflection to do here, and in fact I think the user community needs to take them to task.

Folks in the ARM world crow about the "ARM ecosystem" and how wonderful it is to have multiple chip, development board, and software tool vendors. It was something that attracted me to the ARM architecture for a long time.

The 8-bit world is different. Except for the 8051, the popular 8-bit architectures - HC08, PIC, AVR - are single-sourced. Every architecture has a development tool story as well. Except for AVR, there are few to no free tools for the 8-bit chips.

In the Freescale world, people pretty much use CodeWarrior (CW) exclusively, and it's easy to buy relatively inexpensive ($50 to 100) "demo" boards from Freescale to try out a chip family. There is a free version of CW, but it is limited in the code size that it will compile. That's another story, and it deserves its own post.

I found out about this "hardware hobbling" by doing some research on Freescale's 8-bit processor forum. I was considering buying a DEMOJM board. This is a "motherboard", with built-in USB-to-BDM converter, some i/o, and a square of headers to plug a doughtercard into. It comes with two daughtercards: one for the S08JM60, and one for the MCF51JM128 (a ColdFire part). It looks like a good way to get started with the JM family - these are the nifty USB parts - and because it has a built-in USB-BDM, I figured I could use it to program parts from other families - at least other 5v parts.

But I found some alarming posts on the forum. People were asking "can I use a DEMOxx board to program other parts?" and others were responding "sure, the DEMOxx has a full USB-BDM interface built in." Then the trouble started. Someone with a DEMOQE board - which, like the DEMOJM comes with two daughtercards - couldn't program, using the built-in USB-BDM, anything other than the chips on the daughtercards. These were S08QE chips too - not chips from some other family.

It turns out that the latest version of CodeWarrior - version 6.2 - has code that intentionally breaks these boards! It's possible, by replacing a few DLLs, to fix the problem.

I really think that the community needs to let Freescale know that this kind of thing is hardly ethical behaviour. I don't want my black-box dev tools, when upgraded, to silently break perfectly good hardware that I've legitimately purchased! At the very least this creates confusion and leaves a bad taste. "This used to work; why doesn't it work any more?" Or, "this board has a USB-BDM; I bet I can use that on these other chips!" There is no official explanation, when it doesn't work, about why, and I think this needs to be remedied.

Having had the experience of trying something that should work, but doesn't, people are going to respond in one of three ways - none of them good:

give up on Freescale and go elsewhere
grumble, buy a USB-Multilink, and continue with Freescale, but with a sour taste
learn the truth - that this is intentional breakage - and get angry

In chronological order, here are a few threads from the forum that document all this craziness.

MC9S08SH8 on a DEMO9S08QG8 board
Trouble programming JM32 with DEMOQE
Using demo boards as BDM interfaces
Embedded DEMOQE BDM is a full Multilink? - explains how to "fix" a 6.2 version of CW
PROBLEM TO PROGRAM A MC9S08QE128 CHIP USING DEMOQE BOARD

Is ColdFire next?

2009-01-18T13:39:00.000-08:00

I spent several hours yesterday carefully reading ColdFire documentation, trying to understand what it was Freescale had accomplished, or at least what they left out of the 68000 to make ColdFire. You have to do more than glance thru the new instruction set. I did that a few weeks back and didn't get the sense that much had changed.

It turns out they left out a lot. But not in a bad way. Now the architecture really is much more a RISC-like load-store machine.

Operand size changes:

except for CLR, TST, and MOV, and a few inherently byte-oriented instructions (such as Scc and TAS), all instructions only work on long operands

Addressing mode changes:

single operand instructions - NEG, NOT, ASL, ASR, etc - and immediate instructions - ANDI, ORI, etc - only work on D registers

Instructions left out entirely:

BCD arithmetic (no surprise)
DBcc - decrement and branch
CHK
CMPM
EXG
MOVEP
RESET
ROL, ROR, ROXL, and ROXR (hmm - because they don't exist in C?)
RTD, RTR
TRAPV - again, we're programming in C, not Pascal ;-)

Instructions added:

BITREV
BYTEREV
FF1 (find first one bit)
MOV3Q (move 3 bit immediate to EA)
MVS (move with sign-extend)
MVZ (move with zero-extend)
TPF - trap false - a nop that can consume one or two following instructions!

Because the JM family (the USB parts) has ColdFire parts as well as S08, and they are not much more expensive - I may be experimenting with Forth on the ColdFire too. Especially if I get a JMBADGE.

It's all about the timing

2009-01-16T13:46:00.000-08:00

I think my current interest in Freescale's HC08 has everything to do with timing. I've looked at the architecture before, and haven't been excited about it until now.

In fact, I've looked at it several times, initially in 2001 or 2002. I was overwhelmed then with how many families there were, and the lack of easily available development hardware. Also, the parts at that time were quite slow - I believe this was before the introduction of the S08. I ended up instead buying a Cygnal (now SiLabs) board, which I played around with a tiny bit. I was impressed with the idea of an 8-bit CPU running at 25MHz, but unimpressed with the idea of writing code for the 8051. And their parts are $15 in quantity one. The S08JM32 - a pretty comparable part, running at 24MHz, with 12-bit ADC, and the added joy of a USB interface - is $3 in quantity one.

I just discovered Freescale's agitprop zine Beyond Bits, which has been published for the past three summers. Looking at back issues I realised that only now is their lineup compelling to me. The JM USB parts are new, the Flexis (AC, JM, QE) are new. The S08 is fairly new. All of this is recent.

When I looked at the HC08 in 2006, and sampled the 908QB8 and S08QG8 parts, there were very few parts that seemed interesting. For my purposes (wanting DIP16, wanting small package etc) these were really the only matches. They seemed like nice parts, but I wasn't impressed that this was an architecture I could really grow with. There was a way better selection then of PIC and AVR parts, esp in DIPs.

Now things are dramatically different. I'm looking forward to digging deeply into this architecture!

Putting Forth on the Freescale S08

2009-01-14T21:54:00.000-08:00

I've been looking, literally for years, for an 8-bit microcontroller to get excited about, dig into deeply, and turn into a convivial tool. I still haven't found what I'm looking for (apologies to U2 and Negativland!) and went as far as writing an essay entitled "Eight bit microcontrollers are obsolete!", based mostly on my belief that the ARM7 was going to be the next 8051.

I still really like the little 8-bit machines. They are cheap - still cheaper than their 32-bit "cousins" - and are much more approachable by novices to the art - something that is actually important to me, as I'm not thinking about using them as black boxes buried deep inside something else - the classic embedded control model - but instead as paint and canvas. I want them to be convivial tools - both for me and for others as well.

Recently disillusioned with the ARM world - both ARM7 and Cortex-M3 - and irritated again with the PIC and AVR options - the PICs are too expensive, and AVRs have crummy Flash, and are also expensive - I recently took another deep look at Freescale's HCS08 families, and for the most part liked what I saw.

What can these materials do?

Before getting into the meat of things, I'd like to acknowledge something that Steve Davee said in his recent DorkbotPDX talk that continues to inspire me. He was talking about teaching art to kids. Instead of putting a bunch of art materials in front of them and saying "What can you do with these materials?" - which would cause me, even as an adult, to freeze up creatively - he says "What can these materials do? Let's find out!"

That's what I feel I'm doing right now. I'm exploring the HCS08 with the mantra in my head: What can these materials do?

I'm finding it very helpful, and also sustaining.

Shoehorning Forth

What are the issues we face when considering putting Forth on the S08?

When figuring out how to put Forth on any architecture, there are two main issues to consider, and they are intertwingled:

native or threaded code?
how are we using the registers?

With the HC08, there are so few index (pointer) registers, that any kind of threading, while possible, seems inordinately expensive. The HX register will have to double duty (at least) as the IP and a stack pointer; all the saving & restoring will take time; and it will prove difficult - as in the case of NEST, which needs to push the IP onto the return stack - to have a "place to stand" as we're copying values around.

Don't get me wrong - it's doable, and may even be worth doing. But the high overhead doesn't appeal to me. I'd like Forth on this machine to rival C - and it might even beat it.

Native code compilation means that the body of a high-level word (a so-called "colon" word, because they are defined by the : operator) is going to be simply code - mostly a series of calls to other pieces of code.

It's sometimes useful even when not doing any kind of threading to represent literals in the normal Forth way:

  jsr push_literal
  <literal value>

Again, on this architecture, the overhead of popping an address (to get the address of the literal), fetching both bytes, and pushing them onto the data stack, is quite unappealing. On a machine with more registers, we would simply do one or more load immediates, and then call or jump into the word that expects a literal. On the HC08 there is only one 8-bit register free for this purpose: the accumulator (A). This works fine for 8-bit literals. But what about 16-bit?

I found a nice solution, but before I explain it, let's talk about the register allocation. Because we are compiling native-code, the hardware stack is the return stack, and we simply use jsr/rts to do our "threading". There is only one option for the data stack pointer, and that's HX, which will occasionally have to do double duty as a generic pointer register, and in those cases we'll have to save and restore its value around these other uses so we don't clobber the data stack pointer. That's overhead we'll have to live with.

We can reduce it somewhat by adopting the convention that the data stack lives in the zero page. In this case, H is always 00, and X is really the stack pointer. (Just like the HC05 from which the HC08 is descended!) When we have to use HX to point to arbitrary locations, we only have to save X; when we're done with HX, we restore X, and clear H (which only takes one cycle, instead of the three or more needed to restore its value from the R stack or memory). I'll try it both ways and maybe try to measure the performance difference.

So, given that HX points to the data stack, we could adopt the (obvious) convention that 0,x and 1,x are the top of stack; 2,x and 3,x are the second; etc. But it turns out that wonderful things happen if we instead pre-allocate a 16-bit cell as scratch space at the top of the stack. The real top moves to 2,x and 3,x; and second moves to 4,x and 5,x.

Why is this so useful?

Because we are so register starved, it's often nice to have an "extra hand" - a place to put a byte of literal data, a count, the saved value of A, etc. Having this scratch space available makes programming in assembler much easier, and it allows for the arithmetic and logical operators to share code between their literal and non-literal versions. I'll explain.

Let's first look at the code for + (add), which consumes the top two values on the stack, adds them, and pushes the result. Another way to think about it is that it pops the top value and adds it to the second, making the sum the new top. Here is add:

  ; stack (,x) offsets - this is a big-endian machine!
  ; 0  scratch_hi
  ; 1  scratch_lo
  ; 2  top_hi
  ; 3  top_lo
  ; 4  second_hi
  ; 5  second_lo

  add:      lda 3,x    ; top_lo into A
            aix #2     ; add 2 to HX; top becomes scratch; second becomes top
  add_imm:  add 3,x    ; A = top_lo + prev_top_lo
            sta 3,x
            lda 0,x    ; scratch_hi = old top_hi
            adc 2,x    ; add top_hi
            sta 2,x    ; save sum_hi
            rts

I can write subtract, and, or, xor the same way. What's neat about this is that instead of calling add to add two values on the stack, I can add a literal to the top value by doing this:

            lda #lit_hi
            sta 0,x     ; save in scratch_hi
            lda #lit_lo
            jsr add_imm

There is a subtle correspondence here. The literal version loads the high half of the literal into scratch_hi (0,x), and the low half into A, then calls add_imm. The code at add does a very similar thing: it loads top_lo into A, and leaves top_hi where it is; but by popping the stack, top_hi "moves" into scratch_hi, where add_imm expects it - but it hasn't actually moved! It just "happens" to end up in the right place.

I think this is a very neat solution to the problem of having too few registers on the HC08. I wonder if any other compilers leave this kind of "scratch space" on the stack by convention? I'm stunned at how useful it is.

However, it doesn't end there! We can do more. By recognising that each byte of a bitwise logical operation is independent, if we want to compile a logical op with a literal value that only affects one byte, we can inline:

  lda #$FE
  and 2,x     ; and with top_hi
  sta 2,x

On a threaded Forth with two-byte (16 bit) literals, this would take six bytes of code - which is exactly what our inline version takes! But it runs much faster.

We can do this optimisation if we are adding or subtracting a literal with zeroes in the low byte. Carry propagates from less significant bits upward, not the other way! So there is no way that an add or subtract in the high byte can affect the low byte.

We can make another small optimisation as well. Loading a full 16-bit literal (into A and scratch_hi) takes 5 bytes. What about shorter literals? Are there other (shorter) code sequences we can use? It turns out that there are. For literal values between $FF00 (-256) and $01FF (511) we can load the value in 3 or 4 bytes. Here is the code:

  clr 0,x     ; clear scratch_hi
  lda #lo     ; low half of literal
              ; for values between 0 and $FF; 3 bytes

  clr 0,x     ; clear scratch_hi
  com 0,x     ; scratch_hi = FF; we could use dec 0,x as well
  lda #lo     ; low half of literal
              ; for values between $FF00 and $FFFF; 4 bytes

  clr 0,x     ; clear scratch_hi
  inc 0,x     ; scratch_hi = 01
  lda #lo     ; low half of literal
              ; for values between $0100 and $01FF; 4 bytes

These code sequences are short because the "indexed with zero offset" instructions are only a byte long!

One last comment about literals. Mostly we want to load a literal value and then call some code that consumes it, and we avoid pushing it onto the stack first. On the rare occasions that we want to leave a literal value on the stack - as a return value or flag, eg - we simply compile the load literal, as above - either 16-bit or 8-bit versions - followed by code to "push" the loaded literal, which is these two instructions:

  sta 1,x     ; save literal_lo in scratch_lo
  aix #-2     ; "promote" scratch to top by allocating a stack cell

I think this is pretty compelling demonstration that with a bit of thought and cleverness we can fit Forth pretty nicely on this machine. I'm hoping it won't take up too much space, and that it'll run like a cat with its tail on fire! (apologies to PETA and the SPCA).

Deep in a multi-stage bootstrap

2009-01-11T17:37:00.000-08:00

I'm trying something slightly crazy, and progress is going slowly.

I want to get Forth and a simple USB bootloader running on Freescale's S08JM parts. But instead of doing that the easy way, and buying P+E Micro's USB Mulitlink programmer, I decided it would be interesting to see if I can bootstrap from nothing.

Well, not quite nothing. But to explain that I have to explain some history. The HC08 is really two sub-families: the original HC08 and the newer HCS08. I refer to these here as "908" and "S08". (The "9" means Flash memory, rather than ROM or OTP EPROM. The 908 and S08 parts both use the same Flash.)

The S08s are nicer in almost every respect than the 908s, but they are harder to bootstrap. In fact, without the use of another microcontroller they are basically impossible to bootstrap. Rather than having a bootloader in ROM, like the 908s, they have a piece of hardware on-chip - the background debug controller (BDC). This bit of logic is really a poor man's JTAG. It allows access to the CPU registers, can start and stop the core, read and write memory, and because it can do all these things, it can run code that writes to Flash.

These are really nice features, but the downside is that the custom protocol required to talk to the BDC needs to be driven, with fairly tight timing, by another microcontroller.

Could I use a 908 to bootstrap an S08?

In 2006 I sampled two flavors of 8k Flash, DIP16 parts: the 908QB8, and the S08QG8. Since the 908 has a built-in bootloader, I thought I would start by talking to it, then put just enough code on the 908 to talk the BDC protocol to program the S08... Then put a nice, simple, UART-based bootloader on that chip that I could use to program the S08JM parts - my eventual goal. A long road, but mostly an interesting one.

I met with some initial success. Using two resistors and an HCT244 I was able to connect an RS232 level converter to the 908's PTA0, which the bootloader bitbangs at 9600 bps.

I wrote a bit of Forth code that allowed me to read and write memory, the registers on the stack, and was able within a couple of days of fiddling around to download code into RAM and run it successfully.

Flushed with success, I breadboarded an S08QG8 next to the 908QB8. Since most of the S08 families run at 3v (1.8 to 3.6), I had a voltage-conversion problem. Since I already had an HCT244 on the breadboard, I thought I would use that in the S08 to 908 direction, since the HCT input levels will be just about right for 3v logic. Driving the S08's BKGD pin to 3v is a bit trickier, and I'm not sure I've come up with a good way to do it.

Freescale describe the BDC protocol as "quasi-open-drain". Since it's a one-wire protocol with multiple senders, it needs to have a pullup to Vdd (which is internal to the S08), and the senders need to be able to drive it high, drive it low, and 3-state it.

My approach involved using three port pins: one directly connected to BKGD to pull it low; and a voltage divider between two others - one driven high and one low - that would put approx 3v on BKGD. This doesn't seem to work, and I'm not sure why.

Of course, there is another issue that I haven't mentioned that consumed quite a bit of time to debug. Since the BDC protocol is based on the target's clock speed - each bit transmitted taking 16 BDC cycles - I need to drive BKGD fast enough to keep up with the target. The "host" - in this case the 908QB8 - has an internal oscillator that at the fastest setting yields a bus clock (instruction clock) of 3.2MHz. The 9S08QG8 runs its BDC clock at 8MHz. The tightest code I could write for sending and receiving bits was too slow, so I spent a bunch of time debugging oscillator settings.

After finding a 25Mhz crystal, I'm now running the 908 with a 6.25MHz bus clock, which is fast enough to keep up - but it still doesn't work.

Which is why I thought I'd take a break and document my progress, instead of driving myself crazy.

I missed an historic opportunity

2008-12-24T19:45:00.000-08:00

I'm working on porting Forth to Freescale's 908 and 9S08 parts. I got my first samples in 2006, got discouraged, and stopped working on the project. For mostly unknown reasons I've been recently inspired, and have picked the work back up.

Now that it's the end of 2008, I wish I had worked on it in 2007 and early 2008 so that on 2008-08-08 I could have released my code to the world!

Oh well. Better late than never?

Why am I bothering with Freescale's HCS08?

2008-12-19T13:54:00.000-08:00

If I had any sense, I would be using Atmel AVR parts. Why do I say this?

there are a bunch of folks in dorkbotpdx using Atmel AVR, including building nifty boards, so there is local expertise and groovy local hardware available;
the AVR is fast, and it's a good match with Forth - at least from the perspective of register allocation
there are decent USB parts available, as well as low pin count DIP packages for easy experimentation.

Why, then, am I concentrating on Freescale's HCS08 instead?

I've mentioned elsewhere that I like the 6800 heritage. It's nostalgic for me. But that's a ridiculous reason, right? So what are some compelling reasons to use these parts?

Attractions of the S08

Cost

They are quite cheap - cheaper than comparable PIC or AVR parts - and you get a lot of bang for your buck. For instance, the S08QG parts in DIPs have a UART, SPI, and I2C, all standard, a good selection of timers, 8 channel 10 bit ADC, and some nice add-on features, such as clever compare logic in the ADC that causes an interrupt only when a value crosses a threshold. The USB parts have 12 bit ADC, 2 UARTS, and 2 SPI ports.

Also, it's quite easy to get free samples, directly from Freescale, and also to buy directly from them, for very competitive prices, even down to quantity one.

Flash

Unlike Atmel's Flash, which is only good to 10k cycles (if you're lucky) and has a short retention lifetime (10 years?), Freescale's Flash, like much of Microchip's, is good to 100k cycles and has a retention of more like 100 years. This might not sound like a big deal, esp for hobbyists, but philosophically I find it unappealing to use something so obsolescent as Atmel's chips.

Von Neumann architecture

A feature that is here in the "pro" column but also has a dark underside. Since RAM and Flash exist in the same memory map, it's possible to execute code out of RAM - and in fact it is necessary to do so when executing code to program the Flash. Code in RAM make several things possible:

self-modifying code
"specialised code", generated on the fly on the target
nice interactive development - ie, the ability to define and execute new bits of code while "online" with the target

Built-in debug architecture

This is an attractive feature, esp if you're going to use Freescale's development software, which can take full advantage of it.

Basically, every S08 part - even the lowliest 8 pin part - has not only a background debug mode (BDM) single-pin interface (which can be used to Flash empty parts, to reprogram parts, and to debug running code!) but also a tiny built-in in-circuit emulater (ICE). There are two code breakpoints and one data watchpoint available in hardware, with sophisticated chaining available between two of them. There is also an 8 entry trace buffer, allowing the capture of address and data bus values.

Considering how "low-end" some of these parts are, this is quite a stunning feature.

Quality peripherals

It's more "gut feeling" than real empirics, but the quality and flexibility of the on-chip peripherals seems very high, compared to AVR and PIC. Maybe when I re-develop all my code for AVR I'll have harder data on this one. ;-)

Upward migration path

Freescale have introduced several new S08 families with the so-called Flexis architecture. The new USB parts - the S08JM family - are Flexis parts.

Flexis families have both S08 members, and 32-bit ColdFire members, which are pin-compatible and only a bit more expensive than their 8-bit cousins, making it easy (assuming your code is not in assembler!) to upgrade to a fast 32-bit core. The peripherals are exactly the same 8-bit peripherals in the S08 parts, so all that code - again, assuming it's in a high-level language - will port over trivially.

Or so the story goes. It's a nice idea, and if you're not interested in ever using the 8-bit version, it's a way to get quite cheap 32-bit parts.

And since ColdFire is essentially 68k, it's a good target for Forth and C (and everything else) and also nice for nostalgia buffs.

Distractions of the S08

Ok, so how about downsides to the HCS08? There are several, and a few are big ones.

lack of community
lack of open source support
lack of cheap, universal, and easy-to-use development hardware
lack of off-the-shelf "target" hardware (eg, like Arduinos and clones)
lack of registers! making Forth a bit tricky
von Neumann architecture slows execution considerably
most S08 parts are 3v only (1.8v to 3.6v), though there are a few 5v families (intended mostly for automotive customers)

I think the hardest for me right now is the lack of development hardware. I have some chips, but I have to do some work to be able to try them out. I'm currently busy writing code, but soon I'll want to try it on some real hardware!

Freescale USB parts are on their way

2008-12-17T13:33:00.000-08:00

My samples of the S08JM16 and S08JM32 parts are on their way. Sadly I don't have a way, when they arrive, of doing anything with them. I lack the breakout boards to make them useful - they are in LQFP32 and LQFP44 packages - and I lack a way to talk to and program them. S08 parts use a Background Debug Mode (BDM) - a single-wire, half-duplex interface that is used both to program the Flash and also to debug live code.

One annoyance with Freescale is that there isn't a single super-cheap board to talk to BDM via USB. Each subfamily (such as JM) has its own DEMO board (costing $30 to 90), which has a USB to BDM connection, but usually somehow castrated to work only with that subfamily. The one "universal" BDM board is about $99. I may have to get one. It's the AVR Dragon of the S08 world - but more expensive!

I just realised that a fun way to play with this stuff might be the JMBADGE. These are quite cheap ($30 I think), and have loads of possibility. The JM family is one of the "Flexis" families, which consist of S08 (8 bit) parts on the low end (8k to 60k Flash), and ColdFire V1 parts on the high end (50MHz CPU, 64k to 256k Flash). I could misuse this to program other parts via BDM. And it looks like a fun gizmo.

Using git svn

2008-12-13T17:38:00.000-08:00

By using git-svn I was able to convert my Subversion repo – which contains several projects, some with multiple branches – into Git repos, one per project. This page details the history of that effort, and the Git config setup necessary for success.

Using my Lua code – to parse the dump file created by running “svnadmin dump” on my repo and feed a stream of commands to git-fast-import – showed promise and ran extremely fast (30 commits/s), but because of the twisted structure of my Subversion repository, and because of the way that paths are represented in the dump, it finally became obvious that that approach wasn’t going to work.

It was quite depressing to realise this. I liked my Lua code, and I had almost enjoyed trying to understand the Subversion dump file format. git-fast-import is cool too. Oh well.

git-svn actually rocks. It is intended to be used as a two-way conduit between Subversion and Git. The idea is that a Git user could work on a Subversion-hosted project by using git-svn to clone the repo into Git, and by carefully creating only linear history on the Git side (by rebasing and not doing Git-style merges), make local commits to Git that can subsequently be pushed to the Subversion repo. This kind of usage has lots of advantages, not least of which is that the entire history of the Subversion repo is available in the local Git repo, and commits to the local repo can be made offline and later pushed. Subversion users have neither of these – they are tethered to the central repo for both browsing history and committing.

However, these lovely features of git-svn were irrelevant to me, as I was trying to get my code out of Subversion so I can leave Subversion behind, forever.

git-svn works well, but setting it up is a bit strange. The documentation suggests that it can understand and parse a Subversion repo (or project in a subdirectory) with the “standard” layout: trunk/, branches/, tags/. I was unable to get this to work, at least with the muFORTH code, which has been moved around a lot – checked into CVS, moved around; converted to Subversion, and then moved around some more, partly to fix how cvs2svn interpreted my repo, and partly to fix mistakes I had made along the way.

Using git-svn to “clone” single branches worked perfectly, though. So I simply set up by hand all the branches that I wanted, and then one by one did

  git svn fetch <branch>

The whole process went something like this. First I made a new Git repo for the project.

  mkdir muforth
  cd muforth
  git init

If you want decent commit messages with real email addresses you need to create an “authors” file that maps Subversion user names to email addresses. It looks like this:

  david   = David Frech <david@example.com>
  john    = John Deere <john@example.com>
  (no author) = some lousy conversion tool <none>

The "(no author)" entry is necessary because cvs2svn and also some Subversion tools create log entries like that. Yes, it’s lame.

There are two ways to tell git-svn about our authors file. I created an entry in my global Git config (ie, the file .gitconfig in my home directory), like this:

  [svn]
        authorsfile = /home/david/.svn-authors-for-git

I did this so I wouldn’t forget to set the right command line switches; but it’s also possible on each invocation of git-svn to specify

  -A<filename>

  --authors-file=<filename>

on the command line.

After playing around with git-svn init to see what kind of config entries it creates, I manually edited muforth/.git/config and added lines like these, one per branch:

  [svn-remote "trunk"]
        url = file:///home/david/svn/muforth/trunk
        fetch = :refs/remotes/muforth-trunk
  [svn-remote "itc"]
        url = file:///home/david/svn/muforth/branches/itc
        fetch = :refs/remotes/muforth-itc
  [svn-remote "x86-native"]
        url = file:///home/david/svn/muforth/branches/x86_native
        fetch = :refs/remotes/muforth-x86-native

I used file:// URIs because the Subversion repo was local, but any scheme that Subversion understands will work.

The “fetch =" lines tell Git where to put the heads for the imported branches.

With the above config I then did:

  git svn fetch trunk
  git svn fetch itc
  git svn fetch x86-native

I probably then did something like

  git branch master muforth-trunk

to create a local master branch based on muforth-trunk.

Great minds think alike

2008-12-13T16:40:00.000-08:00

Once I finally understood what a Benito board was, I started thinking "well, that's pretty neat, but it lacks any ADC on-chip; otherwise it would make a nice replacement for the Arduino, at a fraction of the price".

Well, clearly Don (a fellow dorkbotpdxer) has similar ideas.

But it doesn't end there. I'm currently exploring the idea of using Freescale's S08JM family of USB microcontrollers - in part because they are retro & nostalgic (for me - I wrote my first program for the 6800), and in part because they are true von Neumann machines (one memory space for both code and data), rather than Harvard machines like their AVR and PIC competitors. A true Harvard architecture makes interactive Forths difficult to write, since RAM is always in data space, and Flash is in code space. For interaction I want to be able to execute code out of RAM - but I can't.

There are some serious downsides to using these Freescale parts. First, Freescale's toolchain - the bloated Codewarrior - while free is not open source, and is Windows-only. Second, there isn't a single piece of cheap hardware - like the AVR Dragon or PICkit2 - that can program all of Freescale's HC08 parts. Each family has its own DEMOxxx board, and each one costs $50. There is a nifty thing called the USBSPYDER08, but it only works with lowest-end parts - not with the JM family.

Wondering about this once again, I did the following Google search: "open source hc08 tools" and landed on a Freescale forum devoted to the OSBDM08 - an "open source programming cable". Sadly, it seems to be supported only under Codewarrior. But, rather interestingly, the latest thread is by "Brad R" who "writes Forth compilers" and is interested in Linux support for the OSBDM08!

This is, of course, none other than Brad Rodriguez, long-time Forth author and explainer, who created the CamelForth series of 8-bit Forths.

Pretty cool.

Lua programming language

2008-12-12T03:32:00.000-08:00

I’m officially excited about the Lua programming language. Who wouldn’t get excited about a language whose principal author and designer is from Brazil and looks like this? ;-)

Seriously, though, Lua is a small, elegant language, very carefully designed and implemented, with many, if not all, of the features that I want:

anonymous functions (lambdas) and closures;
lexical scoping;
proper tail-recursion;
a resident compiler;
embeddability, in both senses: it can be used as an extension language, and can run on “embedded” (and rather modest) hardware;
extensibility – it can be extended, both in itself (ideally via some kind of metaprogramming), or in C.

Lua has three “implementation” features that are important to me:

the primary documentation (book and reference manual) is written by the designer and primary implementor;
there is an authoritative, high-performance implementation by the language’s designers (unlike, say, Scheme, which has so many implementations that it’s impossible to choose a “good” one);
the implementation is small enough to understand and modify.

Philosophically Lua appeals to me. Everything about it is small. There are few key ideas, but they are good ones. The implementation is tiny. The language is tuned for size and speed, but because it’s designed to be linked to other code, it’s also designed for comprehensibility. And again, that means the smaller the better (since we can only use masterfully something we deeply understand). Lua is a convivial tool.

While most languages are profligate, Lua is parsimonious. The syntax is simple, the semantics straighforward; there few rules and fewer exceptions. The implementation is, by modern standards, tiny: a full host (x86) binary is ~140k bytes. It’s straightforward to slim this down to ~40k or less for use on small embedded devices.

Like all interpreted languages, Lua has the feature that data constructors and function definitions are executable code. This means that configuration files (for instance) are executable code. No longer is it necessary to write parsers for them; simply read them into Lua!

This fact, coupled with Lua’s clever – and only! – structured data type – the table – makes possible not only self-documenting XML-like structured data, but also data that, when executed, can produce an HTML (or XML – think RSS!) rendering of itself.

Resources:

An executive summary of Lua
Programming in Lua, first edition. Covers version 5.0. The second edition is out in paperback; it covers Lua 5.1.
Lua 5.1 reference manual
An interesting history of Lua’s development
A description of Lua 5’s innovative virtual machine
Two articles championing its use (same author, different years) (broken links?)
What users are saying
Some applications of Lua (lots of computer games!)
Lua user’s wiki

Convivial tools

2008-12-12T02:35:00.000-08:00

What are “convivial” tools? What a strange term! Ivan Illich coined this usage in his book Tools for conviviality, in which he says

After many doubts, and against the advice of friends whom I respect, I have chosen “convivial” as a technical term to designate a modern society of responsibly limited tools.

and then adds

I am aware that in English “convivial” now seeks the company of tipsy jollyness, which is distinct from that indicated by the OED and opposite to the austere meaning of modern “eutrapelia,” which I intend. By applying the term ?convivial? to tools rather than to people, I hope to forestall confusion.

Does that help? Probably not.

The idea is basically this: unless the people making things have complete control over the tools that they use, personal freedom and personal relatedness are impossible. A tool that has this malleable quality – and by extension, a society using those tools – he calls convivial.

I came across the book quite by accident. In 1988, disgusted with technology and programming, I quit my job at Microsoft, and went to Europe. When I returned to North America I stayed with a Canadian friend, at his parents’ house. His mom had a copy. (I wonder why – I should ask her.) I read it, enraptured.

It forever changed my life. Since then I have not been able to see technology except through its lens. This made me a dour critic of technology; it prevented me from getting excited about things that excite others; it even, perhaps, prevented me from having a “career” in technology. I was (and am!) deeply, and almost unconsciously, philosophically opposed to most aspects and artifacts of technology. I went apostate. I read lots of books, most of them written in 1973. Everything pointed to real food (ie, organic, grown from heirloom seed), to craft, to livable (walking) cities, to human scale. I rode my bicycle everywhere. I felt that technology was the enemy of everything I found myself caring about.

Some time in 1990 (I think it was) an odd conjunction occurred. I visited a family friend and talked about Forth – he wanted to make his own fuel-injection computer. Returning to Seattle, I read everything I could find about Forth. The Engineering library at the University of Washington has a lot: copies of the Caltech manual; the FORML (Forth modification lab) proceedings; copies of Forth Dimensions (the Forth Interest Group’s ex-magazine).

I had known about Forth since I was sixteen; but I hadn’t plumbed its depths until that week. I made up a packet of articles and hand-written notes and sent it off.

Shortly thereafter I ran into Trimpin, a local artist. I had seen a performance of a work of his – Circumference – at Bumbershoot (a big arts festival in Seattle) a year or two before. This performance involved a roomful of percussion – drums, timpani, garbage cans – all played by “robot hands”. Each instrument had a drumstick poised over it, attached to a solenoid, which was attached via a homemade interface to Trimpin’s computer. Using MIDI and MIDI sequencing software (Performer) he controlled this roomful of instruments. It was breathtaking.

His studio was in my neighborhood, and one day I met him on the street, getting an espresso – remember, this was Seattle!

We fell into a collaboration and I was able to realize two dreams: to use Forth, and to program the Motorola 6809, a processor I had been fascinated by since reading about it in Byte magazine in the late 1970s. I wrote a Forth development system, based on a terrible “object-oriented” Forth for the Macintosh. In a few days’ time I was able to write an assembler for the 6809, a Forth cross-compiler, and a simple “chat” protocol that allowed me to type commands on the host machine (the Mac) that were executed on the target (a small 6809-based embedded computer). All this was possible because of the simplicity and transparency of Forth.

I enjoyed this process tremendously. I felt empowered. I had never enjoyed programming in C this much. I went back to the library to figure out why Forth was so much fun, and discovered ... but that's another story that I'll write some day.

Over the next ten years we collaborated on a series of projects. These were mostly handmade, idiosyncratic – and often musical – kinetic sculptures. Trimpin would collect modern industrial “junk” parts, and piece together a contraption; I would program it, always in Forth. I enjoyed aspects of this collaboration immensely. I worked in his studio – an appealing environment, much more Gepetto’s workshop than high-tech office. I was combining technology and art. I was able to try out new (for me) ideas. I wrote a multitasker. I wrote code to generate and consume MIDI. I wrote an aleatoric music generator.

The largest program I wrote – which sequenced a forty-foot-tall sculpture consisting of four octaves of marimbas, xylophones, and wooden milk bottles, and composed simple musical sequences on the fly – consumed 6k of object code. (This is a permanent installation in the lobby of the Science Museum of Minnesota, in Saint Paul.)

Forth impressed me. It liberated and empowered me. It was a small tool – one that I could understand deeply and entirely. I could make it entirely my own and I have. I have written several Forths – two for the PC, and three for “target” machines: 6809, embedded 8086, and ARM.

Why is this interesting? Because I realize, looking back, that Forth is a convivial tool. I didn’t know that going in; I wasn’t sure, at the time, of the source of my sense of joyful liberation; but now I am sure that that is the reason.

Fast-forward to the present. After “looking in on it” several times over the last few years, I have finally taken the plunge and really dug into the Lua programming language, and I love it. I haven’t programmed much in it yet, but the bit I have done I have enjoyed immensely. I’m having the same feelings about Lua that I had about Forth. Is there a similar explanation? I think there is.

Lua is small – its implementation is about 16000 lines of C code – very modest for a “modern” program. This is about six times the size of my latest Forth (also written in C), which I wanted to be as small as possible. So Lua is only six times bigger than the smallest Forth I was able to write in C.

Lua, like Forth, has a few deep ideas that, once mastered, open infinite possibilities. Like Forth, it was designed to be extensible. Lua is well-documented, well-written, and has a simple C API, so it can be easily “glued” to other programs.

Lua is, by design and intention, a malleable tool, of such modest scope that it is possible for someone, investing a few weeks of study, to learn the system so thoroughly as to “call it their own”. At this point Lua too becomes a convivial tool.

I am excited about Lua in a way that I have not been about Python, OCaml, or Haskell – three languages that intellectually and philosophically interest me. I have been confused by this sudden passion (about Lua), and wondered at its source. This idea of conviviality is so familiar to me, so ingrained, so unconscious, that I don’t consider it any more. But it rules my (technological) passions, even when I don’t recognize it.

Somehow, last night I was able to step back and see the answer, in Lua’s conviviality.

Hosting git repositories

2008-12-12T02:26:00.000-08:00

It’s easy to host git repositories.

There are two hosted services that I know of:

Both services are free, but only conditionally. repo.or.cz is free for open-source projects; github.com – which allows the creation of private repos – is free if you use less than 100MB of space. Private repos are not available for the free accounts, however.

If you want to use your own machine, it’s quite easy to set up git-daemon for read-only access to repos. It listens on port 9418, and serves content using git’s native transport protocol (the git scheme in HTTP URIs). This allows anyone to clone and fetch from the repo. To push, ssh access is necessary – and this doesn’t involve git-daemon in any way.

Generally one would set up a bare repo in a directory that git-daemon is aware of, and push (from a “development” repo) to it when ready to publish. This way it’s easy to control when code becomes visible, and which branches are public.

Changes coming

2008-12-11T00:54:00.000-08:00

I plan to start working on muFORTH again, and created this blog to have a place to share my thoughts about it and my progress.

What I plan to work on in the very near future:

update muFORTH's license, changing from MIT X to two-clause BSD
update the boilerplate that's supposed to refer to a place on the web where info about muFORTH lives, and which now points into etherspace
add support for Freescale's HCS08 architecture so I can start to fiddle around with the MC9S08GQ8 chips I have, and also with some MC9S08JM16 USB chips that I plan to sample

With luck I won't derail myself before I make any progress. Wish me godspeed.