We've had sporadic Altus Metrum customer reports about RF susceptibility issues with their TeleMetrum installations. In almost every case, these problems have been completely resolved by either making sure the system battery has sufficient charge before launch, or through the application of standard engineering techniques such as twisting wire pairs to reduce differential coupling. However, even when every technique we could think of had been applied, once in a while someone still had issues.

Around the time of LDRS this year, the incidence of such reports seemed to increase. One customer, in particular, had an installation in which he virtually always saw continuous resets of the board once his 54mm airframe was put on a launch rail, and several customers reported seeing board resets during ejection charge firing. Keith and I saw a board reset during main charge firing happen in person at NCR's Oktoberfest, and with a couple days available to work together after that launch, we decided it was time to figure out what was really going on.

Here's what we've learned.

In bench testing, it quickly became clear that the problem was the 3.3 volt power supply rail getting pulled down far enough to reset the CPU. This most frequently happened during ejection charge firing, when the input of the LDO regulator is pulled down by the near-short presented by the e-match when a pyro FET is turned on. To keep the 3.3 volt rail voltage up during firing, we include a 100uF bulk capacitor on the regulator's output. In all of our prior bench testing, we never saw the 3.3 volt rail droop significantly. Clearly something had changed... or maybe several things had changed?

The first thing I wondered about was whether the new Kalman filtering code, which requires more compute cycles from the processor, might be consuming enough more power to pull the rail down faster during charge firing. After poking around at it, though, we have no data to suggest the new code makes a measurable difference in power consumption.

The next thing we pondered was that at least some of the e-matches we and others are using in the hobby now come from the fireworks industry, where it is apparently considered a feature for the match to retain continuity after firing. This means the input of the LDO gets held down for longer than with the e-matches we used to use and Quest Q2G2 igniters that open when fired. But that still didn't make sense as the root cause, as we chose the FET firing time such that even with a dead short across the igniter terminals, the 3.3 volt rail wouldn't be pulled down far enough to cause trouble during firing.

One of the big changes between v1.0 and v1.1 on TeleMetrum was that the newer boards incorporate a better reset circuit. This helps ensure the GPS chip always comes up running at power on, which was a problem at temperature extremes with older boards. However, a side-effect of this change is that a v1.1 board will reset any time the 3.3 volt rail drops below 3.15 volts, whereas older boards didn't trip until a much lower voltage. So the recent increase in reports might just be related to more v1.1 boards being placed in service?

While experimenting on the bench, we observed that injecting RF energy into the input of the LDO regulator had the effect of pulling down the output voltage, presumably because the internal reference source accumulates charge and is fooled into thinking the output is too high. Since our designs all have the power switch contacts ahead of the LDO, the wires going out to the switch and back are effectively an antenna... as are, to a lesser extent, the wires going to the e-matches. There is some variability from part to part in just how badly the LDO reacts. But by attaching a tuned length of wire as an antenna to the LDO input and playing around, we were finally able to reproduce the problem reliably on a test board at my bench!

On further analysis, we realized that the output of the USB battery charger chip and the input of the LDO both expect a 1uF bypass cap to ground. At some point, those looked redundant and we eliminated one of the two. Unfortunately, we weren't internalizing the fact that the switch leads were between the two caps, and the one we left was on the output of the charger and not at the input of the LDO. Placing a suitable bypass cap right at the input of the LDO turns out to have a truly dramatic effect on RF immunity!

Once we realized that RF getting into the LDO input was the problem, Keith pointed out that we used to see "noise" in the accelerometer data on earlier boards that was caused by the 3.3 volt rail moving slightly during radio transmit, which we fixed with a hardware change on v1.1. We are now convinced that this was at least partly related to RF coupling to the LDO input, not just the change in power consumption on the LDO output. We didn't realize what was going on in earlier testing because we often didn't have ematches wired up, so RF coupling was minimal. But going back to flights logged with v1.0 boards that included deployment, and studying the magnitude of the "steps" in acceleration data observed when the transmitter was on, Keith was able to compute the amount the 3.3 volt rail must have sagged if the real acceleration wasn't changing... which in some cases was as much as 180mV! We think this proves that RFI could cause the LDO to drop its output voltage below the reset controller set point on v1.1 boards.

Based on these observations, I'm making two hardware changes for the next version of TeleMetrum (version 1.2), and Keith is also making a software change. We have tested all of these changes on real boards both on the bench and in test flights, and the net effect is a major improvement in the RF immunity of TeleMetrum.

The first hardware change is moving to a slightly lower trip voltage on the reset controller. Instead of 3.15, the new part trips at 3.00 volts nominal. This gives us more "headroom" to tolerate 3.3 volt rail droop during charge firing, and will allow the board to operate longer on a given battery charge. This change is not relevant for v1.0 and prior.

The second hardware change is adding an appropriate bypass capacitor right at the LDO input. This requires a PCB update, but it's possible for me to update existing production boards by adding an 0402 cap right across the appropriate pins on the regulator chip.

The software change prevents our altimeters from turning on the radio transmitter while an ejection charge is firing. Since the RF transmitter draws substantial additional power, this should help keep the 3.3 volt rail from drooping. This may not really matter, but it feels like the right thing to do. This change will be part of our next stable firmware release.

We think most TeleMetrum customers need not worry about these updates. But if you have seen odd resets on the rail or during ejection charge firing in flight with a TeleMetrum v1.1, feel free to contact me about updating your existing board to include these improvements.

Keith and I released version 1.0.2 of AltOS, the open source firmware and software system associated with our Altus Metrum hobby rocket avionics systems.

This release includes a fix for the defect in version 1.0.1 that prevented rebooting an altimeter from idle to pad mode over the radio link. This affects both TeleMetrum and TeleMini boards, and re-flashing the altimeter firmware is required to pick up this fix.

We also restored the pre-1.0 mode selection behavior for TeleMetrum at power on. This is also an altimeter firmware change, but does not affect TeleMini.

FreedomBox activities at Debconf11

I spent the last two weeks of July 2011 in Banja Luka. The occasion was the annual Debian developer's conference, Debconf11 and preceding work week known as Debcamp. This was my tenth successive year attending Debconf, and I had a very productive and pleasant time! The facilities were good, the local team was friendly, enthusiastic, and very helpful, and in addition to giving three talks and hosting a couple panel discussions, I managed to put a burst of energy into work on FreedomBox. Several other developers working on FreedomBox were also present, and a good number of Sheeva and Dream plugs were evident in the hacklabs sporting new FreedomBox stickers. Working together in the same place for several days, we made good progress on several projects, and also had some great discussions about what we want to do going forward.

image building tools

For some time, I've been working towards a light-weight tool set to build FreedomBox software images. Shortly before Debconf started, I chose the name 'freedom-maker' for this tool and shared a link to a readable copy of my git repository with other developers I expected to work with in Banja Luka. With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker went from almost useful to actually useful. It still deserves work to be more useful to others, but I have now pushed a copy of the git repository to git.debian.org so that we can take advantage of the tools supported there to enable others to more easily contribute to the code.

Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius' vmdebootstrap to build x86 images suitable for testing in a virtualized environment. At the same time, we plan to refactor the existing code slightly to enable lists of desired packages for the various image flavors we expect to produce independently of the configuration for each specific image building tool.

Jonas continued in parallel to work on his alternate packaging toolset boxer. It offers some potentially interesting features for the future, and we may eventually merge some or all of it into freedom-maker, but for now it remains a separate utility.

uAP user space tools

Several weeks ago, we received from Marvell the source code to two user space programs that are necessary for configuring and monitoring the binary firmware provided for the uAP wireless chip used in the DreamPlug. Early during my stay in Banja Luka, I packaged these for Debian as uaputl and uapevent, and I am pleased to note that they were quickly accepted into the archive and are now present in Debian mirrors.

u-boot

Another bit of code received very shortly before Debconf started was the source for the version of u-boot shipped by Globalscale in the DreamPlug units we're working with. During Debcamp, Clint Adams passed a copy of this source to Jason Cooper, who was already trying to add support for the DreamPlug to upstream u-boot, but had stalled due to a lack of information. Jason has now merged his own work with the sources we got from the manufacturer, and is making good progress towards merging DreamPlug support into upstream u-boot. Once that happens, we should be able to flash our Sheeva and Dream plug devices with a u-boot image built from the source in the Debian u-boot package, in the process enabling things that matter to us like the ability to boot from an ext2 partition, and hopefully the ability to execute command scripts from that partition instead of having to hard-code kernel filenames in flash. This will allow us to support the ongoing effort in Debian to move away from the need for kernel symlinks.

DreamPlug kernels

With respect to kernels, another work stream at Debconf primarily involving Héctor Orón and Nick Bane was to analyze the current state of the patches from Marvell and Globalscale used to support the DreamPlug against both upstream and current Debian kernel sources. To my surprise and our collective pleasure, the remaining patch set required against current upstream kernels is much smaller than we previously believed! There are still several patches critical to us that are not merged upstream, but the work remaining to be able to build images for our devices from mainline and Debian kernel source trees now seems like something we might be able to complete before Debian's next stable release.

One of our discoveries during the u-boot and kernel work during Debconf was that Globalscale did not obtain a new machine id for the DreamPlug, but instead re-used the one for the GuruPlug series, despite there being some differences in the hardware that require at least one additional driver. After much discussion, we plan to continue using the existing machine id instead of requesting another, particularly because the ARM kernel community has apparently stopped issuing new ids for the moment. We will add a new kernel config option for the DreamPlug, however, and are likely to build distinct Sheeva and Dream kernel packages that do not require initrd for use in FreedomBox images, even if doing so is not strictly necessary. This will allow us to optimize both the in-memory footprint and boot times for our devices.

software configuration

Another area of investigation in Banja Luka was technology for package configuration. Mirsal Ennaime performed various tests using debconf and Config::Model, with some results reflected in this commit relating to configuring the bitcoind daemon in the bitcoin package for Debian.

identity and trust management

While we did not actually do any FreedomBox specific work on the trust management layer we know is necessary, after several rounds of conversation, I am now more convinced than ever that the right path forward is to base our trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting software elements. Our thinking to date is captured on an Identity Management page in the wiki.

communication services

Another thing that became fairly clear to me during discussions at Debconf is that in the near term, planning to build communication services around XMPP is the approach most likely to give good results. Investigating the software choices available to build an interesting XMPP infrastructure is now a high priority for me. Jonas has done some work towards configuring and integrating ejabberd or Prosody, I've started studying yate as a possible call manager and VoIP server choice with XMPP/jingle support, and we await with great interest a release from the Buddycloud developers to evaluate as a possible basis for deploying social network services.

Some of these software choices will lead us to use Apache as our web services base technology because of the need for features that only it supports well among daemons that are Free Software.

Jonas completed packaging GNU Sip Witch for Debian, and it is now available in the mirror network. Tzafrir Cohen and Jonas did some initial testing on its use.

documentation

A number of new wiki pages were written (or at least started) in order to sum up ideas, design various aspects of FreedomBox, and reflect discussions that happened during DebConf11. A lot of work is needed to complete these pages though, as well as others to capture more of the current state of the project.

press coverage

Finally, while in Banja Luka I got some great press coverage for FreedomBox! On Sunday the 24th, I was interviewed by the main television network serving the Republika Srpska. This led to a couple of minutes of coverage near the top of the national news program that night, immediately following the lead story about the President and several ministers appearing at Debian Day that morning to help open the conference. This interview was later re-used in another TV program that summarized Debconf11. On the morning of Thursday the 28th, I was part of a small group that spent more than an hour meeting with the Minister of Science and Technology in his office, and the relationship between Debian and our work on FreedomBox was one of the items of discussion in that meeting and the associated press conference. I'm told this resulted in more press coverage, but if true I have not seen it yet.

summary

On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf!

All in all, we got lots of work done on FreedomBox in Banja Luka, enough that I think at least the next few steps along the road towards an eventual "1.0" release of a reference implementation are now much clearer than they were two weeks ago!

Earlier this year, I agreed to join the board of Eben Moglen's FreedomBox Foundation, and to chair the recently announced Technical Advisory Committee.

To date, most of the time I've been able to personally invest in the foundation has gone towards work "behind the scenes", all of which was necessary, but little of which is worthy of external report or attention. I now believe it's time to take a more active role in communicating what we're actually trying to do, and how I think we can get there.

To that end, I intend to be more visible in discussions on the freedombox-discuss mailing list. I'm also accumulating a list of interesting technical challenges that I'll articulate here over time, along with status reports I believe are worthy of broad dissemination.

Keith and I just released version 0.8 of AltOS, the open source firmware and software system associated with our TeleMetrum hobby rocket avionics system.

The AltosUI ground station user interface is significantly improved, with lots of new features including a much cleaner view of what's happening during different stages of the flight experience, a post-flight data plotting interface with pan and zoom, and a very cool live view of where the rocket is overlayed on data from Google Maps! All existing TeleMetrum owners are encouraged to download, install, and begin using this new version immediately.

About a week ago, Altus Metrum announced that our 2nd generation ground software was available, and that it runs identically on Linux, Mac, and Windows computers.

Thanks to feedback from early users of this new software, particularly those who flew rockets at various sites over the holiday weekend and reported on their experiences, we're found and fixed several bugs, and made a significant improvement to the Windows installation experience.

Version 0.7.1 is now available for download from our AltOS page. All existing TeleMetrum owners are encouraged to download, install, and begin using this new version immediately.

Exciting news! As of today, TeleMetrum starter kits are available through Apogee Components in addition to the Garbee and Garbee web store!

Since October of 2002, HP has sponsored a "corporate subscription" to LWN on behalf of Debian, and I recently renewed the subscription through April of 2011. As of this moment, 571 Debian Developers and Debian Maintainers are enabled to take advantage of this subscription.

From discussion on IRC this morning, I gather one recent change hasn't been adequately communicated: I now allow Debian Maintainers to access the subscription. The process remains unchanged:

If you are a Debian Developer or Debian Maintainer and want full LWN access at HP's expense, just go to lwn.net and create an account for yourself (no money is required to create a user account). Then, send email to

lwn-subscription@debian.org

containing your LWN account name. Sign this email with your key on the appropriate Debian keyring. Then, exercise patience. Eventually, I will process your request, and add you to the "Debian Project group" and send you an email acknowledgement.

Likewise, if you retire from being a DD or DM, please let me know at the same address so that I can take you off the Debian subscription.

I believe I've caught up on all pending requests, but sometimes things get mis-filed, so if you're still waiting and I haven't replied, please re-send your request.

Contratulations to Bob Finch and Nathan Dalrymple for becoming our first customers to fly a rocket using a TeleMetrum board last Saturday! The rocket was Nathan's stretched BSD Horizon flying on a CTI Pro29 6xl 305-H226-14A Skidmark motor as shown in this launch photo.

I met both Bob and Nathan at the monthly club launch run by the Albuquerque Rocket Society at the same site in Rio Rancho, NM, earlier this year. That was the day I flew my first test flight with on-board GPS. They got excited about what they saw, and became two of our first customers for production boards.

Other than some rough edges with our first-generation ground software, Bob reported that everything worked as expected, and the GPS location received in the downlink got them to within about 10 feet of where the rocket set down.

In communicating their success to us on IRC, Bob said "MANY thanks to you guys, not only for the product but also for all the help getting me to the point of being able to use it !!". The pleasure is all ours, though, because after Bob asked us lots of new-user questions online, he wrote up his notes for us as "documentation for mere mortals" that will soon form the basis of a "getting started" section in our user manual.

Pretty cool stuff!

First, I was pleasantly surprised to discover that TeleMetrum is now the subject of a post on the Make magazine site. Very cool!

As reported last week, I've been waiting for better GPS antennas to arrive, but heavy snow here in Colorado last Friday delayed delivery. They finally arrived, and as I hoped, they appear to completely solve our signal strength problem!

So... [drum roll, please!]... I am very happy to announce that the first production build of TeleMetrum boards and starter kits are now "in stock" at the Garbee and Garbee web store, along with other products designed by the Altus Metrum community!

Keith and I are still working on the new Java-based ground station software and user documentation for the system. Watch this space for updates. I hope we'll have both ready for download by the time customers actually receive hardware...