Firmata integration testing

[soundanalogous]

In addition to the firmata test application, we need integration tests and possibly also unit tests.

Integration tests would test the serial interface with Firmata and the StandardFirmata sketch. The tests should be run on a PC and exercise the Firmata API over serial or by using fixtures (data captured from the serial port that can be used to test the serial API entirely on the PC). Tests should be written in an existing, well supported test framework in a language that has good cross-platform serial support (I think python is a good candidate).

The reason for integration testing is that Firmata has become large enough to be difficult to test manually. It's difficult to determine if a change in one part of the code affects another part of the code. Having integration tests in place will also help in expanding Firmata to additional microcontroller architectures as the interface should be the same regardless of microcontroller (aside for any limitations imposed by microcontrollers such as ATTiny).

[soundanalogous]

It will be helpful to have at integration tests in place before any new features are added to Firmata.

[soundanalogous]

If anyone wants to take this on, please start a discussion here before beginning work.

[ntruchsess]

Any ideas how to automatically verify that such integration-test executes successfully? With my own code (OneWireFirmata) I've written end-to-end tests that verifiy the response from a physically existing one-wire device. But such code is worthless for someone else who is integrating a completely different thing and just want's to verify it doesn't brake the existing stuff without attaching the appropriate hardware to his arduino. (Not to say the test itself is worhless, but it comes with the price-tag of real hardware).

[soundanalogous]

I haven't put a ton of thought into this yet. I wasn't thinking of testing individual devices (a specific OneWire or specific I2C device for example) but rather the higher level serial API. However the more I think about how to create tests for Firmata, the more complex it becomes. Maybe "integration tests" aren't the right answer. Basically I want to figure something out that enables developers to confidently refactor and and update Firmata. There really isn't much in place right now other than the Firmata test application and any up-to-date client libraries (for which there is often a bias in favor the client library so it's not always the most effective test, yet better than nothing).

I would like something that ensures that changing one part of the code does not affect another part of the code and that can be tested relatively easily. Maybe it's identifying a simple hardware test platform that any Firmata contributor could easily build with components they would likely have laying around and run this against a test application. Or it could be something that runs entirely on a PC and uses data captured from serial communication to create fixtures to test against.

I'm open to ideas here.