Public Lab Research note


Browser-based Arduino sensor data transfer using WebRTC and headphone jack

by rmeister | March 24, 2016 19:45 | 610 views | 14 comments | #12883 | 610 views | 14 comments | #12883 24 Mar 19:45

Read more: i.publiclab.org/n/12883


Name: Richard Meister

Affiliation (school/degree) B.sc. in Electrical Engineering and Information Technology, Technical University Darmstadt

Location (where you are): Darmstadt, Germany

Email: richard.meister.de@gmail.com

Phone: +491769144XXXX

Project title: Browser-based Arduino sensor data transfer using WebRTC and headphone jack

Project description

Abstract/summary:

This project allows web pages to access sensor data from Arduino via WebRTC and the common audio jack of smartphones.

Describe the need your project fulfills:

Many open data projects rely on volunteers spending time to gather data, e.g. from remotely placed sensor nodes that capture environmental parameters. Generally, the more data the better conclusions can be drawn. However, many projects suffer from having not enough volunteers and therefore collected data. If there is a lack of volunteers the barrier to contribute to a project needs to be lowered. This projects approach is to reduce the barrier by removing the necessity for contributors to install a specific software and to possess particular hardware. People often unwillingly install additional software. Besides this, it is a time consuming task that does not contribute to a project itself. But the even bigger obstacle is hardware. If it comes to buying additional hardware, most people will think twice before starting to contribute. A solution for these two problems would be appreciated by a large number of projects.

How will your project meet this need:

Making this project as independent as possible from the underlying hardware, a wired connection via headset socket was chosen. Not only that headset sockets are highly common in past and present smartphones - probably more common than Bluetooth or NFC as other candidates for this purpose - the communication via audio signals also has another huge benefit: Recording and processing audio from a web page with JavaScript and WebRTC makes a completely barrier-free solution. Instead of installing an application the user simply visits a web page.

As data transmission via audio cable is not that usual, the question may arise what data transfer rate will be possible. To answer this, I looked for some ratings of software serial implementation. Considering that there needs to be some space for other operations, for soft serial a data rate of 57,6kbit/s is feasible. But in this projects scenario we have a capacitive coupled link, which means the signal has to be modulated to a carrier, as DC signals do not pass capacitors. Therefore a carrier has to be generated, consuming CPU time. This makes clear why the existing SoftModem library merely supports up to 1200 bit/s. Sufficient for transmission of individual values but maybe a little slow for larger amount of buffered data and definitely too slow for real time streaming of captured data (Arduino Unos sampling rate is approx. 9600 samples per second, resulting in 96kbit data per second).

Getting a first working prototype should be straight forward, as there already is a counterpart to SoftModem: modem.js. Subsequent, the focus is on a clean implementation giving reliable communication and reproducible results. Afterwards, I would like to investigate two possible enhancements: On the one hand speeding up transfer rate, e.g. by using a different modulation scheme. On the other hand, probably more important, the Firmata protocol would make it possible to write a general purpose Arduino sketch, allowing the user to dynamically choose a sensor source. Should there be still time left, I would like to find out if a wireless solution with audio signals is feasible, because it will further reduce the barrier for users.

Timeline/milestones:

The time schedule presented here is a relative optimistic estimation, as until now the coding seems to be straight forward. Occurring problems are taken into account with the last weeks being more flexible and scheduled with extended goals. Also I have to mention that there will be one week between June the 23th and august the 3rd that I have to spend for an advanced training course in the Alps.

  • Community Bonding period (22 April - 23 May): As the proposed project is not directly related to other projects from PublicLab, I will use this period to take a closer look at the community in general. To be more accurate, my aim is to identify at which projects my contribution can be utilized and if there are more specific requirements to be taken into account.
  • Week 0 (16 May - 20 May): Understand SoftModem, (re)writing documentation, wiring components together, eventually testing CPU load and looking on waveforms more closely.
  • Week 1 (23 May - 29 May): Understand modem.js, design of a clean demo web page.
  • Week 2 (30 May - 5 June): Finishing basic demo web page.
  • Week 3 (6 June - 12 June): Beginning of Firmata integration, making necessary changes to the different libraries.
  • Week 4 (13 June - 19 June): Necessary code changes will be completed. Testing advanced Firmata features like I2C, servos, stepper motors, etc.
  • Week 5 (20 June - 26 June): Adding advanced Firmata capabilities to the demo web page. Mid term evaluation.
  • Week 6 (27 June - 3 July): Unavailable due to advanced training course.
  • Week 7 (4 July - 10 July): More tests and error rate evaluation.
  • Week 8 (11 July - 17 July): Enhancing stability, eventually implementing error detection/correction.
  • Week 9 (18 July - 24 July): Time for further improvements.
  • Week 10(25 July - 31 July): RPC (e.g. Firmata) compatibility and example Arduino sketches.
  • Week 11 (1 Aug - 15 Aug): Write a tutorial based on the example sketch and investigate extended goals/future work.
  • Week 12 (15 Aug - 23 Aug): Cleaning up code and documentation.

What broader goal is your project working towards?

The promotion of open science can be seen as the main goal. Getting more people involved in open projects is not an easy task. Lowering the barrier for non-technical people will lower the effort that needs to be taken to get enough volunteers. As side-effect, more people get in contact with science and they will see that it does not need an academical background to be a scientist. This surely is a benefit in multiple ways.

What resources will you need: people, documentation, literature, sample data, hardware if applicable:

I will need a shield that extends the Arduino with a headphone jack, linked below. The shield is not necessary in the first place, as I am able to verify waveforms with an oscilloscope.

Setup

I forked SoftModem and modem.js. In the next days I will program an Arduino with SoftModem and capture some output signals with an oscilloscope.

Experience

As a master student of electronics and data engineering, my technical background includes several programming languages. Regarding digital signal processing, I am familiar with Matlab/Octave. In last my last university project I used Octave to generate signals for brute-forcing garage door openers. I gained experience in Python during a project for acoustic device classification and also web related projects like my bachelor thesis. The language I am most experienced in is Java, because I spend two years as research assistant, programming back end and client applications for the Simpli-City project. Last semester I worked on a smaller project that included JavaScript and particularly Angular programming, so JavaScript is not completely new to me but I am sure my skills regarding it can be enhanced. What should not be missing is C, where I have basic knowledge, including tasks like implementing SPI or I2C drivers. I am confident to transfer my DSP knowledge from Matlab to JavaScript and C.

I have read the PublicLab guidelines and albeit my GitHub account is still empty, I know the concept of pull requests through the use of GitLab for university projects.

Teamwork

Besides university projects, I worked in the international team of the Simpli-City project. It began as a closed source project, but to my knowledge some of the code was published. My involvement in the project included writing an sensor abstraction layer as background service for an Android application. In the first place I had to familiarize with the Android SDK. Furthermore I developed multiple back end services running on a central server, making sensor data available to third party applications. The contribution started during my bachelor studies, but required techniques not covered yet at university, so I had to teach them myself. Regarding teamwork I found it very motivating to get support by more experienced developers.

Expertise

During my studies I had a strong focus and interest in communication schemes. This covers a wide range of technologies I used in several projects. Publish-subscribe mechanisms like MQTT are well known to me, since I implemented an energy-efficient push notification service in the context of my bachelor thesis. Besides network protocols that enable the so-called Internet of Things, I am very familiar with radio signals. Within the scope of a course about physical layer security I implemented a program to hack remotely controlled garage door openers. For my private projects and experiments with microcontrollers, serial communication is a basic and well known task. My expertise is logically connecting remote hardware and making it available in software, which is why I am suited for this project.

Interest

In the past there were no larger contributions to open science projects. Once in a while I installed an app for crowdsourcing purposes, e.g. sensing network speed and availability during large public events.

Audience

Generally this project targets everyone, but especially non-technical users will profit. For them it will become dramatically easier to contribute to open data projects. In consequence such projects will profit from this work. Additionally developers will be equipped with a new way of communication.

Context

I have the time to do this project, it is for a good purpose and I can explore a topic in the broad field of embedded electronics. I would like to further dive into this field, as I can envision doing a related master thesis.

Ongoing involvement

Two months after this project I am going to start my master thesis. Until then I will be available for code improvements and other tasks. I can imagine further contribution in the context of a related thesis topic.

Commitment

I am well aware this project will be a three month full time job. Accordingly I will choose only few university courses that will not be in conflict with this project.


14 Comments

Hi, this sounds well thought out and is an exciting project. A student named Bittu did some research I linked to from the ideas page that indicated that modem.js might not be the same encoding as SoftModem. SoftModem is FSK, but modem.js seems to be something called Bell or something... Sorry, you'll have to look at the original email they sent. Just a heads up.

I agree that while the low bitrate is something we could hope to improve on, it's plenty for polling a 0-255 value every second or so, which is probably a high data rate for many environmental sensors. So I agree that a slow initial implementation that works is a good first step, and speed later.

Finally, as there are different encodings to think about, it's a good idea to think about developing this in a very modular way which could swap out modem.js for something else, for example. In general, this project might benefit from being broken into several smaller self-contained modules, too.

Thanks for your proposal!

Reply to this comment...


Hi, Richard -- be sure you get your proposal in to Google's site in PDF format, i believe it's due today! Thanks!

Reply to this comment...


Hi Jeffrey, the Bell modem standard is basically FSK with the frequencies 1200 and 2200 Hz. SoftModem uses different frequencies for FSK, but that is just a matter of configuration.

I concur that modularity will be beneficial for the JavaScrip implementation. At the moment I can't see where higher modularity for the Arduino library can be achieved or is necessary.

Reply to this comment...


Super, that's exciting to hear about the FSK thing. I was worried we'd have to ditch modem.js or heavily mod it.

Agreed on modularity. Thanks!

Reply to this comment...


Hi, @rmeister - was doing a little more research into this in my spare time, and found a few things --

The first was that I used an audio recording program to record the output of an Arduino running the SoftSerial library on a couple pins connected to a headphone jack:

Screen_Shot_2016-04-06_at_3.14.06_PM.png

Then I used the free software program fl-digi -- a radio encoding/decoding program -- to look at the spectrum put out by the Arduino:

Screen_Shot_2016-04-06_at_3.14.45_PM.png

I guess this was obvious in retrospect -- you can see "bits" if you turn the baud rate down real low, but they're just on or off, not dual-tone like FSK would be.

On the other hand, if you use fl-digi to record from modem.js, you can see the two distinct frequencies used:

Screen_Shot_2016-04-06_at_3.48.30_PM.png

Just thought this might be a good debugging tool for you as you go.

I also have been using this javascript-based equalizer as a similar sort of "waterfall" interface: https://webaudiodemos.appspot.com/input/

Reply to this comment...


Oh, and the sketch I used on the Arduino was just this: https://codebender.cc/sketch:279140

for a Nanite, not a standard Arduino, but pretty simple -- i just had it send "hello again world" over and over, which is why you see the repeated pattern in the audio waveform in my first screengrab.

Reply to this comment...


A few more resources -- example projects using the softmodem library:

A simple FSK audio generator in JavaScript, specifically for the SoftModem library, in case modem.js doesn't work: https://github.com/NeoCat/FSK-Serial-Generator-in-JavaScript/blob/master/fsk-gen.html

A commented version of the SoftModem library: https://github.com/tatung/SoftModem/blob/master/SoftModem.cpp

Reply to this comment...


Hi @warren, thanks for all the hints and links.

I did some testing as well. Here is a capture from the output signal of softmodem.

softmodem_test.ino softmodem_test_capture.png

This explains the wide spectrum you observed with fl-digi. For simplicity, softmodem does not use sine waves. To some extend, I think it is possible to produce more sine-like signals, but this will also eat up a lot of CPU time. So applying filters at receiver side is what I would like to try first.

For windowing in the frequency domain I will try to use dsp.js. If this is not going to work well, I would start writing my own filter functions.

Reply to this comment...


Oh, cool! To clarify, I haven't yet used SoftModem, just softserial, which doesn't do FSK. This looks great, it's awesome to see the longer and shorter waves.

With filtering, will SoftModem be able to read a nonsquare wave as generated by JavaScript, and I guess, does modem.js actually produce sine or square waves?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Also, do you think filtering will be necessary? Depending on how it's implemented, modem.js might not care.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Ah, I overlooked that it was softserial in your case.

Yes I think modem.js produces sine waves. At least there is a sine function in the code and when I look at your record of modem.js it makes totally sense, as there are two relatively sharp lines.

I'll try to get my hands on a headphone connector in the next days, so I can answer the open questions.

Reply to this comment...


Last week I tried to replicate the circuit of the SoftModem shield. I had not the exact resistor values at hand, so maybe thats why it could not recognize the data correctly.

Luckily the shield from china arrived yesterday. The link browser → Arduino works fine with the FSK generator script, but modem.js does not, so far. Apparently changing the baud and frequency values as suggested in a forum post is not sufficient. I have to look at the code so I fully understand what it is expecting.

I could not get the other direction (Arduino to browser) working yet. Configured with the parameters of modem.js and the demo page opened on the phone, there wasn't any data arriving. I would like to test it with my notebook, but I think the microphone input is broken (even the internal mic is not working and I spend a couple of hours on the PulseAudio configuration...). Next step is getting an adapter cable so I can use an external sound card.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Wow, great start though! One thought for debugging is to record as an audio file the output from a known source (softmodem or the fsk generator) and play it into your phone -- there are things to possibly watch out for on cable type, android vs. iOS (reading into iOS probably won't work at all, as apple hasn't implemented or allowed WebRTC).

Great!

Reply to this comment...


@warren, regarding the cable type: there are two different standards. The main difference is that ground and microphone signal pins are switched. Besides this, many manufacturers implement own protocols or standards for volume/mute buttons and so on. One can not clearly say Apple is using AHJ and Android is using OMTP. For Android it also depends on the manufacturer and production date.

The SoftModem Arduino shield has a switch with a lable saying "Samsung" and "Apple". But what it really does is switching between the AHJ and the OMTP pin assignment. Interestingly the library was able to decode the signal no matter what position the switch was set. I think this behaviour is very beneficial for our purpose :)

Reply to this comment...


Login to comment.

Public Lab is open for anyone and will always be free. By signing up you'll join a diverse group of community researchers and tap into a lot of grassroots expertise.

Sign up