How do I...? real time data display over the internet

Wireless & IoT
1

Trying to figure out how to display data on a webpage in ‘real time’…

Either I don’t know what the method is called, or I just plain don’t know how to do it…

Here’s the scoop…

I’ve got some accelerometers connected to a PIC and a NanoWiReach Wifi module. The accelerometers measure tilt and such, the PIC reads that, crunches the numbers, and send the data out over the Wifi module via FTP to my server.

Currently, I’m using a simple iFrame and the REFRESH meta-tag to update the webpage display. Works good enough, but takes a fair amount of time to update the whole page.

What I’m looking for is to update just a single ‘text box’ almost instantly with real time tilt data (actually, multiple ‘text boxes’ whenever I get it running good) over the 'net. So, if a guy had one of my systems across the state was tilting it around, I would be able to see that data almost instantly.

Make any sense?

I want to say that I need a method to ‘push’ the data over to the browser from the server, but the data on the server will in turn be coming via FTP from the module’s. But, I’m thinking that I’m not thinking correctly…

Any swell ideas that don’t involve a hundred lines of scripting for me to try?

2

Do a Google on AJAX. You can set up a web page that will only update the data the changes on the page (i.e., your control) instead of completely refreshing the page.

3

Like Ralph said, you should use AJAX for this.

I would do some googling on using jQuery to make “AJAX” requests using JSON data.

Instead of loading the whole page you can just load the values you need very quickly.

4

As others mentioned, AJAX is the technology used to update data on a page without refreshing the whole page. One of the best tutorials is at: http://www.w3schools.com/ajax/default.asp

However, there is no way to ‘PUSH’ data from the server to the client. You’ll need to implment a timer in your browser code (using javascript) that will query your server periodically and request data from the server. W3Schools.com also has some good tutorials on javascript and how to do this.

5

This is where I got the idea/method/question…

http://www.youtube.com/watch?v=LjM5eeBe … re=related

(skip ahead to about the 4:30 mark and watch how quickly the tach updates on the iPhone)

I assume this is a fast wifi connection because it’s a dedicated AdHoc connection which has nothing else to do in this world except send that particular chunk of data between here and there.

AJAX - got it, downloading a handful of pdf’s now, gonna have a good read over the next couple of days.

After my quick read, I’m assuming that all of the ‘hard work’ is done at my server. All my remote modules have to do is respond to the requests quickly enough.

One thing I don’t get yet is how a remote browser (like me over at a friends house), which is pointing to my server (which is at my house and is grabbing data from the modules which are located on a wifi somewhere else entirely), would get IT’s updates from. If my thinking is right, there will be 2 AJAX processes running…one on the server grabbing data from the modules and updating whatever files on the server as required, and another AJAX process running on the remote browser (me at a friends house) updating my browser’s screen…

In the end, I’d like to shoot for once-per-second updates, data packets of around 300 bytes each. Too much? Not nearly too much? I know 300 bytes per second isn’t squat in the overall scheme of things, but ya know…don’t want to saturate my own network…

6

You will only need one AJAX session originated by your remote client to your local server.

Some way to quickly accumulate data from the sensors and so your local server can dish the data out when required will be needed. If you know a little something about programming, you could roll a simple webserver that also has a socket listener (on another port) that your remote sensors use to talk with your local server. That would be the quickest and most transparent system. Your remote client (iPhone, etc.) requests a page from your local server that has an AJAX call that your remote client will use to update the client with the current readings.

As far as flooding your local server, I doubt you will do it with 300 byte packets with the update rate that the YouTube video shows. Streaming video takes way more bandwidth. Even if that might be a problem, you could employ a strategy that only sends changes in data or use a more efficient way of communicating deltas in the readings.

7

AJAX is simply a JavaScript method for sending a request to a web server that doesn’t require the page to be refreshed. You don’t need anything special on your server…it just appears as a normal GET (or POST) request.

If you want to get an idea of how fast an AJAX request can be made and your browser updated, just go to Google and type in a long search term. The auto-complete text that is displayed is via an AJAX request. Or display a Google map and drag it…the update of the map is also via an AJAX request.

If you want to learn AJAX in 10 minutes go to the W3Schools site http://www.w3schools.com/ajax/default.asp. They walk you through everything you need to know!

8

AJAX is simply a JavaScript method for sending a request to a web server that doesn’t require the page to be refreshed. You don’t need anything special on your server…it just appears as a normal GET (or POST) request.

If you want to get an idea of how fast an AJAX request can be made and your browser updated, just go to Google and type in a long search term. The auto-complete text that is displayed is via an AJAX request. Or display a Google map and drag it…the update of the map is also via an AJAX request.

If you want to learn AJAX in 10 minutes go to the W3Schools site http://www.w3schools.com/ajax/default.asp. They walk you through everything you need to know!

9

Well…after a year of ‘delays’ (I got me one of them ‘little human beings’ running around the house now!), false starts, ‘feature creep’, etc., I got back on the project a few weeks ago and I’m practically stumped…

To recap, here’s the system as it stands right now -

System 1 - Mobile device, vehicle mounted…tilt angles, GPS lat/long/speed, time, and others, being sampled and sent via a ConnectOne Nano WiReach module via a Mifi2200 over the 'net about once a minute. Doing this using a simple file FTP back to the main server here in my basement. Small file, about 1K total per minute.

System 2 - The server. Simple laptop running Ubuntu 8.04, ProFTP, etc. Serves up a simple page using iFrames (I know, don’t groan so loud) with basically the .txt which was sent from System #1 displayed in the middle of it, and using the Refresh metatag to update the page.

System 3 - The browser PC, wherever I am, I can pull up the page, and have a look-see at what System 1 is doing at the moment.

Problems -

-While uploading a fresh dataset file via FTP from #1 to #2, AND #3 does a Refresh, obviously #3 comes up with a dead file because the file doesn’t really exist at that particular moment on #2 since it’s being re-written. The next time the page refreshes (or I ‘F5’), it’ll likely load just fine, obviously because the file is small and it updates pretty quick.

-Once per minutes sucks…'nuff said.

-The option suggested above in which the server (#2) sends out requests to the datalogger (#1) likely won’t work since the IP address will change fairly often (Mifi2200 internet access), likely every time the system is powered up. Also, if my datasheet reading is correct, the ConnectOne Nano WiReach doesn’t support these methods of accessing files or datasets which aren’t internal to the chip itself.

I read up on AJAX as noted above and it seems to me, unless I’m wrong which is quite possible, that AJAX would work just fine for the browser (#3) and server (#2), but not between the mobile device (#1) and the server (#2). And besides, I don’t get it…which I guess is the big thing. Looks like one helluva steep learning curve there.

So, I guess I need a quick & dirty-er method to “push” data from the mobile server #1 to the server #2, which, when updated would push that data out to a browser #3, which it looks like AJAX will handle if I can ever figure it out. I know the methods are out there. Just gotta get a handle on exactly what they are called/named/whatever.

10

The best way to do what you want is to have a listener on your server that will write what it receives to a database. You can write the listener in python/ruby/perl/etc. You have your webserver pull the data it displays from that database rather than a static text file. It’s fairly simple to do using Django or Rails.

11

Trigger:
The best way to do what you want is to have a listener on your server that will write what it receives to a database. You can write the listener in python/ruby/perl/etc. You have your webserver pull the data it displays from that database rather than a static text file. It’s fairly simple to do using Django or Rails.

And that's the kicker...

Like I said earlier, basic HTML, basic (very basic) Python, a fair amount of VB, a little VC+…no Ruby, no Perl, practically nothing ‘up to date’.

The WiFi modules I’m using are capable of TCP and/or UDP transfers, so I’m guessing that by “listener on your server”, you mean to write a Python something-or-other that would listen to whatever ‘sockets’ I set up and grab data as it becomes available.

I’m also guessing that the word “database” in this instance could mean anything from a full-on multi-MB access type file down to a basic comma-delimited text file.

Not sure how you mean for the server to display data from a database (or text file, or whatever) and have it auto-update. Haven’t looked into Django or Rails other than the Wiki references.

12

The listener is on the server itself, just a port that waits for packets from your sensor. It takes the data it receives and puts it in a text file at the simple level or something a bit more robust like a SQlite database.

If you know VB then write it in VB.

With whatever framework you use you just set an item to update at an interval and have the framework pull the most current data to display.

13

Ok, I think I’m starting (barely) to get the jist of it.

A simple text file will work good enough for me, even if it’s a CSV type or (worst case) fixed length record type of text file.

The server is an old laptop running Ubuntu 8.04 (for now anyways), so VB is out, but Python is still in.

The AJAX will require learning a bit of JAVA, just enough to make me dangerous, something I’ve been putting off for years.

If I got this right so far:

The mobile unit sends data packets using TCP fixed length transfer of a block of data, a couple hundred bytes at the very most.

A Python (or whatever) program running in the background on the server “listens” to whatever TCP port I select for data and writes that data to the text file as required.

(and I think I get a bit cornfuzzed here, not sure…)

The server sends out pages with blank data placeholders in X number of spots where data will eventually show up.

The browser, using AJAX, requests the server to get data from the text file to place in those X number of placeholders at regular intervals.

How am I doing?

14

If you want to listen on a TCP port to log data the simplest way is to use inetd:

  1. Pick an unused port number on your server and give it a “service name”. Add this to /etc/services. Your addition will look something like:
mydata 1234/tcp # My wireless project thingy
  1. Edit /etc/inetd.conf. Add the line:
mydata stream tcp nowait root /usr/bin/tee -a /tmp/datalog

Some Linux installations have xinetd instead of inetd. The concepts are the same but the configuration is just a little different. You’ll have to read the manpages.

  1. Restart inetd; find the PID of the inetd process and kill -HUP

If you did everything right you should be able to “telnet localhost 1234”, type some data, disconnect, and then find that data appended to /tmp/datalog. If it did not work check for errors in the syslog.

“/usr/bin/tee -a /tmp/datalog” just reads from STDIN, echoes it back, and also appends the input to /tmp/datalog. If you know any programming at all it should be very straightforward to replace this with something that would clean up the input or such.

Security notes: With this configuration anyone in the world can connect to this port and start logging data to your server. See the documentation for “tcpd” for ways you can restrict this to certain IP addresses. It’s also not a good idea to have this service running as root as I’ve shown in my example.

15

Ok, that’s another piece of the puzzle potentially solved. Sounds relatively simple enough.

In addition to “inetd” grabbing data for the log file, it’s a piece of cake to write another program to parse that file and trim it down/clean it out as required, rather than piling up data packet after data packet. Might have some issues with file contention (i.e. trying to trim it down while the system is trying to write to it), but I’ll work that when I get there, maybe copy the file over to a temp file and work on it there or something.

Now it’s onto figuring out how to update the page that’s served and/or write the page so the browser updates the page itself…

16

If you understand CGI programming you can write a script with a multipart output. The biggest issue is to watch for output buffering; Apache also likes to buffer output if you’re using mod_deflate or the Includes option (or similar post-processing modules). But anyway, if you can get past the buffering problem, this should update the output in the browser (which could be implemented inside an

17

Nope, haven’t done any CGI at all. I get the jist of what it’s used for, but that’s about it.

Like I said earlier, I’m forcing myself into one helluva steep learning curve trying to absorb like a dozen protocols, methods of web programming, etc. at once.

The more I play, the more stuff makes sense, but haven’t really solidly understood much past a fair amount of HTML.

I’m getting there…just takin’ my time.