esp8266 webImport

UPDATED and BUGFIXED!

MicroPython Repl command line is very nice, because you can telnet to the chip and read-and-evaluate python code. It is a great way of learning embedded IoT. The bad thing you cannot copy file while it is running, so experimenting get bad.

Also an automatic module update would be a very nice thing to have.

So I have created a small module called webImport which is able to

  1. download a module.py from a known http server
  2. write it in the esp8266 and import it
  3. enable your esp8266 to be hacked by a worm :) (see below for caveats)

The idea is to serve your remote module via a HTTP server and load it on the repl (or programmatically) with

from webimport import web_import
web_import('modulename')

After a while you edit your modulename.py and you make a newMethod. So you want to reload it…with…

web_import('modulename')
modulename.method()

The CODE

The code is below and was tested on esp8266. It should work on Pyboard and more powerful board too.

# Implement a web import api
# Rev 1.0.2
# Rev 1.0.1  was bugged 
#import ujson as json
import uos as os
import usocket as socket

debug=True

def say(m):
    if debug:
        print("wi: "+str(m))
    

def extractHeaders(headers_pair2split):    
    hp=headers_pair2split.split(b'\r\n')
    headers={}
    for elem in hp:
        k,v=elem.split(b':',1)
        headers[k]=v
        say(str(k)+" => "+str(v))

def http_get_async(url):
    say("webImport:"+url)
    _, _, host, path = url.split('/', 3)
    if ':' in host:
        host, port = host.split(':')
    else:
        port = 80
    addr = socket.getaddrinfo(host, int(port))[0][-1]
    s = socket.socket()
    s.connect(addr)
    s.send(bytes('GET /%s HTTP/1.0\r\nHost: %s\r\n\r\n' % (path, host),
                 'utf8'))
    headerStr = b''
    body_bytes= b''
    buffer=b''
    ## Skip headerStr: they are tiny, so we get a small chunk
    while True:        
        buffer= buffer + s.recv(250)
        if b'\r\n\r\n' in buffer:
            break
    # Evict headerStr
    headerStr, body_bytes = buffer.split(b'\r\n\r\n',1)
    # GG Consider splitting via b'\r\n' to get headers, get Date:
    # to implement a caching algorithm
    #print("wi: debug: headers:"+str(headerStr)+" SIZE:"+str(len(str(headerStr))))
    response, headers_pair2split=headerStr.split(b'\r\n',1)
    extractHeaders(headers_pair2split)
    if '200 OK' not in response:
        raise ImportError("Response not ok:"+str(response))
    yield body_bytes
    while True:
        data= s.recv(1600)
        if data:
            yield data
        else:
            break

def http_get(url):
    response = b''
    try:
        get = http_get_async(url)
        while True:
            file_bytes = get.send(None)
            response += file_bytes
    except StopIteration:
        pass

    response_str = str(response, 'utf-8')
    return response_str


def ensure_dirs(path):
    split_path = path.split('/')
    if len(split_path) > 1:
        for i, fragment in enumerate(split_path):
            parent = '/'.join(split_path[:-i])
            try:
                os.mkdir(parent)
            except OSError:
                pass


def http_get_to_file(url, path):
    ensure_dirs(path)
    totSize=0
    with open(path, 'w') as outfile:
        try:
            get = http_get_async(url)
            while True:
                file_bytes = get.send(None)
                outfile.write(file_bytes)
                totSize+=len(file_bytes)
        except StopIteration:            
            outfile.close()
            say("Loaded "+path+"#"+str(totSize))




def web_import(moduleName, host="192.168.1.7:8000", on_error_use_cache=True):
    try:
        fname="./"+moduleName+".py"
        url="http://"+host+"/"+fname
        http_get_to_file(url,fname)
        return __import__(moduleName)
    except OSError as e:
        print(str(e))
        if on_error_use_cache:                        
            m= __import__(moduleName)
            print("wi: "+moduleName+" ECONN cache ok")
            return m

 

Optimization

As a bonus you can cross-compile the module, using the MicroPython cross compiler, which runs under any Unix-like system and compiles .py scripts into .mpy files.

https://github.com/micropython/micropython/tree/master/mpy-cross

This step is optional but offer to you less memory usage and better performance

 

Final thoughts

Be warned: having a remote download and execution code is a security risk. Node.js’ Npm package manager face this risk again and again, so you should consider some sort of password protection and / or https authentication using the ussl module which does not protect you but Man in the middle attack.