Updates from admin Toggle Comment Threads | Keyboard Shortcuts

  • admin 17:35 on 2020-03-29 Permalink | Reply
    Tags: , pandas, , twitter   

    SARS-CoV-2, a.k.a “Corona Virus”, pseudo “data science” and Twitter 

    Some terminology (taken from – who would have thought it – t-online.de) and background information:

    • SARS-CoV-2 is the correct name of the virus. Everybody calls him “the Corona Virus” though, which is technically incorrect, cause there’s only the family of corona viruses – it’s a group, not a single one.
    • COVID-19 is the name of the disease. It’s an acronym made of “COrona VIrus Disease 19“. (It’s basically the same as HIV and AIDS – one is the infection, the other the actual illness).
    • It’s not the flu, but I think everybody has that down by now
    • It’s R0-value (how many people are infected by one person on average) is about 3 (source: RKI Germany)

    All of us are following the development of this very closely, cause we have no choice since we’re all basically locked at home. And even if we would go out everything’s closed. That gives us a ton of time to play DOOM Eternal (me), or write twitter bots who regularly publish the numbers of COVID-19 infections in Germany (me as well). As macabre as it is, this is very good oppotunity to start fooling around with data science. But to do this you need datasets, which are surprisingly hard to get. For the current case numbers  I found those:

    There’s probably more, but like I said – it’s surprisingly hard to find regularly updated, publicly available data sets in machine-readable form. (Also, this is the first time I go looking for this stuff, so maybe I just have no clue).

    Now back to the twitter bot. What does it do exactly? Go look for yourself, but if you are too lazy:

    • It prints the current infection rate at 8h, 12h, 16h, 20h. At the last time it will include a graph, a 1-week-forecast and a small evaluation how are stand today in comparison to the forecast which would have been made a week ago.
    • A friend helped me greatly by providing first a Holt-Winters prediction (which is live now), and then upgraded this to a more intelligent ARIMA prediction, which is still buried in a jupyter notebook and waiting for daylight. (Maybe she will write a guest post here to explain what it does and how it works – but I haven’t asked yet).
    • As for the bot’s code – it’s not (yet) public. Which is unusual for me, really. I should remedy that.

    What I learned so far:

    • It’s surprsingly easy and hard at the same time to get a Twitter developer account
    • Twitter does not permit publishing the exact same tweet multiple times in a short period of time
    • Heroku is really really nice for this, as long as you don’t need to pay for it. I would be interested in alternatives.
    • matplotlib is a lot more complicated than I expected
      • but strangely neither bokeh, nor plotly can actually export png graphics without either a separate electron app (WTF?) or a headless browser and selenium (W-T-F?!?)
    • pandas rocks, or more precisely: pandas data frames rock.
    • there does not seem to be a single properly maintained Twitter library for JavaScript, but there is at least tweepy for Python. (I mean I want to try something in JS, but honestly, if everything I find is outdated, I just stay with good old Python …)
    • The infection growth is intensely exponetial – almost a straight line on a log scale plot.

    Let’s see where this goes, and I hope you all stay healthy.

     

  • admin 08:32 on 2020-03-22 Permalink | Reply
    Tags: ,   

    Configure Python on Windows 

    All right, I have a Windows machine. It’s a PITA, but it’s here. And for some reason I started doing some Python testing on it. So this is how I managed to do it:

    Preparation:

    • Install python with choco (choco install -y python)
    • Run PowerShell as Administrator
      • Execute Set-ExecutionPolicy -ExecutionPolicy Unrestricted (we’ll see why in a very short time)

    Now to code it’s pretty similar to *NIX:

    • Create your code folder
    • Set up a python venv (python -m venv .env)
    • In VS Code, choose this interpreter

    So why the PowerShell stuff? Cause to activate the environment VS Code needs to execute a .ps1 script. Which it can’t, cause “executing scripts is disabled on this machine”, which seems to be the default setting.

    All in all, surprisingly straightforward. And I just noticed even the *NIX keyboard shortcuts (CTRL-A, CTRL-K, for example) work in the terminal window now. Crazy.

     

  • admin 12:22 on 2019-08-31 Permalink | Reply
    Tags: azure, functions, javascript, serverless   

    The 6 ways of returning data from an Azure Function 

    Why have one if you can have many? Well, beats me, but I thought I’d collect them here so I have my personal reference. Disclaimer: This is only for JavaScript based functions (most of the examples on the MS pages are C#).

    So, most of them have to do with the file function.json  file:

    {
    "bindings": [{
        "type": "httpTrigger",
        "name": "req",
        "and": "so on ..."
    }, {
        "type": "http",
        "name": "thisIsAnOutputBinding",
        "direction": "out"
    }, {
        "type": "someOtherType",
        "name": "anotherBinding",
        "direction": "out"
    }]
}

    More precisely, with the “out” type binding that is defined in the 2nd “bindings” object. The “out” binding has a “name” property, which is basically relevant for all of the methods.

    !!!!! WARNING !!!!! – It seems the name of the binding property must be camelCase. At least I consistently get “invalid binding” errors when I use snake_case

    Method 1 – context property

    The context object has one property per defined “out” binding. So if our property is named “thisIsAnOutputBinding”, our result property is context.thisIsAnOutputBinding , or as MicroSoft puts it:

    Outputs (bindings of direction === “out” ) can be written to by a function in a number of ways. In all cases, the name property of the binding as defined in function.json corresponds to the name of the object member written to in your function.

    Simply put, you just assign the value to a context  member and you’re done:

    module.exports = function(context, req) {
    context.thisIsAnOutputBinding = {
        "my": "return value",
        "in this case": "an object"
    }; 
    // contect.done() is only needed for synchronous functions context.done();
}

    Source here.

    Method 2 – return an object (async functions only)

    With the bindings given above, you can assign the values to the context property, but you can also return an object whose keys correspond to the binding names:

    module.exports = async function(context) {
  return { thisIsAnOutputBinding: 42, anotherBinding: 43 };
  // of course, instead of 42/43 you can put any js object here.
};

     

    Source here.

    Method 3 – using context.bindings

    This is basically a variant of method 1. Why does it exist? No one knows. Apparently the context object has a .bindings  property, which in turn again has properties which name-match the defined out bindings. So this is another possibility, and I think the return  is unnecessary:

    module.exports = async function(context) {
  let retMsg = "Hello, world!";
  context.bindings.thisIsAnOutputBinding = {
    body: retMsg
  };
  context.bindings.anotherBinding = retMsg;
  return;
};

    Source here.

    Method 4 – using context.done()

    If you’re using a sync method, you can’t return an object, but you can call context.done(err, obj) .

    module.exports = async function(context) {
  context.done(null, {
    thisIsAnOutputBinding: { text: "hello there, world", anotherBinding: 42 }
  });
};

    Source here.

    Method 5 – just return it already (only async functions)

    Probably someone said “Well, if I only have one output binding, why should I explicitly address it?”, which is a valid thought. So another way to return data was invented.

    For this the configuration in function.json looks a bit different:

    {
  "bindings": [
    { "type": "someType", name: "req", direction: "in" },
    { "type": "someOutType", direction: "out", name: "$return" }
  ]
}

    You have to set exactly one “out” type binding, and the name must be “$return”. Then you can do this:

    module.exports = async function(context, req) {
  return "woohooo!!";
};

    Source here.

    Methods 6 – special for http outputs

    You thought we were done? Noooooo. For http methods, you can diretly return the body object which is then used to create the body

    // actually NO IDEA if this works sync, async or both
// my guess is: sync, because "context.done() is implicitly called"
module.exports = async function(context, req) {
  rv = { body: "<html/>", status: 201 };
  context.res.send(rv);
};

    Source here.

    Summary

    Let’s just say it’s a mess, and too many ways ruin the map.

    It works though.

     

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel