Tagged: heroku Toggle Comment Threads | Keyboard Shortcuts

  • admin 17:35 on 2020-03-29 Permalink | Reply
    Tags: heroku, pandas, , twitter   

    SARS-CoV-2, a.k.a “Corona Virus”, pseudo “data science” and Twitter 

    Some terminology (taken from – who would have thought it – t-online.de) and background information:

    • SARS-CoV-2 is the correct name of the virus. Everybody calls him “the Corona Virus” though, which is technically incorrect, cause there’s only the family of corona viruses – it’s a group, not a single one.
    • COVID-19 is the name of the disease. It’s an acronym made of “COrona VIrus Disease 19“. (It’s basically the same as HIV and AIDS – one is the infection, the other the actual illness).
    • It’s not the flu, but I think everybody has that down by now
    • It’s R0-value (how many people are infected by one person on average) is about 3 (source: RKI Germany)

    All of us are following the development of this very closely, cause we have no choice since we’re all basically locked at home. And even if we would go out everything’s closed. That gives us a ton of time to play DOOM Eternal (me), or write twitter bots who regularly publish the numbers of COVID-19 infections in Germany (me as well). As macabre as it is, this is very good oppotunity to start fooling around with data science. But to do this you need datasets, which are surprisingly hard to get. For the current case numbers  I found those:

    There’s probably more, but like I said – it’s surprisingly hard to find regularly updated, publicly available data sets in machine-readable form. (Also, this is the first time I go looking for this stuff, so maybe I just have no clue).

    Now back to the twitter bot. What does it do exactly? Go look for yourself, but if you are too lazy:

    • It prints the current infection rate at 8h, 12h, 16h, 20h. At the last time it will include a graph, a 1-week-forecast and a small evaluation how are stand today in comparison to the forecast which would have been made a week ago.
    • A friend helped me greatly by providing first a Holt-Winters prediction (which is live now), and then upgraded this to a more intelligent ARIMA prediction, which is still buried in a jupyter notebook and waiting for daylight. (Maybe she will write a guest post here to explain what it does and how it works – but I haven’t asked yet).
    • As for the bot’s code – it’s not (yet) public. Which is unusual for me, really. I should remedy that.

    What I learned so far:

    • It’s surprsingly easy and hard at the same time to get a Twitter developer account
    • Twitter does not permit publishing the exact same tweet multiple times in a short period of time
    • Heroku is really really nice for this, as long as you don’t need to pay for it. I would be interested in alternatives.
    • matplotlib is a lot more complicated than I expected
      • but strangely neither bokeh, nor plotly can actually export png graphics without either a separate electron app (WTF?) or a headless browser and selenium (W-T-F?!?)
    • pandas rocks, or more precisely: pandas data frames rock.
    • there does not seem to be a single properly maintained Twitter library for JavaScript, but there is at least tweepy for Python. (I mean I want to try something in JS, but honestly, if everything I find is outdated, I just stay with good old Python …)
    • The infection growth is intensely exponetial – almost a straight line on a log scale plot.

    Let’s see where this goes, and I hope you all stay healthy.

     
  • penguin 18:10 on 2018-08-25 Permalink | Reply
    Tags: , digitalocean, , , heroku, , howto   

    Build your own PaaS with Dokku 

    I was looking for some “play” deployment method for a couple of things I want to try out. Most of them require a database. And it should be cheap, cause I don’t have any load on them and don’t earn any money, so I don’t want to spend basically no money if possible. The usual suspects are too expensive – AWS, Heroku, etc.

    So I looked around and found Dokku.

    Dokku is a set of – hang on – shell scripts – which basically emulate Heroku on a machine of your own. It’s integrated with Digital Ocean droplets out of the box, if you want it. And the whole thing is 5 € / month, which is perfect. It also integrates with a Dockerfile based deployment, so you do git push and everything just works.

    It’s amazing.

    This is how you get started. But before you can get started, you need a domain you control, either on AWS or any other hoster. This is for routing traffic to your deployments later. You also need a public SSH key, or better a public / private key pair. Once you have both you can …

    1. create a Digital Ocean account, and …
    2. add your SSH public key to your account, and …
    3. in that account, create a new droplet with a “Dokku” image preinstalled.
    4. Wait until the droplet finished provisioning.

    While the droplet is being created, you can also create a project locally to test it later:

    $ mkdir dokku-test
    $ cd dokku-test
    $ git init
    $ echo "FROM tutum/hello-world" > Dockerfile
    $ git add Dockerfile
    $ git commit -m "Initial commit"
    

    In this little test project we only create a Dockerfile from an hello-world image which displays “Hello world” in a browser so we can verify it worked.

    Once the droplet is done, you can start setting up your personal little PaaS. First, you have to configure your DNS. We will set up a wildcard entry for our deployments, and a non-wildcard entry for git. Let’s assume your domain is for-myself.com, then you would add …

    • my-paas.for-myself.com , type “A” (or “AAAA” if you are IPv6) to your droplet IP
    • *.my-paas.for-myself.com just the same

    Then you SSH into your droplet, and create your dokku project. (This is something you have to do for every project). All you have to do for this is:

    $ ssh root@DROPLET_IP
    ~# dokku apps:create hello-world
    -----> Creating hello-world... done
    ~# _

    Done.

    Now you configure a git remote URL for your project, and push it:

    $ git remote add dokku dokku@my-paas.for-myself.com:hello-world
    

    Again – done. If you push your project now (assuming DNS is already set), everything should happen automagically:

    $ git push --set-upstream dokku master
    X11 forwarding request failed
    Enumerating objects: 3, done.
    Counting objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done.
    Total 3 (delta 0), reused 0 (delta 0)
    -----> Cleaning up...
    -----> Building hello-world from dockerfile...
    remote: build context to Docker daemon  2.048kB
    Step 1/1 : FROM tutum/hello-world
    latest: Pulling from tutum/hello-world
    658bc4dc7069: Pulling fs layer
    [... TRUNCATED ...]
    983d35417974: Pull complete
    Digest: sha256:0d57def8055178aafb4c7669cbc25ec17f0acdab97cc587f30150802da8f8d85
    Status: Downloaded newer image for tutum/hello-world:latest
     ---> 31e17b0746e4
    Successfully built 31e17b0746e4
    Successfully tagged dokku/hello-world:latest
    -----> Setting config vars
           DOKKU_DOCKERFILE_PORTS:  80/tcp
    -----> Releasing hello-world (dokku/hello-world:latest)...
    -----> Deploying hello-world (dokku/hello-world:latest)...
    -----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
    -----> No Procfile found in app image
    -----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...
    -----> New DOKKU_SCALE file generated
    =====> web=1
    -----> Attempting pre-flight checks
           For more efficient zero downtime deployments, create a file CHECKS.
           See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples
           CHECKS file not found in container: Running simple container check...
    -----> Waiting for 10 seconds ...
    -----> Default container check successful!
    -----> Running post-deploy
    -----> Creating new /home/dokku/hello-world/VHOST...
    -----> Setting config vars
           DOKKU_PROXY_PORT_MAP:  http:80:80
    -----> Configuring hello-world.my-paas.for-myself.com...(using built-in template)
    -----> Creating http nginx.conf
    -----> Running nginx-pre-reload
           Reloading nginx
    -----> Setting config vars
           DOKKU_APP_RESTORE:  1
    =====> Renaming container (14c349cb496d) amazing_snyder to hello-world.web.1
    -----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)
    =====> Application deployed:
           http://hello-world.my-paas.for-myself.com
    
    To my-paas.for-myself.com:hello-world
     * [new branch]      master -> master
    Branch 'master' set up to track remote branch 'master' from 'dokku'.

    And if you open your URL now (which is hello-world.my-paas.for-myself.com) you should see this image:

    Now, for 5 € / month you get:

    • A heroku-like, no-nonsense, fully automated, git-based deployment platform
    • A server which you control (and have to maintain, okay, but on which you can deploy …)
    • A database (or many of them – dokku provides great integration for databases btw; more on that in another post)
    • Publicly reachable deployments (for customers, testing, whatever)
    • Let’s Encrypt certificates (dokku provides support for these as well, again more in a later post)
    • And for 1 € more (it’s always 20% of the base price) you get backups of your system)

    That’s absolutely incredible. Oh, and did I mention that the maintainers are not only friendly, but also super responsive and incredibly helpful on Slack?

     
c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel