notebooks/00. CHECK ME FIRST/.ipynb_checkpoints/00. Service Tests-checkpoint.ipynb

   1 {
   2  "metadata": {
   3   "name": "",
   4   "signature": "sha256:7a1fb9fe0590432e08d22db547bfd8825adc67f4ba099ab1a13587be26b31767"
   5  },
   6  "nbformat": 3,
   7  "nbformat_minor": 0,
   8  "worksheets": [
   9   {
  10    "cells": [
  11     {
  12      "cell_type": "heading",
  13      "level": 1,
  14      "metadata": {},
  15      "source": [
  16       "Testing the Virtual Machine Configuration"
  17      ]
  18     },
  19     {
  20      "cell_type": "markdown",
  21      "metadata": {},
  22      "source": [
  23       "To check the virtual machine (VM) configuration, select *Run All* from the *Cell* menu above and then read through this notebook."
  24      ]
  25     },
  26     {
  27      "cell_type": "heading",
  28      "level": 2,
  29      "metadata": {},
  30      "source": [
  31       "Configuration check"
  32      ]
  33     },
  34     {
  35      "cell_type": "markdown",
  36      "metadata": {},
  37      "source": [
  38       "Show the current directory the notebook server is running in *in the virtual machine*."
  39      ]
  40     },
  41     {
  42      "cell_type": "code",
  43      "collapsed": false,
  44      "input": [
  45       "!pwd"
  46      ],
  47      "language": "python",
  48      "metadata": {},
  49      "outputs": []
  50     },
  51     {
  52      "cell_type": "markdown",
  53      "metadata": {},
  54      "source": [
  55       "Show the version of Python that the notebook is running - it should be a variant of Python 3.4."
  56      ]
  57     },
  58     {
  59      "cell_type": "code",
  60      "collapsed": false,
  61      "input": [
  62       "import sys\n",
  63       "print(sys.version)"
  64      ],
  65      "language": "python",
  66      "metadata": {},
  67      "outputs": []
  68     },
  69     {
  70      "cell_type": "code",
  71      "collapsed": false,
  72      "input": [
  73       "#py3 libraries are still throwing warnings\n",
  74       "#We can disable the display of them within a notebook by using:\n",
  75       "#import warnings\n",
  76       "#warnings.simplefilter(action = \"ignore\", category = FutureWarning)"
  77      ],
  78      "language": "python",
  79      "metadata": {},
  80      "outputs": []
  81     },
  82     {
  83      "cell_type": "code",
  84      "collapsed": false,
  85      "input": [
  86       "sys.getdefaultencoding()"
  87      ],
  88      "language": "python",
  89      "metadata": {},
  90      "outputs": []
  91     },
  92     {
  93      "cell_type": "heading",
  94      "level": 2,
  95      "metadata": {},
  96      "source": [
  97       "Check to see that key libraries are installed"
  98      ]
  99     },
 100     {
 101      "cell_type": "code",
 102      "collapsed": false,
 103      "input": [
 104       "#!pip3 install --upgrade pandas"
 105      ],
 106      "language": "python",
 107      "metadata": {},
 108      "outputs": []
 109     },
 110     {
 111      "cell_type": "code",
 112      "collapsed": false,
 113      "input": [
 114       "import pandas as pd"
 115      ],
 116      "language": "python",
 117      "metadata": {},
 118      "outputs": []
 119     },
 120     {
 121      "cell_type": "code",
 122      "collapsed": false,
 123      "input": [
 124       "import matplotlib.pyplot as plt"
 125      ],
 126      "language": "python",
 127      "metadata": {},
 128      "outputs": []
 129     },
 130     {
 131      "cell_type": "code",
 132      "collapsed": false,
 133      "input": [
 134       "plt.plot([1,2,3,4])\n",
 135       "plt.ylabel('some numbers')\n",
 136       "plt.show()"
 137      ],
 138      "language": "python",
 139      "metadata": {},
 140      "outputs": []
 141     },
 142     {
 143      "cell_type": "heading",
 144      "level": 2,
 145      "metadata": {},
 146      "source": [
 147       "Database server tests"
 148      ]
 149     },
 150     {
 151      "cell_type": "markdown",
 152      "metadata": {},
 153      "source": [
 154       "A handful of quick tests just to see that the database servers are running."
 155      ]
 156     },
 157     {
 158      "cell_type": "heading",
 159      "level": 3,
 160      "metadata": {},
 161      "source": [
 162       "PostgreSQL"
 163      ]
 164     },
 165     {
 166      "cell_type": "markdown",
 167      "metadata": {},
 168      "source": [
 169       "Test that the PostgreSQL database is running... This example also shows how to connect to the database."
 170      ]
 171     },
 172     {
 173      "cell_type": "code",
 174      "collapsed": false,
 175      "input": [
 176       "from pandas import read_sql_query as psql"
 177      ],
 178      "language": "python",
 179      "metadata": {},
 180      "outputs": []
 181     },
 182     {
 183      "cell_type": "code",
 184      "collapsed": false,
 185      "input": [
 186       "from sqlalchemy import create_engine\n",
 187       "engine = create_engine('postgresql://test:test@localhost:5432/tm351test')"
 188      ],
 189      "language": "python",
 190      "metadata": {},
 191      "outputs": []
 192     },
 193     {
 194      "cell_type": "code",
 195      "collapsed": false,
 196      "input": [
 197       "#This alternative method may also be used\"\n",
 198       "#import psycopg2 as pg\n",
 199       "#If connecting to the default port, you can omit the port parameter\n",
 200       "#engine = pg.connect(dbname='tm351test',  host='localhost', user='test', password='test' , port=5432)"
 201      ],
 202      "language": "python",
 203      "metadata": {},
 204      "outputs": []
 205     },
 206     {
 207      "cell_type": "code",
 208      "collapsed": false,
 209      "input": [
 210       "psql(\"SELECT table_schema,table_name FROM information_schema.tables \\\n",
 211       "    ORDER BY table_schema,table_name;\", engine)"
 212      ],
 213      "language": "python",
 214      "metadata": {},
 215      "outputs": []
 216     },
 217     {
 218      "cell_type": "heading",
 219      "level": 3,
 220      "metadata": {},
 221      "source": [
 222       "MongoDB"
 223      ]
 224     },
 225     {
 226      "cell_type": "markdown",
 227      "metadata": {},
 228      "source": [
 229       "Test that the mongoDB database is running... This example also shows how to connect to the database."
 230      ]
 231     },
 232     {
 233      "cell_type": "code",
 234      "collapsed": false,
 235      "input": [
 236       "from pymongo import MongoClient"
 237      ],
 238      "language": "python",
 239      "metadata": {},
 240      "outputs": []
 241     },
 242     {
 243      "cell_type": "code",
 244      "collapsed": false,
 245      "input": [
 246       "#If connecting to the default port, you can omit the second (port number) parameter\n",
 247       "c = MongoClient('localhost', 27017)"
 248      ],
 249      "language": "python",
 250      "metadata": {},
 251      "outputs": []
 252     },
 253     {
 254      "cell_type": "code",
 255      "collapsed": false,
 256      "input": [
 257       "db = c['test-database']"
 258      ],
 259      "language": "python",
 260      "metadata": {},
 261      "outputs": []
 262     },
 263     {
 264      "cell_type": "code",
 265      "collapsed": false,
 266      "input": [
 267       "collection = db.test_collection"
 268      ],
 269      "language": "python",
 270      "metadata": {},
 271      "outputs": []
 272     },
 273     {
 274      "cell_type": "code",
 275      "collapsed": false,
 276      "input": [
 277       "post_id = collection.insert({'test':'asasas'})"
 278      ],
 279      "language": "python",
 280      "metadata": {},
 281      "outputs": []
 282     },
 283     {
 284      "cell_type": "code",
 285      "collapsed": false,
 286      "input": [
 287       "c.database_names()"
 288      ],
 289      "language": "python",
 290      "metadata": {},
 291      "outputs": []
 292     },
 293     {
 294      "cell_type": "code",
 295      "collapsed": false,
 296      "input": [
 297       "for r in collection.find():\n",
 298       "    print(r)"
 299      ],
 300      "language": "python",
 301      "metadata": {},
 302      "outputs": []
 303     },
 304     {
 305      "cell_type": "heading",
 306      "level": 2,
 307      "metadata": {},
 308      "source": [
 309       "Additional Install Tests"
 310      ]
 311     },
 312     {
 313      "cell_type": "markdown",
 314      "metadata": {},
 315      "source": [
 316       "Check to see that we can install other libraries as required - such as ones from github."
 317      ]
 318     },
 319     {
 320      "cell_type": "code",
 321      "collapsed": false,
 322      "input": [
 323       "#!pip3 uninstall -y folium\n",
 324       "#!pip3 install git+https://github.com/tbicr/folium.git@fixed#folium"
 325      ],
 326      "language": "python",
 327      "metadata": {},
 328      "outputs": []
 329     },
 330     {
 331      "cell_type": "code",
 332      "collapsed": false,
 333      "input": [
 334       "import folium\n",
 335       "#This initialisation route appears to be broken at the moment?\n",
 336       "#The tbicr patch above works but I think it's an earlier version that does not support clusters\n",
 337       "#folium.initialize_notebook()"
 338      ],
 339      "language": "python",
 340      "metadata": {},
 341      "outputs": []
 342     },
 343     {
 344      "cell_type": "code",
 345      "collapsed": false,
 346      "input": [
 347       "map = folium.Map(location=[40, -99], zoom_start=4)\n",
 348       "map.simple_marker([40.67, -73.94], popup='Add <b>popup</b> text here.')\n",
 349       "#map"
 350      ],
 351      "language": "python",
 352      "metadata": {},
 353      "outputs": []
 354     },
 355     {
 356      "cell_type": "code",
 357      "collapsed": false,
 358      "input": [
 359       "from IPython.display import HTML\n",
 360       "def inline_map(map):\n",
 361       "    \"\"\"\n",
 362       "    Embeds the HTML source of the map directly into the IPython notebook.\n",
 363       "    \n",
 364       "    This method will not work if the map depends on any files (json data). Also this uses\n",
 365       "    the HTML5 srcdoc attribute, which may not be supported in all browsers.\n",
 366       "    \"\"\"\n",
 367       "    map._build_map()\n",
 368       "    return HTML('<iframe srcdoc=\"{srcdoc}\" style=\"width: 100%; height: 510px; border: none\"></iframe>'.format(srcdoc=map.HTML.replace('\"', '&quot;')))\n",
 369       "\n",
 370       "def embed_map(map, path=\"map.html\"):\n",
 371       "    \"\"\"\n",
 372       "    Embeds a linked iframe to the map into the IPython notebook.\n",
 373       "    \n",
 374       "    Note: this method will not capture the source of the map into the notebook.\n",
 375       "    This method should work for all maps (as long as they use relative urls).\n",
 376       "    \"\"\"\n",
 377       "    map.create_map(path=path)\n",
 378       "    return HTML('<iframe src=\"files/{path}\" style=\"width: 100%; height: 510px; border: none\"></iframe>'.format(path=path))"
 379      ],
 380      "language": "python",
 381      "metadata": {},
 382      "outputs": []
 383     },
 384     {
 385      "cell_type": "code",
 386      "collapsed": false,
 387      "input": [
 388       "inline_map(map)"
 389      ],
 390      "language": "python",
 391      "metadata": {},
 392      "outputs": []
 393     },
 394     {
 395      "cell_type": "heading",
 396      "level": 2,
 397      "metadata": {},
 398      "source": [
 399       "Graphviz"
 400      ]
 401     },
 402     {
 403      "cell_type": "code",
 404      "collapsed": false,
 405      "input": [
 406       "%install_ext https://raw.github.com/cjdrake/ipython-magic/master/gvmagic.py"
 407      ],
 408      "language": "python",
 409      "metadata": {},
 410      "outputs": []
 411     },
 412     {
 413      "cell_type": "code",
 414      "collapsed": false,
 415      "input": [
 416       "%load_ext gvmagic"
 417      ],
 418      "language": "python",
 419      "metadata": {},
 420      "outputs": []
 421     },
 422     {
 423      "cell_type": "code",
 424      "collapsed": false,
 425      "input": [
 426       "%%dot digraph G {\n",
 427       "    a -> b;\n",
 428       "    b -> c;\n",
 429       "}"
 430      ],
 431      "language": "python",
 432      "metadata": {},
 433      "outputs": []
 434     },
 435     {
 436      "cell_type": "code",
 437      "collapsed": false,
 438      "input": [
 439       "%%dot graph G {\n",
 440       "  subgraph clusterA {\n",
 441       "    a -- b;\n",
 442       "    subgraph clusterC {\n",
 443       "      C -- D;\n",
 444       "    }\n",
 445       "  }\n",
 446       "  subgraph clusterB {\n",
 447       "    d -- f\n",
 448       "  }\n",
 449       "  d -- D --F\n",
 450       "}"
 451      ],
 452      "language": "python",
 453      "metadata": {},
 454      "outputs": []
 455     },
 456     {
 457      "cell_type": "heading",
 458      "level": 2,
 459      "metadata": {},
 460      "source": [
 461       "Kick the Shell.."
 462      ]
 463     },
 464     {
 465      "cell_type": "code",
 466      "collapsed": false,
 467      "input": [
 468       "#The shell service isn't being properly started at the moment. We can kick it into action with the following command:\n",
 469       "! /usr/local/lib/node_modules/tty.js/bin/tty.js --port 3000 --daemonize"
 470      ],
 471      "language": "python",
 472      "metadata": {},
 473      "outputs": []
 474     },
 475     {
 476      "cell_type": "heading",
 477      "level": 2,
 478      "metadata": {},
 479      "source": [
 480       "More Py 3 patches for now..."
 481      ]
 482     },
 483     {
 484      "cell_type": "code",
 485      "collapsed": false,
 486      "input": [
 487       "!pip3 uninstall -y statsmodels\n",
 488       "!pip3 install git+https://github.com/statsmodels/statsmodels.git"
 489      ],
 490      "language": "python",
 491      "metadata": {},
 492      "outputs": []
 493     },
 494     {
 495      "cell_type": "code",
 496      "collapsed": false,
 497      "input": [
 498       "#!pip3 uninstall -y ggplot\n",
 499       "!pip3 install git+https://github.com/yhat/ggplot.git"
 500      ],
 501      "language": "python",
 502      "metadata": {},
 503      "outputs": []
 504     },
 505     {
 506      "cell_type": "code",
 507      "collapsed": false,
 508      "input": [
 509       "#I've been filing issues... as updates appear, the quickest way to get the is from master repo\n",
 510       "!pip3 uninstall -y pandasql\n",
 511       "#!pip3 install git+https://github.com/yhat/pandasql.git\n",
 512       "#The original repo keeps breaking things in Python 3 - below is my fix\n",
 513       "#I wonder if we should clone repos and freeze them in a presentation branch to be used for that presentation's VM?\n",
 514       "!pip3 install git+https://github.com/psychemedia/pandasql.git"
 515      ],
 516      "language": "python",
 517      "metadata": {},
 518      "outputs": []
 519     },
 520     {
 521      "cell_type": "code",
 522      "collapsed": false,
 523      "input": [
 524       "#http://nbviewer.ipython.org/gist/pascal-schetelat/8382651\n",
 525       "#No installer - just download file... TO DO\n",
 526       "#!pip3 install git+https://github.com/pascal-schetelat/Slope.git"
 527      ],
 528      "language": "python",
 529      "metadata": {},
 530      "outputs": []
 531     },
 532     {
 533      "cell_type": "heading",
 534      "level": 2,
 535      "metadata": {},
 536      "source": [
 537       "What Next?"
 538      ]
 539     },
 540     {
 541      "cell_type": "markdown",
 542      "metadata": {},
 543      "source": [
 544       "If you saw no major errors or warnings and everything seemed to work okay, carry on with the next activity described in the course materials."
 545      ]
 546     },
 547     {
 548      "cell_type": "markdown",
 549      "metadata": {},
 550      "source": [
 551       "If you did get errors or warning messages, please check the [FAQ wiki](https://learn2.open.ac.uk/mod/ouwiki/view.php?id=556345) and the [Technical Help and FAQs Forum](https://learn2.open.ac.uk/mod/forumng/view.php?id=555623). If there is nothing there that helps, please post details of the issues you encountered, and *don't panic*. There are plenty of other things you can get on with while we try to sort the issue(s) for you - so once you've posted your issues, carry on with the next activity described in the course materials."
 552      ]
 553     }
 554    ],
 555    "metadata": {}
 556   }
 557  ]
 558 }