notebooks/Mongo - Network Failures.ipynb

   1 {
   2  "metadata": {
   3   "name": "",
   4   "signature": "sha256:9a3b3161171e965b84b47c0a89952df318a757552b3ea74c9237bae85da97fce"
   5  },
   6  "nbformat": 3,
   7  "nbformat_minor": 0,
   8  "worksheets": [
   9   {
  10    "cells": [
  11     {
  12      "cell_type": "heading",
  13      "level": 1,
  14      "metadata": {},
  15      "source": [
  16       "Mongo - Network Failures"
  17      ]
  18     },
  19     {
  20      "cell_type": "markdown",
  21      "metadata": {},
  22      "source": [
  23       "MongoDB is designed to support distributed operations and as such can tolerate network failures. In this notebook, we'll see how a MongoDB replica set fares in the face of simulated network failures.\n",
  24       "\n",
  25       "(More technical details about how the MongoDB replica set is configured can be found in the notebook [Dockering](./dockering.ipynb) )."
  26      ]
  27     },
  28     {
  29      "cell_type": "code",
  30      "collapsed": false,
  31      "input": [
  32       "from dockeringMongo import *"
  33      ],
  34      "language": "python",
  35      "metadata": {
  36       "activity": false
  37      },
  38      "outputs": [],
  39      "prompt_number": 1
  40     },
  41     {
  42      "cell_type": "code",
  43      "collapsed": false,
  44      "input": [
  45       "c = docker.Client(base_url='unix://var/run/docker.sock',\n",
  46       "                  version='1.10',\n",
  47       "                  timeout=10)"
  48      ],
  49      "language": "python",
  50      "metadata": {
  51       "activity": false
  52      },
  53      "outputs": [],
  54      "prompt_number": 2
  55     },
  56     {
  57      "cell_type": "code",
  58      "collapsed": false,
  59      "input": [
  60       "#Fire up three MongoDB containers that we'll use in a replica set\n",
  61       "\n",
  62       "#STUB is an identifier used to label the nodes\n",
  63       "#Superstition - short stub requcurl -L  https://gist.githubusercontent.com/psychemedia/b0c218135d2698648cbe/raw/activityCodeCell.js  > $(ipython locate profile)/static/custom/activityCodeCell.jsired?\n",
  64       "STUB='rs3'\n",
  65       "rsc=rs_config(c,STUB,num=3)\n",
  66       "rsc"
  67      ],
  68      "language": "python",
  69      "metadata": {
  70       "activity": false
  71      },
  72      "outputs": [
  73       {
  74        "metadata": {},
  75        "output_type": "pyout",
  76        "prompt_number": 9,
  77        "text": [
  78         "{'members': [{'host': '172.17.0.8:27017', '_id': 0},\n",
  79         "  {'host': '172.17.0.9:27017', '_id': 1},\n",
  80         "  {'host': '172.17.0.10:27017', '_id': 2}],\n",
  81         " '_id': 'rs3'}"
  82        ]
  83       }
  84      ],
  85      "prompt_number": 9
  86     },
  87     {
  88      "cell_type": "code",
  89      "collapsed": false,
  90      "input": [
  91       "showContainers(c)"
  92      ],
  93      "language": "python",
  94      "metadata": {
  95       "activity": false
  96      },
  97      "outputs": [
  98       {
  99        "output_type": "stream",
 100        "stream": "stdout",
 101        "text": [
 102         "['/rs3_srv2'] Up 32 seconds\n",
 103         "['/rs3_srv1'] Up 32 seconds\n",
 104         "['/rs3_srv0'] Up 32 seconds\n"
 105        ]
 106       }
 107      ],
 108      "prompt_number": 17
 109     },
 110     {
 111      "cell_type": "code",
 112      "collapsed": false,
 113      "input": [
 114       "tidyAwayContainers(c,['rs3_srv0','rs3_srv1','rs3_srv2'])"
 115      ],
 116      "language": "python",
 117      "metadata": {
 118       "activity": false
 119      },
 120      "outputs": [],
 121      "prompt_number": 25
 122     },
 123     {
 124      "cell_type": "code",
 125      "collapsed": false,
 126      "input": [
 127       "docker_ps(c)"
 128      ],
 129      "language": "python",
 130      "metadata": {
 131       "activity": false
 132      },
 133      "outputs": [
 134       {
 135        "metadata": {},
 136        "output_type": "pyout",
 137        "prompt_number": 26,
 138        "text": [
 139         "[]"
 140        ]
 141       }
 142      ],
 143      "prompt_number": 26
 144     },
 145     {
 146      "cell_type": "code",
 147      "collapsed": false,
 148      "input": [
 149       "#Initialise the replica set\n",
 150       "from pymongo import MongoClient, ReadPreference\n",
 151       "\n",
 152       "#We'll use the 0th server in the set as a the node\n",
 153       "mc = MongoClient('localhost', get27017tcp_port(c,STUB+'_srv0'))"
 154      ],
 155      "language": "python",
 156      "metadata": {},
 157      "outputs": [],
 158      "prompt_number": 23
 159     },
 160     {
 161      "cell_type": "code",
 162      "collapsed": false,
 163      "input": [
 164       "#The statusReport() command lets you ask a mongoDB node what it thinks the state of the world is\n",
 165       "statusReport(mc)"
 166      ],
 167      "language": "python",
 168      "metadata": {},
 169      "outputs": [
 170       {
 171        "ename": "OperationFailure",
 172        "evalue": "command SON([('replSetGetStatus', 1)]) failed: can't get local.system.replset config from self or any seed (EMPTYCONFIG)",
 173        "output_type": "pyerr",
 174        "traceback": [
 175         "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mOperationFailure\u001b[0m                          Traceback (most recent call last)",
 176         "\u001b[1;32m<ipython-input-24-5d7feaf31867>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m#The statusReport() command lets you ask a mongoDB node what it thinks the state of the world is\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mstatusReport\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
 177         "\u001b[1;32m/vagrant/notebooks/dockeringMongo.py\u001b[0m in \u001b[0;36mstatusReport\u001b[1;34m(client)\u001b[0m\n\u001b[0;32m     95\u001b[0m \u001b[1;31m#Mongo commands\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     96\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mstatusReport\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mclient\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 97\u001b[1;33m     \u001b[0mreport\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mclient\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madmin\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcommand\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"replSetGetStatus\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     98\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0mm\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mreport\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'members'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     99\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;34m'self'\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mm\u001b[0m \u001b[1;32mand\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mm\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'self'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 178         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/database.py\u001b[0m in \u001b[0;36mcommand\u001b[1;34m(self, command, value, check, allowable_errors, uuid_subtype, compile_re, **kwargs)\u001b[0m\n\u001b[0;32m    443\u001b[0m         \"\"\"\n\u001b[0;32m    444\u001b[0m         return self._command(command, value, check, allowable_errors,\n\u001b[1;32m--> 445\u001b[1;33m                              uuid_subtype, compile_re, **kwargs)[0]\n\u001b[0m\u001b[0;32m    446\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    447\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mcollection_names\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minclude_system_collections\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 179         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/database.py\u001b[0m in \u001b[0;36m_command\u001b[1;34m(self, command, value, check, allowable_errors, uuid_subtype, compile_re, **kwargs)\u001b[0m\n\u001b[0;32m    349\u001b[0m             \u001b[0mmsg\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m\"command %s failed: %%s\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mrepr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"%\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"%%\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    350\u001b[0m             helpers._check_command_response(result, self.connection.disconnect,\n\u001b[1;32m--> 351\u001b[1;33m                                             msg, allowable_errors)\n\u001b[0m\u001b[0;32m    352\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    353\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcursor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconn_id\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 180         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/helpers.py\u001b[0m in \u001b[0;36m_check_command_response\u001b[1;34m(response, reset, msg, allowable_errors)\u001b[0m\n\u001b[0;32m    176\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    177\u001b[0m             \u001b[0mmsg\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmsg\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;34m\"%s\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 178\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mOperationFailure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmsg\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0merrmsg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    179\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    180\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
 181         "\u001b[1;31mOperationFailure\u001b[0m: command SON([('replSetGetStatus', 1)]) failed: can't get local.system.replset config from self or any seed (EMPTYCONFIG)"
 182        ]
 183       }
 184      ],
 185      "prompt_number": 24
 186     },
 187     {
 188      "cell_type": "code",
 189      "collapsed": false,
 190      "input": [
 191       "#Initialise the replica set\n",
 192       "from pymongo import MongoClient\n",
 193       "\n",
 194       "#We'll use the 0th server in the set as a the node\n",
 195       "mc = MongoClient('localhost', get27017tcp_port(c,STUB+'_srv0'),replicaset=STUB)"
 196      ],
 197      "language": "python",
 198      "metadata": {
 199       "activity": false
 200      },
 201      "outputs": [
 202       {
 203        "ename": "ConfigurationError",
 204        "evalue": "localhost:49153 is not a member of replica set rs3",
 205        "output_type": "pyerr",
 206        "traceback": [
 207         "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mConfigurationError\u001b[0m                        Traceback (most recent call last)",
 208         "\u001b[1;32m<ipython-input-19-586455dbfe36>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m#We'll use the 0th server in the set as a the node\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mmc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mMongoClient\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'localhost'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mget27017tcp_port\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;34m'_srv0'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mreplicaset\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
 209         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, host, port, max_pool_size, document_class, tz_aware, _connect, **kwargs)\u001b[0m\n\u001b[0;32m    364\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0m_connect\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    365\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 366\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_ensure_connected\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    367\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mAutoReconnect\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    368\u001b[0m                 \u001b[1;31m# ConnectionFailure makes more sense here than AutoReconnect\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 210         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m_ensure_connected\u001b[1;34m(self, sync)\u001b[0m\n\u001b[0;32m    928\u001b[0m         \"\"\"Ensure this client instance is connected to a mongod/s.\n\u001b[0;32m    929\u001b[0m         \"\"\"\n\u001b[1;32m--> 930\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__ensure_member\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    931\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    932\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mdisconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 211         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__ensure_member\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    801\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    802\u001b[0m                 \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 803\u001b[1;33m                     \u001b[0mmember\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnodes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__find_node\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    804\u001b[0m                     \u001b[1;32mreturn\u001b[0m \u001b[0mmember\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    805\u001b[0m                 \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 212         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__find_node\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    861\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mcandidate\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mcandidates\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    862\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 863\u001b[1;33m                 \u001b[0mmember\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnodes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__try_node\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcandidate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    864\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mmember\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_mongos\u001b[0m \u001b[1;32mand\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__direct\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    865\u001b[0m                     \u001b[0mmongos_candidates\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmember\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 213         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__try_node\u001b[1;34m(self, node)\u001b[0m\n\u001b[0;32m    726\u001b[0m                 raise ConfigurationError(\"%s:%d is not a member of \"\n\u001b[0;32m    727\u001b[0m                                          \u001b[1;34m\"replica set %s\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 728\u001b[1;33m                                          % (node[0], node[1], self.__repl))\n\u001b[0m\u001b[0;32m    729\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    730\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;34m\"hosts\"\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 214         "\u001b[1;31mConfigurationError\u001b[0m: localhost:49153 is not a member of replica set rs3"
 215        ]
 216       }
 217      ],
 218      "prompt_number": 19
 219     },
 220     {
 221      "cell_type": "code",
 222      "collapsed": false,
 223      "input": [
 224       "#Set up connections to the other members of the replica set\n",
 225       "client1=MongoClient('localhost', get27017tcp_port(c,STUB+'_srv1'),replicaset=STUB)\n",
 226       "client2=MongoClient('localhost', get27017tcp_port(c,STUB+'_srv2'),replicaset=STUB)"
 227      ],
 228      "language": "python",
 229      "metadata": {
 230       "activity": false
 231      },
 232      "outputs": [
 233       {
 234        "ename": "ConfigurationError",
 235        "evalue": "localhost:49154 is not a member of replica set rs3",
 236        "output_type": "pyerr",
 237        "traceback": [
 238         "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mConfigurationError\u001b[0m                        Traceback (most recent call last)",
 239         "\u001b[1;32m<ipython-input-15-2ed9174bcab3>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m#Set up connections to the other members of the replica set\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mclient1\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mMongoClient\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'localhost'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mget27017tcp_port\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;34m'_srv1'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mreplicaset\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[0mclient2\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mMongoClient\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'localhost'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mget27017tcp_port\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;34m'_srv2'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mreplicaset\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mSTUB\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 240         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, host, port, max_pool_size, document_class, tz_aware, _connect, **kwargs)\u001b[0m\n\u001b[0;32m    364\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0m_connect\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    365\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 366\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_ensure_connected\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    367\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mAutoReconnect\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    368\u001b[0m                 \u001b[1;31m# ConnectionFailure makes more sense here than AutoReconnect\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 241         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m_ensure_connected\u001b[1;34m(self, sync)\u001b[0m\n\u001b[0;32m    928\u001b[0m         \"\"\"Ensure this client instance is connected to a mongod/s.\n\u001b[0;32m    929\u001b[0m         \"\"\"\n\u001b[1;32m--> 930\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__ensure_member\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    931\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    932\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mdisconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 242         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__ensure_member\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    801\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    802\u001b[0m                 \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 803\u001b[1;33m                     \u001b[0mmember\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnodes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__find_node\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    804\u001b[0m                     \u001b[1;32mreturn\u001b[0m \u001b[0mmember\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    805\u001b[0m                 \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 243         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__find_node\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    861\u001b[0m         \u001b[1;32mfor\u001b[0m \u001b[0mcandidate\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mcandidates\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    862\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 863\u001b[1;33m                 \u001b[0mmember\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnodes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__try_node\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcandidate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    864\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mmember\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_mongos\u001b[0m \u001b[1;32mand\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__direct\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    865\u001b[0m                     \u001b[0mmongos_candidates\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmember\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 244         "\u001b[1;32m/usr/local/lib/python3.4/dist-packages/pymongo/mongo_client.py\u001b[0m in \u001b[0;36m__try_node\u001b[1;34m(self, node)\u001b[0m\n\u001b[0;32m    726\u001b[0m                 raise ConfigurationError(\"%s:%d is not a member of \"\n\u001b[0;32m    727\u001b[0m                                          \u001b[1;34m\"replica set %s\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 728\u001b[1;33m                                          % (node[0], node[1], self.__repl))\n\u001b[0m\u001b[0;32m    729\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    730\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;34m\"hosts\"\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 245         "\u001b[1;31mConfigurationError\u001b[0m: localhost:49154 is not a member of replica set rs3"
 246        ]
 247       }
 248      ],
 249      "prompt_number": 15
 250     },
 251     {
 252      "cell_type": "code",
 253      "collapsed": false,
 254      "input": [
 255       "#In the mongo console, we would typically use the command MONGOCLIENT.config() to initialise the replica set\n",
 256       "#Here, we use the replSetInitiate admin command, applying it with the desired configuration\n",
 257       "mc.admin.command( \"replSetInitiate\",rsc);"
 258      ],
 259      "language": "python",
 260      "metadata": {
 261       "activity": false
 262      },
 263      "outputs": []
 264     },
 265     {
 266      "cell_type": "code",
 267      "collapsed": false,
 268      "input": [
 269       "#We may need to wait a minute or two for the configuration to come up\n",
 270       "#If you get an error message that suggests the configuration isn't up yet, wait a few seconds then rerun the cell\n",
 271       "mc.admin.command('replSetGetStatus')"
 272      ],
 273      "language": "python",
 274      "metadata": {
 275       "activity": false
 276      },
 277      "outputs": []
 278     },
 279     {
 280      "cell_type": "code",
 281      "collapsed": false,
 282      "input": [
 283       "#The statusReport() command lets you ask a mongoDB node what it thinks the state of the world is\n",
 284       "statusReport(mc)"
 285      ],
 286      "language": "python",
 287      "metadata": {
 288       "activity": false
 289      },
 290      "outputs": []
 291     },
 292     {
 293      "cell_type": "code",
 294      "collapsed": false,
 295      "input": [
 296       "#If we're quick, we can watch the machines come up into the fold\n",
 297       "statusReport(client1)"
 298      ],
 299      "language": "python",
 300      "metadata": {
 301       "activity": false
 302      },
 303      "outputs": []
 304     },
 305     {
 306      "cell_type": "code",
 307      "collapsed": false,
 308      "input": [
 309       "statusReport(client1)"
 310      ],
 311      "language": "python",
 312      "metadata": {
 313       "activity": false
 314      },
 315      "outputs": []
 316     },
 317     {
 318      "cell_type": "code",
 319      "collapsed": false,
 320      "input": [
 321       "statusReport(client1)"
 322      ],
 323      "language": "python",
 324      "metadata": {
 325       "activity": false
 326      },
 327      "outputs": []
 328     },
 329     {
 330      "cell_type": "heading",
 331      "level": 2,
 332      "metadata": {},
 333      "source": [
 334       "Modelling Network Errors"
 335      ]
 336     },
 337     {
 338      "cell_type": "markdown",
 339      "metadata": {},
 340      "source": [
 341       "In this section we'll use firewall rules between containers to model network partitions."
 342      ]
 343     },
 344     {
 345      "cell_type": "code",
 346      "collapsed": false,
 347      "input": [
 348       "#The blockade library contains functions for setting up and tearing down iptable firewall rulesets\n",
 349       "from blockade import *"
 350      ],
 351      "language": "python",
 352      "metadata": {
 353       "activity": false
 354      },
 355      "outputs": []
 356     },
 357     {
 358      "cell_type": "code",
 359      "collapsed": false,
 360      "input": [
 361       "#As we define blockade rules by IP address, we need to know those addresses\n",
 362       "rsc"
 363      ],
 364      "language": "python",
 365      "metadata": {
 366       "activity": false
 367      },
 368      "outputs": []
 369     },
 370     {
 371      "cell_type": "code",
 372      "collapsed": false,
 373      "input": [
 374       "#Maybe worth creating an object that lists stuff like this by instance name?\n",
 375       "memberIPaddress(rsc,'rs3_srv2')"
 376      ],
 377      "language": "python",
 378      "metadata": {
 379       "activity": false
 380      },
 381      "outputs": []
 382     },
 383     {
 384      "cell_type": "markdown",
 385      "metadata": {},
 386      "source": [
 387       "The `partition_containers()` function creates a named ruleset that puts listed IP addresses into separate partitions.\n",
 388       "\n",
 389       "Let's start by splitting off one of the secondary machines - rs3_srv2, on IP address 172.17.0.4:27017 - into a partition on its own:"
 390      ]
 391     },
 392     {
 393      "cell_type": "code",
 394      "collapsed": false,
 395      "input": [
 396       "#The machines in each partition are put into their own lists\n",
 397       "#partition_containers(RULESET,[ PARTITION_1_LIST, PARTITION_2_LIST ]\n",
 398       "partition_containers('test1w2s2', [ [netobj('172.17.0.4')],[netobj('172.17.0.2'),netobj('172.17.0.3')]])"
 399      ],
 400      "language": "python",
 401      "metadata": {
 402       "activity": false
 403      },
 404      "outputs": []
 405     },
 406     {
 407      "cell_type": "markdown",
 408      "metadata": {},
 409      "source": [
 410       "Wait a few seconds for the machines to realise all is not well before proceeding...\n",
 411       "\n",
 412       "...then check their statuses:"
 413      ]
 414     },
 415     {
 416      "cell_type": "code",
 417      "collapsed": false,
 418      "input": [
 419       "statusReport(mc)"
 420      ],
 421      "language": "python",
 422      "metadata": {
 423       "activity": false
 424      },
 425      "outputs": []
 426     },
 427     {
 428      "cell_type": "code",
 429      "collapsed": false,
 430      "input": [
 431       "statusReport(client1)"
 432      ],
 433      "language": "python",
 434      "metadata": {
 435       "activity": false
 436      },
 437      "outputs": []
 438     },
 439     {
 440      "cell_type": "code",
 441      "collapsed": false,
 442      "input": [
 443       "statusReport(client2)"
 444      ],
 445      "language": "python",
 446      "metadata": {
 447       "activity": false
 448      },
 449      "outputs": []
 450     },
 451     {
 452      "cell_type": "markdown",
 453      "metadata": {},
 454      "source": [
 455       "Let's fix the network..."
 456      ]
 457     },
 458     {
 459      "cell_type": "code",
 460      "collapsed": false,
 461      "input": [
 462       "clear_iptables('test1w2s2')"
 463      ],
 464      "language": "python",
 465      "metadata": {
 466       "activity": false
 467      },
 468      "outputs": []
 469     },
 470     {
 471      "cell_type": "code",
 472      "collapsed": false,
 473      "input": [
 474       "#Wait a second or too then see how things are...\n",
 475       "statusReport(mc)"
 476      ],
 477      "language": "python",
 478      "metadata": {
 479       "activity": false
 480      },
 481      "outputs": []
 482     },
 483     {
 484      "cell_type": "markdown",
 485      "metadata": {},
 486      "source": [
 487       "What happens if we knock out the connection between the primary server and the secondary servers?"
 488      ]
 489     },
 490     {
 491      "cell_type": "code",
 492      "collapsed": false,
 493      "input": [
 494       "#Put the primary server into a partition on it's own\n",
 495       "partition_containers('test1w2s2', [ [netobj('172.17.0.2')],[netobj('172.17.0.3'),netobj('172.17.0.4')]])"
 496      ],
 497      "language": "python",
 498      "metadata": {
 499       "activity": false
 500      },
 501      "outputs": []
 502     },
 503     {
 504      "cell_type": "markdown",
 505      "metadata": {},
 506      "source": [
 507       "Again, wait a few seconds for things to happen... then check the statuses:"
 508      ]
 509     },
 510     {
 511      "cell_type": "code",
 512      "collapsed": false,
 513      "input": [
 514       "statusReport(mc)"
 515      ],
 516      "language": "python",
 517      "metadata": {
 518       "activity": false
 519      },
 520      "outputs": []
 521     },
 522     {
 523      "cell_type": "code",
 524      "collapsed": false,
 525      "input": [
 526       "statusReport(client1)"
 527      ],
 528      "language": "python",
 529      "metadata": {
 530       "activity": false
 531      },
 532      "outputs": []
 533     },
 534     {
 535      "cell_type": "code",
 536      "collapsed": false,
 537      "input": [
 538       "statusReport(client2)"
 539      ],
 540      "language": "python",
 541      "metadata": {
 542       "activity": false
 543      },
 544      "outputs": []
 545     },
 546     {
 547      "cell_type": "code",
 548      "collapsed": false,
 549      "input": [
 550       "clear_iptables('test1w2s2')"
 551      ],
 552      "language": "python",
 553      "metadata": {
 554       "activity": false
 555      },
 556      "outputs": []
 557     },
 558     {
 559      "cell_type": "code",
 560      "collapsed": false,
 561      "input": [
 562       "statusReport(mc)"
 563      ],
 564      "language": "python",
 565      "metadata": {
 566       "activity": false
 567      },
 568      "outputs": []
 569     },
 570     {
 571      "cell_type": "code",
 572      "collapsed": false,
 573      "input": [
 574       "statusReport(client1)"
 575      ],
 576      "language": "python",
 577      "metadata": {
 578       "activity": false
 579      },
 580      "outputs": []
 581     },
 582     {
 583      "cell_type": "code",
 584      "collapsed": false,
 585      "input": [
 586       "statusReport(client2)"
 587      ],
 588      "language": "python",
 589      "metadata": {
 590       "activity": false
 591      },
 592      "outputs": []
 593     },
 594     {
 595      "cell_type": "markdown",
 596      "metadata": {},
 597      "source": [
 598       "Grab a copy of the logs from each of the servers to see what happened..."
 599      ]
 600     },
 601     {
 602      "cell_type": "code",
 603      "collapsed": false,
 604      "input": [
 605       "#In an ssh shell, the following sort of command displays a real time stream of stdio log messages from the container\n",
 606       "#!docker.io logs --follow=true {STUB}_srv1\n",
 607       "!docker.io logs {STUB}_srv0 > {STUB}_srv0_log.txt\n",
 608       "!docker.io logs {STUB}_srv1 > {STUB}_srv1_log.txt\n",
 609       "!docker.io logs {STUB}_srv2 > {STUB}_srv2_log.txt\n",
 610       "#?we may also be able to use mongostat - http://docs.mongodb.org/manual/reference/program/mongostat/"
 611      ],
 612      "language": "python",
 613      "metadata": {
 614       "activity": false
 615      },
 616      "outputs": []
 617     },
 618     {
 619      "cell_type": "markdown",
 620      "metadata": {},
 621      "source": [
 622       "*Maybe we need to use a log parser or highlighter to pull out interesting messages? Maybe we need to visualise the communication between the servers, eg with a combined timeline?*"
 623      ]
 624     },
 625     {
 626      "cell_type": "markdown",
 627      "metadata": {},
 628      "source": [
 629       "See below for the log for rs3_srv0, the original primary. There are several things to note:\n",
 630       "\n",
 631       "* around 11:29:42, it mutters to itself about the state of the network and whether it's electable;\n",
 632       "* between 11:29:48 and 11:29:49 it goes for itself as PRIMARY;\n",
 633       "* through 11:30:03 multiple connections are opened with other nodes in the replica set;\n",
 634       "* by 11:30:06, the replica set is up and running.\n",
 635       "* at 11:32:38, sight of rs3_srv2 is lost and it is reported as DOWN as we partition it;\n",
 636       "* through 11:33, more connections are opened to rs3_srv1 - is this a defensive measure?\n",
 637       "* at 11:34:07, rs3_srv2 comes back as the partition is removed;\n",
 638       "* at 11:34:37, we lose a connection to rs3_srv2, and through 11:34:4x note things are really down as a result of the partition we placed rs3_srv0 into;\n",
 639       "* at 11:34:49.634, rs3_srv0 gives up PRIMARY;\n",
 640       "* at 11:35:39, the partition is removed and the connections to the other machines in the replica set are restored; rs3_srv0 doesn't elect itself back as PRIMARY - rs3_srv2 has that job now."
 641      ]
 642     },
 643     {
 644      "cell_type": "code",
 645      "collapsed": false,
 646      "input": [
 647       "!tail -500 {STUB}_srv0_log.txt"
 648      ],
 649      "language": "python",
 650      "metadata": {
 651       "activity": false
 652      },
 653      "outputs": []
 654     },
 655     {
 656      "cell_type": "markdown",
 657      "metadata": {},
 658      "source": [
 659       "How about rs3_srv1's story?\n",
 660       "\n",
 661       "* through 11:29:4x, it starts to get into the replica set, agreeing to rs3_srv0 as PRIMARY at 11:29:48.472, and rebuilding itself through 11:30:03;\n",
 662       "* at 11:32:38.346 it loses sight of rs3_srv2, resighting it at 11:34:06;\n",
 663       "* at 11:34:40, lose sight of the current PRIMARY, rs3_srv0;\n",
 664       "* at 11:34:40.460, rs3_srv1 chooses not to stand for PRIMARY becuase it thinks rs3_srv2 will veto it, even though the server rs3_srv2 thinks is PRIMARY is the one that rs3_srv2 can't see. *(??Do we need also need an example situation where secondaries can see each other but one of the secondaries can't see the PRIMARY?)*\n",
 665       "* at 11:34:42 it votes for rs3_srv2 as PRIMARY; at 11:34:47, 11:34:47, and 11:34:58 it holds back from nominating itself becuase of its recent vote for someone else;\n",
 666       "* at 11:35:00, rs3_srv2 becomes PRIMARY;\n",
 667       "* at 11:35:39, rs3_srv0 rejoins as a SECONDARY."
 668      ]
 669     },
 670     {
 671      "cell_type": "code",
 672      "collapsed": false,
 673      "input": [
 674       "!tail -500 {STUB}_srv1_log.txt"
 675      ],
 676      "language": "python",
 677      "metadata": {
 678       "activity": false
 679      },
 680      "outputs": []
 681     },
 682     {
 683      "cell_type": "markdown",
 684      "metadata": {},
 685      "source": [
 686       "How did things look from the perspective of rs3_2?\n",
 687       "\n",
 688       "* as with rs3_srv1, we have initialisation up to 11:30:04;\n",
 689       "* at 11:32:34.449, spot the first connection error; this server is lost in the wilderness...\n",
 690       "* at 11:34:07, come back up, not bothering to try to take on the mantle of PRIMARY;\n",
 691       "* at 11:34:40, lose sight of rs3_srv0, the current PRIMARY;\n",
 692       "* at 11:34:42.110, make a pitch to become PRIMARY;\n",
 693       "* at 11:34:58.999, become PRIMARY;\n",
 694       "* at 11:35:40.499, rs3_srv0 comes back, letting us know it thought we were down, and is accepted back in as SECONDARY."
 695      ]
 696     },
 697     {
 698      "cell_type": "code",
 699      "collapsed": false,
 700      "input": [
 701       "!tail -500 {STUB}_srv2_log.txt"
 702      ],
 703      "language": "python",
 704      "metadata": {
 705       "activity": false
 706      },
 707      "outputs": []
 708     },
 709     {
 710      "cell_type": "code",
 711      "collapsed": false,
 712      "input": [
 713       "#Tidy up\n",
 714       "#Remove the blockade rules\n",
 715       "clear_iptables('test1w2s2')\n",
 716       "#Shut down the containers\n",
 717       "tidyAwayContainers(c,['rs3_srv0','rs3_srv1','rs3_srv2'])"
 718      ],
 719      "language": "python",
 720      "metadata": {
 721       "activity": false
 722      },
 723      "outputs": []
 724     },
 725     {
 726      "cell_type": "heading",
 727      "level": 1,
 728      "metadata": {},
 729      "source": [
 730       "Writing to Replica Sets"
 731      ]
 732     },
 733     {
 734      "cell_type": "markdown",
 735      "metadata": {},
 736      "source": [
 737       "Need examples of writing to different servers in the replica set under different partion conditions to show what happens to the replication in each case? Eg in matters of eventual consistency, etc?"
 738      ]
 739     },
 740     {
 741      "cell_type": "heading",
 742      "level": 1,
 743      "metadata": {},
 744      "source": [
 745       "PS"
 746      ]
 747     },
 748     {
 749      "cell_type": "markdown",
 750      "metadata": {},
 751      "source": [
 752       "I think there's a lot to be said for giving students a transcript such as this showing a worked example. Having got their eye in, we can perhaps get them to run slightly different scenarios, eg a disconnect between rs3_srv0 and rs3_srv2, with connections still up between rs3_srv0 and rs3_srv1, and rs3_srv1 and rs3_srv2. I'm not sure if we can do one-way breaks? (eg so rs3_srv1 can see rs3_srv0 but rs3_srv1 can't see rs3_srv0?)\n",
 753       "\n",
 754       "Also, students could run scenarios of inserting data into different machines under different network conditions?\n",
 755       "\n",
 756       "```\n",
 757       "testclient= MongoReplicaSetClient('{0}:{1}'.format(getContainIPaddress(c,STUB+'_srv0'),27017), replicaSet=STUB)\n",
 758       "testdb=testclient.testdb\n",
 759       "testcollection=testdb.testcollection\n",
 760       "testcollection.insert({'name':'test1'})\n",
 761       "testcollection.find()\n",
 762       "```"
 763      ]
 764     },
 765     {
 766      "cell_type": "code",
 767      "collapsed": false,
 768      "input": [],
 769      "language": "python",
 770      "metadata": {
 771       "activity": false
 772      },
 773      "outputs": []
 774     }
 775    ],
 776    "metadata": {}
 777   }
 778  ]
 779 }