4 "signature": "sha256:afb82a020546413ed833589b41bdec8f02beaae66e1085c91a668be000123418"
12 "cell_type": "heading",
16 "Getting Started With ggplot"
20 "cell_type": "markdown",
23 "*ggplot* is a Python port of the popular *ggplot2* R implementation by Hadley of Wickham of many the ideas proposed in Leland Wilkinson's *The Grammar of Graphics*. As such, it attempts to enforce good practice in the generation of charts from appropriately shaped datasets.\n",
25 "The \"default\" graphics library for use with *pandas* is arguably the [matplotlib](http://matplotlib.org/) 2D Python plotting library, but whilst *matplotlib* offers a wide variety of powerful charting capabilities, it is often less concise and more convoluted than *ggplot*. It doesn't produce charts that are quite as pretty (or professional looking) as *ggplot* does out of the box either!\n",
27 "*ggplot* also offers a cleaner separation of data and graphical transformations, in accord with Wilkinson's original model. The full original *ggplot2* implementation was also developed with the production of *statistical graphics* in mind. That is, the library provided a range of statistical transformations that could be applied to a dataset as part of the graphic generation process.\n",
29 "The full *ggplot* documentation can be found here: [*ggplot* documentation](http://ggplot.yhathq.com/docs/).\n",
31 "The python *[statsmodels](http://statsmodels.sourceforge.net/)* library is one of the more widely use statistical computing libraries for Python, providing a range of powerful chart types as well as employing *pandas* based data structres for representing datasets. But whilst it *statsmodels* does support the generation of powerful statistical charts, it is rather lacking in support of simpler chart types.\n",
33 "So on these grounds of what we might term, at worst, *principled expediency*, we will tend to focus on the use of *ggplot* although you are free to explore other charting libraries yourself. If you particularly want to make use of interactive Javascript style charts, howver, [Vincent](http://vincent.readthedocs.org/en/latest/) could be a good choice. If you prefer using *matplotlib*, that's fine too. However, we do expect that you *also* gain an understanding of how to make use of graphics libraries based on the ideas of *The Grammar of Graphics*."
37 "cell_type": "heading",
41 "Finding Some Data..."
55 "cell_type": "markdown",
58 "The data we will be using in this notebook was released under a Freedom of Information request to the Isle of Wight Council and describes the revenue taken by two ticket machines in a particular pay and display car park over a twelve month period. (You can see the original FOI request here: [Pay and display ticket machine logs](https://www.whatdotheyknow.com/request/pay_and_display_ticket_machine_l).)"
62 "cell_type": "markdown",
65 "The data is supplied in a set of Excel spreadsheets. We will open just one for now."
82 "! unzip data/pay_and_display_ticket_machine_l.zip -P "
92 "df=pd.read_excel(\"data/iw_parkingMeterData/4_5_Transaction Report RR Dec 2012 March 2013.xls\")\n",
100 "cell_type": "markdown",
103 "By inspection, we see that there are six rows before the header row. Let's try loading the data in by skipping those rows."
110 "df=pd.read_excel(\"data/iw_parkingMeterData/4_5_Transaction Report RR Dec 2012 March 2013.xls\", \\\n",
113 "language": "python",
118 "cell_type": "markdown",
121 "There appear to be some columns that just contain NaN values - let's tidy the dataset a little by dropping those columns."
128 "df.dropna(how='all',axis=1,inplace=True)\n",
131 "language": "python",
139 "#Check to see how the columns are typed\n",
142 "language": "python",
147 "cell_type": "markdown",
150 "Let's do a little more tidying:"
157 "#Cast the date column as a date type, specifying how to parse the dates.\n",
158 "#Set coerce = True to cast any strings that aren't recognised to a NaT value.\n",
159 "df.Date=pd.to_datetime(df.Date, format=\"%Y-%m-%d %H:%M:%S\",coerce=True)\n",
160 "#The final row - which originally had a \"Date\" labelled Total was actually a total row\n",
163 "language": "python",
171 "#Let's see if any other dates weren;t recognised as such\n",
172 "df[df[\"Date\"].isnull()]"
174 "language": "python",
182 "#Let's also just check the total by summing the values (except the total) in the Cash column\n",
183 "df[['Cash']][:-1].sum()"
185 "language": "python",
193 "#Let's just check the row count too\n",
194 "df[['Cash']][:-1].count()"
196 "language": "python",
204 "#Drop the final total row\n",
205 "df.dropna(subset=[\"Date\"],inplace=True)"
207 "language": "python",
212 "cell_type": "heading",
223 "#The ggplot library is currently under active development\n",
224 "#Grab the most recent version from the github repository\n",
225 "#!pip3 uninstall -y ggplot\n",
226 "#!pip3 install git+https://github.com/yhat/ggplot.git"
228 "language": "python",
233 "cell_type": "markdown",
236 "To get started with ggplot we need to load it in."
243 "from ggplot import *"
245 "language": "python",
250 "cell_type": "markdown",
253 "We call `ggplot` by passing in a dataframe and specifying the aesthetic mappings. We need to make sure we define an appropriate aesthetic mapping for each dimnsion we wish to represent in the final display."
257 "cell_type": "heading",
265 "cell_type": "markdown",
268 "We use `geom_point()` to generate a scatterplot. `geom_point()` requires at least *x* and *y* value mappings. In the `aes()` definition, assign the *x* and *y* coordinate axes to the (quoted) column names you wish to plot from the specified dataset."
275 "ggplot(df, aes(x=\"Date\",y=\"Cash\"))+geom_point()"
277 "language": "python",
282 "cell_type": "heading",
290 "cell_type": "markdown",
293 "`ggplot` charts are constructed on a layered basis. the `geom_title()` layer can be used to add a title to the chart."
300 "ggplot(df, aes(x=\"Date\",y=\"Cash\")) + geom_point() \\\n",
301 " + ggtitle(\"Payments made over time\")"
303 "language": "python",
308 "cell_type": "heading",
316 "cell_type": "markdown",
319 "Another useful layer for styling the presentation of a chart is the `labs()` layer that can be used to set axis labels.\n",
321 "Note that *ggplot* actually returns a chart object, which means that we can assign it to a variable and add further layers or modifcations to that variable or chart object. "
328 "g = ggplot(df, aes(x=\"Date\",y=\"Cash\")) + geom_point()\n",
329 "g = g + ggtitle(\"Payments made over time\")\n",
330 "g = g + labs(\"Transaction Date\", \"Transaction amount (\u00a3)\")\n",
333 "language": "python",
338 "cell_type": "heading",
342 "Bar Charts and Histograms"
346 "cell_type": "markdown",
349 "Whilst a very simple technique, *counting* is often one of the most useful tools in our toolbox. For example, let's count how many tickets were issued by each machine for each tariff by aggregating over each group using the `len` function."
356 "df[[\"Tariff\",\"Machine\"]].groupby(['Tariff',\"Machine\"]).agg(len).sort_index()"
358 "language": "python",
363 "cell_type": "markdown",
366 "Bar charts can be used to provide charts showing counts across categorical variables. Supply the categorical variable you wish to chart as the *x* value and use `geom_bar()`.\n",
368 "Note that if variable (`foo`) you wish to use for the categorical x-values has a numeric type, you can cast it to a categorical variable by calling it as follows: `x='factor(foo)'`."
375 "p = ggplot(aes(x='Tariff'), data=df)\n",
376 "p + geom_bar() + ggtitle(\"Number of Tickets per Tariff\") + labs(\"Tariff Code\", \"Count\")"
378 "language": "python",
383 "cell_type": "markdown",
386 "We can add in a grouping variable to produce a stacked bar chart. Here we can see the contribution to the total count in each tariff made from each machine."
393 "p = ggplot(aes(x='Tariff',fill=\"Machine\"), data=df)\n",
394 "p + geom_bar() + ggtitle(\"Number of Tickets per Tariff\") + labs(\"Tariff Code\", \"Count\")"
396 "language": "python",
401 "cell_type": "markdown",
404 "If the range we want along the horizontal *x*-axis is a continuous one, we can make use of `geom_histogram()`. The *binwidth* will be automatically calculated, but we can also force it to a particular width using the *binwidth* parameter.\n",
406 "Here I set the *binwidth* to 0.1, that is, 10 pence, so we can more closely look for small overpayments."
413 "p = ggplot(aes(x='Cash'), data=df)\n",
414 "p + geom_histogram(binwidth=0.1) "
416 "language": "python",
421 "cell_type": "heading",
425 "Frequency Distributions - geom_density()"
429 "cell_type": "markdown",
432 "Sometimes, whilst we *can* plot a chart, it may not really be meaningful to do so.\n",
434 "For situations where you have a continuous distribution of values along a continuous numerical axis, it may may sense to produce a *frequency distribution chart*. For example, a chart showing the distribution of the height of people in a population. The`geo,_density()` chart works out the relative frequency of each value and produces a smoothed curve that extimates the continuos (*frequency*) distribution. The vertical *y*-axis is the propotion of the population taking the value. The area under the curve should sum to 1.\n",
436 "The *Cash* payment received is, in a sense, a continuous variable (people could pay any amount, at least in steps of 5 pence, the smallest coin accepted by the parking meters) although the expectation is that only discrete amounts (as specified by the tariffs) are required as payment."
443 "p = ggplot(aes(x='Cash'), data=df)\n",
444 "p + geom_density() + ggtitle(\"Number of Tickets per Tariff\") + labs(\"Payment (\u00a3)\", \"Proportion\")"
446 "language": "python",
451 "cell_type": "markdown",
454 "From the peaks in the chart, we see peaks at about 50 pence, just below \u00a32, at about \u00a33.30, a small bump about \u00a34.40 and a final burst at about \u00a36.60.\n",
456 "We can lookup the tariff amounts from the *Description.1* column:"
463 "df['Description.1'].unique()"
465 "language": "python",
470 "cell_type": "markdown",
473 "That is, we have distinct payment amounts at \u00a310, \u00a36.60, \u00a34.50, \u00a33.40, \u00a33.00, \u00a31.90 and \u00a30.60.\n",
475 "We can add additional layers to the chart to hightlight these valuese using `geom_vline()`, which adds a vertical line at a particular *x* value."
482 "p = ggplot(aes(x='Cash'), data=df)\n",
483 "p = p + geom_density() + ggtitle(\"Number of Tickets per Tariff\") + labs(\"Payment (\u00a3)\", \"Proportion\")\n",
484 "p + geom_vline(xintercept=[10, 6.6, 4.5, 3.4, 3.0, 1.9, 1,0, 0.6 ],colour='blue') "
486 "language": "python",
491 "cell_type": "markdown",
494 "We see high frequency spikes at all these amounts apart from at the \u00a33 tariff (short stay coaches)."
498 "cell_type": "markdown",
501 "Note that we can set the extent of the *x* and *y* axes by adding `+ xlim(MIN_X, MAX_X)` and `+ ylim(MIN_Y, MAX_Y)` modification layers to the chart."
505 "cell_type": "heading",
509 "Line Charts - geom_line()"
513 "cell_type": "markdown",
516 "For charting continuous values, particular ones that are plotted over time, a line chart often makes most sense.\n",
518 "When looking at transaction reports including separate amounts for each trasnaction, a chart showing the *running total* or accumulated amount can often be useful\n",
520 "We can add such a value as an additional column by sorting the data frame appropriately and then calculaing the cumulative sun over the *Cash* column using the `cumsum()` method."
527 "df.sort(['Date'],inplace=True)\n",
528 "df['Cash_cumul'] = df.Cash.cumsum()\n",
531 "language": "python",
536 "cell_type": "markdown",
539 "To plot the value as a line chart, we use `geom_line()`."
546 "#As well as passing a dataframe to the ggplot function as the first argument, we can also pass it via the data= attribute \n",
547 "g = ggplot(aes(x=\"Date\",y=\"Cash_cumul\"), data=df )+ geom_line()\n",
550 "language": "python",
555 "cell_type": "heading",
563 "cell_type": "markdown",
566 "Modify the chart generated directly above by adding an appropriate title and tidying up the axis titles."
575 "#Add suitable axis labels, and then display the chart\n"
577 "language": "python",
582 "cell_type": "heading",
590 "cell_type": "markdown",
593 "Judicious use of colour can often help us pack more information into a chart in a way that still allows us to read it. For example, suppose we want to look at the accumulated spend over time by Tariff to see which Tariff appears to be generating most revenue.\n",
595 "We can group on the tariff and calculate the accumulated revenue within each tariff band. "
602 "group=df[['Tariff','Cash']].groupby('Tariff')\n",
603 "#For group of rows, apply the transformation to each row in the group\n",
604 "#The number of rows in the response will be the same as the number of rows in the original data frame\n",
605 "df['Cash_cumul2']=group.transform(cumsum)['Cash']\n",
608 "language": "python",
613 "cell_type": "markdown",
616 "We can now plot the tariff based culumative totals as separate lines, splitting each line out using the `colour` aesthetic."
623 "ggplot(df,aes(x=\"Date\",y=\"Cash_cumul2\",colour=\"Tariff\"))+geom_line()"
625 "language": "python",
630 "cell_type": "heading",
638 "cell_type": "markdown",
641 "On other occasions, we may wish to split out data from different groups into different charts. This is referred to as *faceting*. We can split a datset across separate charts based on the value of a particular group attribute by using the `facet_wrap()` layer."
648 "ggplot(df, aes(x=\"Date\",y=\"Cash_cumul2\")) + geom_line() \\\n",
649 " + ggtitle(\"Payments made over time\") \\\n",
650 " + labs(\"Transaction Date\", \"Transaction amount (\u00a3)\") \\\n",
651 " + facet_wrap(\"Tariff\")"
653 "language": "python",
658 "cell_type": "markdown",
661 "By default, axis values are generated for each chart independently. However, we can also force them to use the same axes by setting the `scales` parameter to `fixed`, as opposed to `free`."
668 "ggplot(df, aes(x=\"Date\",y=\"Cash_cumul2\")) + geom_line() \\\n",
669 " + ggtitle(\"Payments made over time\") \\\n",
670 " + labs(\"Transaction Date\", \"Transaction amount (\u00a3)\") \\\n",
671 " + facet_wrap(\"Tariff\",scales = \"fixed\")"
673 "language": "python",
678 "cell_type": "heading",
686 "cell_type": "markdown",
689 "How many ticket machines is the data collected from and how many transactions are recorded by each one?\n",
691 "How would you generate a faceted chart showing the accumulated transactions over time for each of the ticket machines identified in the ticket column?"
698 "# Identifying the number of distinct machines and number of transactions recorded by each one\n"
700 "language": "python",
708 "# Accumulated total for each machine\n"
710 "language": "python",
718 "#Chart faceted by ticket machine\n"
720 "language": "python",
725 "cell_type": "heading",
733 "cell_type": "markdown",
736 "When it comes to publishing charts in a particular publication, we often require that the chart is presented in a particular *style*. In the same way that we can use different CSS style files to alter the look of a particular HTML document, so we can alter the look of a chart generated using *ggplot* by applying differnt *themes* to the chart.\n",
738 "For example, if we need to inject a little humour or apparent \"casualness\" into a chart, at the expense of some accuracy in the chart, we can use the XCKD theme.\n",
740 "(For more information about the use of such a theme, and techniques for creating such \"sketchy visualiastions\", see *Wood, Jo, Petra Isenberg, Tobias Isenberg, Jason Dykes, Nadia Boukhelifa, and Aidan Slingsby. \"[Sketchy rendering for information visualization.\" IEEE Transactions on Visualization and Computer Graphics](http://hal.archives-ouvertes.fr/docs/00/72/08/24/PDF/Wood_2012_SRI.pdf), 18(12), 2012: 2749-2758*.)"
747 "p = ggplot(aes(x='Tariff',fill=\"Machine\"), data=df)\n",
748 "p = p + geom_bar() + ggtitle(\"Number of Tickets per Tariff\") + labs(\"Tariff Code\", \"Count\") \n",
752 "language": "python",
757 "cell_type": "markdown",
760 "Another very useful theme is the \"clean\" looking `theme_bw()`."
769 "language": "python",
774 "cell_type": "heading",
782 "cell_type": "markdown",
785 "In this notebook,we have introduced some of the basic chart types that are supported by *ggplot*, as well as some of the modifications you can make to the charts. You can find further information, as well as descriptions of additioanl chart types, from the [*ggplot* documentation](http://ggplot.yhathq.com/docs/).\n",
787 "Feel free to extend this notebook as your own personal reference notebook by adding further sections about additional chart types.\n",
789 "If you are working through this notebook as part of an inline exercise, return to the course materials now. If you are working through this set of notebooks as a whole, move on to [4.5.3 Getting Started With Maps - folium](4.5.3%20Getting%20Started%20With%20Maps%20-%20folium.ipynb)."
projects / tm351-notebooks.git / blob