Writing reproducible reports using LaTeX and Python

 Do you know what a reproducible report is? Probably not.

When writing a document we need to include some graphs to represent data, calculations about some dataset, ... To do that we usually use another tool to represent the data, export the graph as a PNG or JPG to include it in the document. The same approach is used for data, we can use tools such as GNU Octave, R, ... to manipulate data and then copy the result to the document.

This is a valid solution but when data change we need to do the same tasks again, integrate in the document, ...

But what if we can generate graphs, process the data with the same tool we use to write the report?

If you know R and R Studio maybe you know about Sweave, Knitr or R Markdown . Theses tools allow to include R code within a LaTeX document which will be evaluated when the document is created allowing to use the R output in the document to include graphs, data manipulation, etc

For instance the following is a report created using R Markdown:

The LaTeX code with the embeded R code:

But it is possible to manipulate data as well:

And the code (most of it):

But if you do not know R or you want to use a different language, for example Python. Is there any alternatives?

Yes, there are alternatives, but they are no such cool as using R Studio. We can use the python sty module. So if you have LaTeX installed you have to install that module. 

To install in in Fedora:

[root@archimedes ~]# dnf install texlive-python -y
...
[root@archimedes ~]#

To install in Debian:

root@gauss:~# apt install texlive-latex-extra -y
...
root@gauss:~#


If you use a different Linux distribution or do not use Linux you should to look for a LaTeX package containing the python.sty module and install it.

You need to include the python package in document's preamble:

\include{python}

Now, whenever you need to make some calculations you can use Python to perform them. You need to add a python block:

\begin{python}
...
\end{python}


Inside the python block we will include python code to perform the calculations. But not only we need to perform calculations but show them as well.

To show them as LaTeX we will print LaTeX code:

\documentclass[12pt,a4paper]{report}

\usepackage{python}
\usepackage[utf8]{inputenc}

\newtheorem{example}{Example}

\begin{document}

\begin{example}}
This is a quite simple example about using python within \LaTeX:}
\end{example}}

\begin{python}
# a ----> c
# b ----> x
#
def proporcionality(a, b, c, unit):
   sol = (c*b/a)
   # we need to print LaTeX commands to produce LaTeX output
   # utf-8 is used
   print(r'Peter is paid %d %s \ for working %d hours. He will be paid %.2f %s \ for working %d hours.' % (b, unit, a, sol, unit, c))
   print(r'Powered by \LaTeX \ and Python.')

proporcionality(10, 120, 40, '\\texteuro')
\end{python}

\end{document}

The LaTeX output will be:

As we can see we can use LaTeX commands within the Python code, printing them to the standard output as we have done with the \LaTeX which will print LaTeX in such a distinctive way.

This is a simple and silly example. Let's go to try a more complex and useful example.

Matrix multiplication is something that is common to many people, which use LaTeX I mean. 

So, we will have to multiply the two matrixes (or matrices if you are keen on Latin) and then use the Python syntax to print the LaTeX commands to create the matrices and the matrix coefficients will be taken from a "2-D python array":

\begin{python}
import numpy as np

# matrix A
A = [[1,1,1],[1,2,-3],[1,-3,18]]
# matrix B
B = [[27,-21,-5],[-21,17,4],[-5,4,1]]

# lets do the multiplication
C = np.matmul(A, B)

print(r'\begin{eqnarray*}')
# print matrix A
print(r'\left( \begin{array}{ccc}')
for i in range(3):
   print(r'%d & %d & %d \\' % (A[i][0], A[i][1], A[i][2]))
print(r'\end{array} \right)')

print(r'\cdot')

# print matrix B
print(r'\left( \begin{array}{ccc}')
for i in range(3):
   print(r'%d & %d & %d \\' % (B[i][0], B[i][1], B[i][2])
) print(r'\end{array} \right)')
print(r'=')

# lets print C = A * B print(r'\left( \begin{array}{ccc}')
for i in range(3):
   print(r'%d & %d & %d \\' % (C[i][0], C[i][1], C[i][2]))
print(r'\end{array} \right)')

print(r'\end{eqnarray*}')

\end{python}


The LaTeX output will be:

Another important task in a document is including images, sometimes manipulating data and creating graphs to understand the data or to show whatever you are working on.

Mathplotlib is a powerful visualization library (or Seaborn). We can also use these libraries to plot data, well in fact to create a PNG/JPG/... file and include it in the document:

\begin{example}
Using matplotlib tools. This example has been taken from \href{https://matplotlib.org/basemap/users/examples.html}{Maplotlib Documentation}:
\end{example}

\begin{python}
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np
# set up orthographic map projection with
# perspective of satellite looking down at 50N, 100W.
# use low resolution coastlines.
map = Basemap(projection='ortho',lat_0=45,lon_0=-100,resolution='l')
# draw coastlines, country boundaries, fill continents.
map.drawcoastlines(linewidth=0.25)
map.drawcountries(linewidth=0.25)
map.fillcontinents(color='coral',lake_color='aqua')
# draw the edge of the map projection region (the projection limb)
map.drawmapboundary(fill_color='aqua')
# draw lat/lon grid lines every 30 degrees
. map.drawmeridians(np.arange(0,360,30))
map.drawparallels(np.arange(-90,90,30))
# make up some data on a regular lat/lon grid.
nlats = 73; nlons = 145; delta = 2.*np.pi/(nlons-1)
lats = (0.5*np.pi-delta*np.indices((nlats,nlons))[0,:,:])
lons = (delta*np.indices((nlats,nlons))[1,:,:])
wave = 0.75*(np.sin(2.*lats)**8*np.cos(4.*lons))
mean = 0.5*np.cos(2.*lats)*((np.sin(2.*lats))**2 + 2.)
# compute native map projection coordinates of lat/lon grid.
x, y = map(lons*180./np.pi, lats*180./np.pi)
# contour data over the map.
cs = map.contour(x,y,wave+mean,15,linewidths=1.5)
plt.title('contour lines over filled continent background')
# create an image file with the plot
plt.savefig('contour.png')
\end{python}

% include the created image file in the document
\begin{center}
\includegraphics{contour.png}
\end{center}

An the LaTeX output:

Last but not least we are going to talk about two more things:

1. How to create the document.
2. Debugging

To create the document we will use pdflatex, as usual, but we need to use a flag:

$ pdflatex -enable-write18 latex-python-example.tex


But, what is this -enable-write18 thing? As we have included Python code within the document when the PDF is being create the TeX compiler will stop to run the Python code. This task is done outside TeX so with this option the TeX compiler is able to stop compiling the document to call external programs "to do things" and to include these "external objects" into TeX to be used to compile the document when compilation resumes. Maybe this explanation is not an accurate one but I think that it is explanatory enough.

For debugging, especially the Python code, let's see first the files created after a LaTeX document compilation:

$ ls -lh
total 440K
-rw-rw-r--. 1 jadebustos jadebustos 169K May 21 23:51 contour.png
-rw-rw-r--. 1 jadebustos jadebustos 31 May 21 23:51 latex.py
-rw-rw-r--. 1 jadebustos jadebustos 686 May 21 23:51 latex-python-example.aux
-rw-rw-r--. 1 jadebustos jadebustos 55K May 21 23:51 latex-python-example.log
-rw-rw-r--. 1 jadebustos jadebustos 0 May 21 23:51 latex-python-example.out
-rw-rw-r--. 1 jadebustos jadebustos 190K May 21 23:51 latex-python-example.pdf
-rw-rw-r--. 1 jadebustos jadebustos 1.3K May 21 23:51 latex-python-example.py
-rw-rw-r--. 1 jadebustos jadebustos 0 May 21 23:51 latex-python-example.py.err
-rw-rw-r--. 1 jadebustos jadebustos 0 May 21 23:51 latex-python-example.py.out
-rw-rw-r--. 1 jadebustos jadebustos 3.2K May 22 00:03 latex-python-example.tex
-rwx------. 1 jadebustos jadebustos 194 May 21 11:08 Makefile
$

As you can see most of the files are common files created when compiling a LaTeX document, but there are some new files:

• latex.py, this file contains the name of the job which is the name of the LaTeX file. Not a important one.
• latex-python-example.py, when compiling every time a Python block is found this file will be written with the code to be executed (remember -enable-write18 will allow to interrupt the compilation to call external programs to "do things" like running Python scripts). So you will find in this file the last Python block that was executed.
• latex-python-example.py.out, this file will contain the output to the standard output for the Python script latex-python-example.py.
• latex-python-example.py.err, this file will contain the output to the standard error for the Python script latex-python-example.py.

When the compilation fails or when the result is not the wanted one we can use these files to check if there was some error in the Python code. We have to take into account that these files will be used for all the Python blocks so the last execution for the Python code will be in these files. If the compilation failed we can check these files due to the Python code that failed (if the error is in the Python code not in the LaTeX code outside the Python blocks) was the last which was executed. On the contrary, if the compilation was successful but the expected result is not the one you wanted you will have to do some tricks as commenting/removing all the Python blocks behind the one which is not producing the expected output (unless you are able to see the problem reading the Python code).

Now you can use Python in your LaTeX documents. 

There are other alternatives as PythonTex. If you are interested you can read this Paper.

You can download the code from here.