Update docs
* Update README * Update index.rst * Update docstrings * Fix typo * Edit docs * Add error messages
This commit is contained in:
Vinayak Mehta
+27
-60
@@ -4,26 +4,24 @@
|
||||
contain the root `toctree` directive.
|
||||
|
||||
==================================
|
||||
Camelot: PDF parsing made simpler!
|
||||
Camelot: pdf parsing made simpler!
|
||||
==================================
|
||||
|
||||
Camelot is a Python 2.7 library and command-line tool for getting tables out of PDF files.
|
||||
Camelot is a Python 2.7 library and command-line tool for getting tables out of pdf files.
|
||||
|
||||
Why another PDF table parsing library?
|
||||
Why another pdf table parsing library?
|
||||
======================================
|
||||
|
||||
We tried a lot of tools available online to get tables out of PDFs, but each one had its limitations. `PDFTables`_ stopped its open source development in 2013. `SolidConverter`_ which powers `Smallpdf`_ is closed source. Recently, `Docparser`_ was launched, which again is closed source. `Tabula`_, though being open source, doesn't always give correct output. In most cases, we had to resort to writing custom scripts for each type of PDF.
|
||||
We tried a lot of tools available online to parse tables from pdf files. `PDFTables`_, `SolidConverter`_ are closed source, commercial products and a free trial doesn't last forever. `Tabula`_, which is open source, isn't very scalable. We found nothing that gave us complete control over the parsing process. In most cases, we didn't get the correct output and had to resort to writing custom scripts for each type of pdf.
|
||||
|
||||
.. _PDFTables: https://pdftables.com/
|
||||
.. _SolidConverter: http://www.soliddocuments.com/pdf/-to-word-converter/304/1
|
||||
.. _Smallpdf: smallpdf.com
|
||||
.. _Docparser: https://docparser.com/
|
||||
.. _Tabula: http://tabula.technology/
|
||||
|
||||
PDFs have feelings too
|
||||
======================
|
||||
Some background
|
||||
===============
|
||||
|
||||
PDF started as `The Camelot Project`_ when people wanted a cross-platform way to share documents, since a document looked different on each system. A PDF contains characters placed at specific x,y-coordinates. Spaces are simulated by placing characters relatively far apart.
|
||||
PDF started as `The Camelot Project`_ when people wanted a cross-platform way for sending and viewing documents. A pdf file contains characters placed at specific x,y-coordinates. Spaces are simulated by placing characters relatively far apart.
|
||||
|
||||
Camelot uses two methods to parse tables from PDFs, :doc:`lattice <lattice>` and :doc:`stream <stream>`. The names were taken from Tabula but the implementation is somewhat different, though it follows the same philosophy. Lattice looks for lines between text elements while stream looks for whitespace between text elements.
|
||||
|
||||
@@ -37,9 +35,9 @@ Usage
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.lattice import Lattice
|
||||
|
||||
>>> extractor = Lattice(Pdf('us-030.pdf'))
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Lattice(), 'us-030.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
:header: "Cycle Name","KI (1/km)","Distance (mi)","Percent Fuel Savings","","",""
|
||||
@@ -51,7 +49,7 @@ Usage
|
||||
"2032_2","0.17","57.8","21.7%","0.3%","2.7%","1.2%"
|
||||
"4171_1","0.07","173.9","58.1%","1.6%","2.1%","0.5%"
|
||||
|
||||
Camelot comes with a command-line tool in which you can specify the output format (csv, tsv, html, json, and xlsx), page numbers you want to parse and the output directory in which you want the output files to be placed. By default, the output files are placed in the same directory as the PDF.
|
||||
Camelot comes with a CLI where you can specify page numbers, output format, output directory etc. By default, the output files are placed in the same directory as the PDF.
|
||||
|
||||
::
|
||||
|
||||
@@ -63,11 +61,23 @@ Camelot comes with a command-line tool in which you can specify the output forma
|
||||
options:
|
||||
-h, --help Show this screen.
|
||||
-v, --version Show version.
|
||||
-V, --verbose Verbose.
|
||||
-p, --pages <pageno> Comma-separated list of page numbers.
|
||||
Example: -p 1,3-6,10 [default: 1]
|
||||
-P, --parallel Parallelize the parsing process.
|
||||
-f, --format <format> Output format. (csv,tsv,html,json,xlsx) [default: csv]
|
||||
-l, --log Print log to file.
|
||||
-l, --log Log to file.
|
||||
-o, --output <directory> Output directory.
|
||||
-M, --cmargin <cmargin> Char margin. Chars closer than cmargin are
|
||||
grouped together to form a word. [default: 2.0]
|
||||
-L, --lmargin <lmargin> Line margin. Lines closer than lmargin are
|
||||
grouped together to form a textbox. [default: 0.5]
|
||||
-W, --wmargin <wmargin> Word margin. Insert blank spaces between chars
|
||||
if distance between words is greater than word
|
||||
margin. [default: 0.1]
|
||||
-S, --print-stats List stats on the parsing process.
|
||||
-T, --save-stats Save stats to a file.
|
||||
-X, --plot <dist> Plot distributions. (page,all,rc)
|
||||
|
||||
camelot methods:
|
||||
lattice Looks for lines between data.
|
||||
@@ -80,7 +90,7 @@ Installation
|
||||
|
||||
Make sure you have the most updated versions for `pip` and `setuptools`. You can update them by::
|
||||
|
||||
pip install -U pip, setuptools
|
||||
pip install -U pip setuptools
|
||||
|
||||
The required dependencies include `numpy`_, `OpenCV`_ and `ImageMagick`_.
|
||||
|
||||
@@ -88,46 +98,10 @@ The required dependencies include `numpy`_, `OpenCV`_ and `ImageMagick`_.
|
||||
.. _OpenCV: http://opencv.org/
|
||||
.. _ImageMagick: http://www.imagemagick.org/script/index.php
|
||||
|
||||
We strongly recommend that you use a `virtual environment`_ to install Camelot. If you don't want to use a virtual environment, then skip the next section.
|
||||
|
||||
Installing virtualenvwrapper
|
||||
----------------------------
|
||||
|
||||
You'll need to install `virtualenvwrapper`_.
|
||||
|
||||
::
|
||||
|
||||
pip install virtualenvwrapper
|
||||
|
||||
or
|
||||
|
||||
::
|
||||
|
||||
sudo pip install virtualenvwrapper
|
||||
|
||||
After installing virtualenvwrapper, add the following lines to your `.bashrc` and source it.
|
||||
|
||||
::
|
||||
|
||||
export WORKON_HOME=$HOME/.virtualenvs
|
||||
source /usr/bin/virtualenvwrapper.sh
|
||||
|
||||
.. note:: The path to `virtualenvwrapper.sh` could be different on your system.
|
||||
|
||||
Finally make a virtual environment using::
|
||||
|
||||
mkvirtualenv camelot
|
||||
|
||||
Installing dependencies
|
||||
-----------------------
|
||||
|
||||
`numpy` can be install using `pip`.
|
||||
|
||||
::
|
||||
|
||||
pip install numpy
|
||||
|
||||
`OpenCV` and `imagemagick` can be installed using your system's default package manager.
|
||||
numpy can be install using `pip`. OpenCV and imagemagick can be installed using your system's default package manager.
|
||||
|
||||
Linux
|
||||
^^^^^
|
||||
@@ -151,17 +125,10 @@ OS X
|
||||
|
||||
brew install homebrew/science/opencv imagemagick
|
||||
|
||||
If you're working in a virtualenv, you'll need to create a symbolic link for the OpenCV shared object file::
|
||||
|
||||
sudo ln -s /path/to/system/site-packages/cv2.so ~/path/to/virtualenv/site-packages/cv2.so
|
||||
|
||||
Finally, `cd` into the project directory and install by doing::
|
||||
Finally, `cd` into the project directory and install by::
|
||||
|
||||
make install
|
||||
|
||||
.. _virtual environment: http://virtualenvwrapper.readthedocs.io/en/latest/install.html#basic-installation
|
||||
.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.io/en/latest/
|
||||
|
||||
API Reference
|
||||
=============
|
||||
|
||||
|
||||
+19
-19
@@ -4,15 +4,15 @@
|
||||
Lattice
|
||||
=======
|
||||
|
||||
Lattice method is designed to work on PDFs which have tables with well-defined grids. It looks for lines on a page to form a table representation.
|
||||
Lattice method is designed to work on pdf files which have tables with well-defined grids. It looks for lines on a page to form a table.
|
||||
|
||||
Lattice uses OpenCV to apply a set of morphological transformations (erosion and dilation) to find horizontal and vertical line segments in a PDF page after converting it to an image using imagemagick.
|
||||
Lattice uses OpenCV to apply a set of morphological transformations (erosion and dilation) to find horizontal and vertical line segments in a pdf page after converting it to an image using imagemagick.
|
||||
|
||||
.. note:: Currently, Lattice only works on PDFs that contain text i.e. they are not composed of an image of the text. However, we plan to add `OCR support`_ in the future.
|
||||
.. note:: Currently, Lattice only works on pdf files that contain text. However, we plan to add `OCR support`_ in the future.
|
||||
|
||||
.. _OCR support: https://github.com/socialcopsdev/camelot/issues/14
|
||||
|
||||
Let's see how Lattice processes this PDF, step by step.
|
||||
Let's see how Lattice processes this pdf, step by step.
|
||||
|
||||
Line segments are detected in the first step.
|
||||
|
||||
@@ -40,7 +40,7 @@ The detected line segments are overlapped again, this time by `or` ing their pix
|
||||
:scale: 50%
|
||||
:align: left
|
||||
|
||||
Since dimensions of a PDF and its image vary; table contours, intersections and segments are scaled and translated to the PDF's coordinate space. A representation of the table is then created using these scaled coordinates.
|
||||
Since dimensions of a pdf and its image vary; table contours, intersections and segments are scaled and translated to the pdf's coordinate space. A representation of the table is then created using these scaled coordinates.
|
||||
|
||||
.. image:: assets/table.png
|
||||
:height: 674
|
||||
@@ -63,9 +63,9 @@ Finally, the characters found on the page are assigned to cells based on their x
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.lattice import Lattice
|
||||
|
||||
>>> extractor = Lattice(Pdf('us-030.pdf'))
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Lattice(), 'us-030.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
:header: "Cycle Name","KI (1/km)","Distance (mi)","Percent Fuel Savings","","",""
|
||||
@@ -82,7 +82,7 @@ Scale
|
||||
|
||||
The scale parameter is used to determine the length of the structuring element used for morphological transformations. The length of vertical and horizontal structuring elements are found by dividing the image's height and width respectively, by `scale`. Large `scale` will lead to a smaller structuring element, which means that smaller lines will be detected. The default value for scale is 15.
|
||||
|
||||
Let's consider this PDF.
|
||||
Let's consider this pdf file.
|
||||
|
||||
.. .. _this: insert link for row_span_1.pdf
|
||||
|
||||
@@ -105,16 +105,16 @@ Voila! It detected the smaller lines.
|
||||
Fill
|
||||
----
|
||||
|
||||
In the PDF used above, you can see that some cells spanned a lot of rows, `fill` just copies the same value to all rows/columns of a spanning cell. You can apply fill horizontally, vertically or both. Let us fill the output for the PDF we used above, vertically.
|
||||
In the file used above, you can see that some cells spanned a lot of rows, `fill` just copies the same value to all rows/columns of a spanning cell. You can apply fill horizontally, vertically or both. Let us fill the output for the file we used above, vertically.
|
||||
|
||||
::
|
||||
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.lattice import Lattice
|
||||
|
||||
>>> extractor = Lattice(Pdf('row_span_1.pdf'), fill='v', scale=40)
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Lattice(fill=['v'], scale=40), 'row_span_1.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
:header: "Plan Type","County","Plan Name","Totals"
|
||||
@@ -162,7 +162,7 @@ In the PDF used above, you can see that some cells spanned a lot of rows, `fill`
|
||||
Invert
|
||||
------
|
||||
|
||||
To find line segments, Lattice needs the lines of the PDF to be in foreground. So, if you encounter a PDF like this, just set invert to True.
|
||||
To find line segments, Lattice needs the lines of the pdf file to be in foreground. So, if you encounter a file like this, just set invert to True.
|
||||
|
||||
.. .. _this: insert link for lines_in_background_1.pdf
|
||||
|
||||
@@ -171,9 +171,9 @@ To find line segments, Lattice needs the lines of the PDF to be in foreground. S
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.lattice import Lattice
|
||||
|
||||
>>> extractor = Lattice(Pdf('lines_in_background_1.pdf'), invert=True)
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Lattice(invert=True), 'lines_in_background_1.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
:header: "State","Date","Halt stations","Halt days","Persons directly reached(in lakh)","Persons trained","Persons counseled","Persons testedfor HIV"
|
||||
@@ -186,8 +186,8 @@ To find line segments, Lattice needs the lines of the PDF to be in foreground. S
|
||||
"Kerala","23.2.2010 to 11.3.2010","9","17","1.42","3,559","2,173","855"
|
||||
"Total","","47","92","11.81","22,455","19,584","10,644"
|
||||
|
||||
Lattice can also parse PDFs with tables like these that are rotated clockwise/anti-clockwise by 90 degrees.
|
||||
Lattice can also parse pdf files with tables like these that are rotated clockwise/anti-clockwise by 90 degrees.
|
||||
|
||||
.. .. _these: insert link for left_rotated_table.pdf
|
||||
|
||||
You can call Lattice with debug={'line', 'intersection', 'contour', 'table'}, and call `plot_geometry()` which will generate an image like the ones on this page, with the help of which you can modify various parameters. See :doc:`API doc <api>` for more information.
|
||||
You can call Lattice with debug={'line', 'intersection', 'contour', 'table'}, and call `debug_plot()` which will generate an image like the ones on this page, with the help of which you can modify various parameters. See :doc:`API doc <api>` for more information.
|
||||
|
||||
+19
-19
@@ -4,20 +4,20 @@
|
||||
Stream
|
||||
======
|
||||
|
||||
Stream method is the complete opposite of Lattice and works on PDFs which have text placed uniformly apart across rows to simulate a table. It looks for spaces between text to form a table representation.
|
||||
Stream method is the complete opposite of Lattice and works on pdf files which have text placed uniformly apart across rows to simulate a table. It looks for spaces between text to form a table representation.
|
||||
|
||||
Stream builds on top of PDFMiner's functionality of grouping characters on a page into words and sentences. After getting these words, it groups them into rows based on their y-coordinates and tries to guess the number of columns a PDF table might have by calculating the mode of the number of words in each row. Additionally, the user can specify the number of columns or column x-coordinates.
|
||||
Stream builds on top of PDFMiner's functionality of grouping characters on a page into words and sentences. After getting these words, it groups them into rows based on their y-coordinates and tries to guess the number of columns a pdf table might have by calculating the mode of the number of words in each row. Additionally, the user can specify the number of columns or column x-coordinates.
|
||||
|
||||
Let's run it on this PDF.
|
||||
Let's run it on this pdf.
|
||||
|
||||
::
|
||||
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.stream import Stream
|
||||
|
||||
>>> extractor = Stream(Pdf('eu-027.pdf'))
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Stream(), 'eu-027.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. .. _this: insert link for eu-027.pdf
|
||||
|
||||
@@ -66,9 +66,9 @@ But sometimes its guess could be incorrect, like in this case.
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.stream import Stream
|
||||
|
||||
>>> extractor = Stream(Pdf('missing_values.pdf'))
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Stream(), 'missing_values.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. .. _this: insert link for missing_values.pdf
|
||||
|
||||
@@ -118,16 +118,16 @@ But sometimes its guess could be incorrect, like in this case.
|
||||
"14...","",""
|
||||
"Chronic...","",""
|
||||
|
||||
It guessed that the PDF has 3 columns, because there wasn't any data in the last 2 columns for most rows. So, let's specify the number of columns explicitly, following which, Stream will only consider rows that have 5 words, to decide on column boundaries.
|
||||
It guessed that the pdf has 3 columns, because there wasn't any data in the last 2 columns for most rows. So, let's specify the number of columns explicitly, following which, Stream will only consider rows that have 5 words, to decide on column boundaries.
|
||||
|
||||
::
|
||||
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.stream import Stream
|
||||
|
||||
>>> extractor = Stream(Pdf('missing_values.pdf'), ncolumns=5)
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Stream(ncolumns=[5]), 'missing_values.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
|
||||
@@ -175,15 +175,15 @@ It guessed that the PDF has 3 columns, because there wasn't any data in the last
|
||||
"14...","","","",""
|
||||
"Chronic...","","","",""
|
||||
|
||||
We can also specify the column x-coordinates. We need to call Stream with debug=True and use matplotlib's interface to note down the column x-coordinates we need. Let's try it on this PDF.
|
||||
We can also specify the column x-coordinates. We need to call Stream with debug=True and use matplotlib's interface to note down the column x-coordinates we need. Let's try it on this pdf file.
|
||||
|
||||
::
|
||||
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.stream import Stream
|
||||
|
||||
>>> extractor = Stream(Pdf('mexican_towns.pdf'), debug=True)
|
||||
>>> extractor.plot_text()
|
||||
>>> manager = Pdf(Stream(debug=True), 'mexican_towns.pdf'), debug=True
|
||||
>>> manager.debug_plot()
|
||||
|
||||
.. image:: assets/columns.png
|
||||
:height: 674
|
||||
@@ -198,9 +198,9 @@ After getting the x-coordinates, we just need to pass them to Stream, like this.
|
||||
>>> from camelot.pdf import Pdf
|
||||
>>> from camelot.stream import Stream
|
||||
|
||||
>>> extractor = Stream(Pdf('mexican_towns.pdf'), columns='28,67,180,230,425,475,700')
|
||||
>>> tables = extractor.get_tables()
|
||||
>>> print tables['page-1'][0]
|
||||
>>> manager = Pdf(Stream(columns=['28,67,180,230,425,475,700']), 'mexican_towns.pdf')
|
||||
>>> tables = manager.extract()
|
||||
>>> print tables['page-1']['table-1']['data']
|
||||
|
||||
.. csv-table::
|
||||
|
||||
|
||||
Reference in New Issue
Block a user