[RFC,libinput] dox: switch to sphinx for the user-visible documentation

Submitted by Peter Hutterer on July 24, 2018, 5:42 a.m.

Details

Message ID 20180724054250.GA27209@jelly
State Accepted
Headers show
Series "dox: switch to sphinx for the user-visible documentation" ( rev: 1 ) in Wayland

Not browsing as part of any series.

Commit Message

Peter Hutterer July 24, 2018, 5:42 a.m.
Sending this out as patch even though what really matters is the
output. Which is... here, tadaaa! 

https://people.freedesktop.org/~whot/libinput-rtd/

Basically the motivation here is to make the user-visible documentation less
awful, especially because these days, 90% of the doc needs are by end users,
not the developers. The API itself is just fine in doxygen, but for prose
doxygen's "Related Pages" features is not perfect.

So I'm wondering if using RTD-style documentation is a better option here.
This patch is a more-or-less 1:1 conversion of the hand-written
documentation to use sphinx with the RTD theme. For a good chuckle, look at
the awk/sed script. (that script would go away anway after the one-time
conversion to an .rst source format).

This is not a final version, it needs massaging of the page hierarchy, right
now everything is dumped into the sidebar which is less than ideal. And of
course there are more corner cases that need manual correction.

The doxygen documentation is still built in full - this should be reduced to
just the API documentation where applicable. Some exceptions are present
there, e.g. seats.dox is handwritten but really more of a developer topic.

Biggest issue is likely to find that magic split between user docs and
developer API and where it applies, I'm sure there are bits that should
kinda be in both. Eventually I'm thinking to put the user documentation
top-level and the API documentation in /api. Resolving the various
libinput_foo() functions to point to doxygen is not trivial - you can't link
to doxygen functions directly without knowing the hash. So either we just
point to the api/index.html or we parse the doxygen xml output somehow.

But let's be honest, there are probably less than 50 people worldwide that
need to care about libinput's API. Giving everyone who's confused about
clickpad software button behaviour a link to the API call isn't really
helpful anyway.

Anyway, right now I'm mostly looking for general comments on whether this is
a good idea or a bad one, whichever comes first. I'm not planning to switch
everything to sphinx btw, too much work to go through every function and fix
it. The developer API stays on doxygen.
---
 doc/conf.py.in    | 166 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/meson.build   | 101 ++++++++++++++++++++++++++++
 doc/sed-script.sh |  68 +++++++++++++++++++
 doc/toctree.in    |   6 ++
 4 files changed, 341 insertions(+)
 create mode 100644 doc/conf.py.in
 create mode 100755 doc/sed-script.sh
 create mode 100644 doc/toctree.in

Patch hide | download patch | download mbox

diff --git a/doc/conf.py.in b/doc/conf.py.in
new file mode 100644
index 00000000..a3b54562
--- /dev/null
+++ b/doc/conf.py.in
@@ -0,0 +1,166 @@ 
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/stable/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = '@PROJECT_NAME@'
+copyright = '2018'
+author = ''
+
+# The short X.Y version
+version = '@PROJECT_VERSION'
+# The full version, including alpha/beta/rc tags
+release = '@PROJECT_VERSION@'
+
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.mathjax',
+    'sphinx.ext.graphviz',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path .
+exclude_patterns = []
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+# html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = '@PROJECT_NAME@doc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, '@PROJECT_NAME@.tex', '@PROJECT_NAME@ Documentation',
+     'Peter Hutterer', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, '@PROJECT_NAME@', '@PROJECT_NAME@ Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, '@PROJECT_NAME@', '@PROJECT_NAME@ Documentation',
+     author, '@PROJECT_NAME@', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+# -- Extension configuration -------------------------------------------------
+
+from recommonmark.parser import CommonMarkParser
+
+source_parsers = {
+    '.md': CommonMarkParser,
+}
diff --git a/doc/meson.build b/doc/meson.build
index a42a2e0d..21d025cf 100644
--- a/doc/meson.build
+++ b/doc/meson.build
@@ -138,3 +138,104 @@  custom_target('doxygen',
 	      install : false,
 	      depends: [doxyfiles, readme],
 	      build_by_default : true)
+
+
+
+# Sphinx build
+sphinx = find_program('sphinx-build', required : false)
+if not sphinx.found()
+	error('Program "sphinx-build" not found or not executable. Try building with -Ddocumentation=false')
+endif
+
+sphinx_config = configuration_data()
+sphinx_config.set('PROJECT_NAME', meson.project_name())
+sphinx_config.set('PROJECT_VERSION', meson.project_version())
+
+sphinx_conf_py = configure_file(input : 'conf.py.in',
+				output : 'conf.py',
+				configuration : sphinx_config,
+				install : false)
+
+# because we currently autogenerate the toctree from this list, the entries
+# are without the .dox suffix - sphinx needs them without for some reason.
+# When we pull the trigger to switch to sphinx, the toctree should be
+# manually maintained (similar to page-hierarchy.dox right now)
+# and this garbage goes away
+src_sphinx = [
+	'absolute-axes',
+	'absolute-coordinate-ranges',
+	'architecture',
+	'building',
+	'button_debouncing',
+	'clickpad-softbuttons',
+	'contributing',
+	'device-configuration-via-udev',
+	'device-quirks',
+	'faqs',
+	'gestures',
+	'middle-button-emulation',
+	'normalization-of-relative-motion',
+	'palm-detection',
+	'pointer-acceleration',
+	'reporting-bugs',
+	'scrolling',
+	'seats',
+	'switches',
+	't440-support',
+	'tablet-support',
+	'tapping',
+	'test-suite',
+	'timestamps',
+	'tools',
+	'touchpad-jumping-cursors',
+	'touchpad-pressure',
+	'touchpad-jitter',
+	'touchpads',
+	'trackpoints',
+	'what-is-libinput',
+]
+
+sed_script = find_program('sed-script.sh')
+sphinx_targets = []
+sphinx_sources = []
+foreach d : src_sphinx
+	f = files('@0@.dox'.format(d))
+	sphinx_sources += [f]
+	t = custom_target(d,
+			  input: f,
+			  output: '@0@.rst'.format(d),
+			  install: false,
+			  command: [sed_script, '@OUTDIR@', '@INPUT@'],
+			 )
+	sphinx_targets += [t]
+endforeach
+
+conf_toctree = configuration_data()
+conf_toctree.set('SOURCES', '\n\t'.join(src_sphinx))
+
+toctree = configure_file(input: 'toctree.in',
+			 output: 'toctree.rst.tmp',
+			 configuration: conf_toctree,
+			 install: false)
+
+index_rst = vcs_tag(command : ['git', 'log', '-1', '--format=%h'],
+		     fallback : 'unknown',
+		     input : '../README.md',
+		     output : 'index.rst.tmp',
+		     replace_string: '__GIT_VERSION__')
+
+# combine toctree and the index page into a single rst file
+index_page = custom_target('index',
+			   input: [toctree, index_rst],
+			   output: 'index.rst',
+			   command: ['cat', toctree, index_rst],
+			   install: false,
+			   capture: true)
+
+# drop '-a' once we are happy with all this
+custom_target('sphinx',
+	      input : [ toctree, sphinx_conf_py, index_page ] + sphinx_targets + sphinx_sources,
+	      output : [ 'sphinx' ],
+	      command : [ sphinx, '-a', meson.current_build_dir(), 'sphinx' ],
+	      depends: sphinx_targets,
+	      build_by_default : true)
diff --git a/doc/sed-script.sh b/doc/sed-script.sh
new file mode 100755
index 00000000..33eff86f
--- /dev/null
+++ b/doc/sed-script.sh
@@ -0,0 +1,68 @@ 
+#!/bin/bash -e
+#
+# Helper script to convert doxygenisms to sphinx-isms
+
+outdir=$1
+fname=$2
+
+! test -z "$outdir" || exit 1
+! test -z "$fname" || exit 1
+
+file="$fname"
+outfile="$(basename --suffix='.dox' $fname).rst"
+
+# awk commands:
+# indent anything between the verbatim tags
+# indent anything between the code tags
+# add a empty line before the first list item (line starting with -)
+awk \
+    '$0 ~ /@verbatim$/ { inside=1; print; next; }
+     $0 ~ /@endverbatim/ { inside=0; }
+     inside == 1 { print "    " $0 }
+     inside == 0 { print }' $file | \
+awk \
+    '$0 ~ /@code$/ { inside=1; print; next; }
+     $0 ~ /@endcode/ { inside=0; }
+     inside == 1 { print "    " $0 }
+     inside == 0 { print }' | \
+awk \
+    '$0 ~ /@dot$/ { inside=1; print; next; }
+     $0 ~ /@enddot/ { inside=0; }
+     inside == 1 { print "    " $0 }
+     inside == 0 { print }' | \
+awk \
+    '/^-/{
+	 if (!in_list && a != "") print ""; in_list=1
+	 }
+     /^$/ {in_list=0}
+     {a=$0; print}' | \
+sed \
+  -e 's|@page \([^ ]*\) \(.*\)|.. _\1:\n\n==============================================================================\n\2\n==============================================================================|' \
+  -e 's|@section \([^ ]*\) \(.*\)|.. _\1:\n\n------------------------------------------------------------------------------\n\2\n------------------------------------------------------------------------------|' \
+  -e 's|@subsection \([^ ]*\) \(.*\)|.. _\1:\n\n..............................................................................\n\2\n..............................................................................|' \
+  -e 's|@ref \(LIBINPUT_[_[:alpha:]]\+\)|**\1**|' \
+  -e 's|@ref \(libinput_[_[:alpha:]]\+\)|**\1**|' \
+  -e 's|\(libinput_[_[:alpha:]]\+()\)|**\1**|' \
+  -e 's|<b>|**|' \
+  -e 's|</b>|**|' \
+  -e 's|\*40|\\*40|' \
+  -e 's|\*50|\\*50|' \
+  -e 's|@note|.. note::|' \
+  -e 's|@dotfile \(.*\)|.. graphviz:: \1|' \
+  -e 's|@dot|.. graphviz::\n|' \
+  -e 's|@enddot||' \
+  -e 's|@code|::\n|' \
+  -e 's|`[^`]\+|`\0|g' \
+  -e 's|[^`]\+`$|\0`|g' \
+  -e 's|@ref \([-[:alnum:]_]*\) "\(.*\)"|:ref:`\2 <\1>`|' \
+  -e 's|@ref \([-[:alnum:]_]*\)|:ref:`\1`|' \
+  -e 's|@endcode||' \
+  -e 's|@tableofcontents||' \
+  -e 's|@verbatim|::\n|' \
+  -e 's|@endverbatim||' \
+  -e 's|@image html \([^ ]*\) "\?\(.*\)"\?|.. figure:: \1\n\t:align: center\n\n\t\2|' \
+  -e 's|<a href="\(.*\)">\(.*\)</a>|`\2 <\1>`_|' \
+  -e 's|\[\(.*\)\](\(.*\))|`\1 <\2>`_|' \
+  -e '/^\*\//d' \
+  -e '/^\/\*\*$/d' \
+  > "$outdir/$outfile"
diff --git a/doc/toctree.in b/doc/toctree.in
new file mode 100644
index 00000000..fa3a4248
--- /dev/null
+++ b/doc/toctree.in
@@ -0,0 +1,6 @@ 
+.. toctree::
+	:maxdepth: 2
+	:hidden:
+
+	@SOURCES@
+

Comments

Hey,

On Tue, 24 Jul 2018 at 06:43, Peter Hutterer <peter.hutterer@who-t.net> wrote:
> Sending this out as patch even though what really matters is the
> output. Which is... here, tadaaa!
>
> https://people.freedesktop.org/~whot/libinput-rtd/
>
> Basically the motivation here is to make the user-visible documentation less
> awful, especially because these days, 90% of the doc needs are by end users,
> not the developers. The API itself is just fine in doxygen, but for prose
> doxygen's "Related Pages" features is not perfect.
>
> So I'm wondering if using RTD-style documentation is a better option here.
> This patch is a more-or-less 1:1 conversion of the hand-written
> documentation to use sphinx with the RTD theme. For a good chuckle, look at
> the awk/sed script. (that script would go away anway after the one-time
> conversion to an .rst source format).

Makes sense, and this does look really good, but ... did you remember
to run the script? The output at the URL above is full of @ref, @dot,
etc. The rest looks good though. :) I'm all for something like Sphinx
- not because I've ever used it or know any of its pros and cons, but
it seems like a more stable tooling which is also more widely used.

Cheers,
Daniel
On Tue, Jul 24, 2018 at 09:27:51AM +0100, Daniel Stone wrote:
> Hey,
> 
> On Tue, 24 Jul 2018 at 06:43, Peter Hutterer <peter.hutterer@who-t.net> wrote:
> > Sending this out as patch even though what really matters is the
> > output. Which is... here, tadaaa!
> >
> > https://people.freedesktop.org/~whot/libinput-rtd/
> >
> > Basically the motivation here is to make the user-visible documentation less
> > awful, especially because these days, 90% of the doc needs are by end users,
> > not the developers. The API itself is just fine in doxygen, but for prose
> > doxygen's "Related Pages" features is not perfect.
> >
> > So I'm wondering if using RTD-style documentation is a better option here.
> > This patch is a more-or-less 1:1 conversion of the hand-written
> > documentation to use sphinx with the RTD theme. For a good chuckle, look at
> > the awk/sed script. (that script would go away anway after the one-time
> > conversion to an .rst source format).
> 
> Makes sense, and this does look really good, but ... did you remember
> to run the script? The output at the URL above is full of @ref, @dot,
> etc. The rest looks good though. :) I'm all for something like Sphinx
> - not because I've ever used it or know any of its pros and cons, but
> it seems like a more stable tooling which is also more widely used.

oh right, the issues here are because this is the README.md which is the
only file that doesn't filter through the sed/awk script. Other pages render
correctly with the odd exceptions here and there. Doxygen allows
@ref\nfoo but my sed script doesn't handle that case correctly.

Cheers,
   Peter
On Tue, Jul 24, 2018 at 03:42:50PM +1000, Peter Hutterer wrote:
> Sending this out as patch even though what really matters is the
> output. Which is... here, tadaaa! 
> 
> https://people.freedesktop.org/~whot/libinput-rtd/

fwiw, I've played around with it a bit and updated the URL above. It now has
a proper hierarchy, more doxygen tags are being parsed.

There are still a few parsing issues that need to be fixed manually. And the
doc needs a general do-over to split it better into user docs vs developer
docs. And of course doxygen itself needs to be separated, generated, and
link to the sphinx docs. But otherwise this looks quite nice IMO.

Cheers,
   Peter
Hi,

On Thu, 26 Jul 2018 at 02:53, Peter Hutterer <peter.hutterer@who-t.net> wrote:
> On Tue, Jul 24, 2018 at 03:42:50PM +1000, Peter Hutterer wrote:
> > Sending this out as patch even though what really matters is the
> > output. Which is... here, tadaaa!
> >
> > https://people.freedesktop.org/~whot/libinput-rtd/
>
> fwiw, I've played around with it a bit and updated the URL above. It now has
> a proper hierarchy, more doxygen tags are being parsed.
>
> There are still a few parsing issues that need to be fixed manually. And the
> doc needs a general do-over to split it better into user docs vs developer
> docs. And of course doxygen itself needs to be separated, generated, and
> link to the sphinx docs. But otherwise this looks quite nice IMO.

Oh, that is really nice. I like that it actually has a navigable
hierarchy now. That was always my biggest issue with Doxygen. Given
that the kernel also uses Sphinx for the DRM/KMS documentation in
particular, it's probably good for us to be using it as well.

If it counts for anything at all:
Acked-by: Daniel Stone <daniels@collabora.com>

Cheers,
Daniel

PS: You might want to disable syntax highlighting on the MIT license;
also 'generated by from git commit' at the very bottom.