Remplacement des caractères problématiques

Bonjour à tous,

Étant donné que le logiciel du forum n'aime pas beaucoup les caractères combinants d'Unicode ainsi que d'autres non combinants, je propose le script suivant, en Python 3, pour transformer un texte de façon à ce que le logiciel du forum ne le rejette pas. Le principe consiste à normaliser le texte (par défaut en NFC, mais l'option --normalization-form, ou -f, permet de choisir une autre forme de normalisation) puis à remplacer certains caractères problématiques par des chaînes de caractères qui passent bien. Pour l'instant, il n'y a qu'une poignée de remplacements de ce type ; si d'autres s'avéraient nécessaires, il faudrait les ajouter au tuple situé à partir de la ligne 95 :

for fromStr, toStr in (("\N{MINUS SIGN}", "-"),
                       ("\N{RIGHTWARDS ARROW}", r"$\rightarrow$"),
                       ("\N{LEFTWARDS ARROW}", r"$\leftarrow$")):

L'aide s'obtient avec l'option --help :

#! /usr/bin/env python3
# -*- coding: utf-8 -*-

# cleanup-for-forum-les-mathematiques.net --- Rewrite UTF-8 input for
#                                             <http://www.les-mathematiques.net/phorum/>
# Copyright (c) 2019, brian du forum http://www.les-mathematiques.net/
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#
# The views and conclusions contained in the software and documentation are
# those of the authors and should not be interpreted as representing official
# policies, either expressed or implied, of the
# cleanup-for-forum-les-mathematiques.net project.

import argparse
import contextlib
import locale
import unicodedata
import sys


def processCommandLine():
    params = argparse.Namespace()

    parser = argparse.ArgumentParser(
        usage="""\
%(prog)s [OPTION ...]
Make text suitable for comments on <http://www.les-mathematiques.net/>.""",
        description="""\
Apply a suitable Unicode normalization form to UTF-8 data read from INFILE,
replace unsupported characters by supported ones and write the result to
OUTFILE.

If INFILE is '-', read from the standard input. If OUTFILE is '-', write
to the standard output.""",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        # I want --help but not -h (it might be useful for something else)
        add_help=False)

    parser.add_argument('-i', '--infile', default="-", help="""\
      Input file (default is '-' and means standard input)""")
    parser.add_argument('-o', '--outfile', default="-", help="""\
      Output file (default is '-' and means standard output)""")
    parser.add_argument('-f', '--normalization-form', default="NFC", help="""\
      UTF-8 normalization form to use for the output (default: %(default)s);
      the argument is converted to upper case, then passed to
      unicodedata.normalize()
      (see <https://docs.python.org/3/library/unicodedata.html>)""")
    parser.add_argument('--help', action="help",
                        help="display this message and exit")

    params = parser.parse_args(namespace=params)

    return params


def doWork(stack):
    # See <https://stackoverflow.com/questions/23301166/context-manager-for-optionally-redirected-i-o>
    # for alternative strategies
    if params.infile == "-":
        ifile = sys.stdin
    else:
        ifile = stack.enter_context(open(params.infile, "r", encoding="utf-8"))

    if params.outfile == "-":
        ofile = sys.stdout
    else:
        ofile = stack.enter_context(open(params.outfile, "w", encoding="utf-8"))

    normForm = params.normalization_form.upper()
    s = unicodedata.normalize(normForm, ifile.read())
    # Replace characters that are unsupported on
    # <http://www.les-mathematiques.net/phorum/> and thus prevent one from
    # posting comments that contain them.
    for fromStr, toStr in (("\N{MINUS SIGN}", "-"),
                           ("\N{RIGHTWARDS ARROW}", r"$\rightarrow$"),
                           ("\N{LEFTWARDS ARROW}", r"$\leftarrow$")):
        s = s.replace(fromStr, toStr)

    ofile.write(s)


def main():
    global params

    locale.setlocale(locale.LC_ALL, '')
    params = processCommandLine()

    with contextlib.ExitStack() as stack:
        doWork(stack)

    sys.exit(0)

if __name__ == "__main__": main()

En espérant que cela servira à d'autres utilisateurs du forum.

Edit: ajout de U+2192 RIGHTWARDS ARROW (remplacé par le code LaTeX produisant « $\rightarrow$ ») et U+2190 LEFTWARDS ARROW à la liste des caractères non combinants à remplacer car non supportés par le logiciel du forum.