Parsing structured text¶
Parsing in general is of interest. One reason for delay in starting this project has been the postponing of the choice between using Python’s built-in parsing modules, or more general parsing components. Although I have now decided to start with Python’s built in parser I remain open to use of other parsers as well. Not much point in having a “structured text editor” which cannot handle structured text such as other langages, marked-up texts, config files, log files, and so on.
Parsing is more than merely interesting, it is required as long as Text Storage is plain.
I also intend to rely as heavily as possible on others’ code in developing pym, and can only expect that to arrive hither as plain text.
Parsing Python¶
Parsing produces Abstract Syntax Trees, which have a reasonable amount of related documentation. Green Tree Snakes was a proximate spur to this project, and the other links in this section were found thence.
- Green Tree Snakes - the missing Python AST docs (and some Reddit comments on them).
- The Design of CPython’s Compiler lays out rationales behind some choices in the design of the compiler, and introduces a developer to the code supporting it. For pym it is particularly relevant in introducing ASDL and SPARK.
- Python internals: Working with Python ASTs
Parsing non-Python¶
Other parsers, written in Python but parsing other structures:
- Allen Shorts’ parsley, which follows on from his Pymeta, and which he introduced at Pycon 2013.
- Ned Batchelder’s Python parsing tools is a page covering many other parsers.
- Juancarlo Añez’ Grako is “a tool that takes grammars in a variation of EBNF as input, and outputs memoizing (Packrat) PEG parsers in Python”.
Meta-Compiling¶
HackerNews told me about META II, which is able to “reproduce its own code from a description”:
At the end of these steps you should be able to:
- Define a compiler description that ejects your favorite language code
- Paste it into the Input window
- Have one of the compilers on these web pages generate the translation to your favorite language
- Cut the code generated in your favorite language from the Output window
- Continue compiler and metacompiler development in your favorite language
Unfortunately I have other work to get back to now, but the tutorial is well-written, enthusiastic about the meta-compiler, and when the tutorials don’t work, it’s because they are meant not to, in the next paragraph.
Excellent site, I hope to get more time on that tool.
ASTs¶
Some ideas around ASTs
- AST Transformation Hooks for Domain Specific Languages
- possible meta coding in Python
- Reddit on pydit
- Reddit on ASTs (in /r/python)
- NuPIC implements cortical algorithms
Higher level structures¶
If the text to be editted is a program then parsing can be extended to include information on symbol tables, scope, control flow, tests, patterns, clichés, …
If the text to be editted has meaning then parsing can be extended to include context, connections, importance, definition, relevance, …