Improve lexer embedding (markdown -> html)

[fischerling]

Hey @orbitalquark, thanks for your work on scintillua.

Unfortunatly, I am really bad with LPEG and therefore reaching out for help.

I would like to have better end rules for embedded lexers, especially latex.

For example the markdown lexer uses a blank line as termination for its embedded html lexer.

-- Embedded HTML.
local html = lexer.load('html')
local start_rule = lexer.starts_line(P(' ')^-3) * #P('<') * html:get_rule('tag') -- P(' ')^4 starts code_line
local end_rule = #blank_line * ws
lex:embed(html, start_rule, end_rule)

It would be really nice if we had a proper way to detect the end of the environment using the embeded lexer (for example the closing html tag or similar the end of a latex environment).

I think the html embedding in markdown can be improved by using html.embed_end_tag.

-- Embedded HTML.
local html = lexer.load('html')
local start_rule = lexer.starts_line(P(' ')^-3) * #P('<') * html:get_rule('tag') -- P(' ')^4 starts code_line
local end_rule = html.embed_end_tag
lex:embed(html, start_rule, end_rule)

I think is is correct but I am not really sure.

Is there any good way to detect the end of an latex environment (\end{...})?

[orbitalquark]

html.embed_end_tag matches a single closing tag, any closing tag. An HTML sequence like <div><p>Something</p></div> will stop the HTML lexer after the </p>. The HTML lexer does not keep track of open and closed HTML tags, so the Markdown lexer cannot rely on it to provide an end point.

I think for LaTeX, we have the same problem of balanced begin an end tags. I don't know of a good way to detect them. Then again, I am not very familiar with LaTeX, so I could be wrong.

[fischerling]

html.embed_end_tag matches a single closing tag, any closing tag. An HTML sequence like <div><p>Something</p></div> will stop the HTML lexer after the </p>. The HTML lexer does not keep track of open and closed HTML tags, so the Markdown lexer cannot rely on it to provide an end point.

I see. For my change I had multi-line html in mind.

<div>

<style type="text/css">
margin: 50;
</style>
</div>

This example is not correctly highlighted because the empty line terminates the html embedding.

I think for LaTeX, we have the same problem of balanced begin an end tags. I don't know of a good way to detect them. Then again, I am not very familiar with LaTeX, so I could be wrong.

You are right latex environments work the same as html tags.
Can we pass data from the starting rule to the end rule to hint the required matching tag/environment end?

[orbitalquark]

No we cannot, because a lexer is not passed the entire document to lex every time (for performance reasons). Tracking state is very difficult and error-prone. Consider the following scenario:

1. Add an open tag to the end of a document. The lexer should record it and start looking for an end tag.
2. Scroll back up to the top of the document and insert something.
3. The lexer will continue to look for the end tag and highlight things incorrectly.
4. Since the lexer is only passed a chunk of text to lex, it doesn't know the line number its on, or that it has to reset prior saved state.