Rewrite Variable Logic From Scratch

[omniomi]

There are multiple issues with variables as they stand now in the grammar some of which have been reported and some of which haven't:

• Punctuation in variables with properties and methods isn't scoped correctly: The "scope" named "variable.other.member" includes the dot separator #131
• The sigil for a sub-expression is not handled the same way as the sigil on a variable: subexpression marker ($ in '$(') is in scope 'punctuation.definition.variable' #132
• Some language variables like $_, $$, $? are treated differently depending on context and are currently scoped incorrectly: $_ scoped differently between inside and outside string.quoted.double #133
• Properties and method highlighting breaks when the variable has an array position specified: sub members (properties/methods) of variables appear different behind an array reference #127
• Multiple legal characters break variable highlighting ($Var?Iable), and special automatic variables like $<drive letter>:folder don't highlight correctly: Handle special characters in variable names. #49

There are also questions about the correct scope of properties and methods (mentioned #131) and questions about to what extent the grammar should be concerned with user expectation/"most themes" vs either correctness (subjective) or consistency (with which languages?) (#130 and #129).

Background

The following cases need to be covered with as little repetition / complexity as possible:

• Regular variables with and without scopes, properties, array position, methods, etc.
• Variables contained within strings.
• Variables contained within sub-expressions with and without scopes, properties, array position, methods, etc.
• Variables in and out of strings and sub-expressions when using the ${Variable} format again with and without scopes, properties, array position, methods, etc.

The following gotchas need to be taken in to account:

• Which special characters are legal within ${Variable} syntax vs $Variable syntax.
• International characters (\p{L} vs [A-z]`).
• Access of variables through splatting where the sigil is @.
• Separate scoping for constants, automatic variables, language variables, etc.
• $$ where "$$Hello" needs to show $$ as the variable and Hello as part of the double quoted string and other similar oddities.
• Lots of others...

Planned Format/Rules

??? tbd

Re: Scope selection -> See #138

[msftrncs]

Is there a means in the grammar system to be able to specify an optional repository object as needing to immediately follow another object? (similar to "include" but working after the "end" match? I think such a means would simplify trying to catch everywhere that a property/method can be used. Obviously the match (or begin) regex would need to start with '\G' (or it would have to be assumed).

[omniomi]

@msftrncs sadly no. tmLanguage is incredibly limiting. You effectively have to use non-capturing lookheads and lookbehinds or wrangle nested includes. There's a slightly more powerful format based on the tmLanguage system called .sublime-syntax that has better context control but VS Code doesn't support it.

[msftrncs]

I mentioned elsewhere that I had improved matching for ${} variables. I think this regex segment would work for the inner part of the variable name: (note it forces the previous section possessive)

+([^`{}]|`[`{}]?)+

instead of:

([^}]*[^}`])

or combine a couple patterns in to one for: (not including optional property)

(?i:(\$)(\{)(?:(global|local|private|script|using|workflow):|((?:\p{L}|\d|_)+:)|:)?+([^`{}]|`[`{}]?)+(\}))

PowerShell appears to accept any and all backticks, translating those that are valid to their corresponding character including linefeed and tabs, in addition to the allowing the { and }, throwing the unmatched ones away. I think it would even be possible to scope the escapes. The only quirky case I am not sure about is that ${`} is an empty name and not accepted. Also, for some reason the original matching was accepting ${env:} which is not valid, so I added the possessive qualifier to the drive portion, which also fixes $env:, then also adding |: so that a blank drive is allowed, but a $: or ${:} is not,

None of this handles the :: static member

This was all tested with VS Code against the 'TheBigTestFile'.

[stoinov]

Just to note that at the moment splatting is incorrectly highlighted when inside quotes:
"Some text with @splatting"
This should be just a normal quoted text instead of variable looking one.