Can I use CodeQL to infer indentation level for each line of a Python script?

[dunalduck0]

I am a new user of CodeQL and I apologize in advance if my question seems naive.

I wanted to infer indentation levels for each line of a Python script, ideally including comment lines. Travelling through Python built-in AST seems a good start, but I found it doesn't model comment or blank lines. Moreover, some data structures need special handling to assess indentation increase/decrease properly. Can I get similar data from AstNode of CodeQL? If yes, can anyone please show me some sample codes or queries?

Thank you!

[hvitved]

I am quite sure the Python CodeQL extractor dismisses token information, so I would not expect this to be possible. @github/codeql-python do you agree?

[tausbn]

Yeah, I don't think this is possible to reliably extract from AstNode. It might be possible get an approximation by looking at each line, and picking the smallest starting column among all AstNodes in that line, but there are a few places where this will fail:

• The location of an if statement is the same as the location of its condition (i.e. the foo in if foo: ...). This means an if statement will look like it's been indented when it in fact has not.
• Some constructs (e.g. else:) do not have AST nodes themselves, as they are simply folded into another node (in this case, the corresponding if, for, or while node). Thus, these lines likely won't have any AST nodes on them.

(These are just a few of the issues that I could think of. There may be more.)

[dunalduck0]

Thank you @hvitved and @tausbn for your answers. "else" being folded into "if" node is the same in Python built-in AST, and it makes life harder for me :(.

class ast.If(test, body, orelse)
An if statement. test holds a single node, such as a Compare node. body and orelse each hold a list of nodes.
elif clauses don’t have a special representation in the AST, but rather appear as extra If nodes within the orelse section of the previous one.

[tausbn]

Here's a thought: It should be fairly simple to get an over-approximation of of the information you want by simply scanning through each Python file and recording the indentation level of each line.

Now, this is only an over-approximation because it doesn't account for lines that appear inside of, say, triple-quoted strings that span multiple lines. However, it should be fairly easy to identify these from the AST, and use this information to filter out the indentations that don't belong. That should give you a pretty accurate result.

Incidentally, I would not use CodeQL for this purpose, as it is likely overkill for such a use case. I would suggest using something like tree-sitter-python instead (which is one of the parsers that the CodeQL Python extractor uses internally).

[dunalduck0]

"scanning through each Python file and recording the indentation level of each line" --> this would be confused by styling indentations like the line for "arg3, arg4", right?

def foo():
    result = func_with_many_args(arg1, arg2,
                                 arg3, arg4)

I will take a look into tree-sitter-python. Thank you for pointing it out.

[tausbn]

"scanning through each Python file and recording the indentation level of each line" --> this would be confused by styling indentations like the line for "arg3, arg4", right?

def foo():
    result = func_with_many_args(arg1, arg2,
                                 arg3, arg4)

Yeah, this is another case where you would need to explicitly filter this line out (which is slightly awkward to detect, since you don't want to filter out the line where the expression starts, only the subsequent lines).

May I ask what your use case is for this? Is it the actual indentation level for each line that you care about, or would knowing the "nesting level" of the various AST nodes be enough?

(I'm wondering if this is a case of "I need to do X, and I can do it if I have Y", where in fact there's a third "Z" option that would also be sufficient to do "X", but which might be easier to implement.)

[dunalduck0]

I want to build a model with Python code as training data. I wanted to replace tabs with tokens such as TAB1, TAB2, ... etc. where TABn replaces n successive tabs. In this example, I hope to detect line1 needs no replacement as there is no tab at the beginning. For line2, line3 and line4, replace the 1st tab (or 4-spaces) with TAB1, leaving remaining spaces in line3 untouched. Finally, for line4, replace the first two tabs (or 4-spaces) with TAB2. In other words, TAB1 is indentation level 1, TAB2 is indentation level 2, ..., etc. Does it make sense to you?

line1: def foo():
line2:    result = func_with_many_args(arg1, arg2,
line3:                                 arg3, arg4)
line4:    for i in result:
line5:        print(i)