Tokenizing the characters in text widget slows the program

Question

When I insert the contents of some python script to the text widget, the contents can be highlighted quickly. Also after inserting the contents, new insertions that typed manually can be highlighted too. However after inserting the contents of some python script to the text widget, the new insertions are inserted quickly but highlighted with delays.

What should I do to solve this problem?

Thanks in advance.

Codes:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
if sys.version_info.major == 2:
    exit()
elif sys.version_info.major == 3:
    import io
    import keyword
    import builtins
    import tokenize
    import threading
    import tkinter as tk


root = tk.Tk()
text = tk.Text(master=root, fg="white", bg="black", font="TkDefaultFont 10")
text.pack(fill="both", expand=True)

count = 0


def colorize(*args):
    global count
    row1, col1 = args[0].start
    start = str(row1) + "." + str(col1)
    row2, col2 = args[0].end
    end = str(row2) + "." + str(col2)
    text.tag_add(str(count), start, end)
    try:
        text.tag_config(str(count), foreground=args[1], font=args[2])
    except IndexError:
        text.tag_config(str(count), foreground=args[1])
    count += 1


def search():
    while True:
        try:
            for i in tokenize.tokenize(io.BytesIO(text.get("1.0", "end").encode("utf-8")).readline):
                if i.type == 1:
                    if i.string in keyword.kwlist:
                        colorize(i, "orange")
                    elif i.string in dir(builtins):
                        colorize(i, "blue")
                    else:
                        colorize(i, "white")
                elif i.type == 2:
                    colorize(i, "cyan")
                elif i.type == 3:
                    colorize(i, "purple")
                elif i.type == 53:
                    if i.string == "," or i.string == "." or i.string == ":":
                        colorize(i, "orange")
                    elif i.string == "(" or i.string == ")" or i.string == "[" \
                            or i.string == "]" or i.string == "{" or i.string == "}":
                        colorize(i, "darkred")
                    else:
                        colorize(i, "green")
                elif i.type == 57:
                    colorize(i, "grey", "TkDefaultFont 10 italic")
        except tokenize.TokenError:
            pass


thread = threading.Thread(target=search)
thread.daemon = True
thread.start()
thread.join(1)
root.mainloop()

dildeolupbiten · Accepted Answer

The problem was solved.

The threading module was removed from the codes. Now, the parsing operations are done with key binding (""). Therefore the while loop that was in search() function was removed too.

The colorize(*args) function was changed to:

def colorize(*args):
    global count
    row = text.index("insert").split(".")[0]
    col1 = args[0].start[-1]
    start = row + "." + str(col1)
    col2 = args[0].end[-1]
    end = row + "." + str(col2)
    text.tag_add(str(count), start, end)
    try:
        text.tag_config(str(count), foreground=args[1], font=args[2])
    except IndexError:
        text.tag_config(str(count), foreground=args[1])
    count += 1

According to this function, the row variable is set to the insert position to stop repeating colorizing the whole content. So only last rows are need to be scanned and colorized.

Two parsing methods are defined for different cases.

The first parser is activated when the insert position is at the last row. The first parser is not changed at the core between the search() method of my previous post. The second parser is activated when one wants to paste copied codes in widget.

Here is the second parser method:

keysym = set()


def parser_2(*args):
    global keysym
    keysym = set()
    for i in range(int(args[0])):
        text.mark_set("insert", "{}.{}".format(i + 1, args[1]))
        parser_1(text.get("{}.0".format(i + 1), "{}.0".format(i + 2)))
        text.mark_set("insert", "{}.{}".format(args[0], args[1]))

After copied codes were inserted to the widget, the insert position changes as can be expected. So if we visit all rows either directly or reversely in a for loop and set the mark of all these visited rows and call the parser_1(*args) function with arguments (arguments are visited rows and column of insert position) and then set the mark to the normal insert position again, the copied content can be colorized for once. This is a shortcut function of coloring when user types ctrl+v or when user wants to use a right click menu to paste codes into widget.

The last function (select_parser(event)) is for selecting the parsers according to different cases.

def select_parser(event):
    row, col = text.index("insert").split(".")
    if event.keysym == "Control_L":
        keysym.add(event.keysym)
    elif event.keysym == "v" or event.keysym == "V":
        keysym.add(event.keysym)
        if "Control_L" in keysym:
            parser_2(row, col)
    elif event.keysym == "Control_R":
        keysym.add(event.keysym)
        if "v" in keysym or "V" in keysym:
            parser_2(row, col)
    else:
        parser_1(text.get("{}.0".format(row), "{}.0".format(int(row) + 1)))

One case is for when user types "Control_L". If user types it, it is added into keysym set which was defined in global.

The other case is when user types "v" or "V". If user types one of these, the event.keysym is added into keysym set too. And there's a different condition defined under this case that checks whether "Control_L" is in keysym set or not. If it is in, the second parse method is called. And the keysym is redefined in the second parser method.

The another case is when user types "Control_R". If user types it, it is added into keysym set too. There's a another condition defined in this that checks whether "v" or "V" is in keysym set or not. (When we type "Control_R + v", first "v" is added into keysym set but when we type "Control_L + v", first "Control_L" is added into keysym set.) If "v" or "V" is in keysym set, the second parse method is called. And the keysym is redefined in the second parser method.

And the last case is when user types different from the above keys. In this case, the first parser method is called.

Codes:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
if sys.version_info.major == 2:
    exit()
elif sys.version_info.major == 3:
    import io
    import keyword
    import builtins
    import tokenize
    import tkinter as tk


root = tk.Tk()
text = tk.Text(master=root, fg="white", bg="black", font="TkDefaultFont 10")
text.pack(fill="both", expand=True)

count = 0


def colorize(*args):
    global count
    row = text.index("insert").split(".")[0]
    col1 = args[0].start[-1]
    start = row + "." + str(col1)
    col2 = args[0].end[-1]
    end = row + "." + str(col2)
    text.tag_add(str(count), start, end)
    try:
        text.tag_config(str(count), foreground=args[1], font=args[2])
    except IndexError:
        text.tag_config(str(count), foreground=args[1])
    count += 1


def parser_1(*args):
    try:
        for i in tokenize.tokenize(io.BytesIO(
                args[0].encode("utf-8")).readline):
            if i.type == 1:
                if i.string in keyword.kwlist:
                    colorize(i, "orange")
                elif i.string in dir(builtins):
                    colorize(i, "blue")
                else:
                    colorize(i, "white")
            elif i.type == 2:
                colorize(i, "cyan")
            elif i.type == 3:
                colorize(i, "purple")
            elif i.type == 53:
                if i.string == "," or i.string == "." or i.string == ":":
                    colorize(i, "orange")
                elif i.string == "(" or i.string == ")" or i.string == "[" \
                        or i.string == "]" or i.string == "{" or i.string == "}":
                    colorize(i, "darkred")
                else:
                    colorize(i, "green")
            elif i.type == 57:
                colorize(i, "grey", "TkDefaultFont 10 italic")
    except tokenize.TokenError:
        pass


keysym = set()


def parser_2(*args):
    global keysym
    keysym = set()
    for i in range(int(args[0])):
        text.mark_set("insert", "{}.{}".format(i + 1, args[1]))
        parser_1(text.get("{}.0".format(i + 1), "{}.0".format(i + 2)))
        text.mark_set("insert", "{}.{}".format(args[0], args[1]))


def select_parser(event):
    row, col = text.index("insert").split(".")
    if event.keysym == "Control_L":
        keysym.add(event.keysym)
    elif event.keysym == "v" or event.keysym == "V":
        keysym.add(event.keysym)
        if "Control_L" in keysym:
            parser_2(row, col)
    elif event.keysym == "Control_R":
        keysym.add(event.keysym)
        if "v" in keysym or "V" in keysym:
            parser_2(row, col)
    else:
        parser_1(text.get("{}.0".format(row), "{}.0".format(int(row) + 1)))


text.bind("", select_parser)
root.mainloop()

Tokenizing the characters in text widget slows the program

Answers (1)

Related Questions