Dramatic Slowdown of LiveCode Create Native: Classic Mode

[GS.Sunatori]

I use LiveCode Create Native (macOS) in Classic Mode only for now. I have a large stack which has slowly but steadily grown to 8 250 cards in 762 MB over the last 15 years or so (since Runtime Revolution). For the last 3 months, I have not changed my MacBook Pro computer, macOS, LiveCode version or any of my handlers in scripts.

Last week, all of a sudden, one of my handlers (function MakeCount, about 150 lines of code as shown attached below) which had been taking 2 to 4 seconds to complete became extremely slow to take 3 minutes. The last change of the handler was definitely more than 3 months ago. It uses several “repeat for each line tCd in tIndex” kind of loops, but never “repeat with i = 1 to XX” which would be disastrously slow.

So, my question is whether there is some kind of threshold of the number of cards or stack size in MB or something, over which some operations become extremely slow like a step function (reminiscent of OS’s virtual memory)??

---

function MakeCount
put “!HyperInfo” into tStack
put fld “Index” into tIndex
put fld “Contents” into tContents
put
fld “Script” of cd “{List:Others}” & return &
fld “Script” of cd “{List:Sci-Fi Central}” & return &
fld “Script” of cd “{List:Tubi Sci-Fi}” & return &
fld “Script” of cd “{List:Watch Now - Sci-Fi & Fantasy}” into tSciFi
put fld “Script” of cd “1996-10-10|Sunatori’s Coined Neologism/Néologisme” into tNeologism
put fld “Script” of cd “2016-10-10|Sunatori’s Coined Text Replacements” into tReplacements
put fld “Script” of cd “2017-06-09|Innovation150” into tInnovation
put fld “Script” of cd “Opinions/Des avis” into tOpinions
put fld “Script” of cd “Portfolio/Portefeuille” into tPortfolio
filter lines of tPortfolio with pattern “

”

put 0 into tApple
put 0 into tLiveCode
repeat for each line tLine in tContents
if (“~/Dropbox/” is in tLine) then
put URL (“binfile:” & tLine) into tTxts
if (tTxts <> “”) then
if (“/bug_reporter_” is in tLine) then
repeat for each line tLinf in tTxts
if (“<span” & quote & “>” is in tLinf) then
put tApple + 1 into tApple
end if
end repeat
put “Apple bugs” & tab & StripEmpty(tApple) & return after tReturn
else if (“/LiveCode Quality Control Center” is in tLine) then
repeat for each line tLinf in tTxts
if (“<a href=” & quote & “show_bug.cgi?id=” is in tLinf) then
put tLiveCode + 1 into tLiveCode
end if
end repeat
put (tLiveCode / 2) into tLiveCode
put “LiveCode bugs” & tab & StripEmpty(tLiveCode) & return after tReturn
end if
end if
end if
end repeat
put “SUM(Apple bugs + LiveCode bugs)” & tab & StripEmpty(tApple + tLiveCode) & return & return after tReturn

put 0 into i
set itemDel to quote
repeat for each line tLine in tInnovation
if (“$<H3 ID=” is in (“$” & tLine)) then
put item 2 of tLine into tLinf
if (tLinf <> “”) and (tLinf <> “XXX”) and ((“#” & tLinf & quote & “>”) is not in tPortfolio) then
put “• Missing in Portfolio: #” & tLinf & return after tCheck
end if
end if
if (“<IMG SRC=” & quote & “…/” is not in tLind) and (“<IMG SRC=” & quote & “…/” is in tLine) and (“_A” is not in tLine) then
put i + 1 into i
end if
put tLine into tLind
end repeat
repeat for each line tLine in tPortfolio
if (“

<A HREF=” is in tLine) and (“#” is in tLine) then
set itemDel to quote
put item 2 of tLine into tLinf
set itemDel to “#”
put item 2 of tLinf into tLinf
if (tLinf <> “”) and (tLinf <> “XXX”) and ((“

”) is not in tInnovation) then
put “• Missing in Innovation: ID=” & quote & tLinf & quote & return after tCheck
end if
end if
end repeat
put i into tTotal
put “Innovation150” & tab & StripEmpty(i) & return after tReturn
put tOpinions & return & tPortfolio into tInnovation

put 0 into tPotential
put 0 into tArts
put 0 into tWonders
put 0 into tConcepts
repeat for each line tCd in tIndex
put fld “Title” of cd tCd into tTitle
put char 1 to 4 of tTitle into tYYYY
put line 1 of fld “Contents” of cd tCd into tFirst
if (tYYYY is an integer) and (“Sunatori’s” is in tTitle) then
repeat for each line tContent in tContents
if (“~/Dropbox/” is not in tContent) and (char 1 of tContent <> “-”) then
if (“X=Potential” is in tFirst) then
put tPotential + 1 into tPotential
exit repeat
else if (“X=Arts” is in tFirst) then
put tArts + 1 into tArts
exit repeat
else if (“X=Wonders” is in tFirst) then
put tWonders + 1 into tWonders
exit repeat
else if (“X=Concepts” is in tFirst) then
put tConcepts + 1 into tConcepts
exit repeat
end if
end if
end repeat
end if
end repeat
put
“X=Potential” & tab & tPotential & return &
“X=Arts” & tab & tArts & return &
“X=Wonders” & tab & tWonders & return &
“X=Concepts” & tab & tConcepts & return after tReturn
put tTotal + tPotential + tArts + tWonders + tConcepts into tTotal
put “SUM(Innovation150 + X=Potential + X=Arts + X=Wonders + X=Concepts)” & tab & StripEmpty(tTotal) & return & return after tReturn

put 0 into i
repeat for each line tLine in tReplacements
if (“$

” is in (“$” & tLine)) and ((“[” is not in tLine) and (“]” is not in tLine)) then
put i + 1 into i
end if
end repeat
put “Replacements” & tab & StripEmpty(i) & return after tReturn

put 0 into i
repeat for each line tLine in tNeologism
if (“$

” is in (“$” & tLine)) and ((“[” is not in tLine) and (“]” is not in tLine)) then
put i + 1 into i
end if
end repeat
put “Neologism” & tab & StripEmpty(i) & return after tReturn

put 0 into i
set itemDel to tab
repeat for each line tLine in tSciFi
put item 1 of tLine into tLine1
put item 2 of tLine into tLine2
if (“A” is in tLine2) or (“B” is in tLine2) or (“C” is in tLine2) then
if (tLine1 is not in tText) then
put i + 1 into i
end if
put tLine & return after tText
end if
end repeat
put “Sci-Fi” & tab & StripEmpty(i) & return after tReturn

put 0 into i
repeat for each line tLine in tIndex
if (“§” is not in tLine) and (“|” is in tLine) and (char 1 to 4 of tLine is an integer) then
put i + 1 into i
end if
end repeat
put “Weblog” & tab & StripEmpty(i) & return after tReturn

if (gDoIt = true) and (tCheck <> “”) then
answer tCheck – as sheet
end if
return tReturn
end MakeCount

[adamLC]

Hi @GS.Sunatori thanks for the post!

It’s possible that a unicode character has slipped into one of the fields. You can experiment using:

put textDecode(textEncode(<text>, "native"), "native") into <var>

for the fields you’re copying into variables and seeing which one speeds it up again. The above will turn a unicode string into a native string, replacing unicode characters with ‘?’

[GS.Sunatori]

Hej Adam,

You are my lifesaver!!! The extreme slowdown was indeed caused by several Unicode characters that had slipped into some of the fields!!

In order to detect where these Unicode characters are, I coded a mouseUp handler with the following line that you gave me.

put textDecode(textEncode(<text>, "native"), "native") into <var>

Out of 8 256 cards, there were various Unicode characters (smileys) in text fields of 28 cards, due to Copy & Paste of webpage text from Safari.app (macOS) browser. The characters were replaced with “?” by your textDecode(textEncode()) code. One of the fields has 794 851 chars, so the task was tedious, but I finally managed to manually eliminate the Unicode characters (smileys)!

Now, my function MakeCount runs in 3 seconds instead of 3 minutes!!

Furthermore, I wrote a routine that checks for the existence of Unicode char every time text in a field is changed.

This Unicode oddity should be documented somewhere. Do you happen to know if there is any plan to make Unicode text run almost as fast as ASCII text?

Thanks again, Adam!!!

[adamLC]

Glad that resolved things! I’ll pass that on to make sure it is documented somewhere. Unicode characters take a lot longer to process and it is something we may be able to improve in future but we aren’t working on this at the moment.