Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Dec 07 06:04
    gfmcknight synchronize #695
  • Dec 07 01:05

    slozier on master

    Implement error handling for (r… (compare)

  • Dec 07 01:05
    slozier closed #694
  • Dec 06 22:56
    BCSharp synchronize #694
  • Dec 05 21:48

    slozier on master

    Normalize access of Bytes/ByteA… (compare)

  • Dec 05 21:48
    slozier closed #693
  • Dec 05 05:13
    gfmcknight opened #695
  • Dec 05 05:12
    BCSharp synchronize #694
  • Dec 05 01:19
    BCSharp opened #694
  • Dec 04 14:20
    slozier synchronize #693
  • Dec 04 00:36
    slozier synchronize #693
  • Dec 03 22:12
    slozier synchronize #693
  • Dec 03 20:53
    slozier opened #693
  • Dec 03 00:54

    slozier on master

    Clean up escape_encode (#692) (compare)

  • Dec 03 00:54
    slozier closed #692
  • Dec 02 23:39
    BCSharp opened #692
  • Dec 02 22:26

    slozier on master

    Fix codecs.escape_decode (#690)… (compare)

  • Dec 02 22:26
    slozier closed #690
  • Dec 02 21:56
    BCSharp synchronize #690
  • Dec 02 20:33
    BCSharp commented #690
slozier
@slozier
SetSearchPaths will set the sys.path value which is used to find files on import
slozier
@slozier
@BCSharp Submitted the file to MS.
Andrew Bastien
@volundmush
thanks. that's what I was looking for. hooray
Andrew Bastien
@volundmush
hrrrrrm
so in the case of System.Collections.Generics, would having IronPython use those WHEN FEASIBLE, such as List[int] would it be more efficient to use those instead of Python's own collections like list() and dict() due to the static typing?
Andrew Bastien
@volundmush
I mean sometimes you wanna be able to stuff anything in that dictionary.... but maybe sometimes it's just Dictionary<string, string>
tinyg
@tinyg
Generally they’re interchangeable - we haven’t seen any performance issues either way (but there might be)
Andrew Bastien
@volundmush
how would I do it, though? a dictionary at least... Dictionary[str][str] ?
tinyg
@tinyg
Dictionary[str,str]
Performance wise we generally find python to be quite a bit slower than straight c# (a bit hard to put a figure on as it depends what you’re doing)
tinyg
@tinyg
The other thing to consider is that each time you load a script (from python text or a file), you lose a bit of memory for good (it creates an in-memory assembly which cannot be released until the application closes) - it’s a tiny amount but, if you’re in a server environment (asp.net / wcf service etc) where you might be running for days or weeks then you need to take it into consideration
In our server code we load all the python customisation at startup and have them register a method / class with the C# code and then we just call the registered code over and over - no leaks this way
Not trying to scare you off - our users love having python available and it’s worked great for us - just trying to give you some of the pitfalls I wish I knew about before we started 🙂
slozier
@slozier
@tinyg Hmm, you have an example where you can't reclaim all memory?
tinyg
@tinyg
Pretty sure I can recreate one - it’s generally only tiny (obviously depends on the compiled size of the script )
When we started using IronPython we just loaded the script every time we wanted to run it but found that after 3 or 4 days we'd gradually have leaked enough that the application would be restarted by IIS (or fall over if it was running as a windows service)
tinyg
@tinyg
I remember seeing the "assemblies can't be unloaded" thing somewhere when we were trying to fix it - that when we ended up changing to loading things on startup and just calling back into the code.
tinyg
@tinyg
I had a look to see if I could find the reference that was talking about not being able to unload assemblies and found this https://docs.microsoft.com/en-us/previous-versions/dotnet/netframework-4.0/dd554932(v=vs.100)?redirectedfrom=MSDN instead
I assume this is what IronPython is using so assemblies should be cleaned up?
If so then the implication is that we're holding onto a reference somewhere that's holding the assembly open - now I'm intrigued 🙂 so I'll try to reproduce
tinyg
@tinyg

OK - here's my attempt at a simple simulation:

Process currentProcess = System.Diagnostics.Process.GetCurrentProcess();

void DumpCurrentMemory()
{
    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();

    using (var counter = new PerformanceCounter("Process", "Private Bytes", currentProcess.ProcessName))
    {
        counter.RawValue.Dump();
    }
}

StringBuilder b = new StringBuilder();
var rnd = new Random();
for (int i = 0; i < 4_000; i++)
{
    b.AppendLine($"a = {rnd.Next()}");
}
var code = b.ToString();

var _engine = IronPython.Hosting.Python.CreateEngine();
var scope = _engine.CreateScope();

for (int i = 0; i < 2000; i++)
{
    var source = _engine.CreateScriptSourceFromString(code, SourceCodeKind.AutoDetect);
    source.Execute(scope);
    scope.RemoveVariable("a");

    DumpCurrentMemory();
}

DumpCurrentMemory();

(running in LinqPad)

Memory looks like:

55574528
56381440
56381440
56381440
56381440
56381440
56381440
56385536
56385536
56385536
56389632
56389632
56389632
56389632
56389632
56442880
56442880
56442880
56442880
57380864
57380864
57380864
57380864
57380864
57434112
57434112
57434112
57434112
57434112

(so gradually increasing).

Interestingly - it doesn't grow every run so either that's just .NET's allocation pattern or there's something else at play.

tinyg
@tinyg
Attaching the dotMemory profiler, I can see managed memory is nice and stable but there's a small growth in unmanaged memory over time (in my run 77MB near the start vs 79MB near the end)
tinyg
@tinyg
Alas I don't have access to ANTS anymore which gives a better breakdown of where unmanaged memory came from....
slozier
@slozier
@tinyg Thanks for trying to repro. I let it run for a while and it seems to go back down. However, if you change your a = {rnd.Next()} to class test(object): a = {rnd.Next()} then there is a noticeable increase.
Andrew Bastien
@volundmush
Man being able to use c# for the robust nitty gritty of the engine and python for game logic and extendability is really just spectacular
Alex Earl
@slide
:+1: for sure
Andrew Bastien
@volundmush
Im trying to make a mud server. So text based, multiplayer rpg played via command input
But im more msking an engine/platform than one specific game
slozier
@slozier
fun times, I used to dev a mud server :smile:
tinyg
@tinyg
@slozier that makes sense reading the article about the collectable assemblies - I guess it’s really hard to track what might hold onto the type reference. I’ve always been surprised that there was no dispose on the scope objects - knowing when to let go of the complex graph of types and stuff that has been stuffed in there must be tricky. I wonder if you did “del test” whether that would fix it somehow
I can try running your example through dotMemory and see what’s holding a reference to that class if that helps?
Иван Сердюк
@oceanfish81_twitter
https://www.youtube.com/watch?v=TgUYcZV-foM - a good guide for designing async/await support, for the DLR, in IronPython
so it would be more syntax support, as for CPython 3.5+
but the underlying concurrency and parallelism approaches would surely be different - those are not intended to be similar
Alex Earl
@slide
@slozier I feel like I am cheating on IronPython...but PythonNET is pretty cool ;)
slozier
@slozier
@slide how's the perf on it? We want to use some python3+numpy+pandas
Alex Earl
@slide
@slozier I haven't done any performance testing, but it seems pretty slick. I guess it would depend on how over you need to cross over between C# and Python
I'm submitting some PR's to do some things that I do in IronPython now that are not available in PythonNET yet (e.g., __len__ for .NET collections)
slozier
@slozier
Worst case we'd be running an independent process with a socket or something for interop. I figure Python.NET would simplify the data exchange.
hivemall
@hivemall
i have a large c# project in VS, we call multiple .py scripts by starting a thread, creating the ironpython engine, then compiling the script and calling .Execute.
  1. what options do i have to reduce the startup time, assuming the script itself (lots of imports) is the bottleneck (not the thread/engine creation). How can i precompile the scripts to speed up execution to mitigate the cumbersome reimporting?
    import clr
    clr.CompileModules("foo.dll", "foo.py")
    clr.AddReference("foo.dll")
    import foo ...
    x = 3
    y = x + 1
    is above still the way to go? (https://blogs.msdn.microsoft.com/srivatsn/2008/08/06/static-compilation-of-ironpython-scripts/) or what are my (best) options here? (or should i use pyc somehow?)
  2. how to pass (if possible) variables in/out of these scripts eg set x=4 and retrieve back y inside the thread
hivemall
@hivemall
anyone here :D ?
slozier
@slozier
@hivemall Yes, clr.CompileModules is the way to go. Although if you re-use the same engine importing the second time the script it called should be cheaper. As for getting variables in and out of the scripts you can use scopes:
Python
ScriptScope scope = engine.CreateScope(new Dictionary<string, object> { { "x", 4} });
source.Execute(scope);
scope.TryGetVariable("y", out int y);
Иван Сердюк
@oceanfish81_twitter
yo people
who is able to give talks/run demos and workshops, on May 22-23?
CFP is open
Graham McKnight
@gfmcknight
Xbvv. Bb
Alex Earl
@slide
@slozier you helping the folks of PythonNET with joining the dotnet foundation?