These are chat archives for spyder-ide/public

13th
Jan 2019
Luca Lanzilao
@lanzi94_twitter
Jan 13 15:30
Hi everyone. I'm currently trying to save the data stored in 'variable explorer' by using ''save data as'', which normally creates a .spydata file. The problem is that no output is generated and no error message is displayed. How can I solve the problem? If this method doesn't work, how can I save my data? The data consist of several arrays, matrices and dictionaries. Thank you in advance for the answer.
CAM Gerlach
@CAM-Gerlach
Jan 13 16:54
@lanzi94_twitter Sorry you're having the issue, and its difficult to debug from here since there's no output of any kind. What version of Spyder, spyder-kernels, Python, and your operating system are you running? Does the path/name you're trying to save to have any special or non-ASCII characters? Is it writable? Can you try opening a new Console, creating e.g. a simple list and trying to save your environment to your home directory, to try to isolate other causes? If that works but saving your current env doesn't can you isolate what might be the issue by using del on each object in turn from your env and seeing if it starts working, and by trying to save it to a different path? Thanks!
Luca Lanzilao
@lanzi94_twitter
Jan 13 19:48
I'm currently using the latest version of Spyder, while the version of Python is 3.7. My operating system is Window 10. Yes, I'm not using special name or path, I'm just trying to save file.spydata on desktop. I've tried to open and close Spyder, change the directory of my project, open a new console, but anything worked yet. I noticed that if I try to save just a scalar or a vector, this method works, but it doesn't with my current data. Since I've a lot of data to store, do a check one by one it would may require hours. Is another method available for storing data? Or, is there a manner for making ''save data as'' works? Thank you.
CAM Gerlach
@CAM-Gerlach
Jan 13 20:07
Thanks for the detailed information. I did some further testing, and was in fact able to reproduce the problem. Specifically, I was able to get it to occur if Exclude unsupported data types was unchecked in the Variable Explorer options menu ("gear" icon), and at least one Python module (numpy, pandas, etc) was present in the namespace. Therefore, if you either deselect this option, or temporarily del any objects of type module (and just re-import them again after you've saved your workspace), you should be alright.
CAM Gerlach
@CAM-Gerlach
Jan 13 21:17
@lanzi94_twitter If that still doesn't work, you should try using del on any custom classes, "unusual" objects, etc. which are likely to be the cause rather than common dicts, lists, arrays, dataframes, etc and let us know what fixes it so we can be aware on our end.
Luca Lanzilao
@lanzi94_twitter
Jan 13 21:34
As you adviced, I tried to delete all the ''unusual'' objects, i.e. all kind of data relative to plot functions that were 'automatically' stored (I dind't even notice that they actually were in my data), and then the command ''Save data as'' perfectly worked. This was the unique thing I had to do. Many thanks for the help, I really appreciated it.
CAM Gerlach
@CAM-Gerlach
Jan 13 21:37
Glad it worked! Did just deleting any Python modules not work? Did Exclude Unsupported Data Types?
Luca Lanzilao
@lanzi94_twitter
Jan 13 21:42
I didn't try, I've just deleted that kind of 'unusual' object and now it works. Exclude Unsupported Data Types can be checked or unchecked, I don't think that this problem is related to this fact. Many thanks again.
CAM Gerlach
@CAM-Gerlach
Jan 13 21:59

Exclude Unsupported Data Types can be checked or unchecked, I don't think that this problem is related to this fact.

If you still experienced the problem with it checked, this implies the issue was fact due to a common data type that is considered "supported" (Python collections, numpy arrays, pandas dataframes, etc) which is unfortunate; it would have been beneficial to know what this was.

In any case; while Spyder can't yet serialize everything, it isn't supposed to just fail silently in such situations, and at least give you a clear error message as to what happened and ideally, just skip the unserializable objects entirely wit h a warning messaging listing what was not saved. Furthermore, a likely cause of this error (attempting to pickle module objects) can be fixed by automatically filtering them from the namespace before saving, since they cannot be serialized and a user would rarely, if ever want to actually save them as part of their session. I've opened a new Issue to this effect, spyder-ide/spyder-kernels#75 , which I should be able to implement soon. Therefore, if this happens again once that is implemented, modules will be skipped, only the objects which Spyder cannot serialize will not be saved, and it should pop up a warning message stating exactly what not could be saved.

Longer term, there are multiple facets to improving this: including an Exclude Python modules filter to the options menu (that would explicitly hide these, without affecting all arbitrary/custom types like Hide 'unsupported' variable types), as discussed in spyder-ide/spyder# 1676 ; adding a configurable name/type blacklist to the session save feature, as discussed in spyder-ide/spyder#7257 ; moving to a more comprehensive serialization library and/or a higher pickle protocol, etc. However, that change will get us most of the way there in terms of fixing the core unable to serialize issues.