Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Brian Carcich
@drbitboy
  return [...]
that way, you can load the kernels once, before the call, and let the [kernels] keyword argument to that routine be equivalent to [False], so nothing is done; OR you can pass in any kernel(s) in a list (e.g. some_spice_routine(arg1,arg2,kernels=['metakernel.tm']), and the routine FURNSHes the kernel, and cleans up after itself at the end.
kidpixo
@kidpixo
@drbitboy nice! thanks
Andrew Annex
@AndrewAnnex
btw furnsh accepts Iterable data types so you can directly pass in a list of strings of kernel paths
kidpixo
@kidpixo
Hi there
I'm trying to do some spiceypy calculation using https://joblib.readthedocs.io/en/latest/parallel.html , essentially I calculate all my sensor pixel corners vectors point and then I loop over all the observations time to get the coordinates at target .
Obviously it isn't working
I'm passing the furnished spiceypy instance to the function calculating the actual geomtry of observation at a given time, bu I believe that both thread-based parallelism and process-based parallelism don't inherit it and use a fresh not-furnished spiceypy.
kidpixo
@kidpixo
any idea ? experience with joblib ?
kidpixo
@kidpixo
slightly adapted code
from joblib import Parallel, delayed
import multiprocessing
num_cores = multiprocessing.cpu_count()

output  = Parallel(n_jobs=num_cores, backend='threading')(delayed(spice4mertis_geometry_wrapper)(utc, k,et,v, target, frame, sensor, observer) for utc,et in TIS_UTC_time )
kidpixo
@kidpixo
data example
import numpy as np
utc, k,et,v, target, frame, sensor, observer = '2020-04-09T04:00:12.105','C22',639676881.2906493, np.array([-4.61646326e-03, 6.13704793e-04,9.99989156e-01]), 'MOON','IAU_MOON','MPO_MERTIS_TIS_SPACE','MPO'

A single run on those data returns :

dict(zip(head,spice4mertis_geometry_wrapper(utc,k,et,v,target,frame,sensor ,sensor_id,observer, spice)))

{'utc': '2020-04-09T04:00:12.105',
 'name': 'C22',
 'et': 639676881.2906493,
 'tarlon': -0.20023645299777082,
 'tarlat': -0.4657544517921964,
 'sublon': -0.09431312287105752,
 'sublat': 0.014900819804988915,
 'sunlon': -0.21279322760557254,
 'sunlat': -0.025399579268598888,
 'taralt': 747621.3385348794,
 'subalt': 0.2668455383644952,
 'sunalt': '12:02:52',
 'tardis': 7.201687663702408,
 'tarang': 28.21652025567335,
 'ltime': 25.240221527886003}

so , it works on single direction

kidpixo
@kidpixo
I think I am making a mess importing spiceypy and passing it around to function as variable....
Brian Carcich
@drbitboy
CSPICE, the C library underneath spiceypy, is not thread-safe.
Also, I am pretty sure it is not necessary to pass the spiceypy module, and importing the module should be adequater.
Brian Carcich
@drbitboy
  • adequate ;-)
kidpixo
@kidpixo
right, thanks
kidpixo
@kidpixo

https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/Tutorials/pdf/individual_docs/45_development_plans.pdf

Develop SPICE 2.0: a re-implementation of the SPICE Toolkit from the ground, up, providing thread-safe and object oriented features
–This is the major NAIF undertaking, started in May 2017
–It is being implemented in C++11
–It is expected to take several years

kidpixo
@kidpixo

uhm I'm using joblib.
As stated here https://joblib.readthedocs.io/en/latest/parallel.html#thread-based-parallelism-vs-process-based-parallelism

By default joblib.Parallel uses the 'loky' backend module to start separate Python worker processes to execute tasks concurrently on separate CPUs.
[...]
When you know that the function you are calling is based on a compiled extension that releases the Python Global Interpreter Lock (GIL) during most of its computation then it is more efficient to use threads instead of Python processes as concurrent workers.
[...]
To hint that your code can efficiently use threads, just pass prefer="threads" as parameter of the joblib.Parallel constructor

I guess the loky backend would be ok, but this import a fresh spiceypy for each process, not furnished.

kidpixo
@kidpixo

@drbitboy ok, not it works, ugly and inefficient as it is . Essentially I furnish the kernel at each loop in the wrapper function and kclean afterward. I have no idea how this interact with furnshd spiceypy in the main code ....

the for k,v in centers.items(): a second loop on all the directions i need to calculate at each time.

def spice4mertis_geometry_wrapper(utc,k,et,v,target,frame,sensor, sensor_id,sensor_frame,observer):
    metakernel = 'PATH_TO_KERNEL
    spice.furnsh(str(metakernel))
    out = [utc, k,et]+spice4mertis.core.geometry.geometry(et,v, target, frame, sensor, observer=observer)
    spice.kclear()
    return out

from joblib import Parallel, delayed
import multiprocessing
num_cores = multiprocessing.cpu_count()
# for k,v in centers.items():
output  = Parallel(n_jobs=num_cores, backend='loky')(delayed(spice4mertis_geometry_wrapper)(utc,k,et,v,target,frame,sensor, sensor_id,sensor_frame,observer) for utc,et in zip(utc_time.values,et_time))
Brian Carcich
@drbitboy
does the delayed wrapper routine have access to its thread-instance (or process-instance) information? we could run tests e.g. spice.ktotal() to see if one thread or process affects another. Also, if the [import spiceypy] was inside the scope of the delayed wrapper routine, would that affect how the module was operated?
kidpixo
@kidpixo

@drbitboy so:

does the delayed wrapper routine have access to its thread-instance (or process-instance) information?

I guess not, because it was not working. Passing the metakernel path and furnishing in the delayed wrapper works.

we could run tests e.g. spice.ktotal() to see if one thread or process affects another.

I writing this down for my test, good idea, thanks.

Also, if the [import spiceypy] was inside the scope of the delayed wrapper routine, would that affect how the module was operated?

sorry, I don't understand the question, which kind of operation do you mean?

Brian Carcich
@drbitboy
def spice4mertis_...(...)
   import spiceypy as spice
   sp.ktotal()
   ...
kidpixo
@kidpixo
ok, I'll try , thanks
me and @AndrewAnnex are part os openplanetary.slack.com | maybe would useful to move this discussion there or to https://forum.openplanetary.org/ for future reference ?
Andrew Annex
@AndrewAnnex
oh yeah I don't really check this gitter frequently, yeah I have never attempted to parallelize spice calls because of all the issues mentioned above, and spicecalls have never been the bottleneck for me, but multiprocessing would presumably have the best chance of working if each process has a separate kernel pool, doing the test @drbitboy suggest for modifying the kernel pool(s) would also be a very good idea to make sure nothing nasty is happening
Quessian-Phaedrus-Sternabite
@Quessian-Phaedrus-Sternabite
error.png
Hey Andrew! I've been following your space science with python tutorial and ran into an issue. Furnsh completely stopped working. For some reason, on part 6 the meta kernel doesn't work. Neither does manually furnshing them. Furthermore, it doesn't work on mac for one of my colleagues. Whenever I try to run part six, I get this error (seen above). Finally, the format used in the tutorial meta kernels doesn't work for me, and it'd be great to figure out how to fix that. Thanks!
Brian Carcich
@drbitboy
I am not sure it is Andrew's tutorial. Is it this: https://github.com/ThomasAlbin/SpaceScienceTutorial ?
Quessian-Phaedrus-Sternabite
@Quessian-Phaedrus-Sternabite
Yes, It's part 6 of that. I'm following it on towardsdatascience.
Quessian-Phaedrus-Sternabite
@Quessian-Phaedrus-Sternabite
I normally manually write it from the boxes, but if I get an error I eventually just copy the code directly from github to see what's different. In this case, nothing was, and even directly copying it wasn't working. I have gotten to line 154.
Sorry I wrote here, I did not know where else to contact help.
Brian Carcich
@drbitboy
you might try contacting the author of the tutorial e.g. create an issue on github. btw, the most likely reason for what you are seeing is a problem with the spicepy or kernel setup. I was looking at it and saw that the _kernels/ subdir did not have one of the SPKs in the meta-kernel. maybe that was intentional to keep the repo small.
and why there is a download script.
I did not dig very deeply into it.
Quessian-Phaedrus-Sternabite
@Quessian-Phaedrus-Sternabite
Yes, the project files originally don't have all the kernels and you must use a download script. I already downloaded them though. The other issue is the tutorial format not working and it completely not working on mac.
Quessian-Phaedrus-Sternabite
@Quessian-Phaedrus-Sternabite
I added an error to the tutorial.
mhmsoota
@mhmsoota

Hello community members!

I am a beginner to SpiceyPy and it's really amazing to see experienced people here. Me and my team are working on a simulation project with STK, to try out missions to Asteroids. We are a bit clueless on how to simulate asteroids for which we found we can upload a .bsp or .pe file. In case anybody can guide us on how to be able to make such fies from given data, we'll be really grateful.

Brian Carcich
@drbitboy
I refer you to the required reading for NAIF/SPICE e.g. here: https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C/req/index.html
the SPK required reading describes how to write your own SPKs (.bsp) files. If you have the data in a flat file there is also the MKSPK utility built-in to SPICE; the user guide is here: https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C/ug/mkspk.html
I am a degreed Chemical Engineer; I first read the required reading files over three decades ago and that became my new career.
Brian Carcich
@drbitboy
SPICE is the meat behind the SpiceyPy wrapper; you will not find many libraries better documented than SPICE.
STK can use SPKs (.bsp files) to simulate trajectories.
Have you found JPL Horizons yet? You can get SPKs direct from there.
mhmsoota
@mhmsoota
@drbitboy that seems new to me... I will be checking this out for now then
Yongzhang Yang
@recobaself_gitlab
@AndrewAnnex , Hi~ when I installed the spiceypy via oofline, it shown " File "/Users/XXX/Downloads/spiceypy/SpiceyPy-main/get_spice.py", line 263
print(f"Found CSPICE source at {cspice_dir}")" SyntaxError: invalid syntax
Yongzhang Yang
@recobaself_gitlab
@AndrewAnnex and My python is 2.7
Brian Carcich
@drbitboy
the f"..." does not work with 2.7. Try 3.x.
Yongzhang Yang
@recobaself_gitlab
@drbitboy many thx!
Andrew Annex
@AndrewAnnex
As a reminder I rarely check this chat (but thanks @drbitboy for holding the fort!), if you need to reach me please open an issue on the spiceypy github or join the OpenPlanetary slack group and message me there.