I backed out of the brew install gdal and went with conda install gdal which solved the issue in much better runtime environment and management. Now I no longer have any dependency issue on the tiny experiment example.
However, I am running into a permission issue where the code wants to create a directory called /opt/data despite the fact that I set that directory some where else via with_root_uri.
Traceback (most recent call last):
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/__main__.py", line 17, in <module>
rv.main()
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/cli/main.py", line 294, in run_command
rv.runner.CommandRunner.run(command_config_uri)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/runner/command_runner.py", line 11, in run
CommandRunner.run_from_proto(msg)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/runner/command_runner.py", line 17, in run_from_proto
command.run()
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/command/train_command.py", line 21, in run
task.train(tmp_dir)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/task/task.py", line 137, in train
self.backend.train(tmp_dir)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/backend/pytorch_semantic_segmentation.py", line 228, in train
self.train_opts.model_arch, num_labels, pretrained=True)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/rastervision/backend/torch_utils/semantic_segmentation/model.py", line 6, in get_model
'deeplabv3', 'resnet50', num_labels, False, pretrained_backbone=True)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/torchvision/models/segmentation/segmentation.py", line 22, in _segm_resnet
replace_stride_with_dilation=[False, True, True])
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/torchvision/models/resnet.py", line 255, in resnet50
**kwargs)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/torchvision/models/resnet.py", line 217, in _resnet
progress=progress)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/site-packages/torch/hub.py", line 470, in load_state_dict_from_url
os.makedirs(model_dir)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/os.py", line 211, in makedirs
makedirs(head, exist_ok=exist_ok)
File "//anaconda3/envs/colab_jupy_raster/lib/python3.7/os.py", line 221, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/data'
make: *** [2] Error 1
2019-10-21 09:35:22:rastervision.utils.misc: DEBUG - Terminating 66156...
I grep for all of the files (source and generated) no where I have defined or used
/opt/data. Any clue where it is statically coded?
Or there is some environment variable I need to override?
I tried to rerun the entire pipeline and it looks I need to
rm -rf ..... to force a rebuild of this pipeline. Is there a nicer flag to use in this situation?
I just set the
with_root_uri to a new data directory so we can rebuild the entire pipeline with all the dependencies now in place.
Training [####################################] 100%
2019-10-21 10:09:50:rastervision.backend.pytorch_semantic_segmentation: INFO - train loss: 0.2515265792608261
Validating [####################################] 100%
2019-10-21 10:10:42:rastervision.backend.pytorch_semantic_segmentation: INFO - validation metrics: {'precision': 0.8326506614685059, 'recall': 0.8441998958587646, 'f1': 0.8383855223655701}
2019-10-21 10:10:42:rastervision.backend.pytorch_semantic_segmentation: INFO - epoch elapsed time: 0:04:03.569688
No statistics for
prediction step.
Running evaluator: SemanticSegmentationEvaluator...
2019-10-21 10:11:05:rastervision.evaluation.semantic_segmentation_evaluator: INFO - Computing evaluation for scene val_scene...
2019-10-21 10:11:05:rastervision.evaluation.semantic_segmentation_evaluation: DEBUG - Evaluating window: [ 0. 0. 1000. 1000.]
2019-10-21 10:11:05:rastervision.data.raster_source.rasterized_source: DEBUG - Rasterizing window: [ 0. 0. 1000. 1000.]
2019-10-21 10:11:05:rastervision.data.raster_source.rasterized_source: DEBUG - Cropping shapes to window...
2019-10-21 10:11:05:rastervision.data.raster_source.rasterized_source: DEBUG - # of shapes in window: 17
2019-10-21 10:11:05:rastervision.data.raster_source.rasterized_source: DEBUG - rasterio.rasterize()...
2019-10-21 10:11:08:rastervision.utils.misc: DEBUG - Terminating 67663...
Is this the expected result?
You'd basically have to create this, but for Azure https://github.com/azavea/raster-vision/blob/master/rastervision/runner/aws_batch_experiment_runner.py
There's not too much code there, which is more about how easy it is to use AWS Batch than other things. There's also some configuration that's set up outside of that code, and having to deploy the AWS resources in the first place - which happens through https://github.com/azavea/raster-vision-aws
However, writing one for Kubernetes that would run on Azure's Kube setup (I think they have one, right?), might be the way to go. I'd be interested in helping you think this through if you would be interested in taking it on
@cequencer_gitlab Just to clarify something: the AWS experiment runner has the ability to submit jobs within an experiment to an AWS Batch queue. There are some advantages to using this (like picking CPU vs. GPU instances) and starting and stopping instances automatically, but it's also possible to just log into a cloud instance (AWS, Azure, or whatever), get the Docker image off Quay, and then use the local experiment runner on that instance. That approach wouldn't involve extending the experiment runner.
@lewfish & @lossyrob Thank you for the ping back and providing a myriad of ways to go about running raster-vision on Azure. (I was out on a trip for a few days and I am back this week.).
I am not familiar with the AWS eco-system and I am relative new to Azure service, so I do not have a 1-to-1 mapping of “AWS Batch queue maps to Azure ???” would be. But it sounds like this approach is the easist to try and shortest effort?
I am unclear if there are 2 or 3 approaches when I tried to group both of your feedback. Here is my attempt to simplify, please do correct me if I am wrong:
- Perform the equivalent conversion of “AWS Batch queue” on “Azure equivalent”
- Build a pipeline using Docker container and services on Azure
- Build a kubernetes pipeline using AKS (Azure Kubernetes Services)
I do have some experiences in Kubernetes (K8S) deployments. Does raster-vision have a kubernetes operator code that is usable to deploy a raster-vision K8S deployment? If not, what is the current state on how raster-vision is working within a K8S stack?
Is this the right forum to continue this discussion?
@cequencer_gitlab I think the easiest to try is just start an Azure instance, ssh into it, and run RV locally there. I think this is what you meant by option #2 (and is what I was saying in my previous message). Then, if RV is providing some value, and you want to put in the effort to write your own ExperimentRunner that uses Azure and the code to setup the Azure resources (option #1 or #3) you could try that. RV doesn't have any Kubernetes-related functionality -- you would have to add an ExperimentRunner that uses it.
This is the right venue
I don't have any experience with Azure, but I found this page for Azure Batch which seems similar to AWS Batch. https://azure.microsoft.com/en-us/services/batch/
Hello guys, I was wondering if there is a article on how to take the next step and use in production the model that has been created using Raster-Vision and develop a prediction service API. Let's say a scenario like:
- take the zip predict package
- deployed in AWS S3
- Create serverless functions using this package
- Develop the right API
How exactly can I run multiple experiments of a single dataset without having seperate chips made for each experiment? The Directed Acyclic Graph looks at the input and output of a command, right? So having two experiments with a different experiment_id would be seen as different input/output as for each experiment a different folder is created, and have the same chips be made for each experiment, right (so double work done)? I am using the same dataset object when configuring the .with_dataset() method of the experiment, but that seems to result in a different chipping process being started for each experiment anyway?
@Charmatzis We don't have an article on this. Assuming you know how to setup an API using something like Flask you would just use the
Predictor class to load the model and make a prediction over a GeoTIFF file. https://docs.rastervision.io/en/latest/api.html#predictor
@lmbak This is an example of a set of experiments with a branching structure, where the chips are shared between the experiments. https://github.com/azavea/raster-vision-examples#spacenet-vegas-hyperparameter-search
The key thing is to use the same
root_uri and chip_key across the experiments.
I am focusing more on using Aws batch as a service called by a lambda. What do you think?
@Charmatzis So something triggers the lambda function which takes a reference to a GeoTIFF, and that kicks off a Batch job which calls the Predictor? That seems to make sense. I think you're right that it would be hard to create a lambda function with all the crazy dependencies that RV has. Also, the jobs can take too long for a lamba fn.
Transfer learning question: In my instance segmentation RV fork I'm trying to use the pretrained torchvision mask rcnn COCO model with the resnet-50 fpn backbone to re-train the 'head' of the model on the 1 m RGB of NAIP. I'm unsure exactly what parameters to freeze (requires_grad_(False)) and what to train.
In RV, the object detection code rastervision/backend/torch_utils/model.py freezes (requires_grad_(False)) where 'layer2' not in name and 'layer3' not in name and 'layer4' not in name, the first layer. Have RV developers found this to be more effective for transfer learning compared to freezing more of the model? Do you have experience with varying the 'depth' of transfer based on the size of the training data or some other constraint? I'd love to hear the reasoning behind this decision. Very fascinating; excuse me if something obvious went over my head. See my experiment https://github.com/dgketchum/raster-vision/blob/instance_seg/rastervision/inseg.py, and what I'm freezing, https://github.com/dgketchum/raster-vision/blob/instance_seg/rastervision/backend/torch_utils/instance_segmentation/model.py
@dgketchum I am pretty confident that you will want to fine-tune more of the model than just the head, especially since satellite imagery is quite different than COCO-like datasets. We always stick with the simple approach of fine-tuning the entire model using the same learning rate for all layers. We use a learning rate that is 1/10 the default value provided by the library when fine-tuning. This provides a big advantage over just training from scratch. However, some people have reported a small benefit from using differential learning rates (ie. higher rates in layers closer to the head), or just freezing the first few layers. They discuss this at some point in the fastai course