@brynpickering Hi Bryn. I've now got things up and running w/ 0.6.3. I've implemented (separate from Calliope) the optimization method introduced by Poncelet (2017) to select representative days and their associated weights. I'd like to test these days out in the uk-calliope much like Stefan did in his similar paper. I've followed the directions outlined in the docs to load user-defined representative days. Some questions:
1) It's not clear to me how to set the timestep_weight values as an input parameter. My input file would ideally have columns of i) Timestep; Cluster Number; Weight.
2) It seems when I use the apply_clustering function (pointing to the file containing the representative days), all of the raw data is first read in during Calliope's first preprocessing step. This requires GBs worth of memory when only the data from the pre-selected representative days need be read in. Not the end of the world, just makes troubleshooting tedious.
Any tips would be helpful. Again, all I'm trying to do is take the output from the separate optimization model (i.e. clusters and their associated weights) and use that as an input to the Calliope-UK model. Thanks!
@_clameron_twitter
1) You can give information on the timestep and cluster number, but it will automatically take the timestep_weight as being equals to the number of timesteps associated with that weight. If you have specific timestep weights that override this, then you'll need to update the resulting data in
2) Yep, that's because it does time clustering after loading in the data. Since there are different ways of selecting the representative days, it needs access to all timesteps to make that decision (and the inbuilt clustering methods work with all timesteps). It also doesn't really work with pandas to only load in 'representative days' into memory when it loads in a CSV; without a lot of work, it's going to have to load in all data, then select a subset of it. So you'll always have a reasonable amount of memory consumption with a model like uk-calliope.
If you're doing all the clustering independently of the in-built calliope functions, the best might be to create CSVs with only the representative days. You'll need to give them a monotonically increasing timeseries (i.e. if it's 5 RDs, the timeseries is the first 5 days of a random year) and then assign the timestep weights as your custom weights after the model has been loaded in. That will reduce memory consumption too. Would require some playing around to make it all work properly, but might be your best bet.
1) You can give information on the timestep and cluster number, but it will automatically take the timestep_weight as being equals to the number of timesteps associated with that weight. If you have specific timestep weights that override this, then you'll need to update the resulting data in
model._model_data. Keep in mind that if timestep weights are updated and the sum of weights doesn't equal to the sum of all original timesteps, calculations further down the line will be erroneous.2) Yep, that's because it does time clustering after loading in the data. Since there are different ways of selecting the representative days, it needs access to all timesteps to make that decision (and the inbuilt clustering methods work with all timesteps). It also doesn't really work with pandas to only load in 'representative days' into memory when it loads in a CSV; without a lot of work, it's going to have to load in all data, then select a subset of it. So you'll always have a reasonable amount of memory consumption with a model like uk-calliope.
If you're doing all the clustering independently of the in-built calliope functions, the best might be to create CSVs with only the representative days. You'll need to give them a monotonically increasing timeseries (i.e. if it's 5 RDs, the timeseries is the first 5 days of a random year) and then assign the timestep weights as your custom weights after the model has been loaded in. That will reduce memory consumption too. Would require some playing around to make it all work properly, but might be your best bet.
@sjpfenninger Thank you for your input regarding the transmissions. Unfortuneally I ran into a problem with the new constraints. I'm using the supply_share_equals constraint and get following error message:
ERROR: Rule failed when generating expression for constraint group_supply_share_equals_constraint with index ('Wind_HU', 'NG', 'equals'): KeyError: ('NG', 'Wind_HU')
ERROR: Constructing component 'group_supply_share_equals_constraint' from data=None failed: KeyError: ('NG', 'Wind_HU')
The constraint I use looks like this:
Wind_HU:
techs: ['ConELC-PP_WON']
locs: ['HU']
supply_share_equals:
electricity: 0.02
It seems to me, that the problem lies with my other carriers (i.e. NG). Is there a good way to fix this?
ERROR: Rule failed when generating expression for constraint group_supply_share_equals_constraint with index ('Wind_HU', 'NG', 'equals'): KeyError: ('NG', 'Wind_HU')
ERROR: Constructing component 'group_supply_share_equals_constraint' from data=None failed: KeyError: ('NG', 'Wind_HU')
The constraint I use looks like this:
Wind_HU:
techs: ['ConELC-PP_WON']
locs: ['HU']
supply_share_equals:
electricity: 0.02
It seems to me, that the problem lies with my other carriers (i.e. NG). Is there a good way to fix this?
Hello everybody, I am still working on the Kenya model that I talked to you about some months ago and I am finalizing it. I just have one small problem with transmission lines definitions that I hope you might help me to solve.
I have 4 regions in my model and two different kinds of transmission lines connecting them, namely 132kV and 220kV, now in the 6.4 version I defined for every couple of regions two different techs per each voltage, one for one direction and one for the other, since they have different capacities in the two directions (an example here)
I have 4 regions in my model and two different kinds of transmission lines connecting them, namely 132kV and 220kV, now in the 6.4 version I defined for every couple of regions two different techs per each voltage, one for one direction and one for the other, since they have different capacities in the two directions (an example here)
WSTR,MTKR:
techs:
hvdc_132:
constraints:
energy_cap_equals: 133183
one_way: true
distance: 0.52 # in hundreds of km
MTKR,WSTR:
techs:
hvdc_132_back:
constraints:
energy_cap_equals: 127868
one_way: true
distance: 1.13 # in hundreds of km But i get the error:
"Index '('MTKR::hvdc_220_new:NBOR::power', Timestamp('2035-06-01 00:00:00'))' is not valid for indexed component 'carrier_prod'"
and I don't get it without "one_way: true", but I get one bidirectional line connecting the two regions with the second values inserted,
has anybody faced this problem before? Any hint will be super appreciated thanks everybody :)
Hi there,
I'm trying to set up a new model in Calliope to understand when it makes sense to connect two microgrids in a remote context. So far I'm just setting up a simple toy model with two nodes before starting to put in some more realistic data, and have been running into some errors.
I was wondering if anyone might have seen similar and be able to offer any advice?
$ ./run_single.sh
Attempting run...
Traceback (most recent call last):
File "/home/spf310/anaconda3/envs/calliope/bin/calliope", line 5, in <module>
from calliope.cli import cli
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/__init__.py", line 2, in <module>
from calliope.core import AttrDict, Model, read_netcdf, _logger, set_log_level
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/__init__.py", line 1, in <module>
from calliope.core.model import Model, read_netcdf
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/model.py", line 18, in <module>
from calliope.analysis import plotting, postprocess
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/analysis/plotting/__init__.py", line 1, in <module>
from calliope.analysis.plotting.plotting import plot_summary, ModelPlotMethods
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/analysis/plotting/plotting.py", line 21, in <module>
from calliope.analysis.plotting.transmission import plot_transmission
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/analysis/plotting/transmission.py", line 16, in <module>
from calliope.core.preprocess.util import vincenty
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/preprocess/__init__.py", line 2, in <module>
from calliope.core.preprocess.model_run import \
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/preprocess/model_run.py", line 24, in <module>
from calliope.core.preprocess import locations, sets, checks, constraint_sets, util
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/preprocess/locations.py", line 17, in <module>
from calliope.core.preprocess.checks import DEFAULTS, POSSIBLE_COSTS
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/preprocess/checks.py", line 31, in <module>
DEFAULTS = AttrDict.from_yaml(_defaults_files['defaults'])
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/attrdict.py", line 167, in from_yaml
loaded = cls(_yaml_load(src))
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/site-packages/calliope/core/attrdict.py", line 40, in _yaml_load
src = src.read()
File "/home/spf310/anaconda3/envs/calliope/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3403: ordinal not in range(128)
Run finishedI've uploaded a minimal example of the model I've been running here https://drive.google.com/drive/folders/1EeHV9BNWP2Zvc6TeOOmBLq_p-m9ZFDeO?usp=sharing
Any advice much appreciated,
Sheridan
so not a problem with the model per se
can you check that all your files are saved as UTF-8 encoded text
@Stevogallo it's difficult to tell why there would be an issue with your model, but can you check that you never define the same technology in both directions of a link? e.g.
NBOR,MTKR:
techs:
hvdc_220_new:
constraints:
energy_cap_equals: 133183
one_way: true
MTKR,NBOR:
techs:
hvdc_220_new:
constraints:
energy_cap_equals: 127868
one_way: true
Hello to you all. I would like to ask if anyone had the following problem and if there is any workaround: During the calliope run and after the solving rpocess it remains an infinite time saving plots in HTML. Specifically it stays in the "Saving HTML file with plots to: file.html" process.
The question is that my model has almost 200 nodes so possibly creating that file is a hard task. Does anyone have experienced this?
The question is that my model has almost 200 nodes so possibly creating that file is a hard task. Does anyone have experienced this?
@luzgui Creating HTMLs of large problems will take a very long time, it's just the price paid for interactive plots. I would suggest not producing the HTML files automatically (i.e., no use of
--save-plots), then either use the Calliope plotting API or your own custom plotting functions in python interactively
Goodmorning, I'm Alessandro, a master thesis student at Politecnico di Milano.
I'm working on a water-energy-nexus model, integrating hydrological balance in hydroelectric technology implementation.
My focus is the Zambezi River basin, in which there are 4 cascade reservoirs. It means that the upstream turbined flows go into the downstream reservoirs.
I modeled the system describing the reservoir as storage tech, the inflow as supply tech and a conversion plus tech to model the electricity produced by the hydropower plant and
to track the turbined flow as a cascade inflow to the downstream reservoir (scheme attached).
The unit of measure of the storage has to be considered in m^3 of water that are converted in kWh through an energy efficiency in the conversion plus tech.
A more precise description wants the hydraulic head varying with the storage water level, it means that the energy efficiency should be a function of the storage variable.
Is it feasible to implement this dependance?
Thanks everybody for your time
I'm working on a water-energy-nexus model, integrating hydrological balance in hydroelectric technology implementation.
My focus is the Zambezi River basin, in which there are 4 cascade reservoirs. It means that the upstream turbined flows go into the downstream reservoirs.
I modeled the system describing the reservoir as storage tech, the inflow as supply tech and a conversion plus tech to model the electricity produced by the hydropower plant and
to track the turbined flow as a cascade inflow to the downstream reservoir (scheme attached).
The unit of measure of the storage has to be considered in m^3 of water that are converted in kWh through an energy efficiency in the conversion plus tech.
A more precise description wants the hydraulic head varying with the storage water level, it means that the energy efficiency should be a function of the storage variable.
Is it feasible to implement this dependance?
Thanks everybody for your time
@Alessandro-Barbieri95 nice - certainly the kind of description hydropower systems should have! As to the question of hydraulic head, there is no way I can think of to link electricity generation efficiency with quantity of stored energy in a linear model (it would require multiplication of decision variables). But I also wouldn't expect the difference in hydraulic head to have such a large impact on output efficiency. Instead, it may noticeably impact the maximum power output (
carrier_prod(t) <= storage(t) * some_constant). I can imagine that dependence being implementable.
@Alessandro-Barbieri95 I confirm what @brynpickering was suggesting, you cannot link the energy storage level with an efficiency without going non-linear. I have similar problems modelling thermal storage and power-to-heat conversion techs. There are ways you can escape that, and we did it for non-linear thermal modelling, but probably not ones that would make you happy and your problem easy to solve. And I also agree that the efficiency shouldn't be particularly pivotal here compared to other things, such as the maximum power output.
Hi everyone. I'm currently learning to use Calliope to model renewable integration in some key areas in Portugal.
My question for the day is: after loading a "model" variable using calliope.read_netcdf(), where in that variable can I find the parent for each tech that was described in the .yaml?
Thanks
My colleague Guilherme Luz (@luzgui) and I will be at the openmod workshop next week. Maybe we'll see each other there :)
Hi everyone, just to let you know that version 0.6.5 is now live! It's a minor release which mostly cleans up existing functionality, but with a couple of backwards-incompatible changes. If you're interested, check out our short release post. Happy modelling :)
Hi everyone, I am Javier Valdes from TH Deggendorf. I just started using Calliope 0.6.5 -amazing tool- for modelling microgrids in a number of locations.
So far we are exploring the urban_scale model. We want to specify revenues for each node when the energy generated in that node is transmitted and consumed in a different node. I am not sure if this is the right place to ask this question, but here it goes:
I realized that for the transmission lines the allowed costs include om_pro. Which can represent the cost of consuming energy from one of the transmission links. If I am getting it right, these costs are associated to the end point, the location where the carrier is consumed. I thought that setting om_con as negative in the transmission line would represent the associated beneffit of sending the energy in the node where the it is generated. Nevertheless, the option om_con is not defined for the transmission tech. There's probably a reason for that, and something wrong with my logic.
Do you know what would be the best option to model this situation without using om_con for the transmission lines?
Thanks for your help!
So far we are exploring the urban_scale model. We want to specify revenues for each node when the energy generated in that node is transmitted and consumed in a different node. I am not sure if this is the right place to ask this question, but here it goes:
I realized that for the transmission lines the allowed costs include om_pro. Which can represent the cost of consuming energy from one of the transmission links. If I am getting it right, these costs are associated to the end point, the location where the carrier is consumed. I thought that setting om_con as negative in the transmission line would represent the associated beneffit of sending the energy in the node where the it is generated. Nevertheless, the option om_con is not defined for the transmission tech. There's probably a reason for that, and something wrong with my logic.
Do you know what would be the best option to model this situation without using om_con for the transmission lines?
Thanks for your help!
Hi @jvValdes , I guess that for transmission techs it would be a bit weird to have both om_con and om_prod, because of the bi-directionality of the flows. Anyway, nothing should prevent you from setting a negative om_prod for any technology (in the urban_scale example PV has a negative om_prod, for instance). This might work well enough for your case, 'cause basically you would have a revenue for each time an energy flow is transmitted from one location to another. You just have to take care when post-processing the results and be sure about which node is earning the revenue (if that is important to you). Let me know if that works
Hi Francesco,
Thanks for your help! Yes, I am mainly interested in the energy flows between locations and the lcoe for each location. I followed your advice and modified the model including a negative cost for the om_prod in the transmission lines below the cost of the national_grid. To double check I also do the same in the urban_scale model including in transmission lines: om_prod = -0.099, which is slightly bellow the cost of the national grid. Is that what you meant?
I run the models and exported the results. When I examine the results_carrier_con for the power grid, the results are extraordinary high and negative for all locations, same for results_carrier_prod. Actually the transmission lines are used until the energy_cap_max is reached. I am not sure if that behaviour is normal. I expected to obtain (lower) negative values only for the locations where the carrier is consumed in the file results_carrier_con and (lower) positive values for the Locations where the carrier is produced in the results_carrier_prod. Finally, when I check the results_cost_var file the negative costs are also incurred in all Locations. I am fairly new to Calliope and I am not understanding these results. Do they make sense to you? Thanks!
Thanks for your help! Yes, I am mainly interested in the energy flows between locations and the lcoe for each location. I followed your advice and modified the model including a negative cost for the om_prod in the transmission lines below the cost of the national_grid. To double check I also do the same in the urban_scale model including in transmission lines: om_prod = -0.099, which is slightly bellow the cost of the national grid. Is that what you meant?
I run the models and exported the results. When I examine the results_carrier_con for the power grid, the results are extraordinary high and negative for all locations, same for results_carrier_prod. Actually the transmission lines are used until the energy_cap_max is reached. I am not sure if that behaviour is normal. I expected to obtain (lower) negative values only for the locations where the carrier is consumed in the file results_carrier_con and (lower) positive values for the Locations where the carrier is produced in the results_carrier_prod. Finally, when I check the results_cost_var file the negative costs are also incurred in all Locations. I am fairly new to Calliope and I am not understanding these results. Do they make sense to you? Thanks!
Hi @jvValdes , it's hard to tell not knowing precisely the structure of your model, but clearly if it doesn't make sense to you then it is probably wrong. From what you say (e.g. "results are extraordinarily high") it seems that setting a revenue for transmission leads to a weird and unwanted behaviour. This could be due to the fact that overproducing electricity with some technology for the sake of trading it leads to revenues that exceed the costs of production; could that be?
Hi @FLomb. yes the system is overproducing electricity, but I am not sure from where/why. The electricity flows between nodes are higher than the electricity produced in the whole system. That is why the results are so weird to me. Let me introduce this toy model. I am actually using a reduced version of the urban scale model with only two nodes. The only differences are the following. First, as you suggested, I added a om_prod to the power_lines:
power_lines:
essentials:
name: 'Electrical power distribution'
color: '#6783E3'
parent: transmission
carrier: electricity
constraints:
energy_cap_max: 2000
energy_eff: 0.98
lifetime: 25
costs:
monetary:
interest_rate: 0.10
energy_cap_per_distance: 0.01
om_prod: -0.05 # the cost/profit are below the 10p/kWh electricity price of supply_grid_powerSecond, I reduced the number and complexity of locations:
X1:
techs:
chp:
supply_grid_power:
supply_gas:
demand_electricity:
constraints.resource: file=demand_power.csv
available_area: 500
coordinates: {x: 2, y: 7}
X2:
techs:
supply_grid_power:
demand_electricity:
constraints.resource: file=demand_power.csv
coordinates: {x: 8, y: 7}
N1:
coordinates: {x: 5, y: 7}
links:
X1,X2:
techs:
power_lines:This model does not produce meaningfull results. When I say meaningful, I mean that the energy produced match the demand and that I can identify the source/end of each flow in the results_carrier_con and results_carrier_prod outputs.
In this model, the power lines flows are equal to the energy_cap_max of the transmissions. The only explanation I find is that the same unit of electricity is continually being sold. Is it possible that in one period the same unit of electricity is traded as many times as posible between the two nodes until reaching the energy_cap_max of the power line? If yes, would not be an option to set an om_con in the power lines to solve the effect of this redundancy?
..and a node N1 if the heat pipes and the heat demand are removes
removed*
but maybe you were only sharing part of the techs? anyway, it seems plausible to me that what is happening is that the model is installing extra capacity to overproduce as much as possible and earn (i.e. mimimise total costs) from it. Hence you see flows across transmission lines that equal max transmission capacity in every time step because that is precisely what is happening: always overproducing as much as possible (i.e. as much as transmission cap allows).
so clearly my suggestion of setting a negative om_prod for transmission techs has not been very smart. And though it might maybe work better in a case in which supply capacity is more constrained, still I am realising that the model would try to move energy from one node to the other even before self-consuming (i.e. one node satisfies the demand of the other, instead of its own, to earn money), which is not what you wanted, I guess
to summarise, I think you would like to have trading with revenue only "after" self consumption, right? something like the "export" feature of PV in the example urban model, but between nodes instead of generic?
You are right, the model will always overproduce a small quantity of energy and trade it between nodes as many times as possible until the maximum capacity is reached. Please note that as the system tries to minimize costs, the quantity of energy overproduced is small compared to the quantity of energy traded. The only penalisation both nodes have is the energy_eff of the transmission lines set to 0.98.
I would like to have a trade between nodes with revenues, that means that some nodes will compute a cost and other nodes a gain. The export feature is similar, but I am not sure if it would be a solution. With such kind of feature I would not be able to identfy the source of the flows and the required transmission capacities between nodes. Which is one of the features that makes Calliope so attractive. Moreover, it may be necessary to define an extra import feature?
in his case "Index '('MTKR::hvdc_220_new:NBOR::power', Timestamp('2035-06-01 00:00:00'))' is not valid for indexed component 'carrier_prod'"
@luzgui and I tried out this one_way solution ourselves just recently, and in our case we got that error whenever we defined an om_prod or an om_con for the transmission tech
our interpretation is that the "one_way" makes the con and prod inexistent for the "blocked" direction and thus calliope cannot apply the costs (which are assumed for both directions)
thanks for the update @ramaroesilva, looks like a bug to create an issue from! I've done so here: calliope-project/calliope#316
Hi everyone. This review work might be of interest for the calliope community:
https://www.sciencedirect.com/science/article/pii/S1364032120302082
https://www.sciencedirect.com/science/article/pii/S1364032120302082