macports-gsoc/gsoc-2019-webapp

Mojca Miklavec

@mojca

This commit (I mean: this functionality) needs unit tests

I'll try to do some testing in the evening

Arjun Salyan

@arjunsalyan

OK, thanks.

Mojca Miklavec

@mojca

Make sure that you also get deletions right

or file moves

Make it configurable to the extent that you clone a specific commit, and then pull to a configurable commit (to which you would usually provide the argument 'HEAD')

it would be nice if the functionality works if you provide arbitrary two commits; by default it could be existing commit vs. HEAD

or rather ... you'll store the commit to the database.

so you'll probably want to compare the stored commit's shasum with HEAD

Arjun Salyan

@arjunsalyan

so you'll probably want to compare the stored commit's shasum with HEAD

Yes, currently it does that only. But the other configurations will be very handy- thank you. I will add them.

Sorry if this is stupid: How do I exactly test this? It depends on cloning something. Do I create an arbitrary repo for this purpose?

Mojca Miklavec

@mojca

You need to specify the source of the git repository (use full git url to macports-ports) and the path where this is executed. One tests needs to start with an empty directory. So your function clones the repository first, then compares two given shasums and gives you the list of ports (you need to know which ones are new and which ones were deleted). For the second test you first clone the repository to some arbitrary old commit (I hope it's possible to ensure that newer commits don't get pulled in), then the function pulls everything else and your function verifies the same two shasums. The third test should start with invalid or broken repository (you can clone a random other small repository, or populate the directory with that name, or both) and make sure that the function would handle that properly (maybe purge the contents, just be careful not to point it to the wrong location, so that you don't end up with your root folder being purged).

Mojca Miklavec

@mojca

You may need to change the function a bit to allow such testing, of course.

Arjun Salyan

@arjunsalyan

Make it configurable to the extent that you clone a specific commit, and then pull to a configurable commit (to which you would usually provide the argument 'HEAD')

As long as we can can provide the hashes to develop required range- would it matter if we pull the entire repo or just till a specific commit?

Arjun Salyan

@arjunsalyan

How the current command works:

python manage.py {new_hash} {old_hash}

If both of them (old hash, new hash) are provided. Then entire repo is pulled and updates are carried out only between the two. New hash gets saved in the database- indicating that till this the updates have taken place.

If only new_hash is provided then old_hash is fetched from the database. And then updates are carried out between the two. new_hash again is saved to the database.

If none is provided then HEAD becomes the new_hash and old_hash is fetched from the database.

Arjun Salyan

@arjunsalyan

Make sure that you also get deletions right

git diff would give path if a file was deleted. But won't find it in the Portindex and so we can mark it deleted.

The third test should start with invalid or broken repository (you can clone a random other small repository, or populate the directory with that name, or both) and make sure that the function would handle that properly (maybe purge the contents, just be careful not to point it to the wrong location, so that you don't end up with your root folder being purged).

I have added some tests- still figuring out a nice way to achieve this one.

BTW: cloning this macports-ports has taken the run-time for tests from 1 second to 37 seconds. I don't know if this is a problem. Should we setup a smaller test repository instead?

Mojca Miklavec

@mojca

Of course you can pull the entire repo, you can ask for arbitrary changes even if you have the full repo cloned

Also, it would make sense to swap new_hash and old_hash, as usually you would run something like

git diff old..new

Arjun Salyan

@arjunsalyan

Yes, old new is more natural. But the only reason I went with reverse was that most of the times we would not enter the old hash and simply pick it from the database- for consistency point of view. And if I use old new then old would become a required argument to even enter new. But since we would mostly be interested in providing the new hash to I went this way. We can avoid this by using --new optional arguments, but that does not look so neat.

This is just my perception... And maybe I am thinking in wrong direction.

Mojca Miklavec

@mojca

https://avilpage.com/2018/05/django-tips-tricks-log-sql-queries-to-console.html

Arjun Salyan

@arjunsalyan

SELECT "ports_buildhistory"."id", "ports_buildhistory"."builder_name_id", 
"ports_buildhistory"."build_id", "ports_buildhistory"."status", "ports_buildhistory"."port_name", 
"ports_buildhistory"."time_start", "ports_buildhistory"."time_elapsed", 
"ports_buildhistory"."watcher_id" FROM "ports_buildhistory" ORDER BY 
"ports_buildhistory"."time_start" DESC

Mojca Miklavec

@mojca

https://github.com/macports/macports-infrastructure/blob/master/jobs/mprsyncup#L177

Mojca Miklavec

@mojca

https://github.com/macports/macports-infrastructure/blob/master/jobs/mprsyncup#L139

Mojca Miklavec

@mojca

https://frozen-falls-98471.herokuapp.com/maintainer/github/mojca/

lang/perl5

Arjun Salyan

@arjunsalyan

L177

${PORTINDEX} -p "macosx_${PLATFORM}" -o "${INDEX}"

After the above line, can I simply run this:

tclsh portindex2json.tcl "${INDEX}"/Portindex > "${INDEX}"/Portindex.JSON

Means after defining the right paths to tclsh and portindex2json.tcl

It would give us portindex.JSON in the location ${INDEX}/portindex.JSON

Mojca Miklavec

@mojca

In this case it would rather be

tclsh portindex2json.tcl "${INDEX}" > "${INDEX}.json"

I don't mind if for simplicity we simply generate json for all

INDEX in the above code is already the filename. But yes, adding just that line should be sufficient.

The other thing is ... the script is cloning the git repository for ports & base. I assume it could clone also the one containing portindex2json. And at some point I would argue that portindex2json could simply become part of macports

Arjun Salyan

@arjunsalyan

INDEX in the above code is already the filename. But yes, adding just that line should be sufficient.

When I ran this locally:

portindex -o something

It created a directory 'something' and put PortIndex in it.

And if we see this: L214- it makes me feel that a file name Portindex exists in the directory named PortIndex_darwin_${PLATFORM} .

Mojca Miklavec

@mojca

oh, sorry, I stand myself corrected

I just guessed how it worked

Arjun Salyan

@arjunsalyan

The other thing is ... the script is cloning the git repository for ports & base. I assume it could clone also the one containing portindex2json. And at some point I would argue that portindex2json could simply become part of macports

So we clone it from somewhere and put in the $PORTS directory

Or for simplicity we can have it in macports-ports, that would make It already available in $PORTS. But this seems to be a bad idea.

On my system, tclsh is here: /usr/bin/tclsh.

Arjun Salyan

@arjunsalyan

Maybe I shall ask Rainer or Ryan for the path to tcl. It is should be /bin/tclsh.

And once we finalise the location of portindex2json.tcl, I will create a PR. After that more lines will need to be added to put it to RSYNCROOT

Mojca Miklavec

@mojca

@umeshksingla: I was wrong: https://devcenter.heroku.com/articles/postgresql-indexes#b-trees-and-sorting

I would simply clone the full macports-contrib repository (or well ... you could fetch just the latest version of that file) and put it next to the cloned macports-ports & base

Don't wait for their answer, just put it somewhere and fix it if someone complains.

Mojca Miklavec

@mojca

If you want to test https://macports-gsoc.zulipchat.com ...

Umesh Singla

@umeshksingla

@umeshksingla: I was wrong: https://devcenter.heroku.com/articles/postgresql-indexes#b-trees-and-sorting

i got to know about the order's relevance to null values myself.

Where communities thrive

People

Repo info

Activity