GSoC Week 11

In my last entry I mentioned I had already sent a patch, which would allow Darcs to have a better handling of bad caches, I'm happy to announce that my patch is now in the current head :).

Now we have a better handling of bad entries in the cache, some of the main benefits of this fix is that some operations that would take more than 10 minutes were now reduced to less than 1 minute, the time changes depending the number of patches in the repository( look my report of week 8 for a better idea of what was happening).

Also I mentioned I was having some issues trying to use the timeout function from System.Timeout in Windows, I wrote to haskell-cafe and someone claimed that It would work for him in Windows 7, and then other person claim the same behaviour I was having with Windows XP, then I tried in a friend's computer, and it'd work intermittently, so the conclusion is that this is a Windows systems related issue and not a GHC's as I though at the beginning, I would really appreciate if someone with more experience with Haskell and Windows systems could point out what is really happening.

For the coming week, I would focus on extending the documentation, sending my implementation for the timeout flag, and completing the section "Future work" in the high level document.

GSoC Report: Week 5

In my last post I said I had complete the warm-up phase and that I will start to think about how to handle caches which are no longer available.

The caching mechanism relies in the files _darcs/prefs/sources and ~/.darcs/sources, basically the content of those file is used to generate the cache entries, each of the entries in that file indicate an alternative source to get files. If we want to specify global caches we put that in ~/.Darcs/sources but if we want an alternative repositories to pull from, we specify that in the repository sources file which is in _Darcs/prefs/sources, also each time we do a pull from an external repository it is added to the sources file
automatically.

The problem of expiring caches is given because sometimes it happens that repositories that were available, can become unavailable. For example if I had pulled from 3 different repositories and 2 of them stop being available, it could take up to 2 minutes to get each patch, because Darcs could try to fetch every patch it needs from those 2 not longer available repositories, it tries to establish a connection, an then waits for a time-out or a bad response code. After the problem I just mentioned, the idea is to design a mechanism which can help Darcs to establish which entries should be expired.


We can split the cache entries in two groups: locals and remotes. Dealing with local non-longer reachable repositories is not a big deal since if we don't find the local entries we can assume they don't exist and we can drop them from the cache and stop trying to fetch files from them. Remote repositories are more tricky, for example I can't eliminate an entry just because it gives a time-out when it tries to establish a connection with it, there are other external factors which could interfere with that particular entry in a given moment
(firewalls).

So as we seem this is not an easy task, handling remote repositories is out of our hands, we don't have control over the external sources, we don't have control over the network configuration and so on. So a first approach to this is to mark the entries which are not working and ignore them for the rest of the pulling since we don't want to try to establish a connection with an entry which we know is not available. If we try to establish a connection and fails we can mark it as a bad entry but also I think it could be awkward to wait for a 60 seconds time out, something we could implement is a default time for waiting for a connection to succeed (10-15 seconds maybe) if it doesn't happen between that time we can skip it, mark it as bad, and don't try for the rest of the patches that particular entry. Other approach suggested in the bug tracker was to try to establish a connection with all of the entries and use the one that responds first but then what if all the entries are bad entries?. I have to think more about it, I will discuss on irc and the mailing and then with a clearer idea I will start to code a patch to solve the issue.

Also I sent a first version of a failing test for the case of unreachable entries, but I need to amend, as there are some missing cases.

More of my progress can be found in the wiki.

GSoC Report: Week 4

I have completed my phase of warm up issues which was oriented to allow me to get familiar with the Darcs system, specifically the cache part. I have sent the patches and they have been applied.

During the week I worked in finishing issue 1176, continued with the documentation part, more specifically describing in a higher level how a patch is fetch with a given hash, making easier for someone with a non-technical background understand what is happening, fixing a test which was failing in Windows when it shouldn't and finding out why the IO operations over hashedRepos were put in a different module.

For fixing issue 1176 I did some modifications over some code I had already sent which was about keeping the caches sorted by locality, the initial idea was that anything which was local should be first in the list and then the remote sources, but we realized that between the remotes also exist a "wanted" hierarchy, basically we would prefer to access first http repos over ssh, so the new sorting keeps all the locals first, http in the middle and ssh repos at the end. One of the problems this solve is that weird behaviour of darcs trying to establish a ssh connection when pulling from a http or local repository.


While I was working in the test which was failing we redefine what gets saved in the _darcs/prefs/sources, the global caches were one of those things, it wasn't necessary to have them there because that's why a global cache configuration file exist (~/.darcs/sources), so basically we just drop anything related with global caches before saving the sources file of a repository.


For the next week I will start to work in the problem of how to deal with the unused cache, the idea is by the end of the week to have a work plan and write a test case.

You can check out more of my progress in the wiki.

Improving Darcs Performance - My GSoC Project.

I was accepted for this year GSoC, I will be working for Haskell.org and my project will be focus in improvements of Darcs performance, specifically I'll work in the global cache, you can find a full specification of my project here.

I'm pretty excited about it and really looking forward to start working on this, also my mentor would by Eric Kow (a.k.a kowey).

I'll try to keep weekly updates of how my work is going, and let you know about the ups and downs that I find in my way.

Thanks Haskell.org, thanks Darcs and last but not least thanks Google for given us this awesome opportunity.