Google Summer of Tor 2010 - JTor Hidden Services

GSoC summer of Tor near the end.

Today is the unofficial pencils down date. Too bad I am not ready to put my pencil down yet. I have until Friday to pull everything together and complete what I have left, because that is the official pencils down date. I have had a lot of obstacles when facing the challenge of completing my project - both personal and comprehensive. For example, my HDD decided to not want to mount last week while I was out of town. Luckily I was able to get a new one salvage the data.

There are a few bugs currently in the JTor code that I am not mantaining. The first being a problem with the directory fetching and storage of the list of routers. The list gets fetched and stored, but, I believe, that when the initial circuits are created the list gets emptied. The fix I have used for this is to just reload the router information before I have to use it.

The second bug is a problem with the circuits connecting a stream to a destination. The circuits are created upon initialization, and when using the API to attempt to connect to a server over a Tor stream the stream rarely connects. This is apparently caused by JTor wanting a certain type of circuit but it not existing, so it waits indefinitely. These bugs among other missing functionality make it difficult to have a fully functioning Hidden Services. But I am providing as much functionality as I can so that when those features are fixed and others finished, it will work. In addition, since I am using test driven development I am writing all the tests to ensure the working state of hidden services come these features.

There are a few things that I have added placeholders for in the JTor code that do not exist yet. The first is creating circuits for Hidden Service connections. Currently the only circuits allowed are three hop ones and there wasn't a spot in the API to do otherwise. Also the CircuitManager class needs the ability to create exclusive circuits that will not be used regularly, like the ones between the hidden service and its introduction points.

Looking at the list of features for my original proposal below - I am almost there. For the publication parts - I have completed the V2 Service Descriptor, the advertising of the descriptor. The bad directory checking needs to be added - that will be simple, because it is just a fetch of the descriptor after it has been published. Establishing of introduction points is difficult to verify because of the bug with sending data over a circuit. Fetching of the service descriptor, parsing, verification and periodic fetching have all been implemented. Only descriptor cookies are supported for authentication mechanisms right now. The cells have been created but not tested. I still need to write the tests for them and make sure they are complete. For configuration, I have implemented the ability to create a hidden service from a Tor client hidden service folder. As for the authentication mechanisms, I am going to have to cut those off at descriptor cookie auth for the summer. I am still yet to merge my changes into the rend-spec that Karsten had modified. I am going to be working on the things mentioned as incomplete above until the end of the week. All these changes will be on my github.

Progress Report 2

I am not posting a pie chart for this progress report, because it would look like pac-man with the big slice reading something like 'silly bug hunt.' I spent way too much time trying to fix the encoded output of the onion address. It turns out that the problem was that I was using the base32 encoding described in rfc4648 instead of rfc3548. It was a very big noob moment for me. Luckily it was a learning experience as I spent a lot of time making sure the PK was encoded correctly and that the hash was being performed correctly. But the reason it was a silly bug was because I hardly considered having the wrong rfc as the source of my problem. Either way I wasted some time that I really didn't want to. The good news is the onion address is now being created correctly (hooray?). I am going try not to dwell on it and focus on what I have done and what I am going to do in the next two weeks.

I believe the output created by the JTor V2 Service Descriptor is accurate. My next step is to write the part of the client that fetches and parses the V2 service descriptor. That way I can run my test Hidden Service through Tor, fetch the descriptor through JTor and test it against the descriptor that I created. I say this so that I can work on parsing/fetching service descriptors, in addition the content of the service descriptor changes over time because of how the descriptor id is calculated. I can't compare the values without getting a recent service descriptor from directory services.

As for the documentation, I have been focusing on rendclient and rendservice, the endpoints of hidden services and not so much on rendmid yet. I have found that the spec is lacking a lot in the parts for client authentication mechanisms. That is pretty obvious when you check out rend-spec and section 2's text just reads 'foo.' I have been making notes about the changes from the proposals and added a new-rend-spec.txt to my git repo. I just need to dig into the tor code to be sure about what was actually implemented from the proposals.

I am planning on working my ass off these next few weeks, because I feel kind of behind from losing a bunch of time this past week due to that silly bug. Also, after talking to my mentor I am trying to make the steps to be more vocal about my road blocks instead of being stubborn for fear of sounding noobish. I had told him about my problem initially, but there was some miscommunication where he thought I had solved the problem. Bottom line is that the more we discuss the better off I am.

Over and out.

Progress Report

Two weeks into SoC already - that was fast. To make this exciting I made a pie chart of what I have been spending my time on (pie charts are exciting).

I have been spending a lot of my time going through the existing Tor code and seeing how it is implemented. This was a very daunting task initially, but Eclipse has made that task much easier. I have been refactoring my old code, making sure it is correct and writing Tests. I wrote some code to initialize a Hidden Service from a Tor hidden service directory, so that it would be easily compatible with that configuration. Currently I am working on making one of my tests pass - creating the same onion address for the service. Which, at the current point is not working - I believe it has something to do with the way I am hashing in the info or how I am converting it to base32 chars.

My plan for the next two weeks is to drive on with tests. The next test I am writing is to make sure that the V2 Service descriptor created is the same as Tor. Once that part is successful, I will write the tests for establishing intro points and publishing the service descriptor. Once all of those pass, publishing will be mostly done (with the exception of client authentication, which will probably not fit in these two weeks). Be sure to check my github to see my code, any suggestions are encouraged.

So it begins...

Hooray, GSoC is finally here - and I am a little behind. Spent the first two days of GSoC moving from Philadelphia to Austin. But now I am settled and I have the internet, and so it begins. I am going to take on the V2 Service Descriptors first. The Service Descriptor holds the information necessary to connect to the Hidden Service. I am going to need a way to ensure that the service descriptor that JTor creates would be the same as the one created (with the same keypair, port etc.) by Tor. So I need to create a method to facilitate this testing along with the actual descriptor code.

Photo of Bob by Alice

My friend whipped up this spoof on the painting "The Son of Man" by Magritte, that is an onion in front of his face.

JTor Google Summer of Code 2010 Proposal

�

Kory Kirk

Google Summer of Tor: JTor's Hidden Services

1.��What project would you like to work on? Use our ideas lists as a starting point or make up your own idea. Your proposal should include high-level descriptions of what you're going to do, with more details about the parts you expect to be tricky. Your proposal should also try to break down the project into tasks of a fairly fine granularity, and convince us you have a plan for finishing it.

�

�� I would like to work on JTor! Last semester I attended a class called Distributed Systems. For part of this class we had to take on a semester-long project where we implemented (part of) a distributed system. Most of the other students in the class took projects where not implementation was needed. I chose to take a go at Hidden Services.� Having recently dined with Roger and Jake, I decided to take on a part of JTor as my�class�project. JTor was just starting out, and because of its infantile nature I was suggested a task that was easily compartmentalized - Hidden Services.�

�� I tried to write�Hidden Services�with the same style as the rest of JTor. I�scoured the�rend-spec and made my own notes. By the end of the semester came, the project was not near usable completion. I had gotten a lot of the basic components of the Hidden Services done, like the V2 descriptor, but did not get a chance to pull it all together into a working part of the library. And some parts were left largely untouched like connecting to a hidden service.��A log of the work (and some assignments) done can be found at�http://korykirk.com/CSC8530/

�� For GSoC, I will complete JTor Hidden services. I believe one of the most important parts will be ensuring that accessing and publishing a HiddenService through JTor should look identical to Tor on the wire.� Being that I will be building this project up from scratch, and referring to the implementation of Tor Hidden Services, I would also like to update rend-spec to reflect any inconsistencies between the spec and actual implementation�(due to possible undocumented revisions, partially implemented proposals, older protocol compatability, etc.).�I foresee the rewrite as a gradual�process that would span�the length of�GSoC.�

Below I have outlined what I believe to be the major components of the project:

JTor Hidden Services

main focus points: Hidden Service publication and Hidden Service access.

�

��Hidden Service Publication
- V2�Service Descriptor
  - �Currently implemented V2 service descriptor�as described in rend-spec.
  - �Need to verify correct encoding of descriptor-id and descriptor content.
- Advertising Service Descriptor
  - �Choosing�non-adjacent directory servers to be responsible�(114)
  - Publishing� to 4 non-consecutive directory nodes, and 2 consecutive (for 0.2.0.x support)
- Bad directory reporting
  - check if responsible directory server 404ing to the service descriptor request.
    - http post complaint if 404�
- Establishing introduction points
- �
  - RELAY_ESTABLISH_INTRO�cell
  - post request to the directory server
  - Organizing new key pairs for each introduction point.
- Accepting hidden service connections from clients.�
Hidden Service Access
- Fetching V2 Service descriptors
  - parsing and� caching V2
  - verifying descriptor-id from v2
  - periodic fetch
- Establishing Rendevous points
  - circuit
- Optional client authentication with auth-data or descriptor cookie
- Connecting to hidden service servers.
Cells
- Lots of cells need to be created (which shouldn't be too hard becauese of CellImpl)
- �
  - RELAY_ESTABLISH_RENDEZVOUS,�RELAY_ESTABLISH_INTRO, INTRODUCE2,�RELAY_INTRODUCE1V,��INTRODUCE2 v3,��ESTABLISH_INTRO v2
Configuration file
- Retain settings�and data�for a specific Hidden Service.
- Allow for granular control� of�HiddenService, its�supported protocols and authentication mechanisms.
- use JTor to interface w/ config or edit by hand
Authorization�mechanisms
- Client authorization at directory
- Client presence of descriptor cookie�(partially implemented)�upload Multiple descriptors with different descriptor cookies delayed (< 30 sec)
- �Client authorization at introduction point
- create abstract class for Authentication Mechanisms to be implemented in the future
- Currently do not plan on implementing any directory or intro point authentication mechanisms, but will allow for use of them. �

�

�Roadmap:

�� Week 1-4: Finish V2 Descriptor, Implement HiddenService publishing to directory services, checking for bad directories, relevant cells and the hidden service configuration file. Finish publish portion of �Hidden Services. �Ensure that rend-spec correctly reflects everything needed to implement publishing.

�� Week 5-9: Descriptor fetching and verification, establishing and creating circuits to rend points, connecting to a hidden services, �implement relevant cells, support for authorization mechanisms, update any part of rend-spec that is inaccurate for accessing hidden services.

�� Week �10-12: Documentation and rigorous testing, prepare for release of JTor - help with any addition documentation, tests or changes that need to be made before release.

2.��Point us to a code sample: something good and clean to demonstrate that you know what you're doing, ideally from an existing project.

http://github.com/koryk/JTor� - my contribution is the hiddenservices package (and I think one method somewhere in crypto).

http://github.com/koryk/GAC�- Genetic Algorithm Comparator � this is a project I am still working on for my graduate thesis � I wrote it using the dANN framework to gather comparative data about the Genetic Wavelet Algorithm and Simple Genetic Algorithm.

https://github.com/ajpalkovic/b2010/tree/kory�- This is the code for my team�s submission to this year�s MIT Battlecode programming competition.

http://korykirk.com/java.html�- this is a page with a bunch of older projects from when I was an undergrad.

�

3.��Why do you want to work with The Tor Project / EFF in particular?

Being involved with the Tor project last summer has really sparked an involvement in the open source community for me. I like the underlying idea behind Tor and especially the idea that it helps people with oppressive governments access the internet (because I want people to visit my website (just kidding, because I love the internet)). I recently became a member of the EFF in December.�

�� I first heard about the EFF in my Ethics in Computing class (Which I am now a teaching assistant for), and was immediately drawn to the EFF after reading �Coming into the Country� by John Perry Barlow.� The metaphor of the cyberspace frontier really resonated with me as to why the internet is so special, how it is a new frontier that needs to be kept free and protected. I think the EFF and Tor are doing a good job of that, and so I want to contribute in any way I can.

4.��Tell us about your experiences in free software development environments. We especially want to hear examples of how you have collaborated with others rather than just working on a project by yourself.

Last summer, working on Torbutton was my first contribution to any open source project. Since then I have a part of a few open source projects including JTor, dANN (Java AI framework) and imgur Firefox extension. All of the collaboration I have done has been over IRC.� I spent a big chunk of this semester working on an Java AI programming competition hosted by MIT called Battlecode. This was a collaboration between a few students, writing our team on and using git for version control. I also participated in the ACM ICPC (international collegiate programming contest), where a team of 3 has 5 hours to do 8 programming problems. My team used Java and placed 2nd in the local competition (greater Philadelphia area) and 9th in the Regional competition (Mid-atlantic region). I have a lot of experience working closely with others on code.

�

5.��Will you be working full-time on the project for the summer, or will you have other commitments too (a second job, classes, etc)? If you won't be available full-time, please explain, and list timing if you know them for other major deadlines (e.g. exams). Having other activities isn't a deal-breaker, but we don't want to be surprised.

This summer I have no obligations. I am finishing up my Masters next month, and moving back to Texas where I plan to spend the summer looking for a job (for after the summer), working on GSoC and maybe doing some freelance stuff.

�

6.��Will your project need more work and/or maintenance after the summer ends? What are the chances you will stick around and help out with that and other related projects?

I think that my project will need maintenance, and I would like to be a part of JTor and work on it in the future, not just this summer. Java is my best language, and I think it would be the area of Tor I can contribute to the most. Now that I am done with school, I will have a lot more time to dedicate myself to personal projects instead of academic projects, and I consider JTor a personal project .

�

7.��What is your ideal approach to keeping everybody informed of your progress, problems, and questions over the course of the project? Said another way, how much of a "manager" will you need your mentor to be?

��I do not think my mentor needs to be much of a manager, I am usually self-motivated. I would like to use my mentor as a backboard for ideas or a feedback machine rather than as a boss-figure. Last year I kept a blog of my progress, and I will do that again and document and describe my progress, changes to the code, and obstacles. My git commit messages are a good way to find information about progress on smaller steps of the project.�

8.��What school are you attending? What year are you, and what's your major/degree/focus? If you're part of a research group, which one?

�

I am attending Villanova University outside of Philadelphia. I am in a Computer Science Masters program that will be finished in May. I am a graduate assistant so I owe the Computer Science Department 20 hours a week. My assignment from the past two semesters includes: teaching assistant for three Algorithm and Data structure classes (Java programming) and Ethics in Computing, the Research Assistant for Machine Learning and Data Mining class, and worked at the computer science help desk. The help desk is a one man tutoring center where any computer science student can come ask help on projects mostly in Java or C.

The research I have been doing has been my own. Last semester I did an independent study which I have continued to research as my Master�s thesis. The project is surrounding a new type of genetic algorithm called the Genetic Wavelet Algorithm (GWA). I have been doing research to formally describe it and compare it to the classical genetic algorithm. The GWA is implemented in a open source AI library called dANN,�http://wiki.syncleus.com/index.php/DANN, parts of my thesis can be found in the Genetic Wavelet section. I have also written code to contribute to the implementation of the Genetic Wavelet Algorithm.

9.��How can we contact you to ask you further questions? Google doesn't share your contact details with us automatically, so you should include that in your application. In addition, what's your IRC nickname? Interacting with us on IRC will help us get to know you, and help you get to know our community.

��I am almost always on freenode (in various channels), usually on oftc in #tor and #tor-dev. I usually operate under the nick koryk or koryk1 (sometimes toryk). You can also e-mail me at�kory.kirk@gmail.com.

�

10.��Is there anything else we should know that will make us like your project more?

�� I believe I can complete this project resulting in a robust and flexible package for the JTor library. Other than that, there is nothing I can think of that will make you like my project more, so here are a few random facts about me:

- My great uncle invented the footlong hotdog (I know... how could someone invent a long hotdog? I was just always told this by my family growing up)

- I can bend my finger back to touch my hand.

- I have a genetic mutation that causes me to have freakishly low cholesterol (I am one of the X-Men)

- I always try to program a task in as few lines of code as possible

��In addition here is a short brainstorm of possible other names for the project beside JTor:

- Onion Jar, layered ogre, �coffee leek (like leek a type of onion), jleek, jbulb, onion bulb, Allium

�� those are all onion or java related, if you want more suggestions let me know. ��

�