I've been looking at clojure for awhile, now. I liked lisp, back in the day, but never got particularly good at it. Since then, I've done some minor projects in Scheme (Chicken scheme, to be specific). The syntax and programming styles really "did it" for me. Problems just seemed simpler to solve.
When I was working on an earlier revision of this project in .NET, I passed up on F#. I used it for some small test apps, liked it a lot, but decided against it. The main reason is that nobody else is using it. This might change with its inclusion in VS.NET 2010, but I'm not going to hold my breath. Besides, C# has a lot of functional-like syntax these days, so while I may miss some of the F# sugar (pattern matching, for example), I don't feel all that hemmed in with C#.
This next chunk of code I have to write for my little indexing project is discrete from the rest of the project. If clojure doesn't take off as more than an interesting niche language, I could easily find myself replacing it with bog-standard Java.
'Cause I have to admit it: Java sucks. It isn't that it is hard, it's that it is a pain in the ass. In some ways, I preferred programming in C - I spent a lot less time working, it seems. Maybe because I did so much in C; I don't know. I do know that just about every language I've tried since then (except PIC) has been less of a hassle.
I do like all the JVM application containers, though. To me, that's the real winner for Java.
Which leaves me with the whole "changing horses in midstream" problem. I really should write the whole thing in one language. You wouldn't think that would be too much to ask, would you? I certainly wouldn't hesitate to ask it of someone else.
So, I'll probably keep poking around through the tutorial. I may end up with working code which I end up using for the next bit. At the very least, I'll have a good idea as to whether or not this was a good idea.
Showing posts with label project. Show all posts
Showing posts with label project. Show all posts
Thursday, October 8, 2009
Monday, August 31, 2009
Where's That File?
This post starts out with a reading of another blog, but it isn't outright babble. It's about what I'm working on.
The author of this article claims "you have to think of content entirely abstractly". While there is some exposition as to what it should look like, it is very vague: "your system should be capable of managing any kind of content."
Fair enough, but how?
Well, that's what I've been working on. I think that the various types of data are best handled by programs specifically designed to handle that data. What we as users need is an easy way to find it.
The current solutions tend to involve centralization, synchronization, and search. You're supposed to keep all the important data centralized, if you need to organize it your own way then you synchronize it, and if you're looking for something you search for it.
Which is great, except that users don't do this, because it all sucks.
If I download file from the internet, that file exists in two places which I can get to. My download folder, and the original link. If I copy it up to a CMS, now it is in three places. If that CMS is backed up, it exists in four places. Copy it to a thumb drive? Now I'm up to five.
Five copies of the same file, in locations which are all equally valid, and all have their strengths and weaknesses. Between them, the data is unlikely to be completely irretrievable.
Now, as a user, all I want to know is "where's that file?" (thus the name of the project)
The author of the original article was correct in that the only important thing is the metadata. What he doesn't seem to get is that the metadata is the only content which needs to be managed.
Currently, the problem I'm solving is strictly a question of duplicate files on the network. I have files that I know must be backed up, but I don't know where all of those copies are. I don't want too many copies, because storage costs are on a rising curve: Each additional terabyte costs more than the previous terabyte.
Turns out, solving this problem isn't easy (my first naive implementations didn't scale), and a whole bunch of the work can be extended to other storage sources.
Having that, though, the next obvious step is to include personal metadata (tags, descriptions) to the files. You have to collect and index metadata, anyway (file name, size, etc.), so why not add user metadata, too?
What I'd expect to see at that point is a UI which reflects the various metadata. If I'm looking for my resume, I should be able to not only find "resume.doc", I should know about all of the copies of "resume.doc" I know about, even if I can't get to them. I'd prefer that the "nearest" one be highlighted in some way, things like that.
What I'd like to do after that (as if I didn't want to do enough), is assign rules to various tags. If I label something with "important", then it should be included in a special backup/sync/whatever. Again, this isn't something that will be particularly difficult, but will require effort.
Well, that's cool, but what about other storage sources? Those are a bit harder, and generally specific to that storage (email, for example). However, things like links to articles and downloads is pretty straightforward, and shouldn't be too hard to include.
Where am I now?
Heh. I mentioned that looking for duplicate files is harder than I thought it would be. I'm actually on my third try. The first one was when I thought "I can do this with a script", the second was with .NET, where I aimed bigger, but found not nearly big enough.
So, I've just completed the work on the file crawler, and the next bit is submitting the crawl results to the index. I've done this part before, and I don't expect it to be particularly hard, but I have to find the time for it. After that, something resembling a UI (I am trying to solve a problem), then put the whole thing out there with a big fat "alpha" disclaimer (probably Apache license, since I'm using so much of their stuff).
And that's what I'm doing, and where I'm at.
The author of this article claims "you have to think of content entirely abstractly". While there is some exposition as to what it should look like, it is very vague: "your system should be capable of managing any kind of content."
Fair enough, but how?
Well, that's what I've been working on. I think that the various types of data are best handled by programs specifically designed to handle that data. What we as users need is an easy way to find it.
The current solutions tend to involve centralization, synchronization, and search. You're supposed to keep all the important data centralized, if you need to organize it your own way then you synchronize it, and if you're looking for something you search for it.
Which is great, except that users don't do this, because it all sucks.
If I download file from the internet, that file exists in two places which I can get to. My download folder, and the original link. If I copy it up to a CMS, now it is in three places. If that CMS is backed up, it exists in four places. Copy it to a thumb drive? Now I'm up to five.
Five copies of the same file, in locations which are all equally valid, and all have their strengths and weaknesses. Between them, the data is unlikely to be completely irretrievable.
Now, as a user, all I want to know is "where's that file?" (thus the name of the project)
The author of the original article was correct in that the only important thing is the metadata. What he doesn't seem to get is that the metadata is the only content which needs to be managed.
Currently, the problem I'm solving is strictly a question of duplicate files on the network. I have files that I know must be backed up, but I don't know where all of those copies are. I don't want too many copies, because storage costs are on a rising curve: Each additional terabyte costs more than the previous terabyte.
Turns out, solving this problem isn't easy (my first naive implementations didn't scale), and a whole bunch of the work can be extended to other storage sources.
Having that, though, the next obvious step is to include personal metadata (tags, descriptions) to the files. You have to collect and index metadata, anyway (file name, size, etc.), so why not add user metadata, too?
What I'd expect to see at that point is a UI which reflects the various metadata. If I'm looking for my resume, I should be able to not only find "resume.doc", I should know about all of the copies of "resume.doc" I know about, even if I can't get to them. I'd prefer that the "nearest" one be highlighted in some way, things like that.
What I'd like to do after that (as if I didn't want to do enough), is assign rules to various tags. If I label something with "important", then it should be included in a special backup/sync/whatever. Again, this isn't something that will be particularly difficult, but will require effort.
Well, that's cool, but what about other storage sources? Those are a bit harder, and generally specific to that storage (email, for example). However, things like links to articles and downloads is pretty straightforward, and shouldn't be too hard to include.
Where am I now?
Heh. I mentioned that looking for duplicate files is harder than I thought it would be. I'm actually on my third try. The first one was when I thought "I can do this with a script", the second was with .NET, where I aimed bigger, but found not nearly big enough.
So, I've just completed the work on the file crawler, and the next bit is submitting the crawl results to the index. I've done this part before, and I don't expect it to be particularly hard, but I have to find the time for it. After that, something resembling a UI (I am trying to solve a problem), then put the whole thing out there with a big fat "alpha" disclaimer (probably Apache license, since I'm using so much of their stuff).
And that's what I'm doing, and where I'm at.
Monday, August 3, 2009
Fun With Projects!
Way back in the day, around April or so, I was talking about a project which was taking up my time. Yes, it is still coming along nicely. We're playing nice with ActiveMQ, Java, the whole bit.
It has taken awhile to get as far as I have. It isn't that the underlying concept is all that difficult ("index files"), it is the scale at which I want to do it. So, there's been a lot of internal abstraction going on, with all of the attendant complexity (lots of little files).
What I'm really happy about is the overall process I've been following. I've been a proponent of tests and mocks, and I've used them a lot in my projects before. The one mistake I always made, that everyone always makes, is losing discipline - giving into the urge to cut a corner. After all, I won't need a mock for that class, it's too simple, right?
I haven't done that this time around. It is really paying off. I haven't been able to devote 100% of my time to this, so I've walked away more than once. I have had no trouble picking up where I was. New pieces work excellently with older pieces, and I barely question the predictability of anything I've done so far.
There's more work to do, of course. I see the light at the end of the tunnel, though. There's some obvious performance changes I can make, but once I've got the basic "duplicate files" functionality going, I'll post it all someplace.
It has taken awhile to get as far as I have. It isn't that the underlying concept is all that difficult ("index files"), it is the scale at which I want to do it. So, there's been a lot of internal abstraction going on, with all of the attendant complexity (lots of little files).
What I'm really happy about is the overall process I've been following. I've been a proponent of tests and mocks, and I've used them a lot in my projects before. The one mistake I always made, that everyone always makes, is losing discipline - giving into the urge to cut a corner. After all, I won't need a mock for that class, it's too simple, right?
I haven't done that this time around. It is really paying off. I haven't been able to devote 100% of my time to this, so I've walked away more than once. I have had no trouble picking up where I was. New pieces work excellently with older pieces, and I barely question the predictability of anything I've done so far.
There's more work to do, of course. I see the light at the end of the tunnel, though. There's some obvious performance changes I can make, but once I've got the basic "duplicate files" functionality going, I'll post it all someplace.
Thursday, April 16, 2009
Solr Indexing
Rather than go the DBMS route for my file index, I decided to go with Solr, from the Apache Foundation. I want to keep the index away from the data - for what I have in mind, they are separate things. It doesn't mean there won't be a regular database sitting around someplace, with its own indexes, it's just that I wanted this specific index to stand on its own.
Solr does everything that I would've written, anyway. It accepts and returns data over HTTP, as XML or JSON. Replication, caching, and a bunch of other shiny and fun things. Since it is running as a Java servlet, there's all kinds of stuff you stick in its way, and mega-configurable.
Retrieving data is pretty fast. Insertions, not so much. There isn't much data going back and forth, and I realize that insertions are, by their nature, slow. I suspect I could get better performance with fewer indexed elements, spread out among solr instances (the Servlet aspect makes this pretty easy).
One of the slowdowns I've pinpointed has to do with the .NET WebClient class. It sucks. It uses some Windows-provided HTTP API, and it isn't happy with multiple threads. It also limits itself to two connections at once (then appears to deadlock - but it might be my code). The HTTP specification says only two connections at once, and I think there's a way to override this in the Windows' registry.
Not interested in that path, especially if it is going to be this slow.
At the moment, there's a commit that occurs after every add. This is another slowdown, but the library I'm using, SolrSharp, doesn't make doing it another way very pretty. It shouldn't be too hard to fix - I've been dinking around in the code, and I see what changes I have to make. Its just a lot of refactoring. Bleh.
So, on the todo list are:
Solr does everything that I would've written, anyway. It accepts and returns data over HTTP, as XML or JSON. Replication, caching, and a bunch of other shiny and fun things. Since it is running as a Java servlet, there's all kinds of stuff you stick in its way, and mega-configurable.
Retrieving data is pretty fast. Insertions, not so much. There isn't much data going back and forth, and I realize that insertions are, by their nature, slow. I suspect I could get better performance with fewer indexed elements, spread out among solr instances (the Servlet aspect makes this pretty easy).
One of the slowdowns I've pinpointed has to do with the .NET WebClient class. It sucks. It uses some Windows-provided HTTP API, and it isn't happy with multiple threads. It also limits itself to two connections at once (then appears to deadlock - but it might be my code). The HTTP specification says only two connections at once, and I think there's a way to override this in the Windows' registry.
Not interested in that path, especially if it is going to be this slow.
At the moment, there's a commit that occurs after every add. This is another slowdown, but the library I'm using, SolrSharp, doesn't make doing it another way very pretty. It shouldn't be too hard to fix - I've been dinking around in the code, and I see what changes I have to make. Its just a lot of refactoring. Bleh.
So, on the todo list are:
- a better HTTP client
- modify the addition of records to allow batching
Subscribe to:
Posts (Atom)