Beside [url=http://dict.leo.org]LEO[/url], there are two very nice services on the web for writing a better language. [url=http://sara.natcorp.ox.ac.uk/lookup.html]Sara Natcorp[/url] searches for a given phrase in english literature text, which allows you to check quickly if a chosen phrase is appropriate or not. Recently I found something similar for [url=http://www.redensarten-index.de/suche.php]german phrases[/url].
Archive for October, 2005
Parallelize me
October 24th, 2005 by PeterThe [url=http://grid-occam.org]GridOccam-Project[/url] brings up some interesting discussions in our group at HPI. We are currently discussing about the applicability of GridOccam as coordination language for distributed tasks. We started from the assumption that there is an already existing implementation of an algorithm, which we want to parallelize. The two examples were ‘Game of life’ in Python and a specific physical simulation, written in C. Two independent persons where equipped with the GridOccam compiler and the original application in source code.
We expected the persons to choose the following approach:
[list]
[*]Identify compute-intensive part in the original code
[*]Develop the parallelism in Occam
[*]Call the original computational code in each parallel task
[/list]
Instead, both of them inspected the algorithm and developed it completely from scratch in Occam. This is also the usual solution in MPI-world – if you want to parallelize your problem with MPI, write a new program from scratch.
The authors told us that after analyzing the algorithm for making it parallel, the step of writing everything down in Occam is more or less a mechanical thing. Things simply become not easier if you can re-use the sequential code of the algo, since the parallelization is the tricky part.
What we now need to do is an investigation of practical use cases for coordination languages (like Condor DagMan). It can be expected that a coordination language is mainly for cases where you have non-dependent parts (like parallel rendering) and the source code is not available. In all other cases, rewriting is easier.
Origin of ‘middleware’ term
October 21st, 2005 by PeterAccording to Wikipedia, the term ‘Middleware’ arised originally in a [url=http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PDF]NATO paper[/url] from 1968. If you take this text as the ultimate truth, then Java & .NET are pure middleware platforms, and J2EE is an application. I know people which would not agree to this 
…
DRMAA job states
October 19th, 2005 by PeterIn today’s DRMAA phone conference, we had an interesting discussion about job states. Two results are important for developers building applications on-top-of DRMAA:
The spec is not able to describe the semantics of the different drmaa_control actions. In fact, it only promises that one state follows after the other and that drmaa_job_ps returns one of them. This means that an application developer should not expect any specific behaviour with HOLD/RELEASE and SUSPEND/RESUME control actions in drmaa_control. Only the TERMINATE action seems to be semantically clear over all DRM systems.
The problem occured in my current work on the Condor DRMAA implementation. Since the library is working in the vanilla universe, checkpointing is not available. To fulfil the DRMAA spec, I will fall back to a [i]condor_hold[/i] in case of a SUSPEND request. This will lead to a restart of the job when drmaa_control(RESUME) is called.
With this issue in mind, a similar thing must be considered in drmaa_job_ps(). If you started your job on hold, drmaa_job_ps() might return DRMAA_PS_SYSTEM_ON_HOLD, DRMAA_PS_USER_ON_HOLD, or DRMAA_PS_USER_SYSTEM_ON_HOLD. All of them are valid results, depending on the underlying DRM-specific implementation.
FTP transfer from a shell script
October 18th, 2005 by PeterIf you want to transfer stuff to a FTP server with a shell script, you can choose the heavy solution with the [url=http://www.faqs.org/faqs/unix-faq/faq/part3/section-9.html]expect tool[/url], or a smart solution with [i]ncftpput[/i] from the [url=http://www.ncftp.com/]ncftp[/url] package.
Condor DRMAA progress
October 17th, 2005 by PeterI know some people wait for the results of my current work on the Condor DRMAA library. As usual, there is much more to do than originally intended. Working on other people’s code base is never fun, but I must admit that [url=http://www.cs.wisc.edu/~njg/]Nicholas Geib[/url] did a great job for the already existing code base. Implementing DRMAA is in fact not an easy task, but with our new DRMAA test suite (donated by the [url=http://gridengine.sunsource.net/]SGE guys[/url]) things are easier to handle.
For the upcoming experience documents, I begin to see some interesting issues. One of them is the [i]join_files[/i] attribute. This feature is not necessarily supported as native function of the DRM, for example with Condor. If you implement it anyway by setting [i]stdout[/i] and [i]stderr[/i] to the same file, you will usually come out with garbage.
Currently I flood the DRMAA list with open questions regarding the spec, which come out during debugging and testing. I think Dan felt the same way one year ago, when he did one of his several DRMAA re-implementations. I plan to reach a somehow stable state until tomorrow’s phone conference, in order to be able to tell some positive things to the DRMAA group …
BTW, if you need a nice overview of Condor status codes, take this list:
http://www.cs.wisc.edu/~adesmet/status.html#condor
Detection of directory changes
October 13th, 2005 by PeterIn my current work on the [url=http://sf.net/projects/condor-ext]Condor DRMAA library[/url], I had the problem of determining if a file in a given directory has changed. Windows has an [url=p://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/obtaining_directory_change_notifications.asp]in-build mechanism[/url] for this, and I used it several years ago to write an enhanced “tail -f” tool for Windows. In Unix, this is not trivial. Linux has an optional [i]file alternation monitor[/i] software, but there is no common (e.g. POSIX-compliant) asynchronous mechanism for this. I ended up with polling the the stat’s of all files in the directory periodically. Any better idea out there ?
Mailman hell
October 12th, 2005 by PeterA user of our [url=http://www.netclients-services.de]ISP services[/url] requested a list renaming and a deletion of archives for his Mailman lists. Both functions are not supported by either the administrative tools or the web configuration frontend. There are several Google hits about this topics, and the [url=http://www.gnu.org/software/mailman/faq.html]Mailman FAQ[/url] has also a very old according entry. After playing around with the tips from the FAQ, I gave up before this collection of filesystem-hacks destroys or list configuration completely. I would assume that writing an according tool is a much better solution than providing a 2 page workaround description.
I also remember that it took quite a lot of time with earlier versions to get an integration in your MTA, without managing all list addresses also in the mailserver. The Postfix guys have some funny how-to pages about possible solutions …
In the end I must acknowledge that mailing list software seems to be a difficult piece of work. Too bad.
EU project review
October 7th, 2005 by PeterAs already stated, I am working in the EU-funded [url=http://www.asg-platform.org/]ASG project[/url]. Last week we had our formal review meeting after the first 12 months. Since this was my first defense meeting at the EU, I learned some simple rules, especially from personal talks with the reviewers:
[list]
[*] Be precise. This sounds natural, but researchers tend to look for complicated explanations to answer simple problems. The reviewers need to ask a well-known set of questions, and you should have the well-known set of answers.
[*] Don’t lie. Simply tell the reasons for a problem in the project, they will find it anyway. Don’t try to generate new explanations based on an assumption of the reviewers mood.
[*] Have a vision. You should be able to express your innovative outcome in not more than 3 sentences. This is a problem, especially in a research-based project.
[*] Give a show. Reviewers are bored, since they have a lot of such meetings over several weeks. Make it interesting, but always keep oriented on your vision.
[*] Know what is around you. Especially in technological research, there is a lot of ongoing parallel activity. You should clearly show that you have investigated other related EU projects – especially those the reviewer know.
[*] Clarify terminology. If you are not able to unify the wording over all presentations, your project will look fuzzy and uncoordinated. Some candidates from our project: “Service”, “Middleware”, “Grid”, “Semantic description”
[*] Have a good moderator. Not anybody, but a person with a good overall understanding of the project.
[/list]
It is needless to say that this rules a quite usual, especially in industry. (European) researchers need more experience in this.
Technology impact
October 7th, 2005 by PeterA really astonishing thing happens to me: People find my blog and ask specific questions. Don’t get me wrong, but I am on the web for nearly 10 years now. I never got such an amount of feedback regarding published information. Blogs seem to work, so I promise to spend more time for this.