UNIX for Advanced & Expert Users

I am posting this gauge the level of interest among the community in forming an open source team to work on an automation harness I am about to make available.

I already have a working POC running at my place of work, but it is not secure enough for production environments. However, I am about to release a more powerful and secure version that is secure enough and offers even more features. I will go into further detail about that later on in this post but in the interest of not wasting anyone's time, here is what I am looking for in the way of people with skills I need:

*) Testers
*) Shell Scripter's (Bash) who can review the code, particularly those with the experience and access to a UNIX environment to make it POSIX compliant. (I am currently developing on Centos). You shouldn't be daunted by the thought of working on a full blown application written in shell either.
*) Project managers with experience in running open source projects.
*) Web masters to put a site together for the project.
*) Code management.
*) Packaging
*) Technical writers for the documentation required.

I would particularly like to hear from anyone with a proven track record in managing an open source project and implementing the processes and procedures required to startup such a venture.

The harness is called MUSE, (Managed Unix Shell Execution) and has the following features:

*) Master - Slave architecture.
*) Plugin code modules.
*) Centralised reporting on master.
*) Automatic report summarisation.
*) Very simple syntax, so shallow learning curve.
*) Agnostic - Will run tools on a distributed cluster of servers that are written in any language, thereby enabling end users to leverage their existing tool sets without refactoring.
*) Event driven, Master and Slave are both implemented as state machines that communicate via a messaging framework.
*) Stateful, current state continuously updated in Sqlite3 databases in master and slave.
*) Secure. Currently implemented via LDAP.
*) Access controlled. Standard NIX user and group mechanisms are used to control who can run what and where.
*) Audited - Everything is recorded internally as well as logged under /var/log/muse.
*) Support for RAD. Feature rich developer tools built in by default but controlled via the access control mechanism mentioned above.

I am currently using the insecure harness to test a Data Warehousing Application in the following areas:

*) Integration testing. Application is distributed across multiple server types.
*) Resilience testing. e.g. Killing processes during data loads, block and unblock ports used by interfaces, consume disc space and memory etc.
*) OAT. Sequencing the upgrade and rollback instructions for operations staff with multiple configurations of distributed servers.

I shall not be releasing the insecure version as it is tightly embedded into my employer's systems and I don't want to waste time anonimising code that I'm not going to release. However the new version is almost ready to share with a team of like minded individuals who would like to be involved in making it generally available.

If you are interested in joining the project then please send me a private message with a potted resume. If I think you are likely to be a help during the early stages of the project, I will ask you for a more formal CV and get in touch with you.

If I think you will be of help later on once the project is set up, I will let you know and keep hold of your details.

Thanks for taking the time to read this.

Brad

We appreciate your interest towards our forum, the best place to discuss these abstract and non-technical career related questions is in our LinkedIn group for forum users, currently boasting over 31,436 members.

We created that LinkedIn group specifically for people to discuss career related questions, speak with headhunters, headhunters to post job listings, ask about business opportunities, and similar career (non-technical) related topics.

In addition, we also have rules and guidelines over at our LinkedIn group that all technical questions be posted here in the forums (because we have better search capabilities, code tags, and a better environment for technical Q&A), so this forum is for technical questions -- Q&A -- and our LinkedIn group is for career and job hunting Q&A.

Thanks Akshay

Actually I'm not offering to hire anyone, I'm looking for volunteers for an open source project. But I will try to find your linked in group and put this up there.

Thanks again

Brad

What you wrote furiously looks like a job posting though.
If you are only looking for volunteers, you should state it very clearly and probably be more realistic in your expectations. Good luck anyway !

First off, i have to say, i am interested. I could put together some CV covering what i have done so far, but i think searching through the 4k posts i have here will tell you far better what i can or cannot do. I have already written and maintained shell scripts of 8k lines of code, so i have no problems doing whole applications in shell.

In normal circumstances i would point you to Neo, the forums owner, for how to deal with job offerings but on the other hand i suppose from the "open source" in the first sentence it will be volunteer work anyway, so this is not a "job offering" in a strict sense. There are no clear rules about this and as a border case i am willing to let it stand for the moment.

What makes me wonder is this: you want to create some open source software package. Why do you need formal CVs? I'd say if you find people willing to work for nothing you shouldn't be too picky about who they are. If they cannot code (good enough) or do not have the necessary experience or whatever their lack of qualification may be - it will show soon enough, wouldn't it?

So why don't you - instead of asking people for CVs - just post/submit the code along with some documentation, wishlists, known bugs, specifications, ... and see what happens?

Without wanting to anticipate his decisions and without promising anything: there might be chance that Neo is willing to adopt such a project as an additional board activity. You really should talk to him (he won't bite - not when the moon is not full ;-)). On the other hand, it might be better for the project to use a well-established platform like sourceforge, but having not worked for any open-sorce-projects i can't tell.

Still, it would be interesting to hear more about the projects background and the environment it is foing to take place in. Please elaborate.


bakunin

Good advice, thanks.

I might be getting ahead of myself here as I'm not able to release the current version I have working at my employers site. The code contains too many references to proprietary server and project names.

It was only ever intended as a Proof of concept, but of course it then became so useful we couldn't do without it.

I should explain that I work as a senior test analyst with a heavy emphasis on automation across a cluster of server types that host an online Security as a Service suite of applications. This is my 41st year in engineering and I have been developing the new automation harness with a view to replacing continuous integration as a code promotion model with something inherently agile.

My idea is that no developer or tester ever works on a cluster that is more than 24 hours behind production.

To do that I have designed an agile code promotion model and am developing an automation harness that will not only test a release but become a part of the deliverable itself. Having tested that the upgrade and rollback work, it will then deliver the code to the production environment and deploy it.

The POC using blocking ssh connections and is not able to maintain state, and so doesn't meet the requirement. Although I am currently using it for integration, resilience and OAT testing. It runs thousands of distributed tests several times a day via Jenkins, so the development team know within a few hours if they have checked in any duff code. Its doing stuff like jumping on to servers and creating firewall rules to block ports being used by the application under test. As well as consuming disk space on critical servers, leaking memory. For integration testing, its configuring a distributed application, (a data warehouse), with different configurations. Loading data, running 800+ queries through a WREST API and confirming that each returns the expected results for that query against that data set.

I did a presentation on this to a large multinational telecom giant recently, and they offered me Technical Release Manager, but I turned them down as they wanted to close source MUSE.
I haven't put this much effort into it to just sign it away like that.

==========

My new version comprises of a master server running as a state machine and it launches slaves that are also state machines. The communicate via a secure messaging service.

Between the state machines and the messaging service, I am able to implement event driven code on the master that reacts to failures on the slaves and takes appropriate action for the failure type, (Fatal, non fatal). All plugins run by the slaves produce reports that are collected on the master and summarised by the application itself. Typically I am summarising tens of thousands of lines of output from a little over 1600 plugins on every run. Because - via a simple mechanism - I can customise what constitutes an error for each plugin, I can reduce all of that output to a summary file that tells me that an error exists in a single report. So I can immediately home in on the problem without having to wade through every report.

I am hoping that the security model I have implemented - using single purpose keys - will prove effective enough for adoption ion the financial services sector.

And I have implemented all of the above in bash would you believe...

I chose bash shell for two main reasons -

1) It is everywhere. At my last place of employment in the banking industry, the security teams would not allow us to install any languages onto the production servers. So we had to implement all of our tools in shell.
2) By using shell, (rather than stepping down to the shell from a high level language and making system calls), I have been able to leverage tools written in any language. Because the plugins run by the slaves provide a SHELL, (not a shell script) on the remote host, I can run any tool written in any language that can be run on that host. Therefore people picking up the harness can leverage their existing toolsets without refactoring and so hit the ground running.

All of the development has taken place in my spare time for my own amusement, (with the exception of the suites and plugins for my employer).
Frankly, I want my life back
So I need help in the areas mention in my OP.

With regard to putting together an open source community, you can see from my OP that I am pretty clueless and would appreciate it if I could find someone with experience of pulling something like this together. Keeping it organised and building up a presence on the web. The thought of doing all of that in addition to all the work I am doing already is enough to make me shudder frankly. You can see from the amount of time it took me to reply that I am swamped right now and just don't have the time to keep up with it. Coming back to this every other weekend has killed my velocity and I don't think I'll get it back without collaborating with someone else.

My top priority is finding someone who can sort out hosting a collaborative development site and someone who is experienced enough to be able to review my code and design. When you develop something this big on your own, you inevitably end up with all sorts of defects and design flaws that you don't get time to address as you need to move onto the next part of the framework.

So if you are a good scripter who enjoys distributed automation, and you know what a state machine is and how to understand it, I'd appreciate hearing from you.

==========

With regard to making it a project for the community here, that's probably a great idea. Particularly if I don't have to organise it.

That is understandable. It reinforces my point, though, see below.

This too. Again, see below.

I can understand this. Maybe i am mistaken, but i sense a bit of ambivalence as to what the course of the further development of your pet project should be. This is not a technical question, but one of what you are most comfortable with and i suggest you think that over thoroughly before deciding finally as this decision most probably will stand, regardless of you liking it or not. You could either:

- develop it yourself and/or with hired staff. It will cost you some money to do that, but you will never give up any decisions to others. You will retain full control as to what your product is going to be. In the end you will also reap the benefits if the finished package is ever going to market.

- you could continue as an employee of some company and bring in your current project as an asset. You just turned down such an offer, but it might appeal to you in some modified way. The downside is: it will not be your project any more and maybe you will not like some or any of the decisions others will make about the package. On the other hand it might land you a well-paying job and this might alleviate soem hurt feelings.

- you could turn it into an open-source project. You will minimize your financial risk (as compared to the first option) but you will give away the power to decide the same way as with turning it over to a company. You should be aware that once you gave your project over to the public you retain some power base on the merit that you created the project, but ultimately others may eventually accumulate the same (or even more) merit and then overtrump your decisions. OSS projects tend to be a meritocracy rather than a democracy. See the dispute between Linus Torvalds and Alan Cox, for instance.

What i have written above is more for your personal benefit. As much as i like the idea of having a collaborative project going on here i don't want to take advantage of you and the other people running the site here see that similar. We are a non-profit installment and if there is anything we can give our members in exchange for the effort they put into here it is doing our utmost to keep them happy and comfortable doing what they do.

Now to my main (technical) point, for a change: it might be a good idea anyway to let your POC be what it is - a POC - and rewrite it again from scratch. First, there is the possibility to wean out all the design flaws/shortcomings one inadvertantly builds into the first design of a project.

Second, i think the shell is a very good tool for doing what you did, but the Korn Shell is way better suited for organized programming than is the bash. The ability to have library-like functions to use via the FPATH-variable for instance. The way you can handle several simultaneous I/O-streams and more. I could go on with this list. The Korn Shell is free (since 2005) like the bash and available everywhere, so this is no advantage of the bash.

But perhaps i take the second step before the first: you should come clear with your own intentions and first decide

Again: once you have finally made up your mind, i suggest you have a talk with Neo. I only moderate this board, so i do not know what he can do and/or is willing to do but in my humble opinion much of the infrastructure you say might be needed seems to be there already. There are a lot of good coders here and we might use our knowledge and abilities for something different than writing the concealed homework for students to lazy to do it themselves, for a change. Neo and scott, the admins here, btw., are very good programmers themselves, alas drowned in keeping this board running in the perfect way it runs right now.

I hope this helps.

bakunin