Micro-payments as an internal company employment structure
I would like to build a system that allowed me to post micro-tasks for my official employees to work on. Smaller tasks could be used to train entry and potential employees, while established employees could have a steady stream of tasks but are not bound to any specific hours or minimum work.
The system requires access to an official people database with characters that represent each of the employees. I have been considering using salesforce.com as the primary people database because they have marvelous api access and I think learning the service will be extremely valuable.
The concept I have been mulling over for several years but was recently well pronounced by Second Life founder Philip Rosedale on my new favorite tv should Foundation http://foundation.bz/8/.
Getting start with nitrous.io, ruby, sinatra, and twitter bootstrap
Recently I have discovered Nitrous.io which provides a hosted cloud development environment for ruby, node.js, python, and go. The environment comes pre-configured with enough software to run a public webserver, and can be accessed from any browser! This services is certainly a top candidate for teaching new students to learn web programming, as the software installation barrier to entry is almost entirely eliminated. Unlike Heroku there is almost no deploy time and the user can see how to actually run a web server rather than just sending code into the black box that is the heroku cedar stack.
To get started you need to sign up for the following accounts, which should all have a free option:
github.com => code sharing & collaboration nitrous.io => cloud development environment codeschool.com => cloud code learning heroku.com => cloud app & database hosting build.phonegap.com => mobile app builder & distributor
My strategy is to teach students entirely in the cloud in order to eliminate the most costly startup tasks such as installing software dependencies on varied platforms. This will especially help to introduce windows users to the unix shell without the myriad hassles involved in setting up a proper we b development environment.
A Proposed Exercise in Software Architecture and Data Recovery
A high-level comprehensive understanding, documentation and scripting of machine infrastructure for web applications is damn hard but is ostensibly of massive value. Institutional security is enhanced every time a software system is documented using executable system management and build scripts, whether those be at the high-level concept of chef-server or a puppet system, or more specific like the underlying shell scripts that are executed by those systems. If anyone is going to approach the infrastructure-as-code challenge they will be forced by the shear gravity of the task to approach it in a methodical fashion by scripting the most valuable institutional software assets in a decreasingly valuable order until the desired level of predictability, automation and backup is achieved or the price of moving forward become prohibitive.
Beginning with the class of software system most valuable to the company I immediately look at the database layer. The entire business of serving precise search results for classified ads is based upon the reliable availability of massive amounts of data stored in hundreds of databases using dozens of database technologies Countless business functions are served by databases and rely critically on consistent access to them. To bring the most valuable portion of data storage into a replicable and uniform system would entail the discovery of the most critical database server machines. Again even the top 80% of the most important database servers is a large set of technologies and a daunting collection to automate. Let us find the twenty perfect most valuable database machines across the various architectures and prove that we can build only those machines using scripts alone.
For many organizations targeting the most valuable databases probably means working with one mysql server or a microsoft sql server. To aim at automating the mysql system entails fir a build of all the requisite software onto the machine, then a creation of the databases, followed finally by a load of the data to bootstrap the database. Each instance of the mysql system should be separate from any application code that accesses it. Usually I predict the system would naturally dicate a topology at such scale where the mysql server software is the only process on the machine that communicates modularly over a unix socket connection with the rest of the system. The boundary of our definition of automation of the mysql layer should be here at the socket level interface.
Several pieces of configuration information are required as inputs to any mysql system and must by supplied by an engineer or hard-coded in config files. From a readable documentation perspective for managers and admins to conceptualize each piece of the automation system as a set of required inputs and predictable outputs yields the most profit and confidence. I propose we set upon automating the mysql system with a firm mysql server generation function the ultimate goal. We can do this with a variety of technologies and I think it best that we test the top strategies. Chef, puppet and capistrano all seem proper for this task, and it will probably suffice to test only the first two and perhaps attempt raw bash shell scripts. We win when we can provision a new ubuntu machine, run our script and then connect using the Mysql client and execute core application queries against the database. The next obvious win would be to connect the newly-created mysql server to a running application stack to perform integration testing before officially releasing our new business robot. Later we will document and script the various ways that our system interacts with the mysql server, such as relating to load balancers and applications servers.
Discussed often is the idea of spending a day learning how to re-build a portion of the infrastructure. I propose a more focused, specific recovery and automation challenge for the team: The scenario:
A malicious or spontaneously-generated virus has deleted all of your mysql database servers and the ‘golden’ machine images that were supposed to guarantee them. Luckily we still expect a delivery feed of new few daily data across the internet in fifteen minutes, and we still have our data loading scripts and our project’s code is intact on github. Can we take a blank copy of the Amazon Ubuntu image and have the database ready for the data feed in fourteen minutes?
Drawing a Line Between the Web Server and App Servers for Instructional Purposes
There exists a fundamental difference between web servers and application servers that until recently I did not fully understand, and which needs to be clearly delineated in the students’ mental construction of the web stack and the lifecycle of a request on the internet.
Students cannot be exposed to servers initially through the entire application server stack that rails provides, but must rather learn about static files served via apache, nginx, or IIS after their introduction to http. Previously I had endeavored to introduce students to the server with the simple http server python tool, which creates a server running on the user’s local machine accessible to the local network. The exercise introduces then to the concept of ip addresses and port numbers and requesting files over http. At first everything is hosted locally, and they can discover http using the curl client tool. This approach worked well to demonstrate the most basic functionality of a client-server interaction and should remain a fundamental exercise, with several tweaks.
I am not yet certain whether I should teach http using fist the browser as the most basic http client, or using the curl tool. Curl does not require html, though neither does the browser, which can server up plain text but also images. The browser however does not allow the user to simulate POST, PUT, and DELETE actions. POST, PUT, and DELETE are necessarily actions served by the application server process while GET can be taught at first using only a web server. I do not need app-level functionality any time near the beginning of the course so both browser and curl can be used to introduce the client.
A perfect exercise to demonstrate actually deploying a web site behind a web server would be to purchase a godaddy domain with hosting. Godaddy serves your static web site using the apache web server, the most popular web server software in the world. Before jumping from the python simple http server to apache on godaddy I will explain the market for web servers and the recent history of Apache, IIS, and Nginx which collectively are run on over 90% of the world’s webservers. Although Microsoft IIS is very popular and powerful, valuable systems are built with its stack our course will not deal with IIS as it is proprietary software, restricted and licensed for a fee. Apache and Nginx on the other hand maintain free, open availability and usage for all, so we will focus primarily on them as our web server.
Recounting my Work History - Upstream Communications
Upon examining my resume I realized that the description of what I did at each of my jobs was severely lacking. Even whitetruffle.com gives my profile only 90% completeness because my descriptions are too short, and they are admittedly vague. Since I have meant to write up the story of every one of my business ventures I might as well do a blog post for every employment on my resume and some that aren’t.
Each post should fully describe my activities at the company along with the technology stack employed. I will describe myt favorite successes and most valuable lessons attained. I can highlight how my mentality changed through each step and why I transitioned between each. My book will be much more complete once these chapters have been codified and I can extract patterns and lessons from my experience in a more concrete manner.
Now that I have decided to dedicate some time to recording the story of my professional experience I want to begin with my first salaried position where I was paid full-time to work with rails, well above the median income for the area. The venture of which I speak was both extremely valuable and mildly disappointing, as I was fired for the first and so far only time from that job.
The reason I would like to write about my time at Upstream Communications is because of the practical tools I picked up from their product offering. The studio produced websites that needed to be updated frequently with new content, calendar events, and legislative alerts. The company derived the value for their existence by building content management system backend software on the same server powering their website. Clients who don’t understand coding could then log in to their site using an administrative email and password and update the data displayed in the various lists, sections, images and videos on the page for on-the-fly customizability. The content management systems we built were very similar in functionality to wordpress but with all the flexibility and power of ruby and mysql on the server.
My responsibility was to build websites for various political campaigns with rails, mysql, html, jquery, and css, designing the site and setting up the administration backend panel. I would then deploy the website to a cloud machine running in Heroku’s webserver farm. After a month of repeatedly cranking out rails-based CMS websites I was pretty good at it, and that specific skill has been extremely valuable to me in the management of my own relational database content and that of my employers since then. I am more confident in my data when I have a slick admin interface on top of it. The admin module itself is a ruby library called
rails_admin, which installs all the functionality for admin sign-in/authentication and the editing of database rows from a user-interface rather than using raw SQL. Rails admin creates a single database table for the user or admin model and users the Devise authentication gem to handle everything authentication, from password resets to access tokens to user registration.
Data integrity and trust therein is important at any business and since my experience working at Upstream Communications I have integrated the rails admin system into nearly every one of my rails applications, to great benefit.
Given that I was fired from Upstream I must examine the reasons for my dismissal and ask whether I was actually qualified for the position when I took it. Interviewing and hiring engineering candidates for specific tasks is often difficult because the hiring managers do not know how to be confident a prospect truly can deliver adequately-working software and the engineering candidate often does not fully understand the magnitude of the employers’ expectations for product delivery.
Despite the fact that I could not deliver results efficiently enough for the business and I should have been fairly dismissed for that along, I believe I was actually fired because I couldn’t help myself from speaking out about politics. The company was comprised of a hand-ful of unstable political junkies who at the time were often occupied worshiping Carl Rove and Mitt Romney. I teased them constantly that Romney woud lose because the Ron Paul supporters were just not going to vote at all. The day before I was fired I upset several other employees and they suggested the company was perhaps not the best place for me to be, and after later discovering that the entire organization was essentially funded by and existed for Karl Rove, I am glad I escaped with I did.
The whole experience of almost two months of full-time employment with a rockin’ salary was enough of a boost to my confidence that I soon thereafter went to the 2012 Rails Conference where I was able to take my understanding of the rails ecosystem to the next level and obtain an awesome job doing what I love. I learned it is best to avoid talking about politics at work but more importantly that if your institution is politically aligned against you it is best to sever the relationship and find a more appropriate setting. You deserve the perfect job and should never settle for spending all day every day in a miserable environment.
A More Robust Media & Data Plan
Recently I discussed the need for a more sound and robust strategy for storage data at Zenterp. Our media and code files represent the majority of the company’s assets and their integrity is of prime importance. After studying the industry I have definitely decied to store the company’s data on cloud file servers. Initially all of our digital assets were stored on Amazon S3, but several payment and access mishaps resulted in the accounts being locked temporarily; I had to diversify amoung cloud storage providers for security through redundancy and for research.
Amazon S3 storage is wonderful from a developer perspective because it provides code API access to everything! This is generally true for all Amazon AWS services and I love to script such services using the official ruby gem (library) provided. Alternatively one can use the marvelous “Fog” ruby library which provides a similar interface to the AWS gem, but is actually an abstraction over many storage and compute providers.
Fog allows us to upload and manage files on Rackspace Files and Google Cloud Storage using the same API as with S3. From a business perspective these three storage systems are the most interesting. I want to research these cloud providers and develop organizational best practices for media and data storage redundnancy in the cloud. AWS is most appealing because they are the leaders and have the most experience in the cloud arena, and I have done a lot of work scripting their services. Rackspace intrigues me because their infrastructure is built on top of OpenStack and a business relationship with them could help Zenterp move into building systems on open-source hardware stacks, a stated purpose for the existance of the company. Finally Google is certainly a top-notice cloud service provider and I believe it will benefit us to study using their Cloud Storage, Compute Engine, and App Engine for potential business solutions in the future. They also provide a third level of redundancy for the media as backup.
Now that I have a strategy for redundant media storage in the cloud, I will move to implement the strategy. I still need to work out a fail-safe backup on a nearby physical drive for emergencies. Once the media files are distrubuted to the three storage providers I will report back and form a strategy for content distribution to the public and with the Zenterp organization.
A Review of and Plan for My Technical Studies
From time to time it really helps to write down all the things I have been studying and decide which of them should be afforded highest priority, and then alter my study patterns accordingly. Although it seems like I have several options for the most-profitable ventures in which to invest my time and energiues, there are in reality probably dozens :-P My task is to decide on which of these paths to focus now and which later.
SQL and the MySQL Database System
I have determined that in the immediate future I need to study further the sql language and the mysql software database software system. There seems to be a great need for architects who understand how to scale mysql in a cloud application environment. A stellar colleage recently recommended on and only one book to me: High Performance MySQL. I am working my way through that book now and I believe it will be of monumental value. The book is quite large so it will have to knock off two other large books I have been reading: The Linux Programming Interface and the Postresql 9 Manual. I will definitely finish The Linux Programming Interface but may not get back to the psql manual. Along the same thread of study I continue to read SQL Antipatters by Pragmatic Studios, and keep Seven Databases in Seven Weeks near the top of my queue.
Electronics and C++
Further down the road for me and not of immediate finanical or business impact is the study of embedded systems and programming in c/c++. For about six months I have experimented with the electronics prototyping platform arduino; I learned to proram and run a server embedded in the microcontroller that controls an LED ligh. I learned how to operate the above server over both Ethernet and Wifi. I figured out how to read data from the button status of wii nunchuck remotes and stream that data to my mac and across the network. I built live graphs of data streamed from a lightmter connected to the arduino and I controlled a small motor. Those experiments are just the beginning and my next project is to build a working, flying quadcopter from scratch. Although I would love to spend more time studying electroics, the cost of starting a business and even prototyping in the electronics sector is much high that in software. A goal of mine is to use the profits from my web/mobile software ventures to edify and electronics research lab for wireless computing, wearable computers, flying robotics, and 3D printing. In the mean time before I can commit many resources to electronics and the study thereof I can learn more c++ and read the pragmatic book Modern C++ Programming with Test-Driven Development. My long-term business strategy extends out this far conceptually, to producing electroics systems for the manufacture of micro drones and other futuristic appliances.
Algorithms & Formal Computer Science
Finally on my mind is that I want my knowledge of algorithm construction and optimization to be more advanced. I want to read the book: Introduction to Algoritms as well as finish the iTunes U Course I began from MIT that parallels the book. Also I will continue to do increasingly difficult codeeval.com challenges, but in c++ this time rather that ruby to further my understanding of that lower-level language. The study of algorithms for mathematical computation and natrual language processing will enhance my intellect in general and my possible ventures widely. I don’t know exactly when or how often I will study algorithms in a theoretical and explicit manner but I am convinced the more I do so the faster I will advance in general.
Obviously I have a lot of work to do to get where I want technically, and now that I have ennumerated chronologically my future research my studies will be more guided and effective.
Next Screencast Notes: Intro to SQL
It has been a while since I produced a video tutorial screencast, though I have several good ideas solicited from my network. I think since I am studying SQL a lot that the next tutorial will be an introduction to SQL and how to build a database-backed application. For the introduction I will definitely teach how to install MySQL and use tools like the mysql command line client interface and the SQLPro software package. I can demonstrate:
1. How to create a table in SQL
2. How to populate a table with data using SQL
3. How to query that table using SQL
4. How to join two tables using SQL
5. How to create a ruby Active Record object corresponding to each table
6. How to make the same queryies using ActiveRecord
7. How to manage the database schema using ActiveRecord::Migrations
8. How ActiveRecord and migrations play in the Ruby on Rails ecosystem
I want to take the students all the way up from data in a format they can understand now (such as .csv or .xls) to how to get that data into MySQL in a low-level fashion, to using the active record pattern in ruby, and finally the sugar on top by learning how to alter your sql workflow using rails.