• I find that a lot of devs have problems with thinking about queues and planning their architecture with them in mind. They get lost in the technical details and fail to see how queues really should be planned and organized to get the most out of them.
• Imagine you’re at a grocery store with a single checkout counter and a single queue, where all the shoppers are lining up. You go up to the queue, and join the back – after all, all these people have been waiting before you, don’t be rude. The check out counter then calls the first person in line forward.
  • The person manning the counter scans all the groceries
  • They charge the customer
  • The customer pays
  • The customer bags their groceries (or the groceries are bagged for them)
  • They leave and the person manning the counter calls the next person forward.\
• This process will continue to repeat for as long as there are people in the queue. As soon as everyone in the queue is gone, the person at the counter will just sit there and wait until someone shows up. That’s their job as a worker.
• Notice that the queue didn’t exist at the checkout counter itself, it existed BEFORE the checkout counter. This is the biggest benefit of having a queue system. Now, if we open a second counter next to the first, they can utilize the same queue and ideally get twice as much work done. Add a third, and three times the work gets done.
• However! That only works when the counters never need to talk to each other. The minute a counter needs to talk to another counter to verify something BOTH counters need to stop working to talk before they can go back to whatever they were doing. This is where a lot of problems happen, and causes slowdowns and forces the queues to backup. Each counter needs to function completely independently for the best performance.

Pros

• It provides an easy way to measure when you need to start looking at scaling your system. If you notice the length of your queue start growing, it means you’re not able to work through your data fast enough, so you can easily add a few more workers to help with the load. And because you’re using a queue…
• Your workers don’t need to reside on the same servers! You can have any number/spec of servers with their own number of workers each and they can all still interact with the same queue. You’ve now allowed your system to scale both horizontally (more systems) AND vertically (better systems).
• One of the favourite features of Queues seems to be overlooked a lot of the time. Queues provide a place for data to wait until you get to processing it. This means that if your processing side ever breaks, or needs to be taken offline temporarily YOU’RE NOT LOSING USER DATA. Data is still being collected in the queue and when your workers come back online they can get right back to work retroactively processing the data that was waiting for them.

Cons

• Of course, with every new component there is some overhead. You will need to devote some resources to just handling the queue. It’s another component you need to manage from an infrastructure point of view.
• Ideally, the queue would be its own server which means there’s additional latency when pushing/pulling data to a queue.

Tools for queuing

• While you can write your own queue service fairly simply (for example, piggy backing on Redis) there are also some great tools for you to use and incorporate into your projects right away. I’ll be honest, I don’t feel comfortable giving a full review of them because I’ve only ever used SQS. Instead, I’d recommend you just do some research and figure out what makes sense for you.
• RabbitMQ – http://www.rabbitmq.com/
• IronMQ – http://www.iron.io/mq
• SQS – http://aws.amazon.com/sqs/

I’m going to be writing a plugin for WordPress that adds better support for Fargo.io. Basically, now when you set the attributes for a node you can add some additional attributes to customize when/where the post appears.

• There are a few post related attributes that I want to support
  • category
    • A list of comma-separated values for the categories that you want this post to appear in
  • private
    • If private is set and is true, then the post is NOT made public. By default, private is NOT set which means any posts are public
  • post-date
    • If this is manually set then the post is scheduled for the date that you specify.

The plugin will also support “Pages”. By adding the “page” attribute to a node everything underneath is passed to the page and it overwrites the existing contents of the page.

In addition to the plugin there will be some code that needs to be added to Fargo to add the new post button to the sidebar.

The idea behind this new plugin/fargo integration is simple: I want to power by blog with Fargo.io. I think having full control of my content at ALL times this way is very powerful and the ability to have a custom domain associated with my outlines even more so.

Eventually I’ll have to add support for multiple WordPress installs, but to start it will only support one.

But we were an API company. The tools were provided were meant to allow for mass adoption of our API’s but unless you wanted something vanilla you would need to dive into our API.

After a lot of market research, and even some successful pilots we decided to stop playing in the API space. See the problem is that our market segment was very niche. We thought it was a lot bigger than it was. Our market segment was really programmers who were building something that was technical enough that they needed this functionality, but non technical enough that they were able to waste time building it all themselves. We required that a company be working on a consumer facing application, wanted to tie in to various cloud storage services but didn’t want to do it themselves. Oh, and the company couldn’t be TOO big because then they’d do it themselves, and they couldn’t be too small or then they would HAVE to do it themselves.

But see, that’s the problem with API companies. They tend to be niche products that will have some success in their niche, and that’s it.

I wondered why for some time – why aren’t there any big API companies? What stops that from happening? And then I realized – As an API company, what we are selling are some awesome made Lego bricks. We even sell them in a pastic bag, so you can take a look at our bricks and see how awesome they are. We provide you instructions on how to use the bricks. We even show you how some bricks work. But you’re not trying to create something from it. You buy our bag of bricks because you saw a brick that you needed and you figured it’s easier to get it from us than actually making your own bricks. We missed the whole point of the bricks.

See Lego doesn’t sell the bricks. They sell the picture on the box. The bricks are just a mechanism to achieving what you want – the picture.

API companies, because of the nature of what they do, don’t provide this picture. That’s not what they do. Instead API company founders tend to be very smart, technical people who see the true beauty of the bricks. They revel in the bricks. In their minds eye that can twist and contort any number of the bricks. They can fit them together and build any number of amazing things. They get caught up in the bricks themselves. They never sell the amazing things that you can build with the bricks – they sell to other programmers and leave it to them to build the products.

One of the things that I want to do with my OPML feed is to keep all my thoughts on various projects together. I use Fargo to keep track of all my thoughts on my projects, and when I DO release project to the public I’m using it to power the “updates” section.

One easy way to do this is to utilizing the new “Names” functionality that was introduced with version 0.61. It allows me to publish my OPML document right on Small Pict and have a public link to it that I can share with users. Of course, this isn’t idea for two reasons.

• I’m using up valuable sub-domains that other users may serve other users better. Not to mention the fact that right now sub-domains are free and I would prefer not needing to impose a payment scheme on it.
• I like controlling the content that I’m creating.

That last point is very powerful. As I said, it’s what led me to start using Fargo. So, what I did was write a small script (literally a single function in PHP) that you point at an OPML feed and it will return a `JSON` or `array` structure that you can consume as you want. I’m linking directly to the OPML document in my dropbox folder that I’ve made public.

Want to see it in action? http://risinglegends.net/game/updates

Gist: https://gist.github.com/AngeloR/5601095

However, one of the things that I worry about is when I eventually get bored of WordPress and decide to try and build something. Or what happens when there’s a new blogging engine out there that’s a lot better than WordPress? Do I have to rely on the next tool builder (me or someone else) to create a WordPress importer so that I don’t lose all my content?

I am also very forgetful. Not a “Did I leave the stove on?” forgetful, but the “Have I eaten today?” kind. As such, I need to write down everything I do. If I don’t, I’m pretty much guaranteed to forget it. I have a bunch of ways that I keep track of things. I use Evernote to capture all of my meetings and ideas and to have references to things. I’ve tried using it to organize what I need to do, or even to organize my thoughts, but my mind doesn’t work that way. I find that old-school Outliners really work the way I think. Multiple thoughts and details can quickly be added without needing to worry about things.

I’ve tried a bunch of different outliners before settling on Workflowy a few months ago. It provides a super simply, web-based outliner that lets me do what I need. They offer me the ability to export my data and link to specific nodes and share things – which is awesome. However, something still irked me about it and always stayed my hand from actually paying for it.

I have no idea why – but I was always hesitant about it.

But yes, back to why i’m blogging with Fargo.

Fargo.io is a new tool released by Dave Winer – the same man who inspired my NewsRiver project. It’s a web interface that writes to YOUR Dropbox account. Basically I can hop in to Fargo from ANY internet connected device (thanks to Bootstrap as well for that functionality) and I can write and organize my thoughts in an Outliner that saves itself to my Dropbox account.

I’ve been using it for a bit transitioning all my stuff out of Workflowy into Fargo and I’m already liking it a lot more. A lot of customization options (Custom CSS? Yes please!) as well as the ability to post right from my outliner to WordPress. That’s right – this post is written in Fargo.io and with a couple clicks it’s imported right in to my WordPress powered site.

Of course – now that the thing that powers my website is no longer locked to WordPress, maybe it’s time to investigate writing my own blogging system that integrates a bit better with Fargo.

From a technical point of view, imagine big-data style analytics in real-time. We’re not machine learning – that is better left to people with PhD’s and research papers. Instead what we do is get down into the nitty-gritty details of your network. We run passively on your network sniffing out everything and then perform analytics on what’s happening to provide you a list of services that your employees are currently using. Of course, the benefit of using our tool is that we provide a way for you find out about new services as soon as they might begin to impact your company.

Forget storing data on the every service out there on the web – that’s a full time job that is better left to Google. We focus on what is impacting you and your company.

However, our solution is a single piece of software (and at some point, possibly hardware) that gets installed on premises at your company. We don’t ever see any of your data – that’s what real privacy looks like. Of course, with that comes its own set of problems. My big question right now is pretty simple:

How do I keep the software updated with bug fixes, security updates and features without forcing you to do anything?

One of the ideas that I’m really interested in is this:

We’re currently using git for our VCS. Does it make sense to keep our master branch releases with branches for security, bugs and features? This way we can merge them back to a test branch when everything is ready for testing. Once that passes we move things to a ‘pre-production’ branch which just sits there for most of the week. Perhaps once a week it merges pre-production and master. This way, the app can simply git-pull against our master branch whenever it feels like it? Perhaps on a scheduled basis once a week it can pull from the branch.

The benefits of this is that we can provide custom SSH keys for each installation of the application. Once a users license has expired or run out, we can simply de-auth them against git and then they no longer get any updates – without impacting their current service.

Another benefit is that if there WAS a bug with a new release we can configure the updater to simply roll-back to a previous version.

Because of our market size our customers will more than likely have their own labs set up where they would like to test updates. The git distribution system will allow us to roll out new updates to test labs if a customer desires. From this point they can easily configure and figure out what updates they want to release internally before pointing the application at their git server internally.

Unfortunately for me – I’m such a big proponent of the system that I can’t really see the downsides. That’s where I’m sort of hoping that you on the internet come to my rescue.

What are some potential problems that you see with using git as a method to update your software applications?

One of the things that I’ve been working on is NewsRiver. It’s a new way to read your feeds that really helped me eliminate the problems I was having with reading hundreds of articles a day. The problem with traditional RSS readers is that they make you feel guilty. When I log in to Google Reader after a couple days off and see THOUSANDS of feeds unread, it’s easier for me to just sheepishly mark them all as read and start from scratch. But if I’m going to be doing that, why does it matter that all my feeds are stored in that manner?

The nice thing about important news is that it doesn’t just show up once. Many sites will pick up the article and publish it with their own opinions. NewsRiver capitalizes on this by NOT storing or sorting your feeds. Instead of accumulating large amounts of items that you’ll never actually read NewsRiver streams the items by you as they happen. If you’re not around when it occurs then NewsRiver assumes that you’re busy with something else. If the news is important enough, it will make its way around again.

If you haven’t played around with it yet, I’d highly suggest taking a look. NewsRiver has been around for some time, but it’s always been something I’ve used personally – never really marketing it. Today, however, I’m happy to announce that I’ve been working with Cochran Media on product roadmaps, design, marketing strategies and product positioning.

We’re hoping that together we’ll be able to take this product from a great hobby to something that really solves a need with the demise of Google Reader.

Choice is a funny thing. We all want it. We want to know that whatever happens it was our choice that it happened. We like knowing that we controlled the outcome. We like knowing that it was up to us that something happened. We like choice. We believe choice gives us power.

The problem is, every choice we are given paralyzes us. Every time we are faced with a decision we must sit and think. We have to evaluate our options based on the choices at hand. And every choice given adds more time to our decision process – it decreases our conversion rate if you will.

I propose that in product design, despite when a user says that they WANT choice and they WANT features, what they really want is you to tell them what they want. And I think there’s a very good reason for this.

When a user arrives at your application they are not a new user. They have habits that they’ve picked up along the way. Maybe they used similar software to yours before, maybe they only learnt how to use a mouse last week. Either way, they come with baggage. Every choice you offer your users forces them to go through this baggage and figure out what the best option for them is. And every choice you give a user forces that user to evaluate whether or not it is even worth it for them to go through this baggage and decide what to do next.

This is not a good way to run a business and the underlying problem of asking users what they want simply exacerbates it. When you ask a user what they would like to do, they shuffle through their baggage and pull out a few problem points that they have. Then they rank them and tell you the biggest problem they face with their current tool-set. And then you research their current tool-set and accordingly make decisions about your product.

I urge you to look at companies that truly revolutionized an industry – how many of them conducted ‘focus groups’? How many of them did ‘market research’? I think that companies that do ‘focus groups’ to help them drive product innovation are really driving product evolution. They are taking a product and moving it to its next, logical stage. But evolution is a lumbering beast. Evolution of a product takes years of development and the end result is only marginally different. Look at the difference between v2 and v3 of a product. Was it really that different? Did anything REALLY change?

Conversely, look at v1 of a truly revolutionizing product – the lightbulb, the steam engine, the internal combustion engine and radio. There was nothing like them before. They weren’t born out of ‘market research’. No one interviewed a bunch of people to find out what they wanted next. If you HAD interviewed someone before the lightbulb, isn’t it conceivable to say that they would have asked for a longer burning lamp?

I think what I’m trying to say is that no one remembers the middle-grounds of evolution – they remain lost in the annals of history. Instead we remember the beginning.

Don’t strive to evolve a product to its next stage – revolutionize an industry.

Breeding homing pigeons that could cover a given space with ever
increasing rapidity did not give us the laws of telegraphy, nor did
breeding faster horses bring us the steam locomotive.
– Edward J. v. K. Menge, “The Quarterly Review of Biology”

Rather than focusing on how a VC fund works, I want to talk about how they choose companies for funding, and why that system is flawed.

Motivation

The current VC system is focused around making money. But not just making money, specifically making a ton of money with as little input as possible. A VC firm will have a bunch of partners who have to spend X million dollars. They have two options here:

1. Make an educated guess as to what will take off, so they can get the most return for the least effort
2. Throw money at everyone and hope someone (or everyone) makes it big

The problem is that the first situation takes too much time. In order for a VC to make an “educated” guess, they need a lot of information. They need to know a lot about you starting a company, they need to know a lot about the space, they need to know a lot about other companies that made it and didn’t. This method, as awesome as it sounds, just isn’t feasible. And I agree 100%. There may be a few founders-turned-VCs who HAD knowledge about the space at one point, but by that very nature they are biased. You’ll notice that these people tend to be narcissistic in investments. They invest in people who remind them of themselves, and whose ideas align with theirs. This is a problem, and I think that most VC’s and founders have figured this part out.

The second situation makes a lot more sense. You stop being a strategic thinker and instead rely on numbers. If you invest in 100 companies there’s a much higher chance of success than if you invest in 2. It makes sense from a mathematical point of view, as well as from a time point of view. Now you can just spray investments across the board quickly and sit back and wait for something to happen. Because something is almost guaranteed to happen.

The Approach

Since there is no real way to judge whether a company will succeed or fail, VCs do what everyone else would do. They look back at previous successful companies in the same space as you and try and figure out if you fit that criteria. They look at you as a person and they try and compare you to people who are already successful. It’s called pattern matching and you learn it in pre-school. See the flawed system that I believe VCs use is simply that they believe pattern matching will give them the results they want. After all, if it looks like a duck and acts like a duck, it’s probably a duck.

The problem, however, is that you shouldn’t be looking for a duck.

The easiest way to explain a VCs investment process is simply a standard data-set distribution of people who are successful. Just take a bunch of data-points, throw them on a graph and see where they clump together. Then ignore the points outside and focus on people who fit within the clump – after all if 75/100 successful people have similar traits you would think that the next successful person will have similar traits right?

The Turkey Problem

First off let me say that this example is ENTIRELY lifted from one of the smartest men I’ve heard of. Never met the man, but if someone can get me an introduction it would be most appreciated – and I will send you a pizza.

Nassim Talib puts forth a very interesting argument to pattern matching. Imagine a turkey, being raised on a farm. Every day that turkey is fed, it’s given a place to sleep and is completely protected from all predators. That turkey has it pretty good. If this happens every day, that turkey will continue to believe that being a turkey is pretty awesome. Infact, that turkey will ALWAYS believe that it has it pretty awesome. Until thanksgiving. Then some event that the turkey can’t even comprehend will occur, and the turkey is done for. However, up until thanksgiving if you could talk to a turkey, it would believe that being a turkey is pretty awesome. Someone takes care of you ALL the time, why WOULDN’T you want to be a turkey?

The Simple Solution

So how do turkeys apply to VCs? Simple. The turkey is essentially doing the same thing the VC is. They look back at historical “evidence” and try and fit the future in to it. The problem is this: That outlook only works until it doesn’t. But it’s considered a “statistical anomaly”. It’s not part of the pattern so we ignore it and move forward. The problem is that these statistical anomalies account for the MOST change. What VCs are hoping for are big payouts, but they ignore the outliers that can cause the GREATEST change.

We’re literally held at the mercy of the outliers, but after they happen we either choose to ignore them or we make them the norm. The problem is that the very nature of an outlier means that we never see them coming.

What does this mean to our VC problem?

VCs need to stop using an average to figure out if a company will be successful. They need to start looking at the outliers on either side of the spectrum. Either the ones who will be amazingly famous, or the ones that will fall flat on their face. The thing is, statistically, the companies that you want to invest in are the ones that either closely mirror the outliers or doesn’t mirror anything on the market at all. Those are what will have the potential to give you the most return on your investment.

PS. I hate statistics. I don’t understand how a branch of study can exist PURELY to ignore the things that cause the most change? It’s like a study in how to remain mediocre. But at the same time I absolutely adore that there’s a branch of math dedicated to the mediocre so that we can flip the suggestions on its head and ignore the mediocre.

I stated that I feel part of the problem lies in the strange importance that an entire generation has placed on the “degree”. I feel like I need to clarify why I think that and what needs to happen – mostly because it isn’t what I thought it would be.

I think the core of the problem is really the commoditization of knowledge. Knowledge is clearly power, but rather than working to bring that knowledge to everyone, they’re locking it away. Don’t get me wrong, I’ve always said – and I stand by it – that I don’t believe that an industry can exist without people making money. We’re a capitalist society, and a capitalistic society drives innovation. However, lets explore that idea briefly and see how it affects us.

Above you’ll find outlined the standard deviation on bell curve. The numbers themselves are not that important, but rather the idea of the bell curve. In any non-biased distribution, after you ignore the outliers (that’s why I hate statistics by the way, perhaps I’ll cover that another day), what you’re left with is the above representation. There are a few people that fall to the left, the majority in the middle, and then the rest to the right. The left and right simply refer to the lower and upper bounds of your scale. In our case (capitalism), our bounds simply measure money.

In a theoretical capitalistic society we can expect either a standard deviation bell curve, or a curve that’s slanted slightly to the right. But, ideally, it would be a standard deviation. There are a lot of reasons for this, mostly economical – you can’t have an entirely high-income society. The difference is that the bounds would be a lot higher. However, we all know that pure models only work when we remove human stupidity so what we’re left with is the more practical distribution of capitalism.

Ahh the double bell curve.

I feel that as soon as you add humans to the situation, capitalism looks more the double bell curve – sure the bounds are higher, but there is no middle ground. This is exactly what we are seeing today with people talking about the disappearance of the “middle class”, this is what they mean. They’re talking about this distribution specifically. We see more people who would be considered “well off”, and more that would be considered “poor”. There’s more to it than just this of course (to really research this phenomenon we’d have to look at economic changes, lifestyle changes and behavioral changes), but the end goal is that practical capitalism forces the distribution noticed above.

I think I’ve managed to convey a brief understanding of my view on capitalism, lets bring that back around to how it affects knowledge.

Knowledge lockers exist to enforce the divide. Conspirational? I don’t think so (if you do, drop me an email, I’d love to hear your thoughts). However, I don’t blame either side for the creation of the divide. I think it was just a natural progression based on human nature. If we look back through history we can see a clear rising where there are some who have, and some who have not and the tension that creates. And those who have, have more access to knowledge.

Degrees, in their purest form, represent a person who has attained a certain amount of knowledge in a particular topic. In order to receive said knowledge we’ve engineered a system where you need to be on the right side of our wealth distribution curve. We’ve turned learning into a business – and this is what happens when a business can manufacture demand and is put in command of the supply.

That’s why I feel like educational reform, and all the companies that are trying to bring about education innovation are not helping. They’re not making things better, they’re simply trying to inject themselves into the profit that these knowledge lockers are bringing in.

What I want to advocate is Educational Revolution, throwing away the standard the idea that we need to turn this current commoditization of knowledge on its head and start giving it back to people. We need to bring back the “middle” class, but we can do that, while shifting our bounds upwards.