Skip to main content

How-to-fail: My First AI-Startup

In 2012/2013, I helped found a startup that focused on product identification of Pinterest images using computer vision and machine learning. In those days, convolution neural networks were still largely a guarded secret, with access to Google's DistBelief code (a private precursor to Tensorflow) being only wished for. MATLAB was the primary tool for which our small team of University of Toronto master students explored CNN with. I myself was writing our production code using Python-based SVMs, histograms, perceptual hashes, segmentation systems, and more.

Those early days were confusing for me as the machine learning establishment was still fairly deeply entrenched in classical machine learning and computer vision approaches. CNNs were hard to understand, let alone give consideration to, and the lack of support available to guide me was virtually non-existent. In fact, I received quite a bit of push back in my deep belief that they were the future. I did receive support from two master student interns at that time, but their communication skills were abysmally absent.

At the start of 2014 we brought on a Chief Science Officer to lead the growing research team, although computer hardware and deep learning were not his expertise. I was generally opposed to the cost of acquiring this CSO, feeling like we were investing into the wrong skill set, but it helped the investor story regardless and gave me the opportunity to focus on improving the production-systems.

2014 was the year we finally started to have some cash though, and the research team had started to grow into a small army; there was still very little effort put towards CNNs however. The R&D team would showcase improvements to the existing systems during our biweekly demo-days, but I mainly got empty responses when I talked neural networks. I felt then the best way to encourage the research team to try out neural networks more was to provide them with the tools to explore with.

The first system I bought for the team was an Intel-X79 based workstation with 64-GB of RAM, a Nvidia GTX 770 4GB, 240GB SSD, and 3-TB HDD. The system had as 1250-watt power supply and could be scaled up to house 4-GPUs if needed. I also made sure everyone on the R&D team, regardless of system or role, had an Nvidia GPU. The pick I made there was the Nvidia 750 Ti 2GB, as it had enough VRAM to tinker with and had support for the newer CUDA 3.5. 

Such systems and parts were easy purchases for me to make, going so far as to just walk down to the local computer store and putting the expense onto my own credit card. I was really hoping they would start using what I bought them and provide feedback that they needed more VRAM and more power... but it never happened. 

The real system I had in mind for the R&D team to train on, and for me to use in production model training, was quite a bit beefier and very expensive. I sought out and received approval for what I costed out to be a roughly $45,000: a 4-GPU system with quad-Nvidia Tesla V40 GPUs, each with 12-GB VRAM. I preemptively even had a small server room created to house it: air-conditioned and sound-dampened.

I never pulled the trigger on the purchase though. With no feedback from the R&D team and their efforts being spent on super-pixel segmentation and who knows what else, I was becoming somewhat frustrated. My late evenings were spent trying to push the CNN path forward, hacking out my own CPU-based neural networks in Python, or trying to get what open-source neural network code existed to work on my system. I wanted to showcase how effective CNNs could be for our problem.

By mid-2014, a few months after the company I helped build went public, I was replaced by some ex-Microsoft executive in a show to investors against my will. The board had control of the company at this point, not me, and they wanted to put in place someone who they felt could scale the company faster I suppose. To add insult to injury, I was put in charge of what was called the "legacy platform", as my replacement intended to rebuild the entire system from the ground up with his own team of developers.

As quickly as I legally could, I exited the company, and at the end of 2014 I was on to another AI startup. In the following months, the R&D team was disbanded as my replacement could not understand any of the work being done by them. He chose instead to double down on using an army of humans, a hundred or so, to button-press all day on incoming product images. Cost of service went up, as did query response times, but hey, it worked. Kinda. A couple years later they outsourced to a competing company who had built CNNs to do proper image recognition work. I have no idea what happened to that army of humans; where their souls are now I hope are in a better place.

In hindsight though, seemingly small failures on my part snowballed out of control very quickly. Trust in my abilities to tackle hard problems or to scale quickly help led to me being replaced, even though I feel my replacements were quite ill-suited. No one wants to hear a problem or product will take 2 to 5 years to finalize, nor does someone want to hear that you feel uncertain about your choices. 


Popular posts from this blog

How to login to an expired Windows

Microsoft has a neat little way to prevent software piracy of their Windows operating system. "Windows Genuine Advantage"[edit: I stand corrected, it has nothing to do with WGA] with its "Product Activation" requirement. Essentially, even with a valid product key, you still need to activate your Windows to ensure that only one computer is using that specific product key. If you can't activate your Windows, there being many reasons for this, you are left with a 30 day grace period to change your product key to one that is fully valid or get in contact with Microsoft and plea your case. Once your grace period is up, Windows refuses to let you login anymore. You cannot access your files. You cannot go on the Internet. You cannot do anything, except the thrill of trying to activate Windows. Well, luckily for me I do not have to worry about this issues, as my Windows is valid and activated. However, for those who do not and have been so unfortunate enough to have

ATi Radeon 5770: PCIe x16 vs PCIe x8

In this article I will reveal the differences one should expect from running an ATi Radeon HD 5770 1GB graphics card at full and half PCIe 2.0 bandwidth. This is particularly important data for those considering a dual ATI 5770's Crossfire setup on today's Intel-based P55 motherboards which generally offer only a single PCIe x16 slot or two PCIe x8 slots. Are you better off with a X58 motherboard for Crossfire that offers dual PCIe x16 slots? Would just a more powerful single slot graphics card, such as an ATI 5870, be a better call? I can't tell you that, since you may find Crossfire annoying or you may want the absolute best performance from your rig. What I can show you though is if Crossfire on a P55 motherboard is a viable option, especially for those trying to save a quick buck. Read on! While other reviews of limited PCIe bandwidth have been addressed prior, such as at Tom's Hardware, these looked at either now-out-dated video cards or very expensive graphics

The Hard Drive Capacity Calculator

Curious why when you buy an iPod, computer hard drive, or memory stick there is less memory actually available then what was advertised? For example, you buy a new laptop, advertised having 100GB of hard drive space, actually only seems to have 93GB available? It almost seems as if there is false advertising involved. I'm sorry to say, this isn't the case, rather just a tricky usage of terminology used by memory manufactures. Let me try to explain... You see, digital data is of binary numbers. 1's and 0's, also called bits. A group of eight (8 bits) 1's or/and 0's describe what is known as a byte. There are hence, 256 different values for a byte. ie: 00000000, 00000001, 00000010, 00000011, ... 11111110, 11111111. Digital memory is measured in the number of bytes, although it can also be described in bits. This is where it gets a bit more confusing.... Just as with our decimal number system, when a very large binary number needs to be described, such as 123,456,7

Some features of digital cameras

Most of us already have a digital camera that suits our needs. Quite possibly it's a brand of camera you bought at Walmart on sale that no one has ever heard of before, maybe it's the one integrated into your cell phone, or maybe your camera doesn't suck and you are thinking to yourself, what is the point of this article!? Well, I'm hoping to share a short list of some of the commonly overlooked or new features of digital cameras today that you might be missing out on. You might even find them all just enough of a reason to change what you are hoping for this Christmas. 1. Image Stabilization What was once only found in professional cameras has now found its way to the masses. Image stabilization technologies could very well be the biggest change for digital cameras since the auto-focus. With image stabilization you can take photos or video clips without a tripod that would otherwise be blurry due to an unsteady hand. Even with a steady hand, trying to take a photo whi

Intel Core i7 860: Stock Cooler and Power Usage

I'm just throwing this out there for anyone else running into similar problems as I have been having with my Intel Core i7 860 system. I'm noticing that a lot of heat is being produced from these chips when at full load and especially when overclocked, to the point that I feel the stock CPU cooler from Intel is simply inadequate. The included heatsink/fan included with the Intel Core i7 860 has a low-profile design that is smaller than previous Intel coolers, which seems a bit strange in my opinion considering that these chips are rated at 95W TDP. Anyways, I bought two Core i7 860 systems, one featuring a Gigabyte P55M-UD4 motherboard and another one using a Gigabyte P55A-UD4P motherboard (primary board used in this review). Using the stock Intel cooler, I could effortlessly overclock these systems to 3.5GHz using stock voltages. While the systems were stable for all intensive purposes, running Prime95 would cause the core CPU temperatures to shoot up quickly to 100*C (usin

Review of the New Dell Studio 15 (1555) Notebook

Hi everyone, welcome to my review of the new Dell Studio 15 (1555) notebook. I have had the notebook in my possession for just a few days as of this writing, so the review at this time will focus mostly on my first impressions of the device. I will update this article as I gain more experience using this system, so check back soon for added information or send me your questions via email for a prompt reply. The system under review has the following specifications: Processors Intel® Core™ 2 Duo P8600 (2.4GHz/1066Mhz FSB/3MB cache) Memory 4GB Shared Dual Channel DDR2 at 800MHz Keyboard

Review: Lenovo Bluetooth Laser Mouse

In an attempt to give back to the community a bit, I am going to giving a review of my experience with Lenovo Canada and two new products I have recently ordered from it online. The two products are the Lenovo T400 14-inch notebook and a Lenovo Bluetooth Laser Mouse. Purchasing I ordered online both the T400 and the mouse late Monday night, from Lenovo .com. The total price, after tax, was about $1,250. Before tax, the T400 cost me $1070, while the mouse cost me $27 (regular $45). Shipping was free. I scored a bit of a deal on the purchase thanks to some e-coupon codes that were mentioned over at Redflagdeals .ca. These e-coupons saved me a lot of money, but it still took about a month of constant checking before Lenovo had what I wanted in stock and at a decently low price. Shopping at Lenovo .com is a bit like playing the stock market; there are ups and downs. One day the LED back-lighting feature would cost $370, the next day it would be just $30. In the rare case, it

Lenovo T400 Review

So, let my review of the Lenovo T400 begin. Configuration under review Lenovo T400 14.1-inch LED 1440x900 display Intel Core 2 Duo P8600 CPU (2.4 GHz, 3 MB L2 Cache, 25 W TDP) 3 GB of DDR3-1066 MHz RAM Switchable graphics: Intel GMA X4500MHD or ATI Mobility 3470 /w 256 MB Bluetooth Intel 5300 WiFi 80 GB 5400 HDD Webcam Media Card Reader Windows Vista Home Basic 9-cell extended battery Price: $1070 CAD (Ordered November 24th, Delivered December 4th) Also ordered a Lenovo Bluetooth laser mouse. Price: $33 (Delivered in 1.5 days) Prices were substaintially discounted, thanks to a friends of employees discount and an addition e-coupon code. CPU Performance: 1M on SuperPI Mod 1.5 scored about 21 seconds. For comparison, my 3 year old AMD Sempron 3000+ overclocked to 2.4 GHz scored 50 seconds and my brother's 2 year old Intel Core 2 Duo E6300 overclocked to 3.0 GHz scored 19 seconds. The processor is very efficient in both terms of clock cycles and power savings, but compared to modern-d

Google's Web Toolkit 1.3 RC1 now out

Not everyone knows what AJAX is, not everyone will care, but for those who do and don't know this already, check out Google's Web Toolkit . As a hobbyist web developer, I know just how frustrating it can be to design a website with really neat functionality and layout. I have been really amazed at how Google creates their many websites, having them all lack that rudimentary feel most older websites today have. Google has really gone above and beyond by releasing the same tools to the public, for free, that they have used to make their very own websites. To top it off, they even have now released the source code for this Web Toolkit of theirs. The toolkit simplifies the programming process by converting easy-to-write Java code into the painfully tricky to perfect AJAX stylings. What does this mean to anyone who doesn't speak Geek? Well, expect fancier, more powerful, and more intuitive websites to start showing up on Internet that don't have multimillion dollar developer

Logitech LX 500 Review

So after a recent incident with my old keyboard involving a kitten and a bottle of diet coke, it was time for a new keyboard. Enter the Logitech LX 500. I found the keyboard for sale at my local computer shop as a weekend door crasher special for 15$ - strange how they still had plenty in stock despite it being Sunday afternoon. It fit my price range and included a decent mouse, so... :$: Cha Ching :$: The keyboard and mouse unboxed The 800 dpi wireless optical mouse is a dark blue and black in colour, with two main buttons, a horizontally and vertically scrolling capable clicking scroll wheel, and a smallish button underneath it for tabbing between windows. The tabbing button does not function without the included software being installed, but the basic mouse functions were working right away. A pair of regular Duracell batteries were included with it. Overall, my impression of the mouse is good for what it is. It feels nice and, even for it not being a very large mouse, it still fit