nathan_marz_storm nathan_marz_storm-2012 nathan_marz_storm-2012-33 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Someone asked me an interesting question the other day: "How did you justify taking such a huge risk on building Storm while working on a startup ?" (Storm is a realtime computation system). I can see how from an outsider's perspective investing in such a massive project seems extremely risky for a startup. From my perspective, though, building Storm wasn't risky at all. It was challenging, but not risky. I follow a style of development that greatly reduces the risk of big projects like Storm. I call this style "suffering-oriented programming." Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment. I ha
sentIndex sentText sentNum sentScore
1 Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment. [sent-10, score-0.568]
2 " First make it possible When encountering a problem domain with which you're unfamiliar, it's a mistake to try to build a "general" or "extensible" solution right off the bat. [sent-14, score-0.671]
3 You just don't understand the problem domain well enough to anticipate what your needs will be in the future. [sent-15, score-0.554]
4 You'll make things generic that needn't be, adding complexity and wasting time. [sent-16, score-0.455]
5 As you're hacking things out, you'll learn more and more about the intricacies of the problem space. [sent-19, score-0.596]
6 The "make it possible" phase for Storm was one year of hacking out a stream processing system using queues and workers. [sent-20, score-0.826]
7 At the same time, developing our product drove us to discover new use cases in the "realtime computation" problem space. [sent-29, score-0.49]
8 Then make it beautiful You develop a "map" of the problem space as you explore it by hacking things out. [sent-35, score-0.868]
9 Over time, you acquire more and more use cases within the problem domain and develop a deep understanding of the intricacies of building these systems. [sent-36, score-1.092]
10 The key to developing the "beautiful" solution is figuring out the simplest set of abstractions that solve the concrete use cases you already have. [sent-38, score-0.575]
11 As a rule of thumb, the bigger the investment you're trying to make, the deeper you need to understand the problem domain and the more diverse your use cases need to be. [sent-40, score-0.73]
12 "Making it beautiful" is where you use your design and abstraction skills to distill the problem space into simple abstractions that can be composed together. [sent-42, score-0.574]
13 I view the development of beautiful abstractions as similar to statistical regression: you have a set of points on a graph (your use cases) and you're looking for the simplest curve that fits those points (a set of abstractions). [sent-43, score-0.88]
14 A big part of making it beautiful is understanding the performance and resource characteristics of the problem space. [sent-46, score-0.855]
15 This is one of the intricacies you learn in the "making it possible" phase, and you should take advantage of that learning when designing your beautiful solution. [sent-47, score-0.475]
16 With Storm, I distilled the realtime computation problem domain into a small set of abstractions: streams, spouts, bolts, and topologies. [sent-48, score-0.714]
17 The beautiful systems you build give you new capabilities, which allow you to "make it possible" in new and deeper areas of the problem space. [sent-64, score-0.66]
18 By its nature, hacking things out in a problem domain you don't understand so well and constantly iterating can lead to some sloppy code. [sent-71, score-0.73]
19 Then you learn about making generic abstractions and using encapsulation to make it easier to reason about systems. [sent-81, score-0.601]
20 It recognizes that attempts to make things generic without a deep understanding of the problem domain will lead to complexity and waste. [sent-84, score-1.087]
wordName wordTfidf (topN-words)
[('beautiful', 0.3), ('storm', 0.241), ('domain', 0.24), ('abstractions', 0.21), ('problem', 0.21), ('cases', 0.187), ('stream', 0.18), ('queues', 0.173), ('realtime', 0.154), ('phase', 0.139), ('complexity', 0.139), ('generic', 0.139), ('programming', 0.131), ('processing', 0.122), ('hacking', 0.12), ('computation', 0.11), ('understanding', 0.11), ('anticipate', 0.104), ('risk', 0.097), ('use', 0.093), ('system', 0.092), ('things', 0.091), ('allow', 0.09), ('acquire', 0.09), ('intricacies', 0.09), ('workers', 0.09), ('reach', 0.088), ('make', 0.086), ('characteristics', 0.085), ('concrete', 0.085), ('curve', 0.085), ('guaranteeing', 0.085), ('risky', 0.085), ('learn', 0.085), ('making', 0.081), ('possible', 0.075), ('deep', 0.072), ('reduces', 0.069), ('direct', 0.069), ('pain', 0.069), ('streams', 0.069), ('extensible', 0.069), ('resource', 0.069), ('sloppy', 0.069), ('points', 0.066), ('fast', 0.066), ('space', 0.061), ('learned', 0.061), ('build', 0.06), ('graph', 0.06)]
simIndex simValue blogId blogTitle
same-blog 1 1.0000004 33 nathan marz storm-2012-02-06-Suffering-oriented programming
Introduction: Someone asked me an interesting question the other day: "How did you justify taking such a huge risk on building Storm while working on a startup ?" (Storm is a realtime computation system). I can see how from an outsider's perspective investing in such a massive project seems extremely risky for a startup. From my perspective, though, building Storm wasn't risky at all. It was challenging, but not risky. I follow a style of development that greatly reduces the risk of big projects like Storm. I call this style "suffering-oriented programming." Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment. I ha
2 0.31853619 39 nathan marz storm-2014-02-12-Interview with "Programmer Magazine"
Introduction: I was recently interviewed for "Programmer Magazine", a Chinese magazine. The interview was published in Chinese, but a lot of people told me they'd like to see the English version of the interview. Due to the Google translation being, ahem, a little iffy, I decided to just publish the original English version on my blog. Hope you enjoy! What drew you to programming and what was the first interesting program you wrote? I started programming when I was 10 years old on my TI-82 graphing calculator. Initially I started programming because I wanted to make games on my calculator – and also because I was bored in math class :D. The first interesting game I made on my calculator was an archery game where you'd shoot arrows at moving targets. You'd get points for hitting more targets or completing all the targets faster. A couple years later I graduated to programming the TI-89 which was a huge upgrade in power. I remember how the TI-82 only let you have 26 variables (for the character
3 0.21880129 34 nathan marz storm-2012-09-19-Storm's 1st birthday
Introduction: Storm was open-sourced exactly one year ago today. It's been an action-packed year for Storm, to say the least. Here's some of the exciting stuff that's happened over the past year: 27 companies have publicized that they're using Storm in production . I know of at least a few more companies using it that haven't published anything yet. O'Reilly published a book on Storm. The Storm mailing list has over 1300 members, with over 500 messages per month. The @stormprocessor account has over 1200 followers. More than 4000 people have starred the project on Github . There's a regular Storm meetup in the Bay Area with over 230 members. I've also seen lots of Storm-focused meetups happen all over the world over the past year. 29 people all over the world have contributed to the codebase We released Trident , a high level abstraction for realtime computation, that is a major leap forward in what's possible in realtime. Libraries have been released integrating Stor
4 0.18491732 31 nathan marz storm-2011-10-13-How to beat the CAP theorem
Introduction: The CAP theorem states a database cannot guarantee consistency, availability, and partition-tolerance at the same time. But you can't sacrifice partition-tolerance (see here and here ), so you must make a tradeoff between availability and consistency. Managing this tradeoff is a central focus of the NoSQL movement. Consistency means that after you do a successful write, future reads will always take that write into account. Availability means that you can always read and write to the system. During a partition, you can only have one of these properties. Systems that choose consistency over availability have to deal with some awkward issues. What do you do when the database isn't available? You can try buffering writes for later, but you risk losing those writes if you lose the machine with the buffer. Also, buffering writes can be a form of inconsistency because a client thinks a write has succeeded but the write isn't in the database yet. Alternatively, you can return errors ba
5 0.11195478 37 nathan marz storm-2013-04-02-Principles of Software Engineering, Part 1
Introduction: This is the first in a series of posts on the principles of software engineering. There's far more to software engineering than just "making computers do stuff" – while that phrase is accurate, it does not come close to describing what's involved in making robust, reliable software. I will use my experience building large scale systems to inform a first principles approach to defining what it is we do – or should be doing – as software engineers. I'm not interested in tired debates like dynamic vs. static languages – instead, I intend to explore the really core aspects of software engineering. The first order of business is to define what software engineering even is in the first place. Software engineering is the construction of software that produces some desired output for some range of inputs. The inputs to software are more than just method parameters: they include the hardware on which it's running, the rate at which it receives data, and anything else that influences the oper
6 0.095923632 19 nathan marz storm-2010-07-12-My experience as the first employee of a Y Combinator startup
7 0.09391652 30 nathan marz storm-2011-03-29-My talks at POSSCON
8 0.079001531 36 nathan marz storm-2013-04-01-My new startup
9 0.074905932 22 nathan marz storm-2010-10-05-How to get a job at a kick-ass startup (for programmers)
10 0.069815084 38 nathan marz storm-2013-04-12-Break into Silicon Valley with a blog
11 0.069071256 20 nathan marz storm-2010-07-30-You should blog even if you have no readers
12 0.066994049 2 nathan marz storm-2010-01-03-Tips for Optimizing Cascading Flows
13 0.063408941 7 nathan marz storm-2010-03-04-Introducing "Nanny" - a really simple dependency management tool
14 0.063076667 23 nathan marz storm-2010-10-27-Fastest Viable Product: Investing in Speed at a Startup
15 0.062585562 25 nathan marz storm-2010-12-06-You Are a Product
16 0.055358585 3 nathan marz storm-2010-01-13-Mimi Silbert: the greatest hacker in the world
17 0.055231616 21 nathan marz storm-2010-08-20-5 Tips for Thinking Under Uncertainty
18 0.052409135 18 nathan marz storm-2010-06-16-Your company has a knowledge debt problem
19 0.052009627 40 nathan marz storm-2014-02-24-The inexplicable rise of open floor plans in tech companies
20 0.051089037 41 nathan marz storm-2014-05-10-Why we in tech must support Lawrence Lessig
topicId topicWeight
[(0, 0.409), (1, 0.131), (2, -0.019), (3, 0.459), (4, -0.103), (5, 0.13), (6, 0.003), (7, -0.124), (8, 0.178), (9, -0.033), (10, 0.017), (11, 0.002), (12, -0.164), (13, -0.038), (14, -0.151), (15, 0.043), (16, -0.095), (17, 0.096), (18, -0.045), (19, 0.023), (20, 0.068), (21, -0.115), (22, 0.05), (23, -0.022), (24, 0.075), (25, 0.004), (26, -0.052), (27, 0.053), (28, 0.027), (29, 0.07), (30, -0.059), (31, -0.354), (32, -0.231), (33, 0.155), (34, 0.028), (35, -0.122), (36, 0.303), (37, -0.018), (38, 0.32), (39, 0.022), (40, 0.029)]
simIndex simValue blogId blogTitle
same-blog 1 0.98928463 33 nathan marz storm-2012-02-06-Suffering-oriented programming
Introduction: Someone asked me an interesting question the other day: "How did you justify taking such a huge risk on building Storm while working on a startup ?" (Storm is a realtime computation system). I can see how from an outsider's perspective investing in such a massive project seems extremely risky for a startup. From my perspective, though, building Storm wasn't risky at all. It was challenging, but not risky. I follow a style of development that greatly reduces the risk of big projects like Storm. I call this style "suffering-oriented programming." Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment. I ha
2 0.38807574 39 nathan marz storm-2014-02-12-Interview with "Programmer Magazine"
Introduction: I was recently interviewed for "Programmer Magazine", a Chinese magazine. The interview was published in Chinese, but a lot of people told me they'd like to see the English version of the interview. Due to the Google translation being, ahem, a little iffy, I decided to just publish the original English version on my blog. Hope you enjoy! What drew you to programming and what was the first interesting program you wrote? I started programming when I was 10 years old on my TI-82 graphing calculator. Initially I started programming because I wanted to make games on my calculator – and also because I was bored in math class :D. The first interesting game I made on my calculator was an archery game where you'd shoot arrows at moving targets. You'd get points for hitting more targets or completing all the targets faster. A couple years later I graduated to programming the TI-89 which was a huge upgrade in power. I remember how the TI-82 only let you have 26 variables (for the character
3 0.24914907 34 nathan marz storm-2012-09-19-Storm's 1st birthday
Introduction: Storm was open-sourced exactly one year ago today. It's been an action-packed year for Storm, to say the least. Here's some of the exciting stuff that's happened over the past year: 27 companies have publicized that they're using Storm in production . I know of at least a few more companies using it that haven't published anything yet. O'Reilly published a book on Storm. The Storm mailing list has over 1300 members, with over 500 messages per month. The @stormprocessor account has over 1200 followers. More than 4000 people have starred the project on Github . There's a regular Storm meetup in the Bay Area with over 230 members. I've also seen lots of Storm-focused meetups happen all over the world over the past year. 29 people all over the world have contributed to the codebase We released Trident , a high level abstraction for realtime computation, that is a major leap forward in what's possible in realtime. Libraries have been released integrating Stor
4 0.24595979 31 nathan marz storm-2011-10-13-How to beat the CAP theorem
Introduction: The CAP theorem states a database cannot guarantee consistency, availability, and partition-tolerance at the same time. But you can't sacrifice partition-tolerance (see here and here ), so you must make a tradeoff between availability and consistency. Managing this tradeoff is a central focus of the NoSQL movement. Consistency means that after you do a successful write, future reads will always take that write into account. Availability means that you can always read and write to the system. During a partition, you can only have one of these properties. Systems that choose consistency over availability have to deal with some awkward issues. What do you do when the database isn't available? You can try buffering writes for later, but you risk losing those writes if you lose the machine with the buffer. Also, buffering writes can be a form of inconsistency because a client thinks a write has succeeded but the write isn't in the database yet. Alternatively, you can return errors ba
5 0.1784838 36 nathan marz storm-2013-04-01-My new startup
Introduction: There's been a lot of speculation about what my new startup is doing, so I've decided to set the record straight and reveal all. We are working on one of the biggest problems on Earth, a problem that affects nearly every person on this planet. Our products will significantly improve the quality of life for billions of people. We are going to revolutionize the bedsheet industry. Think about it. There's been almost no innovation in bedsheets in thousands of years. There's nothing worse than waking up to discover one of the corners of your Egyptian cotton fitted sheets has slipped off the mattress. How is this not a solved problem yet? Why are we still using sheets with that annoying elastic in it to secure them to our mattresses? They slip all the time – and if you have a deep mattress, good luck finding sheets that even fit. You're just screwed. Consider the impact of solving this problem, of a bedsheet product that never slips, that always stays secure on the mattre
6 0.1676154 37 nathan marz storm-2013-04-02-Principles of Software Engineering, Part 1
7 0.16187274 19 nathan marz storm-2010-07-12-My experience as the first employee of a Y Combinator startup
8 0.14614742 22 nathan marz storm-2010-10-05-How to get a job at a kick-ass startup (for programmers)
9 0.13978402 3 nathan marz storm-2010-01-13-Mimi Silbert: the greatest hacker in the world
10 0.12891763 2 nathan marz storm-2010-01-03-Tips for Optimizing Cascading Flows
11 0.11760958 30 nathan marz storm-2011-03-29-My talks at POSSCON
12 0.11541726 23 nathan marz storm-2010-10-27-Fastest Viable Product: Investing in Speed at a Startup
13 0.11415955 7 nathan marz storm-2010-03-04-Introducing "Nanny" - a really simple dependency management tool
14 0.11405587 18 nathan marz storm-2010-06-16-Your company has a knowledge debt problem
15 0.10999078 38 nathan marz storm-2013-04-12-Break into Silicon Valley with a blog
16 0.10904706 9 nathan marz storm-2010-03-10-Thrift + Graphs = Strong, flexible schemas on Hadoop
17 0.10836838 41 nathan marz storm-2014-05-10-Why we in tech must support Lawrence Lessig
18 0.10465762 40 nathan marz storm-2014-02-24-The inexplicable rise of open floor plans in tech companies
19 0.10375202 21 nathan marz storm-2010-08-20-5 Tips for Thinking Under Uncertainty
20 0.10219275 25 nathan marz storm-2010-12-06-You Are a Product
topicId topicWeight
[(0, 0.019), (5, 0.018), (10, 0.036), (26, 0.022), (41, 0.057), (48, 0.019), (59, 0.043), (68, 0.567), (76, 0.025), (82, 0.04), (88, 0.072)]
simIndex simValue blogId blogTitle
same-blog 1 0.98643702 33 nathan marz storm-2012-02-06-Suffering-oriented programming
Introduction: Someone asked me an interesting question the other day: "How did you justify taking such a huge risk on building Storm while working on a startup ?" (Storm is a realtime computation system). I can see how from an outsider's perspective investing in such a massive project seems extremely risky for a startup. From my perspective, though, building Storm wasn't risky at all. It was challenging, but not risky. I follow a style of development that greatly reduces the risk of big projects like Storm. I call this style "suffering-oriented programming." Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment. I ha
2 0.36398149 39 nathan marz storm-2014-02-12-Interview with "Programmer Magazine"
Introduction: I was recently interviewed for "Programmer Magazine", a Chinese magazine. The interview was published in Chinese, but a lot of people told me they'd like to see the English version of the interview. Due to the Google translation being, ahem, a little iffy, I decided to just publish the original English version on my blog. Hope you enjoy! What drew you to programming and what was the first interesting program you wrote? I started programming when I was 10 years old on my TI-82 graphing calculator. Initially I started programming because I wanted to make games on my calculator – and also because I was bored in math class :D. The first interesting game I made on my calculator was an archery game where you'd shoot arrows at moving targets. You'd get points for hitting more targets or completing all the targets faster. A couple years later I graduated to programming the TI-89 which was a huge upgrade in power. I remember how the TI-82 only let you have 26 variables (for the character
3 0.35141051 31 nathan marz storm-2011-10-13-How to beat the CAP theorem
Introduction: The CAP theorem states a database cannot guarantee consistency, availability, and partition-tolerance at the same time. But you can't sacrifice partition-tolerance (see here and here ), so you must make a tradeoff between availability and consistency. Managing this tradeoff is a central focus of the NoSQL movement. Consistency means that after you do a successful write, future reads will always take that write into account. Availability means that you can always read and write to the system. During a partition, you can only have one of these properties. Systems that choose consistency over availability have to deal with some awkward issues. What do you do when the database isn't available? You can try buffering writes for later, but you risk losing those writes if you lose the machine with the buffer. Also, buffering writes can be a form of inconsistency because a client thinks a write has succeeded but the write isn't in the database yet. Alternatively, you can return errors ba
4 0.28585038 7 nathan marz storm-2010-03-04-Introducing "Nanny" - a really simple dependency management tool
Introduction: Dependency management in software projects is a pretty simple problem when you think about it. A tool to manage dependencies just needs to do three things: Provide a mechanism to specify the direct dependencies to a project Download the transitive closure of dependencies to a project Publish packages that can be used as a dependency to other projects Some languages have good dependency management systems - for example, rubygems. Others, like Java, have tools like Maven which I would call a complex solution to a simple problem. You shouldn't need to buy a book to understand the solution to such a simple problem. Plus, these dependency management systems are all language specific. I've seen companies do crazy things to manage their dependencies. One company, to manage their jar files, would put all the jars that any project might need in a special "jars" project. You would then need to setup a JARS_HOME environment variable and be sure to update the jars project if you n
5 0.22801201 9 nathan marz storm-2010-03-10-Thrift + Graphs = Strong, flexible schemas on Hadoop
Introduction: There are a lot of misconceptions about what Hadoop is useful for and what kind of data you can put in it. A lot of people think that Hadoop is meant for unstructured data like log files. While Hadoop is great for log files, it's also fantastic for strongly typed, structured data. In this post I'll discuss how you can use a tool like Thrift to store strongly typed data in Hadoop while retaining the flexibility to evolve your schema. We'll look at graph-based schemas and see why they are an ideal fit for many Hadoop-based applications. OK, so what kind of "structured" data can you put in Hadoop? Anything! At BackType we put data about news, conversations, and people into Hadoop as structured objects. You can easily push structured information about social graphs, financial information, or anything you want into Hadoop. Â That sounds all well and good, but why not just use JSON as the data format? JSON doesn't give you a real schema and doesn't protect against data i
6 0.21999253 29 nathan marz storm-2011-01-19-Inglourious Software Patents
7 0.2185902 1 nathan marz storm-2009-12-28-The mathematics behind Hadoop-based systems
8 0.20990665 3 nathan marz storm-2010-01-13-Mimi Silbert: the greatest hacker in the world
9 0.20939019 23 nathan marz storm-2010-10-27-Fastest Viable Product: Investing in Speed at a Startup
10 0.20642915 37 nathan marz storm-2013-04-02-Principles of Software Engineering, Part 1
11 0.20463572 40 nathan marz storm-2014-02-24-The inexplicable rise of open floor plans in tech companies
12 0.19392847 36 nathan marz storm-2013-04-01-My new startup
13 0.18293135 8 nathan marz storm-2010-03-08-Follow-up to "The mathematics behind Hadoop-based systems"
14 0.17989443 18 nathan marz storm-2010-06-16-Your company has a knowledge debt problem
15 0.16656137 21 nathan marz storm-2010-08-20-5 Tips for Thinking Under Uncertainty
16 0.16413561 25 nathan marz storm-2010-12-06-You Are a Product
17 0.16214782 13 nathan marz storm-2010-04-14-Introducing Cascalog: a Clojure-based query language for Hadoop
18 0.1578099 22 nathan marz storm-2010-10-05-How to get a job at a kick-ass startup (for programmers)
19 0.15308796 10 nathan marz storm-2010-03-17-Proof that 1 = 0 using a common logical fallacy
20 0.14824367 2 nathan marz storm-2010-01-03-Tips for Optimizing Cascading Flows