high_scalability high_scalability-2008 high_scalability-2008-269 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Update 3: Always Refer to Your V1 As a Prototype . You really do have to plan to throw one away. Update 2: Lessons Learned Scaling the Audiogalaxy Search Engine . Things he should have done and fun things he couldn’t justify doing. Update: Design details of Audiogalaxy.com’s high performance MySQL search engine . At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. Search was one of most interesting problems at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows.
sentIndex sentText sentNum sentScore
1 Things he should have done and fun things he couldn’t justify doing. [sent-4, score-0.499]
2 At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. [sent-7, score-2.091]
3 Search was one of most interesting problems at Audiogalaxy. [sent-8, score-0.194]
4 It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. [sent-9, score-1.404]
5 At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. [sent-10, score-2.091]
wordName wordTfidf (topN-words)
[('searches', 0.494), ('search', 0.316), ('couldn', 0.279), ('peak', 0.257), ('justify', 0.202), ('refer', 0.2), ('mysql', 0.195), ('million', 0.193), ('engine', 0.182), ('needed', 0.164), ('somewhere', 0.156), ('throw', 0.151), ('every', 0.145), ('performed', 0.139), ('rows', 0.139), ('second', 0.138), ('times', 0.135), ('handle', 0.129), ('functions', 0.118), ('things', 0.115), ('plan', 0.11), ('lessons', 0.109), ('fun', 0.105), ('learned', 0.102), ('details', 0.089), ('one', 0.08), ('core', 0.079), ('update', 0.079), ('done', 0.077), ('database', 0.073), ('always', 0.063), ('site', 0.062), ('design', 0.059), ('interesting', 0.058), ('problems', 0.056), ('scaling', 0.054), ('really', 0.049), ('high', 0.041), ('performance', 0.034)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
Introduction: Update 3: Always Refer to Your V1 As a Prototype . You really do have to plan to throw one away. Update 2: Lessons Learned Scaling the Audiogalaxy Search Engine . Things he should have done and fun things he couldn’t justify doing. Update: Design details of Audiogalaxy.com’s high performance MySQL search engine . At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. Search was one of most interesting problems at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows.
2 0.21265011 258 high scalability-2008-02-24-Yandex Architecture
Introduction: Update: Anatomy of a crash in a new part of Yandex written in Django . Writing to a magic session variable caused an unexpected write into an InnoDB database on every request. Writes took 6-7 seconds because of index rebuilding. Lots of useful details on the sizing of their system, what went wrong, and how they fixed it. Yandex is a Russian search engine with 3.5 billion pages in their search index. We only know a few fun facts about how they do things, nothing at a detailed architecture level. Hopefully we'll learn more later, but I thought it would still be interesting. From Allen Stern's interview with Yandex's CTO Ilya Segalovich, we learn: 3.5 billion pages in the search index. Over several thousand servers. 35 million searches a day. Several data centers around Russia. Two-layer architecture. The database is split in pieces and when a search is requested, it pulls the bits from the different database servers and brings it together for the user. Languages
3 0.1928591 332 high scalability-2008-05-28-Job queue and search engine
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
4 0.16554686 746 high scalability-2009-11-26-Kngine Snippet Search New Indexing Technology
Introduction: While Kngine just announce some improvement and new features , I would like you take you in small trip in Snippet Search research project at Kngine. What is Kngine? Kngine is startup company working in Searching technologies, We in Kngine aims to organize the human beings Systematic Knowledge and Experiences and make it accessible to everyone. We aim to collect and organize all objective data, and make it possible and easy to access. Our goal is to build Web 3.0 Web Search Engine on the advances of Web Search Engine, Semantic Web, Data Representation technologies a new form of Web Search Engine that will unleash a revolution of new possibilities. Introduction to Snippet Search Today, The Web Search Engine’s is the Web getaway, especially to get specific information. But unfortunately the search engines didn’t changed mush as the Web changed from 90’s. Since the 90’s the Web search engine still provide the same kind of results: Links to documents. We i
5 0.13186802 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
Introduction: This is an interview with Gabriel Weinberg , founder of Duck Duck Go and general all around startup guru , on what DDG’s architecture looks like in 2012. Innovative search engine upstart DuckDuckGo had 30 million searches in February 2012 and averages over 1 million searches a day. It’s being positioned by super investor Fred Wilson as a clean, private, impartial and fast search engine. After talking with Gabriel I like what Fred Wilson said earlier, it seems closer to the heart of the matter: We invested in DuckDuckGo for the Reddit, Hacker News anarchists . Choosing DuckDuckGo can be thought of as not just a technical choice, but a vote for revolution. In an age when knowing your essence is not about about love or friendship, but about more effectively selling you to advertisers, DDG is positioning themselves as the do not track alternative , keepers of the privacy flame . You will still be monetized of course, but in a more civilized and an
6 0.13058004 856 high scalability-2010-07-12-Creating Scalable Digital Libraries
7 0.12291311 196 high scalability-2007-12-30-MySQL clustering strategies and comparisions
8 0.11875255 899 high scalability-2010-09-09-How did Google Instant become Faster with 5-7X More Results Pages?
9 0.11791687 342 high scalability-2008-06-08-Search fast in million rows
10 0.11164499 233 high scalability-2008-01-30-How Rackspace Now Uses MapReduce and Hadoop to Query Terabytes of Data
11 0.1092653 934 high scalability-2010-11-04-Facebook at 13 Million Queries Per Second Recommends: Minimize Request Variance
12 0.10612001 638 high scalability-2009-06-26-PlentyOfFish Architecture
13 0.10389334 1361 high scalability-2012-11-22-Gone Fishin': PlentyOfFish Architecture
14 0.10369477 689 high scalability-2009-08-28-Strategy: Solve Only 80 Percent of the Problem
15 0.10017773 560 high scalability-2009-04-08-Learned lessons from the largest player (Flickr, YouTube, Google, etc)
16 0.098077551 1609 high scalability-2014-03-11-Building a Social Music Service Using AWS, Scala, Akka, Play, MongoDB, and Elasticsearch
17 0.097492047 17 high scalability-2007-07-16-Paper: Guide to Cost-effective Database Scale-Out using MySQL
18 0.097076245 630 high scalability-2009-06-14-kngine 'Knowledge Engine' milestone 2
19 0.095854685 770 high scalability-2010-02-03-NoSQL Means Never Having to Store Blobs Again
20 0.094628334 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
topicId topicWeight
[(0, 0.13), (1, 0.071), (2, -0.063), (3, -0.072), (4, 0.028), (5, 0.02), (6, -0.091), (7, 0.012), (8, 0.054), (9, -0.0), (10, -0.005), (11, -0.038), (12, 0.04), (13, 0.046), (14, 0.003), (15, 0.029), (16, -0.075), (17, -0.011), (18, 0.057), (19, 0.026), (20, 0.131), (21, -0.046), (22, -0.063), (23, 0.008), (24, -0.032), (25, 0.025), (26, -0.119), (27, 0.071), (28, 0.092), (29, 0.082), (30, -0.05), (31, 0.037), (32, -0.052), (33, 0.039), (34, 0.124), (35, 0.038), (36, -0.045), (37, -0.009), (38, -0.046), (39, -0.021), (40, 0.116), (41, -0.065), (42, 0.026), (43, 0.071), (44, -0.029), (45, 0.048), (46, 0.009), (47, 0.095), (48, 0.045), (49, -0.024)]
simIndex simValue blogId blogTitle
same-blog 1 0.98718566 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
Introduction: Update 3: Always Refer to Your V1 As a Prototype . You really do have to plan to throw one away. Update 2: Lessons Learned Scaling the Audiogalaxy Search Engine . Things he should have done and fun things he couldn’t justify doing. Update: Design details of Audiogalaxy.com’s high performance MySQL search engine . At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. Search was one of most interesting problems at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows.
2 0.83714181 258 high scalability-2008-02-24-Yandex Architecture
Introduction: Update: Anatomy of a crash in a new part of Yandex written in Django . Writing to a magic session variable caused an unexpected write into an InnoDB database on every request. Writes took 6-7 seconds because of index rebuilding. Lots of useful details on the sizing of their system, what went wrong, and how they fixed it. Yandex is a Russian search engine with 3.5 billion pages in their search index. We only know a few fun facts about how they do things, nothing at a detailed architecture level. Hopefully we'll learn more later, but I thought it would still be interesting. From Allen Stern's interview with Yandex's CTO Ilya Segalovich, we learn: 3.5 billion pages in the search index. Over several thousand servers. 35 million searches a day. Several data centers around Russia. Two-layer architecture. The database is split in pieces and when a search is requested, it pulls the bits from the different database servers and brings it together for the user. Languages
3 0.77102029 332 high scalability-2008-05-28-Job queue and search engine
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
4 0.76132303 342 high scalability-2008-06-08-Search fast in million rows
Introduction: I have a table .This table has many columns but search performed based on 1 columns ,this table can have more than million rows. The data in these columns is something like funny,new york,hollywood User can search with parameters as funny hollywood .I need to take this 2 words and then search on column whether that column contain this words and how many times .It is not possible to index here .If the results return say 1200 results then without comparing each and every column i can't determine no of results.I need to compare for each and every column.This query is very frequent .How can i approach for this problem.What type of architecture,tools is helpful. I just know that this can be accomplished with distributed system but how can i make this system. I also see in this website that LinkedIn uses Lucene for search .Is Lucene is helpful in my case.My table has also lots of insertion ,however updation in not very frequent.
5 0.70462829 746 high scalability-2009-11-26-Kngine Snippet Search New Indexing Technology
Introduction: While Kngine just announce some improvement and new features , I would like you take you in small trip in Snippet Search research project at Kngine. What is Kngine? Kngine is startup company working in Searching technologies, We in Kngine aims to organize the human beings Systematic Knowledge and Experiences and make it accessible to everyone. We aim to collect and organize all objective data, and make it possible and easy to access. Our goal is to build Web 3.0 Web Search Engine on the advances of Web Search Engine, Semantic Web, Data Representation technologies a new form of Web Search Engine that will unleash a revolution of new possibilities. Introduction to Snippet Search Today, The Web Search Engine’s is the Web getaway, especially to get specific information. But unfortunately the search engines didn’t changed mush as the Web changed from 90’s. Since the 90’s the Web search engine still provide the same kind of results: Links to documents. We i
6 0.69115376 246 high scalability-2008-02-12-Search the tags across all post
7 0.67541522 64 high scalability-2007-08-10-How do we make a large real-time search engine?
8 0.65106177 899 high scalability-2010-09-09-How did Google Instant become Faster with 5-7X More Results Pages?
9 0.64865786 1601 high scalability-2014-02-25-Peter Norvig's 9 Master Steps to Improving a Program
10 0.64545482 630 high scalability-2009-06-14-kngine 'Knowledge Engine' milestone 2
11 0.62498653 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
12 0.5874185 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
13 0.58479267 1295 high scalability-2012-08-02-Ask DuckDuckGo: Is there Anything you Want to Know About DDG?
14 0.57751608 810 high scalability-2010-04-14-Parallel Information Retrieval and Other Search Engine Goodness
15 0.54892772 1233 high scalability-2012-04-25-The Anatomy of Search Technology: blekko’s NoSQL database
16 0.5295018 465 high scalability-2008-12-14-Scaling MySQL on a 256-way T5440 server using Solaris ZFS and Java 1.7
17 0.52807719 1253 high scalability-2012-05-28-The Anatomy of Search Technology: Crawling using Combinators
18 0.52441889 455 high scalability-2008-12-01-MySQL Database Scale-out and Replication for High Growth Businesses
19 0.5238269 331 high scalability-2008-05-27-eBay Architecture
20 0.52035326 926 high scalability-2010-10-24-Hot Scalability Links For Oct 24, 2010
topicId topicWeight
[(0, 0.113), (2, 0.248), (10, 0.239), (61, 0.144), (85, 0.078), (94, 0.027)]
simIndex simValue blogId blogTitle
1 0.91505224 689 high scalability-2009-08-28-Strategy: Solve Only 80 Percent of the Problem
Introduction: Solve only 80% of a problem. That's usually good enough and you'll not only get done faster, you'll actually have a chance of getting done at all. This strategy is given by Amix in HOW TWITTER (AND FACEBOOK) SOLVE PROBLEMS PARTIALLY . The idea is solving 100% of a complex problem can be so hard and so expensive that you'll end up wasting all your bullets on a problem that could have been satisfactoraly solved in a much simpler way. The example given is for Twitter's real-time search. Real-time search almost by definition is focussed on recent events. So in the design should you be able to search historically back from the beginning of time or should you just be able to search for recent time periods? A complete historical search is the 100% solution. The recent data only search is the 80% solution. Which should you choose? The 100% solution is dramatically more difficult to solve. It requires searching disk in real-time which is a killer. So it makes more sense to work on the
same-blog 2 0.90885276 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
Introduction: Update 3: Always Refer to Your V1 As a Prototype . You really do have to plan to throw one away. Update 2: Lessons Learned Scaling the Audiogalaxy Search Engine . Things he should have done and fun things he couldn’t justify doing. Update: Design details of Audiogalaxy.com’s high performance MySQL search engine . At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. Search was one of most interesting problems at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows.
Introduction: Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian , VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale . If you want to know how Twitter works - then start here. It happened gradually so you may have missed it, but Twitter has grown up. It started as a struggling three-tierish Ruby on Rails website to become a beautifully service driven core that we actually go to now to see if other services are down. Quite a change. Twitter now has 150M world wide active users, handles 300K QPS to generate timelines, and a firehose that churns out 22 MB/sec. 400 million tweets a day flow through the system and it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers. A couple o
4 0.84935093 575 high scalability-2009-04-21-Thread Pool Engine in MS CLR 4, and Work-Stealing scheduling algorithm
Introduction: I just saw this article in HFadeel blog that spaek about Parallelism in .NET Framework 4, and how Thread Pool work, and the most faoums scheduling algorithm : Work-stealing algorithm. With preisnation to see it in action.
5 0.8487069 1631 high scalability-2014-04-14-How do you even do anything without using EBS?
Introduction: In a recent thread on Hacker News discussing recent AWS price changes , seldo mentioned they use AWS for business, they just never use EBS on AWS. A good question was asked: How do you even do anything without using EBS? Amazon certainly makes using EBS the easiest path. And EBS has a better reliability record as of late, but it's still often recommended to not use EBS. This avoids a single point of failure at the cost of a lot of complexity, though as AWS uses EBS internally, not using EBS may not save you if you use other AWS services like RDS or ELB. If you don't want to use EBS, it's hard to know where to even start. A dilemma to which Kevin Nuckolls gives a great answer : Well, you break your services out onto stateless and stateful machines. After that, you make sure that each of your stateful services is resilient to individual node failure. I prefer to believe that if you can't roll your entire infrastructure over to new nodes monthly then you're unprepared fo
6 0.83334249 1066 high scalability-2011-06-22-It's the Fraking IOPS - 1 SSD is 44,000 IOPS, Hard Drive is 180
7 0.83001423 1299 high scalability-2012-08-06-Paper: High-Performance Concurrency Control Mechanisms for Main-Memory Databases
8 0.8300029 178 high scalability-2007-12-10-1 Master, N Slaves
9 0.82977349 792 high scalability-2010-03-10-How FarmVille Scales - The Follow-up
10 0.81667763 1585 high scalability-2014-01-24-Stuff The Internet Says On Scalability For January 24th, 2014
11 0.81619769 584 high scalability-2009-04-27-Some Questions from a newbie
12 0.813945 1046 high scalability-2011-05-23-Evernote Architecture - 9 Million Users and 150 Million Requests a Day
13 0.80690849 874 high scalability-2010-08-07-ArchCamp: Scalable Databases (NoSQL)
14 0.80608737 1004 high scalability-2011-03-14-Twitter by the Numbers - 460,000 New Accounts and 140 Million Tweets Per Day
15 0.80524856 271 high scalability-2008-03-08-Product: DRBD - Distributed Replicated Block Device
16 0.80147392 1635 high scalability-2014-04-21-This is why Microsoft won. And why they lost.
17 0.79891825 1331 high scalability-2012-10-02-An Epic TripAdvisor Update: Why Not Run on the Cloud? The Grand Experiment.
18 0.79843879 142 high scalability-2007-11-05-Strategy: Diagonal Scaling - Don't Forget to Scale Out AND Up
19 0.79396379 257 high scalability-2008-02-22-Kevin's Great Adventures in SSDland
20 0.79114097 109 high scalability-2007-10-03-Save on a Load Balancer By Using Client Side Load Balancing