high_scalability high_scalability-2008 high_scalability-2008-332 knowledge-graph by maker-knowledge-mining
Source: html
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
sentIndex sentText sentNum sentScore
1 Hi, I want to implement a search engine with lucene. [sent-1, score-0.84]
2 To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). [sent-2, score-1.535]
3 (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. [sent-8, score-1.439]
4 I can set a short expiration time (~5 min) for each search result, but it's still large. [sent-9, score-1.113]
wordName wordTfidf (topN-words)
[('search', 0.526), ('eg', 0.355), ('min', 0.29), ('expiration', 0.27), ('asynchronously', 0.242), ('results', 0.227), ('queuing', 0.205), ('documents', 0.185), ('execute', 0.184), ('asynchronous', 0.153), ('jobs', 0.148), ('pages', 0.14), ('implement', 0.128), ('short', 0.126), ('engine', 0.121), ('result', 0.118), ('page', 0.105), ('would', 0.102), ('job', 0.095), ('store', 0.095), ('design', 0.079), ('set', 0.078), ('still', 0.073), ('scalable', 0.072), ('think', 0.07), ('per', 0.067), ('know', 0.066), ('want', 0.065), ('large', 0.061), ('good', 0.055), ('need', 0.046), ('system', 0.04), ('time', 0.04), ('use', 0.035), ('like', 0.033)]
simIndex simValue blogId blogTitle
same-blog 1 1.0 332 high scalability-2008-05-28-Job queue and search engine
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
2 0.32043365 746 high scalability-2009-11-26-Kngine Snippet Search New Indexing Technology
Introduction: While Kngine just announce some improvement and new features , I would like you take you in small trip in Snippet Search research project at Kngine. What is Kngine? Kngine is startup company working in Searching technologies, We in Kngine aims to organize the human beings Systematic Knowledge and Experiences and make it accessible to everyone. We aim to collect and organize all objective data, and make it possible and easy to access. Our goal is to build Web 3.0 Web Search Engine on the advances of Web Search Engine, Semantic Web, Data Representation technologies a new form of Web Search Engine that will unleash a revolution of new possibilities. Introduction to Snippet Search Today, The Web Search Engine’s is the Web getaway, especially to get specific information. But unfortunately the search engines didn’t changed mush as the Web changed from 90’s. Since the 90’s the Web search engine still provide the same kind of results: Links to documents. We i
3 0.1928591 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
Introduction: Update 3: Always Refer to Your V1 As a Prototype . You really do have to plan to throw one away. Update 2: Lessons Learned Scaling the Audiogalaxy Search Engine . Things he should have done and fun things he couldn’t justify doing. Update: Design details of Audiogalaxy.com’s high performance MySQL search engine . At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. Search was one of most interesting problems at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows.
4 0.17700855 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
Introduction: This is an interview with Gabriel Weinberg , founder of Duck Duck Go and general all around startup guru , on what DDG’s architecture looks like in 2012. Innovative search engine upstart DuckDuckGo had 30 million searches in February 2012 and averages over 1 million searches a day. It’s being positioned by super investor Fred Wilson as a clean, private, impartial and fast search engine. After talking with Gabriel I like what Fred Wilson said earlier, it seems closer to the heart of the matter: We invested in DuckDuckGo for the Reddit, Hacker News anarchists . Choosing DuckDuckGo can be thought of as not just a technical choice, but a vote for revolution. In an age when knowing your essence is not about about love or friendship, but about more effectively selling you to advertisers, DDG is positioning themselves as the do not track alternative , keepers of the privacy flame . You will still be monetized of course, but in a more civilized and an
5 0.17319921 258 high scalability-2008-02-24-Yandex Architecture
Introduction: Update: Anatomy of a crash in a new part of Yandex written in Django . Writing to a magic session variable caused an unexpected write into an InnoDB database on every request. Writes took 6-7 seconds because of index rebuilding. Lots of useful details on the sizing of their system, what went wrong, and how they fixed it. Yandex is a Russian search engine with 3.5 billion pages in their search index. We only know a few fun facts about how they do things, nothing at a detailed architecture level. Hopefully we'll learn more later, but I thought it would still be interesting. From Allen Stern's interview with Yandex's CTO Ilya Segalovich, we learn: 3.5 billion pages in the search index. Over several thousand servers. 35 million searches a day. Several data centers around Russia. Two-layer architecture. The database is split in pieces and when a search is requested, it pulls the bits from the different database servers and brings it together for the user. Languages
6 0.16292354 342 high scalability-2008-06-08-Search fast in million rows
7 0.14894962 630 high scalability-2009-06-14-kngine 'Knowledge Engine' milestone 2
8 0.14877969 899 high scalability-2010-09-09-How did Google Instant become Faster with 5-7X More Results Pages?
10 0.138042 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
11 0.13609391 406 high scalability-2008-10-08-Strategy: Flickr - Do the Essential Work Up-front and Queue the Rest
12 0.12604415 331 high scalability-2008-05-27-eBay Architecture
13 0.1226569 856 high scalability-2010-07-12-Creating Scalable Digital Libraries
14 0.12243707 64 high scalability-2007-08-10-How do we make a large real-time search engine?
15 0.11645368 674 high scalability-2009-08-07-The Canonical Cloud Architecture
16 0.11299618 1052 high scalability-2011-06-03-Stuff The Internet Says On Scalability For June 3, 2011
17 0.11243387 233 high scalability-2008-01-30-How Rackspace Now Uses MapReduce and Hadoop to Query Terabytes of Data
18 0.1091129 689 high scalability-2009-08-28-Strategy: Solve Only 80 Percent of the Problem
19 0.10907171 365 high scalability-2008-08-16-Strategy: Serve Pre-generated Static Files Instead Of Dynamic Pages
20 0.10120075 834 high scalability-2010-06-01-Web Speed Can Push You Off of Google Search Rankings! What Can You Do?
topicId topicWeight
[(0, 0.11), (1, 0.078), (2, -0.02), (3, -0.037), (4, 0.019), (5, 0.013), (6, 0.004), (7, 0.042), (8, 0.057), (9, 0.035), (10, 0.033), (11, -0.031), (12, -0.022), (13, -0.025), (14, 0.04), (15, -0.015), (16, -0.105), (17, -0.024), (18, 0.115), (19, -0.011), (20, 0.094), (21, -0.096), (22, 0.035), (23, 0.02), (24, -0.077), (25, -0.074), (26, -0.09), (27, 0.056), (28, 0.02), (29, 0.145), (30, -0.061), (31, 0.056), (32, -0.053), (33, 0.046), (34, 0.183), (35, -0.02), (36, -0.035), (37, -0.021), (38, -0.075), (39, -0.072), (40, 0.204), (41, -0.023), (42, -0.023), (43, 0.074), (44, -0.05), (45, 0.072), (46, -0.047), (47, 0.064), (48, 0.076), (49, -0.14)]
simIndex simValue blogId blogTitle
same-blog 1 0.98602849 332 high scalability-2008-05-28-Job queue and search engine
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
2 0.92933118 746 high scalability-2009-11-26-Kngine Snippet Search New Indexing Technology
Introduction: While Kngine just announce some improvement and new features , I would like you take you in small trip in Snippet Search research project at Kngine. What is Kngine? Kngine is startup company working in Searching technologies, We in Kngine aims to organize the human beings Systematic Knowledge and Experiences and make it accessible to everyone. We aim to collect and organize all objective data, and make it possible and easy to access. Our goal is to build Web 3.0 Web Search Engine on the advances of Web Search Engine, Semantic Web, Data Representation technologies a new form of Web Search Engine that will unleash a revolution of new possibilities. Introduction to Snippet Search Today, The Web Search Engine’s is the Web getaway, especially to get specific information. But unfortunately the search engines didn’t changed mush as the Web changed from 90’s. Since the 90’s the Web search engine still provide the same kind of results: Links to documents. We i
3 0.85721332 342 high scalability-2008-06-08-Search fast in million rows
Introduction: I have a table .This table has many columns but search performed based on 1 columns ,this table can have more than million rows. The data in these columns is something like funny,new york,hollywood User can search with parameters as funny hollywood .I need to take this 2 words and then search on column whether that column contain this words and how many times .It is not possible to index here .If the results return say 1200 results then without comparing each and every column i can't determine no of results.I need to compare for each and every column.This query is very frequent .How can i approach for this problem.What type of architecture,tools is helpful. I just know that this can be accomplished with distributed system but how can i make this system. I also see in this website that LinkedIn uses Lucene for search .Is Lucene is helpful in my case.My table has also lots of insertion ,however updation in not very frequent.
4 0.83606058 630 high scalability-2009-06-14-kngine 'Knowledge Engine' milestone 2
Introduction: Kngine is Knowledge Web search engine designed to provide meaningful search results, such as: semantic information about the keywords/concepts, answer the user’s questions, discover the relations between the keywords/concepts, and link the different kind of data together, such as: Movies, Subtitles, Photos, Price at sale store, User reviews, and Influenced story Goals Kngine long-term goal is to make all human beings systematic knowledge and experience accessible to everyone. I aim to collect and organize all objective data, and make it possible and easy to access. Our goal is to build on the advances of Web search engine, semantic web, data representation technologies a new form of Web search engine that will unleash a revolution of new possibilities. Kngine tries to combine the power of Web search engines with the power of Semantic search and the data representation to provide meaningful search results compromising user needs. Status Kngine starts as a research project in O
5 0.77991176 246 high scalability-2008-02-12-Search the tags across all post
Introduction: Let suppose i have table which stored tags .Now user can enter keywords and i have to search through all the records in table and find post which contain tags entered by user .user can enter more than 1 keywords. What strategy ,technique i use to search fast .There maybe more than millions records and many users are firing same query. Thanks
6 0.77268535 258 high scalability-2008-02-24-Yandex Architecture
7 0.72226125 1601 high scalability-2014-02-25-Peter Norvig's 9 Master Steps to Improving a Program
8 0.71793401 899 high scalability-2010-09-09-How did Google Instant become Faster with 5-7X More Results Pages?
9 0.7174499 810 high scalability-2010-04-14-Parallel Information Retrieval and Other Search Engine Goodness
10 0.69223726 269 high scalability-2008-03-08-Audiogalaxy.com Architecture
11 0.69011945 1395 high scalability-2013-01-28-DuckDuckGo Architecture - 1 Million Deep Searches a Day and Growing
12 0.68202722 1295 high scalability-2012-08-02-Ask DuckDuckGo: Is there Anything you Want to Know About DDG?
13 0.67957896 775 high scalability-2010-02-10-ElasticSearch - Open Source, Distributed, RESTful Search Engine
14 0.66655481 64 high scalability-2007-08-10-How do we make a large real-time search engine?
15 0.59585667 1253 high scalability-2012-05-28-The Anatomy of Search Technology: Crawling using Combinators
17 0.55474019 1233 high scalability-2012-04-25-The Anatomy of Search Technology: blekko’s NoSQL database
18 0.54206645 335 high scalability-2008-05-30-Is "Scaling Engineer" a new job title?
19 0.54083383 689 high scalability-2009-08-28-Strategy: Solve Only 80 Percent of the Problem
20 0.5066064 900 high scalability-2010-09-11-Google's Colossus Makes Search Real-time by Dumping MapReduce
topicId topicWeight
[(1, 0.032), (2, 0.237), (51, 0.263), (61, 0.299)]
simIndex simValue blogId blogTitle
same-blog 1 0.90142447 332 high scalability-2008-05-28-Job queue and search engine
Introduction: Hi, I want to implement a search engine with lucene. To be scalable, I would like to execute search jobs asynchronously (with a job queuing system). But i don't know if it is a good design... Why ? Search results can be large ! (eg: 100+ pages with 25 documents per page) With asynchronous sytem, I need to store results for each search job. I can set a short expiration time (~5 min) for each search result, but it's still large. What do you think about it ? Which design would you use for that ? Thanks Mat
2 0.81388104 322 high scalability-2008-05-19-Conference: Infoscale 2008 in Italy (June 4-6)
Introduction: The Third International Conference on Scalable Information Systems will focus on a wide array of scalability issues and investigate new approaches to tackle problems arising from the ever-growing size and complexity of information of all kinds. Looking at their technical program a lot of interesting topics will be covered. I see sensor networks, a subject I'm really interested in, has a number of sessions. That's unusual. And it's in Italy!
3 0.8085289 208 high scalability-2008-01-11-FTP Sanity: Redundancy, archiving, consolidation.
Introduction: Easy FTP redundancy and consolidation with the Open Source project Generic-FTP. Works with probably any Linux FTP Server (ProFTPD only one tested). Get rid of some single points of failure. A very easy to set up solution using scripts written in PHP. Tested thoroughly in a production environment.
4 0.80662298 818 high scalability-2010-04-30-Behind the scenes of an online marketplace
Introduction: In a presentation originally held at the 4. O2 Hosting Event in Hamburg, I spoke about the technology at a large online marketplace in Germany called Hitmeister . Â Some of the topics discussed include: what makes up a marketplace? technically system principles development patterns tools philosophy data model hardware I am looking forward to comments and suggestions for both the presentation and our work.
5 0.8032192 626 high scalability-2009-06-10-Paper: Graph Databases and the Future of Large-Scale Knowledge Management
Introduction: Relational databases, document databases, and distributed hash tables get most of the hype these days, but there's another option: graph databases. Back to the future it seems. Here's a really interesting paper by Marko A. Rodriguez introducing the graph model and it's extension to representing the world wide web of data. Modern day open source and commercial graph databases can store on the order of 1 billion relationships with some databases reaching the 10 billion mark. These developments are making the graph database practical for applications that require large-scale knowledge structures. Moreover, with the Web of Data standards set forth by the Linked Data community, it is possible to interlink graph databases across the web into a giant global knowledge structure. This talk will discuss graph databases, their underlying data model, their querying mechanisms, and the benefits of the graph data structure for modeling and analysis.
6 0.79605591 1287 high scalability-2012-07-20-Stuff The Internet Says On Scalability For July 20, 2012
7 0.79389471 173 high scalability-2007-12-05-Easier Production Releases
8 0.79304218 930 high scalability-2010-10-28-NoSQL Took Away the Relational Model and Gave Nothing Back
9 0.78971815 99 high scalability-2007-09-23-HA for switches
10 0.78627747 675 high scalability-2009-08-08-1dbase vs. many and cloud hosting vs. dedicated server(s)?
11 0.77753645 347 high scalability-2008-07-07-Five Ways to Stop Framework Fixation from Crashing Your Scaling Strategy
12 0.77325463 1303 high scalability-2012-08-13-Ask HighScalability: Facing scaling issues with news feeds on Redis. Any advice?
13 0.77071261 793 high scalability-2010-03-10-Saying Yes to NoSQL; Going Steady with Cassandra at Digg
14 0.76982087 493 high scalability-2009-01-16-Just-In-Time Scalability: Agile Methods to Support Massive Growth (IMVU case study)
15 0.76827264 501 high scalability-2009-01-25-Where do I start?
16 0.76286125 1411 high scalability-2013-02-22-Stuff The Internet Says On Scalability For February 22, 2013
17 0.75987506 510 high scalability-2009-02-09-Paper: Consensus Protocols: Two-Phase Commit
18 0.75625294 150 high scalability-2007-11-12-Slashdot Architecture - How the Old Man of the Internet Learned to Scale
19 0.75611353 1201 high scalability-2012-02-29-Strategy: Put Mobile Video Into Cold Storage After 30 Days
20 0.75591362 549 high scalability-2009-03-26-Performance - When do I start worrying?