Design: 如何设计一个web crawler

对于design的总结，就从我最熟悉的crawl 开始。按照我在工作中的学习和了解实际应用一下。

确定design purpose.
crawler需要crawl什么，是固定的host, ip还是某一个domain.
crawler是为谁服务？
crawl 的scale是多少？几百，几千还是几个B？
crawl后的内容如何使用？
crawl的latency的宽容度是多少？
crawl的seed如何选择？
如何filter junk/spam/bad url?
如何保证url的freshness?
如何dedup crawl的次数？
在crawl之后的document需要进行process吗？

假设我们的crawler是一个private crawler, 只是target某一个特定的host, 比如wikipedia. 有一个seed url list, 然后从parent url开始往下衍生找outlinks.


Comments

Post a Comment

Popular posts from this blog

Unique Binary Search Trees

/*** Given  n , how many structurally unique  BST's  (binary search trees) that store values 1... n ? For example, Given  n  = 3, there are a total of 5 unique BST's.    1         3     3      2      1     \       /     /      / \      \      3     2     1      1   3      2     /     /       \                 \    2     1         2         ...
Read more

Longest prefix matching - trie

/*** Longest prefix matching Given a dictionary of words and an input string, find the longest prefix of the string which is also a word in dictionary. Example: Let the dictionary contains the following words: {are, area, base, cat, cater, children, basement} Below are some input/output examples: -------------------------------------- Input String            Output -------------------------------------- caterer                 cater basemexy                base child                   < Empty > ***/ #include <iostream> #include <string> using namespace std; struct node {        char val; ...
Read more