Design: 如何设计一个web crawler

对于design的总结，就从我最熟悉的crawl 开始。按照我在工作中的学习和了解实际应用一下。

确定design purpose.
crawler需要crawl什么，是固定的host, ip还是某一个domain.
crawler是为谁服务？
crawl 的scale是多少？几百，几千还是几个B？
crawl后的内容如何使用？
crawl的latency的宽容度是多少？
crawl的seed如何选择？
如何filter junk/spam/bad url?
如何保证url的freshness?
如何dedup crawl的次数？
在crawl之后的document需要进行process吗？

假设我们的crawler是一个private crawler, 只是target某一个特定的host, 比如wikipedia. 有一个seed url list, 然后从parent url开始往下衍生找outlinks.


Comments

Post a Comment

Popular posts from this blog

Hanoi Tower

Image
最近学习到了一个很有趣的problem. 汉诺塔。 我认真的想了好几天，终于在今天找IT给我set up machine等着没事干的时候大概想通了。哈哈哈。 解决这个问题的方法让我想到了binary tree traverse 里面的in order的遍历，汉诺塔的solution之一的 recursive 的方式跟二叉树in order就是异曲同工，而且in human mind, 汉诺塔should make more sense to understand compared to binary tree. 在网上看到了 一个印度小男孩对汉诺塔的分析 ，果然是tree. Basically 解决汉诺塔的问题就是一个搬家公司应该怎么搬家的问题，from mission completed to start, which is from bottom to top. Let's define Hanoi move function as: HanoiMove(int leftToSit, int from - 0 , int to - 1, int rest - 2) and we have n pieces in total waiting for the move In theory, to approach the last step nth of move, we just need finish move rest of Hanoi pieces (n-1 in total) from start to rest pile,(简单说来就是，清空路障，直捣虎穴。）and then it should be able to move the last piece from start to To pile. so we just do 1) HanoiMove(n-1,  from, res, to)    ---->>>    left as we planned in theory, we move the nth piece move(from,  to)    after this move, we have...
Read more

Kth smallest in Sorted Array

/*** Given two sorted arrays A, B of size m and n respectively. Find the k-th smallest element in the union of A and B. You can assume that there are no duplicate elements. Solution: 1) linear O(K) two index 2) improved BS O(log(min(m,n,k))) http://leetcode.com/2011/01/find-k-th-smallest-element-in-union-of.html ***/ #include <iostream> #include <algorithm> using namespace std; // time complexity: O(k), space: O(1) int kth_smallest_linear( int * a , int * b , int asz , int bsz , int k ) {        int a_index = 0, b_index = 0;        int num;        //edge case - constant time        if ( asz == 0) return b [ k -1];        if ( bsz == 0) return a [ k -1];        if ( k == 0 && asz + bsz < k ) return INT_MIN ;   ...
Read more

Constant Time Stack

/*** How would you design a stack which, in addition to push and pop, also has a function min which returns the minimum element?  Push, pop and min should all operate in O(1) time. ***/ #include <iostream> #include <climits> using namespace std; struct node { int val; node *next; }; class myStack { public: myStack(); ~myStack(); void push(int); int pop(); int getmin(); private: int min; node *stack; node *minStack; node *top; node *mintop; void update_top(node*); void delete_node(node*); }; myStack::myStack() { top = nullptr; mintop = nullptr; min = INT_MAX; } myStack::~myStack() { while (stack != nullptr) { delete_node(stack); } } void myStack::push(int val) { if (val <= min) { min = val; // push in stack stack = new node(); stack->next = top; stack->val = val; top = stack; minStack = new node(); minStack->next = mintop; minStack->val = val; mintop = minStack; } else { // only push val in stack stack = ...
Read more