libevent主要有三个结构

struct event : event的核心结构，即用户的callback函数绑定到event结构上，当发生一定I/O条件或singal或超时，callback被调用。

struct event_base:event被绑定到event_base,event_base是底层I/O复用函数的包装，它负责管理各种注册的event结构和调度。

struct event_config: 为了避免event_base的构造器太复杂，提供了event_config结构对配置参数进行了包装，设置features和flags。

struct event_base相当于底层I/O复用函数的实例，每个底层I/O复用函数都定义了add,del,dispatch,dealloc等函数，提供了统一的接口供上层调用。event_base中有一个指针指向了某个底层I/O复用函数封装的结构struct eventop.

每次调用event_base的注册事件就相当于调用了某个底层I/O复用函数定义的接口函数。

event_base还保存了若干个event chains，比如active events链，所有events链和pending events链.还有一个小根的时间堆，负责对timeout callback的处理，每次从时间堆中提取最小timeout，放到I/O复用函数中。signal的集成实现在event_base比较有特色，采用了一个socket pair方式，每当收到注册信号时，将对应的信号注册到singal链中，不对相应的信号执行signal()函数，而是统一的触发函数，当信号被触发时，处理函数会将信号写到socket的写端，event_base的I/O复用函数就能收到socket的可读请求，将信号事件的处理添加奥actice链种。

这样，event的读写，timeout，singal就被event_base集成到一起。

下面深入探讨下event的管理，event结构定义如下:

TAILQ_ENTRY(event) ev_active_next;

union {

TAILQ_ENTRY(event) ev_next_with_common_timeout;

int min_heap_idx;

} ev_timeout_pos;

evutil_socket_t ev_fd;

struct event_base *ev_base;

union {

/* used for io events */

struct {

LIST_ENTRY (event) ev_io_next;

struct timeval ev_timeout;

} ev_io;

/* used by signal events */

struct {

LIST_ENTRY (event) ev_signal_next;

short ev_ncalls;

/* Allows deletes in callback */

short *ev_pncalls;

} ev_signal;

} ev_;

short ev_events;

short ev_res;          /* result passed to event callback */

short ev_flags;

ev_uint8_t ev_pri;     /* smaller numbers are higher priority */

ev_uint8_t ev_closure;

struct timeval ev_timeout;

/* allows us to adopt for different types of events */

void (*ev_callback)(evutil_socket_t, short, void *arg);

void *ev_arg;

TAILQ_ENTRY 是一个宏，定义了一个结构，现在只需知道ev_active_next是该event结构在event_base中active链中的索引位置。ev_io和ev_sigal是对立的两种事件类型，采用union结构联合。ev_events就是事件的类型，还有callback函数，参数等，从名字和注释就可以很好理解。

其他部件：

struct eventbuffer: libevent提供I/O读写的抽象，不用直接read或者write到fd，而是用bufferread()和buffer write()进行读写

priority: libevent提供了事件优先级的管理，event_base_priority_init()初始化了event_base的优先级范围，默认的event会采用优先级范围的中值。

现在libevent的dispatch过程就很清晰了，首先初始化event_base结构，event_new()初始化event,event_add()使event pending。event_base_dispatch()开始了事件的循环，event_base首先从激活的事件中收集fd，在小根时间堆中找到最小timeout，然后对每一个信号事件进行注册，当发生可读或可写后，event_base检查socket是否可读，然后收集被触发的信号事件，收集fd可读可写事件和timeout。添加到active链中。然后根据优先级callback。event_base_loop()还提供了一系列的参数进行配置，如是执行高优先事件还是检查是否有新的active事件。或者执行callback后的行为。

From django/core/management/commands/loaddata.py    Line 95

for fixture_label in fixture_labels:

  parts = fixture_label.split('.')

fixture_labels default is [initial_data, ] , here we define fixtures = ['person', 'address'], so fixture_labels is ['initial_data', 'person', 'address']

From django/core/management/commands/loaddata.py    Line 98

if len(parts) > 1 and parts[-1] in compression_types:

    compression_formats = [parts[-1]]

    parts = parts[:-1]

else:

    compression_formats = compression_types.keys()

     

if len(parts) == 1:

    fixture_name = parts[0]

    formats = serializers.get_public_serializer_formats()

else:

    fixture_name, format = '.'.join(parts[:-1]), parts[-1]

if format in serializers.get_public_serializer_formats():

    formats = [format]

else:

    formats = []

If we list in the SampleTestCase.fixtures didn’t have a compression formats. By default, compression_formats is ['gz', 'zip', 'bz2'].

If parts didn’t have formats such as ‘json’, ‘xml’, formats will be ['xml', 'json'], in the later, program will be to guess the possible format.

Here, we only have filename is person, address, so program will guess the formats.

From django/core/management/commands/loaddata.py    Line 126

if os.path.isabs(fixture_name):

    fixture_dirs = [fixture_name]

else:

    fixture_dirs = app_fixtures + list(settings.FIXTURE_DIRS) + ['']

If file in the fixtures is a absolute path, program will search only in the path. Otherwise, program will search all the apps_fixtures paths.

From django/core/management/commands/loaddata.py    Line 136

for combo in product([using, None], formats, compression_formats):

    database, format, compression_format = combo

    file_name = '.'.join(

        p for p in [

            fixture_name, database, format, compression_format

        ]

        if p

    )

combo will be the Cartesian product constructed by filename, formats, compression_formats.

From django/core/management/commands/loaddata.py    Line 148

full_path = os.path.join(fixture_dir, file_name)

open_method = compression_types[compression_format]

try:

    fixture = open_method(full_path, 'r')

full_path is the possible fixture_dir + file_name, program will tend to open it, if failed, program tries another.

Above all, we can get that we had better to decide a absolute path and absolute filename in the fixtures to avoid meaningless guess.

• Tokenization

• Feature Extraction

• Classification using different classifiers

• Naive Bayes

• MaxEnt

• SVM

Sentiment Tokenize Issues

• Deal with HTML and XML mark

• Twitter mark-up (names, hash tags)

• Capitalization (preserve for words in all caps)

• Phone numbers, dates

• Emoticons

Extracting Feature for Sentiment Classifier

• How to handle negation

• Which words to use(all words or adjective words)

Boolean Multinomial Naive Bayes

Hard problem in sentiment

• Subtlety sentiment

• Thwarted Expectations and Ordering Effects

Build lexicons

Hatzivasiloglou and McKeown intuition for identifying word polarity

• Adjectives conjoined by ‘and’ have same polarity

• Adjectives conjoined by ‘but’ do not

• Label seed set of 1336 adjectives

• Expand seed set to conjoined adjectives by Google

• Supervised classifier assigns ‘polarity similarity’  to each word pair, resulting in graph;

• Clustering for partitioning the graph into two

Turney Algorithm

1. Extract a phrasal lexicon from reviews

2. Learn polarity of each phrase

3. Rate a review by the average polarity of its phrasal

Using WordNet to learn polarity

• WordNet: online thesaurus

• Create positive and the negative seed-words

• Find Synonyms and Antonyms

• Positive Set: Add synonyms of positive words and antonyms of negative words

• Negative Set: Add synonyms of negative words and antonyms of positive words.

• Repeat, following chains of synonyms

• Filter

Finding sentiment of a sentence

• Important for finding aspects or attributes or target of sentiment

• Frequent phrases + rules

• Filter by rules like “occurs right after sentiment word”

A example to sentiment analysis

Using Naive Bayes to train.

import sys

Data

I use the writing of secondary-school children. There is a summary of contents:

• A corpus to train language model

• A corpus of spelling errors for development

• A table listing counts of edits ‘x|w’, taken from Wikipedia.

Language Models

• Laplace Unigram Language Model: a unigram model with add-one smoothing. Treat out-of-vocabulary items as a word which zero times in traing.

• Laplace Bigram Language Model: a bigram model with add-one smoothing.

• Stupid Backoff Language Model: use a unsmoothed bigram model combined with backoff to and add-one smoothed unigram model.

• Add-k Trigram Language Model: a trigram model with add-k smoothing.

Every language models conform the same interface, accpte corpus to train and get test sentece to spell correct. finally, return scores for sentences represent the likihood of correction.

Code

import collections, math

return score

import collections, math

return score

import collections, math

import collections, math

Evaluation

• Laplace Unigram Language Model: 0.12

• Laplace Bigram Language Model: 0.13

• Stupid Backoff Language Model: 0.12

• Add-k Trigram Language Model: 0.16

The result is the accuracy of spell correction. Total wrong words are 471, and most of model can spell correct up to 60.

Laplace Unigram model and Laplace Bigram Model can reach my goal, but Stupid Backoff model don’t get the expect result. And Trigram Model too.

Shortcoming

1. Edit model is too simple and limit the spell correct.

2. Because of the too small traing corpus, stupid backoff language model can’t get content result. Stupid backoff model is designed to web-scale corpus.

Referencing:

• 52nlp.com“>我爱自然语言处理

• mit.org“>mit NLP course

• We see an observation of a misspelled word x
• Find the correct word w

w = argmax P(x|w)P(w)

From this equation, we can learn that the correct word w is the probability of w misspelled to x, and the probability of word w.

Candidate generation

1.  words with similar spelling

2.  words with similar pronunciation

We get candidate list from compute edit distance of string.

Compute minimal edit distance between two strings, where edits           are:

• Insertion
• Deletion
• Substitution
• Transposition of two adjacent letters

Language Model

• Unigram, bigram, trigram
• Web-scale spelling correction
• Stupid back-off

For example, we can find P(w) for unigram model:

Channel Model

##Edit probability

P(x|w) = probability of the edit (deletion/insertion/substitution/transposition)

##Error probability

We should compute error probability

How often are above mistake happen.

This table is showing the confusion matrix for spelling errors.

Generating confusion lists by P(x|w)

Channel model for ‘access’

Noisy channel probability for acress

=======================================

This table primarily show that the process of compute the noisy channel probability for acress.

1. List the candidate words by insertion, deletion, substitution, transposition
2. Get the P(word) from language model such as unigram model.
3. From a confusion matrix(the probability of type ‘a’ as ‘b’, insert ‘a’ before ‘b’ etc.), we get P(x|w) for candidate words.
4. w = P(x|w)*P(w)

For more accuracy, we may replace unigram of bigram model.

So the above process may change:

P(word) => P(word|previous word)

What’s more, trigram model have more accuracy for the result.

Two corpora are necessary:

1. The probability of the letters in word operation as insertion, deletion, substitute, transposition.
2. The corpora for the probability of special language model, in detail, the probability of the candidate word show after the previous word or the two previous words.

For the real-word spell correction

We don’t know the wrong word which is in a sentence, we must generate a set of candidates for each word.

• For each word in sentence
• Generate candidate set
• the word itself
• all single-letter edits that are English words
• words that homophones
• Choose best candidates
• Noisy channel model
• Task-specific classifier

Consider the probability of no error

The origin word is correct is taking up most probability

Improvements to channel model

• Allow richer edits
• ent -> ant
• ph -> f
• le -> al
• Incorporate pronunciation into channel
• Factors that could influence p(misspelling|word)
• The source letter
• The target letter
• Surrounding letters
• The position in word
• Nearby keys on the keyboard
• Homology on the keyboard
• Pronunciations
• Likely morpheme transformations

At first, I only want to a rich text editor in django-admin pages. I find tinymce is a good solution to. Then I followed the steps by djangoproject. But failed to see the page display richtext editor.

I searched and failed.

Django-tinymce occupied me, I deploy it. Unluckly, when I did all where document lists. No change the page shows.

I looked up the document again. The configuration said
TINYMCE_JS_URL (default: settings.MEDIA_URL + 'js/tiny_mce/ tiny_mce.js')
The URL of the TinyMCE javascript file.
TINYMCE_JS_ROOT (default: settings.MEDIA_ROOT + 'js/tiny_mce')
The filesystem location of the TinyMCE files.
...

Naturally, ‘Default’ means no extra setting and program will work as default setting. But there is different.

What a suprise! When I did lazily and copied ‘default setting’ to settings.py. It changed, but stranged.

There are some malposition. I looked settings again. And find ‘TINYMCE_JS_URL = ‘http://debug.example.org/tiny_mce/tiny_mce_src.js” is wrong. Originaly, I think it’s my fault. But when I see document site again. It’s examples what document site shows. It absolutly wrong.

Misfortune didn’t leave me, if you want to use popup window such as insert image, view html plugins. You must update the config ‘tiny_mce_popup.js’ and settings.py. But these aren’t show up in the tutorial.

What a terribel process!

Ld build a symbol table and relocate outer modules’s variables addresses. Because static library is combined with main program, so during linking period the executable file get absolute address.

Shared Library

Shared library has two link ways. One is decided when loaded into virtual memory, it’s called load time relocation. Another is via PIC(Positon-independent Code) technology decide address until access it.

1. First, wo will analyze the former way. The executable file when loaded into virtual memory gets the address from shared library. In other words, when module’s absulote addresses in memory is sured, the system relocates all abosulate addresses.
   But there are some issues, if shared library exists instruments are relevance to address, such as when a variable in shared library access other library variable, It may be different in different programs. So it must be different code section in different program’s shared library. It goes against the purpose of the shared library, and system must copy several code to different program using shared library.

2. PIC technoloy separates code from independent code and dependent-address code. When compiled module, module build .got section, and put .got table’s address corresponding name which is refer to outer module.
   So after compiled, dependent-address instruments in module can get address from .got table. When loaded into memory, .got section get absoute address is the same as the former way. As we know, even if shared library, every program must own a data section copy and .got section exists .data section, so different programs own different .got section, there will be non-conflict address in code.
   We can conclude that if module load self’s function or variables, it’s easy to understand that module can get abosulate address directly. If access the shared library’s function or variable, it will get address from .got section. The instrument find address in .got section, then indirectly get the value.

1. Finish learning objective-c, cocoa, mac osx develop skills;
2. Read <程序员的自我修养> three chapters;
3. Read <Unix Network Programming> two chapters;
4. Begin Weibo based on Oauth 2.0 client for osx;

But some goals don’t be achieved,

1. Read <Algorithm: C >

Sina Weibo Oauth 2.0 Framework

I look up the Open Sina, but failed find the Weibo SDK for Mac osx. Then I find the OACosumer library for Oauth 1.0. After a few hours work, the project convert Oauth 1.0 to Oauth 2.0 failed.

Then I decided to convert Weibo SDK for IOS to osx platform.
Next, I changed the code about IOS, and replace them by relevance code.

Now, I have fininshed most of basic for Weibo client.

Unix Network Programming indicator

Recently, I begin to learn about the socket API in C.

Question 1:

The requirement that collections not mutate while you are iterating over them can be inconvenient at times, especially if you are doing a “scan the collection and pull out stuff that shouldn’t be there” operation. What would it take to make a classic array enumerator support having the collection mutated during iteration? What impact would this have on fast enumeration?

While I want to iterate over the mutated collection, it raised exception and error that ***** Terminating app due to uncaught exception ‘NSGenericException’, reason: ‘***
Collection <__NSCFDictionary: 0x100a14fd0> was mutated while being enumerated.’ **

#import <Foundation/Foundation.h>

int main(int argc, const char *argv[])
{
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    NSMutableDictionary *myDictionary = [[ NSMutableDictionary alloc] init];
    [myDictionary setObject:@"Keith" forKey:@"name"];
    [myDictionary setObject:@"hacker" forKey:@"subject"];
    [myDictionary setObject:@"4/9/2002" forKey:@"date"];

    NSMutableDictionary *selectionIndex = [NSMutableDictionary dictionaryWithDictionary:myDictionary];
    NSString *whatever = @"999999";
    id keys;
    NSEnumerator *keyEnum = [selectionIndex keyEnumerator];

    while (keys = [keyEnum nextObject])
    {
        [selectionIndex setObject:whatever forKey:keys];
    }
    [pool drain];
    return 0;
}

So I change the way to mutate collection.

#import <Foundation/Foundation.h>

int main(int argc, const char *argv[])
{
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    NSMutableDictionary *myDictionary = [[ NSMutableDictionary alloc] init];
    [myDictionary setObject:@"Keith" forKey:@"name"];
    [myDictionary setObject:@"hacker" forKey:@"subject"];
    [myDictionary setObject:@"4/9/2002" forKey:@"date"];

    NSMutableDictionary *selectionIndex = [NSMutableDictionary dictionaryWithDictionary:myDictionary];
    NSString *whatever = @"999999";

    for (NSString *temp in selectionIndex.allKeys)
    {
        [selectionIndex setObject:whatever forKey:temp];
    }

    for (NSString *temp in selectionIndex.allValues)
    {
        NSLog(@"%@", temp);
    }
    [pool drain];
    return 0;
}

Above all, we can find that we can’t make any changes to a mutated collection but we can enumerate the keys of the collection and change it.

Question 2:

How to add a property implementation method at runtime.

#import <Foundation/Foundation.h>
#import <objc/runtime.h>

@interface bird : NSObject
{
    int height;
    float n;
}

@property float n;
@property int height;

@end

float dynamicN(id self, SEL _cmd)
{
    NSString *methodName = NSStringFromSelector(_cmd);
    NSLog(@"%@,%@", methodName, [self description]);
    return ((bird *)self)->n;
}

void dynamicSetN(id self, SEL _cmd, float sname)
{
    printf("setName start;\n");
    ((bird *)self)->n = sname;
}

@implementation bird
@synthesize height = height;
@dynamic n;

- (id)init
{
    if (self = [super init]) {
        n = 1.0;
        height = 3;
    }
    return self;
}

+ (BOOL) resolveInstanceMethod:(SEL)aSEL
{
    if (aSEL == @selector(n)) {
        class_addMethod([self class], aSEL, (IMP) dynamicN, "f@:");
        return YES;
    }
    if (aSEL == @selector(setN:)) {
        class_addMethod([self class], aSEL, (IMP) dynamicSetN, "v@:f");
        return YES;
    }
    return [super resolveInstanceMethod:aSEL];
}
@end

int main(int argc, const char *argv[])
{
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    bird *aBird = [[bird alloc] init];
    aBird.n = 9;
    printf("\n%f\n,%d", aBird.n, aBird.height);
    [pool drain];
    return 0;
}

I construct a class called bird, and have two properties ‘height’ and ‘n’. I only synthesize height and marked n as dynamic. Use resolveInstanceMethod: to change the dynamic method look up. When others access the bird instance, messaging must through resolveInstanceMethod and look up method.

There is critical point that C-Style function can’t be dynamic type(id) to access a instance variable but via ‘((bird *)self)->n’ to.

class_addMethod:::: is a low API to change runtime, we can add a method to special class via it. Four parameters respectively are ‘classname’, ‘SEL’, ‘function address(IMP)’, encode.

Reference: Objective-C Runtime Reference

Question 3

NSFastEnumeration protocol is must be conformed if objects used in conjunction with the for language construct used in conjunction with Cocoa objects.

#import <Foundation/Foundation.h>

@interface FibonacciSequence : NSObject <NSFastEnumeration>
{
}
@end

@implementation FibonacciSequence

- (NSUInteger) countByEnumeratingWithState: (NSFastEnumerationState *) state
                                   objects: (id *) stackbuf
                                     count: (NSUInteger) len
{
    printf("%d\n", len);
    assert(len >= 2); // because we pre-populate two values on first-call
    id *scan, *stop;
    if (state->state == 0) {
        // first call, do initializations
        state->state = 1;
        state->mutationsPtr = (unsigned long *)self; // not applicable
        state->itemsPtr = stackbuf; // seed with correct values
        state->extra[0] = 1;
        state->extra[1] = 1;
        // fill in the first two values
        state->itemsPtr[0] = [NSNumber numberWithInt: state->extra[0]];
        state->itemsPtr[1] = [NSNumber numberWithInt: state->extra[1]];
        // tweak the scanning pointers because we've already filled
        // in the first two slots.
        scan = &state->itemsPtr[2];
        stop = &state->itemsPtr[0] + len;
    }
    else {
        // Otherwise we're in the Pink, and do normal processing for
        // all of the itemPtrs.
        scan = &state->itemsPtr[0];
        stop = &state->itemsPtr[0] + len;
    }
    while (scan < stop) {
        // Do the Fibonacci algorithm.
        int value = state->extra[0] + state->extra[1];
        state->extra[0] = state->extra[1];
        state->extra[1] = value;
        // populate the fast enum item pointer
        *scan = [NSNumber numberWithUnsignedLong: value];
        // and then scoot over to the next value
        scan++;
    }
    // Always fill up their stack buffer.
    return len;
}

@end

int main(int argc, const char *argv[])
{
    NSAutoreleasePool *pool =  [[NSAutoreleasePool alloc] init];
    FibonacciSequence *fibby = [[FibonacciSequence alloc] init];
    int boredom = 0;
    for (NSNumber *number in fibby) {
        // NSLog (@"%@", number);
        if (boredom++ > 14) {
            break;
        }
    }
    [pool drain];
    return 0;
}

countByEnumeratingWithState:objects:count:

Return the size of |stackbuf|’s elements
parameters

state:
Context information which is used to hold critical variables when the function is invoked multiple times.

stackbuf:
A C array of objects passed in which the sender is to iterate.But it is not required, you also can store calculated values in other array. The advantage of storing in |stackbuf| is we can learn the max size of special instance storing in |stackbuf|.

len:
The maximum number of objects to return in |stackbuf|, |countByEnumeratingWithState:objects:count:| according to |len| to judge calculate how much values.

|NSFastEnumerationState|
typedef struct {
unsigned long state;
id *itemsPtr;
unsigned long *mutationsPtr;
unsigned long extra[5];
} NSFastEnumerationState;

This struct can be hold all the time. It means this struct can communicate with multiple times of function implement as temporary container.

Fileds

state:
Arbitrary state information used by the iterator. Typically this is set to 0 at the beginning of the iteration.
itemsPtr:

A C array of objects, you can point |itemsPtr| to |stackbuf| or another structures. For example
    for (NSNumber *number in fib) {
        // NSLog (@"%@", number);
        if (boredom++ > 14) {
            break;
        }
    }
During the for loop, |number| get value from |itemsPtr| one by one according to the |countByEnumeratingWithState:objects:count:|'s return value.

extra:
A C array that you can use to hold returned values.

终于完成了CS 61A Fall 2011 SICP课程，在十月份的时候开始，一直到今天，差不多与课程同步，然后完成了所有的项目和作业，看完了所有的textbook，收获颇丰。

地址

http://wla.berkeley.edu/~cs61a/fa11/61a-python/content/www/index.html

收获

课程是首次采用Python来代替以前的Scheme教学，用Python的好处是能更快的理解程序，可以进行实践。
从函数式编程到面向对象到MapReduce，从实现列表、对象系统到简单解释器的实现，从网络编程到多线程，幅度之大远非国内高校课程所比。