Here is Kite’s farewell for you to read.

Kite did open-source many parts of their technology/software stack, though I didn’t check how comprehensive those parts are, and if that is anywhere near enough to fork/continue their work.
I wonder if there already exists an open-source project focusing on ML-based code completion for e.g. Python – let me know in the comments if you know one!

Kite cites two reasons for a shutdown: 1) technology not being quite there yet, and 2) failure to monetize.
Kite had up to 500k daily developers using the platform, but apparently extremely few were willing to pay for it.
If you do look at current ML code assistants, there seems to always exist at least some free tier – I wonder if that is forced by the same lackluster, non-paying developers attitude as for Kite.

Kite’s farewell had another interesting number: 18%.
That is by how much individual developer’s productivity could increase thanks to Kite’s assistance.
This isn’t bad at all; for a team of 5 largely independent developers, it’s almost one extra “affordable” developer.
Kite was striving to achieve a “10x improvement”, but at least to me the 18% improvement sounds good enough for sales.

I’m very curious to try some of these assistants out.
I can imagine them to be very helpful for relatively experienced developers when starting to work with a new library/ecosystem – for example, OpenVision Python bindings.
Even the common autocomplete can significantly simplify “onboarding” to a new library – and a more intelligent autocomplete should be able to help with boilerplate code (that you usually don’t have when you begin), as well as with some idiomatic expressions and statements.

Have you already played with some of the smarter code assistants?
What was your experience?
Please share

Just came across the Akaike’s Infor­ma­tion Criterion (AIC) and Schwarz Bayesian Infor­ma­tion Criterion (BIC). Citing robjhyndman,

Asymp­tot­i­cally, min­i­miz­ing the AIC is equiv­a­lent to min­i­miz­ing the CV value. This is true for any model (Stone 1977), not just lin­ear mod­els. It is this prop­erty that makes the AIC so use­ful in model selec­tion when the pur­pose is prediction.
…
Because of the heav­ier penalty, the model cho­sen by BIC is either the same as that cho­sen by AIC, or one with fewer terms. Asymp­tot­i­cally, for lin­ear mod­els min­i­miz­ing BIC is equiv­a­lent to leave–v–out cross-​​validation when v = n[1-1/(log(n)-1)] (Shao 1997).

Want to try AIC and maybe BIC on my models. Conveniently, both functions exist in R.

I do find the promise to go open-source somewhat naive, but who knows…

It is a small command-line tool to easily convert your image-only DJVU files into image+text DJVU files. In Debian testing, there are language packages for (in no specific order) German, English, French, Spanish, Vietnamese, Brasilian Portuguese, Dutch, and Italian. The original tesseract-ocr software includes training data & code, so it should be (at least in theory) easy to add more recognition languages.

i-SOBOT is a tiny 165-mm robot with 17 servo-engines. i-SOBOT can be programmed and controlled at distance (using the LCD-equipped remote control), and understands up to 10 voice commands. This robot will be available in Japan and US in October 2007.

According to the Guinness Book Of Records, i-SOBOT is the smallest robot in the world.

Here’s the official site.

The bot can play music, dance, and respond to some user actions (like applause). There are two versions: the basic one, and the CAMVersion, which differs by it’s built-in camera and WiFi connectivity, to beam you the shots robot makes.

• Open Content Free Web Books (on practical AI programming)
• Artificial Intelligence Lab (with some practical applications)

Looks like it isn’t widely acknowledged, that AI is, in fact, quite widely used. Though primarily in OCR, TTS, STT :), and NLP (including machine translation).

Consciousness is not necessarily the derivative of complexity; it can be rather the derivative of the world’s model and the subject’s placement in that model, which causes consciousness to arise. (In other words, consciousness equals to the ability of the subject to place himself within the constantly self-re-approving environment model.) Thus, the requirement for ANNs use is not convincing: one can ensure that the appropriate world model is created without ANNs. I would even say that ANNs are just a kind of a “black box”, by using which we try to avoid the really obvious complexity, which can nevertheless be solved purely algorithmically, with no extra overhead from ANN-like simulators and wrappers.

There appears to be a specific double-dichotomy in ANN versus Algorithmic approaches for AI development: ANN considers the brain to be a collection of individual neurons (or “perceptrons”? for this case), while algorithmic approach considers the brain to be a collection of “modules”, each performing some quite narrow function. At the same time, we are told that algorithmic approaches cannot foresee unforeseen circumstances, thus ANNs are better for AI development (that’s the second dichotomy). However, modern “intelligent” software (here I mean first of all cognitive-functions software) rather successfully uses “learning algorithms”, “pattern matching algorithms”, “inference algorithms”, “prediction algorithms” and many more other “algorithms”. At the same time, I’m unaware of the successful (or at least impressive) software tool built using ANNs.

(Well, unwinding the above paragraph may lead to a controversy: ANNs’ implementations are algorithms themselves. However, I would make a clear distinction here: I consider ANN to be a rather generic simulator of inter-neuronal interactions and signal-response circuits; the same type of ANN could be applied to several different tasks (well, different instances of the same-type ANN). But if a generic ANN is trained for a specific task, and then optimized for that task only, and then probably also simplified and extended with fixed-value tables to avoid recalculating static relations – this is not ANN, but an algorithm, as being task-specific it falls under the definition of the algorithm much better than under the definition of the ANN.)

I was given a reference to Daniel Dennett by Bernardo Kastrup, the author of the “postulates”. Daniel Dennett is, like me, a proponent of algorithmic approaches to AI. However, I didn’t read any of his works yet. As soon as I do, I’ll add more to the ANN vs Algorithms topic.

In simpler words, nonanticipatory systems can “take into account” only past and present, and cannot base their behaviour/decisions on future expectations.

Nonanticipatory systems are also known as causal systems.

I found the text of this article available in PostScript format, and assume it’s OK to duplicate it here in PDF format: A Universal Predictor Based on Pattern Matching. Please contact me if there is a need to remove this link.

For part 2, I wanted to start (and probably also end) with Cybula’s AURA (universal pattern matcher, white-paper dated 2004). AURA is said to be built around Correlation Matrix Memory (CMM). CMMs were developed (or picked up for development?) by Prof. Austin, the founder of Cybula, in 1986.

The white paper tells us that

The now ubiquitous neural network methods such as Kohonen Networks, Radial Basis Function networks and Kohnen networks all allow users develop good pattern matching systems for small problems, where they excel. However, when the problems grow to large datasets, and where very high performance is needed, they become limited. … The well known k-Nearest Neighbour methods (k-NN) is a relatively good pattern matching method that has been constantly shown to operate well on many problems, however, it suffers from slow operation on large data problems.

The document claims that “The AURA technology has its origins in neural networks but draws upon pattern recognition methods and parallel processing for its fundamental operation“. I’m not a pro in pattern matching and NNs, but I’m aware of the both approaches and how they basically operate; in the light of my knowledge, it appears hard to complement NN with some external pattern recognition methods – just because NN is a pattern recognizing method itself, so it would be stacking two similar methods (which won’t improve performance).

I was impressed by the list of data types AURA can handle:

AURA can be applied to almost any data type. Currently, the technology has the following application components:

• Signal Data (time varying data)
• Text strings (strings of symbolic data)
• Document sets
• Form Data
• Graphs (applicable to images and multidimensional data)

I’ll continue citing the original white paper, as it appears to me a valuable resource, and I’m interested in the techniques used in AURA.

The core of AURA is a storage and retrieval engine based on a Correlation Matrix Memory. This system allows large amounts of data to be saved and retrieved quickly and efficently. Unlike a database, AURA is designed first and foremost to deal with large incomplete data. (…) The power of Cybulas approach is to combine the CMM with methods that prepare the data correctly to get the best out of the network and to use the CMM as a part of a more sophisticated data access system.

Next, the ability of AURA to process huge loads of data is said to be due to the getting-more-specific scheme, where initial pattern matching is done by approximate methods, following with more accurate/specific, and completing recognition with the most accurate methods. The process is as following:

1. Data to be matched
2. Pre-processor
3. CMM
4. Back check
5. Candidate Matches
6. Final Matches

Multiple data types are supported in AURA thanks to different-data-types preprocessors, which convert input data into some single data type, which is actually used for further pattern matching. (At most, there are two such data types – to accommodate for graphs with arcs joining nodes. Or it could be the only type, with lists represented as the simplest uni-directional graph.)

The main emphasis of text searching is that it allows the user to match parts of words, i.e. the data items used by the system are composed of individual letters, rather than whole words as found in the document components.(…) In contrast to the text matching components, the document components use words as the atomic elements of the search, rather than the individual letters.

AURA is also a versatile pattern classifier, allowing the identification of an unknown item of data. (…) AURA differs from other classification methods in that it allows data to be added at any time to the classifier. No rebuilding of the classifier is required. This allows its use in on line applications where new data is constantly arriving.

Hmm, is the description too good or AURA is really a kind of “Easy to use, WYSIWYG Universal Pattern Matcher, database and classifier”? All in all, AURA does look an attractive pattern matching solution.

The technologies mentioned in AURA description need overview as well, so there might be also “part 3″ in the Pattern matching and prediction series. Hopefully, I’ll have time and desire to document my findings.

Please comment if you happened to use any “universal” pattern-matching tools in your activities (except for usual regexps, of course). This would be a valuable information for me.

I decided to find out more about pattern matchers of nowadays… definitely not focusing too much on regular expressions, which are of no interest to me in the light of possible applications.

First of all, the links directory: Pattern Matching Pointers is a long list of links, related to combinatorial pattern matching:

Combinatorial Pattern Matching addresses issues of searching and matching strings and more complicated patterns such as trees, regular expressions, graphs, point sets, and arrays. The goal is to derive non-trivial combinatorial properties for such structures and then to exploit these properties in order to achieve improved performance for the corresponding computational problem.

A steady flow of high-quality research on this subject has changed a sparse set of isolated results into a full-fledged area of algorithmics with important applications. This area is expected to grow even further due to the increasing demand for speed and efficiency that comes from molecular biology, but also from areas such as information retrieval, pattern recognition, compiling, data compression, program analysis and security.

(Further I may link to some of the resources already listed in Pattern Matching Pointers.)

The directory mentioned focuses in part on the computational/molecular biology uses of pattern matching. The following sections are represented in the Pointers:

• people who work in the area (last updated in Dec 2006, and no longer maintained)
• a list of presentations and papers from the CPM conferences (1992-2006)
• a list of thematic conferences, at the moment of writing – for the year 2007:
  • Jan 3-7: PSB’07 (Wailea, Maui)
  • Jan 7-9: SODA’07 (New Orleans, Lousiana)
  • Jan 15-17: APCB’07 (Hong Kong)
  • Mar 27-29: DCC’07 (Snowbird, UT)
  • Apr 1-5: CIBCB’07 (Honolulu, Hawaii)
  • Apr 21-25: RECOMB’07 (San Francisco, CA)
  • Apr 26-28: SDM’07 (Minneapolis, Minnesota)
  • Jul 9-11: CPM’07 (London, Western Ontario)
  • Jul 16-19: COCOON’07 (Banff, Alberta)
  • Aug 13-17: CBS’07 (San Diego, CA)
  • Jul 21-25: ISMB/ECCB’7 (Vienna, Austria)
  • Aug 12-15: KDD’07 (San Jose, CA)
• a huge list of resources on the subject:
  • bibliographies
  • link collections (doh!)
  • interest groups
  • books
  • journals
  • proceedings
  • software
  • news groups and mailing lists

After a brief review, the list appears to be primarily related to regex-like matching, which was a disappointment for me personally. Still, the list is long, and you may find more than I did.

Next thing I checked was the book Bioinformatics: The Machine Learning Approach (MITPress). Chapters 2 through 4 are devoted to machine learning and prediction, and a number of chapters are devoted to HMM and neural networks and even combined HMM-NN approaches. Unfortunately, I didn’t read the book yet (btw, it can be found on the internet in electronic downloadable form); but the contents do look very promising, both for the bioinformaticians and those who just want to learn more about some modern algorithms and approaches.

Next one at hand was “A universal predictor based on pattern matching for mixing sources” (by P. Jacquet , W. Spankowski, I. Apostol). This one is a generic algorithm for predicting the x(n+1) value in the series of x1, x2, …, x(n). I found a PostScript version on the internet, so I assume it’s legal to give you the pdf version of this text. (If you know some restrictions exist, do let me know to remove the PDF version.) The proofs in the article might be hard to follow, but the idea is understood and can be used to the benefit of your specific applications.

(I also came across “A Universal Online Caching Algorithm Based on Pattern Matching“, which may be interesting for browser and web-cache developers )

There’s also an article “Universal lossless compression via multilevel pattern matching”, which is highly similar to the above-mentioned “A universal predictor…”. However, both can be used not only for compression.

Next article I briefly scanned was “A Simple Universal Pattern-Matching Automaton” by Daniel J. Bernstein – so far the best match to what I was looking for. (You may want to read about automata – not to widen the knowledge, but only to find out how this word is used.) I didn’t manage to grasp everything from the first reading, but from what I understood it appears that the automaton described is worth attempting to employ to solve the AI-related tasks in the field of “recognizing” something (or someone).

This is the end of Part 1. Continue to Pattern matching and prediction (part 2).

In this post I’ll try to figure out (primarily for myself) what is Artificial Intelligence.

Evidently, the “artificial” part requires no explanation, and the real problem is only with the “intelligence” part.

An extremely over-simplified, and actually incorrect definition would be “Intelligence is the ability to think logically”. Evidently, logic cannot be the sole basis of intelligence, at least because intelligence requires an ability to comprehend the environment, not only deduct. Moreover, logic itself is not an ultimate intelligence resource – it cannot explain the environment. Even planning an experiment – a generic method of studying the environment – requires not only logic, but also some kind of stimulus to learn the environment (possibly derived from the adaptation requirement which, in turn, is one of the mechanisms of self-preservation and self-defense).

One more definition I came across was “Intelligence is the ability to act purposefully in unknown situations“, which is not strictly correct. If the situation is unknown, it might be not intelligence, but instincts defining actions. If we omit the ambiguity of “unknown situation”, the definition seems good: if “unknown situation” has long enough time duration, and the tested (observed) being is able to find the dependencies between the elements of the previously unknown environment, and use those dependencies to the benefits of the being, then that being is intelligent.

In my opinion, one of the best definitions would be: “Intelligence is the possession of a model of reality and the ability to use this model to conceive and plan actions and to predict their outcomes. The higher the complexity and precision of the model, the plans, and the predictions, and the less time needed, the higher is the intelligence” (taken from transition). This definition should be enhanced with “the ability of intelligent being to acquire the model of reality”, because otherwise intelligence is incomplete. This definition appears correct. The only problem is that it also does not allow the quantification of intelligence.

The above definition is also what is described in wikipedia as Strong AI: “Strong AI [approach] supposes that it is possible for machines to become sapient, or self-aware, but may or may not exhibit human-like thought processes”. Strong AI is opposed to Weak AI, the proponents of which consider the creation of self-aware and reasoning machines impossible, and leave only minor functions to AI.

The AI is sometimes associated with the so-called Singularity, which implies that the creation of AI (evidently, Strong AI) will lead to the loss by humanity of its leading and defining role. As a result, AI will in some way supersede mankind, and, eventually, mankind may be isolated to degrade, or destroyed by AI.

In the light of permanent improvements in AI-related fields (such as various recognition tasks) the so-called AI Effect: “The AI effect is a term for the tendency for individuals to discount advances in artificial intelligence after the fact”.

Thus, “Artificial Intelligence is the ability of a [human-]created entity to acquire the model of reality based on observations and interactions with reality. Artificial Intelligence possesses the created models of reality, and is able to use the models to conceive and plan actions and to predict their outcomes. The higher the complexity and precision of the models, the plans, and the predictions, and the less time needed, the higher is the strength of the Artificial Intelligence“.

In this post, some definitions and examples are given. This is an introductory text.

First of all there is a need to explain why “intelligent” is in braces in the title. Well, it’s simple: whatever the agents are at the moment of writing, they are just specific, narrow algorithms with no signs of intelligence. As soon as I come across the evidence of the contrary, I will happily remove the braces around “intelligent”. But for now – braces stay.

What is an agent? According to the numerous sources I checked, agent is an entity with some characteristic features. These fundamental features are:

• agent acts on behalf of others. For example, you may hire a person who would attend parents’ meetings at the school of your children: in this case, that would be a “parental agent” , who comes to the meetings on your behalf.
• agents are to some extent autonomous (i.e., enjoy some degree of autonomy). In our example, the “parental agent” has freedom to act and respond the way he feels appropriate in his communication with other parents at the meeting; but at the same time he must follow the behaviour strategy you outlined for the official messages announced by the school staff.
• agents are proactive and reactive. Proactive means that agent may exhibit his own independent initiative, which is not (at least directly) related to the delegated tasks of the agent. Reactive tells us that agents will respond to stimuli – e.g., given a task, agent will try to do that task.
• agents are able to learn – that is, they have memory, which influences their further actions.
• agents may be cooperative – help each other or just join efforts to complete given tasks.
• agents are mobile.

Wikipedia (3) gives a more specific, but also seemingly narrow list of agent features.

Software agents (or just “agents”) are software systems that loosely conform to the above definition and features list. The main idea behind agents is to make computers active in human-computer communication. If we take a look at how we use computers nowadays, that could be best described as “point-and-click”: we do everything ourselves, only rarely using some specific, narrow-use tools for separate small operations (as a rule, computationally intensive). Simple example can be any wordprocessor: imagine you typed your article for submission to some journal, and then the journal decides to change the rules for manuscript submission. You get the example of a new format, and see that it is just differently formatted, with the order of parts changed. Now, to fix this, you “point-and-click” to move the parts and change formatting. If there was an agent, you could ask it to make your article conform to the new sample – and that would be all.

Probably this is where the notion of “interface agents” appeared. It has nothing more behind than just the ability of any untrained person to interact with computer systems (software systems). I suppose this is a legacy entity from the times of command-line-only or even punch-card interfaces, and should not be further considered. The only point to be said here is that intelligent user interfaces are expected to be adaptive – that is, in the course of time of interacting with the user, change themselves for the maximal efficiency of mutual operation. As for me, this boils down to actually some “intelligence” of the agent, and is worth being separated into a distinct field as long as it needs to be manually programmed.

Coming back to agents, there are other reasons for their importance and necessity in them. The spell of “information overload” comes in handy here – agents could help here, as well, by filtering data based on the individual preferences they remember about their user (or should it be Master? ).

Somewhat overlapping is also the process of overall computerization, which means no more hand-work, but lots of typing and clicking instead. Speech recognition and synthesis are available, but they do not yet enable us to talk to computers when we want something a bit more complicated than “printing a page” to be done.

The above-mentioned agent cooperation can be sub-divided into three smaller entities, such as communication, coordination and agreement negotiation. Being capable of communicating grants ability to exchange information; coordination enables tasks sharing, and negotiation helps resolve inconsistencies, which may arise due to different factors (unreliable or controversial data, conflicting task results etc).

As for the mobility feature from the list above, I see no clear picture. Thus I’ll provide some definitions and will let you decide whether an internet worm (computer worm) is a mobile agent or not:
A mobile agent is a software entity which exists in a software environment. It inherits some of the characteristics of an Agent. A mobile agent must contain all of the following models: an agent model, a life-cycle model, a computational model, a security model, a communication model and finally a navigation model.
A mobile agent environment is a software system which is distributed over a network of heterogeneous computers. Its primary task is to provide an environment in which mobile agents can execute. The mobile agent environment implements the majority of the models which appear in the mobile agent definition. It may also provide: support services which relate to the mobile agent environment itself, support services pertaining to the environments on which the mobile agent environment is built, services to support access to other mobile agent systems, and finally support for openness when accessing non-agent-based software environments. (Source: reference 1)

In reality, a discrete number of practically applied agents (or simply bots) can be distinguished (2-4):

1. Shop bots, or buyer agents. These are systems, which crawl internet shops comparing prices for items, and presenting that combined info to the user. These systems are highly popular and widely used nowadays. Amazon is one of these – when you look up something, you get offers on “both new and used”, and these are found by Amazon shopbot. Another feature you already know of is the ability of the shopbot to provide personalized recommendations – e.g., if you buy a book in statistics, you’ll get some more recommendations on statistics books; this eases the task of browsing the catalog, and lets you focus on what matters for you. Shopbot can also be a standalone program – downloadable to your computer, and acting as a simple search utility. Refer to Agentland for samples.
2. Personal/assistant agents. These are actually “personal secretaries” or actually assistants, which combine the abilities and features of other bots listed here. These are expected to be rather versatile tools, being the closest to what might be considered “intelligent” in my opinion. Personal assistant could get a spoken inquiry from you, then search the internet with a number of keywords and phrases (generating and improving search phrases in the process), and finally prepare a report for you on the subject you requested. Personal assistant could also remind you of the forthcoming events, keep track of important notes, records and dates, check you mail – and check if it’s spam or not, as well. What is also important, personall assistant should be capable of conducting at least somewhat intelligent chat with you on a variety of topics. Based on my experience, there are currently no personal assistants which would be close to all the listed criteria (wikipedia adds some more to the list, e.g. personal assistant as an opponent in a multiplayer computer game). One of the earliest and good-looking attempts was Ultra Hal Assistant 5.0 by Zabaware; it was free at that time, now it’s version 6 which is no longer free, and has some improvements over old version, but is still more of a toy to me than a personal assistant.
3. Monitoring agents. These check some finite sources of information for changes and report when changes surpass some arbitrary threshold. These agents can monitor pretty much anything – websites, computer networks, hardware conditions etc. Heed the warning: some monitorig agents are actually spyware/keyloggers/remote shells/remote administrators; there is always a risk of being watched by installing this kind of software to watch others.
4. Data miners, and search agents. These gather data from a multitude of sources to give you the single (hopefully the most relevant) search results set. These can also classify data, and summarize large chunks of data into small abstracts.

I also came across “web/browsing agents” (which are just tools for browsing, not agents), “webmastering agents” (which are either data miners and monitoring agents or just bulk-submit tools), “game agents/AI gaming” (which look to me no different than any GameAI), and “artificial life” agents (which might be both “artificial” and “life” but not agents, as they do not fit the simplest definition).

There is a good discussion of how agents differ from usual computer programs and how agents can be generally classified here.

Instead of the conclusion: The truth is out there, and the “agents” as well . Utility-level “agents” are slowly developing, but I wonder if they can algorithmically supercede the limit of just being a piece of software programmed for a finite set of (even complicated) actions; I think this is not going to happen, however. There are some efforts involving more flexible approaches, which might survive and evolve into quasi-intelligent agents. Personally, I expect that some different approach to agents creation will give the best results, just because practice proves that current approaches are not good.

I intend to study the field of (intelligent) agents and software systems in more details during the forthcoming months; as time permits, I’ll put processed data into this blog. Stay tuned.

Some references:

1. Software Agents: A review. 27 May 1997 (local copy)
2. intelligent agents and bots – agentland
3. http://en.wikipedia.org/wiki/Intelligent_agent
4. Haag, Stephen. “Management Information Systems for the Information Age”, 2006.
5. BotSpot
6. AAAI Agents